A virtual overhead-based load adaptive filtering type vector retrieval method

Abstract

The application discloses a load self-adaptive filtering type vector retrieval method based on virtual overhead, comprising the following steps: S1, constructing a unified virtual overhead model to obtain a single node virtual overhead calculation formula; S2, estimating query local selectivity in real time by first-order neighbor sampling based on a graph-based index structure to obtain local selectivity parameters of a current retrieval task; S3, calculating total virtual overhead of a current traversed node in real time; S4, determining a virtual overhead boundary meeting a target recall rate; S5, completely traversing the first-order neighbor by adopting a mode of first distance calculation and then attribute filtering; S6, performing dynamic expansion of the second-order neighbor by adopting a mode of first attribute filtering and then distance calculation; S7, when the total virtual overhead reaches or exceeds the virtual overhead boundary, terminating expansion of the second-order neighbor of a current path; and S8, outputting Top-K retrieval results meeting attribute filtering conditions and having the highest similarity with a query vector after the traversal is completed. The application has the advantages of simple realization, strong compatibility and easy engineering landing.

Description

Technical Field

[0001] This invention relates to the field of vector retrieval technology, and more specifically, to a load-adaptive filtering vector retrieval method based on virtual overhead. Background Technology

[0002] Vector retrieval technology aims to study how to efficiently find data points that are close to a target vector from massive high-dimensional vector datasets. In recent years, graph-based Approximate Nearest Neighbor (ANN) search methods have significantly improved retrieval efficiency while maintaining high recall rates, and have been widely applied in scenarios such as recommender systems, retrieval-augmented generation (RAG), and agent memory management.

[0003] However, with the continuous development of practical application needs, vector retrieval often relies not only on vector similarity but also on structured attribute constraints, making attribute-constrained filtered vector retrieval (Filtered ANN) an important research direction. Compared to pure vector retrieval, this type of vector retrieval with filtering conditions needs to handle vector similarity calculation and attribute filtering operations simultaneously, making the system face more complex performance trade-offs during retrieval execution.

[0004] Specifically, during the retrieval process, the computational requirements for query vectors and predicates differ for different retrieval tasks, and the system load exhibits complex and dynamic characteristics, leading to significant differences in retrieval efficiency under different load conditions. This is mainly reflected in the following three aspects: (1) Dynamic changes in the selectivity of query predicates: In actual retrieval, the selectivity differences of different query predicates will affect the data distribution of effective candidate nodes, resulting in the proportion of nodes that meet the filtering conditions (selectivity) fluctuating greatly between different query predicates, thus making the candidate space size unstable.

[0005] (2) Differences in attribute filtering overhead: Different types of predicate attribute constraints bring drastically different computational costs. For example, simple numerical comparison has very little overhead, while complex string regular expression matching or multi-condition combination judgment will bring huge filtering overhead.

[0006] (3) Heterogeneity of vector distance calculation overhead: Due to the influence of the dimension size, data type and distance measurement method (such as L2 distance, cosine similarity, inner product, etc.) of high-dimensional vectors, the actual overhead of vector distance calculation varies under different loads.

[0007] However, current mainstream methods for handling filtered vector retrieval typically employ fixed, static search strategies, failing to incorporate the aforementioned dynamic load characteristics into the unified consideration of the search process. These methods are often designed based on a specific assumed load, but in practical applications, they struggle to adapt to these dynamic load characteristics when the load changes, leading to significant fluctuations in retrieval performance. For example, in scenarios with extremely high filtering overhead and very low selectivity, static strategies may waste costly vector distance calculations on subsequently eliminated invalid nodes, thus reducing retrieval efficiency and result quality. Conversely, in scenarios requiring high recall and dynamically fluctuating selectivity, rigid filtering and expansion mechanisms are prone to causing graph connectivity breakdowns, and the additional multi-hop neighbor traversal mechanism introduced to compensate for these breakdowns leads to a surge in invalid overhead.

[0008] Furthermore, existing methods generally lack a unified modeling mechanism for the "computational cost - actual benefit" relationship during the retrieval process, making it impossible to dynamically adjust search behavior based on the specific characteristics of the current retrieval process. Given the real-time changes in the ratio of attribute filtering cost to vector distance calculation cost, and the query attribute predicate selection rate, existing static strategies struggle to achieve adaptive adjustment, making it difficult for the system to achieve a stable and optimal balance between search efficiency and retrieval accuracy under different load conditions. Summary of the Invention

[0009] The purpose of this invention is to provide a load-adaptive filtering vector retrieval method based on virtual overhead, so as to overcome the defects of the existing technology.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A load-adaptive filtering vector retrieval method based on virtual overhead includes the following steps: S1. Construct a unified virtual overhead model, and use a unified measurement and modeling method to measure and model the attribute filtering overhead and vector distance calculation overhead, so as to obtain the single-node virtual overhead calculation formula. S2. Based on the virtual cost model in step S1, the local selectivity of the query is estimated in real time by sampling the first-order neighbors of the graph-based index structure to obtain the local selectivity parameters of the current retrieval task. S3. Substitute the local selectivity parameters obtained in step S2 into the single-node virtual cost calculation formula in step S1 to calculate the total virtual cost of the currently traversed nodes in real time. S4. Determine the Virtual Overhead Boundary (VOB) that satisfies the target recall rate through offline simulation load and performance analysis; S5. In graph-based index retrieval, first-order neighbors are fully traversed by first calculating distance and then filtering by attributes. S6. Based on the total virtual cost in step S3 and the virtual cost boundary (VOB) in step S4, perform dynamic expansion on the second-order neighbors by first filtering attributes and then calculating distance. S7. When the total virtual cost of step S3 reaches or exceeds the virtual cost boundary VOB of step S4, the expansion of the second-order neighbor of the current path is immediately terminated. S8. After the traversal is complete, output the Top-K search results that meet the attribute filtering conditions and have the highest similarity to the query vector.

[0011] Furthermore, the construction steps of the unified virtual overhead model in step S1 are as follows: the single attribute filtering overhead and the single vector distance calculation overhead are incorporated into the unified metric space, and the distance calculation overhead is weighted locally selectively to form a unified representation of the comprehensive computational cost of a single node.

[0012] Furthermore, the formula for the single-node virtual overhead constructed in step S1 is as follows: ,in The single-node virtual cost for node u. The average time cost of performing a single attribute predicate filter, The average time cost of performing a single vector distance calculation, This represents the percentage of nodes within the current search area that meet the filtering criteria.

[0013] Furthermore, the real-time estimation of local selectivity in step S2 specifically involves: using the first-order neighbors in the graph-based index search process as a sampling set, statistically analyzing the proportion of nodes that satisfy the query predicate conditions, and obtaining the local selectivity of the current search region.

[0014] Furthermore, step S3, calculating the total virtual cost, specifically involves summing the virtual costs of each traversed node during the retrieval process to obtain the total virtual cost. , where N is the total number of neighbor nodes that have been traversed in the current search.

[0015] Furthermore, step S4, determining the Virtual Overhead Boundary (VOB), specifically involves: S4.1 Generate a simulated query set by constructing artificial query vectors from sampled vectors in the original dataset; S4.2 Measure the atomic overhead of the current environment in real time to obtain the distance calculation overhead and attribute filtering overhead; S4.3 Construct multiple sets of query loads with different selectivity to generate accurate benchmark results; S4.4 Under a fixed recall target, traverse the search to obtain the optimal virtual overhead value for each load throughput; S4.5. Take the average of the optimal virtual overhead values ​​for multiple scenarios to obtain the Virtual Overhead Boundary (VOB).

[0016] Furthermore, the complete traversal of first-order neighbors in step S5 specifically involves: forcing a full traversal of the first-order neighbors of the current node, first performing vector distance calculation and adding them to the candidate set, and then performing attribute filtering on the nodes in the candidate set.

[0017] Furthermore, the dynamic expansion of second-order neighbors in step S6 specifically involves: first, performing attribute filtering judgment on second-order neighbors, and only performing vector distance calculation on nodes that meet the filtering conditions, while skipping nodes that do not meet the conditions.

[0018] Furthermore, the termination of second-order neighbor expansion in step S7 specifically means: when the total virtual cost is greater than or equal to the virtual cost boundary (VOB), the second-order neighbor traversal of the current path is immediately stopped, and no new nodes are visited.

[0019] Furthermore, steps S1 to S7 are all executed on the existing HNSW graph index structure, with only the addition of virtual cost calculation, local selective estimation and virtual cost boundary judgment logic during the search phase, without modifying the underlying index structure.

[0020] Compared with the prior art, the advantages of the present invention are as follows: This invention introduces virtual overhead as a unified metric to model attribute filtering overhead and distance calculation overhead in a unified manner, avoiding the inconsistency problem caused by optimizing the two types of operations separately in the prior art. This enables unified scheduling of overall computing resources under the same framework and load, thereby improving system throughput and response stability.

[0021] This invention achieves dynamic estimation of query selectivity based on the local sampling mechanism of first-order neighbors, and then estimates the average virtual cost of a single node. It does not rely on prior data distribution or historical query statistics, and can adaptively perceive the current query load characteristics during operation, thereby overcoming the problems of existing methods being sensitive to data distribution and having insufficient generalization ability.

[0022] This invention uses virtual overhead boundaries as dynamic constraints for the search process, replacing the control methods that rely on fixed search depth or fixed neighbor expansion numbers in traditional methods. This allows the search process to adjust the expansion range in real time according to the actual computational overhead, thereby maintaining superior performance in different query scenarios.

[0023] This invention divides the neighbors of a graph index (such as HNSW) into first-order neighbors and second-order neighbors, and adopts different processing strategies for each. During the expansion of second-order neighbors, a virtual overhead accumulation and truncation mechanism is introduced. Under the premise of ensuring search connectivity and result quality, invalid searches can be terminated in a timely manner in scenarios with high filtering overhead or low selectivity, which significantly reduces redundant calculations and improves retrieval efficiency.

[0024] In summary, this invention achieves the above optimizations by adjusting the execution logic of the search phase without increasing system complexity, effectively improving the performance of vector hybrid retrieval with filtering conditions. It has the advantages of simple implementation, strong compatibility, and easy engineering implementation, and has good practical value and promotion prospects. It can be directly applied to existing vector databases or approximate nearest neighbor retrieval systems. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the load-adaptive filtering vector retrieval method based on virtual overhead according to the present invention; Figure 2 This is a graph showing the change in query throughput performance with virtual overhead under different datasets and different selectivity conditions when the retrieval recall rate is kept at a certain level.

[0027] Figure 3 This is a schematic diagram of the nearest neighbor graph index HNSW and an architecture diagram of the filtering vector retrieval system in this invention. Detailed Implementation

[0028] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0029] This invention addresses filtering vector retrieval tasks. In such tasks, the system not only needs to find candidate results similar to the query vector in a high-dimensional vector space, but also requires these candidate results to meet specific attribute filtering conditions. Since attribute filtering and vector distance calculation are two different types of operations, their computational costs often differ significantly. In real-world applications, the query load exhibits obvious dynamic characteristics, specifically: the complexity of attribute filtering varies greatly between different queries (leading to significant changes in filtering costs), the cost of vector distance calculation fluctuates with changes in data dimension and calculation method, and the proportion of data meeting the filtering conditions (i.e., selectivity) varies considerably. These factors collectively cause the cost structure of various operations during query execution to constantly change, making it difficult for traditional retrieval methods relying on fixed strategies or static parameter configurations to achieve consistently stable and efficient performance.

[0030] To address the aforementioned problems, this invention proposes a load-adaptive filtering vector retrieval method based on Virtual Overhead (VO). This method unifies the modeling of attribute filtering overhead and distance calculation overhead to construct a unified virtual overhead metric model. Furthermore, it introduces a Virtual Overhead Boundary (VOB) as a dynamic constraint during the search process, thereby achieving dynamic control of the search process while ensuring retrieval accuracy.

[0031] Specifically, this invention does not rely on pre-known query tasks or load data characteristics, nor does it depend on a fixed neighbor processing strategy. During the traversal of a graph-based index structure (such as HNSW), neighbor nodes are divided into different levels, and differentiated processing methods are applied to different levels. Simultaneously, by calculating the cost of the current query process in real time and comparing it with a preset virtual cost boundary, the invention dynamically determines whether to continue expanding neighbor nodes, thereby achieving adaptive allocation and optimization of computing resources under different load conditions.

[0032] In the process of vector approximate nearest neighbor retrieval with filtering conditions, the overall query cost mainly consists of two parts: the cost of performing attribute filtering on candidate vectors and the cost of calculating the similarity distance between candidate vectors. Empirically, under the premise of a fixed recall rate, the query throughput may exhibit a non-monotonic change with the search cost of a single node, which makes it theoretically challenging to derive the optimal search cost for each neighbor traversal. To address this issue, this embodiment proposes the following assumption: an efficient filtered approximate nearest neighbor (ANN) query should control the cost of each neighbor traversal within a certain boundary range, and a good balance can be achieved between search cost and search completeness.

[0033] Based on the above analysis, this embodiment conducts experiments on six datasets under different query selectivity conditions. The characteristic results of the six workloads are shown in Table 1 below: Table 1

[0034] By analyzing the relationship between virtual overhead and query performance, the above hypothesis was verified. An important pattern was observed: when the retrieval recall (e.g., Recall@K = 90%) remains at a certain level, under different datasets and selectivity conditions, the system's query throughput performance exhibits a clear unimodal characteristic with variations in virtual overhead, and the overhead value corresponding to this optimum is highly stable. This indicates that virtual overhead can serve as an effective performance tuning metric to uniformly characterize the combined cost of filtering operations and distance calculations. Detailed experimental results are as follows... Figure 2 As shown, for each workload, regardless of how the predicate selectivity (Sp) changes, query throughput typically peaks around a certain virtual overhead.

[0035] See Figure 1 As shown, this embodiment discloses a load-adaptive filtering vector retrieval method based on virtual overhead, including the following steps: Step S1: Construct a unified virtual overhead model, and use a unified measurement model to model the attribute filtering overhead and vector distance calculation overhead to obtain the single-node virtual overhead calculation formula.

[0036] In this embodiment, the overhead of single-time attribute filtering and single-time vector distance calculation are incorporated into a unified metric space, and the distance calculation overhead is weighted locally selectively to form a unified representation of the comprehensive computational cost of a single node.

[0037] This is used to uniformly characterize the comprehensive computational cost introduced by a single candidate node during the search process. Specifically, for any candidate node u in the retrieval process, a single-node virtual overhead is defined. The virtual cost of a single node is the sum of the inherent cost of performing attribute filtering and the cost of distance computation, where the expected cost of distance computation is determined by local selectivity, meaning that distance computation is performed only if a node meets the filtering criteria. Therefore, the virtual cost of a single node can be expressed as the weighted sum of the attribute filtering cost and the distance computation cost multiplied by local selectivity.

[0038]

[0039] in: The average time cost for performing a single attribute predicate filter.

[0040] The average time cost for performing a single vector distance calculation.

[0041] This refers to the local selectivity of the predicate, which is the percentage of nodes in the current search area that meet the filtering conditions.

[0042] Step S2: Based on the virtual cost model in step S1, the local selectivity of the query is estimated in real time by sampling the first-order neighbors of the graph-based index structure (such as HNSW graph), and the local selectivity parameters of the current retrieval task are obtained.

[0043] In this embodiment, the real-time estimation of local selectivity specifically involves using the first-order neighbors in the search process of a graph-based index (such as an HNSW graph index) as a sampling set, statistically analyzing the proportion of nodes that satisfy the query predicate conditions, and obtaining the local selectivity of the current search region.

[0044] Step S3: Substitute the local selectivity parameter obtained in step S2 into the single-node virtual cost calculation formula in step S1 to calculate the total virtual cost of the currently traversed nodes in real time.

[0045] The total virtual cost is calculated as follows: Based on the virtual cost of a single node, the virtual cost is further defined. Throughout the search path, the system accumulates the virtual costs of all visited nodes (including first-order and second-order neighbors) to obtain the virtual cost of the current search task. , where N is the total number of neighbor nodes that have been traversed in the current search, and this metric reflects the real-time status of computing resources for the current search path.

[0046] Step S4: Determine the Virtual Overhead Boundary (VOB) that satisfies the target recall rate through offline simulation load and performance analysis.

[0047] In this embodiment, the Virtual Overhead Boundary (VOB) is a key control parameter used to constrain the computational cost of the neighbor traversal process. It characterizes the cost boundary at which query throughput performance reaches its optimal level while satisfying the target recall rate. It provides a unified and stable termination criterion for the search process; when the system's virtual overhead value reaches or exceeds the preset virtual overhead boundary, the current search is considered to have reached the optimal cost range, and further expansion should cease, thus avoiding unnecessary computational waste. The virtual overhead boundary does not depend on specific data distributions or query selectivity, but rather serves as a stable optimization target across different scenarios, thereby significantly improving the system's adaptability to complex load environments.

[0048] The virtual overhead boundary can be obtained through offline performance analysis or online estimation. Specifically, it includes: constructing multiple representative workloads under different predicate selectivity conditions, establishing a functional relationship between query performance and virtual overhead, and determining the virtual overhead value that makes the query throughput reach its peak or close to its peak as the virtual overhead boundary. The detailed estimation algorithm includes steps S4.1-S4.5.

[0049] In practical applications, since query load and data distribution are usually unknown, it is impossible to directly obtain accurate attribute filtering overhead, distance calculation overhead, and query selectivity. Therefore, in order to obtain the optimal virtual overhead boundary value in advance when the actual online workload is unknown, this embodiment further proposes a virtual overhead estimation method based on online sampling and statistical analysis.

[0050] Step S4.1: Generate a simulated query set by constructing artificial query vectors from sampled vectors in the original dataset. This step involves generating artificial simulated load. The system first constructs representative simulated query requests from the original dataset to ensure that the profiling process does not depend on real external loads. Specifically, during system initialization, multiple tests are performed on different types of filtering conditions to obtain the average filtering time as an estimate. The specific query request generation includes: ① Simulated Vector Generation: Randomly sample triple vectors (v1, v2, v3) from the original vector dataset, calculate their average value, and normalize them to generate artificial query vectors. This method can simulate query hotspots that are located within the original data distribution space but do not completely overlap.

[0051] ② Simulated Attribute and Predicate Generation: The system randomly generates sets of integer attributes with different cardinality. By adjusting the maximum value range of the attributes (e.g., {2, 4, 8, ..., 64}), simulated query scenarios covering high selectivity to extremely low selectivity (e.g., 1 / 2 to 1 / 64) are artificially constructed.

[0052] Step S4.2: Measure the atomic overhead of the current environment in real time to obtain the distance calculation overhead and attribute filtering overhead.

[0053] In this embodiment, since the cost ratio of distance calculation to filtering operations varies across different hardware platforms, the nanoseconds for a single vector distance calculation and the nanoseconds for a single attribute filtering operation are statistically analyzed. For example, the cost of distance calculation is affected by the vector dimension and the calculation method, so it can be obtained by measuring the average time taken per unit distance calculation operation.

[0054] Meanwhile, a binary search method is used to find suitable simulation parameters so that the filtering delay in the simulation environment is consistent with the filtering cost in the real system, ensuring that the virtual overhead calculated later has actual physical meaning.

[0055] Step S4.3: Construct multiple sets of query loads with different selectivity to generate accurate benchmark results.

[0056] In this embodiment, for each set of simulated vectors and simulated predicates, a precise Top-K result set is generated using a full linear search, which serves as the benchmark for calculating recall, thus generating benchmark results (Golden Results).

[0057] Step S4.4: Under the fixed recall target, traverse and search to obtain the virtual overhead value with the optimal throughput for each load.

[0058] In this embodiment, the parameter search for a fixed recall target (e.g., Recall@10 = 90%) finds the critical point that just meets the target recall rate by dynamically adjusting the retrieval parameters (e.g., efSearch) under different selectivity conditions (Sp).

[0059] At the critical point where recall is satisfied, record the system's query time (T) and the total virtual overhead (VOsum) at that point. Core logic: Find the VOsum that minimizes the single query time T (i.e., the peak throughput).

[0060]

[0061] Step S4.5: Take the average of the optimal virtual overhead values ​​for multiple scenarios to obtain the Virtual Overhead Boundary (VOB).

[0062] In this embodiment, all artificially constructed selective scenarios are traversed, and the "best VirtualOverhead value" under each scenario is collected. Finally, by taking the average value, an average virtual overhead boundary value (Avg Virtual Overhead Boundary) that spans different selectivity and has universality is obtained.

[0063] The specific algorithm is shown in Table 2.

[0064] Table 2

[0065] Step S5: In the graph-based index (HNSW graph) retrieval, the first-order neighbors are fully traversed by first calculating the distance and then filtering the attributes.

[0066] In this embodiment, as Figure 3 As shown, in the HNSW graph search process, for each visited candidate node, its first-order neighbors are first traversed. For first-order neighbors, this embodiment adopts a "distance calculation followed by filtering" approach, that is, first calculate the distance between the neighbor and the query vector and add it to the candidate set, and then only perform attribute filtering on nodes that are likely to enter the final result set. This strategy can ensure sufficient expansion of candidate nodes during the search process, thereby maintaining the connectivity and stability of the graph search.

[0067] Step S6: Based on the total virtual cost in step S3 and the virtual cost boundary (VOB) in step S4, perform dynamic expansion on the second-order neighbors by first filtering attributes and then calculating distance.

[0068] In this embodiment, after processing the first-order neighbors, the system further expands its second-order neighbors. During this process, this embodiment adopts a "filter-then-calculate" approach, that is, it first performs attribute filtering on the second-order neighbors, and only calculates the distance if a node meets the filtering conditions, thereby effectively reducing unnecessary computational overhead. However, unlike traditional methods, this embodiment does not unconditionally traverse all second-order neighbors, but instead introduces virtual overhead boundaries as constraints.

[0069] This embodiment, through the mechanism of steps S5 and S6, implements a hybrid retrieval strategy of "forced traversal of first-order neighbors and controlled expansion of second-order neighbors." On the one hand, the complete traversal of first-order neighbors ensures the connectivity of the search space and the quality of the retrieval results; on the other hand, the controlled expansion of second-order neighbors effectively limits the computational overhead and avoids performance degradation in scenarios with low selectivity or high filtering overhead. A key feature of this method is that it does not rely on fixed parameters (such as the number of neighbors traversed or the search depth), but dynamically determines the search range based on the virtual overhead calculated in real time. This allows for automatic adjustment of the strategy under different query conditions, achieving true load adaptability.

[0070] Step S7: When the total virtual cost of step S3 reaches or exceeds the virtual cost boundary (VOB) of step S4, immediately terminate the expansion of the second-order neighbor of the current path.

[0071] When visiting a second-order neighbor, the system updates the cumulative virtual cost based on the currently estimated virtual cost per node and checks if this cumulative value exceeds a preset boundary. If it does, the expansion of second-order neighbors is immediately stopped, thus achieving dynamic control over the search depth. The cumulative virtual cost is calculated as follows:

[0072] ①If VOtotal < VOB, continue the second-order neighbor retrieval with "filtering first, then distance calculation".

[0073] ② If VOtotal ≥ VOB, immediately stop neighbor expansion for the current path.

[0074] In this embodiment, steps S1 to S7 are all executed on the existing HNSW graph index structure. Only the virtual overhead calculation, local selectivity estimation and virtual overhead boundary judgment logic are added in the search phase, without modifying the underlying index structure.

[0075] Step S8: After the traversal is completed, output the Top-K search results that meet the attribute filtering conditions and have the highest similarity to the query vector.

[0076] Through the above technical solution, the present invention can achieve stable and efficient retrieval performance under various complex query loads. Specifically, in scenarios with low filtering overhead and high selectivity, the system can automatically increase the search depth to improve the recall rate; while in scenarios with high filtering overhead or low selectivity, the system can terminate the search in a timely manner to avoid unnecessary computational overhead, thereby improving the overall throughput performance.

[0077] In terms of system implementation, this invention, without relying on complex index structure modifications or altering existing graph index structures, only requires adding virtual overhead calculation and judgment logic during the search phase to adaptively control the neighbor traversal process in the graph index. This results in low implementation complexity and high engineering feasibility. Furthermore, this method is independent of specific data distributions or application scenarios, making it widely applicable to various vector database systems and demonstrating good versatility and scalability.

[0078] In summary, by introducing virtual overhead modeling and constraint mechanisms, this invention achieves dynamic optimization of the near nearest neighbor retrieval process with filtering conditions under different data distributions and query loads. This ensures retrieval quality while significantly improving system performance, and has important practical application value.

[0079] This invention is applicable to various vector retrieval scenarios with attribute filtering conditions, including but not limited to: vector database retrieval, recommendation system retrieval, and retrieval enhancement generation. Under different data distributions and query loads, this invention can achieve adaptive control of the search process.

[0080] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, the patent owner may make various modifications or alterations within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the present invention, they shall be within the protection scope of the present invention.

Claims

1. A virtual overhead based load adaptive filtering type vector retrieval method, characterized by, Includes the following steps: S1. Construct a unified virtual overhead model, and use a unified measurement and modeling method to measure and model the attribute filtering overhead and vector distance calculation overhead, so as to obtain the single-node virtual overhead calculation formula. S2. Based on the virtual cost model in step S1, the local selectivity of the query is estimated in real time by sampling the first-order neighbors of the graph-based index structure to obtain the local selectivity parameters of the current retrieval task. S3. Substitute the local selectivity parameters obtained in step S2 into the single-node virtual cost calculation formula in step S1 to calculate the total virtual cost of the currently traversed nodes in real time. S4. Determine the Virtual Overhead Boundary (VOB) that satisfies the target recall rate through offline simulation load and performance analysis; S5. In graph-based index retrieval, first-order neighbors are fully traversed by first calculating distance and then filtering by attributes. S6. Based on the total virtual cost in step S3 and the virtual cost boundary (VOB) in step S4, perform dynamic expansion on the second-order neighbors by first filtering attributes and then calculating distance. S7. When the total virtual cost of step S3 reaches or exceeds the virtual cost boundary VOB of step S4, the expansion of the second-order neighbor of the current path is immediately terminated. S8. After the traversal is complete, output the Top-K search results that meet the attribute filtering conditions and have the highest similarity to the query vector. 2.The virtual overhead based load adaptive filtering type vector search method according to claim 1, wherein, The steps for constructing the unified virtual overhead model in step S1 are as follows: the single attribute filtering overhead and the single vector distance calculation overhead are incorporated into the unified metric space, and the distance calculation overhead is weighted locally selectively to form a unified representation of the comprehensive computational cost of a single node.

3. The virtual overhead based load adaptive filtering type vector retrieval method according to claim 2, wherein, The single-node virtual overhead formula constructed by the step S1 is: wherein is the single-node virtual overhead of the node u, is the average time overhead of performing a single attribute predicate filtering, is the average time overhead of performing a single vector distance calculation, is the proportion of nodes in the current search area that satisfy the filtering condition.

4. The virtual overhead based load adaptive filtering type vector retrieval method according to claim 1, wherein, The real-time estimation of local selectivity in step S2 specifically involves using the first-order neighbors in the graph-based index search process as a sampling set, statistically analyzing the proportion of nodes that satisfy the query predicate conditions, and obtaining the local selectivity of the current search region.

5. The virtual overhead based load adaptive filtering type vector retrieval method according to claim 1, wherein, The step S3 calculates the total virtual overhead, specifically, the single-node virtual overheads of all the traversed nodes in the retrieval process are accumulated to obtain the total virtual overhead where N is the total number of the traversed neighbor nodes in the current retrieval. 6.The virtual overhead based load adaptive filtering type vector search method according to claim 1, wherein, The specific steps of determining the Virtual Overhead Boundary (VOB) in step S4 are as follows: S4.1 Generate a simulated query set by constructing artificial query vectors from sampled vectors in the original dataset; S4.2 Measure the atomic overhead of the current environment in real time to obtain the distance calculation overhead and attribute filtering overhead; S4.3 Construct multiple sets of query loads with different selectivity to generate accurate benchmark results; S4.4 Under a fixed recall target, traverse the search to obtain the optimal virtual overhead value for each load throughput; S4.

5. Take the average of the optimal virtual overhead values ​​for multiple scenarios to obtain the Virtual Overhead Boundary (VOB).

7. The virtual overhead based load adaptive filtering type vector retrieval method according to claim 1, wherein, The complete traversal of first-order neighbors in step S5 specifically involves: forcing a full traversal of the first-order neighbors of the current node, first performing vector distance calculation and adding them to the candidate set, and then performing attribute filtering on the nodes in the candidate set. 8.The virtual overhead based load adaptive filtering type vector search method according to claim 1, wherein, The dynamic expansion of second-order neighbors in step S6 is specifically as follows: first, attribute filtering judgment is performed on second-order neighbors, and vector distance calculation is performed only on nodes that meet the filtering conditions, while nodes that do not meet the conditions are skipped directly.

9. The virtual overhead based load adaptive filtering type vector search method according to claim 1, wherein, The termination of second-order neighbor expansion in step S7 specifically means: when the total virtual cost is greater than or equal to the virtual cost boundary (VOB), immediately stop the second-order neighbor traversal of the current path and no longer add new nodes for access.

10. The virtual overhead based load adaptive filtering type vector retrieval method according to claim 1, wherein, Steps S1 to S7 are all executed on the existing HNSW graph index structure. Only the virtual cost calculation, local selective estimation and virtual cost boundary judgment logic are added during the search phase, without modifying the underlying index structure.

Images