Vector searching method under filtering scene based on scalar and vector combined navigation

Abstract

The invention relates to a vector search method in a filtering scene based on scalar and vector combined navigation, which comprises the following steps of: dividing a data set according to a label set associated with each vector, and classifying the vectors with the same label set into one group; constructing a directed acyclic graph, namely a label navigation graph LNG, based on the relationship of the minimum supersets among all the label sets for determining an entrance label set; constructing cross-group edges according to edges in the label navigation graph, constructing a neighbor graph by using an existing ANNS algorithm, and merging to obtain a unified label navigation graph UNG; and obtaining query vectors, directly covering the entry tag set of the query tag set by retrieving, and executing greedy search from the entry vectors to obtain a search result. Compared with the prior art, the method has the advantages that the search efficiency can be improved while the accuracy is ensured, various labels can be supported, and the method is adaptive to various graph-based ANNS algorithms.

Description

Technical Field

[0001] The present invention relates to the field of data search technology, and in particular to a vector search method in a filtering scenario based on scalar and vector joint navigation. Background Art

[0002] With the development of artificial intelligence technology, the importance of unstructured data has become increasingly prominent. Compared with structured data, unstructured data is more diverse, including but not limited to images, videos, audio, and text. Although these data are rich in information, they are quite complex to process. Traditional database technology is only good at processing structured data and is often unable to cope with unstructured data. The emergence of vector search and vector databases has provided a new way to solve this problem by converting unstructured data into vectors and then searching using vector retrieval.

[0003] At present, Approximate Nearest Neighbor Search (ANNS) is widely used in recommendation systems, search engines, and retrieval enhancement generation of large language models. The core idea of ​​ANNS is not limited to returning the most accurate result item, but to search for nearby data items in the vector space, sacrificing accuracy within an acceptable range to improve range retrieval efficiency. After a given query vector, the goal of ANNS is to retrieve vectors with similar distances from a set of high-dimensional vectors. However, the traditional ANNS algorithm only focuses on vector similarity and cannot meet the needs of complex scenarios that also require attribute filtering. In order to address this problem, filtered ANNS allows retrieval of vectors that meet attribute constraints. Specific application scenarios include attribute screening in e-commerce, domain name or user permission filtering in search engines, and academic research to retrieve relevant literature by specific conditions such as publication year or conference venue. That is to say, in order to query vector data and structured data at the same time, label constraints such as attributes and keywords are usually imposed on the search to achieve filtering ANNS, but this method has low search efficiency and it is difficult to guarantee the accuracy of search results. In addition, some existing ANNS algorithms that support label filtering scenarios are severely limited in terms of label types and can only solve very special types of labels (completely equal length and very few types), and cannot adapt to various graph-based search applications. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a vector search method in a filtering scenario based on scalar and vector joint navigation, which can improve the search efficiency while ensuring accuracy and support a variety of tags.

[0005] The object of the present invention can be achieved by the following technical solution: A vector search method in a filtering scenario based on scalar and vector joint navigation, comprising the following steps:

[0006] S1. Divide the dataset according to the label set associated with each vector, and group vectors with the same label set together.

[0007] S2. Based on the minimum superset relationship between all label sets, a directed acyclic graph is constructed, namely, a label navigation graph (LNG), which is used to determine the entry label set;

[0008] S3, constructing cross-group edges according to the edges in the label navigation graph, using the existing ANNS algorithm to construct a neighbor graph, and merging them to obtain a unified label navigation graph (UNG);

[0009] S4. Obtain the query vector, retrieve the entry tag set that directly covers the query tag set, perform greedy search starting from these entry vectors, and obtain search results.

[0010] Furthermore, the specific process of step S1 is as follows:

[0011] Given a vector data set D = {(v 1 ,f 1 ),(v 2 ,f 2 ),…,(v |D| ,f |D| )} includes |D| high-dimensional vectors, each vector v i Uniquely corresponds to a tag set where |f i | represents vector v i The corresponding number of labels, let represents the set of all labels, then the total number of labels in the dataset is |L|;

[0012] In order to avoid iterating over all vectors, the corresponding vector set is calculated for each possible label set in the data set. For each possible label set f, the corresponding vector set V is calculated in the indexing stage. f = {v i ∈V|f i =f}.

[0013] Furthermore, in the label navigation graph constructed in step S2, the directed path from the label set f to f″ is composed of the edges e(f,f′) and e(f′,f″), which means that f″ is the smallest superset of f′, f′ is the smallest superset of f, and all its supersets are all its descendants in the label navigation graph.

[0014] Furthermore, the step S2 specifically includes the following steps:

[0015] S21, for the label set f q , get the corresponding minimum superset;

[0016] S22. Using the prefix tree method, a label navigation graph is constructed by calculating the minimum superset for each label set to be inserted.

[0017] Furthermore, the specific process of step S21 is as follows:

[0018] Get the tag set f q The largest label l max And the corresponding inverted list I(l max ), represents the set of corresponding nodes in the prefix tree;

[0019] For each I(l max ) in the node u, if we traverse all the parents from the prefix tree upwards, and the label set of the parents is f q When there is a superset of u, search deep from u downward to obtain all supersets;

[0020] Finally, for all supersets, all minimal supersets are filtered out according to the definition.

[0021] Furthermore, the specific process of step S22 is: traverse each tag set f in descending order of the tag set size, for each tag set f, calculate all minimum supersets of f, and directly connect edges with f, then parse the tag set f in a prefix tree manner and insert it into the tree, and update the inverted list according to each tag.

[0022] Furthermore, in step S2, if the queried tag set f q If it has appeared in the original dataset, it will be directly used as the entry label set;

[0023] If the query label set f q If it has not appeared in the original data set, all f q The minimal superset of is used as the entry label set.

[0024] Furthermore, the specific process of constructing the cross-group edge in step S3 is as follows: if there is a label navigation graph from the label set f i to f j For the directed edge of the label set f i Each vector v in i , select the label set f j Some vectors v whose distance is less than or equal to the preset threshold j , establish vi to v j These edges are cross-group edges. Cross-group edges are added along the direction of the edges in the label navigation graph, and vectors from different groups are connected using cross-group edges.

[0025] Furthermore, the specific process of merging to obtain a unified label navigation graph in step S3 is: for different label set groups, different threads are allocated to different groups in a parallel manner to construct a neighbor graph, single-thread processing is used in each group, and finally all constructed neighbor graphs are merged to obtain a unified label navigation graph.

[0026] Furthermore, the step S4 specifically includes the following steps:

[0027] S41. For the query vector x q and query tag set f q , first determine its entry label set, randomly select σ vectors from all vectors corresponding to these entry label sets as entry points, and add them to the queue pq;

[0028] S41. When there are still unexplored vectors in the priority queue pq, repeat the following operations:

[0029] Pop the nearest unexplored vector v from pq i , and mark it as explored;

[0030] v i All out-degree neighbors in the unified label navigation are added to pq;

[0031] Keep pq and the query vector v q The first w most recent vectors;

[0032] S43. Return the number of nodes in the priority queue pq relative to the query vector v q The most recent k vectors are used as search results.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] The present invention first divides the data set and groups the label set, and then constructs a label navigation graph to encode the inclusion relationship between the label sets of all vectors. On this basis, a general framework, a unified navigation graph, is proposed to bridge the inclusion relationship of the label set and the vector neighbor relationship. Thus, the label navigation graph is used as a dynamic guidance mechanism. By using the entry label set retrieved by the query label, the ANNS can be strictly constrained to explore in the filtered search space, and a "unified label navigation graph" framework suitable for filtering ANNS is proposed, which can adapt to various graph-based ANNS algorithms, support query labels of any size and distribution, and use the inclusion relationship between label sets to avoid exploring label sets that cannot cover the query label, thereby accelerating the search process, ensuring that only vectors that meet the filtering conditions are searched, providing complete answers, and ensuring versatility, accuracy, completeness and adaptability, while ensuring the accuracy of search results, improving search efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0036] Figure 2 This is a schematic diagram of tag set grouping in the embodiment;

[0037] Figure 3 A schematic diagram of the process of constructing a tag navigation map in an embodiment;

[0038] Figure 4 This is a label navigation diagram in the embodiment;

[0039] Figure 5 Schematic diagram of the cross-group edge in the embodiment;

[0040] Figure 6 Schematic diagram of the greedy search process in the embodiment. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example

[0043] like Figure 1 As shown, a vector search method in a filtering scenario based on scalar and vector joint navigation includes the following steps:

[0044] S1. Divide the dataset according to the label set associated with each vector, and group vectors with the same label set together.

[0045] S2. Based on the minimum superset relationship between all label sets, a directed acyclic graph is constructed, namely, the label navigation graph LNG, which is used to determine the entry label set;

[0046] S3, construct cross-group edges according to the edges in the label navigation graph, use the existing ANNS algorithm to build a neighbor graph, and merge them to obtain a unified label navigation graph UNG;

[0047] S4. Obtain the query vector, retrieve the entry tag set that directly covers the query tag set, perform greedy search starting from these entry vectors, and obtain search results.

[0048] This embodiment applies the above solution, and the main contents are:

[0049] 1. Dataset Partitioning and Label Set Grouping

[0050] Based on the label set associated with each vector, the dataset is partitioned so that vectors with the same label set are grouped together. This grouping operation facilitates efficient management and retrieval of data in subsequent steps.

[0051] like Figure 2 As shown, the known vector data set D = {(v 1 ,f 1 ),(v 2 ,f 2 ),…,(v |D| ,f |D| )} includes |D| high-dimensional vectors. Each vector v i Uniquely corresponds to a tag set where |f i | represents vector v i The corresponding number of labels. represents the set of all labels, then the total number of labels in the dataset is |L|.

[0052] To avoid iterating over all vectors, this solution calculates the corresponding vector set for each possible label set in the dataset. Specifically, for each possible label set f, in the indexing phase, the corresponding vector set V is calculated. f = {v i ∈V|f i =f}.

[0053] 2. Build a label navigation map LNG

[0054] Consider that if the label set f covers all query labels f q , then any superset of f will also cover the label set f q Therefore, this scheme uses the inclusion relationship between different label sets in the entire label set to ensure that only a small number of label sets are needed as entry points to derive all possible answer spaces after filtering.

[0055] This scheme is based on the relationship between the smallest supersets of all possible label sets, which together form a directed acyclic graph (DAG) and are defined as a label navigation graph (LNG). The label navigation graph retains the transitive property of the superset relationship. Specifically, the directed path from f to f″ in LNG consists of the edges e(f,f′) and e(f′,f″), that is, f″ is the smallest superset of f′, and f′ is the smallest superset of f. Therefore, for any label set f, all of its supersets are exactly all of its descendants in LNG. Therefore, descendants can be quickly obtained by using some label sets as entries.

[0056] Among them, for the label set f q The algorithm for obtaining the minimum superset is as follows:

[0057] First get f q The largest label l max And the corresponding inverted list I(l max ), represents the set of corresponding nodes in the prefix tree;

[0058] For each I(l max ) in the node u, if we traverse all the parents from the prefix tree upwards, and the label set of the parents is f q When a superset of u is found, search deep from u downward to obtain all supersets;

[0059] Finally, for all supersets, all minimum supersets are screened out according to the definition.

[0060] The method of constructing LNG in this embodiment is mainly through the prefix tree method. First, each label set f is traversed in descending order of the label set size. For each label set f, the smallest superset of all f′ is traversed and directly connected to f in LNG. The label set f is parsed and inserted into the tree in the general prefix tree manner, and the inverted list is updated according to each label.

[0061] like Figure 3 As shown, when the insertion is carried out to f 2 ={1,4,5}, access from top to bottom in the prefix tree. Since there is a path between 1,4,5 in the prefix tree, no additional insertion is required. 4 ={2,7,8}, since there is no node pointing directly from the root to 2, a new node needs to be created and the inverted list needs to be updated.

[0062] The tag navigation map constructed in this embodiment is as follows Figure 4 As shown in the figure, the selection of the entry label set needs to be discussed in different cases. If the query label set f qIf the query tag set has appeared in the original dataset, it will be directly used as the entry tag set. If the query tag set has not appeared in the original dataset, all f q The minimal superset of is used as the entry label set.

[0063] 3. Generate a unified label navigation map UNG

[0064] The overall idea of ​​building a unified label navigation graph (UNG) is to build a proximity graph within each label set group using existing graph-based ANNS methods. Then, cross-group edges are added along the direction of the edges in LNG (e.g. Figure 5 These edges connect vectors from different groups to form a unified navigation graph (UNG).

[0065] Based on the idea of ​​greedy search in ANNS neighbor graph query, we can greedily identify the groups that contribute to the top-k results of the query. In order to dynamically navigate to these groups during the search process, this scheme sets the group V in each group. f After constructing the proximity graph on the vectors of , we add new edges to bridge the gaps between different groups. To avoid visiting any vector that does not satisfy the filter condition, these new edges must always be connected to the query vector label set F. q Specifically, if there is a vector from f in LNG i to f j , then for the label set f i Each vector v in i , select the label set f j Some vectors v whose distance is less than or equal to the preset threshold j , establish v i to v j The edges of the label navigation graph are called cross-group edges. The cross-group edges are constructed according to the edges in the label navigation graph. Therefore, the final large graph unifies the label navigation graph and the neighbor graph. This graph is called the unified label navigation graph (UNG).

[0066] In each group, the existing ANNS algorithm is directly used to build a neighbor graph, and then these graphs are merged into a large graph. In this embodiment, the Vamana algorithm is used to build the neighbor graph. Through the parallelization method, different threads are assigned to different groups to build the neighbor graph. Single-threaded processing is used in each group, thereby achieving a significant acceleration effect.

[0067] The main idea of ​​constructing the cross-group edge algorithm is to use the current point as the query point, search for the nearest δ points and establish connections with them. Specifically, for point v i And its label set f i , query its out-degree neighbor f on the label navigation graph LNG jAccording to the results of label set grouping, each label set f j Each corresponds to a point set V j And its corresponding ANNS neighbor graph, in all V j On the graph of , query the top-δ nearest neighbors, merge all the results, select the nearest δ points, and compare them with v i Even the edge.

[0068] In order to ensure that there must be cross-group edges between any two sets of corresponding points with a minimum superset relationship in LNG, this scheme is designed to count such edges. If the edge e(f i ,f j )∈LNG but v i and There is no connection between the points in v i , need to Randomly pick a point v j Connect the edges to ensure the connectivity of UNG.

[0069] The unified label navigation graph framework generated above is applicable to query label sets of any size and combination. q , because the filtered search space can be gradually derived from the label set entry. At the same time, UNG can integrate most graph-based ANNS algorithms to build a neighbor graph for each group, such as NSG, KGraph, Vamana, etc.

[0070] 4. Greedy Search

[0071] For a given query vector, greedy search is performed starting from these entry vectors by retrieving the entry label set that directly covers the query label set. The setting of cross-group edges helps the search avoid vectors that are not related to the label, improving the accuracy and efficiency of the search.

[0072] The search process is top-down. For the query vector x q and query tag set f q , first determine its entry label set, randomly select σ vectors from all vectors corresponding to these entry label sets as entry points, and add them to the queue pq. When there are still unexplored vectors in the priority queue pq, repeat the following operations:

[0073] Pop the nearest unexplored vector v from pq i , and mark it as explored;

[0074] v i All out-degree neighbors in the label navigation graph are added to pq;

[0075] Keep pq and the query vector v q The first w most recent vectors.

[0076] Finally, we only need to return the number of nodes in the priority queue pq relative to the query vector v. q The most recent k vectors can be used as the search results.

[0077] like Figure 6 As shown, for the query label f q ={1,4}, its entry label set is f 2 and f 4 , when the parameter σ = 1 (that is, only one point is randomly selected as the entry point), the priority queue can use the vector and Initialize.

[0078] The effectiveness of this solution has not been verified. This embodiment has conducted extensive experiments on 10 real data sets and used Vamana as the underlying graph index to verify that the UNG framework proposed in this solution performs best in the trade-off between answer quality and query efficiency compared to the most advanced filtered ANNS algorithm. While maintaining the same accuracy, this solution achieves a 10-fold speed improvement compared to the baseline method.

[0079] In summary, this scheme introduces a new concept of "label navigation graph" as a dynamic guidance mechanism. By utilizing the entry label set retrieved by the query label, it strictly constrains ANNS to explore within the filtered search space. It also proposes a "unified label navigation graph" framework suitable for filtering ANNS, which can adapt to various graph-based ANNS algorithms and ensure versatility, accuracy, completeness and adaptability.

[0080] The current ANNS algorithms generally focus on scenarios that do not include label filtering. ANNS algorithms that can support label filtering scenarios are very rare and have defects. The proposed unified navigation graph algorithm in this solution fills this gap.

[0081] The ANNS algorithm currently supporting tag filtering scenarios performs poorly in terms of balancing efficiency and recall rate. Compared with other baseline algorithms, the unified navigation graph algorithm proposed in this solution has a 10-fold improvement in efficiency at the same recall rate, achieving the best performance among all current ANNS algorithms for tag filtering scenarios, saving search time to improve user experience, while reducing enterprise resource consumption;

[0082] In addition, some ANNS algorithms that currently support label filtering scenarios are severely limited in terms of label types and can only solve very special types of labels (completely equal length and very few types). The unified navigation graph proposed in this solution can support more general types of labels. The application scenarios are not only in traditional search recommendation and e-commerce fields, but can also be expanded to bioinformatics, financial risk control and other fields.

Claims

1. A vector search method in a filtering scenario based on scalar and vector joint navigation, characterized in that: The following steps are involved: S1. Divide the dataset according to the label set associated with each vector, and group vectors with the same label set together. S2. Based on the minimum superset relationship between all label sets, a directed acyclic graph is constructed, namely, the label navigation graph LNG, which is used to determine the entry label set; S3, construct cross-group edges according to the edges in the label navigation graph, use the existing ANNS algorithm to build a neighbor graph, and merge them to obtain a unified label navigation graph UNG; S4. Obtain the query vector, retrieve the entry tag set that directly covers the query tag set, perform greedy search starting from these entry vectors, and obtain search results.

2. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 1, characterized in that: The specific process of step S1 is as follows: Given a vector data set D = {(v i ,f1),(v2,f2),...,(v |D| , f |D| )} includes |D| high-dimensional vectors, each vector v i Uniquely corresponds to a tag set where |f i | represents vector v i The corresponding number of labels, let represents the set of all labels, then the total number of labels in the dataset is |L|; In order to avoid iterating over all vectors, the corresponding vector set is calculated for each possible label set in the data set. For each possible label set f, the corresponding vector set V is calculated in the indexing stage. f = {v i ∈V|f i =f}.

3. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 2 is characterized in that: In the label navigation graph constructed in step S2, the directed path from label set f to f″ is composed of edges e(f, f′) and e(f′, f″), which means that f″ is the minimum superset of f′, and f′ is the minimum superset of f. For any label set f, all its supersets are all its descendants in the label navigation graph.

4. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 3 is characterized in that: The step S2 specifically includes the following steps: S21, for the label set f q , get the corresponding minimum superset; S22. Using the prefix tree method, a label navigation graph is constructed by calculating the minimum superset for each label set to be inserted.

5. The vector search method in filtering scenarios based on scalar and vector joint navigation according to claim 4, characterized in that: The specific process of step S21 is as follows: Get the tag set f q The largest label l max And the corresponding inverted list I(l max ), represents the set of corresponding nodes in the prefix tree; For each I(l max ) in the node u, if we traverse all the parents from the prefix tree upwards, and the label set of the parents is f q When there is a superset of u, search deep from u downward to obtain all supersets; Finally, for all supersets, all minimal supersets are filtered out according to the definition.

6. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 5, characterized in that: The specific process of step S22 is: traverse each tag set f in descending order of the tag set size, for each tag set f, calculate all minimum supersets of f, and directly connect edges with f, then parse the tag set f in a prefix tree manner and insert it into the tree, and update the inverted list according to each tag.

7. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 4 is characterized in that: In step S2, if the queried tag set f q If it has appeared in the original dataset, it will be directly used as the entry label set; If the query label set f q If it has not appeared in the original data set, all f q The minimal superset of is used as the entry label set.

8. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 4, characterized in that: The specific process of constructing the cross-group edge in step S3 is: if there is a label navigation graph from the label set f i to f j For the directed edge of the label set f i Each vector v in i , select the label set f j Some vectors v whose distance is less than or equal to the preset threshold j , establish v i to v j These edges are cross-group edges. Cross-group edges are added along the direction of the edges in the label navigation graph, and vectors from different groups are connected using cross-group edges.

9. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 8, characterized in that: The specific process of merging to obtain a unified label navigation graph in step S3 is as follows: for different label set groups, different threads are allocated to different groups in a parallel manner to construct a neighbor graph, single-thread processing is used in each group, and finally all constructed neighbor graphs are merged to obtain a unified label navigation graph.

10. The vector search method in a filtering scenario based on scalar and vector joint navigation according to claim 8, characterized in that: The step S4 specifically comprises the following steps: S41. For the query vector x q and query tag set f q , first determine its entry label set, randomly select σ vectors from all vectors corresponding to these entry label sets as entry points, and add them to the queue pq; S41, when there are still unexplored vectors in the priority queue pq, repeat the following operations: Pop the nearest unexplored vector v from pq i , and mark it as explored; v i All out-degree neighbors in the unified label navigation are added to pq; Keep pq and the query vector v q The first w most recent vectors; S43. Return the number of nodes in the priority queue pq relative to the query vector v q The most recent k vectors are used as search results.

Images