Vector hybrid retrieval method and system
By using a two-layer vector hybrid index (TLHI), and leveraging RTBI for attribute filtering and HNSW for vector similarity retrieval, the inefficiency of existing methods is resolved, achieving efficient vector hybrid retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2026-04-10
AI Technical Summary
Existing vector hybrid retrieval methods are inefficient in terms of attribute filtering and vector similarity retrieval, and cannot effectively meet retrieval needs.
A two-layer vector hybrid index, TLHI, is adopted, which includes a radix tree-based index, RTBI, and a hierarchical navigation small-world index, HNSW. Attribute filtering is performed through RTBI, and vector similarity retrieval is performed through HNSW to improve retrieval efficiency.
It achieves efficient attribute filtering and vector similarity retrieval, improves the efficiency of vector hybrid retrieval, and meets the needs of vector similarity retrieval and attribute filtering.
Smart Images
Figure CN121833751A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, in particular to a vector hybrid retrieval method and system. BACKGROUND
[0002] The vector retrieval technology can retrieve TopK vectors with high similarity to a query vector of data. In actual retrieval scenarios, the attribute restrictions of data also need to be considered.
[0003] The commonly used vector hybrid retrieval methods include an exhaustive method, an attribute filtering followed by vector similarity retrieval method, a vector similarity retrieval followed by attribute filtering method, and an attribute partition based vector similarity retrieval method.
[0004] However, the exhaustive method performs attribute filtering and vector similarity retrieval on all data in an exhaustive manner, which is time-consuming. The attribute filtering followed by vector similarity retrieval method marks data that do not meet the attribute restrictions, but still performs vector similarity retrieval on the global vector index, which is inefficient. The vector similarity retrieval followed by attribute filtering method may not obtain TopK retrieval results that pass the attribute filtering, and thus needs to backtrack and re-retrieve or retrieve more data during vector similarity retrieval, which is time-consuming. The attribute partition based vector similarity retrieval method selects frequently queried attributes, and partitions all data according to the attributes. However, if the queried attribute is not a frequently queried attribute or one attribute restriction includes multiple attributes, the exhaustive method needs to be used for attribute filtering, which is time-consuming. Therefore, the existing vector hybrid retrieval methods are inefficient. SUMMARY
[0005] Therefore, the present application provides a vector hybrid retrieval method and system based on a double-layer vector hybrid index TLHI, which can meet the requirements of vector similarity retrieval and attribute filtering, and improve the retrieval efficiency of vector hybrid retrieval.
[0006] To solve the above problems, the technical solutions provided by the present application are as follows:
[0007] In a first aspect, the present application provides a vector hybrid retrieval method, which is applied to a double-layer vector hybrid index TLHI. The TLHI includes a radix tree based index RTBI and a hierarchical navigation small world HNSW index. The method includes the following steps:
[0008] Obtaining vector data and attribute information corresponding to to-be-retrieved data;
[0009] Traversing the attribute information in sequence by using the RTBI to obtain a target partition including the attribute information;
[0010] Perform vector retrieval on the target partition by using the HNSW index to obtain a retrieval result matched with the vector data, the retrieval result including Top K data similar to the vector data, where K > 1.
[0011] In a possible implementation, the attribute information includes M attributes, where M > 1, and the sequentially traversing the attribute information by using the RTBI to obtain the target partition including the attribute information includes:
[0012] defining a pointer pointing to an initial node of the RTBI and an attribute index with an initial value of 1, the attribute index with the initial value of 1 pointing to a first attribute in the attribute information;
[0013] sequentially traversing the attribute information according to a preset attribute sequence order;
[0014] determining that an attribute value corresponding to the attribute index exists in a first node under the initial node of the RTBI;
[0015] adjusting the pointer to point to the first node, and updating the attribute index to 2;
[0016] determining that an attribute value corresponding to the updated attribute index exists in a second node under the first node;
[0017] adjusting the pointer to point to the second node, and updating the attribute index to 3;
[0018] traversing until the updated attribute index is greater than M, and determining that a corresponding node is the target partition.
[0019] In a possible implementation, the method further includes:
[0020] if it is determined that the attribute value corresponding to the attribute index does not exist in the first node under the initial node, returning that the retrieval result is empty, and ending the retrieval.
[0021] In a possible implementation, the method further includes:
[0022] determining a data amount of the target partition;
[0023] if the data amount of the target partition is greater than or equal to a preset data amount, performing vector retrieval on the target partition by using the HNSW;
[0024] if the data amount of the target partition is less than the preset data amount, performing vector retrieval on the target partition by using an exhaustive method.
[0025] In a possible implementation, the vector retrieval on the target partition by using the HNSW index to obtain a retrieval result matching the vector data comprises the following steps.
[0026] constructing a candidate list for storing nodes to be retrieved and a preset-length retrieval list for storing retrieved nodes with similar distances to the vector data, wherein the preset length is greater than or equal to K;
[0027] determining an initial node of a first layer, wherein a bitmap of the initial node indicates that the initial node has been visited;
[0028] adding the initial node to the candidate list and the retrieval list;
[0029] if a first distance between a head element of the candidate list and the vector data is less than a second distance between a head element of the retrieval list and the vector data, enumerating adjacent nodes corresponding to the head element of the candidate list, wherein the adjacent nodes include at least one node;
[0030] if a bitmap of the adjacent nodes indicates that the adjacent nodes have not been visited, updating the bitmap of the adjacent nodes;
[0031] if a third distance between the adjacent nodes and the vector data is less than the second distance or an amount of data in the retrieval list is less than the preset length, writing the adjacent nodes into the candidate list and the retrieval list;
[0032] if an amount of data in the retrieval list after the adjacent nodes are written is greater than the preset length, deleting the head element of the retrieval list;
[0033] returning to performing the judgment on whether the first distance is less than the second distance and subsequent steps until the candidate list is empty or the first distance is greater than the second distance, and ending the retrieval of the first layer, and using the obtained retrieval list for the next layer retrieval;
[0034] iterating the retrieval of all layers to obtain a target retrieval list;
[0035] obtaining the retrieval result from the target retrieval list.
[0036] In a possible implementation, the vector data and attribute information corresponding to the data to be retrieved are obtained by using a preset model to extract a feature vector of the data to be retrieved.
[0037] extracting a feature vector of the data to be retrieved by using a preset model to obtain the vector data, wherein the preset model is a model corresponding to a data type of the data to be retrieved;
[0038] analyze the data to be searched to obtain the attribute information, the attribute information being used to represent multiple dimensions of the data to be searched.
[0039] In a possible implementation, the method further includes:
[0040] obtaining original data;
[0041] constructing the radix tree-based index RTBI by using the original data.
[0042] In a second aspect, an embodiment of the present application provides a vector hybrid search system, the system being applied to a two-layer vector hybrid index TLHI, the TLHI including a radix tree-based index RTBI and a hierarchical navigation small world HNSW index, and the system including:
[0043] an obtaining module, configured to obtain vector data and attribute information corresponding to data to be searched;
[0044] a traversing module, configured to sequentially traverse the attribute information by using the RTBI to obtain a target partition including the attribute information;
[0045] a searching module, configured to perform vector search on the target partition by using the HNSW index to obtain a search result matching the vector data, the search result including Top K data similar to the vector data, where K is greater than or equal to 1.
[0046] In a third aspect, an embodiment of the present application provides a vector hybrid search device, the device including a processor, a memory, and a system bus;
[0047] the processor and the memory being connected through the system bus;
[0048] the memory being configured to store one or more programs, the one or more programs including instructions, the instructions, when executed by the processor, causing the processor to perform the vector hybrid search method in the first aspect.
[0049] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing instructions, the instructions, when executed on a device, causing the device to perform the vector hybrid search method in the first aspect.
[0050] Therefore, the present application has the following beneficial effects:
[0051] The embodiment of the present application provides a vector hybrid retrieval method, which is applied to a double-layer vector hybrid index TLHI, wherein the TLHI comprises a radix tree-based index RTBI and a hierarchical navigation small world HNSW index. Firstly, vector data and attribute information corresponding to to-be-retrieved data are acquired, secondly, attribute information is sequentially traversed by using the RTBI to obtain a target partition comprising the attribute information, and finally, the target partition is subjected to vector retrieval by using the HNSW index to obtain a retrieval result matched with the vector data. The retrieval result comprises Top K data similar to the vector data, wherein K is greater than or equal to 1. In this way, the double-layer vector hybrid index TLHI is used for hybrid retrieval, the first layer of the index uses the RTBI to realize efficient attribute filtering, the second layer of the index uses the HNSW to perform vector similarity retrieval, the demand for vector similarity retrieval and attribute filtering can be met at the same time, and the retrieval efficiency of the vector hybrid retrieval is improved.
[0052] The embodiment of the present application also provides a system corresponding to the above method, which has the same beneficial effects as the above method. BRIEF DESCRIPTION OF DRAWINGS
[0053] Figure 1 A flowchart of a vector hybrid retrieval method provided by the embodiment of the present application is shown in the figure;
[0054] Figure 2 A flowchart of another vector hybrid retrieval method provided by the embodiment of the present application is shown in the figure;
[0055] Fig. 3(a) is a flowchart of inserting first data provided by the embodiment of the present application;
[0056] Fig. 3(b) is a flowchart of inserting second data provided by the embodiment of the present application;
[0057] Fig. 3(c) is a flowchart of inserting third data provided by the embodiment of the present application;
[0058] Figure 4 An index structure diagram of a TLHI provided by the embodiment of the present application is shown in the figure;
[0059] Figure 5 A structure diagram of a vector hybrid retrieval system provided by the embodiment of the present application is shown in the figure;
[0060] Figure 6 A structure diagram of a vector hybrid retrieval device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0061] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present application.
[0062] In the present application, the relational terms such as first and second and the like are used only to differentiate one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by an "including a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0063] The vector mixed retrieval method with attribute filtering generally includes an exhaustive method, a method of attribute filtering followed by vector similarity retrieval, a method of vector similarity retrieval followed by attribute filtering, and a method of vector similarity retrieval based on attribute partitioning.
[0064] The exhaustive method is to perform attribute filtering on each piece of data one by one to obtain a data subset that meets the attribute restrictions, and then perform exhaustive vector similarity retrieval on the data subset to obtain a retrieval result. The retrieval result can completely meet the attribute restrictions and the vector distance is the smallest, but the exhaustive attribute filtering and vector similarity retrieval on all data bring huge time overhead, resulting in low mixed retrieval efficiency.
[0065] The method of attribute filtering followed by vector similarity retrieval needs to pre-construct a traditional relational database index (such as a B+ tree) and an approximate nearest neighbor search (ANNS) index for the attributes and vectors of the full data. The mixed retrieval first filters the attributes by using the traditional relational database index, and uses a bitmap to mark the data that does not meet the attribute restrictions; and then, when performing vector retrieval on the ANNS index, the data that does not meet the attribute restrictions is skipped to obtain a retrieval result. However, when performing vector similarity retrieval, the retrieval is still performed on the index of the global vector, without good use of the characteristics of attribute filtering, resulting in low retrieval efficiency.
[0066] The method of vector similarity retrieval followed by attribute filtering needs to first construct an ANNS index, and then use the ANNS index to retrieve the TopK vectors with high similarity in the mixed retrieval, and then perform attribute filtering on the TopK vectors to obtain the retrieval result. However, the attributes carried by the data corresponding to the TopK vectors retrieved in the vector retrieval may not be able to pass the attribute filtering, resulting in less than K data obtained finally, which needs to backtrack and re-retrieve or retrieve more data in the vector similarity retrieval, and either way needs to consume more time, resulting in low retrieval efficiency.
[0067] The method of vector similarity retrieval based on attribute partition is to partition the frequently searched attribute, and construct an ANNS index for each partition. In the mixed retrieval, the partition corresponding to the query attribute is selected from the constructed partitions, and then ANNS retrieval is performed in the partition corresponding to the query attribute to obtain the retrieval result. Although it can quickly partition and perform ANNS retrieval when the given query attribute is consistent with the frequently searched attribute, if the query attribute is not the frequently searched attribute or the query attribute contains multiple attribute restrictions, attribute filtering needs to be performed by the exhaustive method, resulting in a large time overhead and low retrieval efficiency.
[0068] As can be seen, the current vector mixed retrieval method with attribute filtering has low retrieval efficiency.
[0069] Therefore, the embodiments of the present application provide a vector mixed retrieval method and system, which is applied to a double-layer vector mixed index TLHI. The TLHI includes a radix tree-based index RTBI and a hierarchical navigation small world HNSW index. First, the vector data and attribute information corresponding to the data to be retrieved are obtained, then the attribute information is traversed in turn by using the RTBI to obtain a target partition including the attribute information, and finally the target partition is retrieved by using the HNSW index to obtain a retrieval result matched with the vector data. The retrieval result includes Top K data similar to the vector data, where K≥1. In this way, the mixed retrieval is performed based on the double-layer vector mixed index TLHI. The first layer uses the RTBI to achieve efficient attribute filtering, and the second layer uses the HNSW to perform vector similarity retrieval, which can meet the needs of vector similarity retrieval and attribute filtering at the same time, and improve the retrieval efficiency of the vector mixed retrieval.
[0070] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, a vector mixed retrieval method and system provided by the embodiments of the present application will be described below with reference to the accompanying drawings.
[0071] Referring to Figure 1 , Figure 1A flowchart of a vector hybrid retrieval method provided by an embodiment of the present application. The vector hybrid retrieval method can be implemented based on a two layer hybrid index (TLHI). The method specifically includes S101-S103.
[0072] S101: Obtain vector data and attribute information corresponding to the data to be retrieved.
[0073] The data to be retrieved includes not only vector data but also attribute information. The vector data is a feature vector in the data to be retrieved, and the attribute information is a plurality of dimensions possessed by the data to be retrieved.
[0074] The vector data and the attribute information generally exist in the form of a two-tuple <F, V>. The data to be retrieved can include at least M attribute information, M≥1, and the M attribute information constitutes an attribute information table, that is, F is an attribute information table including M attributes in the data to be retrieved, and the attribute information table includes a plurality of dimensions. For example, in an e-commerce recommendation system, each product has attributes such as price, brand, color, etc. The attribute information and its corresponding attribute value can be represented as: F[price]=100, F[brand]=A, F[color]=red; and V is an n-dimensional feature vector, that is, vector data.
[0075] In a possible implementation manner, obtaining the vector data and the attribute information corresponding to the data to be retrieved includes: extracting a feature vector of the data to be retrieved using a preset model to obtain the vector data, the preset model being a model corresponding to a data type of the data to be retrieved; and analyzing the data to be retrieved to obtain the attribute information, the attribute information being used to represent a plurality of dimensions of the data to be retrieved.
[0076] The feature extraction models used by different data types of data to be retrieved are not the same. For example, in an e-commerce recommendation system, an Img2Vec model can be used to extract a feature vector of a product image; and in retrieval-augmented generation (RAG) of a large model, a BERT (Bidirectional Encoder Representations from Transformers) model can be used to extract a feature vector of text data. That is, a feature vector of the data to be retrieved is extracted using a preset model corresponding to a data type of the data to be retrieved to obtain vector data of the data to be retrieved.
[0077] In an e-commerce recommendation system, the attribute information can include price, brand, color, etc.; in a power system, the attribute information can include valley-flat-peak price, power consumption, power consumption duration, etc. By analyzing the to-be-searched data, attribute information of the to-be-searched data can be obtained, which is used to represent multiple dimensions of the to-be-searched data. That is, in the e-commerce recommendation system, the multiple dimensions include price, brand, color, etc.; in the power system, the multiple dimensions can include valley-flat-peak price, power consumption, power consumption duration, etc. The attribute information involved in the embodiments of the present application is not exemplified one by one, and the attribute information can be determined according to actual search data and the search can be completed.
[0078] S102: sequentially traversing the attribute information by using the RTBI to obtain a target partition including the attribute information.
[0079] The TLHI includes an index RTBI based on a radix tree and a hierarchical navigation small world HNSW index, and a construction process of the TLHI can be referred to below.
[0080] The RTBI is used for attribute filtering, and the HNSW is used for vector similarity search. Using the RTBI for attribute filtering, data with the same attribute information can share storage space when being stored, thereby reducing the size of the index.
[0081] In a possible implementation manner, the attribute information can include M attributes, and the M attributes constitute an attribute information table, M≥1, that is, the attribute information table F includes M attributes.
[0082] When performing search, the search content of a vector hybrid search Q includes a to-be-searched attribute information table Fq, to-be-searched vector data Vq and a query data quantity K. The Fq includes M attributes, the Vq is an n-dimensional feature vector, and the K represents that the search result is to return Top K most similar vector data.
[0083] Sequentially traversing the attribute information by using the RTBI to obtain a target partition including the attribute information includes the following steps.
[0084] 1. A pointer cur is defined to point to an initial node Node0 of the RTBI, and an attribute index s_id with an initial value of 1 is defined, the attribute index s_id with the initial value of 1 points to a first attribute in the attribute information. The first attribute is a first attribute in the to-be-searched attribute information table Fq.
[0085] 2. The attribute information is sequentially traversed in an order of an attribute sequence S created when the RTBI is constructed, that is, M attributes in the to-be-searched attribute information table Fq are sequentially traversed, where the M attributes in the to-be-searched attribute information table Fq can be arranged in the order of the preset attribute sequence S, which is beneficial to improving the traversal efficiency.
[0086] 3. Check whether there is at least one child node with attribute value Fq[s_id] in all child nodes of the node pointed by the pointer cur. s_id ] of the node pointed by the pointer cur.
[0087] If it is determined that there is attribute value corresponding to the attribute index s_id in the first node under the initial node Node0 of the RTBI, the next step 4 is performed. In a possible implementation manner, if it is determined that there is no attribute value corresponding to the attribute index s_id in the first node under the initial node Node0 of the RTBI, it is returned that the retrieval result is empty, and the retrieval is ended.
[0088] It should be noted that in subsequent retrieval, if the attribute index s_id≤M and there is no attribute value corresponding to the attribute index s_id in the child node of the current node, it is directly returned that the retrieval result is empty, and the retrieval is ended.
[0089] When the vector mixed retrieval is performed, if a piece of data that can satisfy the attribute restriction of the query is not retrieved from the RTBI, that is, a partition corresponding to the attribute information is not retrieved, it can be returned that the retrieval result is empty. It is not necessary to continue to traverse the HNSW index, and the retrieval efficiency when the retrieval result is empty is improved.
[0090] 4. The pointer cur is adjusted to point to the first node, and the value of s_id is added by 1, and the attribute index s_id is updated to 2.
[0091] The steps 3-4 are repeatedly performed, for example, if it is determined that there is attribute value corresponding to the updated attribute index s_id 2 in the second node under the first node, the pointer cur is adjusted to point to the second node, and the value of s_id is added by 1, and the attribute index s_id is updated to 3.
[0092] 5. Until the updated attribute index s_id is greater than M, it is indicated that the node satisfying the attribute restriction is found, and it is determined that the node is the target partition satisfying the attribute restriction requirement, and then the vector similarity retrieval is performed on the data in the target partition.
[0093] S103: Perform vector retrieval on the target partition by using the HNSW index, to obtain a retrieval result matched with the vector data, and the retrieval result includes Top K data similar to the vector data, where K≥1.
[0094] It is determined that the attribute index s_id is greater than M, and Top K vector similarity retrieval is performed in the target partition.
[0095] It is determined whether the target partition is constructed with the HNSW index. In a possible implementation manner, before the HNSW index is constructed, whether the HNSW index needs to be constructed is determined by the data amount of the target partition.
[0096] determining the data amount of the target partition, if the data amount of the target partition is greater than or equal to a preset data amount threshold, constructing the HNSW. In the vector similarity retrieval, the HNSW is used for vector retrieval of the target partition. If the data amount of the target partition is less than the preset data amount threshold, the exhaustive method is used for vector retrieval of the target partition.
[0097] Since constructing an index within a partition will bring time and space overhead, and when the data amount of the target partition is small, the retrieval performance of the exhaustive method is not much different from that of the HNSW index, a data amount threshold threshold can be set for the RTBI as a hyperparameter of the index for user adjustment.
[0098] When the data amount in the partition is greater than or equal to threshold, the HNSW index is constructed for vector retrieval. When the data amount in the partition is less than threshold, the HNSW index is not constructed, and the exhaustive method is used for vector retrieval.
[0099] If the partition does not construct the HNSW index, all data in the partition is enumerated to obtain the K vectors with the smallest distance between the feature vectors V corresponding to the data and the to-be-retrieved vector data Vq, and the retrieval result is the final query result of the vector hybrid retrieval Q.
[0100] If the target partition has constructed the HNSW index, the concept of Layers is introduced in the HNSW index. In the bottom layer Layer=0, the connectivity relationship of all nodes is included; it should be noted that the vector data extracted from the original data is represented as a node in the HNSW index, and the connectivity relationship of all nodes is the connectivity relationship of all vectors. The following describes the vector data extracted from the original data as a node.
[0101] With the increase of the number of layers, the number of nodes in each layer gradually decreases and follows the exponential decay law. From the topmost layer where a node is located to all the layers below, the node exists. When using the HNSW index for retrieval, the retrieval starts from the topmost layer. The maximum number of layers of Layers is determined by a random exponential probability decay function.
[0102] In the topmost layer of the constructed HNSW index, an initial node Z is randomly selected, and a priority queue is initialized for each layer as a candidate list, that is, a candidate list is constructed. A visited bitmap is created for each layer. The candidate list is used to store the to-be-retrieved nodes, which are sorted from small to large according to the distance from the vector data. The bitmap is used to mark whether a node has been visited. If the bitmap is 1, the node has been visited. If the bitmap is 0, the node has not been visited.
[0103] From the top layer, mark visited[Z] as 1, the initial node Z has been visited, and add Z to the candidate list. Maintain an object T, and assign the initial node Z to the object T. Take the head element Head in the candidate list, and compare the distance between the head element Head in the candidate list and Q and the distance between the object T and Q.
[0104] If the distance between the head element Head in the candidate list and Q is greater than the distance between the object T and Q, the layer retrieval ends.
[0105] If the distance between the head element Head in the candidate list and Q is less than or equal to the distance between the object T and Q, enumerate all adjacent nodes e of the head element Head in the candidate list. If visited[e] is 1, indicating that the adjacent node e has been visited, skip the node. If visited[e] is 0, mark visited[e] as 1, and if the distance between the adjacent node e and Q is less than the distance between the object T and Q, add the adjacent node e to the candidate list, and assign the adjacent node e to T.
[0106] Repeat the above process until the candidate list is empty or the distance between the head element Head in the candidate list and Q is greater than the distance between the object T and Q, and then end the retrieval of the current layer.
[0107] Continue the final Z obtained in the last layer as the starting point Z of the next layer, and perform the foregoing retrieval process until the bottom layer is reached, and the K data points are retrieved.
[0108] In a possible implementation manner, the HNSW index is used for vector retrieval on the target partition to obtain a retrieval result matched with the vector data, including:
[0109] A candidate list for storing nodes to be retrieved and a priority queue with a preset length ef are constructed as a retrieval list, the retrieval list with the preset length ef can also be a RES list for storing nodes with similar distances to the vector data, and the nodes in the list are sorted in descending order of distance to the vector data, where the preset length ef is greater than or equal to K. The ef is a parameter of the HNSW index, and the larger the ef is set, the more accurate the retrieval result is but the larger the time cost is. The specific value of the ef can be set according to actual requirements.
[0110] Similarly, the retrieval starts from Z, and the initial node Z of the first layer is determined, and the bitmap visited[Z] of the initial node Z is 1, indicating that the initial node Z has been visited. It should be noted that the first layer is the top layer.
[0111] The initial node Z is added to the candidate list and the retrieval list, and then the head element of the candidate list and the head element W of the RES list are taken out, and the first distance between the head element Head of the candidate list and Q and the second distance between the head element W of the RES list and Q are compared.
[0112] If the first distance between the head element Head of the candidate list and Q is greater than the second distance between the head element W of the RES list and Q, the retrieval of the layer ends.
[0113] If the first distance between the head element Head of the candidate list and Q is less than the second distance between the head element W of the RES list and Q, the adjacent node e corresponding to the head element Head of the candidate list is enumerated, and the adjacent node e includes at least one node.
[0114] If visited[e] is 1, it indicates that the adjacent node e has been visited, and the node is skipped. If visited[e] is 0, visited[e] is marked as 1, the head element W of the RES list is taken out again, and if the third distance between the adjacent node e and Q is less than the second distance between the head element W of the RES list and Q, or the length of the RES list is less than ef, the adjacent node e is written into the candidate list and the RES list. The length of the list can be used to represent the amount of data in the list.
[0115] If it is determined that the amount of data in the RES list after the adjacent node e is written is greater than ef, the head element of the RES list is deleted.
[0116] The above process is repeated until the candidate list is empty or the first distance is greater than the second distance, and then the retrieval process of the current layer ends. The obtained retrieval list is used for the next layer retrieval, and the retrieval of all layers is traversed to obtain a target retrieval list.
[0117] After the retrieval process ends, the K elements closest to Q in the target RES list are the retrieval results of Top K of HNSW, and the retrieval results of Top K are the final retrieval results of the vector mixed query Q.
[0118] Referring to Figure 2 , Figure 2 A flowchart of another vector mixed retrieval method provided by the embodiments of the present application is shown.
[0119] The attribute filtering using the RTBI includes: initializing a node pointer cur to point to an initial node Node0, and initializing an attribute index s_id to be 1. It is judged whether the node pointed to by the pointer cur contains a child node with an attribute value Fq[S_(s_id)]. If not, it is returned that the search result is empty, and the search process is ended. If yes, the pointer cur is adjusted to point to the child node, and the attribute index s_id is increased by 1. Until the attribute index s_id is greater than M, the vector similarity search is performed in the target partition.
[0120] The vector similarity search using the HNSW index includes: if the HNSW index is not constructed in the target partition, the vector similarity search is performed using an exhaustive method; if the HNSW index is constructed in the target partition, the vector similarity search is performed using the HNSW index. After the search, a search result is output, and the search result is data with a similarity to the vector data in the Top K.
[0121] Based on the contents of the steps S101-S103, it can be known that the method is applied to a double-layer vector hybrid index TLHI, wherein the TLHI includes an index RTBI based on a radix tree and a hierarchical navigation small world HNSW index. Firstly, vector data corresponding to to-be-searched data and attribute information are acquired, secondly, the attribute information is sequentially traversed using the RTBI to obtain a target partition including the attribute information, and finally, the target partition is subjected to vector search using the HNSW index to obtain a search result matched with the vector data. The search result includes K data similar to the vector data, wherein K≥1. In this way, the hybrid search is performed based on the double-layer vector hybrid index TLHI, the first layer of the index uses the RTBI to achieve efficient attribute filtering, and the second layer of the index uses the HNSW to perform vector similarity search, which can meet the requirements of the vector similarity search and the attribute filtering, and improve the search efficiency of the vector hybrid search.
[0122] The construction process of the double-layer vector hybrid index TLHI used in the embodiments of the application is described below. To construct the double-layer vector hybrid index TLHI, original data used to build the index structure need to be acquired. In a possible implementation manner, the original data is acquired, and the index RTBI based on the radix tree is constructed using the original data. Taking an e-commerce recommendation system as an example, the original data is data related to e-commerce information.
[0123] The RTBI first extracts a corresponding attribute name list S=[S1, S2,..., S m ] from the original data. For each attribute S iThe number of distinct attribute values (Distinct Attribute Number, DAN) recorded in the original data is sorted in ascending order for all attributes, and the sorted attribute sequence is denoted as S, so as to reduce the space overhead to the greatest extent when constructing the index subsequently.
[0124] A globally unique initial node Node0 is newly created as the entry of the RTBI, and the value thereof is set to be empty. For each piece of original data, after the attribute information thereof is extracted, there are totally M different attributes, and the RTBI is constructed according to the attribute order in the sorted attribute sequence S.
[0125] The specific process of constructing the RTBI includes the following steps.
[0126] A pointer cur is defined, and is initially set to be Node0. An attribute index s_id is defined, and the initial value thereof is set to be 1, indicating the first attribute element S1 in the attribute sequence S.
[0127] The RTBI is traversed, and each time it is checked whether there is a child node in the child nodes of the node pointed to by the current cur, the attribute value of the child node being the attribute information F[S_(s_id)] of the current inserted data. If there is, the cur is assigned to be the child, and the s_id is incremented by 1. If there is not, a new child node Node_new is created for the current node, the attribute value of the child node being the attribute information F[S_(s_id)] of the current inserted data, and then the cur is assigned to be Node_new, and the s_id is incremented by 1. The above process is repeated until the s_id is greater than M, indicating that all M attribute information has been processed, and the insertion process of the current data ends.
[0128] The RTBI is traversed again, and when the number of child nodes of a certain node is 1, the current node and the only child node thereof are merged into one node, and two attribute information are expressed at the same time, so as to reduce the height of the index tree, and then reduce the search time and the space occupation of the RTBI index.
[0129] In an embodiment of the present application, the construction process of the TLHI is illustrated by way of example.
[0130] The current attribute list S is [brand, price, color], and currently contains three pieces of attribute data: F1={A, 100, red}, F2={A, 200, blue}, and F3={A, 100, black}. It can be found that the DANs of brand, price and color are 1, 2 and 3 respectively, and are arranged in ascending order.
[0131] An initial node Node0 of the TLHI index is constructed, and the cur pointer is defined to point to Node0 and s_id is 1.
[0132] (1) Inserting a first piece of data F1={A, 100, red}
[0133] Referring to FIG. 3(a), FIG. 3(a) is a flowchart of inserting a first piece of data according to an embodiment of the present application.
[0134] It is checked whether there is a node with a value of A in the child nodes of the node pointed to by the cur pointer, i.e., S[s_id]=S[1]=brand and F1[S[s_id]]=F1[brand]=A. The initial node Node0 has no child node, and a new child node Node_new with a value of A is created. The cur pointer is pointed to Node_new, and s_id+1 is updated to 2.
[0135] It is checked whether there is a node with a value of 100 in the child nodes of the node pointed to by the cur pointer, i.e., S[s_id]=S[2]=price and F1[S[s_id]]=F1[price]=100. There is no node with a value of 100, and a new child node Node_new with a value of 100 is created. The cur pointer is pointed to Node_new, and s_id+1 is updated to 3.
[0136] It is checked whether there is a node with a value of red in the child nodes of the node pointed to by the cur pointer, i.e., S[s_id]=S[3]=color and F1[S[s_id]]=F1[color]=red. There is no node with a value of red, and a new child node Node_new with a value of red is created. The cur pointer is pointed to Node_new, and s_id+1 is updated to 4. At this time, s_id is 4 which is greater than M, M=3, and the insertion of the first piece of data is completed.
[0137] (2) Inserting a second piece of data F2={A, 200, blue}
[0138] Referring to FIG. 3(b), FIG. 3(b) is a flowchart of inserting a second piece of data according to an embodiment of the present application.
[0139] The pointer cur is set to Node0, and s_id is 1. It is checked whether there is a node with a value of A in the child nodes of the node pointed to by the cur pointer, S[s_id]=S[1]=brand and F2[S[s_id]]=F2[brand]=A. It is found that there is already such a node, and the cur pointer is pointed to the node, and s_id is updated to 2.
[0140] Check if the node pointed by cur has a child with value 200, i.e. S[s_id] = S[2] = price, F2[S[s_id]] = F2[price] = 200. There is no child with value 200, create a new child Node_new with value 200, and then point cur to Node_new, s_id is incremented to 3.
[0141] Continue to check if the node pointed by cur has a child with value blue, i.e. S[s_id] = S[3] = color, F2[S[s_id]] = F2[color] = blue. There is no child with value blue, continue to create a new node Node_new with value blue. Then point cur to Node_new, s_id is incremented, and s_id is updated to 4. At this time, s_id is 4 which is greater than M, M = 3, and the second data insertion ends.
[0142] (3) Insert the third data F3 = {A, 100, black}
[0143] Referring to FIG. 3(c), FIG. 3(c) is a flow diagram of inserting the third data according to an embodiment of the present application.
[0144] Point cur to the Node0 node, and define s_id as 1. First, check if the node pointed by cur has a child with value A, i.e. S[s_id] = S[1] = brand, F3[S[s_id]] = F3[brand] = A. It already exists, point cur to the child node, and increment s_id to update s_id to 2.
[0145] Continue to check if the node pointed by cur has a child with value 100, i.e. S[s_id] = S[2] = brand, F3[S[s_id]] = F3[price] = 100. It already exists, point cur to the node, and increment s_id to update s_id to 3.
[0146] Continue to check if the node pointed by cur has a child with value black, i.e. S[s_id] = S[3] = color, F3[S[s_id]] = F3[color] = black. There is no child with value black, create a new child Node_new with value black. Point cur to Node_new, s_id is incremented to update s_id to 4. At this time, s_id is 4 which is greater than M, M = 3, and the third data insertion ends.
[0147] Referring to FIG. 3(c), FIG. 3(c) is a flow diagram of inserting the third data according to an embodiment of the present application. Figure 4 , Figure 4A TLHI index structure diagram provided by an embodiment of the present application.
[0148] After the RTBI is constructed, each node in the last layer of the RTBI corresponds to a partition, representing an attribute combination. Each piece of data is divided into the corresponding partition according to the attribute information, achieving the attribute filtering function.
[0149] Since constructing an index in a partition will bring time and space overhead, and the performance of exhaustive retrieval is not much different from that of HNSW index when the data volume is small, a threshold threshold is set for the RTBI as an index hyperparameter for the user to adjust, and the HNSW vector index is constructed only when the data volume in the partition is greater than the threshold.
[0150] In vector hybrid retrieval, attribute restriction and vector similarity retrieval can be achieved based on the index structure diagram. TLHI can efficiently filter multiple attributes, improving the attribute filtering speed; and only the HNSW index corresponding to the partition meeting the attribute restriction needs to be imported into the memory for retrieval, without importing irrelevant data index into the memory, reducing the memory space size occupied in vector retrieval. For vector hybrid retrieval, high retrieval efficiency and accuracy can be achieved.
[0151] The foregoing embodiment of the present application provides a vector hybrid retrieval method based on the above. Next, a vector hybrid retrieval system provided in the embodiment of the present application is described, which is used to execute the method shown in the foregoing embodiment. Figure 1 The system is applied to a double-layer vector hybrid index TLHI, which includes a radix tree-based index RTBI and a hierarchical navigation small world HNSW index.
[0152] The function of the vector hybrid retrieval system is described. The structure diagram of the vector hybrid retrieval system is shown in FIG. 4, which includes an acquisition module 401, a traversal module 402, and a retrieval module 403. Figure 5
[0153] Among them,
[0154] The acquisition module 401 is configured to acquire vector data and attribute information corresponding to to-be-retrieved data.
[0155] The traversal module 402 is configured to traverse the attribute information in sequence by using the RTBI to obtain a target partition including the attribute information.
[0156] The retrieval module 403 is configured to perform vector retrieval on the target partition by using the HNSW index to obtain a retrieval result matched with the vector data, the retrieval result including Top K data similar to the vector data, where K≥1.
[0157] In a possible implementation, the attribute information includes M attributes, M > 1, and the traversal module 402 is specifically configured to:
[0158] define a pointer pointing to an initial node of the RTBI and an attribute index with an initial value of 1, the attribute index with the initial value of 1 pointing to a first attribute in the attribute information; traverse the attribute information in a preset attribute sequence order; determine that an attribute value corresponding to the attribute index exists in a first node under the initial node of the RTBI; adjust the pointer to point to the first node, and update the attribute index to 2; determine that an attribute value corresponding to the updated attribute index exists in a second node under the first node; adjust the pointer to point to the second node, and update the attribute index to 3; and repeat the above operations until the updated attribute index is greater than M, and it is determined that a corresponding node is the target partition.
[0159] In a possible implementation, the traversal module 402 is specifically further configured to:
[0160] If it is determined that the attribute value corresponding to the attribute index does not exist in the first node under the initial node of the RTBI, return a search result as empty, and end the search.
[0161] In a possible implementation, the system further includes:
[0162] a determination module configured to determine a data amount of the target partition;
[0163] a determination module configured to determine a data amount of the target partition;
[0164] The determination module is further configured to, if the data amount of the target partition is less than the preset data amount, perform vector search on the target partition by using an exhaustive method.
[0165] In a possible implementation, the retrieval module 403 is specifically configured to:
[0166] constructing a candidate list for storing nodes to be retrieved and a preset length retrieval list for storing retrieved nodes with similar distances to the vector data, the preset length being greater than or equal to K; determining an initial node of a first layer, wherein a bitmap of the initial node represents that the initial node has been visited; adding the initial node to the candidate list and the retrieval list; if a first distance between a head element of the candidate list and the vector data is smaller than a second distance between a head element of the retrieval list and the vector data, enumerating adjacent nodes corresponding to the head element of the candidate list, the adjacent nodes including at least one node; updating a bitmap of the adjacent nodes if the bitmap represents that the adjacent nodes have not been visited; writing the adjacent nodes into the candidate list and the retrieval list if a third distance between the adjacent nodes and the vector data is smaller than the second distance or an amount of data in the retrieval list is smaller than the preset length; determining that the amount of data in the retrieval list after the adjacent nodes are written is greater than the preset length, and deleting the head element of the retrieval list; returning to perform the judgment of whether the first distance is smaller than the second distance and subsequent steps until the candidate list is empty or the first distance is greater than the second distance, ending retrieval of the first layer, and using a retrieval list obtained to retrieve a next layer; traversing retrieval of all layers to obtain a target retrieval list; and obtaining the retrieval result from the target retrieval list.
[0167] In a possible implementation, the obtaining module is specifically configured to:
[0168] extracting a feature vector of the data to be retrieved using a preset model to obtain the vector data, the preset model being a model corresponding to a data type of the data to be retrieved; and analyzing the data to be retrieved to obtain the attribute information, the attribute information being used to represent multiple dimensions of the data to be retrieved.
[0169] In a possible implementation, the system further includes:
[0170] The obtaining module 401 is further configured to obtain original data.
[0171] The constructing module is configured to construct the radix tree-based index RTBI by using the original data.
[0172] The embodiment of the application provides a vector hybrid retrieval system, which is applied to a double-layer vector hybrid index TLHI, and the system comprises an acquisition module, a traversal module and a retrieval module. The acquisition module is used for acquiring vector data and attribute information corresponding to to-be-retrieved data. The traversal module is used for sequentially traversing the attribute information by using the RTBI to obtain a target partition comprising the attribute information. The retrieval module is used for performing vector retrieval on the target partition by using the HNSW index to obtain a retrieval result matched with the vector data, wherein the retrieval result comprises Top K data similar to the vector data, and K is greater than or equal to 1. In this way, the double-layer vector hybrid index TLHI is used for hybrid retrieval, the first layer of the index uses the RTBI to realize efficient attribute filtering, the second layer of the index uses the HNSW to perform vector similarity retrieval, the demand for vector similarity retrieval and attribute filtering can be met at the same time, and the retrieval efficiency of vector hybrid retrieval is improved.
[0173] Based on the vector hybrid retrieval method provided in the above method embodiment, the embodiment of the application provides a vector hybrid retrieval device, which is shown in Figure 6 , and the device comprises a processor, a memory and a system bus.
[0174] The processor and the memory are connected through the system bus.
[0175] The memory is used for storing one or more programs, and the one or more programs comprise instructions, which, when executed by the processor, make the processor execute the vector hybrid retrieval method in any one of the above embodiments.
[0176] Based on the vector hybrid retrieval method provided in the above method embodiment, the embodiment of the application provides a computer readable storage medium, which stores instructions, and when the instructions run on a device, the device executes the vector hybrid retrieval method in any one of the above embodiments.
[0177] The various embodiments described in this specification are described in progressive order, and each embodiment can be understood to include any of the features of the other embodiments. In particular, the system or system embodiments can be simpler than the method embodiments, and therefore are described with less detail. The system and system embodiments described above are merely illustrative and are not meant to limit the scope of the application. One skilled in the art will readily recognize from the disclosure herein, the application includes various other embodiments that are neither explicitly described nor necessarily shown in the figures. Accordingly, the application is not to be limited in scope by the illustrative embodiments described herein.
[0178] Those skilled in the art will further appreciate that the functionality of the various examples described in this specification— both alone and in combination— can be implemented by one or more modules. For example, software modules can comprise one or more procedures, functions, classes, or other object implementations that perform the tasks described. The software modules can be stored on computer-readable media such as computer storage media or memories. Other examples are also possible.
[0179] The above description of disclosed embodiments is meant to be illustrative of the application and not limiting. Numerous modifications of the embodiments, as well as numerous implementations of the general principles of the present application, which are essentially within the scope of the application, will be readily apparent to one skilled in the art. Thus, the present application is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A vector hybrid retrieval method, characterized in that, The method is applied to a two-level vector hybrid index TLHI, which includes a radix tree-based index RTBI and a hierarchical navigation small-world index HNSW. The method includes: Obtain the vector data and attribute information corresponding to the data to be retrieved; The attribute information is traversed sequentially using the RTBI to obtain the target partition including the attribute information; The target partition is subjected to vector retrieval using the HNSW index to obtain retrieval results that match the vector data. The retrieval results include the Top K data that are similar to the vector data, where K≥1.
2. The method according to claim 1, characterized in that, The attribute information includes M attributes, where M ≥ 1. The step of sequentially traversing the attribute information using the RTBI to obtain the target partition including the attribute information includes: Define a pointer to the initial node of the RTBI and define an attribute index with an initial value of 1, wherein the attribute index with an initial value of 1 points to the first attribute in the attribute information; The attribute information is traversed sequentially according to the preset attribute sequence. Determine that the first node under the initial node of the RTBI contains an attribute value corresponding to the attribute index; Adjust the pointer to point to the first node and update the attribute index to 2; Determine that a second node under the first node contains an attribute value corresponding to the updated attribute index; Adjust the pointer to point to the second node and update the attribute index to 3; Continue traversing until the updated attribute index is greater than M, then determine the corresponding node as the target partition.
3. The method according to claim 2, characterized in that, The method further includes: If it is determined that there is no attribute value corresponding to the attribute index in the first node under the initial node, then the search result is returned as empty, and the search ends.
4. The method according to claim 1, characterized in that, The method further includes: Determine the amount of data in the target partition; If the data volume of the target partition is greater than or equal to the preset data volume, then the HNSW is used to perform vector retrieval on the target partition; If the data volume of the target partition is less than the preset data volume, then the target partition is searched using a vector search method.
5. The method according to claim 4, characterized in that, The step of using the HNSW index to perform vector retrieval on the target partition to obtain retrieval results matching the vector data includes: Construct a candidate list and a search list of preset length. The candidate list is used to store nodes to be searched, and the search list of preset length is used to store nodes that have been searched and are close to the vector data. The preset length is ≥ K. Determine the initial node of the first layer, wherein the bitmap of the initial node represents that the initial node has been visited; Add the initial node to the candidate list and the search list; If the first distance between the head element of the candidate list and the vector data is less than the second distance between the head element of the retrieval list and the vector data, then enumerate the adjacent nodes corresponding to the head element of the candidate list, and the adjacent nodes include at least one node. If the bitmap of the adjacent node indicates that the adjacent node has not been visited, then update the bitmap of the adjacent node. If the third distance between the adjacent node and the vector data is less than the second distance or the data volume of the retrieval list is less than the preset length, then the adjacent node is written into the candidate list and the retrieval list; If the amount of data in the search list after the adjacent node is determined to be greater than the preset length, then the head element of the search list is deleted. Return to the process of determining whether the first distance is less than the second distance and subsequent steps, until the candidate list is empty or the first distance is greater than the second distance, then end the first layer of retrieval and use the obtained retrieval list for the next layer of retrieval; Iterate through all layers of the search to obtain the target search list; The search results are obtained from the target search list.
6. The method according to claim 1, characterized in that, The step of obtaining the vector data and attribute information corresponding to the data to be retrieved includes: The feature vector of the data to be retrieved is extracted using a preset model to obtain the vector data. The preset model is the model corresponding to the data type of the data to be retrieved. The data to be retrieved is analyzed to obtain the attribute information, which is used to characterize multiple dimensions of the data to be retrieved.
7. The method according to claim 1, characterized in that, The method further includes: Obtain the raw data; The radix tree-based index RTBI is constructed using the original data.
8. A vector hybrid retrieval system, characterized in that, The system is applied to a two-level vector hybrid index TLHI, which includes a radix tree-based index RTBI and a hierarchical navigation small-world index HNSW. The system includes: The acquisition module is used to acquire the vector data and attribute information corresponding to the data to be retrieved; The traversal module is used to sequentially traverse the attribute information using the RTBI to obtain the target partition including the attribute information; The retrieval module is used to perform vector retrieval on the target partition using the HNSW index to obtain retrieval results that match the vector data. The retrieval results include the Top K data that are similar to the vector data, where K≥1.
9. A vector hybrid retrieval device, characterized in that, The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the vector hybrid retrieval method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the device, cause the device to perform the vector hybrid retrieval method as described in any one of claims 1 to 7.