Vector index method in dynamic data scene

Abstract

The application discloses a vector index method in a dynamic data scene, constructs a hybrid step index structure HSG based on coverage, proposes an approximate nearest neighbor search algorithm based on the HSG index structure, a vertex adding algorithm based on relative distance, an HSG vertex deleting algorithm based on an entry edge weight, carries out index graph evaluation and iterative optimization based on coverage, and finally realizes high-performance vector indexing by adding and deleting data in real time under a dynamic data stream mode and iteratively optimizing the index structure based on the HSG index structure. The method can solve the problem that current mainstream algorithms only support static data sets, can add and delete data in real time under a dynamic data stream mode, can iteratively optimize the index structure, can improve search efficiency and recall rate, and can provide effective support for wider application of artificial intelligence and big data technologies and provide a theoretical reference and a practical basis for wider and larger-scale vector similarity retrieval mechanism research.

Description

Technical Field

[0001] This invention belongs to the field of software algorithm technology, and specifically relates to a vector indexing method in dynamic data scenarios. Background Technology

[0002] Recently, vectors have received increasing attention as a data type for representing unstructured data such as images, videos, and text. E-commerce platforms convert images into vectors to enable image search, natural language processing converts text into vectors for semantic analysis, and vector data types are also widely used in fields such as machine vision, speech recognition, and personalized advertising.

[0003] Numerous near-nearest neighbor (NN) search engines have emerged to address the characteristics of different use cases. These NN search engines primarily employ four types of NN algorithms: hash-based, PQ (Product Quantization)-based, graph-based, and tree-based methods. However, existing NN algorithms do not fully support dynamic addition and deletion of data. While most algorithms can achieve high recall and search efficiency simultaneously, they require prior knowledge of the dataset or training set—that is, a general understanding of the data distribution patterns—when building the index. Even when deletion functionality is implemented, most algorithms simply mark the deleted data without actually deleting it from the index. When these algorithms are introduced into database applications, the known approach is to temporarily store newly added vectors, mark deleted vectors, and then periodically perform a full update. This approach is very costly and is clearly not an elegant method when dealing with large datasets.

[0004] Graph-based approximate nearest neighbor algorithms have received considerable attention since their inception. The paper "Hajebi K, Abbasi-Yadkori Y, Shahbazi H, et al. Fast approximate nearest-neighbor search with k-nearest neighbor graph Twenty-Second International Joint Conference on Artificial Intelligence. 2011" proposed the K-NN graph. The paper "Malkov Y, Ponomarenko A, Logvinov A, et al. Approximate nearest neighbor algorithm based on navigable small world graphs. Information Systems, 2014, 45: 61-68" proposed an approximate nearest neighbor search algorithm based on navigable small world (NSW) graphs, where vertices correspond to vectors and edges correspond to connections between them, using a greedy search algorithm when searching for the nearest neighbor. The paper "Malkov YA, Yashunin DA. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 2018, 42(4): 824-836" proposes an approximate nearest neighbor search method for hierarchical navigable small world (HNSW) graphs based on NSW, which greatly accelerates the search efficiency. The paper "Fu C, Xiang C, Wang C, et al. Fast approximate nearest neighbor search with the navigating spreading-out graph. arXiv preprint arXiv: 1707.00143, 2017" first clarifies four aspects after observing previous graph-based approximate nearest neighbor algorithms: ensuring the connectivity of the graph; reducing the average out-degree of the graph; shortening the search path; and reducing the index size.Then, a new graph structure that satisfies the above four aspects is proposed: Navigating Spreading-out Graph (NSG). The paper "Jayaram Subramanya S, Devvrit F, Simhadri HV, et al. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems, 2019, 32" proposes a new graph construction algorithm: Vamana, and extends it to SSD (Solid State Drive).

[0005] Numerous experiments and literature studies have shown that hash-based algorithms are simple, efficient, and easy to implement, but have low accuracy. PQ-based algorithms are simple and accurate, but are almost unusable with large datasets. Graph-based algorithms, while lacking theoretical guarantees, achieve a very good balance between search efficiency and recall; however, they suffer from long graph construction times and high memory consumption. Tree-based algorithms are more efficient at low dimensions, but their efficiency decreases as the dimensionality increases.

[0006] With the widespread application of big data and the increasing demand for vector data management, it is imperative to combine vector data with traditional databases. The main work focuses on modifying existing approximate nearest neighbor search algorithms to suit different usage scenarios. For example, it can be used for hybrid queries after combining with traditional structured data, or for hierarchical system design after combining search engines with databases.

[0007] In summary, while many efficient and flexible approximate nearest neighbor search algorithms and search engines have been proposed, a high-efficiency approximate nearest neighbor algorithm that supports dynamic data flow is still lacking. Therefore, there is an urgent need for an index structure that supports real-time addition and deletion of data, providing an efficient and flexible vector indexing method for applications such as big data and artificial intelligence. Summary of the Invention

[0008] To address the aforementioned technical challenges, this invention provides a vector indexing method for dynamic data scenarios. It aims to overcome the limitations of current mainstream algorithms that only support static datasets, enabling the application of the approximate nearest neighbor algorithm in dynamic data flow modes. This invention explores key technologies for high-performance vector indexing, providing effective support for the wider application of technologies such as artificial intelligence and big data.

[0009] The technical solution adopted in this invention is: a vector indexing method for dynamic data scenarios, the specific steps of which are as follows:

[0010] S1. Construct a hybrid step size index structure based on coverage: Hybrid Step Size Graph (HSG);

[0011] S2. Based on step S1, perform an approximate nearest neighbor search in the hybrid step size graph HSG;

[0012] S3. Based on step S1, implement a vertex addition algorithm based on relative distance;

[0013] S4. Based on step S1, implement the HSG vertex deletion algorithm based on incoming edge weight;

[0014] S5. Based on step S1, perform index graph evaluation and iterative optimization based on coverage.

[0015] S6. Based on steps S2-S5, the approximate nearest neighbor algorithm adds and deletes data in real time under dynamic data flow mode, iteratively optimizes the index structure, and achieves high-performance vector index.

[0016] Furthermore, step S1 is specifically as follows:

[0017] In the HSG, there are both long and short edges, which are separated by relative distance. The short edge comes from the K-ANN graph, that is, for any vertex in the index graph, the k nearest neighbors in the index graph are connected. At the same time, if there is an edge, and any vertex of the edge does not belong to the k nearest neighbors of another vertex, then this edge is a long edge.

[0018] In this system, the shorter side is responsible for searching the k nearest neighbors with the highest similarity to the target, while the longer side is responsible for quickly searching and querying vertices with high similarity to the target starting from the initial vertex of the graph. The HSG selects to limit the average degree of the shorter and longer sides separately, does not require global information of the dataset, and supports incremental updates of the index graph.

[0019] Furthermore, step S2 is specifically as follows:

[0020] S21. Input the query vector, add the starting vertex to the waiting queue, and add the long-side neighbor of the vertex at the head of the waiting queue to the waiting queue.

[0021] For any approximate nearest neighbor search, input the query vector, add the starting vertex to the waiting queue, start from the starting vertex, quickly search for vertices that are closer to the query target by using the long side, and only calculate the distance between the long side neighbor of the current waiting queue head vertex and the target vertex, and add the long side neighbor of the waiting queue head vertex to the waiting queue.

[0022] S22. After one iteration, determine whether the vertex of the waiting queue has changed. If it has changed, repeat step S21 until it has changed. If it has changed, proceed to step S23.

[0023] S23. Take out the head vertex of the waiting queue, add the short-side neighbor of the head vertex to the waiting queue, add the head vertex to the top queue, and determine whether the top queue has been successfully added.

[0024] By searching for k high-quality approximate nearest neighbors along the shortest side, if the distance between the head vertex of the result queue and the target vertex is less than the maximum value of the result queue, then the head vertex of the waiting queue is added to the result queue and removed from the waiting queue. Otherwise, the addition fails, the search ends, and the result queue is returned. If the addition is successful, the shortest-side neighbors of the head vertex are added to the waiting queue, and step S23 is repeated until the addition fails.

[0025] Furthermore, step S3 is specifically as follows:

[0026] S31. Input the vector to be added and perform an approximate nearest neighbor search on the target vector;

[0027] Starting from the starting vertex, search for the k nearest vertices to the target vertex; the returned results include: the k nearest neighbors and all vertices visited during the entire search process.

[0028] In this context, vertices in the top-k list are added as the short-side neighbors of the target vertex, while vertices that have been visited during the search process but are not in the top-k list are added as the long-side neighbors of the target vertex.

[0029] S32. Add the short-side neighbor based on the result returned in step S31;

[0030] When adding a short edge, determine whether a new vertex should be added as a neighbor to the vertices in the top-k list. If the number of short edge neighbors of a vertex in the top-k list has reached the degree limit, the neighbor with the largest distance needs to be deleted, and then a new vertex is added as a neighbor.

[0031] Deleting edges can cause two problems, as follows:

[0032] (1) A connected graph becomes a disconnected graph;

[0033] (2) The short side added earlier is clipped by the vertices added later, while the short side added earlier should exist as a long side.

[0034] If problem (1) occurs, the connectivity of the graph is determined by breadth-first traversal. If the graph becomes disconnected due to the deletion of the edge, the edge is saved to a special set to maintain the connectivity of the graph. If problem (2) occurs, the parameters are adjusted to retain the old long edges when pruning long edges, while the deleted short edges are not actually deleted, but moved to the long edge set.

[0035] S33. Add long-side neighbors based on the results returned in step S31;

[0036] The returned result is an ordered queue sorted from largest to smallest distance from the newly added vertex, divided into two parts: queue one contains vertices traversed by the long side during the search process, and queue two contains vertices traversed by the short side.

[0037] If the number of vertices in queue 2 is less than x, no long edge is added; if the number of vertices in queue 2 is equal to x, a long edge is added from the tail vertex of queue 1 to the newly added vertex; if the number of vertices in queue 2 is greater than x, a long edge is first added from the tail vertex of queue 1 to the x-th vertex in queue 2, and then starting from the x-th vertex in queue 2, a long edge is added every x vertices from the vertex to which the previously added long edge was located.

[0038] Where x represents an optional parameter.

[0039] Furthermore, step S4 is specifically as follows:

[0040] S41. Input the deletion vector. First, delete all short edges of the target vertex. Then, perform a 1-ANN search on the short edge neighbors of the target vertex, starting from itself and the deleted vertex.

[0041] S42. If the degree of a vertex is less than k, perform a short-side neighbor compensation operation. After the short-side compensation is completed, delete all long sides of the target vector.

[0042] S43. According to the search algorithm, long edges are no longer used after step S21. Therefore, to avoid the generation of disconnected long edge graphs, it is necessary to determine whether the product of the number of inbound edges and the number of outbound edges of the long edge of the deleted target vertex is greater than the preset threshold t.

[0043] Where t represents the user-specified parameter; if the product of the number of incoming edges and the number of outgoing edges of the target vertex's long side is less than t, then long side compensation algorithm one is used; if the product of the number of incoming edges and the number of outgoing edges of the target vertex's long side is greater than t, then long side compensation algorithm two is used, as detailed below:

[0044] (1) Long side compensation algorithm one: weight-based compensation algorithm;

[0045] Sum the distances of the incoming edges of the longer edge of the deleted vertex, denoted as ds. Denote the distance of any incoming edge as di. Connect the incoming edge neighbors and outgoing edge neighbors of the deleted vertex with the probability di / ds.

[0046] (2) Long side compensation algorithm two: Replace the target vertex with the nearest vertex;

[0047] Find a vertex to replace the deleted vertex, select the vertex that is closest to the deleted vertex, and move all the long edges of the deleted vertex to that vertex.

[0048] Furthermore, step S5 is specifically as follows:

[0049] S51. Input parameter x, and perform a breadth-first traversal starting from the starting vertex;

[0050] Let Vr be the set of vertices that can be reached by the long side. Starting from the vertices in Vr, perform x rounds of breadth-first traversal to obtain the set of vertices that can be covered, Vc, and output the coverage rate |Vr+Vc| / |V|.

[0051] Where x represents an optional parameter, and the vertex set V represents the set of all vertices.

[0052] If the coverage rate is greater than the preset coverage rate threshold, the coverage rate is output after the traversal is completed. If the coverage rate is less than the preset coverage rate threshold, proceed to step S52.

[0053] S52. Based on the coverage obtained in step S51, perform iterative optimization;

[0054] When calculating the coverage, we obtain the vertex set Vm = V - Vr - Vc, which is the set of missed vertices. For all vertices in Vm, we perform x rounds of breadth-first traversal starting from the vertex itself and using the short edge. We record the number of vertices in Vm encountered by each vertex during the traversal. We select the vertex with the largest number of vertices and add the long edge again. Then we perform x rounds of breadth-first traversal starting from that vertex. We repeat step S52 until the coverage reaches the target.

[0055] The beneficial effects of this invention are as follows: This invention constructs a Hybrid Nearest Neighbor (HSG) index structure based on coverage, proposes an approximate nearest neighbor search algorithm based on the HSG index structure, a vertex addition algorithm based on relative distance, and an HSG vertex deletion algorithm based on incoming edge weights. It also evaluates and iteratively optimizes the index graph based on coverage. Finally, based on the HSG index structure, the approximate nearest neighbor algorithm adds and deletes data in real-time under dynamic data flow mode, iteratively optimizing the index structure to achieve high-performance vector indexing. This invention overcomes the limitation of current mainstream algorithms that only support static datasets, enabling real-time data addition and deletion under dynamic data flow mode, and iterative optimization of the index structure, thus improving search efficiency and recall. This provides effective support for the wider application of technologies such as artificial intelligence and big data, and offers theoretical reference and practical basis for research on broader and larger-scale vector similarity retrieval mechanisms. Attached Figure Description

[0056] Figure 1This is a flowchart of a vector indexing method for dynamic data scenarios according to the present invention.

[0057] Figure 2 This is a flowchart of the HSG search strategy in an embodiment of the present invention.

[0058] Figure 3 This is a flowchart of the vertex addition algorithm based on relatively close distance in an embodiment of the present invention.

[0059] Figure 4 This is a flowchart of the weight-based vertex deletion algorithm in an embodiment of the present invention.

[0060] Figure 5 This is a flowchart illustrating the coverage-based evaluation method and iterative optimization strategy in an embodiment of the present invention. Detailed Implementation

[0061] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0062] like Figure 1 The flowchart of a vector indexing method for dynamic data scenarios according to the present invention is shown below, and the specific steps are as follows:

[0063] S1. Construct a hybrid step size index structure based on coverage: Hybrid Step Size Graph (HSG);

[0064] S2. Based on step S1, perform an approximate nearest neighbor search in the hybrid step size graph HSG;

[0065] S3. Based on step S1, implement a vertex addition algorithm based on relative distance;

[0066] S4. Based on step S1, implement the HSG vertex deletion algorithm based on incoming edge weight;

[0067] S5. Based on step S1, perform index graph evaluation and iterative optimization based on coverage.

[0068] S6. Based on steps S2-S5, the approximate nearest neighbor algorithm adds and deletes data in real time under dynamic data flow mode, iteratively optimizes the index structure, and achieves high-performance vector index.

[0069] In this embodiment, step S1 is specifically as follows:

[0070] Most current Approximate Nearest Neighbor (ANN) algorithms only support static datasets and cannot meet the requirements of dynamically changing data. Existing K-ANN algorithms can support dynamic data flows to some extent, but the index graph maintenance cost is high and the search efficiency is extremely low, making them unsuitable for practical applications on large-scale datasets. Therefore, it is necessary to research index graph structures that efficiently support dynamic data flows while balancing search performance and recall. This embodiment proposes a coverage-based hybrid step size index structure: Hybrid Strides Graph (HSG). By effectively controlling the vertex degree, the long and short edges between vertices are rationally configured to ensure the coverage of the index graph on the vertex set. During the search process, the search first quickly jumps to the vicinity of the target via the long edge, and then filters precise results via the short edge, balancing search performance and recall.

[0071] The construction process of HSG needs to reliably guarantee the configuration of asynchronous long edges. First, the long and short edges of HSG are based on the ideas of NSG (navigating spreading-out graph) and K-ANN algorithms, respectively. However, since NSG relies on global information to construct index edges, it is necessary to study index edge construction methods for dynamic data scenarios.

[0072] In the HSG, there are both long and short edges, which are separated by relative distance. The short edge comes from the K-ANN graph, that is, for any vertex in the index graph, the k nearest neighbors in the index graph are connected. At the same time, if there is an edge, and any vertex of the edge does not belong to the k nearest neighbors of another vertex, then this edge is a long edge.

[0073] Among them, the short side is responsible for searching the k nearest neighbors with the highest similarity to the target, while the long side refers to the NSG graph construction strategy and is responsible for quickly searching and querying vertices with high similarity to the target starting from the initial vertex of the graph.

[0074] Graph-based approximate nearest neighbor algorithms, such as HNSW, NSG, and Vamana (DiskANN), all share a common and clear basic graph construction idea: to minimize the average degree while maintaining connectivity, and to reduce the decrease in recall caused by the lower degree. HSG then faces a choice: either limit the average degree to a level similar to that of most graph-based approximate nearest neighbor algorithms, or individually limit the long and short edges, making their respective average degrees similar to those of NSG and K-ANN.

[0075] To maintain the robustness of the graph after a deletion operation, the HSG selects to limit the average degree of the short and long sides separately, without requiring global information of the dataset, and supports incremental updates of the index graph.

[0076] Graph-based approximate nearest neighbor algorithms face a crucial trade-off: average degree. Increasing the degree of the graph typically improves recall and enhances the robustness of the index graph, reducing the likelihood of disconnected graphs. However, a high degree also leads to more distance calculations in each iteration, which are meaningless and result in significant wasted computational resources, ultimately reducing search efficiency. Conversely, decreasing the average degree of the index graph within a certain range can improve search efficiency, but it also reduces recall to some extent. If the average degree is too low, disconnected graphs are more likely to appear, leading to a substantial drop in recall.

[0077] To mitigate the negative impact of degree increase on search efficiency, observation, experimentation, and theoretical analysis revealed that while longer edges can indeed improve search efficiency, it's not always necessary to traverse both the longer and shorter edges of the current vertex in each iteration. Only one of these edges can be effective in each iteration: if the longer edge's neighbor is closer to the target, then the shorter edge's neighbor must be farther away, and thus the shorter edge's neighbor doesn't need to be calculated; conversely, if the shorter edge's neighbor is closer to the target, then the longer edge's neighbor must be farther away, and thus the longer edge's neighbor doesn't need to be calculated.

[0078] like Figure 2 As shown, in this embodiment, step S2 is specifically as follows:

[0079] S21. Input the query vector, add the starting vertex to the waiting queue, and add the long-side neighbor of the vertex at the head of the waiting queue to the waiting queue.

[0080] For any approximate nearest neighbor search, input the query vector, add the starting vertex to the waiting queue, start from the starting vertex, quickly search for vertices that are closer to the query target by using the long side, and only calculate the distance between the long side neighbor of the current waiting queue head vertex and the target vertex, and add the long side neighbor of the waiting queue head vertex to the waiting queue.

[0081] Since the head vertex of the waiting queue is the vertex closest to the target among all previously calculated vertices, it can be considered as the result of the calculation of the short-side neighbors. This reduces the amount of computation and improves the search speed.

[0082] S22. After one iteration, determine whether the vertex of the waiting queue has changed. If it has changed, repeat step S21 until it has changed. If it has changed, proceed to step S23.

[0083] S23. Take out the head vertex of the waiting queue, add the short-side neighbor of the head vertex to the waiting queue, add the head vertex to the top queue, and determine whether the top queue has been successfully added.

[0084] By searching for k high-quality approximate nearest neighbors along the shortest side, if the distance between the head vertex of the result queue and the target vertex is less than the maximum value of the result queue, then the head vertex of the waiting queue is added to the result queue and removed from the waiting queue. Otherwise, the addition fails, the search ends, and the result queue is returned. If the addition is successful, the shortest-side neighbors of the head vertex are added to the waiting queue, and step S23 is repeated until the addition fails.

[0085] like Figure 3 As shown, in this embodiment, step S3 is specifically as follows:

[0086] S31. Input the vector to be added and perform an approximate nearest neighbor search on the target vector;

[0087] Starting from the starting vertex, search for the k nearest vertices (limited by the degree of the shorter side) to the target vertex; the returned result includes: the k nearest neighbors and all vertices visited during the entire search process.

[0088] In this context, vertices in the top-k list are added as the short-side neighbors of the target vertex, while vertices that have been visited during the search process but are not in the top-k list are added as the long-side neighbors of the target vertex.

[0089] The search process in step S31 is the same as the search algorithm in step S2. The only difference is that the returned result not only returns the k nearest neighbors, but also all the vertices that have been visited during the entire search process.

[0090] S32. Add the short-side neighbor based on the result returned in step S31;

[0091] When adding a short edge, determine whether a new vertex should be added as a neighbor to the vertices in the top-k list. If the number of short edge neighbors of a vertex in the top-k list has reached the degree limit, the neighbor with the largest distance needs to be deleted, and then a new vertex is added as a neighbor.

[0092] Deleting edges can cause two problems, as follows:

[0093] (1) A connected graph becomes a disconnected graph;

[0094] (2) The short side added earlier is clipped by the vertices added later, while the short side added earlier should exist as a long side.

[0095] If problem (1) occurs, the connectivity of the graph is determined by breadth-first traversal. If the graph becomes disconnected due to the deletion of the edge, the edge is saved in a special set, which is used to maintain the connectivity of the graph. Therefore, the set of edges that must exist must be used. If problem (2) occurs, the parameters are adjusted so that the old long edges are retained when pruning long edges. For the deleted short edges, although the absolute distance of the edges does not change, the original short edges become relatively long edges compared with these new shorter edges because shorter edges have appeared. Therefore, the short edges are not actually deleted, but moved to the long edge set.

[0096] S33. Add long-side neighbors based on the results returned in step S31;

[0097] The returned result is an ordered queue sorted from largest to smallest distance from the newly added vertex, divided into two parts: queue one contains vertices traversed by the long side during the search process, and queue two contains vertices traversed by the short side.

[0098] If the number of vertices in queue 2 is less than x, no long edge is added; if the number of vertices in queue 2 is equal to x, a long edge is added from the tail vertex of queue 1 to the newly added vertex; if the number of vertices in queue 2 is greater than x, a long edge is first added from the tail vertex of queue 1 to the x-th vertex in queue 2, and then starting from the x-th vertex in queue 2, a long edge is added every x vertices from the vertex to which the previously added long edge was located.

[0099] Where x represents an optional parameter.

[0100] Most current graph-based approximate nearest neighbor algorithms do not support vertex deletion. Some algorithms only provide marked deletion, that is, they mark deleted data in the index without actually deleting it, and skip it when using the index or filter the marked deleted data when returning results. The drawback is that after frequent addition and deletion of data, the index data only increases and never decreases, which not only wastes a lot of space, but also drastically reduces search efficiency. It often requires the full index to be regenerated periodically, which has a huge time overhead.

[0101] like Figure 4 As shown, in this embodiment, step S4 is specifically as follows:

[0102] S41. Input the deletion vector. First, delete all short edges of the target vertex. Then, perform a 1-ANN search on the short edge neighbors of the target vertex, starting from itself and the deleted vertex.

[0103] S42. If the degree of a vertex is less than k, perform a short-side neighbor compensation operation. After the short-side compensation is completed, delete all long sides of the target vector.

[0104] S43. According to the search algorithm, long edges are no longer used after step S21. Therefore, to avoid the generation of disconnected long edge graphs, it is necessary to determine whether the product of the number of inbound edges and the number of outbound edges of the long edge of the deleted target vertex is greater than the preset threshold t.

[0105] Where t represents the user-specified parameter; if the product of the number of incoming edges and the number of outgoing edges of the target vertex's long side is less than t, then long side compensation algorithm one is used; if the product of the number of incoming edges and the number of outgoing edges of the target vertex's long side is greater than t, then long side compensation algorithm two is used, as detailed below:

[0106] (1) Long side compensation algorithm one: weight-based compensation algorithm;

[0107] According to the definition of a long edge, we want the long edge to be as long as possible. In this case, we sum the distances of the incoming edges of the long edge of the deleted vertex, denoted as ds, and the distance of any incoming edge is denoted as di. We connect the incoming edge neighbors of the deleted vertex and the outgoing edge neighbors of the deleted vertex with the probability di / ds.

[0108] (2) Long side compensation algorithm two: Replace the target vertex with the nearest vertex;

[0109] To reduce the time taken for deletion operations, we can think of it from a graph perspective: find a vertex to replace the deleted vertex, select the vertex that is closest to the deleted vertex, and move all the long edges of the deleted vertex to that vertex.

[0110] The quality of the index graph is crucial. Nearest neighbor search results are the most direct evaluation method; higher recall and search efficiency indicate a better index graph quality. However, this is an abstract result, and it's impossible to obtain information about the index graph from it alone, thus hindering optimization. Therefore, a clear standard is needed to judge the quality of the index graph, and this standard should provide guidance for further optimization. According to the design philosophy of HSG, ideally, sparse and extensive long edges improve search efficiency, while dense and accurate short edges improve recall. Long edges are used to accelerate the search, so the goal is to reach most vertices in the index graph or their vicinity from the starting vertex via long edges.

[0111] like Figure 5 As shown, in this embodiment, step S5 is specifically as follows:

[0112] S51. Input parameter x, and perform a breadth-first traversal starting from the starting vertex;

[0113] Let Vr be the set of vertices that can be reached by the long side. Starting from the vertices in Vr, perform x rounds of breadth-first traversal to obtain the set of vertices that can be covered, Vc. Output the coverage rate |Vr+Vc| / |V|. The higher the coverage rate, the better.

[0114] Where x represents an optional parameter, and the vertex set V represents the set of all vertices.

[0115] If the coverage rate is greater than the preset coverage rate threshold, the coverage rate is output after the traversal is completed. If the coverage rate is less than the preset coverage rate threshold, proceed to step S52.

[0116] S52. Based on the coverage obtained in step S51, perform iterative optimization;

[0117] When calculating the coverage, we obtain the vertex set Vm = V - Vr - Vc, which is the set of missed vertices. For all vertices in Vm, we perform x rounds of breadth-first traversal starting from the vertex itself and using the short edge. We record the number of vertices in Vm encountered by each vertex during the traversal. We select the vertex with the largest number of vertices (or randomly select one if it is not unique) and add the long edge again. Then we perform x rounds of breadth-first traversal starting from that vertex. We repeat step S52 until the coverage reaches the target.

[0118] In summary, the method of this invention solves the problem that current mainstream algorithms only support static datasets. It can add and delete data in real time in dynamic data stream mode, and it can iteratively optimize the index structure to improve search efficiency and recall. It provides effective support for the wider application of technologies such as artificial intelligence and big data, and provides theoretical reference and practical basis for research on vector similarity retrieval mechanisms on a broader and larger scale.

[0119] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A vector indexing method for dynamic data scenarios, the specific steps of which are as follows: S1. Construct a hybrid step size index structure based on coverage: Hybrid Step Size Graph (HSG); The specific steps of S1 are as follows: In the HSG, there are both long and short edges, which are separated by relative distance. The short edge comes from the K-ANN graph, that is, for any vertex in the index graph, the k nearest neighbors in the index graph are connected. At the same time, if there is an edge, any vertex of the edge is not a k nearest neighbor of another vertex, then this edge is a long edge. in, The shorter side is responsible for searching the k nearest neighbors with the highest similarity to the target, while the longer side is responsible for quickly searching and querying vertices with high similarity to the target starting from the initial vertex of the graph. The HSG selects to limit the average degree of the shorter and longer sides separately, does not require global information of the dataset, and supports incremental updates of the index graph. S2. Based on step S1, perform an approximate nearest neighbor search in the hybrid step size graph HSG; S3. Based on step S1, implement a vertex addition algorithm based on relative distance; S4. Based on step S1, implement the HSG vertex deletion algorithm based on incoming edge weight; S5. Based on step S1, perform index graph evaluation and iterative optimization based on coverage. Step S5 is as follows: S51. Input parameter x, and perform a breadth-first traversal starting from the starting vertex; Let Vr be the set of vertices that can be reached by the long side. Starting from the vertices in Vr, perform x rounds of breadth-first traversal to obtain the set of vertices that can be covered, Vc, and output the coverage rate |Vr+Vc| / |V|. Where x represents an optional parameter, and the vertex set V represents the set of all vertices; If the coverage rate is greater than the preset coverage rate threshold, the coverage rate is output after the traversal is completed; if the coverage rate is less than the preset coverage rate threshold, proceed to step S52. S52. Based on the coverage obtained in step S51, perform iterative optimization; When calculating the coverage, we obtain the vertex set Vm = V - Vr - Vc, which is the set of missed vertices. For all vertices in Vm, we perform x rounds of breadth-first traversal starting from the vertex itself and passing through the short edge. We record the number of vertices in Vm encountered by each vertex during the traversal. We select the vertex with the largest number of vertices and add the long edge again. Then we perform x rounds of breadth-first traversal starting from that vertex. We repeat step S52 until the coverage reaches the target. S6. Based on steps S2-S5, the approximate nearest neighbor algorithm adds and deletes data in real time under dynamic data flow mode, iteratively optimizes the index structure, and achieves high-performance vector index.

2. The vector indexing method for dynamic data scenarios according to claim 1, characterized in that, Step S2 is as follows: S21. Input the query vector, add the starting vertex to the waiting queue, and add the long-side neighbor of the vertex at the head of the waiting queue to the waiting queue. For any approximate nearest neighbor search, input the query vector, add the starting vertex to the waiting queue, start from the starting vertex, quickly search for vertices that are closer to the query target by using the long side, and only calculate the distance between the long side neighbor of the current waiting queue head vertex and the target vertex, and add the long side neighbor of the waiting queue head vertex to the waiting queue. S22. After one iteration, determine whether the vertex of the waiting queue has changed. If it has changed, repeat step S21 until it has changed. If it has changed, proceed to step S23. S23. Take out the head vertex of the waiting queue, add the short-side neighbor of the head vertex to the waiting queue, add the head vertex to the top queue, and determine whether the top queue has been successfully added. Search for k high-quality approximate nearest neighbors by short side. If the result queue is less than k or the distance between the head vertex of the waiting queue and the target vertex is less than the maximum value of the result queue, add the head vertex of the waiting queue to the result queue and remove it from the waiting queue. Otherwise, the addition fails, the search ends, and the result queue is returned. If the addition is successful, add the short side neighbor of the head vertex to the waiting queue and repeat step S23 until the addition fails.

3. The vector indexing method for dynamic data scenarios according to claim 1, characterized in that, Step S3 is as follows: S31. Input the vector to be added and perform an approximate nearest neighbor search on the target vector; Starting from the initial vertex, search for the k nearest vertices to the target vertex; the returned results include: the k nearest neighbors and all vertices visited during the entire search process. In this context, vertices in the top-k list are added as the short-side neighbors of the target vertex, while vertices that have been visited during the search process but are not in the top-k list are added as the long-side neighbors of the target vertex. S32. Add the short-side neighbor based on the result returned in step S31; When adding a short edge, determine whether a new vertex should be added as a neighbor to the vertices in the top-k list. If the number of short edge neighbors of a vertex in the top-k list has reached the degree limit, the neighbor with the largest distance needs to be deleted, and then a new vertex is added as a neighbor. Deleting edges can cause two problems, as follows: (1) A connected graph becomes a disconnected graph; (2) The short side added earlier is clipped by the vertices added later, while the short side added earlier should exist as a long side; If problem (1) occurs, the connectivity of the graph is determined by breadth-first traversal. If the graph becomes disconnected due to the deletion of the edge, the edge is saved in a special set to maintain the connectivity of the graph. If problem (2) occurs, the parameters are adjusted to retain the old long edges when pruning long edges, while the deleted short edges are not actually deleted but moved to the long edge set. S33. Add long-side neighbors based on the results returned in step S31; The returned result is an ordered queue sorted from largest to smallest distance from the newly added vertex, divided into two parts: queue one contains vertices traversed by the long side during the search process, and queue two contains vertices traversed by the short side. If the number of vertices in queue 2 is less than x, no long edge is added; if the number of vertices in queue 2 is equal to x, a long edge is added from the tail vertex of queue 1 to the newly added vertex; if the number of vertices in queue 2 is greater than x, a long edge is first added from the tail vertex of queue 1 to the x-th vertex in queue 2, and then starting from the x-th vertex in queue 2, a long edge is added every x vertices from the vertex to which the previous long edge was added. Where x represents an optional parameter.

4. The vector indexing method for dynamic data scenarios according to claim 1, characterized in that, Step S4 is as follows: S41. Input the deletion vector. First, delete all short edges of the target vertex. Then, perform a 1-ANN search on the short edge neighbors of the target vertex, starting from itself and the deleted vertex. S42. If the degree of a vertex is less than k, perform a short-side neighbor compensation operation. After the short-side compensation is completed, delete all long sides of the target vector. S43. According to the search algorithm, long edges are no longer used after step S21. Therefore, to avoid the generation of disconnected long edge graphs, it is necessary to determine whether the product of the number of incoming edges and the number of outgoing edges of the long edge of the deleted target vertex is greater than a preset threshold. ; in, This represents the user-specified parameters; if the product of the number of inbound edges and the number of outbound edges on the longer side of the target vertex is less than... Then use the long-side compensation algorithm one; if the product of the number of inbound edges and the number of outbound edges of the target vertex's long side is greater than... Then, the long-side compensation algorithm two is used, as follows: Long-side compensation algorithm 1: Weight-based compensation algorithm; Sum the distances of the incoming edges of the longer edge of the deleted vertex, denoted as ds. Denote the distance of any incoming edge as di. Connect the incoming edge neighbors of the deleted vertex and the outgoing edge neighbors of the deleted vertex with the probability di / ds. Long-side compensation algorithm 2: Replace the target vertex with the nearest vertex; Find a vertex to replace the deleted vertex, select the vertex that is closest to the deleted vertex, and move all the long edges of the deleted vertex to that vertex.

Images