Neighbor graph index updating method of dynamic vector database

Abstract

The invention belongs to the technical field of database correlation, and discloses a neighbor graph index updating method for a dynamic vector database, which comprises the following steps of: acquiring a to-be-deleted point of a current graph index of the dynamic vector database, which needs to be deleted; deleting incoming edges of all the points to be deleted from the graph index; aiming at each to-be-deleted point p with the outer neighbors, executing the following steps: acquiring points within two hops in the current graph index by taking the to-be-deleted point p as a starting point to construct a candidate set of new inner neighbors in pout, and connecting the points in the candidate set to the pout, wherein the pout is the outer neighbors of the to-be-deleted point p; and deleting all the points to be deleted and all the outbound edges to realize updating of the dynamic vector database graph index. According to the method, the edge beneficial to repairing the monotonous search path can be quickly and accurately found, the updating efficiency is high, and the graph index is still kept to have high precision after the graph index is updated.

Description

Technical Field

[0001] The present invention belongs to the technical field of database-related technologies, and more specifically, relates to a method for updating a neighbor graph index of a dynamic vector database. Background Art

[0002] With the major breakthrough of Large Language Model (LLM), it has demonstrated unprecedented ability in understanding, generating and reasoning human language, which is derived from the knowledge vector representation extracted from multimodal data. When embedded in the vector space, the correlation between different modal data can be measured by the distance between their vector representations. Approximate nearest neighbor (ANN) search can be used to find knowledge related to the prompt words provided by the user in the vector library. It is a basic function of LLM and a core function in applications such as data mining, recommendation and information retrieval.

[0003] Graph-based ANN search methods have become the most promising direction due to their high accuracy and high efficiency. Graph-based ANN search methods mostly build indexes based on neighbor graphs. These methods construct graph structures by treating each high-dimensional vector as a point in the graph and connecting points that are close to each other according to a specific strategy. For each query, the graph index starts from a fixed search starting point and executes a greedy search algorithm to obtain the ANN search results. The greedy search algorithm starts from the starting point and approaches the query target along the monotonic search path in the graph index, checks the distance between the points on the path and the query target in turn, and selects the point closest to the query as the ANN search result.

[0004] Most current research focuses on building high-quality indexes for static datasets. However, static ANN search methods cannot adapt to dynamic scenarios where data is frequently updated, such as e-commerce and news recommendation. In news recommendation, hot topics and breaking news change frequently, and the recommendation system needs to be updated in real time to provide the most relevant content. In e-commerce, order data is updated very quickly, especially during shopping festivals, when a large number of consumers frequently place, modify, or cancel orders. Therefore, dynamic ANN search that can adapt to these rapid changes has become increasingly important and deserves more attention. FreshDiskAnn can improve update performance by modifying the graph structure. It is currently the most advanced graph-based dynamic ANN search method. However, this method still faces two major challenges: First, the deletion operation performed when updating the graph index using this method will destroy the monotonic search path in the graph index that is critical to ANN search, and the repair effect of the monotonic search path is poor when performing path repair, resulting in a decrease in the accuracy of the updated graph index, making it difficult for the greedy search algorithm to approach the query target and difficult to meet the needs of high-precision application scenarios; second, the update efficiency of this method is low, and the time-consuming path repair leads to low update efficiency and difficulty in meeting the needs of real-time update scenarios. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a method for updating a neighbor graph index of a dynamic vector database, which aims to quickly update the graph index while ensuring the correctness of the graph index in the face of a dynamically changing database, so as to quickly adapt to the dynamic changes of the database.

[0006] To achieve the above object, the present invention provides a method for updating a neighbor graph index of a dynamic vector database, which comprises:

[0007] Get the points to be deleted that need to be deleted in the current graph index of the dynamic vector database;

[0008] Deleting all incoming edges of the points to be deleted from the graph index;

[0009] For each point p to be deleted that has an external neighbor, execute: Get the points within two hops from the current graph index starting from the point p to be deleted to construct p out The candidate set of new inner neighbors, p out For the outer neighbors of the point p to be deleted, connect the points in the candidate set to p out ;

[0010] Delete all the points to be deleted and all their outgoing edges to update the graph index of the dynamic vector database;

[0011] Among them, the outer neighbors of any point are the points connected to the outgoing edges of the point, and the inner neighbors of any point are the points connected to the incoming edges of the point.

[0012] Optionally, deleting all incoming edges of the points to be deleted from the graph index includes:

[0013] Scan each point in the current graph index in turn and delete the outgoing edge from the currently scanned point to the point to be deleted.

[0014] Optionally, obtain the points within two hops from the point to be deleted p in the current graph index to construct p out The candidate set of new inner neighbors includes:

[0015] Initialize the candidate set to be empty;

[0016] According to the given point to be deleted, scan the outer neighbors p of the point to be deleted p in turn out , and after scanning the current p out After that, first scan the current p out The outer neighbor and the current p out After all the outer neighbors of point p are scanned, scan and delete the next outer neighbor p of point p. out; Add the scanned points to the candidate set in sequence according to the scanning order until the storage limit of the candidate set is reached.

[0017] Optionally, obtain the points within two hops from the point to be deleted p in the current graph index to construct p out The candidate set of new inner neighbors includes:

[0018] The outer neighbors of the point to be deleted and the points that are related to p out The distance is less than d(p,p out ) are added to the candidate set, and points that are outer neighbors of the outer neighbors of the point to be deleted p and that simultaneously meet three set conditions are added to the candidate set, wherein the three set conditions include:

[0019] d(c,p out ) <d(p,p out )

[0020] d(c,p)>d(p,p out )

[0021] d 2 (c,p out )+d 2 (p,p out )>d 2 (c,p)

[0022] In the formula, p is the point to be deleted, p out is the outer neighbor of p, c is the point of p in the candidate set, d(c,p out ) represents c and p out The distance in the graph index, d(p,p out ) for p and p out The distance in the graph index, d(c,p) is the distance between c and p in the graph index.

[0023] Optionally, obtain the points within two hops from the point to be deleted p in the current graph index to construct p out The candidate set of new inner neighbors includes:

[0024] Initialize the candidate set to be empty;

[0025] According to the given point p to be deleted and one of its outer neighbors p out , scan the other outer neighbors p of the point to be deleted in turn out ', when scanning to the same out The distance is less than d(p,p out ) out 'After that, first scan the current p out 'Outer neighbors until the current p out'After scanning all the external neighbors, scan the next other external neighbor p of the point to be deleted out '; Add the points that meet the conditions for joining the candidate set to the candidate set in sequence according to the scanning order until the storage limit of the candidate set is reached.

[0026] Optionally, the point in the candidate set is connected to p out ,include:

[0027] First, the distance p in the current candidate set out The nearest point Move into the result set and delete the candidates in the current candidate set that match the current point. The distance is less than p out The distance between the candidate set and the result set is 0 is 0. Repeat this step until the current candidate set is empty or the number of nodes in the result set reaches its upper limit.

[0028] Add a point from the result set to p out The outgoing edge.

[0029] Optionally, the method further comprises:

[0030] For the point v in the result set whose outgoing edge is greater than the set upper limit, an edge reduction operation is performed, wherein the edge reduction operation includes: removing the point closest to v in the current outgoing neighbor set of v Move v into the candidate set of outer neighbors and delete the candidates in the current set of outer neighbors that match the current point. The distance from is less than the distance from v and Repeat this step until the current outer neighbor set is empty or the number of nodes in its outer neighbor candidate set reaches the upper limit of the number of outgoing edges, and finally use its outer neighbor candidate set as the outer neighbor set of point v.

[0031] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0033] The present invention also provides a computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the steps of any of the above methods are implemented.

[0034] In general, compared with the prior art, the above technical solutions conceived by the present invention mainly have the following beneficial effects:

[0035] 1. The present invention selects a point within two hops from the point to be deleted p to construct p out The candidate set of new inner neighbors, through analysis, point p is p out One of the inner neighbors of , so point p is located at p out Nearby points, and the neighbors within two hops of point p are also located at p out nearby points, and p out The greedy search result for the query is located at p out Nearby, so the neighbors within two hops of point p are highly overlapped with the greedy search results, therefore, the two-hop neighbors of point p can effectively replace the greedy search results, therefore, constructing a candidate set of repair paths based on the two-hop neighbors of point p can effectively repair the monotone search path destroyed by deleting point p, and ensure the correctness of the graph index update. Moreover, compared to constructing a candidate set through greedy search, the time consumed by obtaining the two-hop neighbors of point p to construct a candidate point set is also greatly reduced. In summary, the present invention effectively repairs the monotone search path destroyed by deletion by repairing the monotone search path to the outer neighbors of the deleted point, thereby achieving a high accuracy of the graph index after updating the graph index, and the candidate set acquisition method used in the present invention can quickly and accurately find edges that help repair the monotone search path. Compared with other methods, the present invention has a high update efficiency.

[0036] 2. The present invention can expand the existing graph index to support the dynamic graph index problem. Compared with other existing solutions that require the redeployment of indexes, the present invention can expand the functions of the already deployed indexes, so it has the advantage of low deployment difficulty.

[0037] 3. Furthermore, the present invention has an upper limit parameter that can be flexibly adjusted, and a trade-off can be made between index accuracy and update efficiency when used. Different parameter settings can be used for different usage scenarios and requirements, so the present invention can adapt to more complex real-world scenario requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flowchart of the steps of a method for updating a neighbor graph index of a dynamic vector database in one embodiment of the present invention;

[0039] Figure 2 is a specific flow chart of constructing a candidate set in one embodiment of the present invention;

[0040] Figure 3 is a specific flow chart of constructing a candidate set in another embodiment of the present invention;

[0041] Figure 4 is a detailed flowchart of a method for updating a neighbor graph index of a dynamic vector database in one embodiment of the present invention;

[0042] Figure 5 is an example diagram of a selected candidate set in an embodiment of the present invention;

[0043] Figure 6 is an example diagram of screening candidate sets and repairing monotonic search paths in one embodiment of the present invention;

[0044] Figure 7 is an example diagram of an effective repair monotonic search path in one embodiment of the present invention;

[0045] Figure 8 This is an experimental comparison chart of the indexing accuracy between the present invention and FreshDiskAnn;

[0046] Fig. 9 This is an experimental comparison chart of the deletion throughput between the present invention and FreshDiskAnn. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Example 1

[0049] like Figure 1 The figure is a flowchart of the steps of a method for updating a neighbor graph index of a dynamic vector database in one embodiment of the present invention, and the steps are described in detail below.

[0050] Step S1: Obtain the points to be deleted that need to be deleted in the current graph index of the dynamic vector database.

[0051] Among them, the vector database is a database specially designed for storing, managing and querying vector data. Its core functions include storing vector data and providing users with similarity query interfaces and data update interfaces. Its data will change dynamically, which is called a dynamic vector database. The graph index is the core component of the vector database. The graph index is a network structure composed of a series of nodes and edges. Different nodes represent different vector representations in the vector library. The edge represents the relationship between two nodes. The length of the edge reflects the strength of the relationship between the two vector representations. Through its graph index, you can search for knowledge related to the prompt words provided by the user in the vector library to realize applications such as data mining, recommendation and information retrieval. When the database changes dynamically, its graph index also needs to be updated. Its update speed and accuracy directly affect the user experience.

[0052] When the dynamic changes of the database involve the deletion of data, the node representing the data in the current graph index needs to be deleted, and the edge connected to the node will also be deleted, which will destroy the original monotonic search path. Therefore, when updating the graph index, it is necessary to delete the node and repair the path, so that the graph index can adapt to the dynamic changes of the current database and ensure its accuracy.

[0053] In the present invention, when the database changes dynamically, the deleted data therein can be directly obtained, and the points that need to be deleted in the current graph index can be clearly identified, which are called points to be deleted.

[0054] Step S2: Delete all incoming edges of the points to be deleted from the graph index.

[0055] Specifically, there may be one or more points to be deleted. After the point to be deleted is identified, the incoming edge and outgoing edge of the point to be deleted in the graph index can be identified. The point to be deleted, all its incoming edges and all its outgoing edges need to be deleted. The present invention implements the deletion of incoming edges and outgoing edges through different strategies. It can be understood that the graph index is a directed graph, the incoming edge of the node is the edge from other nodes pointing to the node, and the outgoing edge is the edge from the node pointing to other nodes. For a certain node, the node at the other end of the incoming edge of the node is the internal neighbor of the node, and the node at the other end of the outgoing edge of the node is the external neighbor of the node.

[0056] This embodiment first deletes all incoming edges of all points to be deleted.

[0057] Specifically, the graph index is usually stored in the form of an edge table. Each node corresponds to an edge table, which records all outgoing edges of the node. The graph index can be updated by modifying the outgoing edges of the storage space.

[0058] In one embodiment, the deletion of the incoming edge can be implemented in the following manner:

[0059] Scan the points to be deleted in turn, obtain all the internal neighbors of the currently scanned point to be deleted, and delete the outgoing edges pointing to the current point to be deleted in its internal neighbor edge table.

[0060] Through the above method, all the edges with points to be deleted can be deleted.

[0061] In another embodiment, the deletion of the incoming edge may also be achieved by:

[0062] Scan each point in the current graph index in turn and delete the outgoing edge from the currently scanned point to the point to be deleted.

[0063] The previous method of deleting incoming edges involves modifying the edge table of the inner neighbors multiple times when some points are inner neighbors of multiple points to be deleted, which is inefficient. Moreover, when multiple threads are used to perform deletion, this method is likely to cause multiple threads to compete for access to the edge tables of these points, which will lead to more serious efficiency problems. In addition, the previous deletion method requires additional maintenance of the incoming edge table of each point in the graph index, which increases the index maintenance overhead and the difficulty of index implementation.

[0064] In the latter method of deleting incoming edges, each point in the graph only needs to modify the edge table once at most, and assigning disjoint point sets to each thread can avoid thread conflicts, which can greatly improve the efficiency of multi-threaded deleting incoming edges. Although this method may require scanning points that do not require modifying the edge table, considering that each point needs to modify the edge tables of multiple points when it is deleted, when the number of points marked for deletion is large, almost all points in the graph need to modify the edge table. Therefore, this method has little efficiency loss due to scanning points that do not need to modify the edge table.

[0065] Step S3: For each point p to be deleted that has an external neighbor, execute: obtain the points within two hops from the point p to be deleted in the current graph index to construct p out The candidate set of new inner neighbors, p out For the outer neighbors of the point p to be deleted, connect the points in the candidate set to p out .

[0066] Specifically, for the convenience of description, p is used to represent the point to be deleted, N out (p) represents the set of outer neighbors of point p, p out represents any outer neighbor of p, p out ′ represents any other outer neighbor of p.

[0067] For each point p to be deleted that has an external neighbor, its external neighbor set N can be obtained out (p), for its external neighbor set N out Every outer neighbor p in (p) out , it is necessary to establish its candidate set and repair its path.

[0068] The traditional way to establish a candidate set and perform path repair is as follows:

[0069] With p out To query the target, the graph index greedy search algorithm is used to start from the graph index search starting point and obtain the monotonic p out It can be understood that the monotone search path means that along the extension direction of the path, the distance between each point on the path and the query target is getting shorter and shorter, that is, each point on the path is getting closer and closer to the query target, so as to be monotonically pout The closest monotone search path to p out A preset number of points are added to the candidate set, that is, each point in the candidate set has a monotonicity of p out Monotonic search path for ;

[0070] This method can ensure that all points in the candidate set are within reach of p out On the monotone search path of out The edges of the graph index can be used to repair the damaged monotone search path. However, although this method can ensure that all points in the candidate set are monotone to p out The monotonic search path has a good degree of repair for the graph index, but the greedy search algorithm has a high time overhead, so the solution of obtaining the candidate set through search limits the deletion efficiency.

[0071] Therefore, in order to improve efficiency and ensure the repair effect at the same time, the present invention proposes another method for quickly constructing a candidate set.

[0072] Specifically, obtain the points within two hops from the point to be deleted p in the current graph index to construct p out Candidate set of new inner neighbors.

[0073] The reasons for the above selection are:

[0074] Point p is p out One of the inner neighbors of , so point p is located at p out Nearby points, and the neighbors within two hops of point p are also located at p out nearby points, and p out The greedy search result for the query is located at p out Therefore, the neighbors within two hops of point p are highly overlapped with the greedy search results. Therefore, the two-hop neighbors of point p can effectively replace the greedy search results. In addition, since obtaining the two-hop neighbors of point p only requires O(1) time complexity, this approximate replacement can greatly improve the efficiency of candidate set generation.

[0075] Specifically, we can directly construct p with points within two hops. out The candidate set of new inner neighbors can also be constructed by further screening some points from points within two hops. The former is more efficient, and the latter has a better repair effect. The specific selection can be flexibly made according to actual needs.

[0076] The selection of data usually has an upper limit. For the graph index of the database, its structure is very complex and each node usually has a large number of neighbors. Considering the data upper limit, points that do not exceed the data upper limit can be selected to construct a candidate set.

[0077] In one embodiment, points within two hops can be directly obtained to construct p out The candidate set of the new inner neighbor and ensure that the number of candidate sets does not exceed its upper limit:

[0078] Initialize the candidate set to be empty;

[0079] According to the given point to be deleted, scan the outer neighbors p of the point to be deleted p in turn out , and after scanning the current p out After that, first scan the current p out The outer neighbor and the current p out After all the outer neighbors of point p are scanned, scan and delete the next outer neighbor p of point p. out ; Add the scanned points to the candidate set in sequence according to the scanning order until the storage limit of the candidate set is reached.

[0080] like Figure 2 The figure shows a specific flow chart of constructing a candidate set in one embodiment of the present invention, where the upper limit of the candidate set size is set to C. s First, input the point p to be deleted and initialize the candidate set C to be empty. Then, take the first outer neighbor in the edge list of point p as p. out Add to candidate set C. If the size of candidate set C reaches C s , then stop building the candidate set and return to the current candidate set C. Otherwise, continue to build p out The outer neighbors of are also added to the candidate set C one by one. In this process, if the size of C reaches C s Then stop building the candidate set and return the current candidate set C, otherwise take the next point in the edge table of point p as p out The process is repeated until the size of the candidate set C reaches C s Or all the outer neighbors of point p are scanned.

[0081] In another embodiment, in order to further improve the accuracy of path repair, conditional constraints are further added to select points that meet the constraint conditions from points within two hops from the deletion point p to form a candidate set. out The distance is less than d(p,p out ) are added to the candidate set, and points that are the outer neighbors of the outer neighbors of the point to be deleted p and meet the three set conditions are added to the candidate set. The three set conditions include:

[0082] d(c,p out ) <d(p,p out )

[0083] d(c,p)>d(p,p out )

[0084] d 2 (c,p out )+d 2 (p,p out )>d 2 (c,p)

[0085] In the formula, p is the point to be deleted, p out is the outer neighbor of p, c is the point of p in the candidate set, d(c,p out ) represents c and p out The distance in the graph index, d(p,p out ) for p and p out The distance in the graph index, d(c,p) is the distance between c and p in the graph index.

[0086] For example, define a candidate region CR(p,p out ):

[0087] CR(p,p out )={x|d(x,p out ) <d(p,p out ),d(x,p)>d(p,p out ),

[0088] d 2 (x,p out )+d 2 (p,p out )>d 2 (x,p)}

[0089] The points in the above candidate areas are the points that meet the three conditions.

[0090] Similarly, considering the data upper limit, the following specific method is used to filter out points that meet the constraints from the points within two hops from the deletion point p and ensure that the number of candidate sets does not exceed its upper limit:

[0091] Initialize the candidate set to be empty;

[0092] According to the given point p to be deleted and one of its outer neighbors p out , scan the other outer neighbors p of the point to be deleted in turn out ', when scanning to the same out The distance is less than d(p,p out ) out 'After that, first scan the current p out 'Outer neighbors until the current p out 'After scanning all the external neighbors, scan the next other external neighbor p of the point to be deleted out '; Add the points that meet the conditions for joining the candidate set to the candidate set in sequence according to the scanning order until the storage limit of the candidate set is reached.

[0093] like Figure 3 The figure shows a specific flow chart of constructing a candidate set in another embodiment of the present invention, setting the parameter C s , first input the point to be deleted p and the current outer neighbor p out , and then calculate the edges of point p one by one in the edge table of point p out distance, and the first distance p out Less than d(p,p out ) out ' is added to the candidate set C. If the size of the candidate set C reaches C s , then stop building the candidate set and return the current candidate set C, otherwise p out ′’s outer neighbors are located in the candidate region CR(p,p out ) points are added to the candidate set. In this process, if the size of C reaches C s Then stop building the candidate set and return the current candidate set C, otherwise the next distance p in the edge table of point p out Less than d(p,p out ) as the point p out ′ and repeat the process until the size of the candidate set C reaches C s Or all the outer neighbors of point p are scanned.

[0094] The following is the candidate region CR(p,p out ) are explained below.

[0095] d(c,p out ) <d(p,p out ) can ensure that the points in the candidate set are as close to p as possible out Since traditional graph index algorithms give higher priority to short edges when pruning edges, that is, they tend to retain shorter edges, in order to successfully repair the monotone search path, it is necessary to make the new connection to p out The edges of the graph index are not deleted as much as possible by the edge pruning strategy. In order to meet this condition as much as possible, the scheme requires that the points in the candidate set need to be at a distance of p out Close enough so that the newly connected edge is as short as possible, and the edge (p, p out ) as an edge that existed in the graph before deletion can be used as a criterion for judging whether it is close enough, that is, it can be considered that it is closer than the edge (p,p out )Longer edges are likely to be removed by the graph index edge pruning algorithm, while shorter edges are more likely to remain in the graph index.

[0096] d(c,p)>d(p,p out) can ensure that the points in the candidate set are as far away from point p as possible. Since the points farther away from point p are deleted, it is likely that the monotonic search path to point c still exists. The points in the candidate set can repair the path to point p. out The premise of a monotone path is that the point has a monotone out If a point in the candidate set does not even have a monotone search path to reach the point after point p is deleted, then it is impossible to repair the point to reach p. out Therefore, the points in the candidate set should still have a monotone search path after point p is deleted. To achieve this goal as much as possible, the scheme requires that the points in the candidate set are as far away from point p as possible.

[0097] d 2 (c,p out )+d 2 (p,p out )>d 2 (c,p) can ensure that the points in the candidate set are located on the side close to point p. out After that, the formed edges are monotone with respect to the endpoints of most of the broken monotone search paths. In order to repair the broken monotone search paths, it is necessary to establish monotone search paths that reach the outer neighbors of the point to be deleted and are also monotone with respect to the endpoints of the broken paths. A necessary condition for achieving this goal is that the newly connected edges to repair the monotone search paths are monotone with respect to the endpoints of these broken paths. For the points located on the side close to point p, since the points pointing from these points to p out The edge has the edge (p,p out ) components, so these edges can be monotonic to the endpoints of most broken paths.

[0098] Through the above method, although further screening is time-consuming, it can significantly improve the repair effect and obtain the most effective candidate set for repairing the monotonic path.

[0099] After determining the candidate set, connect the points in the candidate set to p out , so that p can be repaired out ’s inbound edge, and then the path is repaired.

[0100] Considering connecting a large number of points to p out This will seriously affect the update efficiency, because it is time-consuming to modify the edge table of a large number of points. In one embodiment, after obtaining the candidate set, in order to ensure the repair effect and improve the efficiency, valid points are further selected from the candidate set to connect to p out , that is, increase p by out The incoming edge:

[0101] First, the distance p in the current candidate set outThe nearest point Move into the result set and delete the candidates in the current candidate set that match the current point. The distance is less than p out The distance between the candidate set and the result set is 0 is 0. Repeat this step until the current candidate set is empty or the number of nodes in the result set reaches its upper limit.

[0102] Add a point from the result set to p out The outgoing edge.

[0103] For example, setting the upper limit of the result set δ in When screening the candidate set C, first sort the points in the candidate set according to the distance p out The distances are sorted in ascending order, and then the distance p in C is selected out The nearest point Add to the result set, then delete from C all Point c on the same side, that is, Then, the process of selecting the nearest point and excluding the points on the same side is repeated in the remaining candidate sets until the candidate set is empty or the result set size reaches the upper limit δ in Finally, connect the filtered results to p out , that is, add the edge table of the points in the result set to point to p out edge.

[0104] The reason for screening by the above method is that: since it is necessary to make the newly connected edge as monotone as possible to the end point of the destroyed monotone search path, the final connection to p out The edges of p should point in all directions as much as possible, so these new connections to p out The starting point of the edge should be located at p as much as possible. out In this embodiment, the points in the candidate set are regarded as p out The input edges are selected from the input edges using the above strategy. The above strategy can eliminate the edges with similar directions, so that the results after screening are evenly distributed in p out Therefore, the result after the above strategy screening can satisfy the requirement of making the newly connected edge monotone to the end point of the destroyed monotone search path as much as possible.

[0105] Since existing graph index algorithms usually limit the out-degree upper limit of each point, not every point in the filtered result set can be directly added to the edge table pointing to p out Therefore, when connecting the points in the result set to p out Afterwards, if the out-degree (number of out-edges) of the points in the result set exceeds the graph index out-degree threshold, the edge trimming algorithm is further executed to limit the out-degree, that is, the method further includes:

[0106] For the points in the result set whose outgoing edges are greater than the set upper limit, the edge reduction operation is performed according to the SNG edge trimming algorithm. The specific process is: for the point v in the result set whose outgoing edges are greater than the set upper limit, the edge reduction operation is performed. The edge reduction operation includes: removing the point closest to v in the current outer neighbor set of v Move v into the candidate set of outer neighbors and delete the candidates in the current set of outer neighbors that match the current point. The distance from is less than the distance from v and Repeat this step until the current outer neighbor set is empty or the number of nodes in its outer neighbor candidate set reaches the upper limit of the number of outgoing edges, and finally use its outer neighbor candidate set as the outer neighbor set of point v.

[0107] For example, the SNG edge trimming algorithm removes the original outgoing edges and pointing edges of these points to p out The edges of the node are taken as the candidate set of the outgoing edges of the node, and the outgoing degree upper limit θ of the graph index is set. d , according to the above-mentioned SNG edge trimming algorithm, delete the edges that do not meet the SNG conditions, and use the screening results as the new outgoing edges of the point.

[0108] Step S4: Delete all the points to be deleted and all their outgoing edges to update the graph index of the dynamic vector database.

[0109] Specifically, after executing the above process, the monotone search path with damaged graph index has been repaired, so all outgoing edges of the point to be deleted can be directly deleted, that is, the outgoing edge table of the point to be deleted is set to an empty table.

[0110] Finally, all resources occupied by the point to be deleted can be released. At this point, the deletion process is completed.

[0111] like Figure 4 FIG. 1 is a flowchart of detailed steps of a method for updating a neighbor graph index of a dynamic vector database in an embodiment of the present invention. In a specific embodiment, the method is executed as follows: first, a list of points to be deleted is input, then all the incoming edges of the points to be deleted in the graph are deleted, and then the points to be deleted are processed one by one. The processing process is: the currently processed point to be deleted is set as point p, and one of the outer neighbors p of point p is selected. out Repair the outer neighbors of the monotone search path for the current one and get p out The candidate set of new inner neighbors is then further screened through the following process: first, the distance p in the current candidate set is out The nearest point Move into the result set and delete the candidates in the current candidate set that match the current point. The distance is less than p out Repeat this step until the current candidate set is empty or the number of nodes in the result set reaches its upper limit; after obtaining the screening results, connect the points in the result set to pout , and the out-degree during the connection process exceeds the upper limit of the out-degree of the graph index θ d The point is subjected to the edge reduction operation according to the SNG edge trimming algorithm, and the existing set of external neighbors of the point is used as a candidate set and the number of neighbors less than θ is selected from them. d The outer neighbors of point p are taken as its final outer neighbor set. After the monotone search paths of all outer neighbors of point p are repaired according to the above process, the processing of the point p to be deleted is completed; after processing all the points to be deleted, the outgoing edges of the points to be deleted in the graph and the resources they occupy are released, and the deletion is completed.

[0112] like Figure 5 The figure shows the restoration point p after deleting the point p in one embodiment of the present invention. out Example of a candidate set obtained by the monotone search path. The yellow dots in the figure are the selected candidate points, where c1 and c2 are the distances p among the outer neighbors of point p. out The distance is less than d(p,p out ), c3, c4 are outer neighbors of c1, c2 and are located in the region CR (p, p out ). Figure 6 The figure shows an embodiment of the present invention in which a candidate set is selected and the selection result is connected to p out In the example diagram, the gray points are the points deleted from the candidate set, and the orange points are the final screening results. Figure 7 The figure shows an example diagram of the monotonic search path repair effect in an embodiment of the present invention, wherein s is the starting point, and the orange path in the figure is the repaired monotonic search path. It can be seen that after deleting point p in this example, the damaged monotonic search path in the figure is effectively repaired.

[0113] like Figure 8 The figure shows the experimental comparison of the present invention and FreshDiskAnn in indexing accuracy. Fig. 9 The figure shows the experimental comparison of the deletion throughput between the present invention and FreshDiskAnn. The experiments are based on the HNSW index (the existing approximate nearest neighbor search static index) and are conducted on the SIFT1M dataset. In each round, 1% of the points are randomly selected from the dataset and deleted from the index and then reinserted to ensure that the exact solution of the query remains unchanged. The same query set is used to test the index accuracy. Figure 8 It can be seen from the figure that the FreshDiskAnn index has a problem of decreased accuracy in the first few rounds of updates, while the index accuracy of the present invention is significantly improved after the update and maintained at a high accuracy. Fig. 9 It can be seen from the figure that the deletion throughput of the present invention is much higher than that of FreshDiskAnn, which shows that the present invention has a very high update efficiency.

[0114] Example 2

[0115] The present invention also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0116] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0117] Example 3

[0118] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0119] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0120] Example 4

[0121] The embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present invention.

[0122] The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0123] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for updating a neighbor graph index of a dynamic vector database, characterized in that: include: Get the points to be deleted that need to be deleted in the current graph index of the dynamic vector database; Deleting all incoming edges of the points to be deleted from the graph index; For each point p to be deleted that has an external neighbor, execute: Get the points within two hops from the current graph index starting from the point p to be deleted to construct p out The candidate set of new inner neighbors, p out For the outer neighbors of the point p to be deleted, connect the points in the candidate set to p out ; Delete all the points to be deleted and all their outgoing edges to update the graph index of the dynamic vector database; Among them, the outer neighbors of any point are the points connected to the outgoing edges of the point, and the inner neighbors of any point are the points connected to the incoming edges of the point.

2. The neighbor graph index updating method of the dynamic vector database according to claim 1, characterized in that: The step of deleting all incoming edges of the points to be deleted from the graph index includes: Scan each point in the current graph index in turn and delete the outgoing edge from the currently scanned point to the point to be deleted.

3. The neighbor graph index updating method of the dynamic vector database according to claim 1, characterized in that: Get the points within two hops from the point to be deleted p in the current graph index to construct p out The candidate set of new inner neighbors includes: Initialize the candidate set to be empty; According to the given point to be deleted, scan the outer neighbors p of the point to be deleted p in turn out , and after scanning the current p out After that, first scan the current p out The outer neighbor and the current p out After all the outer neighbors of point p are scanned, scan and delete the next outer neighbor p of point p. out ; Add the scanned points to the candidate set in sequence according to the scanning order until the storage limit of the candidate set is reached.

4. The neighbor graph index updating method of the dynamic vector database according to claim 1, characterized in that: Get the points within two hops from the point to be deleted p in the current graph index to construct p out The candidate set of new inner neighbors includes: The outer neighbors of the point to be deleted and the points that are related to p out The distance is less than d(p,p out ) are added to the candidate set, and points that are outer neighbors of the outer neighbors of the point to be deleted p and that simultaneously meet three set conditions are added to the candidate set, wherein the three set conditions include: d(c,p out )<d(p,p out ) d(c,p)>d(p,p out ) d 2 (c,p out )+d 2 (p,p out )>d 2 (c,p) In the formula, p is the point to be deleted, p out is the outer neighbor of p, c is the point of p in the candidate set, d(c,p out ) represents c and p out The distance in the graph index, d(p,p out ) for p and p out The distance in the graph index, d(c,p) is the distance between c and p in the graph index.

5. The neighbor graph index updating method of the dynamic vector database according to claim 4, characterized in that: Get the points within two hops from the point to be deleted p in the current graph index to construct p out The candidate set of new inner neighbors includes: Initialize the candidate set to be empty; According to the given point p to be deleted and one of its outer neighbors p out , scan the other outer neighbors p of the point to be deleted in turn out ', when scanning to the same out The distance is less than d(p,p out ) out 'After that, first scan the current p out 'Outer neighbors until the current p out 'After scanning all the external neighbors, scan the next other external neighbor p of the point to be deleted out '; Add the points that meet the conditions for joining the candidate set to the candidate set in sequence according to the scanning order until the storage limit of the candidate set is reached.

6. The neighbor graph index updating method of the dynamic vector database according to claim 1, characterized in that: The point in the candidate set is connected to p out ,include: First, the distance p in the current candidate set out The nearest point Move into the result set and delete the candidates in the current candidate set that match the current point. The distance is less than p out The distance between the candidate set and the result set is 0 is 0, and this step is repeated until the current candidate set is empty or the number of nodes in the result set reaches its upper limit; Add a point from the result set to p out The outgoing edge.

7. The method for updating neighbor graph index of a dynamic vector database according to claim 1, characterized in that: The method further comprises: For the point v in the result set whose outgoing edge is greater than the set upper limit, an edge reduction operation is performed, wherein the edge reduction operation includes: removing the point closest to v in the current outer neighbor set of v Move v into the candidate set of outer neighbors and delete the candidates in the current set of outer neighbors that match the current point. The distance from is less than the distance from v and Repeat this step until the current outer neighbor set is empty or the number of nodes in its outer neighbor candidate set reaches the upper limit of the number of outgoing edges, and finally use its outer neighbor candidate set as the outer neighbor set of point v.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images