A dynamic nearest neighbor search method for high-dimensional space vectors based on tree-graph structure

Abstract

The present invention discloses a high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure, comprising: obtaining a high-dimensional space vector set, constructing a global tree index for all vector objects, and constructing a lightweight hierarchical graph index based on the middle level of the tree index to complete the construction of the tree-graph structure; obtaining vector objects to be inserted and deleted, updating the global tree index and the lightweight hierarchical graph index, and completing the dynamic insertion and deletion of the tree-graph structure; using the tree-graph structure to perform an approximate nearest neighbor search or an exact nearest neighbor search of a high-dimensional space vector object, and obtaining a given query vector object. k The present invention can significantly reduce the cost of index construction, flexibly cope with real-time data updates, and support efficient and universal high-dimensional space vector nearest neighbor search.

Description

Technical Field

[0001] The invention belongs to the technical field of database and data mining, and in particular relates to a high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure. Background Art

[0002] Nearest neighbor search is the basis of high-dimensional space retrieval, and has a profound impact on many fields such as pattern recognition, information retrieval, recommendation systems, and vector database management. In the existing technology, graph-based methods undoubtedly show their advantages in query accuracy and efficiency; however, despite their good performance, such methods have to face the dual challenges of high construction cost and complex dynamic data update, which limits their wide application in ever-changing practical scenarios.

[0003] In recent years, academia and industry have conducted many explorations, trying to combine graph methods with hashing, quantization and tree methods to address these problems; although these efforts have alleviated some of the problems to a certain extent, these index structures occupy a large storage space and have slow data updates, which are still key factors restricting the further development of the technology. Therefore, how to design a high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure to achieve lightweight index construction and dynamic real-time update, and support efficient and general nearest neighbor search, has become a difficult problem that needs to be solved urgently in academia and industry. For example, in recommendation systems and e-commerce platforms, a large amount of product information (including pictures, texts, product information, etc.) may be represented as a high-dimensional vector set. The platform needs to perform nearest neighbor search in the product vector set to find the product vector that is closest to the user's frequently purchased products in order to achieve high-quality personalized product recommendations; at the same time, the listing and delisting operations of products on the platform are relatively frequent, that is, it is necessary to frequently perform dynamic operations of data insertion and deletion in the existing data management system.

[0004] Among the existing nearest neighbor search methods for high-dimensional vectors, graph-based methods usually perform better in terms of query accuracy and efficiency. However, such methods face many challenges, including high index construction costs and dynamic update overhead. In recent years, some researchers have tried to combine graph methods with hashing, quantization, and tree methods to address these issues. For example, the clustering-based tree graph combined with index method first uses tree indexes to partition data and constructs corresponding subgraphs in the leaf nodes after partitioning to reduce the graph construction cost. However, this method is inefficient when dealing with frequent data updates. MNG (Monotonic Neighborhood Graph) uses inner product quantization technology to accelerate graph navigation and introduces a delayed deletion strategy to support graph updates. However, the delayed deletion strategy requires regular index reconstruction, resulting in limited update performance. In addition, building a complete graph index on the entire data set will also take up a lot of memory space, resulting in performance bottlenecks. In general, the above methods have technical defects such as large storage space occupied by the index structure and difficulty in coping with real-time data updates.

[0005] In summary, the problem of how to achieve effective data updates and support efficient data retrieval with low index space overhead has not been fundamentally solved, resulting in limited practical application scenarios; for example, e-commerce platforms find it difficult to update product information in a timely manner, resulting in delayed or even incorrect personalized recommendation information for users. Summary of the invention

[0006] In view of the above, the present invention provides a high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure, which can effectively reduce the cost of index construction, support flexible real-time updates to cope with scenarios where data changes frequently, and on this basis realize efficient and general high-dimensional vector nearest neighbor search.

[0007] A high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure comprises the following steps:

[0008] (1) Obtain a high-dimensional space vector set, build a global tree index for all vector objects in the set, and build a lightweight hierarchical graph index for the first layer of the global tree index to complete the construction of the tree-graph structure;

[0009] (2) Obtain the vector objects to be inserted and deleted, update the global tree index and lightweight hierarchical graph index, and realize dynamic insertion and deletion of the tree-graph structure;

[0010] (3) Use the tree-graph structure to perform approximate nearest neighbor search or exact nearest neighbor search for vector objects in high-dimensional space to obtain the nearest neighbor of a given query vector object. k -Neighbor objects.

[0011] Furthermore, the specific implementation process of step (1) is as follows:

[0012] S11: Obtain a high-dimensional space vector set, node capacity parameters, and graph sparsity parameters, and initialize the global tree index and lightweight graph index to empty sets;

[0013] S12: Select any unaccessed vector object, create a leaf entry for it according to the M-tree index strategy, and use the vector object as the center point of the leaf entry;

[0014] S13: inserting the leaf entry into the global tree index according to the M-tree index strategy, where the number of entries accommodated by each node in the global tree index does not exceed the node capacity parameter;

[0015] S14: repeating steps S12 to S13 until all vector objects in the high-dimensional space vector set are inserted into the global tree index, that is, the construction of the global tree index is completed;

[0016] S15: Obtain the first layer of the global tree index, that is, all entries of the layer above the layer (zeroth layer) where the leaf entry is located; initialize the in-degree radius table of the lightweight hierarchical graph index to be empty, and the in-degree radius table stores the radius of the in-degree edge (reverse neighbor) of each vertex on the lightweight hierarchical graph;

[0017] S16: randomly select an unvisited first-layer entry, insert its center point as a vertex into the lightweight hierarchical graph index according to the strategy of the HNSW (Hierarchical Navigable Small World) hierarchical graph index, and update the in-degree radius table; the maximum number of neighbors of each vertex on the lightweight hierarchical graph index is controlled by the graph sparsity parameter;

[0018] S17: adding a pointer to the vertex at the bottom layer of the lightweight hierarchical graph index, pointing to the leaf node corresponding to the first-layer entry in the global tree index;

[0019] S18: Repeat steps S16 to S17 until all entries of the first layer of the global tree index are inserted into the lightweight hierarchical graph index, thus completing the construction of the lightweight hierarchical graph index;

[0020] S19: Merge the global tree index and the lightweight hierarchical graph index to obtain a tree-graph structure.

[0021] Furthermore, in step (2), for the vector object to be inserted, the specific process of implementing the dynamic insertion of the tree-graph structure is as follows:

[0022] S21: Obtain the vector object to be inserted, and obtain the information of the global tree index according to the current tree-graph structure;

[0023] S22: according to the M-tree index strategy, create a leaf entry for the vector object to be inserted, and use the vector object as the center point of the leaf entry;

[0024] S23: inserting the leaf entry into the global tree index according to the M-tree index strategy; if the insertion of the leaf entry does not result in the splitting of the leaf node in the global tree index, that is, the first layer of the global tree index does not change, there is no need to update the lightweight hierarchical graph index, and the process goes to step S26; if the insertion of the leaf entry results in the splitting of the leaf node in the global tree index, that is, a new entry is generated in the first layer of the global tree index, the lightweight hierarchical graph index needs to be updated, that is, steps S24 to S25 are executed;

[0025] S24: Obtain information of the lightweight hierarchical graph index according to the current tree-graph structure, and according to the strategy of the HNSW hierarchical graph index, insert the center point of the new entry of the first layer of the global tree index as a vertex into the lightweight hierarchical graph index, and update the in-degree radius table;

[0026] S25: adding a pointer for the vertex at the bottom layer of the lightweight hierarchical graph index, pointing to the leaf node corresponding to the new entry of the first layer in the global tree index;

[0027] S26: The updated global tree index and lightweight hierarchical graph index are merged and inserted to complete the dynamic insertion of the tree-graph structure.

[0028] Furthermore, in step (2), for the vector object to be deleted, the specific process of implementing dynamic deletion of the tree-graph structure is as follows:

[0029] S31: obtaining the vector object to be deleted and the search candidate set parameters, and obtaining the information of the global tree index and the lightweight hierarchical graph index according to the current tree-graph structure;

[0030] S32: using the current tree-graph structure, taking the vector object to be deleted as the query object, locating the query object through approximate nearest neighbor search (returning the approximate 1-neighbor of the query object); if the vector object is successfully found in the tree-graph structure, the process proceeds to step S34; if it is not successfully found, it is necessary to further locate it using range search, i.e., executing step S33;

[0031] S33: taking the vector object to be deleted as the center, performing an accurate range query with a radius of 0 on the global tree index to locate the vector object; if the vector object cannot be successfully found in the global tree index, it means that the vector object does not exist, and the deletion operation is terminated immediately; if it is successfully found, continue to execute step S34;

[0032] S34: according to the deletion rule of the M-tree index strategy, the successfully located vector object to be deleted is deleted from the global tree index; if the deletion of the vector object does not cause the leaf node underflow in the global tree index, that is, the first layer of the global tree index does not change, there is no need to update the lightweight hierarchical graph index, and the process goes to step S38; if the deletion of the vector object causes the leaf node underflow in the global tree index, that is, an entry in the first layer of the global tree index is deleted, the lightweight hierarchical graph index needs to be updated, that is, step S35 is executed;

[0033] S35: Identify the center of the deleted entry in the first layer of the global tree index, that is, the vertex to be deleted in the lightweight hierarchical graph index, and delete the pointer in the vertex pointing to the leaf node corresponding to the deleted entry in the first layer of the global tree index at the bottom layer of the lightweight hierarchical graph index;

[0034] S36: Taking the vertex to be deleted as the center, perform an accurate range query in the global tree index, where the radius is the in-degree radius stored in the in-degree radius table of the vertex, so as to find all in-degree edges of the vertex to be deleted in the lightweight hierarchical graph index;

[0035] S37: Delete the vertex and its corresponding edges from the lightweight hierarchical graph index, including the in-degree edges obtained in S36 and the out-degree edges directly obtained in the lightweight hierarchical graph index, and update the in-degree radius table; if the number of neighbors of some vertices in the lightweight hierarchical graph index is reduced after deleting the vertex and edge, resulting in a violation of the graph construction rules, then reinsert these vertices with too few neighbors into the lightweight hierarchical graph index according to the strategy of the HNSW hierarchical graph index;

[0036] S38: Merge and delete the updated global tree index and lightweight hierarchical graph index to complete the dynamic deletion of the tree-graph structure.

[0037] Furthermore, the specific implementation process of the approximate nearest neighbor search in step (3) is as follows:

[0038] S41: Get the given query vector object, tree-graph structure, search candidate set parameters and k - Neighborhood parameters;

[0039] S42: Obtain information of a global tree index and a lightweight hierarchical graph index according to the tree-graph structure;

[0040] S43: Initialize the search candidate set to an empty set;

[0041] S44: Randomly select a vertex at the top level of the lightweight hierarchical graph index as the entry point for the search;

[0042] S45: performing a greedy search at the current layer starting from the entry point, and continuously expanding the neighbors of the vertex until an approximate 1-neighbor of the query vector object at the current layer is found, and the length of the candidate set is set to 1 during the search process;

[0043] S46: Using the found approximate 1-nearest neighbor as the entry starting point of the next layer of the lightweight hierarchical graph index;

[0044] S47: Repeat steps S45 to S46 until the first layer is searched from the top layer of the lightweight hierarchical graph index (the bottom layer of the lightweight hierarchical graph index is the zeroth layer);

[0045] S48: Taking the approximate 1-nearest neighbor found in the first layer as the entry starting point of the bottom layer search, greedy search is performed at the bottom layer, and the search results are added to the search candidate set, the length of the candidate set is the search candidate set parameter; the bottom layer search includes not only directly accessing the vertex neighbors on the bottom layer, but also accessing the leaf nodes in the global tree index pointed to by the bottom layer vertices;

[0046] S49: Find the closest object to the query vector from the search candidate set k nearest neighbors as the final search result.

[0047] Furthermore, the specific implementation process of the accurate nearest neighbor search in step (3) is as follows:

[0048] S51: Get the given query vector object, tree-graph structure, search candidate set parameters and k - Neighborhood parameters;

[0049] S52: Use the tree-graph structure to perform approximate nearest neighbor search to find the approximate nearest neighbor of the query vector object k - immediate neighbours;

[0050] S53: Approximately k - The distance from the nearest neighbor to the query vector object is used as the initial radius to perform precise calculations on the global tree index. k -Nearest neighbor search to get the final search results.

[0051] A high-dimensional space vector dynamic nearest neighbor search system based on a tree-graph structure, comprising:

[0052] The tree-graph structure construction module obtains a high-dimensional space vector set to build a global tree index for all vector objects in the set, and builds a lightweight hierarchical graph index for the first layer of the global tree index to complete the construction of the tree-graph structure.

[0053] The tree-graph structure update module obtains the vector objects to be inserted and deleted, which are used to update the global tree index and lightweight hierarchical graph index to realize the dynamic insertion and deletion of the tree-graph structure;

[0054] The nearest neighbor search module uses the tree-graph structure to perform approximate nearest neighbor search or exact nearest neighbor search for high-dimensional space vector objects to obtain the nearest neighbor of a given query vector object. k -Neighbor objects.

[0055] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure.

[0056] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure.

[0057] The present invention designs a tree-graph structure for nearest neighbor search of high-dimensional vectors, including a global tree index built for all vector objects and a lightweight hierarchical graph index built for the first layer of the tree. This design significantly reduces the time and space cost of index construction. At the same time, the present invention designs an effective update strategy for dynamic scenes, including data insertion and data deletion algorithms. These algorithms support real-time dynamic updates on the tree-graph structure, solving the update dilemma of existing methods. Therefore, on the basis of significantly reducing the cost of index construction and supporting dynamic updates, the present invention uses the tree-graph structure to design an efficient approximate nearest neighbor search algorithm and an exact nearest neighbor search algorithm, whose performance is comparable to that of the most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the process of a high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure in an embodiment of the present invention.

[0059] Figure 2 It is a structural block diagram of a high-dimensional space vector dynamic nearest neighbor search system based on a tree-graph structure in an embodiment of the present invention.

[0060] Figure 3 The present invention is a schematic diagram of the structure of an electronic device implemented according to the method of the present invention. DETAILED DESCRIPTION

[0061] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0062] like Figure 1 As shown, the high-dimensional space vector dynamic nearest neighbor search method based on the tree-graph structure in this embodiment includes the following steps:

[0063] Step S1: Obtain a high-dimensional space vector set, build a global tree index for all vector objects, and build a lightweight hierarchical graph index for the first layer of the tree to complete the construction of the tree-graph structure.

[0064] The high-dimensional space vector set is composed of many high-dimensional space vector objects, which can be generated by a variety of practical scenarios. For example, in recommendation systems and e-commerce platforms, users and items can be represented as high-dimensional vectors; in image processing, each image can be converted into a high-dimensional vector; in bioinformatics, gene sequences can be converted into high-dimensional vectors. The specific implementation process of step S1 in this embodiment is as follows:

[0065] S11: Obtain a high-dimensional space vector set, a node capacity parameter, and a graph sparsity parameter. Specifically, the high-dimensional space vector set is composed of high-dimensional space vector objects generated by multiple actual scenarios; the node capacity parameter represents the maximum number of entries that a node in the global tree index can accommodate; and the graph sparsity parameter is used to limit the maximum number of neighbors of each vertex on the lightweight hierarchical graph index.

[0066] S12: Initialize the global tree index and lightweight graph index to empty sets.

[0067] S13: Randomly select an unvisited vector object, create a leaf entry for the object according to the M-tree index strategy, and use the object as the center point of the current leaf entry. Specifically, the M-tree index is an efficient dynamic tree index structure. Each node in the tree contains multiple entries. Each entry represents a hypersphere, and its internal subtrees or data objects maintain local neighborhood relationships. The leaf entry in the leaf node stores the data object as the center point and the distance from the data object to the center of its parent entry. d (initialized to infinity); in addition to the above information, the non-leaf entry in the non-leaf node also stores the center point data object of the hypersphere, the radius of the hypersphere, and a pointer to the subtree node.

[0068] S14: According to the M-tree index strategy, the current leaf entry is inserted into the global tree index, and the number of entries contained in each node in the tree does not exceed the node capacity parameter. Specifically, the M-tree index is constructed incrementally. When the number of entries in a node exceeds the node capacity parameter, the tree will allocate a new node at the same level and redistribute these entries to the original node and the new node; at the same time, the upper layer of the tree needs to generate a corresponding new center point and adjust the radius, that is, a new entry is inserted into the upper parent node; if the split operation causes the parent node to reach its capacity limit, it needs to be recursively split upward.

[0069] S15: Repeat steps S13 to S14 until all objects in the high-dimensional space vector set are inserted into the tree, thus completing the construction of the global tree index.

[0070] S16: Obtain the first layer of the global tree index, that is, all entries of the layer above the layer where the leaf entry is located (layer 0); specifically, find all nodes of the first layer in the global tree index and obtain the entries stored in each node.

[0071] S17: Initialize the in-degree radius table of the lightweight hierarchical graph index to be empty. The in-degree radius table stores the in-degree edge (reverse neighbor) radius of each vertex on the graph. Specifically, the in-degree radius table stores the maximum distance between each vertex on the graph and its reverse neighbor (in-degree edge) between all layers, i.e., the in-degree radius, which helps locate the reverse neighbor of each vertex to support deletion operations on the graph.

[0072] S18: Randomly select an unvisited first-level entry, and according to the strategy of HNSW hierarchical graph index, insert the center point of the first-level entry as a vertex into the graph index, and update the in-degree radius table. The maximum number of neighbors for each vertex on the graph index is controlled by the graph sparsity parameter.

[0073] Specifically, HNSW hierarchical graph index is an advanced nearest neighbor search method. The topmost layer of the graph contains the fewest vertices and the longest edges. As the number of layers decreases, the edges become shorter and the number of vertices increases. The vertices in the upper layer are always included in the lower layer, and the bottom layer (zeroth layer) contains all vertices. The graph is constructed step by step. For each new vertex, a random integer is selected. l , defines the maximum number of layers a vertex belongs to, e.g. l = 2 means that the vertex appears only in two layers, namely the zeroth layer and the first layer; then, using top-down graph search, neighbors are found for the vertex in each layer and a neighbor graph is gradually built within the layer. In the upper layers of the graph, the maximum number of neighbors of a vertex is the graph sparsity parameter, while in the lowest layer it is twice that of the upper layers.

[0074] S19: At the bottom layer of the graph index, add a pointer to the current vertex, pointing to the leaf node corresponding to the current first-layer entry in the tree index. Specifically, each vertex of the bottom-level graph corresponds to the center point of an entry in the first layer of the tree; in addition, each non-leaf entry in the tree has a pointer to a subtree node, and the leaf node corresponding to the first-layer entry is the leaf node pointed to by its pointer. Adding pointers from the bottom-level graph vertices to the leaf nodes can enhance the quality of subsequent nearest neighbor searches.

[0075] S110: Repeat steps S18 to S19 until all entries in the first layer of the tree index are inserted into the graph index, thus completing the construction of the lightweight hierarchical graph index.

[0076] S111: Merge the global tree index and the lightweight hierarchical graph index to obtain the final tree-graph structure.

[0077] Step S2: Get the vector objects to be inserted and deleted, update the global tree index for all vector objects and the lightweight hierarchical graph index for the first layer of the tree, and complete the dynamic insertion and deletion of the tree-graph structure.

[0078] In actual application scenarios, such as recommendation systems or e-commerce platforms, data insertion (i.e., listing of products) and data deletion (i.e., delisting of products) operations are often very frequent. In this embodiment, for the insertion or deletion of objects, the global tree index for all vector objects and the lightweight hierarchical graph index for the first layer of the tree are updated respectively to complete the dynamic insertion or deletion of the tree-graph structure.

[0079] In this embodiment, for the vector object to be inserted, the specific implementation process of completing the dynamic insertion of the tree-graph structure is as follows:

[0080] S211: Obtain the vector object to be inserted and the current tree-graph structure; specifically, the vector object to be inserted is the high-dimensional space vector object to be inserted, and the current tree-graph structure is the tree-graph index composed of the currently existing vector objects (sets).

[0081] S212: Obtain global tree index information according to the current tree-graph structure.

[0082] S213: According to the M-tree index strategy, a leaf entry is created for the vector object to be inserted, and the object to be inserted is used as the center point of the current leaf entry.

[0083] S214: According to the strategy of M-tree index, insert the current leaf entry into the tree index; if the insertion of the leaf entry does not lead to the splitting of the leaf node in the tree index, that is, the first layer of the tree has not changed, there is no need to update the graph index, and the execution goes to step S218; if it leads to the splitting of the leaf node, that is, a new entry is generated in the first layer, the graph index needs to be updated, that is, steps S215-S217 are executed. Specifically, the tree is constructed incrementally, so inserting entries into the tree is consistent with the previous construction process; since the graph is constructed on the first layer of the tree, if the first layer of the tree has not changed, the graph does not need to be updated; otherwise, the newly generated entries in the first layer of the tree need to be converted into new corresponding vertices and inserted into the graph.

[0084] S215: According to the current tree-graph structure, obtain information of the lightweight hierarchical graph index.

[0085] S216: According to the strategy of HNSW hierarchical graph index, the center point of the new entry in the first layer is inserted into the lightweight hierarchical graph index as a vertex, and the in-degree radius table is updated; specifically, the graph is constructed incrementally, so inserting vertices into the graph is consistent with the previous construction process.

[0086] S217: At the bottom layer of the graph index, a pointer is added for the current vertex, pointing to the leaf node corresponding to the new entry in the first layer of the tree index.

[0087] S218: Merge and insert the updated global tree index and lightweight hierarchical graph index to obtain the final tree-graph structure.

[0088] In this implementation, for the vector object to be deleted, the specific implementation process of completing the dynamic deletion of the tree-graph structure is as follows:

[0089] S221: Obtain the vector object to be deleted, the current tree-graph structure and the search candidate set parameters. Specifically, the vector object to be deleted is the high-dimensional space vector object to be deleted; the current tree-graph structure is the tree-graph index composed of the current vector objects (sets); the search candidate set parameters determine the length of the candidate set during the nearest neighbor search. The larger the search candidate set parameters, the higher the search quality and the longer the search time.

[0090] S222: According to the current tree-graph structure, obtain information of the global tree index and the lightweight hierarchical graph index.

[0091] S223: Using the current tree-graph structure, taking the vector object to be deleted as the query object, perform an approximate 1-neighbor search (i.e., return the approximate 1-neighbor of the query object) to locate the object to be deleted; if the object to be deleted is successfully found in the tree-graph structure, execute to step S225; if it is not successfully found, it is necessary to further use range search to locate it, i.e., execute step S224.

[0092] Specifically, before deleting an object, it is first necessary to locate the position of the deleted object in the tree-graph structure. The approximate 1-nearest neighbor search approximately returns an object in the tree-graph structure. The returned object is closest to the query object (the object to be deleted) and is the approximate 1-nearest neighbor of the query object (the object to be deleted). If the returned object is consistent with the object to be deleted, the object to be deleted is successfully located. Otherwise, it is not successfully located and a range search is needed to further find the object to be deleted.

[0093] S224: With the object to be deleted as the center, perform an accurate range query with a radius of 0 on the tree index to locate the object to be deleted; if the object to be deleted is not successfully found in the tree index, it means that the object to be deleted does not exist, and the deletion operation is terminated immediately; if it is successfully found, continue to step S225. Specifically, accurate range query is a common query method in tree index, which will return all objects whose distance from the query object does not exceed the query radius; if the object to be deleted is in the tree-graph structure, that is, in the tree index, the accurate range query will definitely return the result correctly; on the contrary, if the object to be deleted is still not found, it means that the object to be deleted does not exist in the tree-graph structure, and the deletion operation ends early.

[0094] S225: According to the deletion rule of the M-tree index, the successfully located object to be deleted is deleted from the tree index; if the deletion of the object does not cause the underflow of the leaf node in the tree index, that is, the first layer of the tree has not changed, there is no need to update the graph index, and the execution goes to step S231; if it causes the leaf node to overflow, that is, an entry in the first layer of the tree is deleted, then the graph index needs to be updated, that is, steps S226-S230 are executed. Specifically, if the deletion operation causes the leaf entry of the leaf node in the tree to be deleted, and then causes the deletion of the leaf node and its parent entry pointing to the leaf node in the first layer, then the graph needs to be updated; otherwise, there is no change in the first layer of the tree, that is, there is no change in the structure of the graph.

[0095] S226: Identify the center of the first level of deletion entries in the tree, that is, the vertex to be deleted in the graph.

[0096] S227: At the bottom layer of the graph index, delete the pointer in the vertex to be deleted that points to the leaf node corresponding to the first-layer deletion entry in the tree.

[0097] S228: With the vertex to be deleted as the center, perform an exact range query in the tree, with the radius being the in-degree radius of the vertex to be deleted stored in the in-degree radius table, to find all in-degree edges of the vertex to be deleted in the graph. Specifically, the exact range query returns all objects whose distance to the deleted vertex does not exceed the in-degree radius, i.e., the maximum distance between the deleted vertex and its reverse neighbors (in-degree edges) at all levels; that is, the result of the range query includes all reverse neighbors of the object to be deleted on different levels in the graph, i.e., in-degree edges.

[0098] S229: Delete the vertices to be deleted and their corresponding edges from the graph, and update the in-degree radius table; specifically, delete the vertices to be deleted and the corresponding edges from the graph, including: deleting the vertices, deleting the out-degree edges and in-degree edges of the vertices; deleting the vertices and out-degree edges is direct because this information is directly stored in the adjacency list of the graph, while the in-degree edges are found and deleted through the previous precise range query.

[0099] S230: If deleting points and edges causes the number of neighbors of some vertices in the graph to decrease, resulting in a violation of the graph construction rules, then according to the strategy of the HNSW graph, these vertices with too few neighbors are reinserted into the graph. Specifically, if the number of neighbors of some vertices in the graph is too small, the performance of the graph will be degraded, violating the graph construction rules; at the same time, visiting the neighbors of these vertices does not guarantee that their true nearest neighbors will be found. In order to solve this problem, these vertices need to be reinserted into the graph to re-establish their correct neighbor sets.

[0100] S231: Merge and delete the updated global tree index and lightweight hierarchical graph index to obtain the final tree-graph structure.

[0101] Step S3: Using the tree-graph structure, perform an approximate nearest neighbor search or an exact nearest neighbor search for a high-dimensional space vector object to obtain the given query vector object. k -Neighbor objects.

[0102] Nearest neighbor search is a basic problem in database management and artificial intelligence methods, and is widely used in high-dimensional spaces. For example, in a set of product vectors, the product vector closest to the user's frequently purchased products is searched to achieve accurate product recommendations; in face recognition systems, each person's facial features are converted into high-dimensional vectors, and these vectors can be compared to achieve fast face recognition and verification; in medicine, protein structural information is converted into high-dimensional vectors, and protein vectors with known functions are used for nearest neighbor search to predict the function of unknown proteins. Accurate k -The nearest neighbor search problem is defined as: given a set of objects in a high-dimensional space O , a query object q , a distance function d and an integer k , k -Nearest neighbor search in O Find a file containing k A collection of objects S , S The distance of the object in q Recently. In form, k -Nearest neighbor search .

[0103] Accurately perform k -Nearest neighbor search can lead to a lot of computational overhead. A more efficient and practical alternative is approximate nearest neighbor search, which is defined as follows: Given a set of objects in a high-dimensional space O , a query object q , a distance function d , an integer k and a very small constant ε>0, approximately k -Nearest neighbor search in O Find a file containing k A collection of objects S , so that the approximation k -Nearest neighbor search ,in o q Is a collection O middle q In order to simplify modeling and evaluation, recall is often used in practice to replace the exact value of estimating ε.

[0104] The specific implementation process of the approximate nearest neighbor search in this implementation is as follows:

[0105] S311: Obtain query object, tree-graph structure, search candidate set parameters and k - Nearest neighbor parameter; Specifically, the query object is the high-dimensional space vector object to be queried in the actual scene; the tree-graph structure is a tree-graph index composed of the currently existing vector objects (sets); the search candidate set parameter determines the length of the candidate set during the nearest neighbor search. The larger the search candidate set parameter, the higher the search quality and the longer the search time; k - The neighbor parameter indicates the query object to be returned k Approximate nearest neighbors.

[0106] S312: According to the tree-graph structure, obtain information of the global tree index and the lightweight hierarchical graph index.

[0107] S313: Initialize the search candidate set to an empty set.

[0108] S314: Randomly select a vertex at the top level of the graph index as the entry starting point for the search.

[0109] S315: Starting from the current starting point, perform greedy search in the current layer, and continuously expand the neighbors of the vertex until the approximate 1-neighbor of the query object in the current layer is found, and the length of the candidate set of the greedy search is 1. Specifically, each layer of the graph contains an approximate neighbor graph, and the process of greedy search on the neighbor graph of the current layer is: starting from the starting point, check all its neighbors one by one to find the neighbor closest to the query; take the nearest neighbor as the new starting point, and repeat the above process until there is no neighbor closer to the query; then the last starting point is the approximate 1-neighbor of the query.

[0110] S316: Use the approximate 1-neighbor found as the entry starting point of a lower layer of the graph.

[0111] S317: Repeat steps S315-S316 until the first layer is searched from the top layer of the graph index (the bottom layer of the graph is the zeroth layer).

[0112] S318: Use the approximate 1-nearest neighbor found in the first layer of the graph as the search entry starting point for the bottom layer of the graph, perform a greedy search on the bottom layer of the graph, and add the result to the search candidate set. The length of the candidate set is the search candidate set parameter. Unlike before, the search of the bottom layer of the graph not only includes direct access to the vertex neighbors on the bottom layer of the graph, but also includes access to the leaf nodes in the tree pointed to by the vertices of the bottom layer of the graph.

[0113] Specifically, during the search process of the underlying graph, we not only visit the vertices on the underlying graph, but also the leaf nodes in the tree pointed to by the vertex pointers. By simultaneously utilizing the global neighborhood relationship in the graph and the local neighborhood relationship of the leaf nodes in the tree, we can ensure the comprehensive coverage of the data set and ultimately improve the quality of the query results. It is worth noting that visiting the leaf nodes will not affect the convergence process of the greedy search of the underlying graph, but only add the corresponding leaf node objects to the search candidate set.

[0114] S319: Find the closest candidate to the query object in the search candidate set k nearest neighbors as the final search result.

[0115] The specific implementation process of the accurate nearest neighbor search in this embodiment is as follows:

[0116] S321: Obtain query object, tree-graph structure, search candidate set parameters and k - Nearest neighbor parameter; specifically, the query object is the high-dimensional space vector object to be queried, the tree-graph structure is a tree-graph index composed of the currently existing vector objects (sets), and the search candidate set parameter determines the length of the candidate set during the nearest neighbor search. The larger the search candidate set parameter, the higher the search quality and the longer the search time; k - The neighbor parameter indicates the query object to be returned k Exact nearest neighbors.

[0117] S322: Using the tree-graph structure, perform an approximate nearest neighbor search to find the approximate nearest neighbor of the query object. k -Neighbors.

[0118] S323: Approximate k -The distance from the nearest neighbor to the query object is used as the initial radius, and the exact k -Nearest neighbor search to get the final search results.

[0119] Specifically, the traditional tree indexing method will accurately k -Nearest neighbor search is considered a special range query. At the beginning of the search, the radius is set to infinity; as the search progresses, the radius is updated to the current first neighbor. k -The distance of the nearest neighbor is gradually reduced until the true k However, a radius that is too large in the early stages of the search may lead to inefficient tree pruning, resulting in unnecessary computations. To address this inefficiency, we use the approximation from the above k -High-quality approximation obtained from the nearest neighbor search algorithm k -The distance of the nearest neighbor is used to initialize the search radius. Such a high-quality initial radius helps improve the pruning efficiency and speed up the search process.

[0120] like Figure 2 As shown, the high-dimensional space vector dynamic nearest neighbor search system based on the tree-graph structure of this embodiment includes:

[0121] The tree-graph structure construction module obtains a set of high-dimensional space vectors to build a global tree index for all vector objects and a lightweight hierarchical graph index for the first layer of the tree to complete the construction of the tree-graph structure.

[0122] The tree-graph structure update module obtains the vector objects to be inserted and deleted, and is used to update the global tree index for all vector objects and the lightweight hierarchical graph index for the first layer of the tree, thereby completing the dynamic insertion and deletion of the tree-graph structure.

[0123] The nearest neighbor search module uses a tree-graph structure to perform approximate nearest neighbor search or exact nearest neighbor search for vector objects in high-dimensional space to obtain the nearest neighbor of a given query vector object. k -Neighbor objects.

[0124] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0125] Accordingly, the present invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-mentioned high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure. Figure 3 The figure shows a hardware structure of any electronic device with data processing capability where the high-dimensional space vector dynamic nearest neighbor search system based on tree-graph structure provided by an embodiment of the present invention is located. Figure 3 In addition to the processor, memory, internal bus and network interface shown, the electronic device with data processing capability in which the system in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0126] Accordingly, the present invention also provides a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the high-dimensional space vector dynamic nearest neighbor search method based on the tree-graph structure as described above is implemented. The computer-readable storage medium can be an internal storage unit of the aforementioned electronic device with data processing capabilities, such as a hard disk or a memory; or it can be other external storage devices, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. The computer-readable storage medium can also include both an internal storage unit of an electronic device with data processing capabilities and an external storage device, which is used to store computer programs and other programs and data required by the electronic device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0127] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure, applied to a face recognition system, the high-dimensional space vector dynamic nearest neighbor search method comprising the following steps: (1) Obtain a high-dimensional space vector set, build a global tree index for all vector objects in the set, and build a lightweight hierarchical graph index for the first layer of the global tree index to complete the construction of the tree-graph structure. The specific implementation process is as follows: S11: Obtain a high-dimensional space vector set, node capacity parameters, and graph sparsity parameters, and initialize the global tree index and lightweight graph index to empty sets; S12: Select any unaccessed vector object, create a leaf entry for it according to the M-tree index strategy, and use the vector object as the center point of the leaf entry; S13: inserting the leaf entry into the global tree index according to the M-tree index strategy, where the number of entries accommodated by each node in the global tree index does not exceed the node capacity parameter; S14: repeating steps S12 to S13 until all vector objects in the high-dimensional space vector set are inserted into the global tree index, that is, the construction of the global tree index is completed; S15: Obtain the first layer of the global tree index, that is, all entries of the layer above the layer where the leaf entry is located; initialize the in-degree radius table of the lightweight hierarchical graph index to be empty, and the in-degree radius table stores the in-degree edge radius of each vertex on the lightweight hierarchical graph; S16: randomly select an unvisited first-layer entry, insert its center point as a vertex into the lightweight hierarchical graph index according to the strategy of the HNSW hierarchical graph index, and update the in-degree radius table; the maximum number of neighbors of each vertex on the lightweight hierarchical graph index is controlled by the graph sparsity parameter; S17: adding a pointer to the vertex at the bottom layer of the lightweight hierarchical graph index, pointing to the leaf node corresponding to the first-layer entry in the global tree index; S18: Repeat steps S16 to S17 until all entries of the first layer of the global tree index are inserted into the lightweight hierarchical graph index, thus completing the construction of the lightweight hierarchical graph index; S19: Merge the global tree index and the lightweight hierarchical graph index to obtain a tree-graph structure; (2) Obtain the vector objects to be inserted and deleted, update the global tree index and lightweight hierarchical graph index, and realize dynamic insertion and deletion of the tree-graph structure; For the vector object to be inserted, the specific process of implementing dynamic insertion of the tree-graph structure is as follows: S21: Obtain the vector object to be inserted, and obtain the information of the global tree index according to the current tree-graph structure; S22: according to the M-tree index strategy, create a leaf entry for the vector object to be inserted, and use the vector object as the center point of the leaf entry; S23: inserting the leaf entry into the global tree index according to the M-tree index strategy; if the insertion of the leaf entry does not result in the splitting of the leaf node in the global tree index, that is, the first layer of the global tree index does not change, there is no need to update the lightweight hierarchical graph index, and the process goes to step S26; if the insertion of the leaf entry results in the splitting of the leaf node in the global tree index, that is, a new entry is generated in the first layer of the global tree index, the lightweight hierarchical graph index needs to be updated, that is, steps S24 to S25 are executed; S24: Obtain information of the lightweight hierarchical graph index according to the current tree-graph structure, and according to the strategy of the HNSW hierarchical graph index, insert the center point of the new entry of the first layer of the global tree index as a vertex into the lightweight hierarchical graph index, and update the in-degree radius table; S25: adding a pointer for the vertex at the bottom layer of the lightweight hierarchical graph index, pointing to the leaf node corresponding to the new entry of the first layer in the global tree index; S26: merge and insert the updated global tree index and lightweight hierarchical graph index to complete the dynamic insertion of the tree-graph structure; For the vector object to be deleted, the specific process of implementing dynamic deletion of the tree-graph structure is as follows: S31: obtaining the vector object to be deleted and the search candidate set parameters, and obtaining the information of the global tree index and the lightweight hierarchical graph index according to the current tree-graph structure; S32: using the current tree-graph structure, taking the vector object to be deleted as the query object, and locating the query object through approximate nearest neighbor search; if the vector object is successfully found in the tree-graph structure, the process proceeds to step S34; If the location is not found successfully, it is necessary to further locate the location by range search, that is, to execute step S33; S33: taking the vector object to be deleted as the center, performing an accurate range query with a radius of 0 on the global tree index to locate the vector object; if the vector object cannot be successfully found in the global tree index, it means that the vector object does not exist, and the deletion operation is terminated immediately; if it is successfully found, continue to execute step S34; S34: according to the deletion rule of the M-tree index strategy, the successfully located vector object to be deleted is deleted from the global tree index; if the deletion of the vector object does not cause the leaf node underflow in the global tree index, that is, the first layer of the global tree index does not change, there is no need to update the lightweight hierarchical graph index, and the process goes to step S38; if the deletion of the vector object causes the leaf node underflow in the global tree index, that is, an entry in the first layer of the global tree index is deleted, the lightweight hierarchical graph index needs to be updated, that is, step S35 is executed; S35: Identify the center of the deleted entry in the first layer of the global tree index, that is, the vertex to be deleted in the lightweight hierarchical graph index, and delete the pointer in the vertex pointing to the leaf node corresponding to the deleted entry in the first layer of the global tree index at the bottom layer of the lightweight hierarchical graph index; S36: Taking the vertex to be deleted as the center, perform an accurate range query in the global tree index, where the radius is the in-degree radius stored in the in-degree radius table of the vertex, so as to find all in-degree edges of the vertex to be deleted in the lightweight hierarchical graph index; S37: Delete the vertex and its corresponding edges from the lightweight hierarchical graph index, including the in-degree edges obtained in S36 and the out-degree edges directly obtained in the lightweight hierarchical graph index, and update the in-degree radius table; if the number of neighbors of some vertices in the lightweight hierarchical graph index is reduced after deleting the vertex and edge, resulting in a violation of the graph construction rules, then reinsert these vertices with too few neighbors into the lightweight hierarchical graph index according to the strategy of the HNSW hierarchical graph index; S38: merge and delete the updated global tree index and lightweight hierarchical graph index to complete the dynamic deletion of the tree-graph structure; (3) Use the tree-graph structure to perform approximate nearest neighbor search for vector objects in high-dimensional space to obtain the nearest neighbor of a given query vector object. k -Neighbor object, the specific implementation process is as follows: S41: Get the given query vector object, tree-graph structure, search candidate set parameters and k - Neighborhood parameters; S42: Obtain information of a global tree index and a lightweight hierarchical graph index according to the tree-graph structure; S43: Initialize the search candidate set to an empty set; S44: Randomly select a vertex at the top level of the lightweight hierarchical graph index as the entry point for the search; S45: performing a greedy search at the current layer starting from the entry point, and continuously expanding the neighbors of the vertex until an approximate 1-neighbor of the query vector object at the current layer is found, and the length of the candidate set is set to 1 during the search process; S46: Using the found approximate 1-nearest neighbor as the entry starting point of the next layer of the lightweight hierarchical graph index; S47: repeat steps S45 to S46 until the first layer is searched from the top layer of the lightweight hierarchical graph index; S48: Taking the approximate 1-nearest neighbor found in the first layer as the entry starting point of the bottom layer search, greedy search is performed at the bottom layer, and the search results are added to the search candidate set, the length of the candidate set is the search candidate set parameter; the bottom layer search includes not only directly accessing the vertex neighbors on the bottom layer, but also accessing the leaf nodes in the global tree index pointed to by the bottom layer vertices; S49: Find the closest object to the query vector from the search candidate set k Nearest neighbors as the final search result; In a face recognition system, each person's facial features are converted into high-dimensional vectors, and these vectors are compared to achieve face recognition and verification.

2. A high-dimensional space vector dynamic nearest neighbor search system based on a tree-graph structure, used to execute a high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure as claimed in claim 1, characterized in that: The system includes: The tree-graph structure construction module obtains a high-dimensional space vector set to build a global tree index for all vector objects in the set, and builds a lightweight hierarchical graph index for the first layer of the global tree index to complete the construction of the tree-graph structure. The tree-graph structure update module obtains the vector objects to be inserted and deleted, which are used to update the global tree index and lightweight hierarchical graph index to realize the dynamic insertion and deletion of the tree-graph structure; The nearest neighbor search module uses a tree-graph structure to perform an approximate nearest neighbor search for vector objects in high-dimensional space to obtain the nearest neighbor of a given query vector object. k -Neighbor objects.

3. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is used to execute the computer program to implement the high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure as described in claim 1.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the high-dimensional space vector dynamic nearest neighbor search method based on a tree-graph structure as described in claim 1.

Images