Distributed vector indexing and retrieval method and system using mixed memory and disk
By using vector indexing and retrieval methods of mixed memory and disk in distributed clusters, using multi-layer central point neighbor graphs to manage large-scale vector data, the resource consumption and scalability problems are solved, and efficient and stable vector retrieval is achieved.
Patent Information
- Application Number
- CN202410614310.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-05-17
AI Technical Summary
The prior art consumes huge resources when processing large-scale vector data, resulting in hardware pressure, and it is difficult for stand-alone design to adapt to distributed clusters, resulting in insufficient retrieval efficiency and scalability.
Using a distributed vector index and search method that uses memory and disk, cluster nodes independently build indexes, save the center point of vector cluster in memory, and save all vectors of the cluster where the center point is located in disk, and manage it through multi-layer center point neighbor graph. Incremental construction of indexes and asynchronous management operations are adopted to limit cluster sizes, and decompose index management is used to vector, cluster and multi-layer central point neighbor graph operations to realize real-time updates and division and conquer retrieval.
Reduces memory footprint, avoids index reconstruction, improves retrieval efficiency and system stability, realizes efficient and scalable vector retrieval, can process larger data sets, and maintains stable and reliable in distributed clusters.
Smart Images

Figure CN118467544B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and in particular to a distributed vector indexing and retrieval method and system that hybridly uses memory and disk. Background Art
[0002] Vector databases play an important role in building industry intelligent applications based on large language models. They provide efficient similarity search and retrieval functions by storing and processing vector data. Vector data nearest neighbor search plays an important role in information retrieval fields such as multimedia search and entity search, providing relevant results by searching for the vector with the smallest distance to the query vector.
[0003] In today's big data era, the industry has put forward higher requirements for efficient and scalable similarity search. NNS (Nearest Neighbor Search) is one of the commonly used information search methods, which is widely used in image matching and semantic document retrieval of large databases. However, NNS is difficult to process high-dimensional data. It can only maintain high efficiency while ensuring the accuracy of retrieval response when processing low-dimensional data. In order to overcome this problem, the industry has proposed the ANNS (Approximate Nearest Neighbor Search) solution, which relaxes the requirements for precise search. Mainstream ANNS solutions include algorithms based on neighbor graphs, tree-based algorithms, LSH (Locality Sensitive Hashing) algorithms, and product quantization algorithms.
[0004] The HNSW (Hierarchical Navigable Small World) algorithm is an ANNS algorithm based on the whole graph. The graph data structure adopted by HNSW enables it to be easily extended to large-scale data sets and adapt to the dynamic changes of high-dimensional data. The structure consists of a multi-layer neighbor graph, and the upper layer of the neighbor graph is randomly selected from the bottom layer with an exponentially decaying probability distribution. However, the data structure of HNSW can only be maintained in memory, which takes up a lot of memory. Once it is converted to disk storage, since the retrieval process involves multiple rounds of random reading and writing, even access to high-speed solid-state drives cannot guarantee acceptable retrieval efficiency. DiskAnn (Disk-Backed Approximate Nearest Neighbor Search) is an ANNS solution that supports large-scale data resident on disk. However, the performance of DiskAnn depends largely on the speed and performance of the underlying hardware, and there are still some inevitable limitations. The index of DiskAnn mainly consists of raw data and adjacency lists. Too many raw data rows will cause the adjacency list in the single-machine index to be too large. Disk access is based on a page of 4096 bytes. Large-scale adjacency lists will cause a large number of disk accesses, resulting in performance degradation. In addition, DiskAnn is difficult to update the index. The general practice is to regularly rebuild the entire index structure after the vector is updated, which is very expensive.
[0005] In the existing technology, both memory-based and disk-based large-scale vector ANNS solutions face common challenges. First, processing large-scale vector data sets will lead to huge resource consumption, among which the computing and storage resource requirements may put tremendous pressure on the hardware and even cause the machine to crash. The industry generally introduces a distributed cluster architecture to solve the problem of computing and storage resource requirements, that is, to share the huge computing and storage resource consumption through multiple cluster node machines. However, the HNSW algorithm and DiskAnn algorithm mentioned above are designed for single machines, and how to adapt them to distributed clusters is still a problem to be solved. Summary of the invention
[0006] Based on this, it is necessary to propose a distributed vector indexing and retrieval method and system that uses a combination of memory and disk to address the above problems. A distributed vector indexing and retrieval method that uses a combination of memory and disk, the method comprising:
[0007] Cluster nodes independently construct their own indexes. All vectors in the cluster nodes are stored in two storage media: memory and disk. The center point of the vector cluster is stored in the memory and managed by the data structure of a multi-layer center point neighbor graph. All vectors of the cluster where the center point is located are stored in the disk. The index is constructed in an incremental manner. Starting from the empty multi-layer center point neighbor graph, vector addition operations are performed one by one for the existing vector data in the disk of the cluster node, and then corresponding vector operations are triggered when the vector data in the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations. The size of each vector cluster is limited to a range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, a cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, a cluster splitting operation is performed.
[0008] When receiving a query vector for retrieval, the cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0009] The center point stored in the memory is managed using a data structure of a multi-layer center point neighbor graph, specifically including:
[0010] The number of neighbors of each center point in the graph is limited to the upper limit m set by the user; the complete center point neighbor graph is used as the bottom layer, and part of the neighbor graph is added layer by layer to form an L-layer center point neighbor graph, where L is the number of layers set by the user; the upper layer between two adjacent layers is only a subgraph of the lower layer, and the number of center points in the lower layer is t times the number of center points in the upper layer, where t is the user-set multiple; the center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center points coexisting in multiple layers is retained in the memory.
[0011] The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations, specifically including:
[0012] The vector operations include three types: vector addition operations, vector deletion operations and vector modification operations, which may trigger other vector operations or cluster operations when executed.
[0013] The vector adding operation calculates the nearest center point of the vector to be added, and adds it to the cluster where the center point is located, wherein the first added vector becomes the only center point; if the cluster violates the upper limit of the cluster size after the update, the cluster splitting operation is performed.
[0014] The vector deletion operation locates the cluster where the vector to be deleted is located, and removes the vector to be deleted from the cluster; if the cluster violates the lower limit of the cluster size after being updated, the cluster deletion operation is performed.
[0015] The vector modification operation locates the cluster where the vector to be modified is located; if the nearest center point of the modified vector is not the center point of the current cluster, the operation of adding the vector to be modified in the cluster where the nearest center point is located is first performed, and then the operation of deleting the vector to be modified in the cluster where the vector to be modified is located is performed, otherwise the index management operation is not triggered.
[0016] The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations, specifically including:
[0017] The cluster operations include cluster splitting operations and cluster deletion operations, which may trigger vector operations and multi-layer center point neighbor graph operations when executed.
[0018] The cluster splitting operation calls any memory-based k-center point clustering algorithm for all vectors in the cluster to be split, where k is fixed to 2, and obtains 2 new center points. Then, the 2 new center points obtained by clustering are added to each layer of the multi-layer center point neighbor graph, and the clusters where they are located are initialized as empty clusters; then, consider each vector v in the cluster to be split (including the original center point of the cluster to be split), if the distance from v to the original center point is less than the distance from v to the nearest new center point, then perform the vector addition operation of v that does not involve the cluster to be split, otherwise add v to the cluster where the nearest new center point is located; at the same time, consider each neighbor of the original center point of the cluster to be split, that is, For each vector v (including the neighboring center point) of the cluster where the neighboring center point is located, if the distance between v and the center point of the cluster where it is located is greater than the distance between v and the nearest new center point, v is added to the cluster where the nearest new center point is located, and the vector deletion operation of v in its cluster is performed, otherwise no index management operation is performed on v; finally, the original center point of the cluster to be split is deleted from the multi-layer center point neighbor graph, and the two new clusters obtained by the split and the old cluster with modified vectors in the cluster are considered. The cluster splitting operation is performed on the cluster whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user, and the cluster deletion operation is performed on the cluster whose number of vectors in the cluster is less than the lower limit of the cluster size set by the user.
[0019] The cluster deletion operation first performs a vector addition operation on each vector v in the cluster to be deleted (including the center point of the cluster to be deleted), which does not involve the cluster to be deleted, that is, v is added to other vector clusters; then, considering the vector clusters with added elements, the cluster splitting operation is performed on the vector clusters whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user; finally, the center point of the cluster to be deleted is removed in the multi-layer center point neighbor graph.
[0020] The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations, specifically including:
[0021] The multi-layer center point neighbor graph operation consists of three operations: center point addition, center point deletion, and change propagation upwards, and is recursively executed layer by layer starting from the bottom center point neighbor graph upwards.
[0022] The center point adding operation calculates the nearest at most m center points as neighbors for the new center point to be added in the center point neighbor graph of the current layer. If the number of neighbors of the center point taken as a neighbor is less than m, where m is the upper limit of the number of neighbors of each center point set by the user, the new center point is added as a neighbor, otherwise the new center point is replaced by a closer neighbor; the changes are then propagated upward layer by layer.
[0023] The center point deletion operation directly removes the center point to be deleted from the center point neighbor graph of the current layer, and propagates the changes upward layer by layer.
[0024] The change propagation operation considers each center point v in the adjacent upper layer after the center point neighbor graph of the current layer is changed. If v is deleted in the current layer, v is deleted from the adjacent upper layer. If the neighbors of v in the current layer are changed, v's neighbors are changed in the adjacent upper layer. If the center point neighbor graph of the current layer is changed, the number of center points is greater than t times the number of center points in the adjacent upper layer, where t is a user-set multiple, then the ceiling (N cur / tN up ) center points that are not in the adjacent upper layer, and for each extracted center point, perform its addition operation in the adjacent upper layer, where N cur Indicates the number of center points in the current layer, N up Represents the number of center points of the adjacent upper layer. ceiling(x) returns the smallest integer not less than x.
[0025] The cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, specifically including:
[0026] The cluster node starts from a non-empty top-level center point neighbor graph, and searches downward layer by layer for k center points that are closer to the query vector, wherein the top-level neighbor graph uses k randomly selected center points as an entry, and the lower-level neighbor graph uses k center points obtained from the adjacent upper-level neighbor graph as an entry; then, based on the k center points obtained in the lower-level neighbor graph, the k vectors closest to the query vector are found from their clusters as the approximate k-nearest neighbors retrieved from the cluster node.
[0027] The cluster node starts from the non-empty top-level center point neighbor graph and searches for k center points closer to the query vector layer by layer, specifically including:
[0028] When locating k center points closer to the query vector in a single-layer center point neighbor graph, the k center points used as the entry are used as the k-center point set of the first stage, and then the k-center point set is adjusted stage by stage until the k-center point set no longer changes, and then the k-center point set is returned; wherein, the s+1 stage uses the k-center point set of the s stage as input, first calculates the union of all input center point neighbor sets and the s stage center point set, and then selects k center points closest to the query vector from the union as the output of the s+1 stage.
[0029] A distributed vector indexing and retrieval system that uses a combination of memory and disk, the system comprising:
[0030] An index building module is used for cluster nodes to independently build their own indexes. All vectors in the cluster nodes are stored in two storage media: memory and disk. The memory stores the center point of the vector cluster and manages it using the data structure of a multi-layer center point neighbor graph. The disk stores all vectors of the cluster where the center point is located. The index is built in an incremental manner, starting from the empty multi-layer center point neighbor graph, and first performing vector addition operations on the existing vector data in the disk of the cluster node one by one, and then triggering corresponding vector operations when the vector data in the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations. The size of each vector cluster is limited to a range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, a cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, a cluster splitting operation is performed.
[0031] The retrieval module is used to receive a query vector for retrieval. The cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0032] The embodiments of the present invention have the following beneficial effects:
[0033] When the present invention independently constructs respective indexes through cluster nodes, all vectors in the cluster nodes are stored in two storage media, memory and disk. The center point of the vector cluster is stored in the memory and managed by the data structure of a multi-layer center point neighbor graph. The other part is that all vectors of the cluster where the center point is located are stored in the disk. In other words, the memory part manages the multi-layer center point neighbor graph that is similar to the graph data structure in the HNSW algorithm, while the disk part manages the vector data of the cluster where the center point is located. The center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center point coexisting in multiple layers is retained in the memory, which ensures that the memory occupied by the scheme of the present invention is only 1 / N of the memory occupied by the original HNSW algorithm scheme.min About, among which, N min is the user-defined lower limit on the size of a vector cluster.
[0034] Among them, the index is built in an incremental manner, starting from an empty multi-layer center point neighbor graph, and then performing vector addition operations one by one for the existing vector data on the disk of the cluster node. Then, when the vector data on the disk is changed, the corresponding vector operation is triggered, so that the index can be built and updated in real time, without the need for regular index reconstruction, ensuring the operational stability and reliability of the vector retrieval system.
[0035] In addition, index management operations can be decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations, which avoids the long-term monopoly of system resources by index management operations and ensures the smooth execution of concurrent vector retrieval operations. Under the asynchronous processing mechanism, various index management operations can be executed concurrently as much as possible to achieve higher processing efficiency. If combined with the snapshot generation mechanism commonly used in distributed storage systems, vector addition, deletion, modification, and query transaction processing services at the snapshot isolation level can also be provided.
[0036] The vector index management and vector retrieval provided by the present invention are all performed independently in the cluster node. Only in the last step of the vector retrieval are the k-nearest neighbors returned by each cluster node collected from each cluster node, and the overall nearest k-nearest neighbors are selected from them. This divide-and-conquer strategy apportions the huge computing and storage resource consumption generated when processing large-scale vector data sets, and ensures the scalability of the system, that is, as long as the cluster nodes are increased, a larger-scale vector data set can be processed. If the distributed cluster adopts a shard copy mechanism, then even if individual cluster nodes are down and cannot provide services, as long as each shard of the vector data set provides services in at least one normal cluster node, the vector retrieval method proposed by the present invention can still operate normally and maintain a retrieval recall rate close to that when all cluster nodes are in normal service. In other words, the divide-and-conquer strategy adopted by the present invention can be combined with the shard copy mechanism to ensure the stability and reliability of the system. In general, the present invention adopts a mechanism for cluster nodes to independently construct indexes and retrieve nearest neighbors, thereby realizing an efficient and scalable vector retrieval function. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] in:
[0039] Figure 1 A schematic diagram of a flow chart of an embodiment of a distributed vector indexing and retrieval method for hybrid use of memory and disk provided by the present invention;
[0040] Figure 2 The system architecture of the distributed vector indexing and retrieval using a hybrid of memory and disk provided by the present invention;
[0041] Figure 3 A schematic flow chart of another embodiment of the distributed vector indexing and retrieval method for hybrid use of memory and disk provided by the present invention;
[0042] Figure 4 A schematic diagram of locating the 1-nearest neighbor in the neighbor graph of a center point at a certain layer;
[0043] Figure 5 A schematic diagram of a flow chart of an embodiment of a distributed vector indexing and retrieval system that uses a hybrid of memory and disk provided by the present invention. DETAILED DESCRIPTION
[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0045] like Figure 1 As shown, Figure 1 A flow chart of an embodiment of a distributed vector indexing and retrieval method for hybrid use of memory and disk provided by the present invention. A distributed vector indexing and retrieval method for hybrid use of memory and disk, the method comprising:
[0046] S101: Cluster nodes independently construct their own indexes. All vectors in the cluster nodes are stored in two storage media: memory and disk. The center point of the vector cluster is stored in the memory and managed using the data structure of a multi-layer center point neighbor graph. All vectors of the cluster where the center point is located are stored in the disk. The index is constructed in an incremental manner. Starting from an empty multi-layer center point neighbor graph, vector addition operations are performed one by one for the existing vector data on the disk of the cluster node, and then corresponding vector operations are triggered when the vector data on the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations. The size of each vector cluster is limited to a range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, a cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, a cluster splitting operation is performed.
[0047] For example, referring to Figure 2 , Figure 2 The system architecture of distributed vector indexing and retrieval that uses a hybrid of memory and disk provided by the present invention. The vector index is constructed on each cluster node of the distributed cluster. In this system architecture, the vector data has been distributed on the cluster nodes, and the system will construct the index in the cluster node based on all the vectors contained in the cluster node. All vectors in the cluster node are stored in two storage media, memory and disk. The center point of the vector cluster is stored in the memory and managed using a multi-layer center point neighbor graph data structure. All vectors of the cluster where the center point is located are stored in the disk.
[0048] The size of each vector cluster is maintained within the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, the cluster is deleted; if the number of vectors in the cluster is greater than the upper limit, the cluster is split.
[0049] The number of neighbors of each center point is limited to the upper limit m set by the user. The complete center point neighbor graph is used as the bottom layer, and part of the neighbor graph is added layer by layer to form an L-layer center point neighbor graph, where L is the number of layers set by the user. The upper layer in two adjacent layers is only a subgraph of the lower layer, and the number of center points in the lower layer is t times the number of center points in the upper layer, where t is the user-set multiple. The center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center point coexisting in multiple layers is retained in the memory.
[0050] It should be noted that disk storage methods include but are not limited to column storage, row storage, and key-value storage. Different storage methods are only specific implementations of the present invention on disk storage, and are not specifically limited here. In addition, the present invention also allows vector data in memory or disk to be stored using any compression method (including uncompressed storage), such as vector product quantization compression, numerical full integer quantization, and half floating point quantization compression, etc., which are not specifically limited here.
[0051] In addition, the present invention introduces an incremental vector index management mechanism. When constructing a vector index, each cluster node first performs vector addition operations on the existing vector data in the disk of the cluster node one by one, and then triggers the corresponding vector operation when the vector data in the disk is changed. The management operation of the vector index can be decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations.
[0052] Specifically, the asynchronously executed vector operations include three types: vector addition operations, vector deletion operations, and vector modification operations, which may trigger other vector operations or cluster operations when executed.
[0053] The vector addition operation calculates the nearest center point of the vector to be added and adds it to the cluster where the center point is located. The first added vector becomes the only center point; if the cluster violates the upper limit of the cluster size after the update, the cluster splitting operation is performed.
[0054] The vector deletion operation locates the cluster where the vector to be deleted is located and removes the vector to be deleted from the cluster. If the cluster is updated and the lower limit of the cluster size is violated, the cluster deletion operation is performed.
[0055] The vector modification operation locates the cluster where the vector to be modified is located; if the nearest center point of the modified vector is not the center point of the current cluster, the operation of adding the vector to be modified in the cluster where the nearest center point is located is performed first, and then the operation of deleting the vector to be modified in the cluster where the vector to be modified is located is performed, otherwise the index management operation is not triggered.
[0056] Specifically, asynchronously executed cluster operations include cluster splitting and cluster deletion, which may trigger vector operations and multi-layer center point neighbor graph operations during execution. The principle of cluster operations is to ensure that the vector is allocated to the cluster where the nearest center point is located while maintaining high efficiency.
[0057] The cluster splitting operation calls any memory-based k-center point clustering algorithm for all vectors in the cluster to be split, where k is fixed to 2, and obtains 2 new center points. Then, the 2 new center points obtained by clustering are added to each layer of the multi-layer center point neighbor graph, and the clusters where they are located are initialized as empty clusters. Next, consider each vector v in the cluster to be split (including the original center point of the cluster to be split). If the distance from v to the original center point is less than the distance from v to the nearest new center point, perform the vector addition operation of v that does not involve the cluster to be split, otherwise add v to the cluster where the nearest new center point is located. At the same time, consider each vector v (including the neighboring center point) of each neighbor of the original center point of the cluster to be split, that is, the cluster where the neighboring center point is located. If the distance from v to the center point of its cluster is greater than the distance from v to the nearest new center point, then add v to the cluster where the nearest new center point is located, and perform the vector deletion operation of v in its cluster, otherwise do not perform any index management operation on v. Finally, the original center point of the cluster to be split is deleted from the multi-layer center point neighbor graph, and the two new clusters obtained by the split and the old cluster with changed vectors in the cluster are considered. The cluster splitting operation is performed on the cluster whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user, and the cluster deletion operation is performed on the cluster whose number of vectors in the cluster is less than the lower limit of the cluster size set by the user.
[0058] The cluster deletion operation first performs a vector addition operation on each vector v in the cluster to be deleted (including the center point of the cluster to be deleted), which does not involve the cluster to be deleted, that is, v is added to other vector clusters; then, considering the vector clusters with added elements, a cluster splitting operation is performed on the vector clusters whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user; finally, the center point of the cluster to be deleted is removed in the multi-layer center point neighbor graph.
[0059] It should be noted that the cluster splitting operation is one of the key index management operations provided by the present invention, which splits the cluster that exceeds the size limit into two smaller clusters through the k-center point clustering algorithm (k is fixed to 2). Any k-center point clustering algorithm is applicable to this operation, so no matter which specific k-center point clustering algorithm is used, it does not exceed the technical scope protected by the present invention, and the present invention does not make specific restrictions on the k-center point clustering algorithm.
[0060] Specifically, the asynchronously executed multi-layer center point neighbor graph operation consists of three operations: center point addition, center point deletion, and change propagation upward. It is recursively executed layer by layer starting from the bottom-level center point neighbor graph upward, and is suitable for the construction and update of multi-layer center point neighbor graphs.
[0061] In the center point neighbor graph of the current layer, the new center point to be added takes the nearest m center points as neighbors. If the center point taken as a neighbor has less than m neighbors, the new center point is added as a neighbor, otherwise the new center point is replaced by a closer neighbor. Then the changes are propagated upward layer by layer.
[0062] The center point deletion operation directly removes the center point to be deleted from the center point neighbor graph of the current layer, and propagates the changes upward layer by layer.
[0063] After the neighbor graph of the center point of the current layer is changed, consider each center point v in the adjacent upper layer. If v is deleted in the current layer, delete v from the adjacent upper layer; if the neighbors of v in the current layer are changed, change the neighbors of v in the adjacent upper layer. If the number of center points in the neighbor graph of the center point of the current layer is changed, and the number of center points is greater than t times the number of center points in the adjacent upper layer, randomly select the ceiling (N cur / tN up ) center points that are not in the adjacent upper layer, and for each extracted center point, perform its addition operation in the adjacent upper layer, where N cur Indicates the number of center points in the current layer, N up Represents the number of center points of the adjacent upper layer. ceiling(x) returns the smallest integer not less than x.
[0064] S102: When receiving a query vector for retrieval, the cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0065] Exemplarily, when receiving a query vector for retrieval, for a given query vector, the cluster nodes independently retrieve their respective approximate k-nearest neighbors, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node.
[0066] Specifically, the cluster node starts from the non-empty top-level center point neighbor graph, and searches downward layer by layer for k center points that are closer to the query vector. The top-level neighbor graph uses k randomly selected center points as the entrance, and the lower-level neighbor graph uses the k center points output from the adjacent upper-level neighbor graph as the entrance. Then, based on the k center points obtained in the lower-level neighbor graph, the k vectors closest to the query vector are found from their clusters as the approximate k-nearest neighbors retrieved from the cluster node.
[0067] From the above description, it can be known that when the present invention independently constructs respective indexes through cluster nodes, all vectors in the cluster nodes are stored in two storage media, memory and disk. The center point of the vector cluster is stored in the memory and managed by the data structure of a multi-layer center point neighbor graph. The other part is that all vectors of the cluster where the center point is located are stored in the disk. In other words, the memory part manages the multi-layer center point neighbor graph that is similar to the graph data structure in the HNSW algorithm, while the disk part manages the vector data of the cluster where the center point is located. The center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center point coexisting in multiple layers is retained in the memory, which ensures that the memory occupied by the scheme of the present invention is only 1 / N of the memory occupied by the original HNSW algorithm scheme. min About, among which, N min is the user-defined lower limit on the size of a vector cluster.
[0068] Among them, the index is built in an incremental manner, starting from an empty multi-layer center point neighbor graph, and then performing vector addition operations one by one for the existing vector data on the disk of the cluster node. Then, when the vector data on the disk is changed, the corresponding vector operation is triggered, so that the index can be built and updated in real time, without the need for regular index reconstruction, ensuring the operational stability and reliability of the vector retrieval system.
[0069] In addition, index management operations can be decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations, which avoids the long-term monopoly of system resources by index management operations and ensures the smooth execution of concurrent vector retrieval operations. Under the asynchronous processing mechanism, various index management operations can be executed concurrently as much as possible to achieve higher processing efficiency. If combined with the snapshot generation mechanism commonly used in distributed storage systems, vector addition, deletion, modification, and query transaction processing services at the snapshot isolation level can also be provided.
[0070] The vector index management and vector retrieval provided by the present invention are all performed independently in the cluster node. Only in the last step of the vector retrieval are the k-nearest neighbors returned by each cluster node collected from each cluster node, and the overall nearest k-nearest neighbors are selected from them. This divide-and-conquer strategy apportions the huge computing and storage resource consumption generated when processing large-scale vector data sets, and ensures the scalability of the system, that is, as long as the cluster nodes are increased, a larger-scale vector data set can be processed. If the distributed cluster adopts a shard copy mechanism, then even if individual cluster nodes are down and cannot provide services, as long as each shard of the vector data set provides services in at least one normal cluster node, the vector retrieval method proposed by the present invention can still operate normally and maintain a retrieval recall rate close to that when all cluster nodes are in normal service. In other words, the divide-and-conquer strategy adopted by the present invention can be combined with the shard copy mechanism to ensure the stability and reliability of the system. In general, the present invention adopts a mechanism for cluster nodes to independently construct indexes and retrieve nearest neighbors, thereby realizing an efficient and scalable vector retrieval function.
[0071] like Figure 3 As shown, Figure 3 A flow chart of another embodiment of a distributed vector indexing and retrieval method for hybrid use of memory and disk provided by the present invention. A distributed vector indexing and retrieval method for hybrid use of memory and disk, the method comprising:
[0072] S201: Cluster nodes independently construct their own indexes. All vectors in the cluster nodes are stored in two storage media: memory and disk. The center point of the vector cluster is stored in the memory and managed by the data structure of the multi-layer center point neighbor graph. All vectors of the cluster where the center point is located are stored in the disk. The index is constructed in an incremental manner. Starting from the empty multi-layer center point neighbor graph, vector addition operations are performed one by one for the existing vector data in the disk of the cluster node, and then corresponding vector operations are triggered when the vector data in the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations. The size of each vector cluster is limited to the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, a cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, a cluster splitting operation is performed.
[0073] Exemplarily, the vector index is constructed on each cluster node of the distributed cluster. The vector data has been distributed on the cluster nodes. The system will construct the index in the cluster node according to all the vectors contained in the cluster node. All the vectors in the cluster node are stored in two storage media: memory and disk. The center point of the vector cluster is stored in the memory, and the data structure of the multi-layer center point neighbor graph is used for management. All vectors of the cluster where the center point is located are stored in the disk. The size of each vector cluster is maintained within the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, the cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, the cluster splitting operation is performed. The number of neighbors of each center point is limited to the upper limit m set by the user. The complete center point neighbor graph is used as the bottom layer, and part of the neighbor graph is added layer by layer to form an L-layer center point neighbor graph, where L is the number of layers set by the user. The upper layer in the two adjacent layers is only a subgraph of the lower layer, and the number of center points in the lower layer is t times the number of center points in the upper layer, where t is the user-set multiple. The center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center point coexisting in multiple layers is retained in the memory. In addition, the present invention introduces an incremental vector index management mechanism. When constructing a vector index, each cluster node first performs a vector addition operation on the existing vector data in the disk of the cluster node one by one, and then triggers the corresponding vector operation when the vector data in the disk is changed. The management operation of the vector index can be decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations.
[0074] Specifically, vector operations include vector addition, vector deletion, and vector modification. The vector addition operation calculates the nearest center point of the vector to be added and adds it to the cluster where the center point is located, where the first added vector becomes the only center point; if the cluster violates the upper limit of the cluster size after the update, the cluster split operation is performed; the vector deletion operation locates the cluster where the vector to be deleted is located and removes the vector to be deleted from the cluster; if the cluster violates the lower limit of the cluster size after the update, the cluster deletion operation is performed; the vector modification operation locates the cluster where the vector to be modified is located; if the nearest center point of the modified vector is not the center point of the current cluster, the addition operation of the vector to be modified in the cluster where the nearest center point is located is performed first, and then the deletion operation of the vector to be modified in the cluster where the vector to be modified is located is performed, otherwise the index management operation is not triggered.
[0075] Assume that the size of the vector cluster is lower bounded by N min , then the vectors stored in each cluster node are at most 1 / N min The vectors need to be managed as a central point in the memory. For example, in a cluster node, 10 trillion high-dimensional floating-point vectors are indexed. The scheme of the present invention is adopted and the lower limit of the size of the vector cluster is set to N. min =32, the cluster node only needs about 32GB of memory space to store the multi-layer center point neighbor graph.
[0076] Cluster operations include cluster splitting and cluster deletion. The cluster splitting operation calls any memory-based k-center point clustering algorithm for all vectors in the cluster to be split, where k is fixed to 2, and obtains two new center points. Then, the two new center points obtained by clustering are added to each layer of the multi-layer center point neighbor graph, and the clusters where they are located are initialized as empty clusters. Next, consider each vector v in the cluster to be split (including the original center point of the cluster to be split). If the distance from v to the original center point is less than the distance from v to the nearest new center point, perform the vector addition operation of v that does not involve the cluster to be split, otherwise add v to the cluster where the nearest new center point is located. At the same time, consider each neighbor of the original center point of the cluster to be split, that is, the neighbor For each vector v (including the adjacent center point) of the cluster where the near center point is located, if the distance from v to the center point of the cluster where it is located is greater than the distance from v to the nearest new center point, v is added to the cluster where the nearest new center point is located, and the vector deletion operation of v in the cluster where it is located is performed, otherwise no index management operation is performed on v; finally, the original center point of the cluster to be split is deleted from the multi-layer center point neighbor graph, and the two new clusters obtained by the split and the old cluster with modified vectors in the cluster are considered, and the cluster splitting operation is performed on the cluster whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user, and the cluster deletion operation is performed on the cluster whose number of vectors in the cluster is less than the lower limit of the cluster size set by the user. The cluster deletion operation first performs the vector addition operation of v that does not involve the cluster to be deleted for each vector v (including the center point of the cluster to be deleted) in the cluster to be deleted, that is, v is added to other vector clusters; then, considering the vector clusters with added elements, the cluster splitting operation is performed on the vector clusters whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user; finally, the center point of the cluster to be deleted is removed from the multi-layer center point neighbor graph.
[0077] The multi-layer center point neighbor graph operation consists of three operations: center point addition, center point deletion, and change propagation upwards. It is recursively executed layer by layer starting from the bottom center point neighbor graph upwards. In the center point addition operation, in the center point neighbor graph of the current layer, the nearest at most m center points are calculated as neighbors for the new center point to be added. If the center point taken as a neighbor has less than m neighbors, where m is the upper limit of the number of neighbors for each center point set by the user, the new center point is added as a neighbor, otherwise the new center point is replaced by a closer neighbor; then the changes are propagated upwards layer by layer. The center point deletion operation directly removes the center point to be deleted from the center point neighbor graph of the current layer, and propagates the changes upwards layer by layer. After the neighbor graph of the center point of the current layer is changed, consider each center point v in the adjacent upper layer. If v is deleted in the current layer, delete v from the adjacent upper layer. If the neighbors of v in the current layer change, change the neighbors of v in the adjacent upper layer. If the number of center points in the neighbor graph of the center point of the current layer is changed, the number of center points is greater than t times the number of center points in the adjacent upper layer, where t is the user-set multiple, then randomly select the ceiling (N cur / tN up ) center points that are not in the adjacent upper layer, and for each extracted center point, perform its addition operation in the adjacent upper layer, where N cur Indicates the number of center points in the current layer, N up Represents the number of center points of the adjacent upper layer. ceiling(x) returns the smallest integer not less than x.
[0078] S202: When receiving a query vector for retrieval, the cluster node starts from the non-empty top-level center point neighbor graph, and searches downward layer by layer for k center points that are closer to the query vector, wherein the top-level neighbor graph uses k randomly selected center points as the entrance, and the lower-level neighbor graph uses k center points obtained from the adjacent upper-level neighbor graph as the entrance; then, based on the k center points obtained in the lower-level neighbor graph, the k vectors closest to the query vector are found from their clusters as the approximate k-nearest neighbors retrieved from the cluster node, and then the nearest k vectors are selected from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0079] Exemplarily, when locating k center points closer to the query vector in a single-layer center point neighbor graph, the k center points as the entry are used as the k-center point set of the first stage, and then the k-center point set is adjusted stage by stage until the k-center point set no longer changes, and the k-center point set is returned; wherein, the s+1 stage uses the k-center point set of the sth stage as input, first calculates the union of all input center point neighbor sets and the sth stage center point set, and then selects the k center points closest to the query vector from the union as the output of the s+1 stage. This retrieval operation for a single-layer center point neighbor graph can ensure that the k center points found in the current layer will never be farther from the given query vector than the k center points found in the upper layer.
[0080] Combined with reference Figure 4 , Figure 4 Schematic diagram for locating the 1-nearest neighbor in the neighbor graph of a center point at a certain layer. Figure 4 The specific process of locating the 1-nearest neighbor of the query vector in a certain layer of the center point neighbor graph is shown. The system maintains a k-center point set S in memory, where k = 1. In the first stage, S consists of the entry vector, that is, S = {I}. In the second stage, the union of the neighbor set of all vectors in S and S is first calculated, that is, {I, B, C, D}, and then the vector C closer to the query vector is selected as the output of the second stage, that is, S is updated to {C}. In the third stage, the union of the neighbor set of all vectors in S and S is first calculated, that is, {A, B, C, I}, and then the vector B closer to the query vector is selected as the output of the third stage, that is, S is updated to {B}. In the fourth stage, the union of the neighbor set of all vectors in S and S is first calculated, that is, {A, B, C, E, H, I}, and then the vector E closer to the query vector is selected as the output of the fourth stage, that is, S is updated to {E}. In the fifth stage, the union of the neighbor sets of all vectors in S and S is calculated, that is, {B, D, E, G, J}. Then it is found that the vector closest to the query vector in this set is still E. Therefore, the positioning is completed and {E} is returned as the k-center point set output in this layer.
[0081] Furthermore, the k-center point set obtained from the underlying neighbor graph is used as the approximate k-nearest neighbors retrieved from the cluster nodes, and the nearest k vectors are selected from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0082] From the above description, it can be seen that the present invention introduces three vector operations of vector addition, vector deletion and vector modification, two cluster operations of cluster deletion and cluster splitting, and the update operation of the multi-layer center point neighbor graph to realize incremental vector index management, and limits the size of each vector cluster within the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, the cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, the cluster splitting operation is performed. It is an effective solution for integrating memory and disk to provide large-scale vector retrieval. In addition, the present invention adopts a mechanism for cluster nodes to independently build indexes and retrieve nearest neighbors, realizing efficient and scalable vector retrieval functions.
[0083] like Figure 5 As shown, Figure 5 A flow chart of an embodiment of a distributed vector indexing and retrieval system for hybrid use of memory and disk provided by the present invention. A distributed vector indexing and retrieval system 10 for hybrid use of memory and disk, the system comprising:
[0084] The index building module 11 is used for cluster nodes to independently build their own indexes. All vectors in the cluster nodes are stored in two storage media: memory and disk. The center point of the vector cluster is stored in the memory and managed by the data structure of the multi-layer center point neighbor graph. All vectors of the cluster where the center point is located are stored in the disk. The index is built in an incremental manner. Starting from an empty multi-layer center point neighbor graph, vector addition operations are performed one by one for the existing vector data in the disk of the cluster node, and then corresponding vector operations are triggered when the vector data in the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations. The size of each vector cluster is limited to the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, a cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, a cluster splitting operation is performed.
[0085] The retrieval module 12 is used to receive a query vector for retrieval. The cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their own nodes, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0086] Exemplarily, in the index construction module 11, the vector index is constructed on each cluster node of the distributed cluster, and the vector data has been distributed on the cluster nodes. The system will construct the index in the cluster node according to all the vectors contained in the cluster node. All the vectors in the cluster node are stored in two storage media, memory and disk. The center point of the vector cluster is stored in the memory, and the data structure of the multi-layer center point neighbor graph is used for management. All vectors of the cluster where the center point is located are stored in the disk. The size of each vector cluster is maintained within the range set by the user, that is, when the vector in the cluster changes, if the number of vectors in the cluster is less than the lower limit, the cluster deletion operation is performed, and if the number of vectors in the cluster is greater than the upper limit, the cluster splitting operation is performed. The number of neighbors of each center point is limited to the upper limit m set by the user. The complete center point neighbor graph is used as the bottom layer, and part of the neighbor graph is added layer by layer to form an L-layer center point neighbor graph, where L is the number of layers set by the user. The upper layer in the two adjacent layers is only a subgraph of the lower layer, and the number of center points in the lower layer is t times the number of center points in the upper layer, where t is the user-set multiple. The center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center point coexisting in multiple layers is retained in the memory. In addition, the present invention introduces an incremental vector index management mechanism. When constructing a vector index, each cluster node first performs a vector addition operation on the existing vector data in the disk of the cluster node one by one, and then triggers the corresponding vector operation when the vector data in the disk is changed. The management operation of the vector index can be decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations.
[0087] In the retrieval module 12, when receiving a query vector for retrieval, the cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes. The cluster nodes start from the non-empty top-level center point neighbor graph and search for k center points closer to the query vector layer by layer. The top-level neighbor graph uses the randomly selected k center points as the entry, and the lower-level neighbor graph uses the k center points obtained in the adjacent upper-level neighbor graph as the entry; then, based on the k center points obtained in the lower-level neighbor graph, the k vectors closest to the query vector are found from their clusters as the approximate k-nearest neighbors retrieved from the cluster nodes. Further, the nearest k vectors are selected from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
[0088] It is worth noting that the various modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.
[0089] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A distributed vector indexing and retrieval method that uses a combination of memory and disk, characterized in that: The method comprises: Cluster nodes independently construct their own indexes. All vectors in the cluster nodes are stored in two storage media: memory and disk. The memory stores the center point of the vector cluster and manages it using the data structure of a multi-layer center point neighbor graph. The disk stores all vectors of the cluster where the center point is located. The index is constructed in an incremental manner. Starting from an empty multi-layer center point neighbor graph, vector addition operations are performed one by one for the existing vector data in the disk of the cluster node, and then corresponding vector operations are triggered when the vector data in the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations, including: the vector operations include three types: vector addition operations, vector deletion operations, and vector modification operations, which can be executed Can trigger other vector operations or cluster operations; the vector adding operation calculates the nearest center point of the vector to be added and adds it to the cluster where the center point is located, where the first added vector becomes the only center point; if the cluster is updated and the upper limit of the cluster size is violated, the cluster splitting operation is performed; the vector deleting operation locates the cluster where the vector to be deleted is located and removes the vector to be deleted from the cluster; if the cluster is updated and the lower limit of the cluster size is violated, the cluster deleting operation is performed; the vector modifying operation locates the cluster where the vector to be modified is located; if the nearest center point of the modified vector is not the center point of the current cluster, the adding operation of the vector to be modified in the cluster where the nearest center point is located is performed first, and then the deleting operation of the vector to be modified in the cluster where the vector to be modified is located is performed, otherwise the index management operation is not triggered; The size of each vector cluster is limited to the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, the cluster is deleted; if the number of vectors in the cluster is greater than the upper limit, the cluster is split; When receiving a query vector for retrieval, the cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
2. The distributed vector indexing and retrieval method using a combination of memory and disk according to claim 1, characterized in that: The center point stored in the memory is managed using a data structure of a multi-layer center point neighbor graph, specifically including: The number of neighbors of each center point in the graph is limited to the upper limit m set by the user; the complete center point neighbor graph is used as the bottom layer, and part of the neighbor graph is added layer by layer to form an L-layer center point neighbor graph, where L is the number of layers set by the user; the upper layer between two adjacent layers is only a subgraph of the lower layer, and the number of center points in the lower layer is t times the number of center points in the upper layer, where t is the user-set multiple; the center point vector data is not stored repeatedly in the memory, that is, only one copy of the vector data of the center points coexisting in multiple layers is retained in the memory.
3. The distributed vector indexing and retrieval method using a combination of memory and disk according to claim 1, characterized in that: The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations, specifically including: The cluster operations include cluster splitting operations and cluster deletion operations, which may trigger vector operations and multi-layer center point neighbor graph operations when executed; The cluster splitting operation calls any memory-based k-center point clustering algorithm for all vectors in the cluster to be split, where k is fixed to 2, and obtains 2 new center points. Then, the 2 new center points obtained by clustering are added to each layer of the multi-layer center point neighbor graph, and the clusters where they are located are initialized to empty clusters; then, consider each vector v in the cluster to be split, the vectors in the cluster to be split include the original center point of the cluster to be split, if the distance from v to the original center point is less than the distance from v to the nearest new center point, then perform the vector addition operation of v that does not involve the cluster to be split, otherwise add v to the cluster where the nearest new center point is located; at the same time, consider each neighbor of the original center point of the cluster to be split, that is, the neighboring For each vector v of the cluster where the center point is located, the vectors of the clusters where the neighboring center points are located include the neighboring center points. If the distance between v and the center point of the cluster where it is located is greater than the distance between v and the nearest new center point, v is added to the cluster where the nearest new center point is located, and the vector deletion operation of v in the cluster where it is located is performed, otherwise no index management operation is performed on v; finally, the original center point of the cluster to be split is deleted from the multi-layer center point neighbor graph, and the two new clusters obtained by the split and the old cluster with modified vectors in the cluster are considered. The cluster splitting operation is performed on the cluster whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user, and the cluster deletion operation is performed on the cluster whose number of vectors in the cluster is less than the lower limit of the cluster size set by the user; The cluster deletion operation first targets each vector v in the cluster to be deleted, where the vectors in the cluster to be deleted include the center point of the cluster to be deleted, and performs a vector adding operation on v that does not involve the cluster to be deleted, that is, adding v to other vector clusters; then, considering the vector clusters to which elements have been added, the cluster splitting operation is performed on the vector clusters whose number of vectors in the cluster is greater than the upper limit of the cluster size set by the user; finally, the center point of the cluster to be deleted is removed in the multi-layer center point neighbor graph.
4. The distributed vector indexing and retrieval method using a combination of memory and disk according to claim 1, characterized in that: The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations and multi-layer center point neighbor graph operations, specifically including: The multi-layer center point neighbor graph operation consists of three operations: center point addition, center point deletion, and change propagation upward, which are recursively executed layer by layer starting from the bottom center point neighbor graph upward; The center point adding operation calculates the nearest at most m center points as neighbors for the new center point to be added in the center point neighbor graph of the current layer. If the center point taken as a neighbor has less than m neighbors, where m is the upper limit of the number of neighbors of each center point set by the user, the new center point is added as a neighbor, otherwise the new center point is replaced by a closer neighbor; and then the changes are propagated upward layer by layer; The center point deletion operation directly removes the center point to be deleted from the center point neighbor graph of the current layer, and propagates the changes upward layer by layer; The change propagation operation considers each center point v in the adjacent upper layer after the center point neighbor graph of the current layer is changed. If v is deleted in the current layer, v is deleted from the adjacent upper layer. If the neighbors of v in the current layer are changed, v's neighbors are changed in the adjacent upper layer. If the center point neighbor graph of the current layer is changed, the number of center points is greater than t times the number of center points in the adjacent upper layer, where t is a user-set multiple, then the ceiling (N cur / t -N up ) center points that are not in the adjacent upper layer, and for each extracted center point, perform its addition operation in the adjacent upper layer, where N cur Indicates the number of center points in the current layer, N up Represents the number of center points of the adjacent upper layer. ceiling(x) returns the smallest integer not less than x.
5. The distributed vector indexing and retrieval method using a combination of memory and disk according to claim 1, characterized in that: The cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, specifically including: The cluster node starts from a non-empty top-level center point neighbor graph, and searches downward layer by layer for k center points that are closer to the query vector, wherein the top-level neighbor graph uses k randomly selected center points as an entry, and the lower-level neighbor graph uses k center points obtained from the adjacent upper-level neighbor graph as an entry; then, based on the k center points obtained in the lower-level neighbor graph, the k vectors closest to the query vector are found from their clusters as the approximate k-nearest neighbors retrieved from the cluster node.
6. The distributed vector indexing and retrieval method using a combination of memory and disk according to claim 5, characterized in that: The cluster node starts from the non-empty top-level center point neighbor graph and searches for k center points closer to the query vector layer by layer, specifically including: When locating k center points closer to the query vector in a single-layer center point neighbor graph, the k center points used as the entry are used as the k-center point set of the first stage, and then the k-center point set is adjusted stage by stage until the k-center point set no longer changes, and then the k-center point set is returned; wherein, the s+1 stage uses the k-center point set of the s stage as input, first calculates the union of all input center point neighbor sets and the s stage center point set, and then selects k center points closest to the query vector from the union as the output of the s+1 stage.
7. A distributed vector indexing and retrieval system that uses a combination of memory and disk, characterized in that: The system comprises: An index building module is used for cluster nodes to independently build their own indexes. All vectors in the cluster nodes are stored in memory and disk. The memory stores the center point of the vector cluster and manages it using the data structure of a multi-layer center point neighbor graph. The disk stores all vectors of the cluster where the center point is located. The index is built in an incremental manner. Starting from an empty multi-layer center point neighbor graph, vector addition operations are performed one by one for the existing vector data in the disk of the cluster node, and then corresponding vector operations are triggered when the vector data in the disk is changed. The management operation of the vector index is decomposed into asynchronously executed vector operations, cluster operations, and multi-layer center point neighbor graph operations, including: the vector operations include three types: vector addition operations, vector deletion operations, and vector modification operations. They Other vector operations or cluster operations may be triggered during execution; the vector adding operation calculates the nearest center point of the vector to be added and adds it to the cluster where the center point is located, wherein the first added vector becomes the only center point; if the cluster is updated and the upper limit of the cluster size is violated, a cluster splitting operation is performed; the vector deleting operation locates the cluster where the vector to be deleted is located and removes the vector to be deleted from the cluster; if the cluster is updated and the lower limit of the cluster size is violated, a cluster deleting operation is performed; the vector modifying operation locates the cluster where the vector to be modified is located; if the nearest center point of the modified vector is not the center point of the current cluster, the adding operation of the vector to be modified in the cluster where the nearest center point is located is performed first, and then the deleting operation of the vector to be modified in the cluster where the vector to be modified is located is performed, otherwise the index management operation is not triggered; The size of each vector cluster is limited to the range set by the user, that is, when the vectors in the cluster change, if the number of vectors in the cluster is less than the lower limit, the cluster is deleted; if the number of vectors in the cluster is greater than the upper limit, the cluster is split; The retrieval module is used to receive a query vector for retrieval. The cluster nodes independently retrieve the approximate k-nearest neighbors of the query vector in their respective nodes, and then select the nearest k vectors from the union of the approximate k-nearest neighbor sets of each cluster node as the retrieval result.
