External storage vector retrieval method and device based on graph abstraction and storage medium

Abstract

The invention discloses an external storage vector retrieval method and device based on graph abstraction and a storage medium, and relates to the technical field of vector retrieval and external storage. The method comprises the following steps: preprocessing a query vector to obtain a preset type of query vector; inputting the preset type of query vector and the target return quantity into a pre-constructed graph abstract index to obtain an abstract node set corresponding to the preset type of query vector; obtaining a cluster priority score and a search seed of a cluster to which the abstract nodes of the abstract node set belong according to the cluster to which the abstract nodes belong; determining a cluster access sequence according to the cluster priority score; loading and accessing the clusters on the memory according to the sequence of the cluster access sequences, and determining a target cluster sequence corresponding to the query vector; according to the search seeds corresponding to the clusters in the target cluster sequence, determining local candidate pools corresponding to the clusters in the target cluster sequence; and according to the local candidate pool, on the basis of a triangular inequality pruning rule, sequentially carrying out local search on clusters in the target cluster sequence to obtain a global candidate set of the query vector.

Description

Technical Field

[0001] This application relates to the fields of vector retrieval and external storage technology, and more specifically, to an external storage vector retrieval method, apparatus and storage medium based on graph abstraction. Background Technology

[0002] With the rapid development of applications such as large models, recommender systems, and search engines, vectorized representation has become the mainstream method for modeling text, images, audio, and user behavior. In scenarios such as vector databases, retrieval augmentation generation (RAG), and long context attention acceleration, low-latency, high-throughput Approximate Nearest Neighbor Search (ANNS) is required on high-dimensional vector sets of millions or even billions.

[0003] In traditional in-memory ANNS scenarios, graph indexes such as HNSW and NSG, or structures like IVF and LSH, can be relied upon to achieve millisecond or even sub-millisecond query latency while ensuring high recall. However, when the data scale expands to hundreds of millions or billions of vectors, if the solution of having all vectors and indexes resident in memory is still adopted, the required memory capacity often reaches hundreds of GB or even TB, resulting in high hardware costs and difficulty in deployment on a single machine. Therefore, more and more systems are adopting an "out-of-core" solution, persisting most of the vector data and local indexes to external storage such as SSDs, and only keeping a small amount of navigation structure in memory, achieving single-machine retrieval of hundreds of millions of vectors through hierarchical indexing.

[0004] Existing typical external-memory ANNS solutions include: systems based on graph indexes that segment the complete graph into multiple data segments, retaining segment-level abstractions or navigation structures in memory; and systems that first perform IVF or other clustering on the data, then build graph indexes or inverted indexes within each cluster, and construct high-level navigation structures between clusters. These systems typically employ the concept of "hierarchical indexing": upper-level abstract indexes guide queries on which clusters to access, while lower-level local indexes are used for fine-grained searches within selected clusters.

[0005] However, in external storage scenarios, the real performance determinant is often not a single distance calculation, but rather the data migration between memory and SSD. Existing solutions generally lack fine-grained, two-level pruning strategies: on the one hand, the order of cluster access lacks dynamic rearrangement based on query context, easily leading to the search for a valid result only after scanning multiple irrelevant clusters; on the other hand, there is a lack of theoretically provable pruning at the "vector level" during intra-cluster retrieval, resulting in a large number of vectors being unnecessarily read from disk, only to be discarded later. These problems collectively lead to high I / O counts, low memory hit rate, and unstable latency.

[0006] The content of the background section is merely technology known to the public and does not necessarily represent existing technology in the field. Summary of the Invention

[0007] This application aims to provide an external storage vector retrieval method, apparatus, and storage medium based on graph abstraction to solve at least one of the above-mentioned technical problems.

[0008] According to one aspect of this application, an external memory vector retrieval method based on graph abstraction is provided. The method includes: preprocessing a received query vector to obtain a preset type query vector; inputting the preset type query vector and the received target return quantity into a pre-constructed graph abstraction index to obtain a set of abstract nodes corresponding to the preset type query vector; clustering the abstract nodes of the set of abstract nodes according to their clusters to obtain the cluster priority score and search seed of the respective clusters; determining a cluster access sequence according to the cluster priority score; loading and accessing clusters in memory according to the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector; determining a local candidate pool corresponding to the clusters in the target cluster sequence according to the search seed of the clusters in the target cluster sequence; and performing a local search on the clusters in the target cluster sequence according to the local candidate pool and the triangle inequality pruning rule to obtain a global candidate set of the query vector.

[0009] According to some embodiments of this application, determining the local candidate pool corresponding to a cluster in the target cluster sequence based on the search seed corresponding to the cluster in the target cluster sequence includes: determining the search starting point of the cluster in the target cluster sequence based on the search seed corresponding to the cluster in the target cluster sequence; and determining the local candidate pool based on the search starting point.

[0010] According to some embodiments of this application, based on the local candidate pool and the triangle inequality pruning rule, a local search is performed sequentially on the clusters in the target cluster sequence to obtain a global candidate set for the query vector, including: a local candidate pool; determining the target node; determining the difference between the distance between the target node and the query vector of a preset type and the distance between the target node and its neighbors; updating the local candidate pool based on the neighbors of the target node when the difference is less than or equal to the farthest distance threshold of the global candidate set; traversing all clusters in the target cluster sequence to update all local candidate pools; and determining the global candidate set based on the updated local candidate pools.

[0011] According to some embodiments of this application, before preprocessing the received query vectors to obtain a preset type query vector, the method further includes a step of constructing an index directory file. The step of constructing the index directory file includes: preprocessing the original vector set to obtain a preset type original vector set; allocating the preset type original vector set to a preset number of clusters based on a preset clustering algorithm to obtain a cluster set; constructing a local graph index for each cluster in the cluster set based on the preset type original vectors allocated within the cluster set; determining a graph abstraction index based on the cluster set; and determining an index directory file for the local graph index and the graph abstraction index.

[0012] According to some embodiments of this application, based on a preset clustering algorithm, allocating a preset type of original vector set to a preset number of clusters to obtain a cluster set includes: allocating sampled vectors of the preset type of original vector set to a preset number of clusters to obtain an initial cluster set; performing centroid training on the initial cluster set according to a preset number of rounds to determine the centroid of each cluster in the trained initial cluster set; and allocating the preset type of original vectors of the preset type of original vector set to the cluster closest to the centroid of each cluster in the trained initial cluster set to obtain a cluster set.

[0013] According to some embodiments of this application, constructing a local graph index for each cluster of a cluster set based on preset type original vectors allocated within the cluster set includes: determining an initial nearest neighbor graph based on a nearest neighbor graph algorithm and preset type original vectors allocated within the cluster set; and determining a local graph index based on the initial nearest neighbor graph.

[0014] According to some embodiments of this application, determining the graph abstract index based on the cluster set includes: determining the centroid set of the cluster set based on the centroid vector of each cluster in the cluster set; determining the abstract node set based on the sampling vector of each cluster in the cluster set; determining the graph abstract node set based on the centroid set and the abstract node set; and determining the graph abstract index based on the graph abstract node set.

[0015] According to some embodiments of this application, determining a graph abstract index based on a graph abstract node set includes: inserting a graph abstract node set into an initial graph abstract index according to a preset hierarchical rule; determining the adjacent edges of the abstract nodes in the graph abstract node set; and constructing a graph abstract index based on the initial graph abstract index and adjacent edges after inserting the graph abstract node set.

[0016] According to one aspect of this application, an external storage vector retrieval device based on graph abstraction is provided. The device includes a query processing unit. The query processing unit preprocesses the received query vector to obtain a preset type query vector; the query processing unit inputs the preset type query vector and the target return quantity into a pre-constructed graph abstraction index to obtain a set of abstract nodes corresponding to the preset type query vector; the query processing unit clusters the abstract nodes of the abstract node set according to their respective clusters to obtain the cluster priority score and search seed of each cluster; the query processing unit determines a cluster access sequence based on the cluster priority score; the query processing unit accesses clusters in the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector; the query processing unit determines a local candidate pool corresponding to the clusters in the target cluster sequence based on the search seed of each cluster in the target cluster sequence; the query processing unit performs a local search on the clusters in the target cluster sequence sequentially based on the local candidate pool and the triangle inequality pruning rule to obtain a global candidate set for the query vector.

[0017] According to another aspect of this application, this application also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of implementing the external memory vector retrieval method based on graph abstraction as described above.

[0018] The technical solution of this application can obtain a preset type query vector by preprocessing the received query vector. The technical solution of this application can also obtain the set of abstract nodes corresponding to the preset type query vector by inputting the preset type query vector and the received target return quantity into a pre-constructed graph abstract index. Furthermore, the technical solution of this application can obtain the cluster priority score and search seed of the cluster by clustering the abstract nodes of the abstract node set.

[0019] The technical solution of this application can determine the cluster access sequence through cluster priority scoring. This technical solution can load and access clusters in memory according to the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector. This technical solution can determine the local candidate pool corresponding to the clusters in the target cluster sequence through the search seed corresponding to the clusters in the target cluster sequence. This technical solution can then use the local candidate pool, based on the triangle inequality pruning rule, to sequentially perform local searches on the clusters in the target cluster sequence to obtain a global candidate set for the query vector.

[0020] The technical solution of this application can obtain an abstract node set by querying a vector, and obtain the cluster priority score of the cluster by clustering the abstract nodes of the abstract node set, thereby determining the cluster access sequence and reducing the problem of finding a valid result only after scanning multiple irrelevant clusters.

[0021] The technical solution of this application can prune vectors using the triangle inequality pruning rule, providing theoretical support for vector-level pruning, thereby reducing the problem of a large number of vectors being unnecessarily read from the disk and only used for later obsolescence.

[0022] The technical solution of this application can prune clusters by cluster priority scores and prune vectors by triangle inequality pruning rules. Combined with memory mapping and hot spot caching strategies, the average number of external memory accesses per query is significantly reduced under similar recall rates. This can achieve higher QPS and lower tail latency under the constraint of limited memory on a single machine, thereby improving I / O efficiency and throughput. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating method 1000 according to an embodiment of this application is shown; Figure 2 A flowchart illustrating step S160 according to an embodiment of this application is shown; Figure 3 A flowchart illustrating step S170 according to an embodiment of this application is shown; Figure 4 A flowchart illustrating method 2000 according to an embodiment of this application is shown; Figure 5 A flowchart illustrating step S210 according to an embodiment of this application is shown; Figure 6 A flowchart illustrating step S212 according to an embodiment of this application is shown; Figure 7 A flowchart illustrating step S213 according to an embodiment of this application is shown; Figure 8 A flowchart illustrating step S214 according to an embodiment of this application is shown; Figure 9 A flowchart illustrating step S2144 according to an embodiment of this application is shown; Figure 10 A schematic diagram of the external storage vector retrieval device according to an embodiment of this application is shown.

[0025] Explanation of reference numerals in the attached figures: External storage vector retrieval device 30; query processing unit 31; index building unit 32; storage management unit 33; configuration monitoring unit 34. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0027] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0028] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0029] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order.

[0030] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0031] The English terms used in this application, their full English names, and their corresponding Chinese definitions are as follows: HNSW, Hierarchical Navigable Small World; NSG, Navigating Spreading-out Graph; IVF, Inverted File Index; LSH, Locality-Sensitive Hashing; I / O, Input / Output; Mmap, Memory-mapped, refers to memory mapping. MADV_RANDOM, Memory Advice Random, provides memory access suggestions for random access. MADV_DONTNEE, Memory Advice Don't Need, memory access advice - no longer needed (recommends the kernel to release the memory). QPS, Queries Per Second; K-Means, K-Means Clustering, K-means clustering algorithm; kNN-Descent, k-Nearest Neighbors Descent, is a K-nearest neighbor descent algorithm. API, Application Programming Interface; RAG, Retrieval-Augmented Generation; SSD, Solid State Drive.

[0032] See Figure 10 The external storage vector retrieval device 30 based on graph abstraction provided in this application includes a query processing unit 31, an index building unit 32, a storage management unit 33, and a configuration monitoring unit 34.

[0033] The following is combined with Figure 9 This application describes an external memory vector retrieval method 1000 based on graph abstraction. See also... Figure 1 Method 1000 includes steps S110-S170.

[0034] In step S110, the received query vector is preprocessed to obtain a query vector of a preset type.

[0035] According to the example embodiment, a query vector can be a vectorized representation of finding and locating relevant nodes, subgraphs, or patterns in graph-structured data. Query vectors can be semantic query vectors, image query vectors, etc. Preprocessing can include normalization, standardization, dimensionality checking, and data cleaning. A preset type query vector can be a preprocessed vector with the same vectorized form (e.g., dimensions). The original vector can be stored in external storage.

[0036] For example, in step S110, the query processing unit 31 performs preprocessing on the received query vector, such as normalization, standardization, dimension checking, and data cleaning, to obtain a query vector of a preset type.

[0037] In step S120, the preset type query vector and the number of received target returns are input into the pre-constructed graph abstract index to obtain the set of abstract nodes corresponding to the preset type query vector.

[0038] According to the example embodiment, the target number of results returned can be the upper limit of the number of most relevant (or most similar) results to be returned in the retrieval of the query vector. The pre-built graph abstract index can be an index structure with an abstract hierarchy built based on the original vectors; for example, the graph abstract index can be a structure such as HNSW. The graph abstract index can include local index files and vector data files. The local index files and vector data files are mapped to a virtual address space via mmap. The set of abstract nodes can be a set of abstract nodes in the graph abstract index that are similar to the query vector of the preset type.

[0039] For example, in step S120, the query processing unit 31 inputs a preset type query vector and the number of received target returns into a pre-constructed graph abstraction index to perform an approximate k-nearest neighbor search. The graph abstraction index outputs a set of abstract nodes, thus obtaining the abstract node set. Search parameters may include a probe budget and a candidate pool size. The probe budget controls the number of abstract nodes accessed, and the candidate pool size controls the search width of the abstraction layer.

[0040] In step S130, the cluster priority score and search seed of the cluster to which the abstract node of the abstract node set belongs are obtained.

[0041] According to the example embodiment, the abstract nodes in the abstract node set may belong to different clusters. The graph abstract index includes a number of clusters. The cluster priority score refers to a quantitative evaluation score of the relevance of a cluster to a preset type query vector. The abstract nodes of a selected cluster can be called hit nodes, and the number of hit nodes can be used as the cluster priority score (denoted as C_P) of that cluster. i The search seed can be the starting point for searching a cluster. The few abstract nodes among the hit nodes that are closest to the preset type query vector can be used as the search seed.

[0042] For example, in step S130, the query processing unit 31 can cluster the clusters to which the abstract nodes of the abstract node set belong, so as to obtain the number of hit nodes in the cluster as the cluster priority score, and take the several abstract nodes among the hit nodes that are closest to the preset type query vector as search seeds.

[0043] In step S140, the cluster access sequence is determined based on the cluster priority score.

[0044] According to the example embodiment, the cluster access sequence can be a sequential sequence of accessing clusters. For example, in step S140, the query processing unit 31 can sort the cluster priority scores in descending order to determine the cluster access sequence. Clusters with higher cluster priority scores, i.e., clusters with more hit nodes, are more likely to contain the true nearest neighbor vectors of the preset type query vector. Therefore, prioritizing access to clusters with higher cluster priority scores can obtain high-quality candidate results faster, improve the success rate of subsequent early stopping, and reduce unnecessary cluster access and I / O. When accessing clusters, the query processing unit 31 performs pointer operations based on the node ID and offset, without directly calling the read system call. All I / O is automatically triggered by the operating system through page faults, such as MADV_RANDOM, MADV_DONTNEED, etc., which prompt optimization of kernel prefetching and reclamation strategies.

[0045] In step S150, clusters are loaded and accessed in memory according to the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector.

[0046] According to the example embodiment, the target cluster sequence can be a candidate cluster sequence of the true nearest neighbor vectors of the preset type query vector. That is, the target cluster sequence can be a cluster order sequence with higher cluster priority scores. For example, in step S150, the query processing unit 31 can load and access clusters in memory according to the order of the cluster access sequence, prune clusters with lower cluster priority scores, and retain clusters with higher cluster priority scores, thereby determining the target cluster sequence corresponding to the query vector.

[0047] In step S160, the local candidate pool corresponding to the clusters in the target cluster sequence is determined according to the search seed corresponding to the clusters in the target cluster sequence.

[0048] According to the example embodiment, the local candidate pool can be a set of vector candidates that have been initially screened. For example, in step S160, the query processing unit 31 determines the local candidate pool corresponding to the clusters in the target cluster sequence based on the search seed corresponding to the clusters in the target cluster sequence. The query processing unit 31 can determine the search starting point of the target cluster based on the search seed of the target cluster. The query processing unit 31 can determine the local candidate pool based on the search starting point.

[0049] In step S170, based on the local candidate pool and the triangle inequality pruning rule, the clusters in the target cluster sequence are searched locally in turn to obtain the global candidate set of the query vector.

[0050] According to the example embodiment, the triangle inequality pruning rule can be a rule for pruning vectors using the property of the triangle inequality satisfied by distance. Local search can be a search of the original vectors within a cluster. The global candidate set can be the set of nearest neighbor vectors after searching the query vector. For example, the global candidate set can be the global top-k set. The global candidate set can be obtained from the local candidate pools of all clusters in the target cluster sequence.

[0051] For example, in step S170, the query processing unit 31 can perform local searches on the clusters in the target cluster sequence according to the local candidate pool and the triangle inequality pruning rule to obtain the global candidate set of the query vector.

[0052] Through the above embodiments, the technical solution of this application can obtain a preset type query vector by preprocessing the received query vector. The technical solution of this application can input the preset type query vector and the received target return quantity into a pre-constructed graph abstract index to obtain the set of abstract nodes corresponding to the preset type query vector. The technical solution of this application can obtain the cluster priority score and search seed of the cluster by clustering the abstract nodes of the abstract node set.

[0053] The technical solution of this application can determine the cluster access sequence through cluster priority scoring. This technical solution can load and access clusters in memory according to the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector. This technical solution can determine the local candidate pool corresponding to the clusters in the target cluster sequence through the search seed corresponding to the clusters in the target cluster sequence. This technical solution can then use the local candidate pool, based on the triangle inequality pruning rule, to sequentially perform local searches on the clusters in the target cluster sequence to obtain a global candidate set for the query vector.

[0054] The technical solution of this application can obtain an abstract node set by querying a vector, and obtain the cluster priority score of the cluster by clustering the abstract nodes of the abstract node set, thereby determining the cluster access sequence and reducing the problem of finding a valid result only after scanning multiple irrelevant clusters.

[0055] The technical solution of this application can prune vectors using the triangle inequality pruning rule, providing theoretical support for vector-level pruning, thereby reducing the problem of a large number of vectors being unnecessarily read from the disk and only used for later obsolescence.

[0056] The technical solution of this application can prune clusters by cluster priority scores and prune vectors by triangle inequality pruning rules. Combined with memory mapping and hot spot caching strategies, the average number of external memory accesses per query is significantly reduced under similar recall rates. This can achieve higher QPS and lower tail latency under the constraint of limited memory on a single machine, thereby improving I / O efficiency and throughput.

[0057] Optionally, see Figure 2 Step S160 may include steps S0161 and S162.

[0058] In step S161, the search starting point of the clusters in the target cluster sequence is determined according to the search seed corresponding to the clusters in the target cluster sequence.

[0059] According to the example embodiment, the search starting point can be a specific search starting point for a cluster. The search seed is the set of IDs (Seeds) in the local index of the clusters in the target cluster sequence. i The query processing unit 31 selects the ID set Seeds. i The one or more closest to the preset type query vector in the middle distance are used as the search starting point for this cluster.

[0060] In step S162, a local candidate pool is determined based on the search starting point.

[0061] According to the example embodiment, the local candidate pool is a priority queue with a maximum length of L. The local candidate pool includes several candidate vectors. The query processing unit 31 can insert the search starting point into the local candidate pool and mark it as visited as the initial local candidate pool. The query processing unit 31 can expand other vectors within the cluster according to the edges of the NSG policy, thereby determining the local candidate pool.

[0062] Through the above embodiments, the technical solution of this application can determine the search starting point of clusters in the target cluster sequence using the search seed corresponding to the clusters in the target cluster sequence. The technical solution of this application can determine a local candidate pool using the search starting point.

[0063] Optionally, see Figure 3 Step S170 may include steps S171-S175.

[0064] In step S171, the target node is determined based on the local candidate pool.

[0065] According to the example embodiment, the target node can be the node in the cluster that is closest to the preset type query vector in the current access. For example, the query processing unit 31 can calculate the distance between the candidate nodes in the local candidate pool and the preset type query vector, and select the node closest to the preset type query vector as the target node.

[0066] In step S172, the difference between the distance between the target node and the preset type query vector and the distance between the target node and its neighboring nodes is determined.

[0067] According to the example embodiment, the neighbors of the target node can be the neighboring nodes of the target node. When the application primarily uses semantic retrieval, inner product distance or cosine similarity distance can be used. When the application primarily uses image feature retrieval, Euclidean distance can be used.

[0068] For example, let the target node popped from the local candidate pool be v_i, and the distance between target node v_i and the preset type query vector q be d(q, v_i). Let v_j be a neighbor of target node v_i, and the distance between target node v_i and neighbor v_j be d(v_i, v_j). Let d_max be the distance between the farthest vector in the current global candidate set (i.e., the global top-k set) and the preset type query vector q. The triangle inequality is given as follows: .

[0069] Wherein, d(v_i, v_j) has been pre-computed and stored when constructing the graph abstract index. Query processing unit 31 can calculate the difference (denoted as LB(q, v_j)): .

[0070] In step S173, if the difference is less than or equal to the farthest distance threshold of the global candidate set, the local candidate pool is updated based on the neighboring nodes of the target node.

[0071] According to the example implementation, the farthest distance threshold of the global candidate set can be the distance between the farthest vector in the current global candidate set and the query vector of the preset type, i.e., d_max.

[0072] For example, if LB(q, v_j) > d_max, then v_j is unlikely to enter the final global candidate set, and therefore there is no need to load the original vector of v_j from external memory; the candidate v_j is directly pruned. Only when LB(q, v_j) ≤ d_max does the query processing unit 31 trigger an external memory read operation to load the original vector of v_j and calculate the precise distance d(q, v_j).

[0073] The query processing unit 31 can compare the maximum distance between d(q, v_j) and the candidate vectors in the local candidate pool with the preset type query vector q.

[0074] If d(q, v_j) is greater than or equal to the maximum distance between the candidate vector in the local candidate pool and the query vector q of the preset type, and the local candidate pool is full, then the neighbor v_j of the target node will not be added to the local candidate pool.

[0075] If the local candidate pool is not full, the neighboring node v_j of the target node is added to the local candidate pool.

[0076] If d(q, v_j) is less than the maximum distance between the candidate vector in the local candidate pool and the preset type query vector q, then the neighbor v_j of the target node is added to the local candidate pool. At this time, the candidate vector corresponding to the maximum distance between the candidate vector in the local candidate pool and the preset type query vector q needs to be deleted.

[0077] The query processing unit 31 can traverse all the neighbors of the target node in the cluster to determine the local candidate pool of the cluster. When the local candidate pool is updated, the global candidate set is updated accordingly.

[0078] In step S174, all clusters in the target cluster sequence are traversed to update the entire local candidate pool.

[0079] According to the example embodiment, the query processing unit 31 can traverse all clusters in the target cluster sequence to determine all local candidate pools.

[0080] During the local search process of traversing all clusters in the target cluster sequence, whenever the local search of a cluster updates the global candidate set, the query processing unit 31 resets the "unimproved cluster count". If the global candidate set is not updated for N consecutive clusters (e.g., N = the target return number or a fixed constant), the remaining clusters are considered to have a low probability of contributing, and cluster access can be terminated early to reduce I / O.

[0081] In step S175, the global candidate set is determined based on the updated local candidate pools.

[0082] For example, query processing unit 31 can determine the global candidate set based on the updated candidate vectors of all local candidate pools.

[0083] Through the above embodiments, the technical solution of this application can determine the target node by determining a local candidate pool. The technical solution of this application determines the difference between the distance between the target node and a preset type query vector and the distance between the target node and its neighbors. If the difference is less than or equal to the farthest distance threshold of the global candidate set, the technical solution of this application updates the local candidate pool based on the neighbors of the target node. The technical solution of this application traverses all clusters in the target cluster sequence to update all local candidate pools. The technical solution of this application determines the global candidate set using the updated local candidate pools.

[0084] The technical solution of this application can prune vectors using the triangle inequality pruning rule. The vector pruning process only depends on the calculated distance between the target node and the preset type query vector and the pre-stored distance between the target node and its neighbors. It does not involve additional external storage access and therefore does not introduce additional I / O overhead.

[0085] Optionally, the query processing unit 31 can receive multiple query vectors simultaneously. In scenarios with concurrent query vectors, the same original vector may be accessed multiple times. The query processing unit 31 can maintain a lightweight cache table in user space, which only caches hot vectors that are "not pruned and have been hit multiple times". Since the number of vectors accessed in each query is usually in the O(log n) range, the cache size can be controlled within a small range, without causing significant memory pressure. When the query processing unit 31 receives multiple query vectors simultaneously, it can independently execute steps S110-S170 according to each query vector.

[0086] The following is combined with Figure 10 This application describes an external memory vector retrieval method based on graph abstraction, 2000. See also... Figure 4 Method 2000 includes steps S210-S280.

[0087] Step S210 is the step of building the index directory file. See also Figure 5 Step S210 includes steps S211-S215.

[0088] In step S211, the original vector set is preprocessed to obtain a preset type of original vector set.

[0089] According to the example embodiment, the original vector set can be a collection of original data objects. Preprocessing may include standardization, dimensionality checking, and data cleaning. A preset type original vector set can be a collection of preprocessed original vectors with a uniform type. The preset type original vector set can be a collection of original vectors of type float32, etc.

[0090] For example, the index building unit 32 can perform preprocessing such as standardization, dimension checking and data cleaning on the original vector set to make the dimensions of all original vectors consistent, thus obtaining a set of original vectors of a preset type.

[0091] In step S212, based on a preset clustering algorithm, the preset type of original vector set is assigned to a preset number of clusters to obtain a cluster set.

[0092] According to the example implementation, the preset clustering algorithm can be an algorithm such as K-Means or Inverted Index File (IVF). The preset number of clusters can be the number of clusters. The preset number of clusters can be set according to the data size, target query performance, and memory budget. The cluster set can be a set of clusters containing several preset type original vectors.

[0093] For example, index building unit 32 can use the K-Means algorithm to allocate a set of original vectors of a preset type to clusters of a preset number to obtain a set of clusters.

[0094] Optionally, see Figure 6 Step S212 may include steps S2121-S2123.

[0095] In step S2121, the sampled vectors of the original vector set of the preset type are assigned to the clusters of the preset number of clusters to obtain the initial cluster set.

[0096] According to the example embodiment, the initial cluster set can be a set of clusters obtained after initially allocating a portion of the preset type original vectors in the preset type original vector set. For example, the index building unit 32 can extract a portion of the preset type original vectors and allocate them to a preset number of clusters to obtain the initial cluster set.

[0097] In step S2122, the initial cluster set is trained with centroids according to a preset number of rounds to determine the centroid of each cluster in the trained initial cluster set.

[0098] According to the example embodiment, the centroid of a cluster can be the geometric center of the cluster. The preset round can be the round of centroid training. For example, the index building unit 32 can continue to extract the unassigned portion of the preset type original vectors, and based on the distance between the preset type original vectors and the centroids of the clusters in the previous round, assign the unassigned portion of the preset type original vectors to the nearest cluster, and then calculate the centroid of the clusters in the current round. After the index building unit 32 performs centroid training on the initial cluster set according to the preset round, it can optimize the centroids of the clusters.

[0099] In step S2123, the preset type original vectors of the preset type original vector set are assigned to the nearest cluster to the centroid of each cluster of the initial cluster set after training, so as to obtain the cluster set.

[0100] According to the example embodiment, after centroid training in a preset number of rounds, the index building unit 32 can assign the original vector of the preset type to the cluster closest to the centroid to obtain a cluster set.

[0101] The index building unit 32 counts the number of preset type original vectors and the ID range of preset type original vectors within each cluster and stores the information in the cluster metadata table.

[0102] In step S213, a local graph index for each cluster in the cluster set is constructed based on the preset type original vectors allocated within the cluster set.

[0103] According to the example embodiment, the local graph index can be a clustering of the local subgraphs formed by a node and its neighbors in a graph (network). The local graph index can be an NSG-type graph or similar structure. For example, the index building unit 32 can construct the local graph index of each cluster in the cluster set based on the nearest neighbor graph algorithm and according to the preset type of original vectors allocated within the cluster set.

[0104] Optionally, see Figure 7 Step S213 may include steps S2131-S2132.

[0105] In step S2131, based on the nearest neighbor graph algorithm, the initial nearest neighbor graph is determined according to the preset type of original vectors allocated within the cluster set.

[0106] According to the example embodiment, the nearest neighbor graph algorithm can be an algorithm such as kNN-Descent. The initial nearest neighbor graph can be an initial k-neighbor graph.

[0107] For example, the index building unit 32 can use the kNN-Descent algorithm to build an initial k-neighbor graph based on the original vectors of a preset type allocated within the cluster set.

[0108] In step S2132, the local graph index is determined based on the initial nearest neighbor graph.

[0109] According to the example embodiment, the index building unit 32 can perform sparsification and topology optimization on the initial k-neighbor graph to generate an NSG-type graph with good connectivity and search performance, i.e., a local graph index.

[0110] During the local graph index construction process, the index construction unit 32 can assign a local index ID to each preset type of original vector, and record the ID of the adjacent vector and the distance between them (or a compressed representation of the distance) for each edge.

[0111] After the local graph index is constructed, the index construction unit 32 can write the local graph index to external memory in a sequential layout. The local graph index written to external memory can include an adjacency list area, an edge weight area, and a metadata area. The adjacency list area can store the outgoing edge list of each preset type original vector node. The edge weight area can store the pre-calculated distances between preset type original vectors. The metadata area can store the number of nodes (i.e., the number of preset type original vectors), the average degree of preset type original vectors, the offset of preset type original vectors, etc.

[0112] In step S214, the graph abstract index is determined based on the cluster set.

[0113] According to the example embodiment, a graph abstraction index can be a structure that optimizes graph data storage and retrieval through hierarchical abstraction. For example, index building unit 32 can determine the graph abstraction index based on the cluster set.

[0114] Optionally, see Figure 8 Step S214 may include steps S2141-S2144.

[0115] In step S2141, the centroid set of the cluster set is determined based on the centroid vector of each cluster in the cluster set.

[0116] According to the example embodiment, the centroid vector can be a vector representing the center position of a cluster. The centroid set can be a set of centroid vectors. For example, the index building unit 32 can take the centroid vector of each cluster in the cluster set to form a centroid set.

[0117] In step S2142, the abstract node set is determined based on the sampling vector of each cluster in the cluster set.

[0118] According to the example embodiment, the sampling vector can be a number of real, preset-type original vectors randomly or uniformly sampled from the cluster. The number of preset-type original vectors in the sampling vector can be adjusted according to the size of the cluster. The abstract node set can be a set of nodes representing the cluster.

[0119] For example, the index building unit 32 can randomly or uniformly sample several real preset type original vectors from each cluster of the cluster set to form the sampling vector of the cluster, and then incorporate the sampling vector into the abstract node set of the cluster.

[0120] In step S2143, the set of abstract nodes of the graph is determined based on the set of centroids and the set of abstract nodes.

[0121] According to the example embodiment, the graph abstraction node set can be a higher-level node set that aggregates original nodes (i.e., original vectors of a preset type) through hierarchical abstraction.

[0122] For example, index building unit 32 can merge the centroid set and all abstract node sets to form a graph abstract node set. For example, the graph abstract node set can be represented as:

[0123] Where G_A_nodes is the set of abstract nodes in the graph; CT is the set of centroids; S i Let be the set of abstract nodes for the i-th cluster in the cluster set.

[0124] In step S2144, the graph abstract index is determined based on the graph abstract node set.

[0125] According to the example embodiment, the graph abstract index can be a multi-level abstract representation structure of a graph structure. For example, the index building unit 32 can adopt the HNSW strategy to build the graph abstract index based on the set of graph abstract nodes.

[0126] Optionally, see Figure 9 Step S2144 may include steps S21441-S21443.

[0127] In step S21441, a set of graph abstract nodes is inserted into the initial graph abstract index according to a preset hierarchical rule.

[0128] According to the example embodiment, the initial graph abstract index can be a blank HNSW index structure. The preset hierarchy rule can be a rule that proceeds layer by layer from the bottom up.

[0129] For example, the index building unit 32 can insert abstract nodes from the graph abstract node set into the initial graph abstract index layer by layer from the bottom layer upwards, with the bottom layer having the densest abstract nodes and the top layer having the sparsest abstract nodes.

[0130] In step S21442, the adjacent edges of the abstract nodes in the graph abstract node set are determined.

[0131] According to the example embodiment, the index building unit 32 can select several nearest neighbors as edges for each abstract node in the graph abstract node set, and select the nearest abstract node in the current layer as the neighboring edge of the abstract node.

[0132] In step S21443, a graph abstract index is constructed based on the initial graph abstract index and adjacent edges after inserting the graph abstract node set.

[0133] According to the example embodiment, the index building unit 32 can form a hierarchical graph structure with small-world characteristics based on the initial graph abstract index and adjacent edges after inserting the graph abstract node set, thereby constructing the graph abstract index.

[0134] The index building unit 32 can record the cluster number to which the node belongs and the information mapped back to the preset type of original vector or local graph index ID in the abstract node metadata.

[0135] In step S215, the index directory file is determined for the local graph index and the graph abstraction index.

[0136] According to the example embodiment, the index directory file can be a metadata file that manages index information. The index directory file may include cluster metadata, local index file paths, and vector data file paths.

[0137] The index building unit 32 can serialize the graph abstract index and store it on the local disk in a compact structure. The index building unit 32 can uniformly register cluster metadata, local index file paths, and vector data file paths in the index directory file. The storage management unit 33 can manage the external storage layout and memory mapping of the index directory file and the original vector files. The configuration monitoring unit 34 can verify, statistically analyze, and record version information of the index directory file to ensure secure online loading.

[0138] See Figure 4 Steps S220-S280 are the same as steps S110-S170 in method 1000 above, so they will not be described again.

[0139] Through the above embodiments, the technical solution of this application can capture global and local structural information simultaneously by incorporating the cluster centroid and the intra-cluster sampling vector into the graph abstract index. This allows for a more accurate assessment of the relevance of each cluster to the query vector during the global cluster search phase, thereby reducing access to irrelevant clusters and external storage I / O.

[0140] The technical solution of this application can decouple clustering, graph abstraction index, local graph index and distance metric through modular API, support multiple index structures and hardware configurations, facilitate integration with existing vector databases, search systems or RAG frameworks, and has good engineering implementation value and scalability.

[0141] According to another aspect of this application, this application also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of implementing the external memory vector retrieval method based on graph abstraction as described above.

[0142] According to another aspect of this application, this application also provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the external memory vector retrieval method based on graph abstraction as described above.

[0143] According to another aspect of this application, this application also provides a computer program product, including: a computer program stored on a computer-readable storage medium; the computer program includes program instructions that, when executed by a computer, cause the computer to perform the external storage vector retrieval method based on graph abstraction as described above.

[0144] Finally, it should be noted that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions of the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for external memory vector retrieval based on graph abstraction, characterized in that, The method includes: The received query vector is preprocessed to obtain a query vector of a preset type; The preset type query vector and the number of target returns received are input into the pre-constructed graph abstract index to obtain the set of abstract nodes corresponding to the preset type query vector; Based on the cluster cluster to which the abstract nodes of the abstract node set belong, the cluster priority score and search seed of the respective cluster are obtained; The cluster access sequence is determined based on the cluster priority score; The clusters are loaded and accessed in memory according to the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector; Based on the search seed corresponding to the cluster in the target cluster sequence, determine the local candidate pool corresponding to the cluster in the target cluster sequence; Based on the local candidate pool, and using the triangle inequality pruning rule, local searches are performed sequentially on the clusters in the target cluster sequence to obtain the global candidate set of the query vector.

2. The method according to claim 1, characterized in that, The step of determining the local candidate pool corresponding to the clusters in the target cluster sequence based on the search seed corresponding to the clusters in the target cluster sequence includes: The starting point for searching clusters in the target cluster sequence is determined based on the search seed corresponding to the clusters in the target cluster sequence; The local candidate pool is determined based on the search starting point.

3. The method according to claim 1, characterized in that, The step involves performing a local search on each cluster in the target cluster sequence based on the local candidate pool and the triangle inequality pruning rule to obtain a global candidate set for the query vector, including: The target node is determined based on the local candidate pool; Determine the difference between the distance between the target node and the preset type query vector and the distance between the target node and its neighbors; If the difference is less than or equal to the farthest distance threshold of the global candidate set, the local candidate pool is updated based on the neighbors of the target node; Traverse all clusters in the target cluster sequence to update the entire local candidate pool; The global candidate set is determined based on the updated local candidate pools.

4. The method according to claim 1, characterized in that, Before preprocessing the received query vector to obtain a query vector of a preset type, the method further includes: The steps for building the index directory file include: The original vector set is preprocessed to obtain a pre-defined original vector set; Based on a preset clustering algorithm, the original vector set of the preset type is assigned to a preset number of clusters to obtain a cluster set; Based on the preset type of original vectors allocated within the cluster set, construct the local graph index of each cluster in the cluster set; Determine the graph abstraction index based on the cluster set; For the local graph index and the graph abstraction index, determine the index directory file.

5. The method according to claim 4, characterized in that, The method of allocating the original vector set of the preset type to a preset number of clusters based on a preset clustering algorithm to obtain a cluster set includes: The sampled vectors of the original vector set of the preset type are assigned to clusters of a preset number to obtain an initial cluster set; The initial cluster set is trained with centroids according to a preset number of rounds to determine the centroid of each cluster in the trained initial cluster set. The preset type original vectors of the preset type original vector set are assigned to the nearest cluster to the centroid of each cluster of the initial cluster set after training, so as to obtain the cluster set.

6. The method according to claim 4, characterized in that, The step of constructing a local graph index for each cluster in the cluster set based on the pre-defined type of raw vectors allocated within the cluster set includes: Based on the nearest neighbor graph algorithm, the initial nearest neighbor graph is determined according to the preset type of original vectors allocated within the cluster set; The local graph index is determined based on the initial nearest neighbor graph.

7. The method according to claim 4, characterized in that, The step of determining the graph abstract index based on the cluster set includes: The centroid set of the cluster set is determined based on the centroid vector of each cluster in the cluster set; Based on the sampling vector of each cluster in the cluster set, determine the abstract node set; Based on the centroid set and the abstract node set, determine the graph abstract node set; The graph abstract index is determined based on the set of graph abstract nodes.

8. The method according to claim 7, characterized in that, Determining the graph abstract index based on the graph abstract node set includes: Insert the set of graph abstract nodes into the initial graph abstract index according to a preset hierarchical rule; Determine the adjacent edges of the abstract nodes in the set of abstract nodes in the graph; The graph abstract index is constructed based on the initial graph abstract index after inserting the set of graph abstract nodes and the adjacent edges.

9. An external memory vector retrieval device based on graph abstraction, characterized in that, The device includes: The query processing unit preprocesses the received query vector to obtain a query vector of a preset type. The query processing unit inputs the preset type query vector and the target return quantity into the pre-constructed graph abstract index to obtain the set of abstract nodes corresponding to the preset type query vector; The query processing unit clusters the abstract nodes of the abstract node set according to their respective clusters to obtain the cluster priority score and search seed of each cluster. The query processing unit determines the cluster access sequence based on the cluster priority score; The query processing unit accesses clusters in the order of the cluster access sequence to determine the target cluster sequence corresponding to the query vector; The query processing unit determines the local candidate pool corresponding to the clusters in the target cluster sequence based on the search seed corresponding to the clusters in the target cluster sequence; The query processing unit performs local searches on the clusters in the target cluster sequence according to the local candidate pool and the triangle inequality pruning rule, so as to obtain the global candidate set of the query vector.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the external memory vector retrieval method based on graph abstraction as described in any one of claims 1-8.

Images