Vector retrieval method and device, equipment and storage medium
By constructing graph indexes in vector retrieval and storing raw and compressed vectors in a hierarchical manner, the problem of high vector retrieval delay in shared storage environment is solved, and more efficient data access performance is achieved.
Patent Information
- Application Number
- CN202411013010.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
The existing vector search solutions have high latency and poor adaptability in shared storage application scenarios, resulting in poor performance in data centers, cloud computing environments and other scenarios.
By constructing a graph index, the original vector is stored in the second-level storage resource, the compressed vector is stored in the first-level storage resource, and after obtaining the compressed vector with the highest similarity from the first-level storage resource during query, Obtain corresponding original vectors from the secondary storage resources, reduce synchronous data access operations during the search process, and improve data access performance.
It effectively reduces the delay of vector retrieval, so that the vector retrieval scheme can better adapt to the shared storage environment, and improves data access efficiency.
Smart Images

Figure CN118964686A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present application relate to the field of database technology, and in particular, to a vector retrieval method, apparatus, device, and storage medium. Background Art
[0002] Vector retrieval is a technique that searches for the vector that is most similar to a given query vector in a high-dimensional space. This technique has applications in many fields, including but not limited to computer vision, natural language processing, recommendation systems, and information retrieval. In vector retrieval, data is usually represented as multidimensional vectors that exist in a multidimensional space, where each dimension can represent a feature, and then similarity measures are performed on these vectors to find the most similar vectors. The goal of vector retrieval is to find a set of vectors that are most similar to the query vector in a large vector database.
[0003] Shared storage refers to storage resources that are shared among multiple computers. This storage method allows multiple servers or workstations on a network to access the same data set without duplicating the data on each node. Shared storage is a key component in many IT architectures and system designs, especially in data centers, cloud computing environments, and high-performance computing scenarios. The main purpose of shared storage is to improve data availability, accessibility, and fault tolerance.
[0004] Shared storage has the characteristics of high latency and high throughput, which results in poor adaptability of existing vector retrieval solutions in shared storage application scenarios. Therefore, it is desired to provide a vector retrieval solution that can better adapt to shared storage. Summary of the invention
[0005] One or more embodiments of the present application provide the following technical solutions:
[0006] The present application provides a vector retrieval method; wherein a graph index constructed based on an original vector and a compressed vector corresponding to the original vector are stored in a first-level storage resource; a node in the graph index is associated with a compressed vector corresponding to the original vector represented by the node, and an edge in the graph index represents a similar relationship between the original vectors corresponding to the nodes connected by the edge; the original vector is stored in a second-level storage resource; the data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; the method comprises:
[0007] In response to a query vector, acquiring, from the first-level storage resource, at least one target compressed vector having the highest similarity to the query vector based on the graph index, and after acquiring the at least one target compressed vector, acquiring, from the second-level storage resource, at least one target original vector corresponding to the at least one target compressed vector;
[0008] The similarities between the query vector and the at least one target original vector are respectively calculated, and the at least one target original vector is sorted based on the calculated similarities, so as to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0009] The present application also provides a vector retrieval device; wherein a graph index constructed based on an original vector and a compressed vector corresponding to the original vector are stored in a first-level storage resource; a node in the graph index is associated with a compressed vector corresponding to the original vector represented by the node, and an edge in the graph index represents a similar relationship between the original vectors corresponding to the nodes connected by the edge; the original vector is stored in a second-level storage resource; the data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; the device comprises:
[0010] an acquisition module, in response to a query vector, acquiring, from the first-level storage resource and based on the graph index, at least one target compressed vector having the highest similarity to the query vector, and acquiring, after acquiring the at least one target compressed vector, from the second-level storage resource, at least one target original vector corresponding to the at least one target compressed vector;
[0011] A determination module calculates the similarity between the query vector and the at least one target original vector respectively, and sorts the at least one target original vector based on the calculated similarity, so as to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0012] The present application also provides an electronic device, including:
[0013] processor;
[0014] a memory for storing processor-executable instructions;
[0015] The processor implements the steps of any of the above methods by running the executable instructions.
[0016] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.
[0017] In the above technical solution, the graph index constructed based on the original vector and the compressed vector corresponding to the original vector can be stored in the first-level storage resource, and the original vector can be stored in the second-level storage resource, while ensuring that the data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; wherein the nodes in the graph index can be associated with the compressed vector corresponding to the original vector represented by the node, and the edges in the graph index can indicate that there is a similarity relationship between the original vectors corresponding to the nodes connected by the edges. In this case, in response to the query vector, at least one target compressed vector with the highest similarity to the query vector can be obtained from the first-level storage resource based on the graph index, and after all the target compressed vectors are obtained, at least one target original vector corresponding to the at least one target compressed vector can be obtained from the second-level storage resource, and subsequently the similarity between the query vector and the at least one target original vector can be calculated respectively, and the at least one target original vector can be sorted based on the calculated similarity, so as to determine the vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0018] By adopting the above method, the nearest neighbor search process based on the graph index can be decoupled from the data access operation for the second-level storage resource such as shared storage for storing the original vector. It is no longer necessary to perform synchronous data access operations for the second-level storage resource during the search process. Instead, after obtaining the compressed vector with the highest similarity to the query vector from the first-level storage resource such as memory for storing the compressed vector corresponding to the original vector and the graph index built based on the original vector through the search process, the original vector corresponding to the compressed vector is obtained from the second-level storage resource. Therefore, the delay of vector retrieval can be reduced, so that this vector retrieval solution can be better adapted to shared storage. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The following is a description of the accompanying drawings required for describing the exemplary embodiments, wherein:
[0020] Figure 1 It is a schematic diagram of the architecture of a database system shown in an exemplary embodiment of the present application.
[0021] Figure 2 It is a data deployment diagram of a vector database system shown in an exemplary embodiment of the present application.
[0022] Figure 3 It is a schematic diagram of a graph index shown in an exemplary embodiment of the present application.
[0023] Figure 4 It is a schematic diagram of another graph index shown in an exemplary embodiment of the present application.
[0024] Figure 5 It is a flowchart of a vector retrieval method shown in an exemplary embodiment of the present application.
[0025] Figure 6 It is a structural schematic diagram of a device shown in an exemplary embodiment of the present application.
[0026] Figure 7 It is a block diagram of a vector retrieval device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0027] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. On the contrary, they are only examples consistent with some aspects of one or more embodiments of the present application.
[0028] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.
[0029] Vector retrieval is a technique that searches for the most similar vector to a given query vector in a high-dimensional space. This technique has applications in many fields, including but not limited to computer vision, natural language processing, recommendation systems, and information retrieval. In vector retrieval, data is usually represented as multidimensional vectors that exist in a multidimensional space, where each dimension can represent a feature. Similarity measures are then performed on these vectors (such as calculating the distance between these vectors) to find the most similar vectors. The goal of vector retrieval is to find a set of vectors that are most similar to the query vector in a large vector database.
[0030] Vector retrieval can be divided into two main types, namely Exact Nearest Neighbor Search (ENS) and Approximate Nearest Neighbor Search (ANNS). Exact nearest neighbor search returns the point that is exactly closest to the query vector. However, in high-dimensional space, exact nearest neighbor search can be very time-consuming because it is necessary to calculate the similarity between the query vector and each vector in the vector database. Therefore, approximate nearest neighbor search is usually used in practical applications, aiming to quickly find the vector that is relatively most similar to the query vector, but it does not guarantee to find the absolutely most similar vector; approximate nearest neighbor search speeds up the search process by organizing data into special index structures, such as KD trees (K-Dimensional Trees), hash tables, or HNSW (Hierarchical Navigable Small World graphs) graphs.
[0031] The HNSW algorithm is a memory-based solution to the approximate nearest neighbor search problem that is particularly suitable for large-scale data sets in high-dimensional space. The HNSW algorithm combines the characteristics of small-world networks with a multi-level index structure to achieve fast and accurate search. Among them, the small-world network is a graph consisting of some nodes and the connections between them, with two important characteristics: short path length and high clustering coefficient. Short path length means that the shortest path length between any two nodes is usually small, even when the network is large. High clustering coefficient means that the nodes in the network tend to form dense local clusters.
[0032] The HNSW algorithm builds a multi-level graph index, where each level contains a subgraph, and the subgraphs at higher levels are sparser than those at lower levels. Each node (representing a high-dimensional vector) can have multiple links in different levels, which enables the algorithm to search from global to local, thereby quickly converging to the nearest neighbor node.
[0033] The HNSW graph consists of multiple layers, each containing fewer nodes, and the lowest layer contains all the nodes. In each layer, the nodes form a connected graph, where each point is connected to a certain number of neighboring points, which are called links. The link density of each layer usually increases gradually. In this way, when searching, you can start from the high level and gradually go down to the lower level until you reach the lowest level, thereby reducing the number of candidate points that need to be checked.
[0034] The indexing process in the HNSW algorithm can include: selecting a node as the root node and starting to build the highest level of the HNSW graph; when a new node needs to be inserted, it starts from the highest level, randomly selects a point as the starting point, and then moves to the lower level through a greedy strategy until a suitable position is found to insert the new point. During the insertion process, the algorithm calculates the distance between the new point and each point in the current layer (indicating the distance between the corresponding vectors) and decides whether to establish a new link.
[0035] The search process in the HNSW algorithm can include: starting the search from the highest level, selecting a starting point (entrypoint), which can be randomly selected or selected by some strategy, such as selecting the known point closest to the query point; in the current level, checking all nodes connected to the starting point, and calculating the distance between these nodes and the query point, selecting the node closest to the query point as the target of the next search, and moving to the next level, repeating this process; in each level, the algorithm will traverse the neighboring nodes of the current node, looking for a node closer to the query point, if a closer node is found, it will be set as the current node and continue searching in this level; eventually, the search will reach the lowest level, which contains all the nodes, in which the search process will traverse the nodes adjacent to the current best node again to further narrow the range of the nearest neighbors. During the entire search process, the algorithm will record the several candidate points closest to the query point encountered, and when the search is completed, these candidate points will be sorted according to the distance to the query point, and the closest point will be regarded as the approximate nearest neighbor of the query point.
[0036] Due to the multi-level structure and short path length, the HNSW algorithm is able to achieve fast approximate nearest neighbor search in high-dimensional space. In addition, the HNSW algorithm can effectively process large-scale data sets because it only needs to maintain a small number of links associated with each node.
[0037] The DiskANN scheme is a disk-based vector retrieval scheme.
[0038] The DiskANN (Disk-based Approximate Nearest Neighbors) algorithm is an approximate nearest neighbor search algorithm that is particularly suitable for processing large-scale, high-dimensional data sets that cannot be fully loaded into memory. Traditional approximate nearest neighbor searches are usually performed entirely in memory. When the data set is too large to be stored entirely in memory, traditional approximate nearest neighbor searches are no longer applicable. The DiskANN algorithm overcomes this limitation by storing data in layers in memory and disk and using multi-level indexes.
[0039] The DiskANN algorithm uses a hierarchical storage structure to keep the most frequently accessed data in memory, while less active data is stored on disk, which enables it to process data sets that are much larger than memory. The DiskANN algorithm can also use HNSW graph indexes to quickly locate the nearest neighbors, and the complete HNSW graph index is usually stored on disk. Therefore, for the DiskANN algorithm, it is necessary to perform disk I / O (Input / Output) operations multiple times during the search process, and the disk I / O operations performed are synchronous I / O operations, that is, each time an I / O request is initiated to the disk, it is necessary to wait for the corresponding I / O result to determine the next search target through the HNSW graph index stored on the disk.
[0040] Existing vector retrieval schemes such as the DiskANN algorithm usually require multiple synchronous disk I / O operations during the search process. Therefore, these vector retrieval schemes usually have a high disk I / O cost, that is, more disk resources are consumed when performing I / O operations, and it takes a long time, resulting in a high delay in vector retrieval.
[0041] Shared storage refers to storage resources that are shared among multiple computers. This storage method allows multiple servers or workstations on a network to access the same data set without duplicating the data on each node. Shared storage is a key component in many IT architectures and system designs, especially in data centers, cloud computing environments, and high-performance computing scenarios. The main purpose of shared storage is to improve data availability, accessibility, and fault tolerance.
[0042] Shared storage can be classified according to different technical implementations, including Network Attached Storage (NAS), Storage Area Network (SAN), Object Storage (Object Storage), Software-Defined Storage (SDS), Distributed File System (DFS) and cloud storage.
[0043] Shared storage has the characteristics of high latency and high throughput. When applying existing vector retrieval solutions in the application scenario of shared storage, it usually means storing the vector database in the shared storage. In actual applications, the storage medium of shared storage is usually a disk. Therefore, in addition to the disk I / O cost of the existing vector retrieval solution itself, it may also bring additional network I / O costs due to the need to transmit data packets on the network in the application scenario of shared storage, that is, it may be necessary to consume additional network resources and time when performing I / O operations, resulting in further increase in the latency of vector retrieval. In addition, since shared storage itself has the characteristics of high latency, applying existing vector retrieval solutions in the application scenario of shared storage will further increase the latency of shared storage, which may make the latency of shared storage difficult for users to accept, affecting the user experience when using shared storage. Therefore, the existing vector retrieval solutions are poorly compatible with shared storage.
[0044] The present application provides a technical solution for vector retrieval, aiming to reduce the latency of vector retrieval so as to better adapt to shared storage.
[0045] In the vector retrieval scheme provided in the present application, a graph index constructed based on the original vector and a compressed vector corresponding to the original vector can be stored in a first-level storage resource, and the original vector can be stored in a second-level storage resource, while ensuring that the data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; wherein the nodes in the graph index can be associated with the compressed vector corresponding to the original vector represented by the node, and the edges in the graph index can indicate that there is a similarity relationship between the original vectors corresponding to the nodes connected by the edges. In this case, in response to the query vector, at least one target compressed vector with the highest similarity to the query vector can be obtained from the first-level storage resource based on the graph index, and after all the target compressed vectors are obtained, at least one target original vector corresponding to the at least one target compressed vector can be obtained from the second-level storage resource, and subsequently the similarity between the query vector and the at least one target original vector can be calculated respectively, and the at least one target original vector can be sorted based on the calculated similarity, so as to determine the vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0046] By adopting the above method, the nearest neighbor search process based on the graph index can be decoupled from the data access operation for the second-level storage resource such as shared storage for storing the original vector. It is no longer necessary to perform synchronous data access operations for the second-level storage resource during the search process. Instead, after obtaining the compressed vector with the highest similarity to the query vector from the first-level storage resource such as memory for storing the compressed vector corresponding to the original vector and the graph index built based on the original vector through the search process, the original vector corresponding to the compressed vector is obtained from the second-level storage resource. Therefore, the delay of vector retrieval can be reduced, so that this vector retrieval solution can be better adapted to shared storage.
[0047] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the architecture of a database system shown in an exemplary embodiment of the present application.
[0048] A database system is a software system used to store, manage and retrieve data. Figure 1 As shown, the database system may at least include a database, a database management system (DBMS) and a database application.
[0049] A database is a collection of actual stored data. The data is organized in a database according to a specific pattern or model for easy retrieval and management. In the present application, the database may be a vector database. In a vector database, data is usually represented as multidimensional vectors, which exist in a multidimensional space, and each dimension may represent a feature.
[0050] A database management system is a software system that is the core component for controlling and managing a database. It can be used to define, create, maintain, and operate a database. A database management system provides a series of tools and interfaces to manage a database, such as interfaces for creating, reading, updating, and deleting operations. A database management system is mainly used to control the storage, retrieval, updating, and management of data. It is also responsible for the accuracy, consistency, security, and backup and recovery of data.
[0051] A database application is a front-end application that interacts with a database to implement specific business logic and functions. It can provide a user-oriented interface, which can be a graphical user interface, command line, or other forms of interface.
[0052] In actual applications, users can initiate specific operations on the database (such as index building operations, data retrieval operations, etc.) through database applications, and the database management system can perform these operations on the database.
[0053] The above database and the above database management system can be deployed on a server. The server can be a server including an independent physical host or a server cluster consisting of multiple independent physical hosts; or the server can be a virtual server, cloud server, etc. hosted by a host cluster. The above database application can be deployed on a terminal device that has established a network connection (including various types of wired networks or wireless networks) with the server. The terminal device can be a personal computer (PC), a desktop computer, a laptop computer, a tablet computer, a smart phone, a PDA (Personal Digital Assistants), etc.
[0054] Please refer to Figure 2 , Figure 2 It is a data deployment diagram of a vector database system shown in an exemplary embodiment of the present application.
[0055] A vector database system is a database system specifically used to store, manage, and retrieve vector data. Figure 1 Similar to the database system shown, the vector database system may also include a vector database and a vector database management system. In practical applications, the vector database is a collection of actually stored vectors, and the vector database management system may maintain indexes created for the vectors in the vector database, so that the vector database management system may use the indexes to perform vector retrieval, thereby speeding up the vector retrieval.
[0056] like Figure 2 As shown, in a vector database system, a hierarchical storage method can be used to store vectors and indexes.
[0057] Specifically, on one hand, the vectors stored in the above-mentioned vector database may be referred to as original vectors, and these original vectors may be stored in the second-level storage resources, that is, the vector database may be deployed in the second-level storage resources.
[0058] On the other hand, an index can be created for the original vectors in the above-mentioned vector database, and specifically, a graph index can be constructed based on these original vectors. The graph index can be similar to the HNSW graph index. In addition, a compressed vector corresponding to the original vector in the vector database can be calculated. The graph index and all or part of these compressed vectors can be stored in the first-level storage resource, that is, the above-mentioned vector database management system can maintain the graph index and all or part of these compressed vectors in the first-level storage resource.
[0059] Among them, storage resources refer to various hardware and software systems used to store data.
[0060] For the above-mentioned graph index, the nodes in the graph index can represent original vectors and be associated with compressed vectors corresponding to the original vectors represented by the nodes. For example, the compressed vector corresponding to the original vector represented by the node can be used as a label or an attribute of the node; the edges in the graph index can indicate that there is a similarity relationship between the original vectors corresponding to the nodes connected by the edges (i.e., the original vectors represented by the nodes).
[0061] It should be noted that the original vectors represented by the nodes in the above graph index are not stored in the above first-level storage resources, but only the compressed vectors associated with the nodes are stored in the first-level storage resources to reduce the occupancy of the first-level storage resources; the original vectors represented by the nodes are stored in the above second-level storage resources.
[0062] In some embodiments, the compressed vector corresponding to an original vector may be a vector obtained by performing product quantization (PQ) on the original vector.
[0063] Product quantization is an effective technique for compressing high-dimensional vectors, mainly used for approximate nearest neighbor search. When using product quantization to compress the original vector, the high-dimensional vector space can be first divided into multiple low-dimensional subspaces (assuming that the dimension of the original vector is D, it can be divided into M subspaces, and the dimension of each subspace is D / M); for each subspace, a clustering algorithm (such as the K-means algorithm) is used to build a codebook, and each element in the codebook (which can be called a codeword) represents a cluster center of the subspace; after the original vector is divided into M subvectors, the closest codeword is found in the corresponding subspace codebook for each subvector. This process is called quantization. After quantization, each subvector will be replaced with its corresponding codeword index; finally, the quantized codeword index is saved to form the final compressed representation of the original vector, that is, the compressed vector corresponding to the original vector. Since the storage space occupied by the codeword index is usually much smaller than the storage space occupied by the floating-point value of the original vector, this achieves data compression for the original vector.
[0064] Correspondingly, when performing an approximate nearest neighbor search, the query vector is also segmented, quantized, and encoded. Then, a similarity measure based on the codeword index is searched on the codebook index to find possible approximate nearest neighbors.
[0065] That is, when performing a nearest neighbor search based on the above-mentioned graph index, a similarity measurement is performed on the above-mentioned query vector (specifically, the compressed vector corresponding to the query vector) and the compressed vectors associated with the nodes in the graph index, so as to determine at least one compressed vector that is most similar to the query vector.
[0066] It should be noted that the data access performance of the first-level storage resources may be better than that of the second-level storage resources, that is, the delay in accessing data stored in the first-level storage resources may be lower than the delay in accessing data stored in the second-level storage resources.
[0067] In some embodiments, since the processor can directly access the memory but needs to access the disk through I / O operations, the latency of accessing data stored in the memory is lower than the latency of accessing data stored in the disk. Therefore, the first-level storage resource can be the memory, and the second-level storage resource can be the storage resource of the shared storage service. The shared storage service refers to a service that provides shared storage, and the storage resource of the shared storage service is the shared storage itself.
[0068] In practical applications, the first-level storage resource may specifically be the memory of the server used to deploy the vector database management system. The second-level storage resource may be a local disk; that is, the vector database and the vector database management system may be deployed on the same server, in which case the disk of the server may be used as the second-level storage resource. Alternatively, the second-level storage resource may be a remote disk; that is, the vector database may be deployed on a disk connected to the server used to deploy the vector database management system via a network, in which case the disk may be used as the second-level storage resource. Remote disks may be implemented using a variety of technologies, such as network attached storage, storage area network, or cloud storage. Remote disks are typically accessed via network protocols such as TCP / IP, iSCSI, NFS, CIFS, and the like. Alternatively, the second-level storage resource may be a disk used by a distributed file system, object storage, or the like.
[0069] In some embodiments, when constructing the above-mentioned graph index based on the above-mentioned original vectors, specifically, on the one hand, each node in the graph index can be constructed based on each original vector, that is, each original vector is represented by each node in the graph index, and on the other hand, the similarity between each two original vectors can be calculated, and the edges in the graph index can be constructed based on the calculated similarity. In the graph index, the distance between two nodes can be inversely proportional to the similarity between the original vectors represented by the two nodes; if two nodes are connected by an edge, it means that the similarity between the original vectors represented by the two nodes is high, that is, the original vectors represented by the two nodes are neighbors, and this edge can indicate that there is a similarity relationship (or a neighbor relationship) between the original vectors represented by the two nodes.
[0070] Please refer to Figure 3 , Figure 3 It is a schematic diagram of a graph index shown in an exemplary embodiment of the present application.
[0071] like Figure 3 As shown, in the above graph index, a node can represent an original vector, and this node is associated with a compressed vector corresponding to the original vector. For example, this compressed vector can be used as a label or an attribute of this node; an edge can represent that there is a similarity relationship between the original vectors corresponding to the two nodes connected by this edge.
[0072] For Figure 3 As for the graph index shown, the first-level storage resource stores the graph index and the compressed vectors corresponding to all the original vectors.
[0073] In some embodiments, when constructing the graph index based on the original vectors, the original vectors may be first divided into at least one vector group, and an aggregate vector corresponding to each vector group is determined.
[0074] It should be noted that, for a vector group, the original vectors included in the vector group are similar to each other; that is, the original vectors similar to each other in the above original vectors can be divided into a vector group. The aggregation vector corresponding to the vector group can be a representative original vector in the original vectors included in the vector group.
[0075] Then, on the one hand, each node in the graph index can be constructed based on the aggregate vectors corresponding to each vector group, that is, each node in the graph index represents each aggregate vector, and the node used to represent each aggregate vector is associated with the vector group corresponding to each aggregate vector. Specifically, each original vector contained in each vector group can be converted into a compressed vector corresponding to each original vector to obtain a vector group containing a compressed vector, and the node used to represent each aggregate vector is associated with the compressed vector contained in the vector group corresponding to each aggregate vector. On the other hand, the similarity between each two aggregate vectors in these aggregate vectors can be calculated, and the edge in the graph index can be constructed based on the calculated similarity. In the graph index, the distance between two nodes can be inversely proportional to the similarity between the aggregate vectors represented by the two nodes; if two nodes are connected by an edge, it means that the similarity between the aggregate vectors represented by the two nodes is high, that is, the aggregate vectors represented by the two nodes are neighbors, and this edge can indicate that there is a similarity relationship (or a neighbor relationship) between the aggregate vectors represented by the two nodes.
[0076] In some embodiments, when the original vectors are divided into at least one vector group and the aggregate vectors corresponding to each vector group are determined, the vector groups may be divided first and then the aggregate vectors may be determined, that is, the original vectors are first divided into at least one vector group, and then after all the vector groups are divided, an original vector is selected from the original vectors included in each vector group as the aggregate vector corresponding to each vector group. Taking any divided vector group as an example, an original vector may be selected from the original vectors included in this vector group according to a certain rule or randomly as the aggregate vector corresponding to this vector group.
[0077] Alternatively, the aggregate vectors may be determined first and then the vector groups may be divided, that is, at least one original vector may be selected from the original vectors according to a certain rule or randomly as an aggregate vector corresponding to at least one vector group, wherein the selected aggregate vectors correspond to the vector groups one-to-one, and then original vectors similar to each aggregate vector may be added to the vector group corresponding to each aggregate vector. Taking any selected aggregate vector as an example, original vectors similar to the aggregate vector may be added to the vector group corresponding to the aggregate vector from the remaining original vectors except the aggregate vectors.
[0078] In some embodiments, for a divided vector group, the original vectors included in this vector group are similar to each other, which means that the similarity between the original vectors included in this vector group is not less than a preset first threshold, that is, the original vectors included in this vector group can be considered to be the nearest neighbors of each other; or, the number of repetitions of the original vectors whose similarity with each original vector included in this vector group is not less than a preset second threshold is not less than a preset third threshold, that is, the number of repetitions of the nearest neighbors of each original vector included in this vector group is not less than the third threshold.
[0079] Please refer to Figure 4 , Figure 4 It is a schematic diagram of another graph index shown in an exemplary embodiment of the present application.
[0080] like Figure 4 As shown, in the above graph index, a node can represent an aggregate vector, and this node is associated with a compressed vector contained in the vector group corresponding to this aggregate vector (converted from the original vector contained in the vector group). For example, the compressed vector contained in the vector group corresponding to this aggregate vector can be used as a label or an attribute of this node; an edge can indicate that there is a similarity relationship between the aggregate vectors corresponding to the two nodes connected to this edge.
[0081] For Figure 4As for the graph index shown, the first-level storage resource stores the graph index and compressed vectors corresponding to all aggregated vectors (ie, a part of all original vectors).
[0082] By aggregating the above-mentioned original vectors and constructing the above-mentioned graph index based on the aggregated vectors, the scale of the graph index can be reduced, which not only saves the storage overhead of the graph index, but also reduces the computing resource consumption when performing nearest neighbor search based on the graph index, and increases the scale of data that can be processed by a single machine (for example, the number of vectors contained in the vector database).
[0083] Please refer to Figure 5 , Figure 5 It is a flowchart of a vector retrieval method shown in an exemplary embodiment of the present application.
[0084] like Figure 5 The embodiment shown can be applied to Figure 2 The vector database system shown adopts a hierarchical storage method to store vectors and indexes.
[0085] It should be noted that in product quantization, a high-dimensional vector is decomposed into several low-dimensional vectors, each of which is quantized and represented by an index. Therefore, a complete vector can be represented by a combination of these indices. This quantization method can greatly reduce storage and computing costs, but it also introduces errors. Since some information is lost during the quantization process, the similarity calculated based on the vector after product quantization may be different from the similarity calculated based on the original vector.
[0086] Similarly, the similarity calculated based on the compressed vector obtained by other methods may also have a certain gap with the similarity calculated based on the original vector.
[0087] After obtaining at least one compressed vector that is most similar (may be relatively most similar) to the query vector through nearest neighbor search, it is usually necessary to rearrange at least one original vector corresponding to the at least one compressed vector according to the similarity with the query vector to determine the final vector retrieval result corresponding to the query vector.
[0088] like Figure 5 As shown, the above vector retrieval method may include the following steps:
[0089] Step 502: In response to a query vector corresponding to the vector database, obtain at least one target compressed vector having the highest similarity to the query vector from the first-level storage resources based on the graph index, and after obtaining the at least one target compressed vector, obtain at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resources.
[0090] In this embodiment, when a query vector for vector retrieval in the above-mentioned vector database is obtained, a response can be made to it. Specifically, a nearest neighbor search can be performed from the above-mentioned first-level storage resources based on the above-mentioned graph index to obtain at least one compressed vector (which can be referred to as a target compressed vector) with the highest similarity to the query vector, and after the nearest neighbor search is completed and all target compressed vectors are obtained, at least one original vector (which can be referred to as a target original vector) corresponding to at least one target compressed vector is obtained from the above-mentioned second-level storage resources. Among them, the original vector corresponding to a target compressed vector can be the original vector directly corresponding to the target compressed vector, or it can be the original vector included in the vector group to which the aggregated vector corresponding to the target compressed vector belongs. In this way, it is no longer necessary to perform synchronous data access operations on the second-level storage resources during the search process.
[0091] It should be noted that in the K-nearest neighbor search, the K value usually refers to the number of nearest neighbors to be found. In the present application, the K value can be pre-set according to actual needs. In this case, the K (which may also be greater than K, for example, an integer multiple of K) compressed vectors with the highest similarity to the query vector can be obtained from the first-level storage resources based on the graph index as the at least one target compressed vector.
[0092] Specifically, if a node in the graph index represents an original vector, the K compressed vectors with the highest similarity to the query vector can be obtained from the first-level storage resource based on the graph index as the at least one target compressed vector. If a node in the graph index represents an aggregated vector, a certain number of compressed vectors with the highest similarity to the query vector can be obtained from the first-level storage resource based on the graph index, so that the total number of original vectors contained in the vector group to which the aggregated vectors corresponding to these compressed vectors belong is not less than the K value, and the compressed vectors contained in the vector groups corresponding to these aggregated vectors can be used as the at least one target compressed vector.
[0093] In some embodiments, after acquiring the at least one target compression vector, when acquiring at least one target original vector corresponding to the at least one target compression vector from the second-level storage resource, specifically, after acquiring all the target compression vectors, a data access request corresponding to the second-level storage resource may be initiated (assuming that the second-level storage resource is a distributed file system, the data access request may be an I / O request for the distributed file system), and at this time, the data access request may be used to acquire at least one target original vector corresponding to the at least one target compression vector from the second-level storage resource; subsequently, a data access result corresponding to the data access request may be acquired, so that at least one target original vector corresponding to the at least one target compression vector may be acquired from the data access result.
[0094] That is, after completing the nearest neighbor search based on the graph index from the first-level storage resource to obtain all the target compressed vectors with the highest similarity to the query vector, the target original vectors corresponding to these target compressed vectors can be obtained from the second-level storage resource through a data access to the second-level storage resource.
[0095] In some embodiments, after acquiring the above-mentioned at least one target compression vector, when acquiring at least one target original vector corresponding to the at least one target compression vector from the above-mentioned second-level storage resource, specifically, after each target compression vector is acquired, an asynchronous data access request corresponding to the second-level storage resource may be initiated, wherein asynchronous means that after initiating the data access request, there is no need to block and wait for the request to return. At this time, the data access request can be used to acquire the target original vector corresponding to this target compression vector from the second-level storage resource; and after acquiring all the target compression vectors, since each data access request initiated corresponding to the second-level storage resource can be used to acquire the target original vector corresponding to a target compression vector from the second-level storage resource, in this case, all data access results corresponding to all the initiated data access requests can be acquired, so that at least one target original vector corresponding to the at least one target compression vector can be acquired from all the data access results.
[0096] That is, after each target compression vector is obtained from the first-level storage resource through a nearest neighbor search based on the graph index, an asynchronous data access request can be initiated for the second-level storage resource. However, before the nearest neighbor search is completed, there is no need to block and wait for the data access result corresponding to the data access request. Instead, after all the target compression vectors with the highest similarity to the query vector are obtained from the first-level storage resource through a nearest neighbor search based on the graph index, the target original vectors corresponding to each of these target compression vectors can be obtained from the second-level storage resource through the data access results corresponding to all the initiated data access requests.
[0097] Step 504: Calculate the similarity between the query vector and the at least one target original vector respectively, and sort the at least one target original vector based on the calculated similarity, so as to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0098] In this embodiment, for the at least one target original vector corresponding to the at least one target compressed vector obtained from the second-level storage resource, the similarities between the query vector and these target original vectors can be calculated respectively, and these target original vectors can be sorted based on the calculated similarities to determine the vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0099] In some embodiments, when sorting the at least one target original vector based on the calculated similarity to determine the vector retrieval result corresponding to the query vector based on the at least one sorted target original vector, the target original vectors can be sorted in descending order according to the calculated similarity, and based on the at least one sorted target original vector, a preset number (which can be the K value) of target original vectors arranged in front can be determined as the vector retrieval result corresponding to the query vector.
[0100] In the vector retrieval scheme provided in the present application, a graph index constructed based on the original vector and a compressed vector corresponding to the original vector can be stored in a first-level storage resource, and the original vector can be stored in a second-level storage resource, while ensuring that the data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; wherein the nodes in the graph index can be associated with the compressed vector corresponding to the original vector represented by the node, and the edges in the graph index can indicate that there is a similarity relationship between the original vectors corresponding to the nodes connected by the edges. In this case, in response to the query vector, at least one target compressed vector with the highest similarity to the query vector can be obtained from the first-level storage resource based on the graph index, and after all the target compressed vectors are obtained, at least one target original vector corresponding to the at least one target compressed vector can be obtained from the second-level storage resource, and subsequently the similarity between the query vector and the at least one target original vector can be calculated respectively, and the at least one target original vector can be sorted based on the calculated similarity, so as to determine the vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0101] By adopting the above method, the nearest neighbor search process based on the graph index can be decoupled from the data access operation for the second-level storage resource such as shared storage for storing the original vector. It is no longer necessary to perform synchronous data access operations for the second-level storage resource during the search process. Instead, after obtaining the compressed vector with the highest similarity to the query vector from the first-level storage resource such as memory for storing the compressed vector corresponding to the original vector and the graph index built based on the original vector through the search process, the original vector corresponding to the compressed vector is obtained from the second-level storage resource. Therefore, the delay of vector retrieval can be reduced, so that this vector retrieval solution can be better adapted to shared storage.
[0102] Corresponding to the above-mentioned embodiment of the vector retrieval method, the present application also provides an embodiment of a vector retrieval device.
[0103] Please refer to Figure 6 , Figure 6It is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608 and a non-volatile memory 610, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.
[0104] Please refer to Figure 7 , Figure 7 It is a block diagram of a vector retrieval device shown in an exemplary embodiment of the present application.
[0105] The above vector retrieval device can be applied to Figure 6 The device shown is used to implement the technical solution of the present application. In which, a graph index constructed based on the original vector and a compressed vector corresponding to the original vector are stored in a first-level storage resource; the node in the graph index is associated with a compressed vector corresponding to the original vector represented by the node, and the edge in the graph index represents a similar relationship between the original vectors corresponding to the nodes connected by the edge; the original vector is stored in a second-level storage resource; the data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; the device includes:
[0106] The acquisition module 702 acquires, in response to the query vector, at least one target compressed vector having the highest similarity to the query vector from the first-level storage resource based on the graph index, and acquires, after acquiring the at least one target compressed vector, at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource;
[0107] The determination module 704 calculates the similarity between the query vector and the at least one target original vector respectively, and sorts the at least one target original vector based on the calculated similarity, so as to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
[0108] In some embodiments, a graph index is constructed based on the original vector by the following steps:
[0109] Divide the original vector into at least one vector group, and determine an aggregate vector corresponding to each vector group; wherein the original vectors included in each vector group are similar to each other; and the aggregate vector is an original vector in the original vectors included in the vector group;
[0110] Building nodes in the graph index based on the aggregate vectors corresponding to each vector group, and associating the nodes representing each aggregate vector with the compression vectors corresponding to each aggregate vector;
[0111] The similarity between every two aggregated vectors in the aggregated vectors is calculated, and an edge in the graph index is constructed based on the calculated similarity.
[0112] In some embodiments, dividing the original vector into at least one vector group and determining an aggregate vector corresponding to each vector group includes:
[0113] The original vectors are divided into at least one vector group, and after the at least one vector group is divided, an original vector is selected from the original vectors included in each vector group as an aggregate vector corresponding to each vector group; or at least one original vector is selected from the original vectors as an aggregate vector corresponding to at least one vector group, and an original vector similar to each aggregate vector is added to the vector group corresponding to each aggregate vector.
[0114] In some embodiments, the similarity between the original vectors included in the vector group is not less than a preset first threshold; or, the number of repetitions of the original vectors whose similarity with each original vector included in the vector group is not less than a preset second threshold is not less than a preset third threshold.
[0115] In some embodiments, acquiring at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource includes:
[0116] At least one target aggregate vector corresponding to the at least one target compressed vector is obtained from the second-level storage resource, and original vectors included in the vector group to which each target aggregate vector belongs are further obtained as at least one target original vector corresponding to the at least one target compressed vector.
[0117] In some embodiments, after acquiring the at least one target compressed vector, acquiring at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource includes:
[0118] After acquiring the at least one target compression vector, initiating a data access request corresponding to the second-level storage resource; wherein the data access request is used to acquire at least one target original vector corresponding to the at least one target compression vector from the second-level storage resource;
[0119] A data access result corresponding to the data access request is obtained; wherein the data access result includes the at least one target original vector.
[0120] In some embodiments, after acquiring the at least one target compressed vector, acquiring at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource includes:
[0121] Initiating an asynchronous data access request corresponding to the second-level storage resource each time a target compressed vector is obtained; wherein the data access request is used to obtain a target original vector corresponding to the target compressed vector from the second-level storage resource;
[0122] After acquiring the at least one target compression vector, a data access result corresponding to the initiated data access request is acquired; wherein the data access result includes at least one target original vector corresponding to the at least one target compression vector.
[0123] In some embodiments, the step of sorting the at least one target original vector based on the calculated similarity to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector includes:
[0124] The at least one target original vector is sorted in descending order according to the calculated similarity, and based on the sorted at least one target original vector, a preset number of target original vectors arranged in front are determined as vector retrieval results corresponding to the query vector.
[0125] In some embodiments, the compressed vector is a vector obtained by performing product quantization on the original vector.
[0126] In some embodiments, the first-level storage resources include memory; and the second-level storage resources include storage resources of a shared storage service.
[0127] For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of the present application.
[0128] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.
[0129] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0130] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0131] Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0132] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0133] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0134] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" refers to and includes any or all possible combinations of one or more associated listed items.
[0135] The description of the terms "one embodiment", "some embodiments", "example", "specific example" or "an implementation method" etc. used in one or more embodiments of the present application means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic description of these terms is not necessarily for the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without contradicting each other.
[0136] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".
[0137] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.
[0138] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
Claims
1. A vector search method; wherein: A graph index constructed based on the original vector and a compressed vector corresponding to the original vector are stored in the first-level storage resource; nodes in the graph index are associated with compressed vectors corresponding to the original vectors represented by the nodes, and edges in the graph index represent similar relationships between the original vectors corresponding to the nodes connected by the edges; The original vector is stored in a second-level storage resource; The data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; the method comprises: In response to a query vector, acquiring at least one target compressed vector having the highest similarity to the query vector from the first-level storage resource based on the graph index, and after acquiring the at least one target compressed vector, acquiring at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource; The similarities between the query vector and the at least one target original vector are respectively calculated, and the at least one target original vector is sorted based on the calculated similarities, so as to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
2. The method according to claim 1, constructing a graph index based on the original vector by the following steps: The original vector is divided into at least one vector group, and an aggregate vector corresponding to each vector group is determined; wherein, The original vectors included in each vector group are similar to each other; the aggregate vector is an original vector in the original vectors included in the vector group; Building nodes in the graph index based on the aggregate vectors corresponding to each vector group, and associating the nodes representing each aggregate vector with the compression vectors corresponding to each aggregate vector; The similarity between every two aggregated vectors in the aggregated vectors is calculated, and an edge in the graph index is constructed based on the calculated similarity.
3. The method according to claim 2, wherein dividing the original vector into at least one vector group and determining an aggregate vector corresponding to each vector group comprises: The original vectors are divided into at least one vector group, and after the at least one vector group is divided, an original vector is selected from the original vectors included in each vector group as an aggregate vector corresponding to each vector group; or at least one original vector is selected from the original vectors as an aggregate vector corresponding to at least one vector group, and an original vector similar to each aggregate vector is added to the vector group corresponding to each aggregate vector.
4. According to the method of claim 2, the similarity between the original vectors contained in the vector group is not less than a preset first threshold; or, the number of repetitions of the original vectors whose similarity with each original vector contained in the vector group is not less than a preset second threshold is not less than a preset third threshold.
5. The method according to claim 2, wherein obtaining at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource comprises: At least one target aggregate vector corresponding to the at least one target compressed vector is obtained from the second-level storage resource, and original vectors included in the vector group to which each target aggregate vector belongs are further obtained as at least one target original vector corresponding to the at least one target compressed vector.
6. The method according to claim 1, wherein after obtaining the at least one target compressed vector, obtaining at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource comprises: After acquiring the at least one target compression vector, initiating a data access request corresponding to the second-level storage resource; wherein the data access request is used to acquire at least one target original vector corresponding to the at least one target compression vector from the second-level storage resource; A data access result corresponding to the data access request is obtained; wherein the data access result includes the at least one target original vector.
7. The method according to claim 1, wherein after obtaining the at least one target compressed vector, obtaining at least one target original vector corresponding to the at least one target compressed vector from the second-level storage resource comprises: Initiating an asynchronous data access request corresponding to the second-level storage resource each time a target compressed vector is obtained; wherein the data access request is used to obtain a target original vector corresponding to the target compressed vector from the second-level storage resource; After acquiring the at least one target compression vector, a data access result corresponding to the initiated data access request is acquired; wherein the data access result includes at least one target original vector corresponding to the at least one target compression vector.
8. The method according to claim 1, wherein the step of sorting the at least one target original vector based on the calculated similarity to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector comprises: The at least one target original vector is sorted in descending order according to the calculated similarity, and based on the sorted at least one target original vector, a preset number of target original vectors arranged in front are determined as vector retrieval results corresponding to the query vector.
9. According to the method of claim 1, the compressed vector is a vector obtained by performing product quantization on the original vector.
10. According to the method of claim 1, the first-level storage resource includes memory; the second-level storage resource includes storage resources of a shared storage service.
11. A vector search device; wherein: A graph index constructed based on the original vector and a compressed vector corresponding to the original vector are stored in the first-level storage resource; nodes in the graph index are associated with compressed vectors corresponding to the original vectors represented by the nodes, and edges in the graph index represent similar relationships between the original vectors corresponding to the nodes connected by the edges; The original vector is stored in a second-level storage resource; The data access performance of the first-level storage resource is better than the data access performance of the second-level storage resource; the device comprises: an acquisition module, in response to a query vector, acquiring, from the first-level storage resource and based on the graph index, at least one target compressed vector having the highest similarity to the query vector, and acquiring, after acquiring the at least one target compressed vector, from the second-level storage resource, at least one target original vector corresponding to the at least one target compressed vector; A determination module calculates the similarity between the query vector and the at least one target original vector respectively, and sorts the at least one target original vector based on the calculated similarity, so as to determine a vector retrieval result corresponding to the query vector based on the sorted at least one target original vector.
12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.
13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Vector range retrieval method and device, equipment, medium and program product
CN115129949A
Vector coding learning method and device for neighbor graph vector retrieval
CN115408571A
Cited By
Query method, processor, processing system, storage medium and program product
CN120067177A
Vector set-based data processing method and device, medium and product
CN120256435A
Query processing method and device in graph database
CN120523982A
Graph database-oriented vector embedding management and query method
CN120973981A
