Vector retrieval method and device

Abstract

The vector retrieval method comprises the following steps: acquiring a query object; n vector identifiers corresponding to a query object are obtained by querying a first index, the first index is stored in a memory, N is a positive integer, the N vector identifiers comprise a plurality of first vector identifiers, and the first vector identifiers are vector identifiers which are not determined to be recalled; second vector identifiers corresponding to the multiple first vector identifiers are determined by querying index entries corresponding to the multiple first vector identifiers in a second index, the second vector identifiers are determined to be recalled vector identifiers, the second index is stored in a persistent storage medium, and the compression ratio of vectors in the first index entries is larger than that of vectors in the second index; and obtaining a query result of the query object based on the second vector identifier. According to the vector retrieval method provided by the invention, the retrieval time delay is effectively reduced for the scene of mixed storage of the memory and the persistent storage medium.

Description

Vector search method and device Technical Field

[0001] The present application relates to the field of retrieval technology, and in particular to a vector retrieval method and device. Background Art

[0002] In the field of vector retrieval, the industry's common vector index storage medium is memory. However, as data volumes grow from tens of millions to hundreds of millions, and even billions, memory costs continue to rise. Hybrid storage, combining hard drives, such as solid-state drives (SSDs), and memory has become an industry trend for solving large-capacity data problems. However, this comes with a significant performance degradation. Utilizing SSDs to achieve performance comparable to pure memory storage presents higher technical requirements.

[0003] Summary of the Invention

[0004] The embodiments of the present application provide a vector search method and apparatus, which, for scenarios where memory and persistent storage media are mixed, ensure search accuracy and reduce search latency, thereby improving search performance.

[0005] In a first aspect, the present application provides a vector retrieval method, including: obtaining a query object; obtaining N vector identifiers corresponding to the query object by querying a first index, the first index being stored in a memory, N being a positive integer, the N vector identifiers including multiple first vector identifiers, the first vector identifier being an uncertainly recalled vector identifier; determining second vector identifiers corresponding to the multiple first vector identifiers by querying index items corresponding to the multiple first vector identifiers in a second index, the second vector identifier being a certainty of being recalled vector identifier, the second index being stored in a persistent storage medium, the compression ratio of the vector in the first index item being greater than that of the vector in the second index; and obtaining a query result for the query object based on the second vector identifier.

[0006] By storing the first index data with high compression ratio and low storage overhead in memory, and storing the second index data with low or no compression ratio but high storage overhead in a persistent storage medium, the problem of high memory overhead in vector retrieval is solved. At the same time, the results retrieved from the first index are classified and divided, and only the vectors with uncertain categories are identified in the persistent storage medium for more accurate retrieval, reducing the amount of data accessed to the persistent storage medium and the number of accesses, effectively reducing the retrieval latency while ensuring retrieval accuracy.

[0007] Exemplarily, feature extraction is performed on the query object to obtain a query vector; based on a similarity measurement result between the query vector and a first vector in a first index, N vector identifiers are determined from the first index, the first index includes multiple vector identifiers and a first vector corresponding to each vector identifier, the first index is stored in a memory, and N is a positive integer; the categories of the N vector identifiers are identified, and the categories of the vector identifiers are valid categories, uncertain categories, or invalid categories; based on a similarity measurement result between the query vector and a second vector corresponding to a vector identifier of an uncertain category in a second index, the vector identifier of the uncertain category is determined to be a valid category or an invalid category, the second index includes multiple vector identifiers and the second vector corresponding to each vector identifier, the second index is stored in a persistent storage medium, and the compression ratio of the first vector is greater than the compression ratio of the second vector; the vector identifiers of the valid category are recalled to obtain K target vector identifiers, and K is a positive integer less than N.

[0008] In one possible implementation, identifying the categories of the N vector identifiers includes: determining that the vector identifier is a valid category, an uncertain category, or an invalid category based on a quantization loss value corresponding to the vector identifier; wherein the quantization loss value corresponding to the vector identifier is determined based on the loss between the first vector corresponding to the vector identifier and the vector before quantization of the first vector corresponding to the vector identifier.

[0009] In this possible implementation, the quantization loss value can measure the loss before vector quantization (i.e., the original vector) and after vector quantization. The larger the quantization loss value, the greater the difference between the quantized vector and the original vector, and the less accurate the retrieval result matched by the similarity measurement result. Therefore, the category of the vector identifier can be identified by the quantization loss value between the vector before quantization and the vector after quantization. For example, the vector identifier of the partial vector with the largest quantization loss value is classified as an invalid category, the vector identifier of the partial vector with the second largest quantization loss value is classified as an uncertain category, and the vector identifier of the partial vector with the smallest quantization loss value is classified as a valid category.

[0010] In another possible implementation, based on the quantization loss value corresponding to the vector identifier, the vector identifier is determined to be a valid category, an uncertain category, or an invalid category, wherein the quantization loss value corresponding to the vector identifier of the valid category is greater than the quantization loss value corresponding to the vector identifier of the uncertain category, and the quantization loss value corresponding to the vector identifier of the uncertain category is greater than the vector identifier of the invalid category.

[0011] Exemplarily, the N vector identifiers are sorted from small to large based on the loss value, the categories of the first M vector identifiers are determined as valid categories, the categories of the last J vector identifiers are determined as invalid categories, and the categories of the remaining vector identifiers are determined as uncertain categories, where M and J are both positive integers less than N.

[0012] In this possible implementation, the search results retrieved from the first index data are identified and divided into three categories: a valid category, an uncertain category, and an invalid category. The search results of the invalid category are directly discarded, reducing the number of precision searches required from the second index in the persistent storage medium, further reducing the amount of data accessed to the persistent storage medium, and improving the retrieval latency.

[0013] In another possible implementation, the second index includes multiple clusters, each cluster includes multiple index items, and each index item includes a vector identifier and its corresponding second vector; in vector retrieval, each index item in the cluster has the same probability of being accessed.

[0014] In this possible implementation, by clustering index items that may be accessed simultaneously, the amount of data scanned during disk access is reduced, thereby reducing retrieval latency.

[0015] In another possible implementation, the index item includes a vector and an index pointer corresponding to the vector, and the index pointer is related to the identifier of the cluster and the identifier of the vector in the cluster. In one example, multiple index items in the cluster are stored in the same storage area in the persistent storage medium.

[0016] In another possible implementation, the vector search method provided by an embodiment of the present application further includes aggregating access requests to a persistent storage medium, grouping access requests for index items in the same cluster into a single access request batch. Aggregating access requests improves disk read throughput and further reduces search latency.

[0017] In another possible implementation, a vector retrieval method provided by an embodiment of the present application further includes: reading a set of candidate index items in the second index, the index items in the candidate index item set including a candidate vector identifier and its corresponding second vector, and selecting the category of the vector identifier as an uncertain category.

[0018] In another possible implementation, reading the second vector corresponding to the set of uncertain vector identifiers in the second index includes reading a set of candidate index items from a first-level cache in memory; wherein the first-level cache stores data in the second index that is accessed more frequently than a preset frequency. In other words, frequently accessed, high-frequency data in the second index is stored in the cache. If the cache hits the data, there is no need to access the persistent storage medium, further reducing retrieval latency.

[0019] In another possible implementation, the first-level cache space also stores the first index item in the second index, the probability that the category of the vector identifier in the first index item is an uncertain category is greater than a preset probability, and the first index item includes the vector identifier and its corresponding second vector, thereby increasing the read hit rate of the first-level cache.

[0020] In another possible implementation, reading the candidate index item set in the second index includes: reading the candidate index item set in the second index from a first-level cache space in the memory; when the candidate index item set in the second index is not read from the first-level cache space, reading the candidate index item set in the second index from a second-level cache space in the memory; when the candidate index item set in the second index is not read from the second-level cache space, reading the candidate index item set in the second index from a persistent storage medium; wherein the second-level cache space stores data read from the persistent storage medium by a previous vector retrieval.

[0021] In this possible implementation, the second index data is preloaded through a multi-level cache logic mechanism. While caching high-frequency data, low-quality data and data read in the last retrieval are also cached, reducing the number of accesses to the persistent storage medium and the retrieval delay, ensuring controllable retrieval delay for both high-frequency and unpopular retrieval scenarios.

[0022] In another possible implementation, the vector search method provided by an embodiment of the present application further includes: updating the data in the secondary cache space with the data read from the persistent storage medium. For example, the data read from the persistent storage medium during the current search process replaces the data in the secondary cache space.

[0023] Optionally, the persistent storage medium is a solid-state drive.

[0024] In a second aspect, the present application provides a vector retrieval device, comprising: an acquisition module, a first query module, a second query module and a recall module; wherein the acquisition module is used to acquire a query object; the first query module is used to obtain N vector identifiers corresponding to the query object by querying a first index, the first index being stored in a memory, N being a positive integer, the N vector identifiers including multiple first vector identifiers, the first vector identifier being an uncertainly recalled vector identifier; the second query module is used to determine the second vector identifiers corresponding to the multiple first vector identifiers by querying index items corresponding to the multiple first vector identifiers in the second index, the second vector identifier being a determinedly recalled vector identifier, the second index being stored in a persistent storage medium, the compression ratio of the vector in the first index item being greater than that of the vector in the second index; the recall module is used to obtain a query result of the query object based on the second vector identifier.

[0025] In a possible implementation, the second index includes multiple clusters, each cluster includes multiple index items; in vector retrieval, each index item in a cluster has the same probability of being accessed.

[0026] In a possible implementation, the index item includes a vector and an index pointer corresponding to the vector, and the index pointer is related to the identifier of the cluster and the identifier of the vector in the cluster.

[0027] In another possible implementation, multiple index items in the cluster are stored in the same storage area in the persistent storage medium.

[0028] In another possible implementation, an aggregation module is further included, which is used to aggregate access requests to the persistent storage medium and aggregate access requests for index items in the same cluster into the same access request batch.

[0029] In another possible implementation, the N vector identifiers also include a third vector identifier that is determined to be recalled; and the query result of the query object is related to the third vector identifier.

[0030] In another possible implementation, an identification module is further included, which identifies the N vector identifiers to obtain the first vector identifier and a third vector identifier, wherein the third vector identifier is the vector identifier determined to be recalled.

[0031] In another possible implementation, the identification module is specifically used to identify the first vector identifier and the third vector identifier based on the quantization loss value corresponding to each vector identifier in N vector identifiers, and the quantization loss value of the third vector identifier in the first index is less than the quantization loss value of the first vector identifier in the first index; the quantization loss value corresponding to each vector identifier is determined based on the loss between the vector in the first index corresponding to each vector identifier and the original vector corresponding to each vector identifier.

[0032] In another possible implementation, the method further includes: a reading module configured to read a candidate index item set in the second index, where the candidate index item set includes index items in the second index corresponding to the multiple first vector identifiers.

[0033] In another possible implementation, the reading module is specifically configured to read the candidate index item set from a first-level cache space in the memory; wherein the first-level cache space stores data in the second index that is accessed more frequently than a preset frequency.

[0034] In another possible implementation, the first index item in the second index is further stored in the first-level cache space, where the first index item is an index item corresponding to a vector identifier whose probability of being determined to be the first vector identifier is greater than a preset probability.

[0035] In another possible implementation, the reading module is specifically used to read the candidate index item set in the second index from the first-level cache space in the memory; when the candidate index item set in the second index is not read from the first-level cache space, read the candidate index item set in the second index from the second-level cache space in the memory; when the candidate index item set in the second index is not read from the second-level cache space, read the candidate index item set in the second index from the persistent storage medium; wherein the second-level cache space stores data read from the persistent storage medium by a previous vector retrieval.

[0036] In another possible implementation, an update module is further included, which is used to update the data in the secondary cache space with the data read from the persistent storage medium.

[0037] Optionally, the persistent storage medium is a solid-state drive.

[0038] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method provided in the first aspect of the present application.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method provided in the first aspect of the present application.

[0040] In a fifth aspect, the present application provides a computer program or a computer program product, wherein the computer program or computer program product includes instructions, and when the instructions are executed, the method provided in the first aspect of the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a schematic diagram of a vector retrieval method;

[0042] FIG2 is a system architecture diagram of a retrieval system provided in an embodiment of the present application;

[0043] FIG3 is a schematic diagram of a flow chart of a vector search method provided in an embodiment of the present application;

[0044] FIG4 is a schematic diagram of storing the second index data in the form of multiple clusters on a solid state drive;

[0045] FIG5 is a schematic diagram of an implementation process of a vector search method provided in an embodiment of the present application;

[0046] FIG6 is a schematic diagram of the structure of a vector search device provided in an embodiment of the present application;

[0047] FIG7 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solution of the present application is further described in detail below through the accompanying drawings and examples.

[0049] To facilitate understanding of the vector retrieval method and apparatus provided in the embodiments of the present application, some technical terms involved in the embodiments of the present application are briefly explained below.

[0050] Vector, a set of numbers is used to represent an object. Such a set of numbers is a vector. If a vector consists of n numbers, it is an n-dimensional vector.

[0051] Vector retrieval refers to retrieving K vectors that are close to the query vector in a given vector dataset according to a certain metric (K-Nearest Neighbor, KNN). However, due to the high computational complexity of KNN, we usually only focus on the approximate nearest neighbor (ANN) problem.

[0052] In order to solve the problems existing in the background technology, the following solutions can be adopted:

[0053] The first solution is a disk + memory hybrid storage solution that uses graph-based vector retrieval methods (such as DiskANN or Zoom retrieval algorithms). It utilizes the local neighbor relationships of the neighbor graph and gradually approaches the query point through a greedy search method, achieving a high recall rate and competitive query latency.

[0054] Specifically, the original vectors or high-precision compressed encoding vectors of the data points are stored on the SSD, and most of the data structures representing the edge relationships are also stored on the SSD. The low-precision encoding of all data points and some edge information that exists as a cache are retained in the memory. As shown in Figure 1, in a standard query logic, triangle points and square points are points on the access path, and both the triangle points and the edges related to the triangle points need to access SSD information. Therefore, the IO volume is closely related to the number of these triangle points and the edges related to the triangle points. In recall scenarios with topk=100 or less, DiskANN shows good performance. However, for high recall scenarios (such as top1000+), due to the linear growth of the number of triangle points and the edges related to the triangle points, its performance deteriorates more seriously.

[0055] As can be seen, the first solution, based on graph vector retrieval, primarily addresses nearest neighbor retrieval within a small recall range (e.g., topk = 100) in high-volume scenarios. However, performance degrades significantly for high-volume recalls, such as those above the top 1000. For example, in a top 1000+ recall scenario, the number of triangle points outside the circle can be significant (at least 2000), along with the number of adjacent edge relationships. This creates redundant I / O overhead, exponentially increasing retrieval latency compared to a top 100 recall.

[0056] The second solution is to provide hybrid storage capabilities through system-level vector engines such as Vearch and Milvns. It mainly uses common I / O models, such as Mmap or RocksDB, for SSD reading. The algorithm process is basically the same as that of pure memory scenarios and is not restricted by feature index types.

[0057] However, when Vearch and Milvus use Mmap as the IO model for hybrid storage, if faced with a memory shortage scenario, due to insufficient page cache resources occupied by the kernel, the overhead of the retrieval process is mainly on eliminating and updating pages, and the original ANN retrieval performance deteriorates by dozens of times.

[0058] It can be seen that the second solution does not take into account the media characteristics of SSDs and does not involve the local similarity of vector indexes. It also has obvious disadvantages in ultra-large-scale scenarios and suffers from severe performance degradation.

[0059] In summary, the above solution has the problem that in high recall scenarios (for example, top 1000+), vector retrieval using hybrid memory and SSD storage has difficulty simultaneously meeting both retrieval accuracy and retrieval latency. To ensure high retrieval accuracy, retrieval latency will increase exponentially. Therefore, achieving high retrieval accuracy and low retrieval latency in scenarios with limited memory resources is a very challenging problem.

[0060] In order to solve this problem, an embodiment of the present application provides a vector retrieval method, which performs a preliminary search in the memory to obtain a rough recall result, classifies and identifies the rough recall result, determines the recall of valid results, truncates the invalid results, and sends the uncertain results as a candidate set to the hard disk. Then, a high-precision search is performed on the hard disk to determine the valid results in the candidate set for recall. By classifying and identifying the rough recall result, determining the recall of valid results, and truncating the invalid results, only the uncertain results need to be sent to the hard disk for retrieval, thereby reducing a large number of invalid hard disk access times. While ensuring the retrieval accuracy, the retrieval delay is effectively reduced.

[0061] Figure 2 shows the system architecture of a retrieval system. As shown in Figure 2, the retrieval system includes an encoder, a vector retrieval engine, a vector model, and a database. The retrieval system includes an offline preparation phase and an online retrieval phase.

[0062] During the offline preparation phase, the following operations are performed on samples in the database: feature data is extracted from the samples. This feature data is fed into an encoder, which outputs the vector data corresponding to the sample (for example, the 128-dimensional embedding vector in Figure 2). This vector data is then used as input for the vector model, which outputs a vector index. Once all samples in the database have completed these operations, a vector index is successfully constructed for vector retrieval.

[0063] During the online retrieval phase, a query object is received from the user and fed into an encoder. The encoder then encodes and outputs a query vector for the query object. The vector engine searches the vector index based on the query vector (for example, using an approximate nearest neighbor search (ANN)) to obtain the identifiers of the top k similar vectors. These top k vector identifiers are the retrieved recall vectors. The object corresponding to the recall vector is then searched in the database and presented to the user as a retrieval result.

[0064] It is understandable that the retrieval system provided in the embodiment of the present application is not limited to vector retrieval of a certain modality. For example, the retrieval system can be applicable to vector retrieval of image modality, vector retrieval of text modality, and vector retrieval of video modality, etc.

[0065] For example, when the retrieval system is applied to image-based retrieval, the sample in the offline preparation stage is an image, the extracted feature data is the pixel data of the target image, and the encoder is an image encoder, which is used to encode the feature pixel data into vector data; the input query object in the online retrieval stage is the query image, the database is the image database and the forward database, and the retrieval result is the topk images similar to the query image, realizing image search.

[0066] When the retrieval system is applied to text-based retrieval, the sample in the offline preparation stage is text, the extracted feature data is the keywords of the text, and the encoder is a word encoder (such as the BERT model), which is used to encode the keywords into vector data; the input query object in the online retrieval stage is the query text, the database is a text database (summary database) and a forward database (forward database), and the retrieval result is the topk text data similar to the query text.

[0067] Figure 3 is a flow chart of a vector search method provided in an embodiment of the present application. This method can be applied to the vector engine and vector model modules of the search system shown in Figure 2 to achieve high-precision and low-latency vector search. The search system can be deployed on any device, equipment, platform, or device cluster with computing capabilities. The present embodiment of the application does not specifically limit the specific computing device used to deploy the search system; appropriate computing devices can be selected for deployment as needed. As shown in Figure 3, the vector search method includes at least steps S301-S305.

[0068] In step S301, a query object is obtained.

[0069] A query object input by a user is received. The query object may be in various modalities such as text, image, voice, and video. The embodiment of the present application does not limit the modality of the query object.

[0070] In other words, the vector search method provided in the embodiments of the present application can implement unimodal queries such as "searching for text by text" or "searching for images by images," as well as multimodal queries such as "searching for images by text" or "searching for text by images." In step S302, N vector identifiers corresponding to the query object are obtained by querying a first index. The first index is stored in memory, where N is a positive integer. The N vector identifiers include multiple first vector identifiers, which are vector identifiers that are uncertainly recalled.

[0071] The query object entered by the user is subjected to feature extraction to obtain a vector representation of the query object, which is the query vector. For example, the query vector is input into a trained encoder, which outputs the query vector corresponding to the query object.

[0072] In the embodiments of the present application, there is no limitation on the type of query object. For example, the query object can be text, image, video, etc.

[0073] It is understood that when the query object is text, the encoder is an encoder that has a text feature vector extraction function, for example, the encoder can be a BERT model. When the query object is an image, the encoder is an encoder that has an image feature extraction function, for example, the encoder is an image encoder. When the query object is a video, the encoder is an encoder that has a video feature vector extraction function, for example, the encoder is a video encoder.

[0074] In the subsequent steps, a specific implementation of a vector retrieval method provided in an embodiment of the present application is introduced by taking a picture as an example of a query object.

[0075] Exemplarily, based on a similarity measurement result between the query vector and a first vector in a first index, N vector identifiers are determined from the first index, and the first index is stored in a memory.

[0076] A first index is stored in the memory, and the first index includes multiple vector identifiers (identity documents, IDs) and a first vector corresponding to each vector identifier. The vector identifier is used to uniquely identify an original vector, which refers to a vector representation of an image in an image database output by an encoder. The first vector is a vector obtained by quantizing the original vector, also known as a quantized vector. The first vector corresponding to the vector identifier is a vector obtained by quantizing the original vector corresponding to the vector identifier.

[0077] Since the storage space of the memory is limited, in order to save memory overhead, the quantized vector can optionally be quantized by using a product quantization (PQ) method to quantize the original vector to obtain a quantized vector with a high compression ratio, thereby saving memory space.

[0078] In another example, in order to further speed up the efficiency of vector retrieval, the quantized index can adopt a hybrid index structure of inverted index + hierarchical navigable small world network algorithm (hierarchical navigable small world, HNSW). For example, multiple inverted tables are constructed in the memory, each of which includes low-precision coding vectors of PQ quantization coding. The number of vectors in each inverted table can be controlled within 1000, and PQ quantization coding can adopt PQ16*8 coding mode for 32-dimensional original vectors. When performing vector retrieval in the memory, the HNSW graph is responsible for selecting a certain number of inverted tables based on the query vector, traversing the PQ coding vectors in the selected inverted tables in turn, and calculating the similarity between the query vector and each PQ coding vector, determining N vector identifiers based on the similarity measurement results, and using the selected N vector identifiers as the recall results based on the first index retrieval.

[0079] The embodiment of the present application does not limit the specific calculation method of similarity. A suitable similarity calculation method can be selected according to actual conditions, such as Euclidean distance, cosine, inner product, and Hamming distance.

[0080] To reduce the memory overhead of the first index, the first index data is subjected to PQ quantization with a high compression ratio. This inevitably sacrifices a certain degree of vector retrieval accuracy. Therefore, the recall result of the vector retrieval based on the first index can also be called a coarse recall result. The coarse recall result can be used in a subsequent step to perform a high-precision search on the second index stored in a persistent storage medium (such as a solid-state drive) to select the top k vector identifiers with higher accuracy from the coarse recall result.

[0081] Therefore, the recall number topN of the rough recall result must be greater than the recall number topK of the final vector search. In other words, the recall number topN of the rough recall result must have a certain degree of redundancy. For example, if the final recall number topK of the vector search is required to be 1000, the recall number topN based on the first index search can be determined to be 4000.

[0082] It should be noted that the N vector identifiers corresponding to the query object mean: the rough recall result obtained by querying the first index, for example, the vector identifiers corresponding to the vectors in the first index whose similarity with the query vector is greater than a threshold.

[0083] In some other examples, it also includes identifying N vector identifiers, setting a classification module, and classifying the vector identifiers in the rough recall results through the classification module into "valid", "uncertain" and "invalid" categories. The vector identifier of the valid category indicates that the accuracy meets the requirements and can be used as the vector identifier recalled in the final recall result. The vector identifier of the invalid category indicates that the accuracy does not meet the requirements. The vector identifier determined to be invalid can be discarded; the vector identifier of the uncertain category indicates that the accuracy is between meeting the requirements and not meeting the requirements. It cannot be determined as a vector identifier with no valid or invalid category. The vector identifier of the uncertain category needs to continue to perform high-precision vector retrieval from the second index stored in the persistent storage medium to confirm the vector identifier of the valid category.

[0084] Optionally, the classification module may determine the category of the vector identifier in the coarse recall result based on the quantization loss value corresponding to the vector identifier. For example, the classification module calculates the loss value between the first vector corresponding to the vector identifier in the coarse recall result and the original vector, sorts the vector identifiers in the coarse recall result in descending order based on the loss value, determines the categories of the first M vector identifiers as valid categories, determines the categories of the last J vector identifiers as invalid categories, and determines the remaining vector identifiers as uncertain categories.

[0085] The vector identifiers of valid categories can be used as part of the final topk recall results, and the vector identifiers of invalid categories are discarded. Only the vector identifiers of uncertain categories are sent to the solid-state drive as a candidate set for high-precision retrieval to determine the vector identifiers of valid categories. This reduces the amount of data access to the solid-state drive. Only the index data corresponding to the vector identifiers of uncertain categories needs to be accessed, reducing the amount of IO data and thus reducing the retrieval latency of the entire vector retrieval.

[0086] In step S303, by querying the index items corresponding to the multiple first vector identifiers in the second index, the second vector identifier corresponding to the multiple first vector identifiers is determined, the second vector identifier is the vector identifier determined to be recalled, the second index is stored in a persistent storage medium, and the compression ratio of the vector in the first index item is greater than that of the vector in the second index.

[0087] For example, based on the similarity measurement result between the query vector and the second vector corresponding to the vector identifier of the uncertain category in the second index, the vector identifier of the uncertain category is determined to be a valid category or an invalid category. The second index includes multiple vector identifiers and a second vector corresponding to each vector identifier. The second index is stored in a persistent storage medium, and the compression ratio of the first vector is greater than the compression ratio of the second vector.

[0088] The second index data stored on the solid-state drive has a low compression ratio and can provide an index structure with high precision but poor query efficiency. The second index includes a vector identifier and a second vector corresponding to each vector identifier. The vector identifier is the same as the vector identifier in the first index, which is used to uniquely identify the original vector. The second vector can be the original vector or a vector obtained through scalar quantization (SQ). The quantized vector after SQ encoding has a lower compression ratio, is closer to the original vector, and has higher retrieval accuracy, but also has a higher storage overhead, which the solid-state drive can provide.

[0089] The second index data corresponding to the candidate set is read, and based on the similarity measurement result between the query vector and the second vector corresponding to the vector identifier of the uncertain category in the second index, the vector identifier in the candidate set is determined to be a valid category or an invalid category, and the valid category is recalled.

[0090] For example, the second index data corresponding to the candidate set is read directly from the solid-state drive. The similarity between the query vector and the second vector is calculated. The similarity results are sorted in descending order. The first E vector identifiers are identified as valid categories, and the remaining vector identifiers are identified as invalid categories. The E vector identifiers and the M vector identifiers constitute the final topK recalled vector identifiers.

[0091] In another example, in order to increase IO efficiency, the second index data is clustered, and index items with roughly the same probability of being accessed are clustered into a cluster, and stored in the form of multiple clusters on the solid-state drive. For example, the index item data in the same cluster is stored in the same storage area on the solid-state drive to ensure the storage continuity of the data in the same cluster (see Figure 4).

[0092] In one example, the index item in the second index includes a vector and an index pointer corresponding to the vector. The index pointer is related to the identifier of the cluster and the identifier of the vector in the cluster. For example, the index pointer in the index item of the second index can be cluster+inner_id as shown in Figure 4. An index pointer includes the cluster number and the inner number of the vector in the cluster. The cluster+inner_id can indicate the storage location of an index item in the second index on the solid-state hard disk, which facilitates quick search of any vector in the second index.

[0093] Exemplarily, the clustering of the second index data is implemented as follows: the original vector corresponding to the image in the database is used as the query vector to perform vector retrieval, and the retrieval obtains the vector identifier of the second index stored on the solid-state hard disk that is ultimately needed to access the data; these vector identifiers are abstracted as points, and each point represented by the vector identifier is connected to each other by edges, and the weight of the edge is increased by 1; after all query vectors are queried, a weighted graph is obtained; through a clustering algorithm (such as a community discovery algorithm), the weighted graph is subjected to minimum cut partitioning or graph clustering, and a certain number of clusters are obtained according to the clustering algorithm. If there are many isolated clusters, the above steps are continued. The final layout obtains multiple clusters as shown in Figure 4. Second vectors and vector identifiers with spatial similarity are often divided into the same cluster cluster, and the data in the same cluster cluster is stored in the same storage area of ​​the solid-state hard disk and stored continuously.

[0094] To further improve I / O efficiency, access requests to solid-state drives are aggregated during vector retrieval. Access requests for index items in the same cluster are aggregated into the same access request batch, reducing I / O times, increasing disk read throughput, and further reducing retrieval latency.

[0095] In another example, as shown in Figure 5, in order to further reduce the retrieval latency, frequently accessed high-hot data can be cached. For example, the frequency of accessing index items in the second index on the solid-state drive during multiple retrieval processes is counted, and the index item data with an access frequency greater than a preset frequency is determined as high-hot data and cached in the memory.

[0096] In this way, when the candidate set is determined, the index item data of the second index corresponding to the candidate set is read from the cache first, which is likely to be read successfully. When the cache hits, there is no need to access the SSD, which greatly improves the retrieval efficiency. If the cache misses, the data is read from the SSD.

[0097] In another example, to increase the cache hit rate, low-quality data is also cached in the memory, namely, index item data in the second index corresponding to vector identifiers with a high probability of being identified as uncertain categories. For example, the probability of vector identifiers being identified as uncertain categories during historical retrieval is calculated, and the index item data in the second index corresponding to vector identifiers with a probability greater than a preset value is determined as low-quality data and stored in the cache.

[0098] In another example, a multi-level cache mechanism is set up to store the above-mentioned high-heat data and low-quality data in the first-level cache, and cache the data read from the solid-state drive during the last retrieval process in the second-level cache.

[0099] Once the candidate set is determined, data is first read from the primary cache. If a hit is not found, data is read from the secondary cache. If a hit is still not found, data is finally read from the solid-state drive, further increasing the cache hit rate.

[0100] Since the cache not only stores frequently accessed hot data, it still has a high hit rate when facing some unpopular queries or hot migration scenarios, and still has a good optimization effect on retrieval latency.

[0101] The cache data in the secondary cache is updated with the number of vector retrievals. For example, the data read from the solid-state drive during the current retrieval process is updated to the secondary cache.

[0102] By setting up a multi-layer cache mechanism, the second index data is preloaded to reduce the number of accesses to the solid-state drive. The multi-layer cache not only caches high-temperature data, but also caches low-quality data and last loaded data, ensuring controllable retrieval latency in both high-frequency and unpopular query scenarios.

[0103] In step S304, a query result of the query object is obtained based on the second vector identifier.

[0104] The vector identifier of the valid category determined in step S302 (ie, the third vector identifier) ​​and the vector identifier of the valid category determined in step S303 (ie, the second vector identifier) ​​are taken together as topK target vector identifiers for recall.

[0105] Subsequently, based on the similarity measurement results between the query vector and the original vectors corresponding to the K recalled target vector identifiers, the K target vector identifiers can be re-ranked to obtain a ranking result to ensure retrieval accuracy. The images corresponding to the K target vector identifiers are then found in the image database and displayed to the user according to the ranking results.

[0106] Alternatively, the sorting result is obtained by re-sorting the original vectors corresponding to the query vector and the K target vectors, as well as other user-entered sorting criteria. For example, the K target vectors are first sorted in descending order based on the similarity results, meaning that the higher the similarity, the higher the ranking. Then, based on the user-entered criteria, the images corresponding to the K target vectors are re-sorted in the positive sorting database and displayed to the user. For example, if the user-entered query image is a shoe and the condition is red, then in the positive sorting database, images of red shoes are sorted first among the K target vectors, and the shoes that deviate from red are sorted further back.

[0107] Based on the same concept as the aforementioned embodiment of the vector retrieval method, the embodiment of the present application further provides a vector retrieval device 600, which includes units or modules for implementing each step in the vector retrieval method shown in Figures 3-5.

[0108] Figure 6 is a schematic diagram of the structure of a vector search device provided in an embodiment of the present application. The device can be deployed on any device, equipment, platform, or device cluster with computing capabilities, such as a computer or server, to implement high-recall (e.g., topk = 1000+) vector search scenarios, ensuring high search accuracy and low search latency for vector searches.

[0109] As shown in FIG6 , the vector retrieval device 600 includes at least: an acquisition module 601, a first query module 602, a second query module 603, and a recall module 604; wherein the acquisition module 601 is used to acquire a query object; the first query module 602 is used to obtain N vector identifiers corresponding to the query object by querying a first index, wherein the first index is stored in a memory, N is a positive integer, and the N vector identifiers include multiple first vector identifiers, wherein the first vector identifier is a vector identifier that is uncertain to be recalled; the second query module 603 is used to determine the second vector identifiers corresponding to the multiple first vector identifiers by querying index items corresponding to the multiple first vector identifiers in the second index, wherein the second vector identifier is a vector identifier that is certain to be recalled, wherein the second index is stored in a persistent storage medium, and the compression ratio of the vectors in the first index items is greater than that of the vectors in the second index; and the recall module 604 is used to obtain a query result for the query object based on the second vector identifier.

[0110] In a possible implementation, the second index includes multiple clusters, each cluster includes multiple index items; in vector retrieval, each index item in a cluster has the same probability of being accessed.

[0111] In a possible implementation, the index item includes a vector and an index pointer corresponding to the vector, and the index pointer is related to the identifier of the cluster and the identifier of the vector in the cluster.

[0112] In another possible implementation, multiple index items in the cluster are stored in the same storage area in the persistent storage medium.

[0113] In another possible implementation, the vector search apparatus 600 further includes an aggregation module 605 , which is configured to aggregate access requests to the persistent storage medium, and aggregate access requests for index items in the same cluster into a same access request batch.

[0114] In another possible implementation, the N vector identifiers also include a third vector identifier that is determined to be recalled; and the query result of the query object is related to the third vector identifier.

[0115] In another possible implementation, the vector retrieval device 600 further includes an identification module 606 , which identifies the N vector identifiers to obtain the first vector identifier and a third vector identifier, wherein the third vector identifier is the vector identifier determined to be recalled.

[0116] In another possible implementation, the identification module 606 is specifically used to identify the first vector identifier and the third vector identifier based on the quantization loss value corresponding to each vector identifier in the N vector identifiers, and the quantization loss value of the third vector identifier in the first index is less than the quantization loss value of the first vector identifier in the first index; the quantization loss value corresponding to each vector identifier is determined based on the loss between the vector in the first index corresponding to each vector identifier and the original vector corresponding to each vector identifier.

[0117] In another possible implementation, the vector search apparatus 600 further includes a reading module 607 configured to read a candidate index item set in the second index, where the candidate index item set includes index items in the second index corresponding to the multiple first vector identifiers.

[0118] In another possible implementation, the reading module 607 is specifically configured to read the candidate index item set from a first-level cache space in the memory; wherein the first-level cache space stores data in the second index that is accessed more frequently than a preset frequency.

[0119] In another possible implementation, the first index item in the second index is further stored in the first-level cache space, where the first index item is an index item corresponding to a vector identifier whose probability of being determined to be the first vector identifier is greater than a preset probability.

[0120] In another possible implementation, the reading module 607 is specifically used to read the candidate index item set in the second index from the first-level cache space in the memory; when the candidate index item set in the second index is not read from the first-level cache space, the candidate index item set in the second index is read from the second-level cache space in the memory; when the candidate index item set in the second index is not read from the second-level cache space, the candidate index item set in the second index is read from the persistent storage medium; wherein the second-level cache space stores the data read from the persistent storage medium by the previous vector retrieval.

[0121] In another possible implementation, the vector search apparatus 600 further includes an update module 608 , which is configured to update the data in the secondary cache space with the data read from the persistent storage medium.

[0122] Optionally, the persistent storage medium is a solid-state drive.

[0123] The vector search device 600 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the vector search device 600 are respectively for implementing the corresponding processes of each method in Figures 3-5. For the sake of brevity, they are not further described here.

[0124] Based on the same concept as the embodiment of the aforementioned method, a computing device is also provided in the embodiment of the present application. The computing device includes at least a processor and a memory, and a program is stored in the memory. When the processor executes the program, it can implement the units or modules of each step in the method shown in Figures 3-5.

[0125] FIG7 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application.

[0126] As shown in Figure 7, the computing device 700 includes at least one processor 701, a memory 702, a communication interface 703 and a persistent storage medium 704. Among them, the processor 701, the memory 702, the communication interface 703 and the persistent storage medium 704 are communicatively connected, and the communication connection can be achieved by a wired manner (such as a bus) or by a wireless manner. The communication interface 703 is used to receive data sent by other devices (such as query pictures or query text data); the memory 702 stores first index data, the persistent storage medium 704 stores second index data, and the persistent storage medium 704 also stores computer instructions. The processor 701 executes the computer instructions to execute the method in the aforementioned method embodiment.

[0127] It should be understood that in the embodiment of the present application, the processor 701 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0128] The memory 702 is a random access memory that can provide instructions and data to the processor 701. The persistent storage medium 704 is a non-volatile random access memory that can store data persistently.

[0129] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The memory 702 may be a volatile memory such as a random access memory (RAM), which is used as an external cache memory. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0130] It should be understood that the computing device 700 according to the embodiment of the present application can execute the method shown in Figures 3-5 in the embodiment of the present application. The detailed description of the implementation of the method is given above and will not be repeated here for the sake of brevity.

[0131] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned method is implemented.

[0132] An embodiment of the present application provides a chip, which includes at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0133] An embodiment of the present application provides a computer program or a computer program product, which includes instructions. When the instructions are executed, the computer is caused to perform the above-mentioned method.

[0134] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0136] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A vector retrieval method, characterized in that: include: Get the query object; Obtaining N vector identifiers corresponding to the query object by querying a first index, wherein the first index is stored in a memory, N is a positive integer, and the N vector identifiers include a plurality of first vector identifiers, each of which is an uncertainly recalled vector identifier; Determining second vector identifiers corresponding to the multiple first vector identifiers by querying index items corresponding to the multiple first vector identifiers in a second index, where the second vector identifiers are identifiers of vectors determined to be recalled, the second index being stored in a persistent storage medium, and the compression ratio of the vectors in the first index items being greater than that of the vectors in the second index; A query result of the query object is obtained based on the second vector identifier.

2. The method according to claim 1, characterized in that The second index includes multiple clusters, each cluster includes multiple index items; in vector retrieval, each index item in the cluster has the same probability of being accessed.

3. The method according to claim 2, characterized in that The index item includes a vector and an index pointer corresponding to the vector, and the index pointer is related to the identifier of the cluster and the identifier of the vector in the cluster.

4. The method according to claim 2 or 3, wherein: The multiple index items in the cluster are stored in the same storage area in the persistent storage medium.

5. The method according to any one of claims 2 to 4, characterized in that: Also includes: Aggregation processing is performed on the access requests to the persistent storage medium, and the access requests for the index items in the same cluster are aggregated into the same access request batch.

6. The method according to any one of claims 1 to 5, characterized in that The N vector identifiers also include a third vector identifier determined to be recalled; The query result of the query object is related to the third vector identifier.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: The N vector identifiers are identified to obtain the first vector identifier and the third vector identifier, wherein the third vector identifier is the vector identifier determined to be recalled.

8. The method according to claim 7, characterized in that Also includes: The identifying the N vector identifiers to obtain the first vector identifier and the third vector identifier includes: Based on the quantization loss value corresponding to each vector identifier in the N vector identifiers, identifying the first vector identifier and the third vector identifier, the quantization loss value of the third vector identifier in the first index being less than the quantization loss value of the first vector identifier in the first index; The quantization loss value corresponding to each vector identifier is determined based on the loss between the vector in the first index corresponding to each vector identifier and the original vector corresponding to each vector identifier.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: A candidate index item set in the second index is read, where the candidate index item set includes index items in the second index corresponding to the multiple first vector identifiers.

10. The method according to claim 9, wherein: The reading of the candidate index item set in the second index includes: Reading the candidate index item set from the first-level cache space in the memory; The first-level cache space stores data in the second index whose access frequency is greater than a preset frequency.

11. The method according to claim 10, characterized in that The first-level cache space further stores a first index item in the second index, where the first index item is an index item corresponding to a vector identifier whose probability of being determined to be the first vector identifier is greater than a preset probability.

12. The method according to any one of claims 9 to 11, characterized in that: The reading of the candidate index item set in the second index includes: Reading a candidate index item set in the second index from the first-level cache space in the memory; When the candidate index item set in the second index is not read from the first-level cache space, reading the candidate index item set in the second index from the second-level cache space in the memory; When the candidate index item set in the second index is not read from the secondary cache space, reading the candidate index item set in the second index from the persistent storage medium; The secondary cache space stores data read from the persistent storage medium by a previous vector search.

13. The method according to claim 12, characterized in that Also includes: The data in the secondary cache space is updated using the data read from the persistent storage medium.

14. A vector search device, characterized in that: include: Acquisition module, used to obtain query objects; A first query module is configured to obtain N vector identifiers corresponding to the query object by querying a first index, where the first index is stored in a memory, N is a positive integer, and the N vector identifiers include a plurality of first vector identifiers, where the first vector identifiers are vector identifiers that are uncertainly recalled; a second query module, configured to determine second vector identifiers corresponding to the multiple first vector identifiers by querying index items corresponding to the multiple first vector identifiers in a second index, where the second vector identifiers are identifiers of vectors determined to be recalled, the second index being stored in a persistent storage medium, and the compression ratio of vectors in the first index items being greater than that of vectors in the second index; A recall module is configured to obtain a query result of the query object based on the second vector identifier.

15. The device according to claim 14, characterized in that The second index includes multiple clusters, each cluster includes multiple index items; in vector retrieval, each index item in the cluster has the same probability of being accessed.

16. The device according to claim 15, characterized in that The index item includes a vector and an index pointer corresponding to the vector, and the index pointer is related to the identifier of the cluster and the identifier of the vector in the cluster.

17. The device according to claim 15 or 16, characterized in that The multiple index items in the cluster are stored in the same storage area in the persistent storage medium.

18. The device according to any one of claims 15 to 17, characterized in that Also includes: The aggregation module is configured to aggregate the access requests to the persistent storage medium, and aggregate the access requests for the index items in the same cluster into a same access request batch.

19. The device according to any one of claims 14 to 18, characterized in that The N vector identifiers also include a third vector identifier determined to be recalled; The query result of the query object is related to the third vector identifier.

20. The device according to claim 19, characterized in that Also includes: The identification module identifies the N vector identifiers to obtain the first vector identifier and the third vector identifier, wherein the third vector identifier is the vector identifier determined to be recalled.

21. The device according to claim 20, characterized in that The identification module is specifically configured to identify, based on the quantization loss value corresponding to each vector identifier in the N vector identifiers, the first vector identifier and the third vector identifier, where the quantization loss value of the third vector identifier in the first index is less than the quantization loss value of the first vector identifier in the first index; The quantization loss value corresponding to each vector identifier is determined based on the loss between the vector in the first index corresponding to each vector identifier and the original vector corresponding to each vector identifier.

22. The device according to any one of claims 14 to 21, characterized in that Also includes: A reading module is configured to read a candidate index item set in the second index, where the candidate index item set includes index items in the second index corresponding to the multiple first vector identifiers.

23. The device according to claim 22, characterized in that The reading module is specifically used for: Reading the candidate index item set from the first-level cache space in the memory; The first-level cache space stores data in the second index whose access frequency is greater than a preset frequency.

24. The device according to claim 23, characterized in that The first-level cache space further stores a first index item in the second index, where the first index item is an index item corresponding to a vector identifier whose probability of being determined to be the first vector identifier is greater than a preset probability.

25. The device according to any one of claims 22 to 24, characterized in that The reading module is specifically used for: Reading a candidate index item set in the second index from the first-level cache space in the memory; When the candidate index item set in the second index is not read from the first-level cache space, reading the candidate index item set in the second index from the second-level cache space in the memory; When the candidate index item set in the second index is not read from the secondary cache space, reading the candidate index item set in the second index from the persistent storage medium; The secondary cache space stores data read from the persistent storage medium by a previous vector search.

26. The device according to claim 25, characterized in that Also includes: An update module is used to update the data in the secondary cache space using the data read from the persistent storage medium.

27. An electronic device comprising a memory and a processor, characterized in that: The memory stores executable code, and the processor executes the executable code to implement the method according to any one of claims 1 to 13.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 13.

29. A computer program or computer program product, characterized in that The computer program or computer program product comprises instructions, and when the instructions are executed, the method according to any one of claims 1 to 13 is implemented.

Images