A hybrid vector retrieval method and device for high-concurrency scenarios

Abstract

The present invention discloses a hybrid vector retrieval method and device for high-concurrency scenarios. For vector data, the method uses graph and quantization coding to calculate distance to construct graph and quantization indexes, and persistently stores the constructed indexes on an SSD hard disk. During a query, given a large number of high-concurrency query vectors, search candidate points are first obtained based on the quantization index. Then, multiple queues are established and the search candidate points are assigned to the corresponding queues. For each queue, the search candidate point at the head of the queue is assigned to read the hard disk to obtain persistently stored graph index neighbor information. Finally, a greedy algorithm is used to search and return the approximate nearest neighbors of the query point. The present invention is targeted at application scenarios of large-scale, high-concurrency queries. By efficiently allocating and scheduling queries, the risk of delay caused by query congestion is avoided. At the same time, the reading strategy of the SSD hard disk is optimized, achieving a balance between hard disk reading and vector calculation, and improving the search speed of vectors.

Description

Technical Field

[0001] The present invention belongs to the field of approximate nearest neighbor search, and in particular relates to a hybrid vector retrieval method and device for high-concurrency scenarios. Background Art

[0002] In the digital economy era, new technologies such as big data, cloud computing, and mobile internet have spawned massive amounts of unstructured data online, including images, videos, and text. To effectively retrieve this data, people typically use vector retrieval methods based on trees, hashes, quantization, and neighbor graphs. Currently, algorithms that build indexes based on neighbor graphs have become the mainstream in vector retrieval due to their excellent retrieval capabilities.

[0003] However, current methods based on nearest neighbor graphs rely too heavily on memory, leading to high memory costs as data scale increases. To address this issue, a mainstream approach is to use a hybrid vector retrieval strategy that combines memory and disk. This strategy saves memory during retrieval by storing the memory-intensive nearest neighbor graph index on disk.

[0004] However, as the number of queries increases dramatically and queries become more concurrency-intensive, efficient allocation and scheduling of queries and optimization of hard disk reads are increasingly becoming bottlenecks in retrieval speed. To address this issue, the present invention proposes a hybrid vector retrieval method and device for high-concurrency scenarios. Summary of the Invention

[0005] In view of the shortcomings and deficiencies of the above-mentioned existing technologies, the present invention aims to provide a hybrid vector retrieval method for high-concurrency scenarios:

[0006] (1) Obtain a vector dataset V, construct an index for the data in the vector dataset V using a neighbor graph and product quantization according to vector distance, and obtain a corresponding graph index and a product quantization index; store the corresponding graph index and product quantization index in an SSD hard disk for persistent storage;

[0007] (2) Get the query vector group Q = {q1,q2,q3,...,q i}, the query vector group Q contains several query vectors q i ;

[0008] Perform an approximate nearest neighbor search:

[0009] In the first step, several search candidate points are obtained based on the product quantization index;

[0010] The second step is to establish several queues, where the queue is a data structure;

[0011] Step 3: for each of the plurality of search candidate points, assign the current search candidate point to the most idle queue according to the idleness of the queues;

[0012] Step 4: For each of the queues established in the second step where the search candidate point exists, the search candidate point at the head of the queue is assigned to read the persistently stored graph index to obtain neighbor information in the graph index;

[0013] The head of the queue refers to: the first data in the queue;

[0014] (3) For each query vector q in the query vector group Q i , according to the neighbor information in the graph index read, perform greedy algorithm search, record the nearest neighbor and query vector q i The vector distance to the nearest neighbor is sorted from near to far to obtain the query vector q i The approximate nearest neighbors of are used as the retrieval results.

[0015] As a preferred solution, in step (2), in the second step, the queue includes: a queue using a linked list structure, a queue using a sequential list structure, and the queue only allows the head to exit and the tail to enter.

[0016] As a preferred solution, in step (2), the specific allocation strategy of allocating the current search candidate point to the most idle queue according to the idleness of the queue in the third step includes:

[0017] For the query vector q that needs to be allocated currently i If there is an empty queue, the empty queue will be loaded first; if the queues are not empty, the query vector q will be loaded according to the current queue status of the queue. i Assign to the queue with the smallest queue length;

[0018] The queues and query vectors q are stored by several hosts. i .

[0019] As a preferred solution, the specific reading process described in the fourth step of step (2) is as follows: for the candidate search point that has currently reached the head of each queue, a CPU thread resource is allocated to the queue where each candidate search point is located to interact with the read-write thread on the SSD hard disk controller, the location of the neighbor information is obtained from the graph index, and according to the location of the neighbor information, the neighbor information is read from the storage area corresponding to the SSD hard disk.

[0020] The present invention also provides a hybrid vector retrieval device for high-concurrency scenarios, comprising:

[0021] Index building blocks for:

[0022] Obtain a vector dataset V, index the data in the vector dataset V using a neighbor graph and product quantization according to vector distance, and obtain a corresponding graph index and a product quantization index; store the corresponding graph index and product quantization index in an SSD hard disk for persistent storage;

[0023] Neighbor information reading module, used for:

[0024] Get the query vector group Q = {q1,q2,q3,...,q i}, the query vector group Q contains several queries q i ;

[0025] Perform an approximate nearest neighbor search:

[0026] In the first step, several search candidate points are obtained based on the product quantization index;

[0027] The second step is to establish several queues, where the queue is a data structure;

[0028] Step 3: for each of the plurality of search candidate points, assign the current search candidate point to the most idle queue according to the idleness of the queues;

[0029] Step 4: For each of the queues established in the second step where the search candidate point exists, the search candidate point at the head of the queue is assigned to read the persistently stored graph index to obtain neighbor information in the graph index;

[0030] Search and sort module for:

[0031] For each query vector q in the query vector group Q i , according to the neighbor information in the graph index read, perform greedy algorithm search, record the nearest neighbor and query vector q i The vector distance to the nearest neighbor is sorted from near to far to obtain the query vector q i The approximate nearest neighbors of are used as the retrieval results.

[0032] Beneficial effects of the present invention:

[0033] Aiming at application scenarios of large-scale, high-concurrency queries, the present invention avoids the risk of delay caused by query congestion by efficiently allocating and scheduling queries. At the same time, it optimizes the reading strategy of the SSD hard disk, achieves a balance between hard disk reading and vector calculation, and improves the vector search speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific embodiments or the description of the prior art. Hereinafter, some specific embodiments of the present invention will be described in detail in an illustrative, non-limiting manner with reference to the drawings. The same reference numerals in the drawings indicate the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale.

[0035] In the attached figure:

[0036] Figure 1 is a flow chart of the present invention;

[0037] Figure 2 It is a flow chart of an embodiment of image search according to the present invention. DETAILED DESCRIPTION

[0038] In order to make the scheme and features of the present invention more clear, the following Figure 2 The present invention is further described in detail with reference to the accompanying drawings.

[0039] The technical solution of the present invention can be applied to scenarios requiring vector search, where the vector is a corresponding feature vector extracted from any data type such as picture, voice, text, etc., and the feature vector constructs a vector dataset V.

[0040] The following examples illustrate vector search using image search scenarios:

[0041] (1) Obtain an image vector dataset V, where the image vector data is obtained by vectorizing the image data in the image dataset using machine learning technology, construct an index for the data in the vector dataset V using a neighbor graph and product quantization according to the vector distance, and obtain a corresponding graph index and a product quantization index; store the corresponding graph index and product quantization index in an SSD hard disk for persistent storage;

[0042] (2) For the image data that needs to be queried, the query image is converted into a query vector using machine learning technology, that is, the query vector group Q = {q1,q2,q3,...,q i}, the query vector group Q contains several query vectors q i ;

[0043] Perform an approximate nearest neighbor search:

[0044] In the first step, several search candidate points are obtained based on the product quantization index;

[0045] The second step is to establish several queues, where the queue is a data structure;

[0046] Step 3: for each of the plurality of search candidate points, assign the current search candidate point to the most idle queue according to the idleness of the queues;

[0047] Step 4: For each of the queues established in the second step where the search candidate point exists, the search candidate point at the head of the queue is assigned to read the persistently stored graph index to obtain neighbor information in the graph index;

[0048] The head of the queue refers to: the first data in the queue;

[0049] (3) For each query vector q in the query vector group Q i , according to the neighbor information in the graph index read, perform greedy algorithm search, record the nearest neighbor and query vector q i The vector distance to the nearest neighbor is sorted from near to far to obtain the query vector q i The approximate nearest neighbors of are used as the retrieval results.

[0050] In step (2), the specific allocation strategy for allocating the current search candidate point to the most idle queue according to the idleness of the queue in the third step includes:

[0051] For the query vector q that needs to be allocated currently i If there is an empty queue, the empty queue will be loaded first; if the queues are not empty, the query vector q will be loaded according to the current queue status of the queue. i Assign to the queue with the smallest queue length;

[0052] The queues and query vectors q are stored by several hosts. i .

[0053] In step (2), the specific process of reading described in the fourth step is: for the candidate search point that has currently reached the head of each queue, a CPU thread resource is allocated to the queue where each candidate search point is located to interact with the read-write thread on the SSD hard disk controller, the location of the neighbor information is obtained from the graph index, and according to the location of the neighbor information, the neighbor information is read from the storage area corresponding to the SSD hard disk.

[0054] A hybrid vector retrieval device for high-concurrency scenarios, comprising:

[0055] Index building blocks for:

[0056] Obtain an image vector dataset V, where the image vector data is obtained by vectorizing the image data in the image dataset using machine learning techniques, index the data in the image vector dataset V using a neighbor graph and product quantization according to vector distance, and obtain corresponding graph indexes and product quantization indexes; store the corresponding graph indexes and product quantization indexes on an SSD hard drive for persistent storage;

[0057] Neighbor information reading module, used for:

[0058] Get the query vector group Q = {q1,q2,q3,...,q i}, the query vector group Q contains several query vectors q i ;

[0059] Perform an approximate nearest neighbor search:

[0060] In the first step, several search candidate points are obtained based on the product quantization index;

[0061] The second step is to establish several queues, where the queue is a data structure;

[0062] Step 3: for each of the plurality of search candidate points, assign the current search candidate point to the most idle queue according to the idleness of the queues;

[0063] Step 4: For each of the queues established in the second step where the search candidate point exists, the search candidate point at the head of the queue is assigned to read the persistently stored graph index to obtain neighbor information in the graph index;

[0064] Search and sort module for:

[0065] For each query vector q in the query vector group Q i , according to the neighbor information in the graph index read, perform greedy algorithm search, record the nearest neighbor and query vector q i The vector distance to the nearest neighbor is sorted from near to far to obtain the query vector q i The approximate nearest neighbors of are used as the retrieval results.

[0066] Aiming at application scenarios of large-scale, high-concurrency queries, the present invention avoids the risk of delay caused by query congestion by efficiently allocating and scheduling queries. At the same time, it optimizes the reading strategy of the SSD hard disk, achieves a balance between hard disk reading and vector calculation, and improves the vector search speed.

[0067] The above description is only part of the specific implementation methods of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person familiar with the art within the technical scope disclosed in the present invention should be covered by the protection scope of the present invention.

Claims

1. A hybrid vector retrieval method for high-concurrency scenarios, characterized by This method includes the following steps: (1) Obtain a vector dataset V, construct an index for the data in the vector dataset V using a neighbor graph and product quantization according to vector distance, and obtain a corresponding graph index and a product quantization index; store the corresponding graph index and product quantization index in an SSD hard disk for persistent storage; (2) Get the query vector group Q = {q1,q2,q3,...,q i }, the query vector group Q contains several query vectors q i ; Perform an approximate nearest neighbor search: In the first step, several search candidate points are obtained based on the product quantization index; The second step is to establish several queues, where the queue is a data structure; Step 3: for each of the plurality of search candidate points, assign the current search candidate point to the most idle queue according to the idleness of the queues; Step 4: For each of the queues established in the second step where the search candidate point exists, the search candidate point at the head of the queue is assigned to read the persistently stored graph index to obtain neighbor information in the graph index; The head of the queue refers to: the first data in the queue; (3) For each query vector q in the query vector group Q i , according to the neighbor information in the graph index read, perform greedy algorithm search, record the nearest neighbor and query vector q i The vector distance to the nearest neighbor is sorted from near to far to obtain the query vector q i The approximate nearest neighbors of are used as the retrieval results.

2. A hybrid vector retrieval method for high-concurrency scenarios according to claim 1, characterized in that: In step (2), in the second step, the queue includes: a queue using a linked list structure, a queue using a sequential list structure, and the queue only allows the head to exit and the tail to enter.

3. The hybrid vector retrieval method for high-concurrency scenarios according to claim 1, characterized in that: In step (2), the specific allocation strategy for allocating the current search candidate point to the most idle queue according to the idleness of the queue in the third step includes: For the query vector q that needs to be allocated currently i If there is an empty queue, the empty queue will be loaded first; if the queues are not empty, the query vector q will be loaded according to the current queue status of the queue. i Assign to the queue with the smallest queue length; The queues and query vectors q are stored by several hosts. i .

4. The hybrid vector retrieval method for high-concurrency scenarios according to claim 1 is characterized by: The specific process of reading described in the fourth step in step (2) is as follows: for the candidate search point that has currently reached the head of each queue, a CPU thread resource is allocated to the queue where each candidate search point is located to interact with the read-write thread on the SSD hard disk controller, the location of the neighbor information is obtained from the graph index, and according to the location of the neighbor information, the neighbor information is read from the storage area corresponding to the SSD hard disk.

5. A hybrid vector retrieval device for high-concurrency scenarios, characterized in that: include: Index building blocks for: Obtain a vector dataset V, index the data in the vector dataset V using a neighbor graph and product quantization according to vector distance, and obtain a corresponding graph index and a product quantization index; store the corresponding graph index and product quantization index in an SSD hard disk for persistent storage; Neighbor information reading module, used for: Get the query vector group Q = {q i ,q2,q3,...,q i }, the query vector group Q contains several query vectors q i ; Perform an approximate nearest neighbor search: In the first step, several search candidate points are obtained based on the product quantization index; The second step is to establish several queues, where the queue is a data structure; Step 3: for each of the plurality of search candidate points, assign the current search candidate point to the most idle queue according to the idleness of the queues; Step 4: For each of the queues established in the second step where the search candidate point exists, the search candidate point at the head of the queue is assigned to read the persistently stored graph index to obtain neighbor information in the graph index; Search and sort module for: For each query vector q in the query vector group Q i , according to the neighbor information in the graph index read, perform greedy algorithm search, record the nearest neighbor and query vector q i The vector distance to the nearest neighbor is sorted from near to far to obtain the query vector q i The approximate nearest neighbors of are used as the retrieval results.

Images