A method for asynchronous vector index construction and vector retrieval

Abstract

The present invention discloses an asynchronous vector index construction and vector retrieval method, belonging to the technical field of databases. The construction of the asynchronous vector index is to use a time series database as the underlying storage engine of the vector database, and the time series database includes a memory buffer, a background working thread and a disk; taking the data storage unit of the time series database as a unit, when the size of the memory buffer reaches a buffer threshold, the background working thread converts the vector data in the memory buffer into the storage format of the data storage unit. At this time, if the scale of the vector data reaches the scale threshold, a vector index is constructed for it and written to the disk; the index construction parameters are dynamically adjusted according to the scale of the vector data, and the time series database asynchronously merges the data storage units in the disk and rebuilds the vector index during the merging period; and the asynchronous vector retrieval is to perform partition query on the asynchronous vector index constructed based on the asynchronous vector index construction method.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular relates to an asynchronous vector index construction and vector retrieval method. Background Art

[0002] In the prior art, some databases support vector data storage, but lack the ability to efficiently retrieve vector data. Vector retrieval is to find several vectors that are most similar to the query vector from a large amount of vector data based on the query vector. The vector similarity is usually measured using methods such as Euclidean distance and cosine similarity. The current vector retrieval method relies on brute force search of all stored vector data, resulting in extremely low retrieval efficiency.

[0003] Vector index is an index structure designed for vector data (usually high-dimensional data, such as feature vectors of images, text or other complex objects). By building an index, it can speed up similarity queries between vectors and improve retrieval efficiency. However, building a vector index often requires a lot of computing resources. If the vector index is maintained every time data is written, it will have a significant negative impact on the overall write performance of the system. In addition, to ensure the retrieval efficiency of the vector index, it is usually necessary to build the index after the data scale reaches a certain threshold (the scale depends on the index algorithm). If the data scale is too small, the query performance of the index may be the same as brute force search, and it may even reduce the recall rate. The recall rate is an evaluation indicator in vector retrieval, which is used to measure the ability of the vector retrieval system to retrieve all relevant vector data. The higher the recall rate, the higher the accuracy of the retrieval of the vector retrieval system; if the data scale is too large, the process of building the index will be very time-consuming.

[0004] Therefore, how to achieve millisecond-level high-recall vector data retrieval while ensuring system write performance is a difficult problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the shortcomings of the prior art and achieve the purpose of ensuring efficient write performance of the database system while supporting vector data retrieval with high recall rate in milliseconds, the present invention adopts the following technical solutions:

[0006] An asynchronous vector index construction method is applied to a vector database VectorDB. The vector database VectorDB uses a time series database TSDB as an underlying storage engine and has efficient writing performance and point query performance. The time series database TSDB includes a memory buffer, a background working thread, and a disk.

[0007] Taking the data storage unit Level File of the time series database TSDB as the unit, when the memory buffer size reaches the buffer threshold, the background worker thread converts the vector data in the memory buffer into the storage format of the data storage unit Level File. At this time, if the scale of the vector data reaches the scale threshold, a vector index is constructed for it and written to the disk;

[0008] Dynamically adjust index building parameters according to the scale of vector data. As the stored vector data continues to increase, the vector index originally built and written to disk may no longer adapt to the new data distribution. In this case, the time series database TSDB asynchronously merges the data storage unit Level File in the disk and rebuilds the vector index during the merge to adapt the vector index to the new data distribution and improve the index's retrieval accuracy and recall rate.

[0009] The present invention uses an asynchronous method to build vector indexes, avoiding conflicts between vector writing and index building. When the Level File of TSDB is generated, the background index building and data persistence are automatically triggered according to the vector data scale to ensure that the writing performance is not affected.

[0010] Furthermore, the vector database VectorDB is a storage engine implemented based on the log structure merge tree LSM-Tree with the time series database TSDB as the bottom layer. The storage unit Level File of the time series database TSDB is arranged in a hierarchical structure on the disk. The data is first cached in the memory buffer in the order of writing. When the memory buffer reaches the cache threshold, it is flushed to the disk to form the lowest level of data storage unit Level File. As the amount of data increases, the low-level data storage unit Level File gradually migrates to the next higher level through the merge operation. The volume of the data storage unit Level File at each level is significantly larger than that of the previous level, usually increasing at a fixed rate (such as 10 times). Through the combination of LSM-Tree and vector index, using the efficient writing and storage management capabilities of the LSM-Tree architecture, combined with vector index technology, the present invention achieves efficient and accurate vector data retrieval while maintaining the powerful writing performance of TSDB.

[0011] Furthermore, when the time series database TSDB constructs a vector index for the data storage unit Level File, it analyzes key indicators such as the quantity and dimension of the vector data contained in the data storage unit Level File, and dynamically adjusts key parameters for vector index construction based on the key indicators. The key parameters include the number of clusters, blocking strategy, etc., so as to achieve a balance between retrieval efficiency and resource consumption for data of different data sizes. By dynamically adjusting the index construction parameters, it can effectively solve the problems of reduced index accuracy and performance bottlenecks caused by uneven vector data size, and enable the constructed vector index to adapt to different data distributions.

[0012] Furthermore, the cached vector training results are obtained through the quantizer to improve the efficiency of constructing the vector index. The quantizer trains the vector data through the index algorithm (including the algorithm that relies on vector quantization or clustering). By discretizing or clustering the high-dimensional vector data, it is used to effectively compress the data to reduce storage and computing costs, while retaining the similarity between vectors. The time series database TSDB caches the trained quantizers and directly uses the corresponding cached quantizers when constructing the vector index next time without the need for vector training, thereby greatly reducing the time consumption of constructing the vector index. The present invention caches the trained vector training results (quantizers), reuses the cached quantizers in the subsequent index construction process, avoids repeated training, greatly reduces the computational overhead and time consumption of index construction, and improves the overall write performance of the system.

[0013] Furthermore, the time series database TSDB selects the index construction parameters such as the number of clusters and the number of training iterations that perform best within the range of vector data scales contained in each data storage unit Level File, and uses them as the unique identifier of the quantizer cache to construct a vector index;

[0014] Before training vector data, the time series database TSDB combines the range of vector data size and the corresponding index construction parameters into a quantizer cache identifier, and checks whether a corresponding quantizer cache exists; if a corresponding quantizer cache exists, the cached quantizer is directly used to build the vector index; if a corresponding quantizer cache does not exist, vector data training is performed, and the trained quantizer and its identifier are cached in the memory for use in subsequent vector index construction.

[0015] An asynchronous vector retrieval method, which performs partition query based on an asynchronous vector index constructed by the asynchronous vector index construction method, comprises the following steps:

[0016] Step 1: Determine whether the column in the query condition is a partition column;

[0017] Step 2: If it is a partition column, partition pruning is performed according to the query conditions, the storage information of the column is extracted from the metadata of each partition, and the partitions that do not meet the conditions are filtered out to reduce the query scope;

[0018] Step 3: Concurrently query the vector data in the partitions. Perform concurrent queries on the partitions that meet the query conditions and return a set of data that is most similar to the query vector in the query conditions.

[0019] The present invention combines the distributed computing framework of the database system DolphinDB to achieve efficient vector retrieval under large-scale vector data: large-scale data is partitioned. When retrieving vectors, the partitioned data is first pre-filtered according to the query conditions to narrow the search scope; then, each partition executes the vector retrieval task in parallel, making full use of computing resources, thereby greatly improving the retrieval efficiency.

[0020] Furthermore, in step 3, the query within each partition includes the following steps:

[0021] Step 3.1: Filter data according to the query conditions and obtain the global row number of the partition data where the data that meets the conditions is located;

[0022] Step 3.2: The partitioned data includes two parts: the memory buffer and the disk data storage unit Level File. According to the global line number, the vector database VectorDB locates the local line number in each data storage unit Level File in the disk;

[0023] Step 3.3: For the vector data in the memory buffer, the vector index has not been built yet, and the vector database VectorDB will not retrieve the vector data, but directly return the row number that matches the query condition;

[0024] Step 3.4: For the data in the disk data storage unit Level File, according to the local row number, perform vector retrieval on each data storage unit Level File to obtain a set of data that is most similar to the query vector;

[0025] Step 3.5: Summarize the vector data retrieved from the data storage unit Level File;

[0026] Step 3.6: Merge the data in the memory buffer and the data on disk and return it as the final query result of the partition.

[0027] The pre-filtering strategy used within the partition is to first filter the data in the partition according to the query conditions and then perform vector retrieval. This ensures the efficiency of the retrieval while avoiding redundant calculations and improving the retrieval accuracy.

[0028] Furthermore, in step 3.4, if the number of data in the data storage unit Level File that meets the query conditions is less than the quantity threshold, the data in the data storage unit Level File is directly returned; otherwise, approximate retrieval is performed using the local row number and the vector index in the data storage unit Level File.

[0029] The advantages and beneficial effects of the present invention are:

[0030] The present invention significantly improves the retrieval performance of vector data by introducing vector indexes in the Level File of TSDB; dynamically adjusts the index construction parameters for vector data of different scales to balance the construction efficiency and retrieval accuracy of the vector index; and further reduces the overhead of index construction by caching the training results of the vector, so that TSDB can still maintain high writing and retrieval performance in the scenario of rapid growth of vector data. In addition, the asynchronous vector index construction and data merging mechanism of the present invention not only ensures the high efficiency of TSDB writing performance, but also improves the adaptability of the vector index to changes in data distribution. By regularly rebuilding the vector index, the system can continuously optimize the retrieval accuracy of the index to meet the needs of fast and accurate retrieval under large-scale vector data. On the whole, the present invention effectively solves the problems of high training overhead and slow index construction in vector retrieval, and realizes vector data retrieval with high recall rate in milliseconds while ensuring the efficient writing performance of TSDB. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flowchart of a method for constructing an asynchronous vector index according to an embodiment of the present invention.

[0032] Figure 2 Schematic diagram of a quantizer cache in an embodiment of the present invention.

[0033] Figure 3 It is a flow chart of an asynchronous vector retrieval method in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0035] like Figure 1As shown in the figure, an asynchronous vector index construction method is applied to the vector database VectorDB. The vector database VectorDB uses the time series database TSDB (Time Series Database) as the underlying storage engine and combines the vector index to achieve single-machine billion-level vector data storage and millisecond-level vector retrieval. The time series database TSDB is a storage engine implemented based on the log structure merge tree LSM-Tree (LogStructuredMergeTree) structure at the bottom of the DolphinDB database system. It has efficient writing performance and point query performance. The time series database TSDB includes a memory buffer, a background worker thread, and a disk.

[0036] The vector index is constructed based on the data storage unit Level File of the time series database TSDB. When the memory buffer size of the time series database TSDB reaches a certain threshold, the background working thread will convert the vector data in the memory buffer into the storage format of the data storage unit Level File. At this time, if the scale of the vector data reaches a certain threshold, a vector index will be constructed based on this batch of vector data and persisted (written) to the disk.

[0037] Periodically merge vector data and rebuild indexes, and dynamically adjust index building parameters based on the size of vector data:

[0038] As the amount of stored vector data continues to increase, the vector index that was originally built and written to disk may no longer be suitable for the new data distribution. In this case, the time series database TSDB will asynchronously merge the data storage unit LevelFile on the disk and rebuild the vector index during the merge to adapt the vector index to the new data distribution and improve the index's retrieval accuracy.

[0039] As a storage engine based on the log structure merge tree LSM-Tree architecture, the storage unit Level File of the time series database TSDB is arranged in a hierarchical structure on the disk. This hierarchical structure is a typical implementation in the existing technology. The data is first cached in the memory buffer in the order of writing. When the memory buffer is full, the data will be flushed to the disk to form the lowest level of data storage unit Level File. As the amount of data increases, the low-level data storage unit Level File gradually migrates to a higher level through merging operations. The size of each level of data storage unit Level File will be significantly larger than the previous level, usually increasing at a fixed rate (such as 10 times).

[0040] In order to ensure the query performance and efficiency of each vector index, the time series database TSDB will dynamically adjust the construction parameters of the vector index according to the scale of the vector data in each data storage unit Level File. Specifically, when the time series database TSDB builds a vector index for a data storage unit Level File, it analyzes key indicators such as the number and dimension of the vector data contained in the data storage unit Level File, and dynamically adjusts the key parameters of the vector index construction based on these key indicators, such as the number of clusters of the vector index, the block strategy, etc. This strategy ensures a balance between retrieval efficiency and resource consumption for data of different data sizes.

[0041] By dynamically adjusting index construction parameters, we can effectively solve the problems of reduced index accuracy and performance bottlenecks caused by uneven vector data size, and enable the constructed vector index to adapt to different data distributions.

[0042] Through the quantizer, the cache vector training results are obtained to improve the efficiency of constructing the vector index. The quantizer is one of the key technologies in vector nearest neighbor retrieval. It discretizes or clusters high-dimensional vector data to effectively compress data to reduce storage and computing costs while retaining the similarity between vectors.

[0043] For indexing algorithms that require training of vector data (usually algorithms that rely on vector quantization or clustering), the time series database TSDB will cache the trained quantizers. This way, the cached quantizers can be used directly the next time a vector index is built without the need for vector training, thus greatly reducing the time spent on building vector indexes. Figure 2 shown.

[0044] Taking a single vector database VectorDB table as an example, the time series database TSDB selects the index construction parameters (usually empirical values) such as the number of clusters and the number of training iterations that perform best within the range of vector data scales contained in each data storage unit Level File as the unique identifier of the quantizer cache to construct a vector index, including the following steps:

[0045] Step 1: Before training vector data, the time series database TSDB will first combine the vector data scale range and the corresponding index construction parameters into a quantizer cache identifier, and check whether there is a corresponding quantizer cache. For example, the data storage unit Level File for constructing the vector index currently contains 30,000 vector data, which is in the vector data scale range [25,600, 102,400). According to experience, the quantizer in this range performs best when the number of clusters is 256 and the number of training iterations is 12. Therefore, the quantizer trained with these empirical value parameters is selected as the training cache for this range.

[0046] Step 2: If the quantizer cache in step 1 exists, skip the training process and directly use the cached quantizer to build the vector index; if the quantizer cache in step 1 does not exist, perform vector data training and cache the trained quantizer and its identifier in the memory for subsequent use in building the vector index. For example, if the data storage unit Level File for building the vector index contains 30,000 vector data, which is in the vector data scale range [25,600, 102,400), the quantizer will be trained with the best training parameters in this range (for example, the number of clusters is 256 and the number of training iterations is 12 times). After the training is completed, the trained quantizer will be cached in the memory. The next time the vector data is trained, if the scale of the vector data involved in the training is in the range [25,600, 102,400), the quantizer can be directly used to build the vector index.

[0047] like Figure 3 As shown in FIG. 1 , an asynchronous vector retrieval method combines the vector database VectorDB with the distributed computing framework of the database system DolphinDB to achieve efficient vector retrieval. The vector retrieval query format of the database system DolphinDB is:

[0048] Select{columns}fromtable{where}orderbyrowEuclidean(vector_column,query_vector)limitk

[0049] Among them, vector_column represents the name of the vector data column, query_vector represents the query vector, and k represents the number of vectors that are most similar to the query vector.

[0050] Take the following query statement as an example:

[0051] Select*fromtablewhereidin[1,3,5]orderbyrowEuclidean(vector_column,[10,11,12,5,7.1])limit10

[0052] The semantics of this query statement is: retrieve the 10 data that are most similar to the query vector [10,11,12,5,7.1] and whose id is in [1,3,5]. The specific query process of partition query is as follows:

[0053] Step 1: The computing layer determines whether the id column in the query filter condition is a partition column.

[0054] Step 2: If the id column is a partition column, the computing layer performs partition pruning based on the query condition "id in [1, 3, 5]". The storage information of the id column is extracted from the metadata of each partition, and the partitions that do not meet the condition are filtered out to reduce the query scope. Assuming that there are currently 10 partitions, after partition pruning, it can be determined that the id values ​​of 7 partitions are not in the range of [1, 3,5]. Therefore, only the data of 3 partitions need to be queried in the end.

[0055] Step 3: Concurrently query the vector data in the partitions. Concurrently query the three partitions selected in step 2 to find the 10 data that are most similar to the query vector. After the query is completed, the computing layer will summarize the query results of each partition and finally obtain 30 candidate data. Then, the computing layer will calculate the 10 data that are most similar to the query vector from these 30 data and return them to the user.

[0056] For queries within each partition, the specific steps are as follows:

[0057] Step 3.1: Filter the data according to the query condition "id in [1, 3, 5]" to obtain the global row number of the partition data where the data that meets the condition is located.

[0058] Step 3.2: The partitioned data includes two parts: the memory buffer and the disk data storage unit Level File. According to the global line number obtained in step 3.1, the vector database VectorDB will locate the local line number in each data storage unit LevelFile on the disk.

[0059] Step 3.3: Since the vector data in the memory buffer has not yet been indexed, the vector database VectorDB does not retrieve the vector data, but directly returns the row number that matches the query condition.

[0060] Step 3.4: Perform vector search for the data in the disk data storage unit Level File. Perform vector search for each data storage unit Level File based on the local row number obtained in step 3.2. Each data storage unit Level File will retrieve 10 data that are most similar to the query vector. If the number of data that meets the conditions in a data storage unit Level File is less than 10, the data in the data storage unit Level File will be directly returned; otherwise, the local row number and the vector index in the data storage unit Level File will be used for approximate search.

[0061] Step 3.5: Summarize the vector data obtained from the data storage unit Level File search. Assume that there are 3 data storage unit Level Files on the partition disk, and each data storage unit Level File will search for the 10 data that are most similar to the query vector, for a total of 30 data. The vector database VectorDB will calculate the 10 data that are most similar to the query vector.

[0062] Step 3.6: Merge the data in the memory buffer and the disk. Merge the 10 data obtained in step 3.5 with the data in the memory buffer, calculate the 10 data in the partition that are most similar to the query vector, and return them to the computing layer as the final query result of the partition.

[0063] In order to verify the high efficiency of the vector retrieval of the present invention, a performance test was conducted on the vector retrieval of the vector database VectorDB.

[0064] Experimental environment:

[0065] CPU: Intel Core i5-12500

[0066] Memory: 64 GiB

[0067] Disk: PM9B1 NVMe Samsung 256GiB

[0068] Dataset size: 1,000,000 vector data with a dimension of 128

[0069] Table 1. Vector retrieval time and recall rate

[0070] Find the most similar vector Query time Recall 1 8ms 100% 16 11ms 87.6%

[0071] The experimental results are shown in Table 1. The vector database VectorDB shows high recall rate and excellent performance in vector retrieval.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An asynchronous vector index construction method, applied to a vector database, characterized in that: The vector database uses a time series database as an underlying storage engine, and the time series database includes a memory buffer, a background working thread, and a disk; Taking the data storage unit of the time series database as a unit, when the memory buffer size reaches the buffer threshold, the background worker thread converts the vector data in the memory buffer into the storage format of the data storage unit. At this time, if the scale of the vector data reaches the scale threshold, a vector index is constructed for it and written to the disk; Dynamically adjust index building parameters according to the scale of vector data. The time series database asynchronously merges the data storage units on the disk and rebuilds the vector index during the merging process. The cached vector training results are obtained through the quantizer. The quantizer trains the vector data through the index algorithm to compress the data while retaining the similarity between vectors. The time series database caches the trained quantizer and directly uses the corresponding cached quantizer when constructing the vector index next time. The time series database selects the index construction parameter with the best performance within the range of vector data size contained in each data storage unit, and uses it as the unique identifier of the quantizer cache to construct the vector index; Before performing vector data training, the time series database combines the quantizer cache identifier according to the range of the vector data scale and the corresponding index construction parameters, and checks whether there is a corresponding quantizer cache; if there is a corresponding quantizer cache, the cached quantizer is directly used to construct the vector index; if there is no corresponding quantizer cache, vector data training is performed, and the trained quantizer and its identifier are cached in the memory.

2. The method for constructing an asynchronous vector index according to claim 1, characterized in that: The vector database is a storage engine implemented based on a log structure merge tree with a time series database as the underlying layer. The storage units of the time series database are arranged in a hierarchical structure on the disk. The data is first cached in the memory buffer in the order of writing. When the memory buffer reaches the cache threshold, it is flushed to the disk to form the lowest level of data storage units. As the amount of data increases, the low-level data storage units gradually migrate to the next higher level through merge operations, and the volume of the data storage units at each level increases compared to the previous level.

3. The method for constructing an asynchronous vector index according to claim 1, characterized in that: When constructing a vector index for a data storage unit, the time series database analyzes key indicators of the vector data contained in the data storage unit, and dynamically adjusts key parameters for constructing the vector index according to the key indicators, so as to achieve a balance between retrieval efficiency and resource consumption for data of different data sizes.

4. An asynchronous vector retrieval method, characterized in that: Performing partition query based on an asynchronous vector index constructed by the asynchronous vector index construction method according to any one of claims 1 to 3 comprises the following steps: Step 1: Determine whether the column in the query condition is a partition column; Step 2: If it is a partition column, partition pruning is performed according to the query conditions, the column storage information is extracted from the metadata of each partition, and the partitions that do not meet the conditions are filtered out; Step 3: Concurrently query the vector data in the partitions. Perform concurrent queries on the partitions that meet the query conditions and return a set of data that is most similar to the query vector in the query conditions.

5. The asynchronous vector search method according to claim 4, characterized in that: In step 3, the query within each partition includes the following steps: Step 3.1: Filter data according to the query conditions and obtain the global row number of the partition data where the data that meets the conditions is located; Step 3.2: The partitioned data includes a memory buffer and a disk data storage unit. According to the global row number, the vector database is located at the local row number in each data storage unit on the disk; Step 3.3: For the vector data in the memory buffer, directly return the row number that matches the query condition; Step 3.4: for the data in the disk data storage unit, according to the local row number, perform vector retrieval on each data storage unit to obtain a set of data that is most similar to the query vector; Step 3.5: Summarize the vector data retrieved from the data storage unit; Step 3.6: Merge the data in the memory buffer and the data on disk and return it as the final query result of the partition.

6. The asynchronous vector search method according to claim 5, characterized in that: In step 3.4, if the number of data in the data storage unit that meets the query condition is less than the quantity threshold, the data in the data storage unit is directly returned; otherwise, approximate retrieval is performed using the local row number and the vector index in the data storage unit.

Images