A query distribution-aware based vector database system

Abstract

The application relates to the field of vector databases, and aims to solve the problems of high delay and low recall rate caused by different distribution user queries, and provides a vector database system based on query distribution perception, which comprises an index construction module, a vector retrieval module, an interval measurement module, a log recording module, a distribution perception module, a distribution detection module, a distribution mapping module, a storage module, an interface agent module and a graphical management interface; the application provides an efficient solution for storage and retrieval of vector data, simultaneously, the distribution of user queries is monitored in real time, the vector distribution is effectively measured, the vector index structure is optimized through historical user queries, and the stability and reliability of vector database query quality are guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vector database, in particular to a vector database system based on query distribution perception. BACKGROUND

[0002] With the vigorous development of artificial intelligence, machine learning and deep learning, the importance of vector data in many fields is increasingly prominent. However, traditional database technology has obvious shortcomings in storing and retrieving vector data, so the emerging technology of vector database has emerged as the times require.

[0003] In the practical application of vector database, the difference between user query and vector distribution in the database is very common, for example: the modal of query vector and database vector is inconsistent, the user query distribution changes over time, etc. However, the current vector similarity retrieval in vector database is usually based on the assumption that the query vector and the vector in the database follow the same distribution. Once the query vector and the database vector distribution are different, the query performance will decrease sharply, and the performance gap may reach an order of magnitude or even more.

[0004] Therefore, the efficient storage and accurate retrieval of vector data are crucial, and the diversity of user query distribution will greatly affect the query quality of vector database. We must take targeted measures to provide an efficient solution for the storage and retrieval of vector data to meet the query requirements of different distributions, which not only can guarantee the stability and reliability of query quality, but also can significantly improve user experience, so it has very high urgency and importance. SUMMARY

[0005] In view of the shortcomings and deficiencies existing in the prior art, the present application provides a vector database system based on query distribution perception, which comprises an index construction module, a vector retrieval module, an interval measurement module, a log recording module, a distribution perception module, a distribution detection module, a distribution mapping module, a storage module, an interface agent module and a graphical management interface. An efficient solution is provided for the storage and retrieval of vector data, and the distribution of user query is monitored in real time, the difference between the distribution of query vector and the vector in the database is intuitively measured, the problem of high delay and low recall rate caused by different distribution of user query is effectively solved on the basis of ensuring the retrieval quality of the same distribution of user query, and the stability and reliability of vector database query quality are guaranteed.

[0006] The present application provides a vector database system based on query distribution perception, which comprises an index construction module, a vector retrieval module, an interval measurement module, a log recording module, a distribution perception module, a distribution detection module, a distribution mapping module;

[0007] The index building module is used to: index the target set V = {v1, v2, ... v} n} Construct an index and record the index name, where n is the number of vectors in the target set V to be retrieved;

[0008] The vector retrieval module is used to: receive the user-input query dataset Q = {q1, q2, ..., q} m} and query index name, m is the number of vectors in the query dataset Q; for each element in the query dataset Q, query the index according to the query index name to obtain the top-k approximate nearest neighbor search results;

[0009] The interval measurement module is used to: traverse each vector v in the retrieval target set V. i Calculate vector v i Distance d from the target set V Mi This forms the distance set U(d) M After traversing the target set V, record the distribution range of the target set V, which represents the degree of distribution offset of the target set V.

[0010] The log recording module is used to: record query logs, the query logs including query vector q. i Query index name, query vector q i The distribution distance from the target set V, and the number of out-of-distribution queries, h; the number of out-of-distribution queries, h, represents the number of queries whose query vectors are inconsistent with the distribution of the target set V.

[0011] The distributed sensing module is used to: calculate the query vector q i The distribution distance (distance) between the target set V and the target set V is used to update the number of out-of-distribution queries (h). It is then determined whether the number of out-of-distribution queries (h) is greater than a threshold. If it is less than or equal to the threshold, the query log is recorded using the logging module. If it is greater than the threshold, the corresponding query vector (q) is retrieved from the logging module based on the query index name. i As t i This constitutes the historical query set T = {t1, t2, ..., t} l The distribution detection module is used to perform the following processing:

[0012] The distribution detection module is configured to calculate a distribution difference coefficient differ between the historical query set T and the search target set V, determine whether the distribution of the search target set V needs to be adjusted by the historical query set T, construct a distribution optimization index by the distribution mapping module when the difference coefficient differ is greater than 0, update the number of out-of-distribution queries h to an initial value 0, and update the distribution interval of the search target set V according to the historical query set T; and when the difference coefficient differ is less than or equal to 0, do not adjust the existing index.

[0013] The distribution mapping module is configured to construct a distribution optimization index for the search target set V based on the historical query set T and the search target set V.

[0014] The construction of the distribution optimization index comprises the following steps:

[0015] a, perform a brute-force search on each vector v in the search target set V i In the historical query set T, j accurate nearest neighbors {z1, z2…z j}, calculate the mean vector

[0016] b, calculate the mapping set F = {f1, f2…f n} of the search target set V, each f i in the mapping set F corresponds to a v i in the search target set V;

[0017] The calculation formula of the mapping set F of the search target set V is as follows:

[0018] F = V + βZ

[0019] wherein, Z = {z 1avg ,z 2avg …z navg} is composed of the mean vector z iavg ; and β is an adjustable fusion coefficient.

[0020] c, input the mapping set F into the index construction module to construct an index, and after the construction is completed, replace the data in the index from the mapping set F to the search target set V, i.e., replace F = {f1, f2…f n} to V = {v1, v2…v n}, each f i corresponds to a v i ; at the same time, the structure information of the index is retained, the structure information comprises distribution information of the search target set V and the historical query set T, and the search step number between the search target vectors in the index is reduced; and the log recording module is used to record the query log.

[0021] Preferably, the query distribution perception based vector database system further comprises a storage module, an interface agent module and a graphical management interface; the storage module is used for persistently storing the retrieval target set V, index data and query logs; the graphical management interface provides an operable user interface and visualizes the vector database; the interface agent module is used for receiving requests from the graphical management interface and forwarding them to corresponding modules for execution.

[0022] Preferably, in the interval measurement module, the distance d Mi is calculated as follows:

[0023]

[0024] wherein μ represents the mean vector of the retrieval target set V, and S represents the covariance matrix of the retrieval target set V.

[0025] The distribution interval of the retrieval target set V is denoted as [low, high]; low represents the lower limit of the distribution offset of the retrieval target set V, and is calculated as follows:

[0026] low = quantile (U (d M ), percent)

[0027] high represents the upper limit of the distribution offset of the retrieval target set V, and is calculated as follows:

[0028] high = quantile (U (d M ), 1 - percent)

[0029] wherein quantile (set, percent) represents the value of the percent percentile of the elements in the set set arranged in ascending order.

[0030] Preferably, the distance distance is calculated as follows:

[0031]

[0032] wherein μ represents the mean vector of the retrieval target set V, and S represents the covariance matrix of the retrieval target set V.

[0033] The calculation method of the number h of queries outside the distribution includes: the initial value of h is 0, and h is increased by 1 when the distance distance is less than the distribution interval low or greater than the distribution interval high.

[0034] Preferably, the calculation method of the distribution difference coefficient differ includes the following steps:

[0035] S1, sampling two disjoint sets V1, V2 from the search target set V, each of the sets containing k vectors, calculating the Wasserstein distance d of set V1 and set V2 w (V1,V2);

[0036] S2, randomly sampling a set T1 from the historical query set T, the set T1 containing k vectors, calculating the Wasserstein distance d of set T1 and set V1 w (T1,V1), calculating the Wasserstein distance d of set T1 and set V2 w (T1,V2);

[0037] S3, the difference coefficient differ is calculated according to the following formula:

[0038]

[0039] Wherein, alpha is an adjustable coefficient.

[0040] Compared with the prior art, the present application has the beneficial effects that: the present application proposes a vector database system based on query distribution perception, which is composed of an index construction module, a vector retrieval module, an interval measurement module, a log recording module, a distribution perception module, a distribution detection module, a distribution mapping module, a storage module, an interface agent module and a graphical management interface. Through the index construction, vector retrieval and storage module, the vector similarity calculation and data persistent storage problem is solved; through the log recording module and the distribution perception module, the user's query history is recorded, the distribution change of the user query is perceived in time, the index structure in the vector database is dynamically adjusted, and the retrieval quality of the user's same distribution and different distribution query is guaranteed; through the distribution detection module, the difference coefficient is used to intuitively measure the distribution of the user's historical query record and the vector in the library; through the distribution mapping module, the distribution of the search target set and the user's historical query record is fused, the search path during retrieval is effectively reduced, the retrieval efficiency and recall rate are improved, and the index construction method does not need to be changed.

[0041] And the existing vector database is usually based on the assumption that the query vector and the vector in the database follow the same distribution, once the query vector and the database vector distribution are different, the query performance will be sharply decreased. The present application provides an efficient solution for the storage and retrieval of vector data, at the same time, the distribution of the user query is monitored in real time, the high delay, low recall rate and other problems caused by different distribution user query are effectively solved on the basis of ensuring the quality of the same distribution user query, and the stability and reliability of the query quality are guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings required to be used in the specific embodiments or prior art description will be briefly introduced below, and some specific embodiments of the present application will be described in detail below with reference to the drawings in an exemplary but non-limiting manner. The same reference signs in the drawings indicate the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0043] Figure 1 is a system architecture diagram of the present application,

[0044] Figure 2 is an execution flow diagram of the present application. The embodiments of the present application are as follows: DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] In order to make the technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings. Figure 1 is a system architecture diagram of the present application, Figure 2 is an execution flow diagram of the present application. The embodiments of the present application are as follows:

[0047] The embodiment of the present application proposes a vector database system based on query distribution perception, which includes an index construction module, a vector retrieval module, an interval measurement module, a log recording module, a distribution perception module, a distribution detection module, a distribution mapping module, a storage module, an interface agent module and a graphical management interface.

[0048] The index construction module is used to construct an index for a retrieval target set V={v1, v2…v n} and record the index name, wherein n is the number of vector in the retrieval target set V; the retrieval target set V is composed of the base set in the Laion data set; the index name is input by the user;

[0049] The method of constructing the index includes a graph-based method, a tree-based method and a quantization-based method, which are supported by the Faiss and Annoy open source libraries;

[0050] The vector retrieval module is used to receive a query data set Q={q1, q2…q m} and query index name, m is the number of vectors in the query dataset Q; the query dataset Q is composed of the query set in the Laion dataset, which belongs to a different modality from the retrieval target set V and has a certain distribution difference; for each element in the query dataset Q, the index is queried according to the query index name to obtain the top-k approximate nearest neighbor retrieval results;

[0051] The interval measurement module is used to traverse each vector v in the retrieval target set V. i Calculate vector v i Distance d from the target set V Mi This forms the distance set U(d) M After traversing the target set V, record the distribution interval [low, high] of the target set V, which represents the degree of distribution shift of the target set V.

[0052] The distance d Mi The calculation method is as follows:

[0053]

[0054] Where μ represents the mean vector of the target set V, and S is the covariance matrix of the target set V; the distance d Mi It reflects the vector v i The degree of distribution deviation in the target set V;

[0055] In the distribution interval [low, high], low represents the lower limit of the distribution offset of the retrieval target set V, which is calculated as follows:

[0056] low = quantile(U(d) M ),0.05)

[0057] In the distribution interval [low, high], hiht represents the upper limit of the distribution offset of the retrieval target set V, which is calculated as follows:

[0058] high = quantile(U(d) M ),1-0.95)

[0059] Where quantile(set, percent) represents the percentile value of the elements in set set after sorting them from smallest to largest;

[0060] The log recording module is used to record query logs, which include query vector q. i Query index name, query vector q iThe distribution distance distance of the retrieval target set V, and the number of out-of-distribution queries h; the number of out-of-distribution queries h represents the number of queries that are inconsistent with the distribution of the retrieval target set V;

[0061] The distribution perception module is configured to calculate the query vector q i The distribution distance distance of the retrieval target set V, and the number of out-of-distribution queries h,

[0062] The calculation method of the distribution distance distance is as follows:

[0063]

[0064] Wherein, μ represents the mean vector of the retrieval target set V, and S is the covariance matrix of the retrieval target set V;

[0065] The calculation method of the number of out-of-distribution queries h is that the initial value of h is 0, and h is increased by 1 when the distribution distance distance is less than the distribution interval low or greater than the distribution interval high;

[0066] It is judged whether the number of out-of-distribution queries h is greater than a threshold threshold, in the embodiment, the threshold threshold is n*0.3, if it is less than or equal to the threshold threshold, the query log is recorded by using the log recording module, if it is greater than the threshold threshold, the corresponding query vector q in the log recording module is obtained from the log recording module according to the query index name i As t i , the historical query set T = {t1, t2…t l} is constituted, and the following processing is performed by using the distribution detection module:

[0067] The distribution detection module is configured to calculate the distribution difference coefficient differ between the historical query set T and the retrieval target set V, and to determine whether the distribution of the retrieval target set V needs to be adjusted by the historical query set T. When the difference coefficient differ is greater than 0, a distribution optimization index is constructed by the distribution mapping module, the number of out-of-distribution queries h is updated to the initial value 0, and the distribution interval [low, high] is updated according to the historical query set T; when the difference coefficient differ is less than or equal to 0, the existing index is not adjusted;

[0068] The calculation method of the distribution difference coefficient includes the following steps:

[0069] S1, two disjoint sets V1 and V2 are sampled from the retrieval target set V, each of the sets contains k vectors, and the Wasserstein distance d w (V1, V2) of the sets V1 and V2 is calculated.

[0070] S2, randomly sample a set T1 from the historical query set T, the set T1 contains k vectors, calculate the Wasserstein distance d w (T1, V1), calculate the Wasserstein distance d w (T1, V2) of set T1 and set V2;

[0071] S3, the difference coefficient differ is calculated as follows:

[0072]

[0073] Wherein, a is an adjustable coefficient, and a is 0.125 in the embodiment; the Wasserstein distance can effectively capture the spatial distribution relationship between the vector sets, so as to judge the distribution difference between the vector sets;

[0074] The distribution mapping module is used to construct a distribution optimization index for the retrieval target set V based on the historical query set T and the retrieval target set V.

[0075] The construction of the distribution optimization index includes the following steps:

[0076] a, brute force search each vector v in the retrieval target set V i In the historical query set T, j accurate nearest neighbors {z1, z2…z j}, calculate the mean vector In the embodiment, j is 7; the mean vector z iavg is consistent with the distribution characteristics of the historical query set T, and is similar to the vector v i , used to represent the distribution of the vector v i in the historical query set T;

[0077] b, calculate the mapping set F = {f1, f2…f n} of the retrieval target set V, f i in the mapping set F corresponds to v i in the retrieval target set V one by one;

[0078] The calculation formula of the mapping set F of the retrieval target set V is as follows:

[0079] F = V + βZ

[0080] Wherein, Z = {z 1avg ,z 2avg …z navg}, the mean vector z iavgThe distribution of the set Z is consistent with the historical query set T, and contains vector information in the retrieval target set V; β is an adjustable fusion coefficient for adjusting the weight, and in the embodiment, the value is 0.67; the mapping set F contains the distribution characteristics of the retrieval target set V and the historical query set T, and for the top-k retrieval results in the retrieval target set V returned by each query in the historical query set T, the distance between the mapping vectors corresponding to the retrieval results is closer, so that it is easier to connect with each other when the index is constructed, thereby improving the retrieval efficiency and recall rate;

[0081] c, inputting the mapping set F into the index construction module, constructing the index, and after the construction is completed, replacing the data in the index by the mapping set F with the retrieval target set V, that is, replacing F={f1, f2…f n} with V={v1, v2…v n}, f i and v i correspond to each other; at the same time, the structure information of the index is retained, the structure information contains the distribution information of the retrieval target set V and the historical query set T, and the retrieval step number between the retrieval target vectors in the index is shortened, and the mapping vector cannot be used to replace the original vector for retrieval distance and return retrieval results, so it is necessary to replace the original vector; using the log recording module to record the query log;

[0082] The storage module is used for persistently storing the retrieval target set V, the index data and the query log in the form of a file;

[0083] The graphical management interface provides an operable user interface, and visualizes the vector database;

[0084] The interface agent module is used for receiving a request of the graphical management interface and forwarding to a corresponding module for execution;

[0085] The above is only part of the specific embodiments of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A query distribution-aware vector database system, characterized in that, It includes an index building module, a vector retrieval module, an interval measurement module, a log recording module, a distribution awareness module, a distribution detection module, and a distribution mapping module; The index building module is used to: index the target set V = {v1, v2, ... v} n } Construct an index and record the index name, where n is the number of vectors in the target set V to be retrieved; The vector retrieval module is used to: receive the user-input query dataset Q = {q1, q2, ..., q} m } and query index name, m is the number of vectors in the query dataset Q; for each element in the query dataset Q, query the index according to the query index name to obtain the top-k approximate nearest neighbor search results; The interval measurement module is used to: traverse each vector v in the retrieval target set V. i Calculate vector v i Distance d from the target set V Mi This forms the distance set U(d) M After traversing the target set V, record the distribution range of the target set V, which represents the degree of distribution offset of the target set V. The log recording module is used to: record query logs, the query logs including query vector q. i Query index name, query vector q i The distribution distance from the target set V, and the number of out-of-distribution queries, h; the number of out-of-distribution queries, h, represents the number of queries whose query vectors are inconsistent with the distribution of the target set V. The distributed sensing module is used to: calculate the query vector q i The distribution distance (distance) between the target set V and the target set V is used to update the number of out-of-distribution queries (h). It is then determined whether the number of out-of-distribution queries (h) is greater than a threshold. If it is less than or equal to the threshold, the query log is recorded using the logging module. If it is greater than the threshold, the corresponding query vector (q) is retrieved from the logging module based on the query index name. i As t i This constitutes the historical query set T = {t1, t2, ..., t} l The distribution detection module is used to perform the following processing: The distribution detection module is used to calculate the distribution difference coefficient differ between the historical query set T and the retrieval target set V, and determine whether the distribution of the retrieval target set V needs to be adjusted through the historical query set T. When the difference coefficient differ is greater than 0, the distribution mapping module constructs a distribution optimization index, updates the number of out-of-distribution queries h to the initial value 0, and updates the distribution range of the retrieval target set V according to the historical query set T. When the difference coefficient differ is less than or equal to 0, the existing index is not adjusted. The distribution mapping module is used to construct a distribution-optimized index for the retrieval target set V based on the historical query set T and the retrieval target set V; The construction of the distributed optimized index includes the following steps: a) Brute-force search retrieves each vector v in the target set V. i In the historical query set T, find the j exact nearest neighbors {z1, z2, ... zj} j }, calculate the mean vector b. Calculate the mapping set F = {f1, f2, ..., f} of the retrieval target set V. n }, f in the mapping set F i With calculation of v in the target set V i One-to-one correspondence; The formula for calculating the mapping set F of the target set V is as follows: F=V+βZ Where Z = {z 1avg ,z 2avg …z navg }, by the mean vector z iavg Composition; β is an adjustable fusion coefficient; c. Input the mapping set F into the index building module to build the index. After the index is built, replace the data part of the index with the retrieval target set V. At the same time, retain the structural information of the index, which includes the distribution information of the retrieval target set V and the historical query set T, and shorten the retrieval steps between the retrieval target vectors in the index. Use the log recording module to record the query log.

2. The query distribution-aware vector database system as described in claim 1, characterized in that, It also includes a storage module, an interface proxy module, and a graphical management interface; The storage module is used to persistently store the retrieval target set V, index data, and query logs; The graphical management interface provides an operable user interface and visualizes the operation of the vector database; The interface proxy module is used to receive requests from the graphical management interface and forward them to the corresponding module for execution.

3. The query distribution-aware vector database system as described in claim 1, characterized in that, In the interval measurement module, the distance d Mi The calculation method is as follows: Where μ represents the mean vector of the search target set V, and S is the covariance matrix of the search target set V; The distribution interval of the search target set V is denoted as [low, high]; low represents the lower limit of the distribution offset of the search target set V, and its calculation method is as follows: low=quantile(U(d M ),percent) high represents the upper limit of the distribution offset of the target set V, and its calculation method is as follows: high=quantile(U(d M ),1-percent) Here, quantile(set,percent) represents the percentile value of the elements in set set after sorting them from smallest to largest.

4. The query distribution-aware vector database system as described in claim 3, characterized in that, The method for calculating the distribution distance is as follows: Where μ represents the mean vector of the search target set V, and S is the covariance matrix of the search target set V; The method for calculating the number of out-of-distribution queries h includes: h is initially 0, and h is incremented by 1 when the distribution distance is less than the distribution interval low or greater than the distribution interval high.

5. A query distribution-aware vector database system as described in claim 4, characterized in that, The calculation method for the distribution difference coefficient includes the following steps: S1, sample two disjoint sets V1 and V2 from the target set V, each set containing k vectors, and calculate the Wasserstein distance d between set V1 and set V2. w (V1,V2); S2, randomly sample a set T1 from the historical query set T, where set T1 contains k vectors, and calculate the Wasserstein distance d between set T1 and set V1. w Given sets (T1, V1), calculate the Wasserstein distance d between sets T1 and V2. w (T1,V2); S3, the formula for calculating the difference coefficient is as follows: Where α is an adjustable coefficient.

Images