Vector query method of vector database and related product

Abstract

The invention provides a vector query method of a vector database and a related product. The method comprises a step of constructing a reverse index and a step of carrying out vector query based on the reverse index. In the step of establishing the reverse index, preprocessing each data vector to reduce the dimension of each data vector, and then clustering; in the vector query step based on the inverted index, the vector to be queried is preprocessed to reduce the dimension of the vector to be queried, and then the neighbor node of the vector to be queried is obtained. And finally, restoring the vector to be queried and each neighbor vector, and sorting each neighbor vector according to the distance between the vector to be queried and each neighbor vector. According to the technical scheme, the vector query accuracy of the vector database can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a vector query method for a vector database and related products. Background Art

[0002] The working principle of the vector database is to convert unstructured data such as images and text into structured data vectors and store the data vectors. The method for executing data query in the vector database includes: after receiving the query instruction of the user's image or text, the image or text to be queried will also be converted into a corresponding data vector, and the data vector will be used as the vector to be queried; then the data vector closest to the vector to be queried will be found from the vector database and returned to the user. Compared with other types of databases, vector databases can improve the work efficiency of executing data queries.

[0003] In the process of data query in the vector database in the prior art, the query method based on the IVF (Inverted File Index) algorithm is a commonly used vector query method. This query method needs to cluster each vector in the vector database to obtain multiple clusters, and obtain the centroid vector of each cluster to establish an inverted index. Then, after receiving the vector query instruction, the distance between each centroid vector and the query vector is calculated to obtain the centroid vector with the smallest distance to the query vector, and the neighbor vector of the query vector is found in the cluster corresponding to the centroid vector. For data vectors with higher dimensions, the query method based on the IVF algorithm will occupy a large amount of storage space for the constructed inverted index, and in the process of using the inverted index query, a large amount of calculations are required to obtain the neighbor vectors of the vector to be queried, so there is a problem of low work efficiency.

[0004] In order to improve the working efficiency of vector query, before clustering the vectors in the vector database, the data vectors in the vector database can be preprocessed to reduce the dimension of each data vector, for example, the dimension of each data vector in the vector database can be compressed by using the product quantization method or the scalar quantization method, so as to reduce the storage space occupied by the constructed inverted index. And before performing the vector query, the dimension of the vector to be queried is compressed to reduce the amount of data calculation in the query process and improve the working efficiency of the vector query. However, after compressing the data vector and the vector to be queried, a certain degree of distortion will be caused in the data vector and the vector to be queried, which may lead to certain errors in the query results. Summary of the invention

[0005] The purpose of the present invention is to provide a vector query method and related products for a vector database, so as to solve the problem that in the process of performing vector query in the vector database in the prior art, the dimension of the data vector is compressed in the process of building an inverted index, resulting in inaccurate query results.

[0006] Furthermore, another object of the present invention is to expand the search range during the vector query process to reduce the impact of edge effects on the accuracy of the query results.

[0007] Furthermore, another object of the present invention is to enable the vector query process to automatically adjust the search range according to the distribution of each data vector in the vector database to further improve the accuracy of the vector query result.

[0008] In a first aspect, the present invention provides a vector query method for a vector database, comprising:

[0009] Obtaining an inverted index of the vector database, the inverted index including a plurality of clusters obtained by clustering the data vectors after dimensionality reduction processing, and a centroid vector of each of the clusters;

[0010] Obtaining a vector to be queried, and performing dimensionality reduction processing on the vector to be queried;

[0011] Calculating the distance between the query vector and each of the centroid vectors, taking a first set number of clusters with the closest distance between the centroid vector and the query vector as target clusters, and taking the data vectors in each of the target clusters as target data vectors;

[0012] Calculating the distance between each of the target vectors and the vector to be queried, and taking a second set number of target data vectors closest to the vector to be queried as neighbor vectors of the vector to be queried;

[0013] Restoring the dimensions of the query vector and each of the neighbor vectors, and calculating the distance between each of the neighbor vectors and the query vector;

[0014] The neighbor vectors are sorted according to the distances between the neighbor vectors and the vector to be queried.

[0015] Furthermore, after the step of calculating the distance between the query vector and each of the centroid vectors, the method further includes:

[0016] Obtaining the predicted number of the target clusters, and determining whether the predicted number is greater than the first set number;

[0017] If yes, the predicted number of clusters with the closest distance between the centroid vector and the query vector are taken as the target clusters;

[0018] If not, the step of taking the first set number of clusters with the shortest distance between the centroid vector and the query vector as the target cluster is executed.

[0019] Furthermore, the step of obtaining the predicted number of the target clusters includes:

[0020] A pre-trained gradient boosting decision tree model is obtained, and the gradient boosting decision tree model is used to obtain the predicted quantity at least according to the distance between each of the data vectors and the query vector.

[0021] Furthermore, the step of performing dimensionality reduction processing on the vector to be queried includes:

[0022] The vector to be queried is processed by product quantization method to reduce the dimension of the vector to be queried.

[0023] Furthermore, the step of restoring the dimensions of the query vector and each of the neighbor vectors includes:

[0024] The step of restoring the dimensions of the query vector and each of the neighbor vectors includes:

[0025] Obtaining index items of the vector to be queried and each of the neighbor vectors, and obtaining a vector pointer in each of the vectors;

[0026] The vector data corresponding to each of the vector pointers are read respectively to obtain the query vector and each of the neighbor vectors after dimension restoration.

[0027] Furthermore, the step of using a second set number of target data vectors that are closest to the vector to be queried as neighbor vectors of the vector to be queried includes:

[0028] Creating a data pile, and storing a second set number of the target data vectors having the smallest distances from the vector to be queried in the data pile;

[0029] The target data vector in the data pile is used as the neighbor vector.

[0030] Furthermore, after the step of sorting the neighbor vectors according to the distances between the neighbor vectors and the query vector, the method further includes:

[0031] A neighbor vector list of the vector to be queried is generated according to each of the neighbor vectors.

[0032] Furthermore, after the step of obtaining the vector to be queried after the dimension reduction process, the method further includes:

[0033] The configuration data of the database is obtained, and the values ​​of the first set quantity and the second set quantity are obtained according to the configuration data.

[0034] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned vector query methods are implemented.

[0035] In a third aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned vector query methods.

[0036] In the technical solution provided by the present invention, the inverted index of the vector database is an inverted index constructed based on the data vector after dimensionality reduction processing, so the storage space occupied by the constructed inverted index can be reduced; and in the process of querying based on the constructed inverted index, the amount of calculation of the distance between the vector to be queried and the centroid vector of each cluster can be reduced. On the other hand, the technical solution of the present invention adopts a two-stage query method to query the query vector, in which the neighbor vectors of the vector to be queried are obtained in the first stage, and the dimensions of the vector to be queried and the neighbor vectors of the vector to be queried are restored in the second stage, and the distance between the restored vector to be queried and its neighbor vectors is calculated, and the neighbor vectors of the vector to be queried are re-sorted according to the distance, so that the accuracy of the query of the vector to be queried can be improved.

[0037] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0039] Figure 1 is a schematic flow chart of a vector query method for a vector database according to an embodiment of the present invention;

[0040] Figure 2 is a schematic flow chart of constructing an inverted index of a vector database in a vector query method according to an embodiment of the present invention;

[0041] Figure 3 is a schematic flow chart of performing vector query based on a constructed inverted index in a vector query method according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of performing vector query based on an inverted index according to an embodiment of the present invention;

[0043] Figure 5 is a schematic flow chart of performing vector query based on the constructed inverted index in a vector query method according to another embodiment of the present invention;

[0044] Figure 6 is a schematic flow chart of restoring the dimensions of a vector to be queried and a neighbor vector of the vector to be queried in a vector query method according to an embodiment of the present invention;

[0045] Figure 7 is a schematic diagram of a computer program product according to an embodiment of the present invention; and

[0046] Figure 8 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] Refer to the following Figures 1 to 8 To describe a vector query method and related products of a vector database of an embodiment of the present invention. In the description of this embodiment, it should be understood that the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features, that is, include one or more of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. When a feature "includes or contains" one or some of the features it covers, unless otherwise specifically described, this indicates that other features are not excluded and may further include other features.

[0048] See also Figure 1 , Figure 1 The present invention is a schematic flow chart of a vector query method for a vector database according to an embodiment of the present invention, wherein the inverted index of the vector database includes a plurality of clusters obtained by clustering the data vectors after dimensionality reduction processing, and the centroid vectors of each cluster, thereby reducing the storage space occupied by the constructed inverted index and improving the working efficiency of the vector query. In addition, the vector query method of the present embodiment is a two-stage query method, which can improve the accuracy of the vector query.

[0049] exist Figure 1 In the process shown, the vector query method of the vector database of the present invention may generally include the following steps:

[0050] Step S102: obtaining an inverted index of a vector database;

[0051] Step S104: performing a vector query based on the obtained inverted index.

[0052] The inverted index of the vector database obtained in the above step S102 includes multiple clusters obtained by clustering the data vectors after dimensionality reduction processing, and the centroid vector of each cluster. Since the inverted index in this embodiment is obtained by clustering the data vectors after dimensionality reduction processing, the storage space occupied by the inverted index can be reduced.

[0053] In this embodiment, the method for constructing the inverted index of the vector database is as follows: Figure 2 As shown, the following steps are included:

[0054] Step S112: obtaining a data set of a vector database, wherein the data set includes a plurality of data vectors;

[0055] Step S114: performing dimensionality reduction processing on each data vector in the data set;

[0056] Step S116: clustering each data vector in the data set to obtain a plurality of clusters;

[0057] Step S118: Calculate the centroid vector of each cluster to construct an inverted index of the vector database.

[0058] In this embodiment, in step S116, a k-means clustering algorithm may be used to cluster each data vector in the data set so as to assign each data vector to a corresponding cluster, specifically including:

[0059] First, s initial centroid vectors are selected from each data vector of the data set, where s is a positive integer greater than 1.

[0060] In this embodiment, the method for selecting s initial centroid vectors includes: first randomly selecting a data vector from the data set as the first initial centroid vector, and then calculating the distance between the remaining data vectors and the first initial centroid vector, and taking the data vector with the farthest distance from the first initial centroid vector as the second initial centroid vector; then, from the remaining data vectors, selecting the data vector with the farthest distance from the existing initial centroid vector as the new centroid vector, and so on, until s initial centroid vectors are obtained.

[0061] Then, each data vector is clustered according to each initial centroid vector to obtain s clusters. Taking one of the data vectors as an example, the method for clustering the data vector includes: calculating the distance between the data vector and each initial centroid vector, and clustering the data vector into the cluster corresponding to the initial centroid vector closest to the data vector. And so on, until all data vectors are clustered.

[0062] Finally, each cluster is updated until the centroid vector of each cluster no longer changes, or the number of clustering times for each data vector reaches a set threshold.

[0063] In this embodiment, the method for updating each clustering cluster includes: calculating the centroid vector of each clustering cluster, and then re-clustering each data vector according to the calculated centroid vector. Among them, the average vector of each data vector in the clustering cluster can be used as the centroid vector of the corresponding clustering cluster. If the change of the centroid vector of each clustering cluster is within the preset change threshold after two consecutive clustering, it is determined that the centroid vector of each clustering cluster no longer changes, and clustering is not continued, and it is determined that the clustering of each data vector is completed.

[0064] After completing the clustering of each data vector in the data set, k clusters can be obtained, and then the average vector of the data vector in each cluster is calculated respectively, and each average vector is used as the centroid vector of the corresponding cluster.

[0065] In this embodiment, the method for performing vector query based on the constructed inverted index in step S104 is specifically as follows: Figure 3 As shown, the following steps are included:

[0066] Step S122: obtaining a vector to be queried, and performing dimensionality reduction processing on the vector to be queried;

[0067] Step S124: Calculate the distance between the query vector and the centroid vector of each cluster;

[0068] Step S126: taking a first set number of clusters whose centroid vectors are closest to the query vector as target clusters;

[0069] Step S128: taking the data vectors in each target cluster as the target data vector, and calculating the distance between each target data vector and the query vector;

[0070] Step S130: taking a second set number of target data vectors that are closest to the vector to be queried as neighbor vectors of the vector to be queried;

[0071] Step S132: restore the dimensions of the query vector and the neighbor vectors of the query vector, and calculate the distance between the query vector and each of its neighbor vectors;

[0072] Step S134: sorting neighbor vectors of the query vector according to their distances from the query vector.

[0073] According to the above content, the method for performing vector query based on the constructed inverted index in this embodiment is a two-stage query method, in which the first stage finds the second set number of neighbor vectors closest to the query vector through the dimensionality reduction processing of the query vector, and the neighbor vector is also a data vector after dimensionality reduction processing. Then in the second stage, the dimensions of the query vector and the neighbor vectors of the query vector are restored, and the distance between the restored query vector and its neighbor vectors is calculated, and the neighbor vectors of the query vector are re-sorted according to the distance to ensure the accuracy of the vector query.

[0074] Taking an application scenario as an example, assuming that the query vector obtained in the above step S122 is q, after the dimensionality reduction processing of the query vector q, the low-dimensional query vector obtained is q', the above-mentioned first set number is m, and the second set number is k. Therefore, in this embodiment, the m clusters with the closest distance between the centroid vector and the query vector q' are used as target clusters, and the k data vectors with the closest distance to the query vector q' are selected from the m target clusters as the neighbor vectors of the query vector q', completing the first stage of the vector query method.

[0075] Assume that the k neighbor vectors of the query vector q′ obtained in the first stage are the top-k data vectors. These top-k data vectors are compressed data vectors. Therefore, in the second stage, the query vector q′ is restored to the query vector q, and the dimensions of the top-k data vectors obtained in the first stage are restored. Then, the distances between the restored top-k data vectors and the query vector q are calculated respectively, and the top-k data vectors are arranged in ascending order according to the distances.

[0076] In summary, in the technical solution provided by this embodiment, the inverted index of the vector database is an inverted index constructed based on the data vector after dimensionality reduction processing, so the storage space occupied by the constructed inverted index can be reduced, and it is more suitable for high-dimensional or large-volume data vectors, and in the process of querying according to the constructed inverted index, the amount of calculation of the distance between the vector to be queried and the centroid vector of each cluster can be reduced. In addition, the technical solution of this embodiment adopts a two-stage query method to query the query vector, in which the neighbor vectors of the vector to be queried are obtained in the first stage, and the dimensions of the vector to be queried and the neighbor vectors of the vector to be queried are restored in the second stage, and the distance between the restored vector to be queried and its neighbor vectors is calculated, and the neighbor vectors of the query vector are re-sorted according to the distance, so that the accuracy of the query vector to be queried can be improved.

[0077] When using the constructed inverted index to perform vector query, the query results obtained by the vector query may be inaccurate due to edge effects. Figure 4 Taking the application scenario shown as an example, each circular point represents a data vector in the vector database, a star-shaped point represents a vector to be queried, a dotted circle represents a cluster, and a square point represents the centroid vector of the corresponding cluster. Figure 4 The query vector belongs to the first cluster, so the first cluster should be used as the target cluster of the query vector, and the neighbor vector of the query vector should be searched in the first cluster; however, since the query vector is at the edge of the first cluster, and the distance between the query vector and the centroid vector of the second cluster is smaller than the distance between the query vector and the execution vector of the first cluster, in the existing vector query method, the second cluster will be used as the target cluster of the query vector, and the neighbor vector of the query vector will be searched in the second cluster, which will cause the problem of inaccurate query results.

[0078] In order to solve the problem of inaccurate query results caused by edge effects, in some embodiments of the present invention, the query method of the vector database of the present invention is as follows: Figure 5 As shown, after calculating the distance between the query vector and the centroid vector of each cluster in the above step S124, the following is further included:

[0079] Step S136: obtaining the predicted number of target clusters, and determining whether the predicted number is greater than a first set number;

[0080] If yes, execute step S138; if no, execute step S126;

[0081] Step S138: taking the predicted number of clusters with the closest distance between the centroid vector and the query vector as the target cluster.

[0082] In this embodiment, the predicted number of target clusters probes is a hyperparameter. The setting of this hyperparameter can make the number of target clusters obtained by the database during the vector query process not less than the predicted number, thereby expanding the search range of the neighbor vectors of the query vector, reducing the influence of the edge effect on the vector query result, and achieving the purpose of improving the accuracy of the vector query.

[0083] In this embodiment, after obtaining the k neighbor vectors of the vector q′ to be queried in the above step S130, a data pile can be created first, and the k neighbor vectors of the vector q′ to be queried can be stored in the data pile; then it is determined whether the predicted number probes of the target clusters is greater than the first set number m; if not, step S312 is executed; if greater, the m+1th target cluster to the probesth target cluster are selected according to the distance between each centroid vector and the vector to be queried, and multiple data vectors with the closest distance to the vector q′ to be queried are selected from the m+1th target cluster to the probesth target cluster, and the selected data vectors are stored in the established data pile, so that the number of data vectors in the data pile is h. Then the k data vectors with the closest distance to the vector q′ to be queried in the data pile are used as the neighbor vectors of the vector q′ to be queried.

[0084] Still Figure 4 Taking the application scenario shown as an example, if the predicted number of target clusters is set to 2, the first cluster and the second cluster will be used as target clusters at the same time to find the neighbor vectors of the query vector in the first cluster and the second cluster, thereby improving the accuracy of the query result.

[0085] In some embodiments of the present invention, in the above step S136, the method for obtaining the predicted number of target clusters includes:

[0086] First, obtain the pre-trained gradient boosting decision tree model;

[0087] Then, the obtained gradient boosting decision tree model is used to obtain the predicted number of target clusters according to the distance between each data vector and the query vector.

[0088] The above-mentioned gradient boosting decision tree model is an algorithm model based on ensemble learning, which achieves better prediction performance by constructing multiple decision tree models and combining the decision tree models together. In this embodiment, the gradient boosting decision tree model can be trained using a preset training data set to obtain a pre-trained gradient boosting decision tree model.

[0089] In this embodiment, the input data of the gradient boosting decision tree model can be constructed according to the distance between each data vector and the query vector, including:

[0090] First, obtain the nearest centroid vector to the query vector, calculate the distance between the centroid vector and the third set number of centroid vectors, and use the distance as the first input data of the gradient boosting decision tree model; for example, assuming that the third set number is X, and the centroid vector closest to the query vector q′ is the Yth centroid vector, then use the distance between the Xth centroid vector and the Yth centroid vector as the first input data of the gradient boosting decision tree model;

[0091] Then, a data vector closest to the query vector is obtained, and the data vector is used as the second input data of the gradient boosting decision tree model;

[0092] Next, a fourth set number of data vectors closest to the query vector is obtained, for example, the 10th data vector closest to the query vector, and the data vector is used as the third input data of the gradient boosting decision tree model;

[0093] Finally, the distance between the second input data and the third input data is used as the fourth input data of the gradient boosting decision tree model, the distance between the second input data and the third set number of centroid vectors is used as the fifth input data of the gradient boosting decision tree model, and the query vector is used as the sixth input data of the gradient boosting decision tree model.

[0094] After obtaining the above input data, the gradient boosting decision tree model can automatically obtain the predicted number of target clusters according to the above input data, so that the predicted number matches the distribution of each data vector.

[0095] Through the technical solution of this embodiment, a pre-trained gradient boosting decision tree model can be used to adaptively obtain the predicted number of target clusters according to the distance between each data vector and the vector to be queried, and then in the process of vector query, the end condition of the query can be adaptively obtained according to the distribution of data vectors in each cluster. Therefore, the technical solution of this embodiment can solve the problem that the predicted number of target clusters cannot be accurately set because the user is not clear about the distribution of data vectors, and achieve the purpose of improving the accuracy and reliability of vector query.

[0096] In some embodiments of the present invention, the method of performing dimensionality reduction processing on the query vector in step S122 includes:

[0097] The product quantization method is used to perform operation processing on the query vector to reduce the dimension of the query vector.

[0098] Accordingly, the method of performing dimensionality reduction processing on each vector in the data set in the above step S114 includes:

[0099] The product quantization method is used to operate on each data vector in the data set to reduce the dimension of each data vector.

[0100] The above-mentioned product quantization method is an algorithm for accelerating the retrieval speed, and its core idea is segmentation and clustering. In this embodiment, taking the query vector as an example, the method of using the product quantization method to reduce the dimension of the query vector includes:

[0101] First, a sample vector set is obtained, where the sample vector set includes multiple sample vectors and the dimension of each sample vector is D.

[0102] Then, the dimensions of each sample vector in the sample vector set are spatially segmented. Assuming that the dimensions of each sample vector are segmented into M spaces, and the dimensions in each space are the same, the number of dimensions d in each space is D / M, that is, there are D / M sub-vectors in each space.

[0103] Next, each space is clustered respectively. For example, a k-means algorithm may be used to cluster each space respectively to obtain a spatial cluster corresponding to each space, and obtain a centroid vector of each spatial cluster and an ID of the centroid vector.

[0104] Since the dimension of each sample vector is divided into M spaces in this embodiment, after clustering each space, each space corresponds to multiple spatial clustering clusters. In the process of dimensionality reduction processing of the query vector, the dimension of the query vector is first divided into M spaces, and then the centroid vector corresponding to each space is calculated, and the ID of the centroid vector is used to represent the corresponding sub-vector, thereby obtaining M sub-vectors of the query vector, so as to compress the dimension of the query vector from D to M.

[0105] After the data vectors are compressed, in the process of building the inverted index, a symmetric distance algorithm can be used to calculate the distance between the data vectors to cluster the data vectors, thereby improving the efficiency of clustering the data vectors. Similarly, after the query vector is compressed, a symmetric distance algorithm can be used to calculate the distance between the query vector and the centroid vector in the process of querying the query vector to improve the efficiency of vector query.

[0106] Taking two data vectors as an example, in the process of building an inverted index, the method of using a symmetric distance algorithm to calculate the distance between the two data vectors includes:

[0107] First, the distance between each corresponding sub-vector in the two data vectors is obtained by querying the distance table. Since each sub-vector in the compressed data vector is the ID of the corresponding centroid vector, the distance between the centroid vectors of each spatial clustering cluster can be calculated first in this embodiment, and each distance is stored in the distance query table according to the ID of the corresponding centroid vector, so as to query the distance between each sub-vector.

[0108] Then, the distance between the two data vectors is obtained by adding the distances between the corresponding sub-vectors in the two data vectors.

[0109] In the technical solution of this embodiment, the product quantization method is used to preprocess the data vectors of the database and the vector to be queried. Compared with other methods such as the scalar quantization method, it can improve the compression of each data vector, reduce the storage space occupied by the inverted index, and reduce the amount of data calculation in the query process, thereby improving the work efficiency of the query vector query.

[0110] In some embodiments of the present invention, the method for restoring the dimensions of the query vector and the neighbor vectors of the query vector in step S132 is as follows: Figure 6 As shown, the following steps are included:

[0111] Step S142: obtaining index items of the vector to be queried and each neighboring vector of the vector to be queried, and obtaining a vector pointer in each index item;

[0112] Step S144: respectively read the vector data corresponding to the vector pointers in the index items of the query vector and each neighbor vector of the query vector to obtain the query vector and each neighbor vector of the query vector after dimension restoration.

[0113] In this embodiment, after the dimensionality reduction processing is performed on each data vector in the data set in the above step S114, the vector pointers of each data vector before the dimensionality reduction processing are also stored in the index items of the corresponding data vectors; correspondingly, after the dimensionality reduction processing is performed on the query vector in the above step S122, the vector pointers of the query vector before the dimensionality reduction processing are also stored in the index items of the query vector.

[0114] In the above step S144, the storage address of the query vector before dimensionality reduction in the disk can be obtained according to the vector pointer in the index item of the query vector, and then the restored query vector can be obtained by reading the data vector stored in the storage address.

[0115] Similarly, the storage addresses of the neighbor vectors of the query vector before dimensionality reduction in the disk can be obtained according to the vector pointers in the index items of the neighbor vectors of the query vector, and then the data vectors stored in each storage address can be read to obtain the restored neighbor vectors of the query vector.

[0116] The technical solution of this embodiment can prevent the loss of pre-processed data vectors and pre-processed query vectors, and can increase the speed of restoring the query vector and its neighbor vectors, thereby achieving the purpose of improving the work efficiency of database vector query.

[0117] In some embodiments of the present invention, the method of using the second set number of target data vectors closest to the query vector as neighbor vectors of the query vector in step S130 includes:

[0118] First, a data pile is constructed, the capacity of the data pile being a second set amount;

[0119] Then, a second set number of target data vectors having the smallest distances from the query vector are stored in the constructed data pile;

[0120] Finally, the target data vector in the data pile is used as the neighbor vector of the query vector.

[0121] In this embodiment, the capacity of the data pile can be k, and then the k target data vectors with the smallest distance to the vector to be queried are stored in the constructed data pile. Therefore, the k data vectors in the data pile are the k neighbor vectors of the vector to be queried.

[0122] Through the technical solution of this embodiment, a data pile can be used to pre-store neighbor vectors of a vector to be queried, thereby improving the convenience of vector query.

[0123] In some embodiments of the present invention, after sorting the neighbor vectors of the query vector in the above step S134, the method further includes:

[0124] Generate a neighbor vector list of the vector to be queried according to the neighbor vector of the vector to be queried.

[0125] In this embodiment, an empty list may be created first, and after obtaining the neighbor vectors of the vector to be queried, the neighbor vectors of the vector to be queried may be stored in the empty list. For example, k neighbor vectors of the vector to be queried may be stored in the empty list in ascending order of distance from the vector to be queried, so as to generate a list containing the neighbor vectors of the vector to be queried, so that the user can intuitively view the query results for the vector to be queried, thereby achieving the purpose of improving user experience.

[0126] In some embodiments of the present invention, before the above step S104, the method further includes:

[0127] The configuration data of the database is acquired, and the values ​​of the first set quantity and the second set quantity are obtained according to the configuration data.

[0128] In this embodiment, the user can adjust the first set number and the second set number by setting the configuration data of the database, wherein the first set number is the number of target clusters, which is a fixed value used for preliminary search in the vector query process, and the value can be set arbitrarily, for example, it can be set to any one of 3, 4, 5, 6, 7, 8, 9, and 10, so the number of clusters to be searched in the query process can be adjusted by setting the configuration parameters. The second set number is the number of neighbor vectors of the vector to be queried obtained, so the number of neighbor vectors of the vector to be queried in the query result can be adjusted by setting the configuration parameters.

[0129] Therefore, through the technical solution of this embodiment, by setting the configuration parameters of the database, the accuracy of the query vector query and the number of query results obtained can be controlled, thereby improving the flexibility and applicability of the database.

[0130] The flow chart provided by the present embodiment is not intended to indicate that the operation of the method will be performed in any particular order, or that all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical ideas provided by the present embodiment method, additional changes can be made to the above method.

[0131] It should be understood that in some embodiments, each part can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0132] This embodiment also provides a computer program product 10 , a computer readable storage medium 20 , and a computer device 30 . Figure 7 is a schematic diagram of a computer program product 10 according to an embodiment of the present invention, Figure 8 is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which implements the steps of any of the above-mentioned vector query methods for a vector database when the computer program 11 is executed by a processor 32. The computer-readable storage medium 20 stores the above-mentioned computer program 11 thereon, which implements the steps of any of the above-mentioned vector query methods for a vector database when the computer program 11 is executed by a processor 32. The computer device 30 may include a memory 31, a processor 32, and a computer program 11 stored in the memory 31 and running on the processor 32.

[0133] The computer program 11 for performing the operation of the present invention may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, a configuration data of an integrated circuit, or a source code or an object code written in any combination of one or more programming languages ​​and process programming languages. The computer program 11 may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider via the Internet). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0134] In the description of this embodiment, the computer program product 10 is a related product including the computer program 11 .

[0135] For the purpose of the description of the present embodiment, the computer readable storage medium 20 is a tangible device capable of retaining and storing the computer program 11, which can be any device that can contain, store, communicate, propagate or use the computer program 11 for an instruction execution system, device or apparatus or in conjunction with these instruction execution systems, devices or apparatuses. More specific examples (a non-exhaustive list) of the computer readable storage medium 20 include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the above.

[0136] At this point, those skilled in the art should recognize that, although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived based on the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized as covering all these other variations or modifications.

Claims

1. A vector query method for a vector database, characterized in that: include: Obtaining an inverted index of the vector database, the inverted index including a plurality of clusters obtained by clustering the data vectors after dimensionality reduction processing, and a centroid vector of each of the clusters; Obtaining a vector to be queried, and performing dimensionality reduction processing on the vector to be queried; Calculating the distance between the query vector and each of the centroid vectors, taking a first set number of clusters with the closest distance between the centroid vector and the query vector as target clusters, and taking the data vectors in each of the target clusters as target data vectors; Calculating the distance between each of the target vectors and the vector to be queried, and taking a second set number of target data vectors closest to the vector to be queried as neighbor vectors of the vector to be queried; Restoring the dimensions of the query vector and each of the neighbor vectors, and calculating the distance between each of the neighbor vectors and the query vector; The neighbor vectors are sorted according to the distance between them and the vector to be queried.

2. The vector query method according to claim 1, characterized in that: After the step of calculating the distance between the vector to be queried and each of the centroid vectors, the method further includes: Obtaining the predicted number of the target clusters, and determining whether the predicted number is greater than the first set number; If yes, the predicted number of clusters with the closest distance between the centroid vector and the query vector are taken as the target clusters; If not, the step of taking the first set number of clusters with the shortest distance between the centroid vector and the query vector as the target cluster is executed.

3. The vector query method according to claim 2, characterized in that: The step of obtaining the predicted number of target clusters includes: A pre-trained gradient boosting decision tree model is obtained, and the gradient boosting decision tree model is used to obtain the predicted quantity at least according to the distance between each of the data vectors and the query vector.

4. The vector query method according to claim 1, characterized in that: The step of performing dimensionality reduction processing on the vector to be queried includes: The vector to be queried is processed by product quantization method to reduce the dimension of the vector to be queried.

5. The vector query method according to claim 1, characterized in that: The step of restoring the dimensions of the query vector and each of the neighbor vectors includes: Obtaining index items of the vector to be queried and each of the neighbor vectors, and obtaining a vector pointer in each of the vectors; The vector data corresponding to each of the vector pointers are read respectively to obtain the query vector and each of the neighbor vectors after dimension restoration.

6. The vector query method according to claim 1, characterized in that: The step of using a second set number of target data vectors closest to the vector to be queried as neighbor vectors of the vector to be queried includes: Creating a data pile, and storing a second set number of the target data vectors having the smallest distances from the vector to be queried in the data pile; The target data vector in the data pile is used as the neighbor vector.

7. The vector query method according to claim 1, characterized in that: After the step of sorting the neighbor vectors according to the distances between the neighbor vectors and the vector to be queried, the method further includes: A neighbor vector list of the vector to be queried is generated according to each of the neighbor vectors.

8. The vector query method according to claim 1, characterized in that: After the step of obtaining the vector to be queried after the dimension reduction process, the method further includes: The configuration data of the database is obtained, and the values ​​of the first set quantity and the second set quantity are obtained according to the configuration data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the vector query method according to any one of claims 1 to 8 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the vector query method according to any one of claims 1 to 8 are implemented.

Images