Data query method and related products of vector database

Abstract

The present invention provides a data query method and related products for a vector database. The method includes: the step of obtaining a query index of the vector database, and the step of performing a vector query based on the query index; the step of obtaining the query index of the vector database includes: obtaining a long-term index of the vector database, the long-term index being an index constructed based on the original data vectors of the vector database; detecting whether the vector database has a newly added data vector; if so, constructing a temporary index of the vector database based on the newly added data vector. The technical solution of the present invention can avoid the problem of consuming too much time in the process of inserting the newly added data vector into the long-term index, and can improve the availability of the vector database compared to the prior art, thereby achieving the purpose of improving the performance of the vector database.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a data query method for a vector database and related products. Background Art

[0002] Vector databases are becoming increasingly important as the need for efficient processing of vector similarity queries continues to increase, such as in the field of artificial intelligence systems led by OpenAI's ChatGPT, and in the field of large language model technology, where deep neural network models are used to convert different types of complex objects (such as images of faces and objects) into data vectors, thereby producing vectors that accurately represent the meaning of text. Vector databases are designed and optimized to process and store high-dimensional vectors, so they are suitable for supporting a variety of applications, such as semantic search, retrieval enhancement generation, recommendation systems, face recognition, and image search.

[0003] The working principle of the vector database is to convert unstructured data such as images and text into structured vector data, and store the vector data in the database. After receiving the user's image or text query instruction, the image or text to be queried will also be converted into vector data, and then the query index will be used to find one or more vector data closest to the vector data from the database and return it to the user, thereby improving the efficiency of the database in executing data queries.

[0004] Query index is an important tool for vector database to perform data query. Common query indexes in the prior art include IVF (Inverted File) index, HNSW (Hierarchical Navigable SmallWorld) graph index, etc. Vector databases generally only build query indexes once. If new data vectors are added to the vector database, the built query index will be updated according to the newly added data vectors. Inserting data vectors into a vector database takes a long time, and the vector database cannot provide external services during the insertion process, which will reduce the availability of the database and affect the performance of the database. Summary of the invention

[0005] An object of the present invention is to provide a data query method and related products for a vector database, which are used to solve the problem of reduced availability of the vector database due to the long time required to insert new data vectors into the query index, so as to achieve the purpose of improving the performance of the vector database.

[0006] In a first aspect, the present invention provides a data query method for a vector database, comprising:

[0007] A step of obtaining a query index of the vector database, and a step of performing a vector query according to the query index;

[0008] The step of obtaining the query index of the vector database includes:

[0009] Acquire a long-term index of the vector database, where the long-term index is an index constructed according to original data vectors of the vector database;

[0010] Detecting whether the vector database has a new data vector;

[0011] If yes, construct a temporary index of the vector database according to the newly added data vector;

[0012] The step of performing a vector query according to the query index includes:

[0013] After receiving the vector to be queried, determining whether the temporary index exists in the query index;

[0014] If so, obtaining a neighbor vector of the vector to be queried according to the long-term index and the temporary index;

[0015] If not, the neighboring vector of the vector to be queried is obtained according to the long-term index.

[0016] Furthermore, before the step of constructing a temporary index of the vector database according to the newly added data vector, the step further includes:

[0017] Determining that the data volume of the newly added data vector is greater than a set data volume threshold;

[0018] If yes, executing the step of constructing a temporary index of the vector database according to the newly added data vector;

[0019] If not, the long-term index is updated according to the newly added data vector.

[0020] Furthermore, after the step of constructing a temporary index of the vector database according to the newly added data vector, the method further includes:

[0021] Detecting whether the load of the vector database is less than a set load threshold;

[0022] If so, the temporary index is merged into the long-term index.

[0023] Furthermore, the step of merging the temporary index into the long-term index includes:

[0024] Inserting each data vector in the temporary index into the long-term index one by one;

[0025] After all the data vectors in the temporary index are inserted into the long-term index, the temporary index is deleted.

[0026] Furthermore, the step of obtaining the long-term index of the vector database includes:

[0027] Constructing a HNSW graph index according to the original data vectors of the vector database, and using the HNSW graph index as the long-term index; and

[0028] The step of constructing a temporary index of the vector database according to the newly added data vector comprises:

[0029] An IVF index is constructed according to the newly added data vector to obtain the temporary index.

[0030] Furthermore, the step of acquiring the neighbor vector of the vector to be queried according to the long-term index and the temporary index includes:

[0031] Acquire a first set number of candidate neighbor vectors of the vector to be queried according to the long-term index, and acquire a second set number of candidate neighbor vectors of the vector to be queried according to the temporary index;

[0032] The nearest neighbor vector is obtained from each of the candidate nearest neighbor vectors.

[0033] Furthermore, the step of obtaining the nearest neighbor vector from each of the candidate nearest neighbor vectors includes:

[0034] Calculating the distance between each candidate neighbor vector and the vector to be queried, and sorting each candidate neighbor vector according to the distance between the candidate neighbor vector and the vector to be queried;

[0035] According to the sorting, a third set number of candidate neighbor vectors having the shortest distances to the query vector are selected as the neighbor vectors.

[0036] Furthermore, before the step of obtaining a third set number of neighbor vectors of the query vector from each of the candidate neighbor vectors, the step further includes:

[0037] Detecting whether there are repeated data vectors in each of the candidate neighbor vectors;

[0038] If yes, de-duplication processing is performed on each of the candidate neighbor vectors.

[0039] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above data query methods are implemented.

[0040] In a third aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above data query methods.

[0041] The technical solution provided by the present invention, when there are newly added data vectors in the vector database, constructs a temporary index of the vector database according to the newly added data vectors of the vector database, so that the query index of the vector database includes both the long-term index and the temporary index. In the process of querying the query vector, the neighbor vectors of the query vector in the original vector data of the vector database can be obtained according to the long-term index of the vector database, and the neighbor vectors of the query vector in the newly added vector data of the vector database can be obtained according to the temporary index of the vector database. Due to the technical solution of the present invention, the problem of consuming too much time in the process of inserting the newly added data vector into the long-term index can be avoided. Therefore, compared with the prior art, the availability of the vector database can be improved, and the purpose of improving the performance of the vector database can be achieved.

[0042] Furthermore, the technical solution of the present invention, when there are many new data vectors in the vector database, constructs a temporary index of the vector database to improve the performance of the vector database; when there are few new data vectors in the vector database, inserts the new data vectors into the long-term index of the vector database to improve the accuracy of data query in the vector database.

[0043] Furthermore, the technical solution of the present invention can also update the long-term index of the vector database according to the temporary index of the vector database when the load of the vector database is small, so as to improve the efficiency of the vector database in executing data query and reduce the impact on the performance of the vector database.

[0044] Furthermore, in the technical solution of the present invention, since the HNSW graph index has the advantages of high query accuracy and recall rate, and the IVF index has the advantage of fast construction speed, the technical solution of the present invention adopts the HNSW graph index as the long-term index of the vector database and adopts the IVF index as the temporary index of the vector database, which can improve the accuracy of data query and improve the work efficiency of data query, thereby achieving the purpose of improving the performance of the database.

[0045] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0047] Figure 1 is a schematic flow chart of a method for querying data in a vector database according to an embodiment of the present invention;

[0048] Figure 2 is a schematic flow chart of obtaining a query index of a vector database in a data query method according to an embodiment of the present invention;

[0049] Figure 3 is a schematic flow chart of performing vector query according to a query index of a vector database in a data query method according to an embodiment of the present invention;

[0050] Figure 4 is a schematic flow chart of performing vector query according to a query index of a vector database in a data query method according to another embodiment of the present invention;

[0051] Figure 5 is a schematic flow chart of obtaining neighboring vectors of a vector to be queried in a data query method according to an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of an HNSW graph index in a data query method according to an embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of an IVF index in a data query method according to an embodiment of the present invention;

[0054] Figure 8 is a schematic flow chart of performing vector query according to a query index of a vector database in a data query method according to another embodiment of the present invention;

[0055] Fig. 9 is a schematic flow chart of performing vector query according to a query index of a vector database in a data query method according to another embodiment of the present invention;

[0056] Fig.10 is a schematic diagram of a computer program product according to an embodiment of the present invention; and

[0057] Fig.11 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In existing vector databases, commonly used query indexes include IVF indexes and HNSW graph indexes, wherein the IVF index is constructed by first clustering each vector in the vector database to obtain multiple clusters, and obtaining the centroid vector of each cluster to obtain the IVF index. The method of using the IVF index for data query in a vector database includes: after receiving a vector to be queried, first calculating the distance between each centroid vector and the vector to be queried to obtain the cluster with the smallest distance between the centroid vector and the vector to be queried, and then using one or more data vectors in the cluster that are closest to the vector to be queried as the neighbor vectors of the vector to be queried.

[0059] The HNSW graph index distributes data vectors in a vector database to a hierarchical graph structure. The method for performing data query using the HNSW graph index in a vector database includes: after receiving a vector to be queried, searching for vectors closest to the vector to be queried from the top graph to the bottom graph in turn, and using the vector closest to the query vector in each graph layer as the entry point of the next graph layer, and using one or more data vectors closest to the vector to be queried in the bottom graph as neighbor vectors of the query vector.

[0060] The above IVF index and HNSW graph index each have their own advantages and disadvantages. The advantages of IVF index are fast search speed and less memory usage, while the disadvantages are low search accuracy and the clustering in the IVF index does not change even if new vector data is added. The advantages of HNSW graph index are fast query speed, especially for high-dimensional data vectors, which can speed up the query speed and support incremental indexing. The disadvantage is that a large amount of memory is required because a graph structure with multiple levels needs to be maintained.

[0061] Generally, the query index of a vector database is constructed once, that is, it will not change after the query index is constructed. But in fact, as an engineering software, new data vectors will be continuously input into the vector database, so the query index of the vector database needs to be updated according to the newly added data vectors. For the IVF index, if there is a new data vector, the distance between the new data vector and the centroid vector of each cluster is calculated, and then the new data vector is assigned to the cluster corresponding to the nearest centroid vector. Therefore, if a new data vector is added, it will cause a large change in the IVF index, and the clusters and centroid vectors formed with the original data will become invalid.

[0062] For HNSW graph index, if there is a new data vector, it will be inserted into the multi-layer graph structure of HNSW graph index, and the adjacency relationship between the new data vector and its neighbor vector will be established. If there are many new data vectors, it will take a lot of time to update the HNSW graph index, and the vector database will not be able to provide external services during the update process, which will reduce the availability of the database and affect the performance of the database.

[0063] In order to solve the above technical problems, the present invention provides a data query method and related products of a vector database, as shown below. Figures 1 to 11 To describe a data query method and related products of a vector database of an embodiment of the present invention. In the description of this embodiment, it should be understood that the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features, that is, include one or more of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. When a feature "includes or contains" one or some of the features it covers, unless otherwise specifically described, this indicates that other features are not excluded and may further include other features.

[0064] See also Figure 1 , Figure 1 is a schematic flow chart of a method for querying data in a vector database according to an embodiment of the present invention. The method may generally include:

[0065] like Figure 1 As shown, the data query method of the vector database of this embodiment includes the following steps:

[0066] Step S102: obtaining a query index of a vector database;

[0067] Step S104: performing a vector query according to the query index of the vector database.

[0068] In this embodiment, the method for obtaining the query index of the vector database in the above step S102 is as follows: Figure 2 As shown, the following steps are included:

[0069] Step S112: obtaining a long-term index of the vector database, where the long-term index is an index constructed based on original vector data in the vector database;

[0070] Step S114: Check whether there is any new data vector in the vector database;

[0071] If yes, execute step S118;

[0072] Step S118: construct a temporary index of the vector database according to the newly added data vector of the vector database.

[0073] In this embodiment, a preset first index construction method can be used to construct an index based on the original data vector of the vector database to obtain a long-term index of the vector database, wherein the original data vector of the vector database is a data vector before the vector database has a newly added data vector. In the case where the vector database has a newly added data vector, in order to avoid the process of inserting the newly added data vector into the long-term index taking too long, in this embodiment, the newly added data vector is no longer inserted into the long-term index of the vector database, but a temporary index of the vector database is constructed based on the newly added data vector of the vector database, so that during the data query process, the newly added data vector of the vector database can be queried based on the temporary index.

[0074] In the above step S104, the method for performing vector query according to the query index of the database is as follows: Figure 3 As shown, the following steps are included:

[0075] Step S122: obtaining a vector to be queried, and determining whether there is a temporary index in the query index of the vector database;

[0076] If it exists, execute step S124; if it does not exist, execute step S126;

[0077] Step S124: Obtaining neighbor vectors of the vector to be queried according to the long-term index and temporary index of the vector database;

[0078] Step S126: Obtain neighbor vectors of the vector to be queried according to the long-term index of the vector database.

[0079] Since the query index of the database may or may not have a temporary index, after obtaining the vector to be queried, it is first determined whether there is a temporary index in the query index of the vector database.

[0080] If there is no temporary index in the query index of the vector database, the query vector can be queried using the long-term index according to the query method corresponding to the long-term index, and the neighbor vectors of the query vector can be obtained. If there is a temporary index in the query index of the vector database, the query vector can be queried according to the long-term index and the temporary index of the vector database respectively to obtain the neighbor vectors of the query vector.

[0081] The long-term index of the vector database is an index constructed based on the original vector data of the vector database. According to the long-term index of the vector database, the vectors of the vector to be queried in the original vector data of the vector database can be queried as much as possible. The temporary index of the vector database is an index constructed based on the newly added data vectors of the vector database. According to the temporary index of the vector database, the vectors of the vector to be queried in the newly added database of the vector database can be queried as much as possible. Therefore, obtaining the neighboring vectors of the vector to be queried based on the long-term index and temporary index of the vector database can expand the coverage area of ​​the query and improve the accuracy of the query.

[0082] In summary, the technical solution provided by this embodiment, when there are newly added data vectors in the vector database, constructs a temporary index of the vector database according to the newly added data vectors of the vector database, so that the query index of the vector database includes both the long-term index and the temporary index. In the process of querying the query vector, the neighbor vectors of the query vector in the original vector data of the vector database can be obtained according to the long-term index of the vector database, and the neighbor vectors of the query vector in the newly added vector data of the vector database can be obtained according to the temporary index of the vector database. Due to the technical solution of this embodiment, it is possible to avoid affecting the availability of the vector database due to the long time consumed in the process of inserting the newly added data vector into the long-term index, so compared with the prior art, it is possible to improve the performance of the vector database.

[0083] In some embodiments of the present invention, Figure 4 As shown, before constructing a temporary index of the vector database according to the newly added data vector of the vector database in the above step S118, it also includes:

[0084] Step S116: determining whether the amount of the newly added data vector in the vector database is greater than a set data amount threshold;

[0085] If yes, go to step S118; if no, go to step S120;

[0086] Step S120: updating the long-term index of the vector database according to the newly added data vectors of the vector database.

[0087] In this embodiment, the data volume threshold is preferably set to 500. When the data volume of the newly added data vector of the vector database is greater than the set data volume threshold, it can be considered that the newly added data vector of the vector database is large, and inserting the newly added data vector into the long-term index of the vector database will consume too much time, so step S118 is executed to construct a temporary index of the vector database to improve the performance of the vector database.

[0088] When the data volume of the newly added data vector in the vector database is not greater than the set data volume threshold, it can be considered that the newly added data vector in the vector database is small, and inserting the newly added data vector into the long-term index of the vector database will not take too long. Therefore, step S120 is executed to insert the newly added data vector into the long-term index of the vector database to improve the accuracy of data query in the vector database.

[0089] In summary, the technical solution of this embodiment can improve the accuracy of vector database data query and improve the performance of the vector database.

[0090] In some embodiments of the present invention, after constructing a temporary index of the vector database according to the newly added data vector of the vector database in the above step S118, the following further includes:

[0091] Step S122: Detect whether the load of the vector database is less than a set load threshold;

[0092] If yes, execute step S124;

[0093] Step S124: Merge the temporary index of the vector database into the long-term index.

[0094] Although setting a temporary index in the vector database can prevent the availability of the vector database from being affected by the long update time of the long-term index, using both the long-term index and the temporary index for data query will increase the workload of the vector database in executing data query compared to using only the long-term index. Therefore, it is necessary to merge the temporary index of the vector database into the long-term index.

[0095] In this embodiment, whether the load of the vector database is less than the set load threshold can be obtained according to the CPU usage of the vector database or the transaction volume executed by the vector database. For example, if the CPU usage of the vector database is less than the set usage threshold, it can be determined that the load of the vector database is less than the set load threshold; or if the number of transactions being executed by the vector database is less than the set transaction volume threshold, it can also be determined that the load of the vector database is less than the set load threshold.

[0096] If the load of the vector database is less than the set load threshold, it can be determined that the vector database is in an idle state. At this time, updating the long-term index of the vector database has a small impact on the performance of the vector database. Therefore, in this embodiment, when the load of the vector database is less than the set load threshold, the temporary index of the vector database is merged into the long-term index to update the long-term index of the vector database.

[0097] Through the technical solution of this embodiment, when the load of the vector database is small, the long-term index of the vector database can be updated according to the temporary index of the vector database, so as to improve the efficiency of the vector database in executing data query and reduce the impact on the performance of the vector database.

[0098] In some embodiments of the present invention, the method for updating the long-term index of the vector database according to the temporary index of the vector database includes:

[0099] Each data vector in the temporary index of the vector database is inserted into the long-term index of the vector database, and after all the data vectors in the temporary index are inserted into the long-term index, the temporary index is deleted.

[0100] In this embodiment, each data vector in the temporary index of the vector database is inserted into the long-term index of the vector database one by one. For example, assuming that the index type of the long-term index of the vector database is an HNSW graph index, and there are multiple layers in the index, the method of inserting one of the data vectors of the temporary index of the vector database into the long-term index includes:

[0101] The data vector to be inserted into the long-term index is taken as the vector to be inserted, and then the distance between each data vector in the long-term index and the data vector to be inserted is calculated, and the data vector with the closest distance to the vector to be inserted is taken as the neighbor vector of the vector to be inserted; finally, the vector to be inserted is inserted into the layer where its neighbor vector is located, and an adjacency relationship between the vector to be inserted and its neighbor vector is established.

[0102] After all data vectors in the temporary index of the vector database are inserted into the long-term index, the neighboring vectors of the vector to be queried can be obtained based on the long-term index. Therefore, in this embodiment, the temporary index is deleted to avoid the temporary index affecting the data query of the vector database.

[0103] In some embodiments of the present invention, the method for obtaining the long-term index of the vector database in step S112 includes:

[0104] An HNSW graph index is constructed according to the original data vectors of the vector database, and the HNSW graph index is used as a long-term index of the vector database.

[0105] The method for constructing a temporary index of the vector database according to the newly added data vector of the vector database in the above step S118 includes:

[0106] An IVF index is constructed according to the newly added data vector of the vector database, and the IVF index is used as a temporary index of the vector database.

[0107] In this embodiment, the method for constructing an HNSW graph index based on the original data vectors of the vector database includes: first, setting the number of layers for obtaining the HNSW graph index; then, inserting each data vector in the vector database into the bottom layer graph of the HNSW graph index; then, determining whether each data vector enters other layers based on the exponential probability distribution. If one of the data vectors is determined to enter the Ith layer, then the data vector will also exist in each layer from the I-1th layer to the bottom layer; finally, setting the adjacency relationship between the data vectors in each layer.

[0108] In this embodiment, the method for constructing an IVF index based on the newly added data vectors of the vector database includes: clustering each newly added data vector of the vector database to obtain a plurality of clusters, and calculating the centroid vector of each cluster.

[0109] In this embodiment, the k-means algorithm can be used to cluster the newly added data vectors of the vector database. The clustering method includes: first obtaining the number of clusters, then randomly selecting the number of vectors from the newly added data vectors, and using the selected newly added data vectors as the initial centroid vectors of each cluster. Then, the distance between each newly added data vector and each initial centroid vector is calculated, and each newly added data vector is assigned to the cluster with the closest centroid vector distance. Finally, each cluster is updated until the number of updates to each cluster reaches a set number, or the centroid vector of each cluster no longer changes. Among them, the method for updating each cluster includes: calculating the centroid vector of each cluster, then calculating the distance between each newly added data vector and each centroid vector, and assigning each newly added data vector to the cluster with the closest centroid vector distance.

[0110] The technical solution of this embodiment adopts the HNSW graph index as the long-term index of the vector database and adopts the IVF index as the temporary index of the vector database. Since the HNSW graph index has the advantages of high query accuracy and recall rate, and the IVF index has the advantage of fast construction speed, the technical solution of this embodiment can improve the accuracy of data query and improve the work efficiency of data query, thereby achieving the purpose of improving the performance of the database.

[0111] In some embodiments of the present invention, the method for obtaining the neighbor vectors of the query vector according to the long-term index and the temporary index of the vector database in step S126 is as follows: Figure 5 As shown, the following steps are included:

[0112] Step S212: acquiring a first set number of candidate neighbor vectors of the vector to be queried according to the long-term index of the vector database;

[0113] Step S214: acquiring a second set number of candidate neighbor vectors of the vector to be queried according to the temporary index of the vector database;

[0114] Step S216: Obtaining a neighbor vector of the vector to be queried from each candidate neighbor vector of the vector to be queried.

[0115] In this embodiment, the values ​​of the first set number and the second set number may be the same, and both are the numbers of neighboring vectors of the query vector that need to be obtained.

[0116] In the above step S212, the original data of the vector database may be queried according to the long-term index of the vector database to obtain a first set number of candidate neighbor vectors of the vector to be queried.

[0117] In this embodiment, the long-term index of the database is an HNSW graph index as an example. Assume that the long-term index has a three-layer graph structure, namely, a top-level graph, a middle-level graph, and a bottom-level graph. Figure 6 As shown, the method for obtaining a first set number of candidate neighbor vectors of the vector to be queried according to the long-term index includes:

[0118] First, the data vector closest to the query vector is found from the top-level graph, and the data vector is used as the entry vector of the middle-level graph; then, the neighbor vector of the entry vector in the middle-level graph is obtained, and the data vector closest to the query vector is found in the neighbor vector, and the data vector is used as the entry vector of the bottom-level graph; finally, the first set number of neighbor vectors of the entry vector in the bottom-level graph are used as candidate neighbor vectors of the query vector.

[0119] In the above step S214, the newly added data vector of the vector database may be queried according to the temporary index of the vector database to obtain a second set number of candidate neighbor vectors of the vector to be queried.

[0120] In this embodiment, the temporary index of the vector database is an IVF index as an example. Assume that there are a first cluster, a second cluster, and a third cluster in the temporary index. Figure 7 As shown, each cluster is obtained by clustering the newly added data vectors of the vector database, and each cluster has a corresponding centroid vector. Then, the method for obtaining a second set number of candidate neighbor vectors of the query vector according to the temporary index of the vector database includes:

[0121] First, the distance between the centroid vector of each cluster and the query vector is calculated to obtain the cluster with the smallest distance between the centroid vector and the query vector, and this cluster is used as the target cluster;

[0122] Then, the distance between each data vector in the target cluster and the query vector is calculated, and a second set number of vectors in the target cluster that are closest to the query vector are used as candidate neighbor vectors of the query vector.

[0123] Through the above steps S212 and S214, the number of candidate neighbor vectors of the vector to be queried is obtained to be the sum of the first set number and the second set number.

[0124] In the above step S216, the distances between the query vector and its candidate neighbor vectors may be calculated first, and a third set number of candidate neighbor vectors with the shortest distances to the query vector may be used as neighbor vectors of the query vector.

[0125] The technical solution of this embodiment can obtain candidate neighbor vectors of the vector to be queried in the original data vectors of the vector database according to the long-term index of the vector database, and obtain candidate neighbor vectors of the vector to be queried in the newly added data vectors of the vector database according to the temporary index of the vector database, thereby ensuring that the query range of the vector to be queried can cover the original data vectors and the newly added data vectors of the vector database; and then obtain the neighbor vector of the vector to be queried from each candidate neighbor vector, which can improve the accuracy and reliability of the query of the vector to be queried.

[0126] In some embodiments of the present invention, the method of obtaining the neighbor vector of the query vector from each candidate neighbor vector of the query vector in step S216 includes:

[0127] First, the distance between the query vector and each of its candidate neighbor vectors is calculated, and the candidate neighbor vectors of the query vector are sorted according to the size of the distance between them and the query vector;

[0128] Then, a third set number of candidate neighbor vectors having the smallest distances to the query vector are selected according to the sorting to serve as neighbor vectors of the query vector.

[0129] In this embodiment, the distance between the query vector and each of its candidate neighbor vectors may be a Euclidean distance, a cosine distance, or a negative inner product distance, or may be other distances defined by the user. After calculating the distance between the query vector and each of its candidate neighbor vectors, the candidate neighbor vectors of the query vector may be sorted in ascending order of the distance between the query vector and the query vector, and then the first third set number of candidate neighbor vectors in the sorting may be used as the neighbor vectors of the query vector.

[0130] After the candidate neighbor vectors of the query vector are sorted, the sorting can reflect the distance between the candidate neighbor vectors of the query vector and the query vector, and the neighbor vectors of the query vector can be quickly and accurately obtained according to the sorting. Therefore, through the technical solution of this embodiment, the work of obtaining the neighbor vectors of the query vector can be improved, and the work efficiency of data query can be improved.

[0131] In some embodiments of the present invention, before obtaining a third set number of neighbor vectors of the query vector from each candidate neighbor vector of the query vector in step S216, the method further includes:

[0132] Detect whether there are repeated data vectors in each candidate neighbor vector of the query vector;

[0133] If so, the candidate neighbor vectors of the query vector are deduplicated.

[0134] In this embodiment, duplicate data vectors may exist in the long-term index and temporary index of the vector database. For example, in the process of inserting the data vectors in the temporary index into the long-term index, the query index of the vector database includes both the long-term index and the temporary index. Since some data vectors in the temporary index have been inserted into the long-term index, the same data vectors may exist in the long-term index and the temporary index. The candidate neighbor vectors of the vector to be queried obtained according to the long-term index may be duplicated with the candidate neighbor vectors of the vector to be queried obtained according to the temporary index.

[0135] If there are duplicate data vectors in the candidate neighbor vectors of the query vector, there may also be duplicate data vectors in the neighbor vectors of the query vector obtained from the candidate neighbor vectors of the query vector, which will cause the actual number of neighbor vectors of the query vector obtained to be not the third set number.

[0136] Therefore, before obtaining the third set number of neighbor vectors of the vector to be queried from the candidate neighbor vectors of the vector to be queried, it is first detected whether there are repeated data vectors in the candidate neighbor vectors of the vector to be queried, and if there are repeated data vectors, the candidate neighbor vectors of the vector to be queried are deduplicated.

[0137] In this embodiment, the method for deduplicating candidate neighbor vectors of a query vector includes: first obtaining a candidate neighbor vector of the query vector, and then detecting whether there is a candidate neighbor vector that is the same as the candidate neighbor vector; if so, deleting the candidate neighbor vector that is the same as the candidate neighbor vector and retaining only the candidate neighbor vector.

[0138] The technical solution of this embodiment can perform deduplication processing on each candidate neighbor vector of the vector to be queried before obtaining the neighbor vector of the vector to be queried from each candidate neighbor vector of the vector to be queried, so as to reduce the interference of repeated data vectors in the candidate neighbor vector of the vector to be queried, and improve the accuracy and reliability of the query on the vector to be queried.

[0139] In some embodiments of the present invention, in the data query method of the vector database of the present invention, the method for obtaining the query index of the vector database is as follows: Figure 8 As shown, the following steps are included:

[0140] Step S302: constructing an HNSW graph index according to the original data vectors of the vector data volume, and using the HNSW graph index as a long-term index of the vector database;

[0141] Step S304: Check whether there is any new data vector in the vector database;

[0142] If yes, execute step S306;

[0143] Step S306: determining whether the data volume of the newly added data vector in the vector database is greater than a set data volume threshold;

[0144] If not, execute step S308; if yes, execute step S310;

[0145] Step S308: updating the long-term index of the vector database according to the newly added data vector of the vector database;

[0146] Step S310: constructing an IVF index according to the newly added data vector of the vector database, and using the IVF index as a temporary index of the vector database;

[0147] Step S312: Detect whether the load of the vector database is less than a preset load threshold;

[0148] If yes, execute step S314;

[0149] Step S314: inserting the data vector in the temporary index of the vector database into the long-term index of the vector database, and deleting the temporary index of the vector database after the insertion is completed.

[0150] In the data query method of the vector database of the present invention, the method of performing vector query according to the query index of the vector database is as follows: Fig. 9 As shown, the following steps are included:

[0151] Step S322: obtaining a vector to be queried, and determining whether there is a temporary index in the query index of the vector database;

[0152] If not, execute step S324; if yes, execute step S326;

[0153] Step S324: obtaining neighboring vectors of the vector to be queried according to the long-term index of the vector database;

[0154] Step S326: acquiring a first set number of candidate neighbor vectors of the vector to be queried according to the long-term index of the vector database;

[0155] Step S328: acquiring a second set number of candidate neighbor vectors of the vector to be queried according to the temporary index of the vector database;

[0156] Step S330: Detect whether there are repeated data vectors in each candidate neighbor vector of the query vector;

[0157] If yes, execute step S332; if no, execute step S334;

[0158] Step S332: performing deduplication processing on each candidate neighbor vector of the query vector;

[0159] Step S334: Calculate the distance between the query vector and each candidate neighbor vector, and sort the candidate neighbor vectors of the query vector according to the size of the distance between the query vector and the query vector;

[0160] Step S336: selecting a third set number of candidate neighbor vectors having the smallest distances to the query vector according to the sorting to serve as neighbor vectors of the query vector.

[0161] In this embodiment, the long-term index of the vector database is the HNSW graph index, and the temporary index is the IVF index. These two setting methods utilize the characteristics of the two indexes, that is, the query results of the HNSW graph index are more accurate, and the IVF index is very fast to build and search for small batches of data, and occupies less memory. In addition, the solution of this embodiment is universal, optimized for the situation of frequent updates, and also applicable to the situation of infrequent updates. After inserting data, very accurate results can be obtained in time. Users are actually not aware of the changes in the index structure. What users pursue is actually accurate results and real-time performance, and this method happens to meet these two requirements at the same time.

[0162] The flow chart provided by the present embodiment is not intended to indicate that the operation of the method will be performed in any particular order, or that all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical ideas provided by the present embodiment method, additional changes can be made to the above method.

[0163] It should be understood that in some embodiments, each part can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0164] This embodiment also provides a computer program product 10 , a computer readable storage medium 20 , and a computer device 30 . Fig.10 is a schematic diagram of a computer program product 10 according to an embodiment of the present invention, Fig.11 is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which implements the steps of any of the above-mentioned vector database data query methods when executed by a processor 32. The computer-readable storage medium 20 stores the above-mentioned computer program 11 thereon, which implements the steps of any of the above-mentioned vector database data query methods when executed by a processor 32.

[0165] The computer program 11 for performing the operation of the present invention may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, a configuration data of an integrated circuit, or a source code or an object code written in any combination of one or more programming languages ​​and process programming languages. The computer program 11 may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider via the Internet). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0166] In the description of this embodiment, the computer program product 10 is a related product including the computer program 11 .

[0167] For the purpose of the description of the present embodiment, the computer readable storage medium 20 is a tangible device capable of retaining and storing the computer program 11, which can be any device that can contain, store, communicate, propagate or use the computer program 11 for an instruction execution system, device or apparatus or in conjunction with these instruction execution systems, devices or apparatuses. More specific examples (a non-exhaustive list) of the computer readable storage medium 20 include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the above.

[0168] At this point, those skilled in the art should recognize that, although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived based on the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized as covering all these other variations or modifications.

Claims

1. A data query method for a vector database, characterized in that: include: A step of obtaining a query index of the vector database, and a step of performing a vector query according to the query index; The step of obtaining the query index of the vector database includes: Acquire a long-term index of the vector database, where the long-term index is an index constructed according to original data vectors of the vector database; Detecting whether the vector database has a new data vector; If yes, construct a temporary index of the vector database according to the newly added data vector; The step of performing a vector query according to the query index includes: After receiving the vector to be queried, determining whether the temporary index exists in the query index; If so, obtaining a neighbor vector of the vector to be queried according to the long-term index and the temporary index; If not, the neighboring vector of the vector to be queried is obtained according to the long-term index.

2. The data query method according to claim 1, characterized in that: Before the step of constructing a temporary index of the vector database according to the newly added data vector, the method further includes: Determining that the data volume of the newly added data vector is greater than a set data volume threshold; If yes, executing the step of constructing a temporary index of the vector database according to the newly added data vector; If not, the long-term index is updated according to the newly added data vector.

3. The data query method according to claim 1, characterized in that: After the step of constructing a temporary index of the vector database according to the newly added data vector, the method further includes: Detecting whether the load of the vector database is less than a set load threshold; If so, the temporary index is merged into the long-term index.

4. The data query method according to claim 3, characterized in that: The step of merging the temporary index into the long-term index comprises: Inserting each data vector in the temporary index into the long-term index one by one; After all the data vectors in the temporary index are inserted into the long-term index, the temporary index is deleted.

5. The data query method according to claim 1, characterized in that: The step of obtaining the long-term index of the vector database includes: Constructing a HNSW graph index according to the original data vectors of the vector database, and using the HNSW graph index as the long-term index; and The step of constructing a temporary index of the vector database according to the newly added data vector comprises: An IVF index is constructed according to the newly added data vector, and the IVF index is used as the temporary index.

6. The data query method according to claim 1, characterized in that: The step of acquiring the neighbor vector of the vector to be queried according to the long-term index and the temporary index comprises: Acquire a first set number of candidate neighbor vectors of the vector to be queried according to the long-term index, and acquire a second set number of candidate neighbor vectors of the vector to be queried according to the temporary index; The nearest neighbor vector is obtained from each of the candidate nearest neighbor vectors.

7. The data query method according to claim 6, characterized in that: The step of obtaining the nearest neighbor vector from each of the candidate nearest neighbor vectors comprises: Calculating the distance between each candidate neighbor vector and the vector to be queried, and sorting each candidate neighbor vector according to the distance between the candidate neighbor vector and the vector to be queried; According to the sorting, a third set number of candidate neighbor vectors having the shortest distances to the query vector are selected as the neighbor vectors.

8. The data query method according to claim 6 or 7, characterized in that: Before the step of obtaining a third set number of neighbor vectors of the vector to be queried from each of the candidate neighbor vectors, the method further includes: Detecting whether there are repeated data vectors in each of the candidate neighbor vectors; If yes, de-duplication processing is performed on each of the candidate neighbor vectors.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data query method according to any one of claims 1 to 8 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data query method described in any one of claims 1 to 8 are implemented.

Images