Vector data query method and device, electronic equipment and storage medium

Abstract

The invention relates to a vector data query method and device, electronic equipment and a storage medium. The method comprises the steps that first-level quantization operation is executed on a difference vector between a vector to be inquired and a global mean value to obtain a first-level quantization index, the first-level quantization index is reconstructed, a corresponding first reconstruction vector is obtained, second-level quantization operation is executed on a residual vector between the difference vector and the first reconstruction vector to obtain a second-level quantization index, and the second-level quantization index is obtained; and obtaining a candidate vector set according to the first reconstruction vector and a pre-constructed hierarchical navigable small-world graph, and finally searching a data query result corresponding to the to-be-queried vector based on the secondary quantization index and the candidate vector set. According to the scheme provided by the invention, the storage space requirement and the calculation complexity of the vector data can be effectively reduced on the premise of ensuring the retrieval accuracy, and the actual user requirement of large-scale and high-dimensional vector retrieval is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a vector data query method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the continuous development of deep learning technology, vector search has been widely applied in the fields of information retrieval, recommendation system and natural language processing. The core of vector search is to convert complex data into vectors in deep learning, and to efficiently find semantically similar content from massive data by calculating the similarity between vectors.

[0003] At present, the mainstream vector search technology has been widely applied in practical applications due to its high retrieval efficiency and good recall rate. However, in the large-scale data scenario, the storage and indexing of high-dimensional vectors require a large amount of memory, and the memory consumption becomes a bottleneck. In the real-time retrieval scenario of high-dimensional vectors, the vector similarity calculation consumes a large amount of computing resources, which greatly affects the performance. In addition, in order to improve the retrieval performance, vector compression means is usually introduced, but this often sacrifices the retrieval accuracy, so that the vector search technology cannot simultaneously consider accuracy and performance. It can be seen that with the continuous expansion of data size and the continuous increase of vector dimension, the limitations of the current vector search technology are increasingly prominent, and it is difficult to simultaneously meet the accuracy and performance requirements of retrieval. SUMMARY

[0004] To solve or partially solve the problems in the related art, the present application provides a vector data query method, device, electronic equipment and storage medium, which can optimize the storage and computing performance of vector indexing while ensuring retrieval accuracy.

[0005] The first aspect of the present application provides a vector data query method, comprising: performing a first quantization operation on a difference vector between a to-be-queried vector and a global mean to obtain a first quantization index; reconstructing the first quantization index to obtain a corresponding first reconstructed vector; performing a second quantization operation on a residual vector between the difference vector and the first reconstructed vector to obtain a second quantization index; obtaining a candidate vector set according to the first reconstructed vector and a pre-constructed hierarchical navigable small world graph; based on the second quantization index and the candidate vector set, finding a data query result corresponding to the to-be-queried vector.

[0006] In an example, the first quantization operation on the difference vector between the to-be-queried vector and the global mean to obtain the first quantization index comprises: calculating a difference vector between the query vector and the global mean value; dividing the difference vector into N difference sub-vectors, N being an integer greater than 1; obtaining a first subspace codebook corresponding to each difference sub-vector, the first subspace codebook being a cluster center vector set of each difference subspace constructed in a training phase; quantizing and encoding the difference sub-vector using the first subspace codebook to generate the first-level quantization index.

[0007] In an example, the reconstructing the first-level quantization index to obtain a corresponding first reconstructed vector comprises: finding each cluster center vector corresponding to the first-level quantization index from the first subspace codebook; sequentially concatenating each cluster center vector to generate the corresponding first reconstructed vector; The first reconstructed vector is used to approximately reconstruct the difference vector between the query vector and the global mean value, and the first reconstructed vector has the same dimension as the difference vector.

[0008] In an example, the performing a second-level quantization operation on a residual vector between the difference vector and the first reconstructed vector to obtain a second-level quantization index comprises: calculating the residual vector between the difference vector and the first reconstructed vector, the residual vector being used to represent information loss of the first-level quantization operation; dividing the residual vector into M residual sub-vectors, M being an integer greater than 1; obtaining a second subspace codebook corresponding to each residual sub-vector, the second subspace codebook being a cluster center vector set of each residual subspace constructed in a training phase; quantizing and encoding the residual sub-vector using the second subspace codebook to generate the second-level quantization index.

[0009] In an example, the obtaining a candidate vector set according to the first reconstructed vector and a pre-constructed hierarchical navigable small-world graph comprises: randomly assigning a layer number to determine an entry point; starting from the entry point, searching for a neighbor node corresponding to the first reconstructed vector layer by layer in a descending order of layer number, and connecting a plurality of neighbor nodes to generate a neighbor node list until the search is completed; extracting a candidate vector associated with each neighbor node in the neighbor node list, and a plurality of the candidate vectors generating the candidate vector set.

[0010] In an example, the searching, based on the secondary quantization index and the candidate vector set, for the data query result corresponding to the to-be-queried vector comprises: obtaining a historical quantization index corresponding to the candidate vector, reconstructing a historical reconstructed vector based on the historical quantization index, and adding the global mean value to the historical reconstructed vector to generate a target calculation vector; reconstructing a second reconstructed vector corresponding to the residual vector based on the secondary quantization index, and adding the global mean value, the first reconstructed vector and the second reconstructed vector to generate a target query vector; calculating the Euclidean distance between the target calculation vector and the target query vector, and screening at least one data query result.

[0011] In an example, the searching, based on the secondary quantization index and the candidate vector set, for the data query result corresponding to the to-be-queried vector comprises: obtaining a historical quantization index corresponding to the candidate vector; searching, based on the historical quantization index, the primary quantization index and the secondary quantization index, for the distance between the candidate vector and the to-be-queried vector from a pre-stored distance lookup table; wherein the distance lookup table stores distance values between codebook vectors corresponding to each quantization index.

[0012] The second aspect of the present application provides a vector data query device, comprising: a primary quantization module configured to perform a primary quantization operation on a difference vector between a to-be-queried vector and a global mean value, and obtain a primary quantization index; a first reconstruction module configured to reconstruct the primary quantization index and obtain a corresponding first reconstructed vector; a secondary quantization module configured to perform a secondary quantization operation on a residual vector between the difference vector and the first reconstructed vector, and obtain a secondary quantization index; a candidate vector generation module configured to obtain a candidate vector set according to the first reconstructed vector and a pre-constructed hierarchical navigable small-world graph; a data query module configured to search, based on the secondary quantization index and the candidate vector set, for a data query result corresponding to the to-be-queried vector.

[0013] The third aspect of the present application provides an electronic device, comprising: a processor; and a memory having executable code stored thereon, the executable code, when executed by the processor, causing the processor to perform the method as described above.

[0014] The fourth aspect of the present application provides a computer readable storage medium, which stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the method described above.

[0015] The fifth aspect of the present application provides a computer program product, which comprises computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] The technical solution provided by the present application can include the following beneficial results: In the embodiments of the present application, a first quantization operation is performed on a difference vector between a to-be-queried vector and a global mean value, a first quantization index is obtained, the first quantization index is reconstructed, a corresponding first reconstructed vector is obtained, a second quantization operation is performed on a residual vector between the difference vector and the first reconstructed vector, a second quantization index is obtained, a candidate vector set is obtained according to the first reconstructed vector and a pre-constructed hierarchical navigable small world graph, and finally, a data query result corresponding to the to-be-queried vector is found based on the second quantization index and the candidate vector set.

[0017] Compared with the related art, the technical solution of the present application has the following advantages: on the one hand, the object of the first quantization operation is the difference vector between the to-be-queried vector and the global mean value, rather than directly quantizing the to-be-queried vector, which effectively preserves the relative position relationship between vectors and avoids destroying the original spatial structure, thereby reducing the influence of quantization error on retrieval accuracy. On the other hand, the first quantization operation can greatly compress data and quickly reduce the representation range of the vector, and the second quantization operation finely encodes the residual vector in the search stage to control error and ensure the accuracy of data query, thereby effectively compensating for the loss of accuracy caused by the first quantization. It can be seen that the technical solution of the present application can effectively reduce the storage space requirement and computational complexity of vector data while ensuring retrieval accuracy, thereby meeting the actual user demand for large-scale and high-dimensional vector retrieval.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the several views, and in which:

[0020] Figure 1 is a flowchart of a vector data query method according to an embodiment of the present application; Figure 2is another flow diagram of a vector data query method according to an embodiment of the present application; Figure 3 is a structural diagram of a vector data query device according to an embodiment of the present application; Figure 4 is a flow diagram of a vector index optimization method based on local adaptive quantization according to an embodiment of the present application; Figure 5 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] Embodiments of the present application will be described in more detail by referring to the drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided so that the present application is more thorough and complete, and the scope of the present application is fully conveyed to those skilled in the art.

[0022] The terminology used in the present application is merely for the purpose of describing specific embodiments and is not intended to limit the present application. The singular forms "a", "an" and "the" used in the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms "first", "second", "third", etc. can be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0024] With the continuous development of deep learning technology, vector search has been widely used in information retrieval, recommendation system and natural language processing, etc. The core of vector search is to convert complex data into vectors in deep learning, and to efficiently find semantically similar content from massive data by calculating the similarity between vectors.

[0025] Currently, the mainstream vector search technology has been widely used in practical applications due to its high retrieval efficiency and good recall rate.

[0026] However, in a large-scale data scenario, the storage and indexing of high-dimensional vectors require a large amount of memory, and memory consumption becomes a bottleneck, and in a real-time retrieval scenario of high-dimensional vectors, vector similarity calculation requires a large amount of computing resources, which greatly affects performance.

[0027] In addition, in order to improve retrieval performance, vector compression means are usually introduced, but this often comes at the expense of retrieval accuracy, so that the vector search technology cannot simultaneously consider accuracy and performance.

[0028] As can be seen, with the continuous expansion of data size and the continuous increase of vector dimension, the limitations of the current vector search technology are increasingly prominent, and it is difficult to simultaneously meet the accuracy and performance requirements of retrieval.

[0029] In related technologies, the limitations of the vector search technology are increasingly prominent, and it is difficult to simultaneously meet the accuracy and performance requirements of retrieval.

[0030] To solve the above problems, the embodiment of the present application provides a vector data query method, which can optimize the storage and computing performance of vector index while ensuring retrieval accuracy.

[0031] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0032] Figure 1 is a flowchart of a vector data query method according to an embodiment of the present application.

[0033] Referring to Figure 1 The scheme at least includes the following steps: Step 101, performing a first quantization operation on the difference vector between the to-be-queried vector and the global mean to obtain a first quantization index.

[0034] In the embodiment of the present application, the to-be-queried vector and the pre-computed global mean are first obtained, and then the difference vector between the two is calculated, and a first quantization operation is performed on the difference vector to obtain a first quantization index.

[0035] Optionally, the to-be-queried vector refers to a text vector input by a user or received by a system, which is usually a high-dimensional floating-point vector used for data query.

[0036] The global mean refers to a statistical mean vector obtained by averaging all vectors in a vector database, denoted as μ. The global mean can be calculated in advance and stored fixedly, which is used to reduce the influence of different data distribution on quantization accuracy.

[0037] The difference vector refers to the difference between the to-be-queried vector and the global mean vector, which reflects the position of the to-be-queried vector relative to the center of the entire vector space, and is conducive to preserving the relative structural relationship in the original vector space.

[0038] The first-level quantization operation refers to a process of quantization encoding of the difference vector. The technical means adopted in the present application is PQ quantization (Product Quantization), and of course, other technical means can also be adopted to realize quantization under the condition that the computing resources are satisfied, such as a quantization mode based on deep learning, VQ quantization (Vector Quantization), etc. The first-level quantization operation can divide the high-dimensional vector into several sub-vectors, and the closest codebook vector is searched in each subspace to obtain the quantization encoding result.

[0039] The first-level quantization index refers to the index set of the codebook vector matched in each subspace after the difference vector is subjected to the first-level quantization, which can be used to reconstruct the approximate vector or as a vector expression in the coarse retrieval process to narrow the search range.

[0040] As an example, text data input by a user through a smart electronic device is acquired, and after data preprocessing such as denoising and filtering of the text data, valid text is extracted, and the valid text is converted into a to-be-queried vector.

[0041] In step 102, the first-level quantization index is reconstructed to obtain a corresponding first reconstructed vector.

[0042] In the embodiment of the present application, after the first-level quantization index is obtained, the first-level quantization index can be reconstructed to obtain a corresponding first reconstructed vector.

[0043] Optionally, the reconstruction refers to extracting a corresponding cluster center vector based on the preset sub-codebook (i.e., the codebook set of each subspace) used in the first-level quantization process, and splicing the cluster center vector to obtain an approximate vector representation.

[0044] The first reconstructed vector refers to an approximate vector obtained by splicing all the sub-codebook vectors corresponding to the first-level quantization index, which is used to approximately restore the difference vector between the original to-be-queried vector and the global mean value.

[0045] In step 103, a second-level quantization operation is performed on the residual vector between the difference vector and the first reconstructed vector to obtain a second-level quantization index.

[0046] In the embodiment of the present application, the second-level quantization operation is performed on the residual vector between the difference vector and the first reconstructed vector to obtain a more refined second-level quantization index, thereby improving the accuracy of vector data query.

[0047] Optionally, the residual vector refers to the difference between the difference vector and the first reconstructed vector.

[0048] The secondary quantization operation refers to dividing the residual vector into multiple subspaces in a manner similar to the primary quantization operation, and finding the closest codebook vector for each subspace to obtain the corresponding index number.

[0049] The secondary quantization index refers to a sequence of numbers representing the corresponding codebook vector of the residual vector in each subspace, which is used for fast recovery of the residual approximation vector and calculation of the vector distance in the subsequent query stage.

[0050] It is worth noting that the difference between the primary quantization operation and the secondary quantization operation is: The goal of the primary quantization operation is to reduce the vector dimension or value range, and to use less memory bandwidth during graph construction and traversal, that is, the primary quantization operation focuses more on reducing the vector range, so it can allow some approximation error.

[0051] The goal of the secondary quantization operation is to perform accurate distance calculation in the candidate vector set to improve the final query accuracy, that is, the secondary quantization operation focuses on improving the search accuracy, so it needs to restore as much information as possible of the original vector. Moreover, the secondary quantization only takes effect in the final comparison stage, which improves the overall efficiency.

[0052] Step 104, obtaining a candidate vector set according to the first reconstructed vector and a pre-constructed hierarchical navigable small world graph.

[0053] In the embodiments of the present application, a hierarchical navigable small world graph is pre-constructed based on the original data set, and in the query stage, a candidate vector set can be obtained according to the first reconstructed vector and the hierarchical navigable small world graph.

[0054] The hierarchical navigable small world graph (HNSW) is a high-efficiency approximate nearest neighbor search index structure, which organizes all the vectors to be searched into a multi-level graph structure, and the higher the level, the sparser the connection, and the bottom layer is the densest. In the present application, the search starts from the top layer and goes down layer by layer.

[0055] Optionally, the candidate vector set refers to the vectors obtained by HNSW graph search, which belong to the neighbors of the vector to be queried.

[0056] In the query stage, the first reconstructed vector obtained in step 102 is used as the starting point of the search, and a group of candidate vectors most likely similar to the vector to be queried is screened out through graph search, thereby greatly reducing the calculation amount and improving the data query efficiency and accuracy.

[0057] Step 105, finding the data query result corresponding to the vector to be queried based on the secondary quantization index and the candidate vector set.

[0058] In the embodiment of the present application, a data query result corresponding to the vector to be queried may be searched based on the secondary quantization index and the candidate vector set.

[0059] Optionally, the data query result refers to the vector most similar to the query vector and its associated identifier, data information, etc., for example, the document number of a text search and the product code of a recommendation system.

[0060] In an embodiment of the present application, a first-level quantization operation is first performed on the difference vector between the query vector and the global mean to obtain a first-level quantization index, and the first-level quantization index is reconstructed to obtain a corresponding first reconstructed vector. Then, a second-level quantization operation is performed on the residual vector between the difference vector and the first reconstructed vector to obtain a second-level quantization index. Based on the first reconstructed vector and a pre-constructed hierarchical navigable small-world graph, a set of candidate vectors is obtained. Finally, based on the second-level quantization index and the set of candidate vectors, a data query result corresponding to the query vector is searched.

[0061] The technical solution of the present application is relative to the related art: on the one hand, the object of the first-level quantization operation is the difference vector between the vector to be queried and the global mean, rather than directly quantizing the vector to be queried, which effectively preserves the relative position relationship between the vectors and avoids destroying the original spatial structure, thereby reducing the impact of quantization error on retrieval accuracy. On the other hand, the first-level quantization operation can greatly compress data and quickly reduce the representation range of the vector, while the second-level quantization operation finely encodes the residual vector in the search phase to control errors and ensure the accuracy of data queries, thereby effectively compensating for the accuracy loss caused by the first-level quantization. It can be seen that the technical solution of the present application can effectively reduce the storage space requirements and computational complexity of vector data while ensuring retrieval accuracy, and meet the actual user needs of large-scale, high-dimensional vector retrieval.

[0062] Figure 2 This is another flowchart of a vector data query method shown in an embodiment of the present application. Figure 2 relatively Figure 1 The technical solutions of the embodiments of the present application are described in more detail.

[0063] Step 201: respond to a data query instruction, where the data query instruction includes a vector to be queried and the number of data query results.

[0064] In an embodiment of the present application, a user uses an intelligent electronic device to enter an information retrieval system or a recommendation system, and enters query information in a user interaction interface provided by the information retrieval system or the recommendation system. When the user touches the request control of the user interaction interface, a data query instruction is automatically triggered.

[0065] At this time, the to-be-queried vector can be a vector representation obtained by converting the query information input by the user into the system through feature extraction and other technical means as input data of the data query.

[0066] The data query result quantity can be the number of results returned by the system or the number of results expected to be obtained by the user, for example, the top 5, top 10, top 50, top 100, etc. most similar, to meet the user's demand for the number of data query results.

[0067] In step 202, a first quantization operation is performed on the difference vector between the to-be-queried vector and the global mean to obtain a first quantization index.

[0068] Before performing the vector data query, statistics are first performed on all original vectors to obtain the global mean μ. The calculation method is as follows: in each dimension, the values of all original vectors are added up, and then divided by the total number of vectors to obtain the average value of each dimension. The vector composed of these average values is taken as the global mean μ.

[0069] After performing the vector data query, according to the working principle described above, the global mean corresponding to the to-be-queried vector is determined, the to-be-queried vector is subtracted from the global mean, a first quantization operation is performed on the difference vector, the difference vector is divided into multiple subspaces, and the sub-vectors in each subspace are quantized and matched respectively to generate a first quantization index.

[0070] As an optional example of an embodiment of the present application, performing a first quantization operation on the difference vector between the to-be-queried vector and the global mean to obtain a first quantization index includes: calculating the difference vector between the to-be-queried vector and the global mean, dividing the difference vector into N difference sub-vectors, N being an integer greater than 1, obtaining a first subspace codebook corresponding to each difference sub-vector, and using the first subspace codebook to quantize and encode the difference sub-vector to generate a first quantization index.

[0071] The difference sub-vector is a segment obtained by dividing the whole difference vector into N segments, and the purpose of dividing multiple difference sub-vectors is to reduce the complexity of a single quantization operation and improve processing efficiency, while improving the overall quantization accuracy through local quantization.

[0072] The first subspace codebook is a set of clustering center vectors of each difference subspace constructed in the training phase, and the codebook can find the clustering center vector closest to the sub-vector, i.e. approximately representing the sub-vector.

[0073] As an example, a data set containing 1 million text vectors is obtained, each vector has a dimension of 768, and is represented using a 32-bit floating point number. The global mean μ is calculated, μ = [0.021, -0.015,..., 0.018].

[0074] For the query vector x, the difference d = x - μ is calculated, in order to reduce the complexity of high-dimensional quantization and improve the accuracy, the PQ quantization method is used to divide the difference vector d into 8 sub-vectors, , each sub-vector has a dimension of 96.

[0075] In the training phase, the K-means clustering algorithm is used for each subspace, and the number of clusters is set to 256, that is, each subspace codebook contains 256 cluster centers, and the index can be represented by 8 bits, to obtain the first subspace codebook set.

[0076] In the quantization encoding phase, for each sub-vector of the difference vector d , find the nearest cluster center (codebook vector) in the corresponding subspace codebook, and encode it with the index number of the cluster center.

[0077] For example, for the first sub-vector d¹, find the nearest cluster center c_1, 42 (index number 42). Similarly, quantize and encode the other sub-vectors, and combine the cluster center indexes of all sub-vectors into a primary quantization index d_Q = [42, 128, 75, 201], which is used to replace the complete query vector x for storage and retrieval.

[0078] In order to ensure that the original vector can be restored or approximately recovered, the global mean μ (stored as a constant) and the primary quantization index d_Q are retained in the query and reconstruction phase, so as to realize efficient and accurate vector query.

[0079] The primary quantization index is used to represent the rough features of the query vector, which facilitates subsequent fast candidate vector retrieval and fine distance calculation. Through the primary quantization operation, the data dimension complexity can be effectively reduced, the query speed can be accelerated, and good query accuracy can be maintained.

[0080] Step 203, reconstruct the primary quantization index to obtain the corresponding first reconstructed vector.

[0081] As an optional example of the embodiment of the present application, the cluster center vectors corresponding to the primary quantization index are found from the first subspace codebook, and the cluster center vectors are sequentially spliced according to the original order of the primary quantization index to generate the corresponding first reconstructed vector.

[0082] The first reconstructed vector is used to approximately reconstruct the difference vector between the query vector and the global mean, and the first reconstructed vector has the same dimension as the difference vector.

[0083] In the present application, the primary quantization index can be reconstructed as d_Q_reconstructed =∑c_i,j, where c_i,j is the codebook vector with index j in the i-th subspace.

[0084] For example, after obtaining the primary quantization index d_Q = [42, 128, 75, 201], the first reconstructed vector d_Q_reconstructed = [c_1, 42, c_2, 128, c_8, 201] is obtained through reconstruction. Compared with the primary quantization index, the first reconstructed vector has a more specific numerical representation and index relationship, which facilitates subsequent distance calculation and retrieval operations.

[0085] By reconstructing the primary quantization index, the first reconstructed vector that approximately represents the primary quantization index is obtained, which is beneficial to improving the calculation efficiency in the subsequent query stage.

[0086] In step 204, a secondary quantization operation is performed on the residual vector between the difference value vector and the first reconstructed vector to obtain a secondary quantization index.

[0087] As an optional example of an embodiment of the present application, performing a secondary quantization operation on the residual vector between the difference value vector and the first reconstructed vector to obtain a secondary quantization index includes: calculating the residual vector between the difference value vector and the first reconstructed vector, dividing the residual vector into M residual sub-vectors, M being an integer greater than 1, obtaining a second subspace codebook corresponding to each residual sub-vector, the second subspace codebook being a set of clustering center vectors of each residual subspace constructed in the training stage, and quantizing and encoding the residual sub-vector using the second subspace codebook to generate the secondary quantization index.

[0088] In the training stage, the subspace codebook of the residual vector is constructed using training data. All difference value vectors in the training set are divided into M residual sub-vectors, and a second subspace codebook is constructed for each residual sub-vector. The primary quantization operation and reconstruction are performed to obtain the corresponding residual vector Each residual vector is segmented into M residual sub-vectors (for example, M = 8), and the residual sub-vector set in each subspace i is clustered using K-means clustering to generate K' clustering centers {c'_i,1, c'_i,2,..., c'_i,K'}. K-means clustering is performed to generate K' clustering centers {c'_i,1, c'_i,2,..., c'_i,K'}, and the operation is repeated until convergence. Each residual sub-vector is assigned to the nearest clustering center. Each clustering center is updated as the mean of all vectors in the class. The final set of clustering centers is the second subspace codebook, with one codebook for each subspace, and a total of N.

[0089] where K' is usually set to be less than the number of clusters of the primary quantization, for example, K' = 16, which can be represented by 4 bits In the query phase: the residual vector e = d - d_Q_reconstructed is calculated, which is used to represent the information loss of the first quantization operation, or the approximation error. The residual vector e is divided into M residual sub-vectors . For each sub-vector , the closest cluster center vector c'_i,j is found from its corresponding second subspace codebook, and the quantization result of this residual sub-vector is represented by the index j, and the obtained second quantization index e_Q = [j_1, j_2,..., j_m] is obtained, such as e_Q = [3, 9, 15,..., 7].

[0090] That is, the second quantization operation can compensate and optimize the information that the first quantization operation fails to accurately restore, further improving the input data accuracy of data query.

[0091] Based on the contents of steps 202, 203 and 204, it can be known that the present application constructs a first subspace codebook for the first quantization operation and a second subspace codebook for the second quantization operation.

[0092] The codebook is composed of a plurality of subspace cluster center vectors. Assuming that the difference vector is divided into m sub-vectors, the first subspace codebook stores m x k codebook vectors in total, and the dimension of each codebook vector is the ratio of the original vector dimension d and the number of subspaces m, that is, d / m.

[0093] In the first quantization process, each difference sub-vector finds the closest cluster center vector in the corresponding subspace by searching, and the result is represented by an index, thereby obtaining the first quantization index array d_Q. To further improve the compression accuracy, the system also introduces a second subspace codebook for quantization and encoding of the residual vector generated in the first reconstruction process. Similarly, the residual vector is divided into m sub-vectors, each of which finds the closest cluster center vector in its corresponding second subspace codebook and is represented by an index, thereby obtaining the second quantization index array e_Q. The second subspace codebook contains m x K' codebook vectors in total, and the dimension of each vector is also d / m, where K' is usually less than the number of clusters K of the first codebook, so as to reduce redundancy and storage overhead.

[0094] Finally, each original vector can be represented by only two index arrays, i.e., the first quantization index d_Q and the second quantization index e_Q, thereby significantly reducing the storage cost. In addition, the entire quantization process adopts a local adaptive quantization strategy, that is, the original vectors are not directly uniformly encoded, but the difference between each original vector and the global mean vector is first calculated, and then only the difference vector is independently quantized. This strategy can better preserve the relative distance relationship between the original vectors, and will not destroy the structure due to global quantization, effectively improving the accuracy performance of the vectors in the approximate retrieval process.

[0095] At step 205, a candidate vector set is obtained according to the first reconstruction vector and the pre-constructed hierarchical navigable small world graph.

[0096] As an optional example of the embodiment of the present application, obtaining the candidate vector set according to the first reconstruction vector and the pre-constructed hierarchical navigable small world graph comprises: randomly allocating a layer number, determining an entry point, starting from the entry point, searching for neighbor nodes corresponding to the first reconstruction vector layer by layer in descending order of the layer number, connecting the neighbor nodes to generate a neighbor node list, and searching until the searching is completed, respectively extracting candidate vectors associated with each neighbor node in the neighbor node list, and generating a plurality of candidate vectors to generate the candidate vector set.

[0097] First, the construction process of the hierarchical navigable small world graph (HNSW graph) is described in detail: Step 1) Vector preprocessing and quantization encoding Perform a first-level quantization operation on all original vectors to be constructed as an index. After the difference value processing of each original vector from the global mean value, a difference vector is obtained. Then, the PQ quantization method is applied to the difference vector to divide it into k subspaces and perform subspace quantization respectively. Each difference vector d_i is finally represented as a first-level quantization index vector of length k: where j_k is the selected codebook index in the kth subspace, indicating the position of the sub-vector in the codebook. The set is the input vector set for constructing the HNSW index structure: {d_Q_1, d_Q_2,..., d_Q_N}.

[0098] Step 2) HNSW multi-layer graph structure construction Input: quantized vector set {d_Q_1, d_Q_2,..., d_Q_N} Parameter setting: M: the maximum number of neighbors connected by each node in each layer, M_L: the maximum number of connections of each node in the bottom layer (0th layer), usually set to a larger value of M, ef_construction: the search range in the construction stage.

[0099] The construction process is as follows: create an empty multi-layer graph structure, and the layers have a nested relationship. The top layer has fewer nodes and sparse connections, and the bottom layer has dense connections.

[0100] Secondly, the process of obtaining the candidate vector set is described in combination with the constructed HNSW graph: Step 3) Randomly allocate layers The maximum layer number l_i to which the node belongs is generated according to an exponential distribution function, and the calculation method is: floor(-ln(uniform(0,1)) x m_L). The layer determines the visible layer number of the node in the HNSW graph, and the higher the layer number, the sparser the node.

[0101] The entry point is determined from the randomly assigned layer, which is used as the starting position of the search. It is usually a representative vector node in the highest layer. The selection of the entry point can be preset or based on the global reference node reserved during the training phase.

[0102] Step 4) Insert nodes layer by layer from the highest layer to the bottom layer In the current layer, the first reconstructed vector d_Q_i is used as the query vector, and the search range is controlled using the ef_construction parameter to find the nearest neighbor nodes from the existing nodes.

[0103] And the heuristic algorithm is used to select the top M optimal neighbor nodes from the candidate set, establish the edge connection between the node and the selected neighbor nodes, and update the neighbor list of the connected nodes symmetrically to ensure the connectivity of the graph.

[0104] The search process starts from the highest layer where the entry point is located and traverses each layer of the graph in turn. Each layer uses a greedy strategy or a heap-based neighbor search algorithm to find a number of neighbor nodes with a closer distance to the query vector (i.e. the first reconstructed vector) in the current layer.

[0105] In each layer, the search process maintains a priority queue (candidate set) and constantly replaces the candidate nodes with a closer distance while limiting the search range to ensure efficiency.

[0106] Step 5) Enter the next layer Repeat the above search and connection steps until the node is inserted into the 0th layer (i.e. the bottom layer).

[0107] Step 6) Generate candidate vector set Sort all candidate vectors according to the exact distance and return the top-k results with the smallest distance to form the candidate vector set.

[0108] Step 206, based on the two-level quantization index and the candidate vector set, find the data query results that match the number of data query results.

[0109] As an optional example of the embodiment of the application, based on the secondary quantization index and the candidate vector set, the process of searching the data query result corresponding to the to-be-queried vector comprises the following steps: obtaining the historical quantization index corresponding to the candidate vector, reconstructing the historical reconstructed vector based on the historical quantization index, adding the global mean value to the historical reconstructed vector to generate a target calculation vector, reconstructing the second reconstructed vector corresponding to the residual vector based on the secondary quantization index, and adding the global mean value, the first reconstructed vector and the second reconstructed vector to generate a target query vector, calculating the Euclidean distance between the target calculation vector and the target query vector, and screening at least one data query result.

[0110] This example adopts a decoding reconstruction and fine distance estimation method to calculate the actual vector space distance between the to-be-queried vector and the candidate vector, and the specific process is as follows: The candidate vector set C={c1, c2,..., ck}, each vector has complete first-level quantization index and secondary quantization index.

[0111] 1) For each vector c_i in the candidate set, extract the stored secondary quantization index e_Q_i.

[0112] 2) Decode the secondary quantization information, and perform decoding reconstruction on the secondary quantization index of the to-be-queried vector and the candidate vector. The residual vector reconstruction of the to-be-queried vector and the candidate vector is actually calculated by summing the corresponding index vectors in each subspace codebook: For the to-be-queried vector: q_e_reconstructed=Σ(j=1tom)c'_j,q_e_Q[j]; For the candidate vector: e_i_reconstructed=Σ(j=1tom)c'_j,e_Q_i[j].

[0113] 3) Combine the first reconstructed vector of the first-level quantization index and the second reconstructed vector of the residual vector, and add the global mean vector to complete the full-amount approximate reconstruction of the query vector and the candidate vector, which are respectively represented as: q_approx (target query vector) = μ (global mean value) + q_d_Q_reconstructed (first reconstructed vector) + q_e_reconstructed (second reconstructed vector); c_i_approx (target calculation vector) = μ (global mean value) + c_i_d_Q_reconstructed (first historical reconstructed vector) + e_i_reconstructed (second historical reconstructed vector).

[0114] 4) Calculate the Euclidean distance: dist(q, c_i) = dist(q, c_i) - dist(q, c_i_approx) + dist(q_approx, c_i_approx).

[0115] Finally, according to the distance value from small to large, the first k distance minimum candidate vectors are selected as the final approximate query result, and the corresponding data item and data number are returned.

[0116] This example restores the reconstruction results of the first-level quantization vector and the second-level residual quantization vector, and then directly calculates the actual distance between the original vectors, thereby significantly improving the accuracy of distance estimation and the accuracy of the retrieval result.

[0117] As another optional example of the embodiment of the present application, based on the second-level quantization index and the candidate vector set, the process of finding the data query result corresponding to the to-be-queried vector can further include: obtaining the historical quantization index corresponding to the candidate vector, and based on the historical quantization index, the first-level quantization index and the second-level quantization index, finding the distance between the candidate vector and the to-be-queried vector from the pre-stored distance lookup table.

[0118] Among them, the distance lookup table stores the distance values between the codebook vectors corresponding to each quantization index.

[0119] This example uses a table lookup distance estimation method, and the specific process is as follows: on a large-scale data set, this example introduces a pre-constructed codebook distance lookup table, skips the reconstruction step, and directly finds the distance through the index position, thereby quickly estimating the approximate distance between the candidate vector and the query vector.

[0120] 1) Use lookup table to speed up: precompute the distance between each subspace codebook vector and store it as a distance lookup table.

[0121] 2) For two quantized vectors d_Q_i and d_Q_j, the distance calculation is: dist(d_Q_i, d_Q_j) = Σ(k=1 to m) distance_table[k][i_k][j_k].

[0122] Among them, i_k and j_k are the indexes of the two vectors in the kth subspace.

[0123] This example does not need to reconstruct the vector, nor does it need to perform actual distance calculation of high-dimensional vectors, and the retrieval speed is extremely fast, suitable for large-scale scenarios.

[0124] Referring to Figure 3 , Figure 3 is a vector index optimization method flowchart based on local adaptive quantization according to an embodiment of the present application. The process includes at least the following steps: Suppose there is a dataset containing 1 million text vectors, each vector is 768-dimensional, represented by 32-bit floating point numbers.

[0125] S31, vector segmentation setup Divide the 768-dimensional vector into 8 sub-vectors, each 96-dimensional.

[0126] Global mean calculation: μ = [0.021, -0.015,..., 0.018].

[0127] S32, first-level quantization process Calculate the difference d = x - μ for the original vector x.

[0128] Divide d into 8 sub-vectors: Each is 96-dimensional.

[0129] Perform K-means clustering on each subspace, Ks = 256 (can be represented by 8 bits). For the first sub-vector d¹, find the nearest cluster center c_1,42, represented by index 42. Similarly, quantize the other sub-vectors, and finally d_Q = [42, 128, 75,..., 201].

[0130] S33, second-level residual quantization process Reconstruct the first-level quantization result: d_Q_PQ = [c_1, 42, c_2, 128,..., c_8, 201].

[0131] Calculate the residual e = d - d_Q_PQ.

[0132] Apply PQ to the residual e as well, divided into 8 subspaces, but use Ks' = 16 to get the quantized representation of the residual e_Q = [3, 9, 15,..., 7].

[0133] S34, index construction and query Use the first-level quantization result d_Q to construct the HNSW graph, set the number of layers to 4, and the maximum number of connections M = 16.

[0134] When querying, perform the same PQ quantization on the query vector q.

[0135] S35, retrieval candidate set Search in the HNSW graph to get 100 candidate vectors.

[0136] For the candidate vectors, combine the first-level and second-level quantization results to reconstruct and calculate the exact distance.

[0137] S35, vector query result.

[0138] Return the 10 results with the smallest distance.

[0139] Through experimental verification, the above method is used to test on a 1 million text vector data set, and the result is compared with that of the accurate search using the original vector, the accuracy only decreases by about 5%, and the query performance is improved by about 30%.

[0140] In the embodiments of the present application, first, unlike the traditional single quantization, the present application proposes a two-stage quantization strategy for different application scenarios. The first-stage quantization is used for index construction, pursuing high compression rate, and the second-stage quantization is used for accurate calculation, pursuing high accuracy. While maintaining the retrieval accuracy, the memory and calculation overhead of index construction are significantly reduced. Second, the difference between each vector and the global mean is calculated before quantization, rather than directly quantizing the original vector, which preserves the relative position relationship between vectors, does not destroy the original neighbor structure, improves the expression ability of the quantized vector, and reduces the quantization error. Third, for the random access mode of HNSW, the core algorithm of HNSW does not need to be modified, and it can be directly applied to the existing HNSW implementation, has good compatibility, reduces the technology migration cost, and improves the practicability.

[0141] Corresponding to the foregoing application function implementation method embodiments, the present application also provides a vector data query device, an electronic device and corresponding embodiments.

[0142] Figure 4 is a structural schematic diagram of a vector data query device according to an embodiment of the present application.

[0143] Referring to Figure 4 A vector data query device includes: A first quantization module 401 is configured to perform a first quantization operation on a difference vector between a to-be-queried vector and a global mean, to obtain a first quantization index. A first reconstruction module 402 is configured to reconstruct the first quantization index, to obtain a corresponding first reconstructed vector. A second quantization module 403 is configured to perform a second quantization operation on a residual vector between the difference vector and the first reconstructed vector, to obtain a second quantization index. A candidate vector generation module 404 is configured to obtain a candidate vector set according to the first reconstructed vector and a pre-constructed hierarchical navigable small world graph. A data query module 405 is configured to search for a data query result corresponding to the to-be-queried vector based on the second quantization index and the candidate vector set.

[0144] As an optional example of the embodiments of the present application, the first quantization module 401 is configured to: calculate the difference vector between the to-be-queried vector and the global mean. The difference vector is divided into N difference sub-vectors, N being an integer greater than 1; A first subspace codebook corresponding to each difference sub-vector is obtained, the first subspace codebook being a cluster center vector set of each difference subspace constructed in a training stage; The first subspace codebook is used to quantitatively encode the difference sub-vector, to generate a first-level quantization index.

[0145] As an optional example of the embodiment of the application, the first reconstruction module 402 is configured to: search each cluster center vector corresponding to the first-level quantization index from the first subspace codebook; splice the cluster center vectors in sequence to generate a corresponding first reconstruction vector; The first reconstruction vector is used to approximately reconstruct the difference vector between the to-be-queried vector and the global mean, and the first reconstruction vector has the same dimension as the difference vector.

[0146] As an optional example of the embodiment of the application, the second-level quantization module 403 is configured to: calculate a residual vector between the difference vector and the first reconstruction vector, the residual vector being used to represent information loss of the first-level quantization operation; divide the residual vector into M residual sub-vectors, M being an integer greater than 1; obtain a second subspace codebook corresponding to each residual sub-vector, the second subspace codebook being a cluster center vector set of each residual subspace constructed in a training stage; The second subspace codebook is used to quantitatively encode the residual sub-vector, to generate a second-level quantization index.

[0147] As an optional example of the embodiment of the application, the candidate vector generation module 404 is configured to: randomly assign a layer number to determine an entry point; start from the entry point, search a neighbor node corresponding to the first reconstruction vector layer by layer in a descending order of the layer number, and connect a plurality of neighbor nodes to generate a neighbor node list until the search is completed; extract a candidate vector associated with each neighbor node in the neighbor node list, and generate a candidate vector set from the plurality of candidate vectors.

[0148] As an optional example of the embodiment of the application, the data query module 405 is configured to: obtain a historical quantization index corresponding to the candidate vector, reconstruct a historical reconstruction vector based on the historical quantization index, and add the global mean and the historical reconstruction vector to generate a target calculation vector; The second reconstruction vector corresponding to the residual vector is reconstructed based on the second quantization index, and the target query vector is generated by adding the global mean, the first reconstruction vector and the second reconstruction vector. The Euclidean distance between the target calculation vector and the target query vector is calculated, and at least one data query result is screened.

[0149] As an optional example of the embodiment of the present application, the data query module 405 is configured to: obtain the historical quantization index corresponding to the candidate vector; based on the historical quantization index, the first quantization index and the second quantization index, find the distance between the candidate vector and the to-be-queried vector from the pre-stored distance lookup table; The distance lookup table stores the distance values between the codebook vectors corresponding to each quantization index.

[0150] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0151] Figure 5 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0152] Referring to FIG. 5, Figure 5 The electronic device 500 includes a memory 510 and a processor 520.

[0153] The processor 520 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The memory 510 can include various types of storage units such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device can be a rewritable storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory can store some or all instructions and data required by the processor during runtime. In addition, the memory 510 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 510 can include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and a transient electronic signal transmitted through wireless or wired transmission.

[0154] The memory 510 stores executable code, which, when processed by the processor 520, can cause the processor 520 to perform part or all of the above-mentioned methods.

[0155] In addition, the method according to the present application can also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing part or all of the steps of the above-mentioned method of the present application.

[0156] Alternatively, the present application can also be implemented as a computer readable storage medium (or non-transitory machine readable storage medium or machine readable storage medium) having executable code (or computer program or computer instruction code) stored thereon, which is executed by a processor of an electronic device (or a server, etc.) to cause the processor to execute part or all of the steps of the above-mentioned method according to the present application.

[0157] The present application also provides a computer program product, which includes computer instructions, which, when executed by a processor, implement the method as described above.

[0158] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only, and that many modifications and variations of the embodiments described herein are possible. It is therefore to be understood that within the scope of the appended claims, and their equivalents, many alternatives to the embodiments described herein are possible. The selection of terms to be used in the description is not intended to limit the scope of the embodiments described herein, but rather to best explain the principles of the embodiments, practical application, or improvement over the technology in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A vector data query method, characterized in that: include: Performing a first-level quantization operation on the difference vector between the query vector and the global mean to obtain a first-level quantization index; Reconstructing the first-level quantization index to obtain a corresponding first reconstructed vector; Performing a secondary quantization operation on a residual vector between the difference vector and the first reconstructed vector to obtain a secondary quantization index; Obtaining a set of candidate vectors according to the first reconstruction vector and a pre-constructed hierarchical navigable small-world graph; Based on the secondary quantization index and the candidate vector set, a data query result corresponding to the vector to be queried is searched.

2. The method according to claim 1, characterized in that The step of performing a first-level quantization operation on a difference vector between a query vector and a global mean value to obtain a first-level quantization index includes: Calculating a difference vector between the query vector and the global mean; Dividing the difference vector into N difference sub-vectors, where N is an integer greater than 1; Obtaining a first subspace codebook corresponding to each difference subvector, where the first subspace codebook is a set of cluster center vectors of each difference subspace constructed in a training phase; The first subspace codebook is used to perform quantization encoding on the difference subvector to generate the first-level quantization index.

3. The method according to claim 2, characterized in that The reconstructing the first-level quantization index to obtain a corresponding first reconstructed vector includes: Searching for each cluster center vector corresponding to the first-level quantization index from the first subspace codebook; Concatenate each cluster center vector in sequence to generate the corresponding first reconstruction vector; The first reconstruction vector is used to approximately reconstruct a difference vector between the query vector and the global mean, and the first reconstruction vector has the same dimension as the difference vector.

4. The method according to claim 1, wherein The performing a secondary quantization operation on a residual vector between the difference vector and the first reconstructed vector to obtain a secondary quantization index includes: Calculating a residual vector between the difference vector and the first reconstructed vector, wherein the residual vector is used to represent the information loss of the first-level quantization operation; Dividing the residual vector into M residual sub-vectors, where M is an integer greater than 1; Obtaining a second subspace codebook corresponding to each residual subvector, where the second subspace codebook is a set of cluster center vectors of each residual subspace constructed in the training phase; The second subspace codebook is used to perform quantization encoding on the residual subvector to generate the secondary quantization index.

5. The method according to claim 1, wherein The step of obtaining a set of candidate vectors according to the first reconstructed vector and a pre-constructed hierarchical navigable small-world graph includes: Randomly assign the number of layers and determine the entry point; Starting from the entry point, searching for neighboring nodes corresponding to the first reconstruction vector layer by layer in descending order of number of layers, and connecting multiple neighboring nodes to generate a neighboring node list until the search is completed; The candidate vectors associated with each neighboring node in the neighboring node list are extracted respectively, and the candidate vector set is generated by a plurality of the candidate vectors.

6. The method according to claim 1, characterized in that The searching for a data query result corresponding to the vector to be queried based on the secondary quantization index and the candidate vector set includes: Obtaining a historical quantization index corresponding to the candidate vector, reconstructing a historical reconstruction vector based on the historical quantization index, and adding the global mean to the historical reconstruction vector to generate a target calculation vector; reconstructing a second reconstruction vector corresponding to the residual vector based on the secondary quantization index, and adding the global mean, the first reconstruction vector, and the second reconstruction vector to generate a target query vector; A Euclidean distance between the target calculation vector and the target query vector is calculated to filter at least one data query result.

7. The method according to claim 1, characterized in that The searching for a data query result corresponding to the vector to be queried based on the secondary quantization index and the candidate vector set includes: Obtaining a historical quantization index corresponding to the candidate vector; Based on the historical quantization index, the primary quantization index, and the secondary quantization index, searching a pre-stored distance lookup table for a distance between the candidate vector and the query vector; The distance lookup table stores distance values ​​between codebook vectors corresponding to various quantization indexes.

8. A vector data query device, characterized in that: include: A first-level quantization module is used to perform a first-level quantization operation on the difference vector between the query vector and the global mean to obtain a first-level quantization index; A first reconstruction module, configured to reconstruct the first-level quantization index to obtain a corresponding first reconstruction vector; a secondary quantization module, configured to perform a secondary quantization operation on a residual vector between the difference vector and the first reconstructed vector to obtain a secondary quantization index; a candidate vector generation module, configured to obtain a set of candidate vectors based on the first reconstruction vector and a pre-constructed hierarchical navigable small-world graph; The data query module is configured to search for a data query result corresponding to the vector to be queried based on the secondary quantization index and the candidate vector set.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable codes stored thereon, wherein when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 7.

Images