A similarity search method and system based on vector database

Abstract

The present application relates to the field of data retrieval technology, and in particular to a similarity search method and system based on a vector database. The method comprises: using the Euclidean distance between the vector data of the original information and the central data as the radius, and the angle with a preset direction as the angle, to obtain the data points of the vector data in polar coordinates; clustering the data points in the polar coordinates to obtain multiple global clusters; using the global cluster where the data points of the query vector are located as the target cluster; and using improved PQ coding to determine the subvectors of the target cluster and the centroid of each subvector, thereby obtaining the search results of the query vector. The technical solution of the present application can improve search accuracy while ensuring search efficiency.

Description

Technical Field

[0001] The present application relates to the field of data retrieval technology, and in particular to a similarity search method and system based on a vector database. Background Art

[0002] With the rapid development of big data technology, the amount of unstructured data, such as images, videos, and audio, has exploded. This unstructured data is stored in vector databases as vectors. When performing data recognition processing on unstructured data, such as face recognition or voice recognition, it is often necessary to perform similarity searches on the vector database to obtain multiple vectors with the highest similarity. Finally, recognition results are obtained based on the data categories of these vectors.

[0003] Currently, PQ coding is a fast similarity search technology for vector databases. Traditional PQ coding first decomposes a high-dimensional vector into M sub-vectors, and uses a clustering algorithm to divide the vector space of each sub-vector into K clusters. The cluster centers of the K clusters are the centroids of the sub-vectors. For any sub-vector, the index of the cluster to which the sub-vector belongs is found in the vector space, and the sub-vector is represented by the index. Ultimately, the high-dimensional vector can be encoded as a combination of M indexes, and then vector search is performed based on the combination of M indexes. For example, if the high-dimensional vector is 128-dimensional, the high-dimensional vector is decomposed into 8 sub-vectors, each with a dimension of 16, and the 128-dimensional high-dimensional vector is encoded as an 8-dimensional index combination.

[0004] However, traditional PQ coding directly specifies the number of sub-vectors and divides high-dimensional vectors into equal-length partitions to obtain multiple sub-vectors. The number of sub-vectors directly affects the search efficiency and accuracy of vector searches. The larger the number of sub-vectors, the higher the search accuracy, but the lower the search efficiency. Therefore, how to determine the number of sub-vectors to ensure the search efficiency and accuracy during vector searches is an urgent problem to be solved. Summary of the Invention

[0005] In order to solve the technical problems of low search efficiency and search accuracy, the present application provides a similarity search method and system based on a vector database, which can improve search accuracy while ensuring search efficiency.

[0006] In a first aspect, the present application provides a similarity search method based on a vector database, the search method comprising: taking the Euclidean distance between the vector data of the original information and the central data as the radius, and the angle with the preset direction as the angle, to obtain the data point of the vector data in the polar coordinates, wherein the original information is text or an image, and the central data is the average value of all vector data; clustering the data points in the polar coordinates to obtain multiple global clusters; taking the global cluster where the data points of the query vector are located as the target cluster; using improved PQ coding to determine the subvectors of the target cluster and the centroid of each subvector, thereby obtaining the search result of the query vector; the improved PQ coding comprises: clustering the dimensions according to correlation to obtain multiple subvectors; performing clustering operations in the subvector space of each subvector to obtain the centroid of each subvector; constructing an objective function by minimizing the centroid error and maximizing the search efficiency to determine the optimal number of subvectors, the centroid error is positively correlated with the Euclidean distance between the subvector and the corresponding centroid, and the search efficiency is negatively correlated with both the number of data points in the target cluster and the number of subvectors.

[0007] The Euclidean distance between the original vector data and the center data is used as the radius, and the angle with the preset direction is used as the angle to obtain the data points of each vector data in polar coordinates, and all vector data are mapped to polar coordinates. Based on the position information of the data points of each vector data in polar coordinates, each vector data is divided into multiple global clusters. The global cluster to which the query vector belongs can be directly located based on the position information of the data points of the query vector in polar coordinates, achieving rapid positioning of the target cluster and improving search efficiency. Furthermore, the similarity between each vector data and the query vector is calculated within the target cluster, avoiding traversing all vector data and further improving search efficiency. Furthermore, in the process of calculating the similarity between each vector data and the query vector within the target cluster, an objective function is constructed based on minimizing the centroid error and maximizing the search efficiency to determine the optimal number of subvectors. The vector data is split into the optimal number of subvectors using an improved PQ coding. The query vector and each vector data in the target cluster are encoded into index combinations based on the subvectors and the centroids of each subvector. The search results of the query vector are obtained by calculating the Hamming distance between the index combinations, improving search accuracy while ensuring search efficiency.

[0008] Preferably, obtaining multiple global clusters includes: setting a first cluster number, dividing all data points into multiple initial clusters based on the spatial distance of the data points in polar coordinates; taking the sum of the silhouette coefficient and the variance of the number of data points in each initial cluster as an evaluation value; adjusting the first cluster number multiple times to draw an evaluation value curve, and taking the first cluster number corresponding to the inflection point in the evaluation value curve as a first target number, and the clustering result of the first target number corresponds to multiple global clusters.

[0009] In the global cluster When searching for a vector in the cluster, it is necessary to traverse the global cluster Calculate the global cluster for all data points within The similarity between all data points in the initial cluster and the query vector is calculated, and the search results are obtained based on the similarity. Therefore, the variance of the number of data points in the initial cluster is used as part of the evaluation value to measure the consistency of the number of data points in each initial cluster and ensure the search efficiency of each query vector.

[0010] Preferably, clustering the dimensions based on correlation to obtain multiple sub-vectors includes: arranging the dimension values ​​of any dimension of each vector data in the target cluster in a preset order to obtain a dimension value sequence of the dimension; defining a clustering distance to divide all dimensions into an initial number of dimension clusters, and the dimension clusters correspond to the sub-vectors; the clustering distance is negatively correlated with the correlation between the dimension value sequences, and positively correlated with the difference between the dimension value distribution sequences.

[0011] Dimensions with greater correlation are divided into the same sub-vector to reduce information loss caused by dimension division. At the same time, dimensions with smaller differences in dimension value distribution sequences are divided into the same sub-vector to effectively reduce the centroid error of the sub-vector and improve search accuracy.

[0012] Preferably, the correlation is the absolute value of the Pearson correlation coefficient; within the dimension value sequence, the number of occurrences of the dimension value within the preset interval of each dimension value is counted to obtain the dimension value distribution sequence.

[0013] Preferably, performing a clustering operation in the sub-vector space of each sub-vector to obtain the centroid of each sub-vector includes: determining the number of centroids of each sub-vector using an elbow method.

[0014] Preferably, the centroid error for: , is the number of sub-vectors, For subvector The number of inner sub-vector data, For subvector Sub-vector data within , For subvector Inner vector data The center of mass.

[0015] Preferably, the search efficiency for: , is the number of sub-vectors, is the number of data points in the target cluster.

[0016] Preferably, the objective function is obtained by subtracting the product of the centroid error and the adjustment weight from the search efficiency.

[0017] The specific calculation formula of the objective function is given. When the search efficiency is high and the centroid error is small, the value of the objective function is a large value. The optimization algorithm is used to obtain the number of subvectors corresponding to the maximum value of the objective function, and the optimal number can be accurately obtained.

[0018] Preferably, adjust the weight for: , is the number of data points in the target cluster, is the average number of data points in all global clusters, for function.

[0019] Since the amount of vector data in different global clusters may be different, the adjustment weight is determined according to the number of data points in the target cluster to ensure that good search efficiency can be obtained in each global cluster.

[0020] In the second aspect of the present application, a similarity search system based on a vector database is also provided, comprising a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, a similarity search method based on a vector database according to the first aspect of the present application is implemented.

[0021] The technical solution of this application has the following beneficial technical effects:

[0022] First, the Euclidean distance between the vector data of the original information and the central data is used as the radius, and the angle with the preset direction is used as the angle to obtain the data points of each vector data in the polar coordinates, and all vector data are mapped to the polar coordinates; each vector data is divided into multiple global clusters according to the position information of the data points of each vector data in the polar coordinates, and the global cluster to which the query vector belongs can be directly located according to the position information of the data points of the query vector in the polar coordinates, thereby realizing rapid positioning of the target cluster and improving search efficiency; further, the similarity between each vector data and the query vector is calculated within the target cluster to avoid traversing all vector data, thereby further improving search efficiency.

[0023] Furthermore, in the process of calculating the similarity between each vector data and the query vector within the target cluster, the objective function is constructed by minimizing the centroid error and maximizing the search efficiency to determine the optimal number of sub-vectors. The vector data is split into the optimal number of sub-vectors using the improved PQ coding. The query vector and each vector data in the target cluster are encoded into index combinations according to the sub-vectors and the centroids of each sub-vector. The search result of the query vector is obtained by calculating the Hamming distance between the index combinations, thereby improving the search accuracy while ensuring the search efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1This is a flowchart of a similarity search method based on a vector database according to an embodiment of the present application.

[0025] Figure 2 This is a structural block diagram of a similarity search system based on a vector database according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0027] According to a first aspect of the present application, the present application provides a similarity search method based on a vector database. Figure 1 FIG is a flow chart of a similarity search method based on a vector database according to an embodiment of the present application. Figure 1 As shown, the similarity search method based on the vector database includes steps S101 to S104, which are described in detail below.

[0028] S101, taking the Euclidean distance between the vector data of the original information and the center data as the radius, and the angle with the preset direction as the angle, to obtain the data point of the vector data in polar coordinates, the original information is text or image, and the center data is the average value of all vector data.

[0029] In one embodiment, the original information is text or an image, and the original information is related to the specific application scenario. In the face recognition scenario, the original information is a face image; in the text recommendation scenario, the original information is a text sample. The vector data of the original information is obtained using an autoencoder network or a PCA algorithm, which is not limited in this application.

[0030] After obtaining the vector data of each original information, the average value of all vector data is calculated to obtain the center data; for the vector data of any original information, the Euclidean distance between the vector data and the center data is used as the radius, and the angle between the vector data and the preset direction is used as the angle to obtain the data point of the vector data in the polar coordinates; wherein, the preset direction is the horizontal right direction, and can also be any other direction, which is not limited in this application.

[0031] In this way, the data points of all vector data in polar coordinates are obtained using the same method.

[0032] S102 , clustering the data points in the polar coordinates to obtain multiple global clusters.

[0033] In one embodiment, obtaining multiple global clusters includes: setting a first cluster number, dividing all data points into multiple initial clusters based on the spatial distance of the data points in polar coordinates; taking the sum of the silhouette coefficient and the variance of the number of data points in each initial cluster as an evaluation value; adjusting the first cluster number multiple times to draw an evaluation value curve, and taking the first cluster number corresponding to the inflection point in the evaluation value curve as a first target number, and the clustering result of the first target number corresponds to multiple global clusters.

[0034] The first cluster number is 2, and each adjustment increases the first cluster number by 1. The silhouette coefficient is a commonly used indicator for evaluating clustering effectiveness. It combines the intra-cluster compactness and inter-cluster separation to quantify the clustering effect. It is a useful technical tool for those skilled in the art and will not be described in detail here.

[0035] The spatial distance of the data points in polar coordinates is the sum of the absolute value of the radius difference and the absolute value of the angle difference. and data points Spatial distance in polar coordinates for:

[0036] ,in, and The data points and data points The radius, and The data points and data points angle.

[0037] Understandably, in the global cluster When searching for a vector in the cluster, it is necessary to traverse the global cluster Calculate the global cluster for all data points within The similarity between all data points in the initial cluster and the query vector is calculated, and search results are obtained based on the similarity. Therefore, using the variance of the number of data points in the initial cluster as part of the evaluation value can measure the consistency of the number of data points in each initial cluster, making the number of data points in each initial cluster basically consistent. In this way, when searching for a vector in any global cluster, the number of traversed vector data is approximately the same, which ensures the search efficiency of each query vector.

[0038] S103: The global cluster where the data point of the query vector is located is used as the target cluster.

[0039] In one embodiment, in a face recognition scenario, the query vector is the vector data of the face image to be queried; in a text recommendation scenario, the query vector is the vector data of the text to be queried.

[0040] The global cluster where the query vector is located is quickly determined according to the position of the query vector's data points in the polar coordinates. The global cluster is used as the target cluster, and the query vector is searched in the target cluster, avoiding traversing all the data points in the polar coordinates and improving the search efficiency.

[0041] S104: Determine the sub-vectors of the target cluster and the centroid of each sub-vector using the improved PQ coding, and then obtain the search results of the query vector.

[0042] In one embodiment, one data point in the target cluster corresponds to one vector data, and the improved PQ coding is used to determine the sub-vectors of the target cluster and the centroids of each sub-vector. The query vector and each vector data in the target cluster are encoded into an index combination according to the sub-vectors and the centroids of each sub-vector, and the search result of the query vector is obtained by calculating the Hamming distance between the index combinations.

[0043] The improved PQ coding includes: clustering dimensions based on correlation to obtain multiple sub-vectors, and performing a clustering operation in the sub-vector space of each sub-vector to obtain the centroid of each sub-vector; constructing an objective function to determine the optimal number of sub-vectors by minimizing the centroid error and maximizing the search efficiency; the centroid error is positively correlated with the Euclidean distance between the sub-vector and the corresponding centroid, and the search efficiency is negatively correlated with both the number of data points in the target cluster and the number of sub-vectors.

[0044] Among them, clustering the dimensions based on correlation to obtain multiple sub-vectors includes: arranging the dimension values ​​of any dimension of each vector data in the target cluster in a preset order to obtain a dimension value sequence of the dimension; defining a clustering distance to divide all dimensions into an initial number of dimension clusters, and the dimension clusters correspond to the sub-vectors; the clustering distance is negatively correlated with the correlation between the dimension value sequences, and is positively correlated with the difference between the dimension value distribution sequences.

[0045] The initial number is randomly set. In the embodiment of the present application, the value of the initial number is 2. After the initial number is set, the Kmeans algorithm or the Kmeans++ algorithm can be used to divide all dimensions into the initial number of dimension clusters. Each dimension cluster contains at least one dimension. If a dimension cluster includes three dimensions, then the cluster corresponds to a subvector composed of these three dimensions.

[0046] The correlation can be expressed using the absolute value of the Pearson correlation coefficient.

[0047] In the dimension value sequence, the number of occurrences of dimension values ​​within the preset interval of each dimension value is counted to obtain the dimension value distribution sequence. For example, the dimension value range is divided into 5 dimension value preset intervals. There are 50 vector data in the target cluster, and the number of occurrences in the 5 dimension value preset intervals is 25, 0, 0, 10, and 15 respectively. Then the dimension value distribution sequence is If the difference between the dimensional value distribution sequences of any two dimensions is small, it means that the dimensional values ​​of the two dimensions are roughly the same. Dividing the two dimensions into the same subvector can effectively reduce the centroid error of the subvector. In this way, the dimensions with high correlation are divided into the same subvector, reducing the information loss caused by dimension division. At the same time, the dimensions with small differences in the dimensional value distribution sequences are divided into the same subvector, effectively reducing the centroid error of the subvector and improving search accuracy.

[0048] After obtaining the initial number of sub-vectors, each sub-vector corresponds to a vector space, and the sub-vector data contained in each vector space is consistent. For example, if there are 50 vector data in the target cluster, and dimension 1 and dimension 3 constitute a sub-vector, then the sub-vector space includes 50 sub-vector data, each of which contains dimension 1 and dimension 3. Further, the Kmeans algorithm or Kmeans++ algorithm is used in the vector space to divide the 50 sub-vector data into K sub-vector clusters, and the cluster center of each sub-vector cluster is calculated as the K centroids of the sub-vectors composed of dimension 1 and dimension 3. Among them, the number of sub-vector centroids K is determined using the elbow method, and the number of centroids corresponding to different sub-vectors is different. In this way, the vector data is split into the initial number of sub-vectors, and the centroid of each sub-vector is obtained.

[0049] In one embodiment, the number of subvectors is adjusted, that is, the initial number is continuously adjusted, and an objective function is constructed to minimize the centroid error and maximize the search efficiency to determine the optimal number of subvectors. The optimal number can balance the centroid error and search efficiency, thereby improving the retrieval efficiency while ensuring the retrieval accuracy.

[0050] The centroid error is positively correlated with the Euclidean distance between the subvector and the corresponding centroid, because the centroid is used to replace the subvector data in the subvector cluster to which the centroid belongs in the subsequent index combination, so the centroid error directly affects the search accuracy. Specifically, the centroid error for:

[0051] , is the number of sub-vectors, For subvector The number of inner sub-vector data, For subvector Sub-vector data within , For subvector Inner vector data The center of mass.

[0052] The search efficiency is negatively correlated with the number of data points and sub-vectors in the target cluster. The larger the number of data points in the target cluster, the more vector data needs to be traversed during the vector search, and the lower the search efficiency. The larger the number of sub-vectors, the longer the index combination, and the greater the amount of calculation required to obtain the search results based on the Hamming distance between the index combinations, and the lower the search efficiency. Specifically, the search efficiency for: , is the number of sub-vectors, is the number of data points in the target cluster.

[0053] Furthermore, the objective function is constructed by minimizing the centroid error and maximizing the search efficiency. That is, the objective function is negatively correlated with the centroid error and positively correlated with the search efficiency. Therefore, the objective function is obtained by subtracting the product of the centroid error and the adjustment weight from the search efficiency. The optimization algorithm is used to obtain the number of subvectors corresponding to the maximum value of the objective function to obtain the optimal number. Specifically, the objective function satisfy:

[0054] , For search efficiency, is the centroid error, The adjustment weight is used to balance the importance of search efficiency and centroid error. The larger the adjustment weight, the higher the importance of search efficiency. The value of is 1.

[0055] In another embodiment, since the number of vector data in different global clusters may be different, in order to ensure that a good search efficiency can be obtained in each global cluster, the adjustment weight is determined according to the number of data points in the target cluster. The adjustment weight is positively correlated with the number of data points in the target cluster. Specifically, in the process of determining the sub-vectors of the target cluster and the centroid of each sub-vector using the improved PQ coding, the adjustment weight is for:

[0056] , is the number of data points in the target cluster, is the average number of data points in all global clusters, for function.

[0057] In one embodiment, after obtaining the optimal number of target clusters, the query vector and all vector data in the target cluster can be split into multiple sub-vectors, and the number of sub-vectors is equal to the optimal number; and the centroid of each sub-vector can be obtained.

[0058] For example, the optimal number is recorded as 5, and the query vector can be split into 5 sub-vectors. Sub-vector 1 is located in the sub-vector cluster where the centroid 2 is located in the corresponding sub-vector space, and sub-vector 2 is located in the sub-vector cluster where the centroid 1 is located in the corresponding sub-vector space. Similarly, sub-vector 3, sub-vector 4 and sub-vector 5 are located in the sub-vector clusters where the centroid 2, centroid 5 and centroid 3 are located in their respective sub-vector spaces. Then the query vector can be encoded as The index combination of all vector data in the target cluster is obtained in the same way, and the similarity between the query vector and each vector data in the target cluster is calculated based on the Hamming distance between the index combinations, and the one or more vector data with the greatest similarity are used as the search result of the query vector.

[0059] According to the second aspect of the present application, the present application also provides a similarity search system based on a vector database. Figure 2 : is a structural block diagram of a similarity search system based on a vector database according to an embodiment of the present application. Figure 2 As shown, the system 50 includes a processor and a memory. The memory stores computer program instructions. When the computer program instructions are executed by the processor, the similarity search method based on a vector database according to the first aspect of the present application is implemented. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are well known in the art and are therefore not described in detail here.

[0060] It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all fall within the scope of protection of the present application.

Claims

1. A similarity search method based on a vector database, characterized in that: The search method includes: using the Euclidean distance between the vector data of the original information and the center data as the radius and the angle with the preset direction as the angle to obtain the data point of the vector data in polar coordinates, the original information is text or image, and the center data is the average value of all vector data; Cluster the data points in polar coordinates to obtain multiple global clusters; The global cluster where the data points of the query vector are located is taken as the target cluster; The improved PQ coding is used to determine the sub-vectors of the target cluster and the centroid of each sub-vector, and then the search results of the query vector are obtained; The improved PQ coding includes: clustering the dimensions according to the correlation to obtain multiple sub-vectors; performing clustering operations in the sub-vector space of each sub-vector to obtain the centroid of each sub-vector; constructing an objective function to determine the optimal number of sub-vectors by minimizing the centroid error and maximizing the search efficiency, the centroid error is positively correlated with the Euclidean distance between the sub-vector and the corresponding centroid, and the search efficiency is negatively correlated with the number of data points in the target cluster and the number of sub-vectors; .... for: , is the number of sub-vectors, For subvector The number of inner sub-vector data, For subvector Sub-vector data within , For subvector Inner vector data The center of mass; Search efficiency for: , is the number of sub-vectors, is the number of data points in the target cluster; The objective function is obtained by subtracting the product of the centroid error and the adjustment weight from the search efficiency.

2. A similarity search method based on a vector database according to claim 1, characterized in that: Obtaining multiple global clusters includes: Set the first cluster number and divide all data points into multiple initial clusters according to the spatial distance of data points in polar coordinates; The sum of the silhouette coefficient and the variance of the number of data points in each initial cluster is used as the evaluation value; The first cluster number is adjusted multiple times to draw an evaluation value curve, and the first cluster number corresponding to an inflection point in the evaluation value curve is used as a first target number, and the clustering result of the first target number corresponds to multiple global clusters.

3. A similarity search method based on a vector database according to claim 1, characterized in that: The dimensions are clustered according to the correlation to obtain multiple sub-vectors, including: Arrange the dimension values ​​of any dimension of each vector data in the target cluster in a preset order to obtain a dimension value sequence of the dimension; A clustering distance is defined to divide all dimensions into an initial number of dimension clusters, where the dimension clusters correspond to the subvectors; the clustering distance is negatively correlated with the correlation between dimension value sequences and positively correlated with the difference between dimension value distribution sequences.

4. A similarity search method based on a vector database according to claim 3, characterized in that: The correlation is the absolute value of the Pearson correlation coefficient; In the dimension value sequence, the number of occurrences of the dimension value in the preset interval of each dimension value is counted to obtain the dimension value distribution sequence.

5. A similarity search method based on a vector database according to claim 1, characterized in that: The performing a clustering operation in the sub-vector space of each sub-vector to obtain the centroid of each sub-vector includes: determining the number of centroids of each sub-vector using the elbow method.

6. A similarity search method based on a vector database according to claim 1, characterized in that: Adjust weight for: , is the number of data points in the target cluster, is the average number of data points in all global clusters, for function.

7. A similarity search system based on a vector database, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a similarity search method based on a vector database according to any one of claims 1 to 6 is implemented.

Images