Vector Retrieval Method, Apparatus, Device, and Readable Storage Medium

Abstract

The present application discloses a vector retrieval method, apparatus, device, and readable storage medium. Index information of vectors to be retrieved is pre-constructed, and the index information includes the mean vector of the vectors to be retrieved, basis vectors, projection values of the vectors to be retrieved after removing the mean vector on each basis vector, and index identifiers corresponding to the respective projection values. After obtaining a vector to be retrieved, for each basis vector, based on the vector to be retrieved, a preset retrieval threshold, and the index information of the vectors to be retrieved, a retrieval result corresponding to the basis vector is determined. Finally, based on the retrieval results corresponding to the respective basis vectors, a final retrieval result corresponding to the vector to be retrieved is determined. Since the preset retrieval threshold can limit the similarity between the retrieval result and the vector to be retrieved, the retrieval result obtained based on the preset retrieval threshold is not a specific TopK. Therefore, in scenarios where high similarity is required without limiting the quantity, the retrieval result obtained based on the preset retrieval threshold has a better recall effect.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and more specifically, to a vector retrieval method, apparatus, device, and readable storage medium. Background Art

[0002] In today's data age, information retrieval has become an essential part of daily life and work, widely used in fields such as search engines, ad recommendations, and intelligent question-answering. There are currently multiple approaches to information retrieval, one of which is vector retrieval. Vector retrieval involves searching for vectors similar to the search vector in a given vector database using a metric (such as Euclidean distance, cosine distance, inner product, or Hamming distance).

[0003] Currently, commonly used vector search methods can be broadly categorized by their algorithm implementation: neighbor graph-based, clustering-based, and spatial partitioning-based. These methods all employ a TopK search approach, which retrieves the K most similar vectors to the search vector for recall. However, for scenarios requiring high similarity without a limit on the number of similarities (e.g., similar text recommendation), this TopK search approach can lead to reduced recall effectiveness.

[0004] Therefore, how to provide a vector retrieval method suitable for scenarios requiring high similarity but not limited in number has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application proposes a vector search method, device, equipment and readable storage medium. The specific solution is as follows:

[0006] A vector retrieval method, comprising:

[0007] Obtaining a vector to be searched, a preset search threshold, and pre-constructed index information of the searched vector, the index information including the mean vector and basis vectors of the searched vector, the projection value of the searched vector on each basis vector after removing the mean vector, and the index identifier corresponding to each projection value;

[0008] For each basis vector, determining a search result corresponding to the basis vector based on the vector to be searched, the preset search threshold, and the index information of the searched vector;

[0009] Based on the search results corresponding to the basis vectors, a final search result corresponding to the vector to be searched is determined.

[0010] Optionally, the mean vector of the retrieved vector is determined as follows:

[0011] Obtain the retrieved vectors;

[0012] Based on the retrieved vectors, determine the feature matrix corresponding to the retrieved vectors, where the number of rows of the feature matrix is the number of retrieved vectors, and the number of columns is the dimension of each retrieved vector;

[0013] Calculate the mean value of each column in the feature matrix to obtain the mean vector of the retrieved vectors.

[0014] Optionally, the method for determining the basis vectors is as follows:

[0015] Randomly select a first preset number of retrieved vectors from the retrieved vectors;

[0016] Perform Schmidt orthogonalization on the first preset number of retrieved vectors to obtain a first preset number of initial basis vectors;

[0017] Perform normalization on each of the initial basis vectors to obtain the basis vectors.

[0018] Optionally, the method for determining the basis vectors is as follows:

[0019] Perform mean removal on the retrieved vectors based on the mean vector of the retrieved vectors to obtain the retrieved vectors after mean removal;

[0020] Determine the feature matrix corresponding to the retrieved vectors after mean removal;

[0021] Based on the feature matrix corresponding to the retrieved vectors after mean removal, determine the covariance matrix;

[0022] Obtain the eigenvalues and eigenvectors of the covariance matrix, arrange the eigenvectors in descending order according to the magnitudes of the eigenvalues, and determine the top second preset number of the eigenvectors as the basis vectors.

[0023] Optionally, the projection values of the retrieved vectors after removing the mean vector on each of the basis vectors are sorted according to the projection value magnitudes and maintained using a skip list.

[0024] Optionally, the method further includes:

[0025] Determine the retrieved vectors to be deleted;

[0026] Delete the projection values of the retrieved vectors to be deleted on each of the basis vectors in the skip list.

[0027] Optionally, the method further includes:

[0028] Determine the retrieved vectors to be added;

[0029] Perform mean removal processing on the to-be-added retrieved vector based on the mean vector of the retrieved vectors, to obtain the to-be-added retrieved vector after mean removal processing;

[0030] Calculate the projection value of the to-be-added retrieved vector after mean removal processing on each of the basis vectors;

[0031] Insert the projection values of the to-be-added retrieved vector after mean removal processing on each of the basis vectors into the skip list in the order of the projection value magnitudes.

[0032] Optionally, determining the retrieval result corresponding to the basis vector based on the to-be-retrieved vector, the preset retrieval threshold, and the index information of the retrieved vectors includes:

[0033] Calculate the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector;

[0034] Determine the retrieval interval based on the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold;

[0035] Determine the target projection values as the projection values within the retrieval interval among the projection values of the retrieved vectors after removing the mean vector on the basis vector;

[0036] Determine the index identifier corresponding to the target projection value as the retrieval result corresponding to the basis vector.

[0037] Optionally, determining the retrieval interval based on the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold includes:

[0038] Determine a first value and a second value, where the first value is the difference between the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold, and the second value is the sum of the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold;

[0039] Determine the interval that is not greater than the first value and not less than the second value as the retrieval interval.

[0040] A vector retrieval device, the device includes:

[0041] An acquisition unit, configured to acquire a to-be-retrieved vector, a preset retrieval threshold, and index information of pre-constructed retrieved vectors, where the index information includes the mean vector of the retrieved vectors, the basis vectors, the projection values of the retrieved vectors after removing the mean vector on each of the basis vectors, and the index identifiers corresponding to each projection value;

[0042] A retrieval result determination unit corresponding to a base vector, configured to determine, for each base vector, a retrieval result corresponding to the base vector based on the vector to be retrieved, the preset retrieval threshold, and the index information of the retrieved vector;

[0043] A final retrieval result determination unit, configured to determine a final retrieval result corresponding to the vector to be retrieved based on the retrieval results corresponding to each base vector.

[0044] Optionally, the device includes a mean vector determination unit;

[0045] The mean vector determination unit is configured to obtain the retrieved vector; determine a feature matrix corresponding to the retrieved vector based on the retrieved vector, where the number of rows of the feature matrix is the number of retrieved vectors, and the number of columns is the dimension of each retrieved vector; calculate the mean of each column in the feature matrix to obtain the mean vector of the retrieved vector.

[0046] Optionally, the device includes a first base vector determination unit;

[0047] The first base vector determination unit is configured to randomly extract a first preset number of retrieved vectors from the retrieved vectors; perform Schmidt orthogonalization on the first preset number of retrieved vectors to obtain a first preset number of initial base vectors; perform normalization on each of the initial base vectors to obtain the base vectors.

[0048] Optionally, the device includes a second base vector determination unit;

[0049] The second base vector determination unit is configured to perform mean removal on the retrieved vectors based on the mean vector of the retrieved vectors to obtain the retrieved vectors after mean removal; determine a feature matrix corresponding to the retrieved vectors after mean removal; determine a covariance matrix based on the feature matrix corresponding to the retrieved vectors after mean removal; obtain eigenvalues and eigenvectors of the covariance matrix, and arrange the eigenvectors in descending order according to the magnitudes of the eigenvalues, and determine the first second preset number of the eigenvectors with the highest rankings as the base vectors.

[0050] Optionally, the projection values of the retrieved vectors after removing the mean vector on each of the base vectors are maintained using a skip list after being sorted according to the projection value magnitudes.

[0051] Optionally, the device further includes a vector deletion unit;

[0052] The vector deletion unit is configured to determine the retrieved vectors to be deleted; delete the projection values of the retrieved vectors to be deleted on each of the base vectors in the skip list.

[0053] Optionally, the device further includes: a vector addition unit;

[0054] The vector addition unit is configured to determine the to-be-added retrieved vector; perform mean removal processing on the to-be-added retrieved vector based on the mean vector of the retrieved vector to obtain the mean-removed to-be-added retrieved vector; calculate the projection values of the mean-removed to-be-added retrieved vector on each of the basis vectors; and insert the projection values of the mean-removed to-be-added retrieved vector on each of the basis vectors into the skip list in the order of the projection value magnitudes.

[0055] Optionally, the retrieval result determination unit corresponding to the basis vector includes:

[0056] A calculation unit, configured to calculate the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector;

[0057] A retrieval interval determination unit, configured to determine a retrieval interval based on the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold;

[0058] A target projection value determination unit, configured to determine the projection values within the retrieval interval among the projection values of the retrieved vector after removing the mean vector on the basis vector as target projection values;

[0059] A retrieval result determination unit, configured to determine the index identifier corresponding to the target projection value as the retrieval result corresponding to the basis vector.

[0060] Optionally, the retrieval interval determination unit is specifically configured to:

[0061] Determine a first value and a second value, where the first value is the difference between the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold, and the second value is the sum of the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold;

[0062] Determine the interval that is not greater than the first value and not less than the second value as the retrieval interval.

[0063] A vector retrieval device includes a memory and a processor;

[0064] The memory is configured to store a program;

[0065] The processor is configured to execute the program to implement each step of the vector retrieval method as described above.

[0066] A readable storage medium stores a computer program thereon. When the computer program is executed by a processor, each step of the vector retrieval method described above is implemented.

[0067] By means of the above technical solution, the present application discloses a vector retrieval method, device, equipment and readable storage medium. In this solution, index information of the retrieved vectors is pre-constructed. The index information includes the mean vector of the retrieved vectors, the basis vectors, the projection values of the retrieved vectors after removing the mean vector on each basis vector, and the index identifiers corresponding to each projection value. When performing vector retrieval, a vector to be retrieved, a preset retrieval threshold and the pre-constructed index information of the retrieved vectors are obtained. For each basis vector, based on the vector to be retrieved, the preset retrieval threshold and the index information of the retrieved vectors, the retrieval result corresponding to the basis vector is determined. Finally, based on the retrieval results corresponding to each basis vector, the final retrieval result corresponding to the vector to be retrieved is determined. In this solution, the preset retrieval threshold can limit the similarity degree between the retrieval result and the vector to be retrieved. The retrieval result obtained based on the preset retrieval threshold is not a specific TopK. Therefore, in scenarios where high similarity is required and the quantity is not limited, the retrieval result obtained based on the preset retrieval threshold has a better recall effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0069] Figure 1 is a schematic flowchart of a vector retrieval method disclosed in an embodiment of the present application;

[0070] Figure 2 is a schematic flowchart of a method for constructing index information of retrieved vectors disclosed in an embodiment of the present application;

[0071] Figure 3 is a schematic flowchart of a method for determining a retrieval result corresponding to a basis vector based on a vector to be retrieved, a preset retrieval threshold and index information of the retrieved vectors disclosed in an embodiment of the present application;

[0072] Figure 4 is a schematic structural diagram of a vector retrieval device disclosed in an embodiment of the present application;

[0073] Figure 5 is a hardware structural block diagram of a vector retrieval device disclosed in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0075] To better understand the solution of the present application, the commonly used vector retrieval methods will be introduced first, as follows:

[0076] The current society is in the data era. Information retrieval has become an important part of daily work and life and is widely used in many fields such as search engines, advertising recommendations, and intelligent question answering. There are currently various information retrieval methods, and vector retrieval is one of them.

[0077] Vector retrieval refers to retrieving vectors similar to the vector to be retrieved in a given vector database according to a certain metric method (such as Euclidean distance, cosine, inner product, Hamming distance, etc.). The vector to be retrieved and the vectors in the vector database can be obtained by converting unstructured data (such as text, audio, video, images, etc.) through some mapping means (such as artificial intelligence technology, machine learning technology, etc.).

[0078] Currently, the commonly used vector retrieval methods can be roughly divided into vector retrieval methods based on a nearest neighbor graph, vector retrieval methods based on clustering, and vector retrieval methods based on space partitioning according to the implementation method of the retrieval algorithm, where:

[0079] In the vector retrieval method based on a nearest neighbor graph, the representative algorithms are the NSW (Navigable small world models) algorithm and the HNSW (Hierarchical Navigable Small World graphs) algorithm. The idea of the NSW algorithm is to establish highway nodes in the nearest neighbor graph to link two nodes that are far apart to improve the retrieval speed; on this basis, the HNSW algorithm constructs a hierarchical navigable nearest neighbor graph. The top-level nodes are used as entry points. As the number of layers increases, the number of nodes in each layer also increases, and the bottom layer graph contains all the nodes. During vector retrieval, only the TopK nearest neighbor nodes associated with the nodes in each layer are retrieved downward.

[0080] In the vector retrieval method based on clustering, vectors are clustered according to the center distance. The result of clustering is that many vectors are divided into multiple clusters. During retrieval, only the most similar cluster needs to be found, and then brute-force search is performed within the cluster to obtain the TopK results. The retrieval effect of this method is greatly affected by parameter configuration.

[0081] Vector retrieval methods based on space partitioning. For example, KD-Tree takes the median value in each dimension to bisect the vectors until the smallest unit is reached, and then retains the results of searching for TopK; VP-Tree (Volumetric Point Tree) is a binary tree partitioning method that queries from the top vertex of the binary tree downwards to obtain the TopK results; the LSH (Locality-Sensitive Hashing) algorithm retrieves the TopK results by hashing and mapping into adjacent spaces.

[0082] The application scenarios of vector retrieval methods vary greatly. Taking the advertising recommendation algorithm as an example, in the initial stage of the algorithm, rough ranking is required, so thousands of similar vectors need to be recalled to ensure the effect of subsequent fine ranking; in the field of similar text recommendation, it is necessary to ensure that the similarity of the recalled vectors is controlled within a certain threshold, and most texts below the threshold are irrelevant texts, so there is no great requirement for the number. In addition, for scenarios such as deduplication of similar elements and judging the existence of similar elements, more emphasis is placed on the similarity degree of the recalled vectors rather than simply retrieving TopK.

[0083] However, the above-mentioned commonly used vector retrieval methods all adopt the TopK retrieval method, that is, K vectors most similar to the vector to be retrieved are retrieved for recall. However, for some scenarios that require high similarity and do not limit the quantity (such as similar text recommendation scenarios, deduplication of similar elements, judging the existence of similar elements, etc.), this TopK retrieval method will lead to a decline in the recall effect.

[0084] Therefore, how to provide a vector retrieval method suitable for scenarios that require high similarity and do not limit the quantity has become a technical problem that needs to be solved urgently by those skilled in the art.

[0085] With the complication of the application scenarios of vector retrieval methods, the capacity of vector databases can reach the level of hundreds of millions. At the same time, most scenarios have high requirements for the real-time performance of vector retrieval. Indexing is a process of effectively organizing data, which can greatly accelerate the query of large data sets. To improve the efficiency of vector retrieval, index identifiers can be constructed for each vector in the vector database. This index identifier is used to link the vector and the unstructured data represented by the vector. Based on the index identifiers of each vector in the vector database, index information can be pre-constructed according to the implementation method of the retrieval algorithm, and vector retrieval can be realized based on the constructed index information.

[0086] In the existing vector retrieval methods, index information is pre-constructed based on the index identifiers of each vector in the vector database. For different vector retrieval methods, the forms of the index information are also different. For different forms of index information, the addition and deletion of vectors will cause changes in the index information, and the changes in the index information have an impact on the retrieval performance. For example, in the HNSW (Hierarchical Navigable Small World graphs) algorithm, if a vector needs to be added, only the index information of the vector to be added needs to be added. If a vector needs to be deleted, the index information of the vector to be deleted is marked for deletion instead of being truly deleted. If vectors are frequently added and deleted, it will cause a sharp increase in memory, thus affecting the retrieval performance. In the VP-Tree (Victory Point Tree) algorithm, if vectors are frequently added and deleted, it will cause the cumulative deterioration of the depth of the tree, ultimately affecting the retrieval performance.

[0087] In view of the above problems, the inventor of this case conducted in-depth research and found that for any two adjacent points in space, the projection distance of these two points on any vector is not greater than the actual distance between the two points. If the projection distances of two points in space on multiple orthogonal vectors are similar, then their actual distances are very likely to be similar. Based on this, a vector retrieval method was finally proposed. This vector retrieval method is applicable to scenarios that require high similarity and do not limit the quantity, and can well handle the addition and deletion of vectors, so that the retrieval performance is not affected by the addition and deletion of vectors.

[0088] Next, the vector retrieval method provided by this application will be introduced through the following embodiments.

[0089] Refer to Figure 1 , Figure 1 which is a schematic flowchart of a vector retrieval method disclosed in an embodiment of this application. The method may include:

[0090] Step S101: Obtain the vector to be retrieved, a preset retrieval threshold, and the index information of the retrieved vectors pre-constructed. The index information includes the mean vector of the retrieved vectors, the base vectors, the projection values of the retrieved vectors after removing the mean vector on each of the base vectors, and the index identifiers corresponding to each projection value.

[0091] In this application, the vector to be retrieved can be a vector corresponding to any unstructured data (such as text, audio, video, image, etc.). The retrieved vectors are part or all of the vectors in the vector database. The preset retrieval threshold is set based on the scenario requirements and is used to represent the similarity degree between the retrieval result and the vector to be retrieved. The construction method of the index information of the retrieved vectors will be described in detail through the following embodiments and will not be elaborated here.

[0092] Step S102: For each basis vector, based on the vector to be retrieved, the preset retrieval threshold, and the index information of the retrieved vector, determine the retrieval result corresponding to the basis vector.

[0093] In this application, the projection value of the vector to be retrieved after removing the mean vector on the basis vector can be calculated, and based on the projection value of the vector to be retrieved after removing the mean vector on the basis vector, the preset retrieval threshold, the projection value of the retrieved vector after removing the mean vector on the basis vector, and the index identifier corresponding to each projection value, determine the retrieval result corresponding to the basis vector.

[0094] In this application, based on the projection value of the vector to be retrieved after removing the mean vector on the basis vector, the preset retrieval threshold, the projection value of the retrieved vector after removing the mean vector on the basis vector, and the index identifier corresponding to each projection value, determine the retrieval result corresponding to the basis vector, that is, from the projection values of the retrieved vector after removing the mean vector on the basis vector, determine the projection values whose similarity degree with the projection value of the vector to be retrieved after removing the mean vector on the basis vector meets the limit of the preset retrieval threshold, and determine the index identifier corresponding to the projection value that meets the limit of the preset retrieval threshold as the retrieval result corresponding to the basis vector. The specific implementation manner will be described in detail in the following embodiments and will not be elaborated here.

[0095] Step S103: Based on the retrieval results corresponding to each basis vector, determine the final retrieval result corresponding to the vector to be retrieved.

[0096] In this application, the intersection of the retrieval results corresponding to each basis vector can be determined as the final retrieval result corresponding to the vector to be retrieved. The retrieval results corresponding to each basis vector can be index identifiers. In addition to the index identifiers, the final retrieval result can also include the corresponding vectors and source data.

[0097] This embodiment discloses a vector retrieval method. In this method, index information of the retrieved vectors is pre-constructed. The index information includes the mean vector of the retrieved vectors, the basis vectors, the projection values of the retrieved vectors after removing the mean vector on each basis vector, and the index identifiers corresponding to each projection value. When performing vector retrieval, a vector to be retrieved, a preset retrieval threshold, and the pre-constructed index information of the retrieved vectors are obtained. For each basis vector, based on the vector to be retrieved, the preset retrieval threshold, and the index information of the retrieved vectors, the retrieval result corresponding to the basis vector is determined. Finally, based on the retrieval results corresponding to each basis vector, the final retrieval result corresponding to the vector to be retrieved is determined. In this solution, the preset retrieval threshold can limit the similarity degree between the retrieval result and the vector to be retrieved. The retrieval result obtained based on the preset retrieval threshold is not a specific TopK. Therefore, in scenarios where high similarity is required without limiting the quantity, the retrieval result obtained based on the preset retrieval threshold has a better recall effect. Additionally, for scenarios where high similarity is required without limiting the quantity, the preset retrieval threshold is generally relatively small, which ensures that the magnitude of the retrieval result corresponding to each basis vector is very small. Therefore, the retrieval performance can be improved.

[0098] In another embodiment of the present application, the construction method of the index information of the retrieved vectors is described in detail.

[0099] Refer to Figure 2 , Figure 2 which is a schematic flowchart of a method for constructing index information of retrieved vectors disclosed in an embodiment of the present application. The method may include:

[0100] Step S201: Obtain the retrieved vectors.

[0101] There may be multiple retrieved vectors. For ease of explanation, it is assumed that there are n retrieved vectors, and the dimension of each retrieved vector is q.

[0102] Step S202: Determine the mean vector of the retrieved vectors.

[0103] As an implementable manner, the determination method of the mean vector of the retrieved vectors is:

[0104] Based on the retrieved vectors, determine the feature matrix corresponding to the retrieved vectors. The number of rows of the feature matrix is the number of retrieved vectors, and the number of columns is the dimension of each retrieved vector; calculate the mean of each column in the feature matrix to obtain the mean vector of the retrieved vectors.

[0105] For ease of understanding, if there are n retrieved vectors and the dimension of each retrieved vector is q, then the feature matrix corresponding to the retrieved vectors is [n, q].

[0106] Step S203: Determine the basis vectors.

[0107] As an implementable manner, Schmidt orthogonalization can be used to determine the basis vectors. That is, the determination method of the basis vectors is as follows: randomly extract a first preset number of retrieved vectors from the retrieved vectors; perform Schmidt orthogonalization on the first preset number of retrieved vectors to obtain a first preset number of initial basis vectors; perform normalization on each of the initial basis vectors to obtain the basis vectors.

[0108] For ease of understanding, if there are n retrieved vectors and the first preset number is t, then t retrieved vectors can be randomly extracted from the n retrieved vectors, and Schmidt orthogonalization is performed on the t retrieved vectors to determine the basis vectors.

[0109] Specifically, first randomly select two groups of vectors from the t retrieved vectors, denoted as S0 and S1; let v0 = S0, according to the formula:

[0110]

[0111] Then, select the third retrieved vector S2, and obtain:

[0112]

[0113] Perform in sequence, and finally obtain v t-1 :

[0114] Perform normalization on v0, v1, …, v t-1 to finally obtain all the basis vectors.

[0115] As another implementable manner, principal component analysis can be used to determine the basis vectors. That is, the determination method of the basis vectors is as follows: perform mean removal on the retrieved vectors based on the mean vector of the retrieved vectors to obtain the retrieved vectors after mean removal; determine the eigenmatrix corresponding to the retrieved vectors after mean removal; determine the covariance matrix based on the eigenmatrix corresponding to the retrieved vectors after mean removal; obtain the eigenvalues and eigenvectors of the covariance matrix, and arrange the eigenvectors in descending order according to the magnitudes of the eigenvalues, and determine the first second preset number of the eigenvectors as the basis vectors.

[0116] For ease of understanding, if there are n retrieved vectors, the first preset number is s, and the eigenmatrix corresponding to the retrieved vectors after mean removal is B, the covariance matrix X can be calculated based on the formula X = B T ·B;

[0117] Calculate the eigenvalues and eigenvectors of the covariance matrix X, and sort the eigenvectors in descending order according to the eigenvalues. Take the first s eigenvectors as the basis vectors.

[0118] It should be noted that since the method of using Schmidt orthogonalization to determine the basis vectors does not consider the eigen - distribution among vectors, it will lead to improper determination of the basis vectors. In this case, generally, the following problems will occur: some vectors are clustered, and many vectors that were originally not close coincide in the projection on the basis, resulting in the vectors recalled on each basis being too concentrated. This will lead to a decline in the performance of real - time retrieval and an increase in the probability of false recall. Therefore, the method of using Schmidt orthogonalization to determine the basis vectors is more suitable for the case where the vector space distribution is decentralized and there is no obvious clustering. When there are obvious feature clusters in the vector space distribution, a more suitable method is to use principal component analysis to determine the basis vectors.

[0119] It should also be noted that for the case where the vector space distribution is decentralized and there is no obvious clustering, the number t of basis vectors can be appropriately selected to be less. This will reduce the memory occupancy of the index and improve the retrieval performance, but will slightly reduce the recall rate.

[0120] Step S204: Determine the projection values of the retrieved vector after removing the mean vector on each of the basis vectors, and the index identifiers corresponding to each projection value.

[0121] It should be noted that the index identifier corresponding to each projection value is the index identifier of the retrieved vector corresponding to this projection value. The projection values of the retrieved vector after removing the mean vector on each of the basis vectors, and the index identifiers corresponding to each projection value can be maintained using any data structure. As an implementable manner, in this application, the projection values of the retrieved vector after removing the mean vector on each of the basis vectors are sorted according to the projection value size and maintained using a skip list, which can improve the retrieval performance and ensure that the complexity of changing the index information when adding or deleting a retrieved vector is at the logarithmic level.

[0122] It should be noted that in this application, for each basis vector, a skip list can be bound to it, and the projection values of the retrieved vector after removing the mean vector on each of the basis vectors, and the index identifiers corresponding to each projection value are maintained in this skip list.

[0123] As an implementable manner, when a certain retrieved vector needs to be deleted, the following method can be used:

[0124] Determine the retrieved vector to be deleted;

[0125] Delete the projection values of the retrieved vector to be deleted on each of the basis vectors in the skip list.

[0126] As another implementable manner, when it is necessary to add a retrieved vector, the following method can be adopted:

[0127] Determine the to-be-added retrieved vector;

[0128] Perform mean removal processing on the to-be-added retrieved vector based on the mean vector of the retrieved vector to obtain the to-be-added retrieved vector after mean removal processing;

[0129] Calculate the projection value of the to-be-added retrieved vector after mean removal processing on each of the basis vectors;

[0130] Insert the projection values of the to-be-added retrieved vector after mean removal processing on each of the basis vectors into the skip list in the order of the projection value magnitudes.

[0131] It should be noted that after adding or deleting a retrieved vector, the index identifier of the retrieved vector needs to be added or deleted synchronously.

[0132] In another embodiment of the present application, the specific implementation manner of determining the retrieval result corresponding to the basis vector in step S102 based on the to-be-retrieved vector, the preset retrieval threshold, and the index information of the retrieved vector is described.

[0133] Refer to Figure 3 , Figure 3 is a schematic flowchart of a method for determining the retrieval result corresponding to the basis vector based on the to-be-retrieved vector, the preset retrieval threshold, and the index information of the retrieved vector disclosed in the embodiment of the present application. The method may include the following steps:

[0134] Step S301: Calculate the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector.

[0135] Step S302: Determine the retrieval interval based on the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold.

[0136] As an implementable manner, the determining the retrieval interval based on the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold includes: determining a first value and a second value, where the first value is the difference between the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold, and the second value is the sum of the projection value of the to-be-retrieved vector after removing the mean vector on the basis vector and the preset retrieval threshold; determining the interval that is not greater than the first value and not less than the second value as the retrieval interval.

[0137] For ease of understanding, assume that the projection value of the vector to be retrieved on the basis vector after removing the mean vector is m, and the preset retrieval threshold is th. Then the retrieval interval is [m - th, m + th].

[0138] Step S303: Determine the projection values within the retrieval interval among the projection values of the retrieved vector on the basis vector after removing the mean vector as the target projection values.

[0139] Step S304: Determine the index identifier corresponding to the target projection value as the retrieval result corresponding to the basis vector.

[0140] After determining the index identifier, the corresponding vector and original data can be determined based on the index identifier.

[0141] Next, a vector retrieval device disclosed in an embodiment of the present application will be described. The vector retrieval device described below can be correspondingly referred to the vector retrieval method described above.

[0142] Refer to Figure 4 , Figure 4 which is a schematic structural diagram of a vector retrieval device disclosed in an embodiment of the present application. As Figure 4 shown, the vector retrieval device may include:

[0143] An acquisition unit 11, configured to acquire a vector to be retrieved, a preset retrieval threshold, and index information of the retrieved vector pre-constructed, where the index information includes the mean vector of the retrieved vector, the basis vector, the projection value of the retrieved vector on each basis vector after removing the mean vector, and the index identifier corresponding to each projection value;

[0144] A retrieval result determination unit 12 corresponding to the basis vector, configured to determine the retrieval result corresponding to the basis vector for each basis vector based on the vector to be retrieved, the preset retrieval threshold, and the index information of the retrieved vector;

[0145] A final retrieval result determination unit 13, configured to determine the final retrieval result corresponding to the vector to be retrieved based on the retrieval results corresponding to each basis vector.

[0146] As an implementable manner, the device includes a mean vector determination unit;

[0147] The mean vector determination unit is configured to acquire the retrieved vector; determine the feature matrix corresponding to the retrieved vector based on the retrieved vector, where the number of rows of the feature matrix is the number of retrieved vectors, and the number of columns is the dimension of each retrieved vector; calculate the mean of each column in the feature matrix to obtain the mean vector of the retrieved vector.

[0148] As an implementable manner, the device includes a first basis vector determination unit;

[0149] The first basis vector determination unit is configured to randomly extract a first preset number of retrieved vectors from the retrieved vectors; perform Schmidt orthogonalization processing on the first preset number of retrieved vectors to obtain a first preset number of initial basis vectors; perform normalization processing on each of the initial basis vectors to obtain the basis vectors.

[0150] As an implementable manner, the device includes a second basis vector determination unit;

[0151] The second basis vector determination unit is configured to perform mean removal processing on the retrieved vectors based on the mean vector of the retrieved vectors to obtain the retrieved vectors after mean removal processing; determine the feature matrix corresponding to the retrieved vectors after mean removal processing; determine the covariance matrix based on the feature matrix corresponding to the retrieved vectors after mean removal processing; obtain the eigenvalues and eigenvectors of the covariance matrix, and arrange the eigenvectors in descending order according to the magnitudes of the eigenvalues, and determine the first second preset number of the eigenvectors ranked at the top as the basis vectors.

[0152] As an implementable manner, the projection values of the retrieved vectors after removing the mean vector on each of the basis vectors are maintained using a skip list after being sorted according to the magnitudes of the projection values.

[0153] As an implementable manner, the device further includes a vector deletion unit;

[0154] The vector deletion unit is configured to determine the retrieved vector to be deleted; delete the projection values of the retrieved vector to be deleted on each of the basis vectors in the skip list.

[0155] As an implementable manner, the device further includes: a vector addition unit;

[0156] The vector addition unit is configured to determine the retrieved vector to be added; perform mean removal processing on the retrieved vector to be added based on the mean vector of the retrieved vectors to obtain the retrieved vector to be added after mean removal processing; calculate the projection values of the retrieved vector to be added after mean removal processing on each of the basis vectors; insert the projection values of the retrieved vector to be added after mean removal processing on each of the basis vectors into the skip list in the order of the magnitudes of the projection values.

[0157] As an implementable manner, the retrieval result determination unit corresponding to the basis vectors includes:

[0158] A calculation unit configured to calculate the projection values of the vector to be retrieved after removing the mean vector on the basis vectors;

[0159] A retrieval range determination unit, configured to determine a retrieval range based on a projection value of the vector to be retrieved after removing the mean vector on the basis vectors and the preset retrieval threshold;

[0160] A target projection value determination unit, configured to determine a projection value within the retrieval range among the projection values of the retrieved vector after removing the mean vector on the basis vectors as a target projection value;

[0161] A retrieval result determination unit, configured to determine an index identifier corresponding to the target projection value as a retrieval result corresponding to the basis vectors.

[0162] As an implementable manner, the retrieval range determination unit is specifically configured to:

[0163] Determine a first value and a second value, where the first value is a difference between a projection value of the vector to be retrieved after removing the mean vector on the basis vectors and the preset retrieval threshold, and the second value is a sum value of a projection value of the vector to be retrieved after removing the mean vector on the basis vectors and the preset retrieval threshold;

[0164] Determine an interval that is not greater than the first value and not less than the second value as the retrieval range.

[0165] Refer to Figure 5 , Figure 5 FIG. Figure 5 is a hardware structure block diagram of a vector retrieval device provided in an embodiment of the present application. Refer to

[0166] In an embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete communication with each other through the communication bus 4;

[0167] The processor 1 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement an embodiment of the present invention;

[0168] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0169] Among them, the memory stores a program, and the processor can call the program stored in the memory. The program is used for:

[0170] Obtain a vector to be retrieved, a preset retrieval threshold, and index information of pre-constructed vectors to be retrieved. The index information includes the mean vector of the vectors to be retrieved, basis vectors, projection values of the vectors to be retrieved after removing the mean vector on each basis vector, and index identifiers corresponding to the respective projection values;

[0171] For each basis vector, determine a retrieval result corresponding to the basis vector based on the vector to be retrieved, the preset retrieval threshold, and the index information of the vectors to be retrieved;

[0172] Based on the retrieval results corresponding to each basis vector, determine a final retrieval result corresponding to the vector to be retrieved.

[0173] Optionally, the refinement function and expansion function of the program can be referred to the above description.

[0174] An embodiment of the present application further provides a readable storage medium, which can store a program suitable for execution by a processor. The program is used for:

[0175] Obtain a vector to be retrieved, a preset retrieval threshold, and index information of pre-constructed vectors to be retrieved. The index information includes the mean vector of the vectors to be retrieved, basis vectors, projection values of the vectors to be retrieved after removing the mean vector on each basis vector, and index identifiers corresponding to the respective projection values;

[0176] For each basis vector, determine a retrieval result corresponding to the basis vector based on the vector to be retrieved, the preset retrieval threshold, and the index information of the vectors to be retrieved;

[0177] Based on the retrieval results corresponding to each basis vector, determine a final retrieval result corresponding to the vector to be retrieved.

[0178] Optionally, the refinement function and expansion function of the program can be referred to the above description.

[0179] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0180] The various embodiments in this specification are described in a progressive manner, and the key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other.

[0181] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vector retrieval method, characterized in that: The method comprises: Obtaining a vector to be searched, a preset search threshold, and pre-constructed index information of the searched vector, the index information including the mean vector and basis vectors of the searched vector, the projection value of the searched vector on each basis vector after removing the mean vector, and the index identifier corresponding to each projection value; For each basis vector, determining a search result corresponding to the basis vector based on the vector to be searched, the preset search threshold, and the index information of the searched vector; Based on the search results corresponding to the basis vectors, a final search result corresponding to the vector to be searched is determined.

2. The method according to claim 1, characterized in that The mean vector of the retrieved vector is determined as follows: Get the retrieved vector; Based on the searched vector, determining a feature matrix corresponding to the searched vector, where the number of rows of the feature matrix is the number of searched vectors, and the number of columns is the dimension of each searched vector; The mean of each column in the feature matrix is calculated to obtain a mean vector of the searched vector.

3. The method according to claim 1, characterized in that The basis vector is determined as follows: Randomly extracting a first preset number of searched vectors from the searched vectors; Performing Schmidt orthogonalization on the first preset number of searched vectors to obtain a first preset number of initial basis vectors; Normalization is performed on each of the initial basis vectors to obtain the basis vector.

4. The method according to claim 1, wherein The basis vector is determined as follows: performing a mean removal process on the searched vector based on the mean vector of the searched vector to obtain a searched vector after the mean removal process; Determine the feature matrix corresponding to the retrieved vector after the mean value processing; Determining a covariance matrix based on a feature matrix corresponding to the retrieved vector after the mean removal process; The eigenvalues and eigenvectors of the covariance matrix are obtained, and the eigenvectors are arranged in descending order according to the magnitude of the eigenvalues, and a second preset number of the eigenvectors with the highest rankings are determined as the basis vectors.

5. The method according to claim 1, wherein The projection values of the searched vector after removing the mean vector on each of the basis vectors are sorted according to the size of the projection values and then maintained using a skip table.

6. The method according to claim 5, characterized in that Also includes: Determine the search vector to be deleted; Delete the projection value of the searched vector to be deleted on each of the basis vectors in the skip list.

7. The method according to claim 5, characterized in that Also includes: Determine the search vector to be added; performing a mean removal process on the to-be-added searched vector based on the mean vector of the searched vector to obtain the to-be-added searched vector after the mean removal process; Calculate the projection value of the search vector to be increased after the mean value removal processing on each of the basis vectors; The projection value of the search vector to be increased after the mean value removal processing on each of the basis vectors is inserted into the skip list in the order of the projection value.

8. The method according to claim 1, characterized in that The determining, based on the vector to be searched, the preset search threshold, and the index information of the searched vector, a search result corresponding to the basis vector includes: Calculate the projection value of the vector to be searched on the basis vector after removing the mean vector; Determining a search interval based on a projection value of the vector to be searched on the basis vector after removing the mean vector and the preset search threshold; Determine, as a target projection value, a projection value on the basis vector that is within the search interval among the projection values of the searched vector after the mean vector is removed; The index identifier corresponding to the target projection value is determined as the search result corresponding to the basis vector.

9. The method according to claim 8, characterized in that The determining of the search interval based on the projection value of the vector to be searched on the basis vector after removing the mean vector and the preset search threshold comprises: Determine a first value and a second value, where the first value is a difference between a projection value of the vector to be searched on the basis vector after removing the mean vector and the preset search threshold, and the second value is a sum of a projection value of the vector to be searched on the basis vector after removing the mean vector and the preset search threshold; A section that is not greater than the first value and not less than the second value is determined as a search section.

10. A vector search device, characterized in that: The device comprises: an acquisition unit, configured to acquire a vector to be searched, a preset search threshold, and pre-constructed index information of the searched vector, wherein the index information includes a mean vector and a basis vector of the searched vector, a projection value of the searched vector on each of the basis vectors after removing the mean vector, and an index identifier corresponding to each projection value; a retrieval result determination unit corresponding to a basis vector, configured to determine, for each basis vector, a retrieval result corresponding to the basis vector based on the vector to be searched, the preset retrieval threshold, and the index information of the searched vector; The final search result determining unit is used to determine the final search result corresponding to the vector to be searched based on the search results corresponding to each basis vector.

11. A vector retrieval device, characterized in that: including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the vector retrieval method according to any one of claims 1 to 9.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the vector search method according to any one of claims 1 to 9 is implemented.

Images