Large-scale high-dimensional vector nearest neighbor data retrieval method and device

By improving product quantization method and dimensionality reduction compression of vector data, the problem of large-scale vector index update in dynamic data environments is solved, and efficient vector retrieval and index update are realized, which is suitable for the ever-changing data environment in the Internet.

CN119089005APending Publication Date: 2024-12-06INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410995162.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When existing large-scale vector index generation algorithms face changing Internet data, they lack effective update support, resulting in reduced query index orientation or require frequent recalculation, and consume a lot of computing power.

Method used

Improve product quantization method, support index updates, and reduce computing power consumption by reducing dimensionality compression of the original vector data, and build a technical verification system containing this algorithm to optimize the search index.

Benefits of technology

It realizes dynamic update of vector retrieval and reduces computing power consumption, improves query efficiency and index accuracy, and is suitable for dynamic databases and high-dimensional vector datasets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089005A_ABST
    Figure CN119089005A_ABST
Patent Text Reader

Abstract

The invention provides a large-scale high-dimensional vector nearest neighbor data retrieval method and device, and aims to solve the problems that for a current vector data set containing massive information, although an existing vector query index creation scheme can give a high-directivity query index based on a static data set, an efficient updating method is lacked when a large batch of newly-added data is added. The directivity of a query index is difficult to maintain. According to the method for creating and dynamically updating the large-scale high-dimensional vector query index based on improved product quantization, high query precision is guaranteed, updating consumption is limited to a low level at the same time, and high directivity of the query index after updating is maintained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing and data retrieval, and in particular to a large-scale high-dimensional vector nearest neighbor data retrieval method and device. Background Art

[0002] Vector nearest neighbor retrieval technology has a wide range of applications in many fields. The problem it solves is to find the data that is most similar to the query data in a large-scale database, usually for vectors. In the field of natural language processing, such as similar text matching, word vectors generated based on semantics are used to find text that is semantically similar to a given query text. Vector nearest neighbor retrieval technology is required when dealing with large-scale data retrieval problems.

[0003] According to different retrieval methods, nearest neighbor retrieval can be divided into precise retrieval and approximate retrieval. Precise retrieval measures the exact distance between the query vector and all vectors in the database and compares them, such as comparing the Euclidean distance and cosine distance between the query vector and the vectors in the database, and then finds the topk nearest neighbors. This method can give the most accurate results, but it consumes a lot of computing resources and takes too long to run. At the same time, the accuracy of its results is only for the vector level, and it is difficult to guarantee the accuracy in the original data type; approximate retrieval maps the original data to a low-dimensional space in various ways to reduce the amount of calculation, and further reduces the amount of calculation of distance comparison transactions through a series of approximate algorithms, such as local sensitive hashing, VP tree, product quantization, etc., which significantly reduces the amount of calculation of the query process with a small loss of accuracy and improves the retrieval speed.

[0004] In summary, the core of nearest neighbor search in large-scale data is the approximate nearest neighbor search problem. There are many research results in this field, including vector retrieval schemes based on space and hashing, which can be subdivided into different methods corresponding to different scopes of application. Research on vector retrieval algorithms for large-scale data sets is of great significance to the study of retrieval problems in natural language processing.

[0005] In the clustering partitioning scheme, the high-dimensional space is divided. For example, IVF (Inverted File) clusters the original data through kmeans and uses the centroid for quantization to reduce the query operation amount. Product quantization is a commonly used nearest neighbor search algorithm, which reduces the amount of calculation involved in the query process by dividing the data set into blocks and quantizing them.

[0006] Currently, the data on the Internet is constantly increasing, which means that the items in various large databases are also increasing. However, the mainstream vector retrieval solutions mainly generate retrieval indexes for static data, lack support for dynamic databases, and have poor support for retrieval index updates. Summary of the invention

[0007] The present invention aims at the problem that when facing the ever-changing Internet data, the existing large-scale vector index generation algorithm lacks effective update support for the query index after the newly added vector, so that the query index directivity gradually decreases with the addition of new vectors or the query index needs to be continuously recalculated to maintain its directivity with a large amount of computing power consumption. The present invention improves on the existing product quantization method for creating query indexes for large-scale high-dimensional vector sets, adds support for index update, and thus proposes a large-scale high-dimensional vector compression retrieval scheme with improved product quantization, so that dynamic update of query index is supported while realizing vector retrieval, and computing power consumption is further reduced by dimensionality reduction compression of original vector data. In addition, the inventor has also constructed a technical verification system containing this algorithm, in which the reasonable values ​​of some hyperparameters in the debugging algorithm can be explored for different types of data sets to achieve further optimization of the retrieval index.

[0008] Specifically, the present invention proposes Figure 5 A large-scale high-dimensional vector nearest neighbor data retrieval method is shown, characterized by comprising:

[0009] The initial step is to obtain the original database, take each feature vector in the original database as a row, and the index of each feature as a column to form an original vector group of the original database, and compress the row vector of the vector group to obtain a compressed vector group;

[0010] In the index construction step, the compressed vector group is divided into m low-dimensional subspaces in the column dimension, and multiple clusters of each low-dimensional subspace are obtained by clustering the subvectors of each low-dimensional subspace in the row dimension; and the central vector of each cluster is used as an index to represent all subvectors in the cluster;

[0011] The retrieval step obtains the query vector to be retrieved and compresses the query vector to obtain a compressed vector, divides the compressed vector into m low-dimensional sub-vectors in the column dimension, calculates the distance between the low-dimensional sub-vector in each column and the central vector of each cluster, selects the cluster with the closest distance as the candidate cluster, obtains the candidate cluster corresponding to each column, counts the row with the most appearances in all candidate clusters as the target row, and outputs the feature vector of the target row in the original vector group as the query result of the query vector.

[0012] The large-scale high-dimensional vector nearest neighbor data retrieval method shown is characterized by comprising:

[0013] In the index update step, a newly added vector is obtained, and after being compressed, the newly added vector is divided into m low-dimensional sub-vectors in the column dimension, which are respectively used as newly added sub-vectors in the m low-dimensional subspaces, and the sub-vectors of each low-dimensional subspace are re-clustered in the row dimension, and multiple clusters of each low-dimensional subspace are updated.

[0014] The large-scale high-dimensional vector nearest neighbor data retrieval method is characterized by comprising:

[0015] The compression includes:

[0016] The size of the compressed vector X is N and the dimension is D:

[0017] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0018] According to the expected μ of the data in the row dimension j Calculate the standard deviation σ of the data in the row dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X in the row dimension:

[0019]

[0020] The row dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain

[0021]

[0022] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0023]

[0024] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0025] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0026] Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″:

[0027] X″=X′W.

[0028] The large-scale high-dimensional vector nearest neighbor data retrieval method is characterized in that the query vector is a feature vector of image, text, video or audio data; and the multiple feature vectors included in the original database are feature vectors of image, text, video or audio data respectively.

[0029] The present invention also proposes Figure 6 A large-scale high-dimensional vector nearest neighbor data retrieval device B is shown, characterized in that it includes:

[0030] The initial module M1 obtains the original database, takes each feature vector in the original database as a row and the index of each feature as a column to form an original vector group of the original database, and compresses the row vector of the vector group to obtain a compressed vector group;

[0031] The index building module M2 divides the compressed vector group into m low-dimensional subspaces in the column dimension, and obtains multiple clusters of each low-dimensional subspace by clustering the subvectors of each low-dimensional subspace in the row dimension; and uses the central vector of each cluster as an index to represent all subvectors in the cluster;

[0032] Retrieval module M3 obtains the query vector to be retrieved and compresses the query vector to obtain a compressed vector, divides the compressed vector into m low-dimensional sub-vectors in the column dimension, calculates the distance between the low-dimensional sub-vector in each column and the central vector of each cluster, selects the cluster with the closest distance as the candidate cluster, obtains the candidate cluster corresponding to each column, counts the row with the most appearances in all candidate clusters as the target row, and outputs the feature vector of the target row in the original vector group as the query result of the query vector.

[0033] The large-scale high-dimensional vector nearest neighbor data retrieval device shown is characterized by comprising:

[0034] The index update module obtains the newly added vector, compresses the newly added vector and divides it into m low-dimensional sub-vectors in the column dimension, which are used as the newly added sub-vectors in the m low-dimensional subspaces, and re-clustering the sub-vectors of each low-dimensional subspace in the row dimension, and updating multiple clusters in each low-dimensional subspace.

[0035] The large-scale high-dimensional vector nearest neighbor data retrieval device is characterized by comprising:

[0036] The compression includes:

[0037] The size of the compressed vector X is N and the dimension is D:

[0038] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0039] According to the expected μ of the data in the row dimension j Calculate the standard deviation σ of the data in the row dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X in the row dimension:

[0040]

[0041] The row dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain

[0042]

[0043] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0044]

[0045] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0046] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0047] Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″:

[0048] X″=X′W.

[0049] The large-scale high-dimensional vector nearest neighbor data retrieval device is characterized in that the query vector is a feature vector of image, text, video or audio data; and the multiple feature vectors included in the original database are feature vectors of image, text, video or audio data respectively.

[0050] The present invention also proposes an electronic device, characterized in that it includes the aforementioned large-scale high-dimensional vector nearest neighbor data retrieval device; the electronic device is connected to an information display device, and the information display device is used to display the query results according to display parameters and attributes set by the user or through an artificial intelligence model.

[0051] The present invention also proposes a storage medium for storing a computer program for executing the large-scale high-dimensional vector nearest neighbor data retrieval method.

[0052] It can be seen from the above scheme that the advantages of the present invention are:

[0053] The present invention mainly aims at the problem that the data dimension in the current Internet is too high to be conducive to storage and computational processing; and the data in the Internet is in the process of continuous generation, while many existing vector similarity retrieval schemes lack support for updating indexes for large-scale new data. The present invention designs corresponding methods from the aspects of dimensionality reduction of high-dimensional vectors, construction of query indexes for large-scale vectors, and dynamic update of indexes for large quantities of new data, and integrates them into an integrated system, which provides solutions for vector compression, similarity retrieval and dynamic update to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the structure of the vector set X″ of the present invention;

[0055] Figure 2 Schematic diagram of the structure of the divided vector set X″ of the present invention;

[0056] Figure 3 It is a structural diagram of a data set after clustering in the divided low-dimensional subspace of the present invention;

[0057] Figure 4 It is a structural diagram of the large-scale high-dimensional vector data compression retrieval technology verification system of the present invention;

[0058] Figure 5 is a flow chart of the method of the present invention;

[0059] Figure 6 It is a module diagram of the device of the present invention;

[0060] Figure 7 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0061] Figure 8 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0062] Fig. 9 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0063] Reference numerals:

[0064] A-First electronic device;

[0065] B-Large-scale high-dimensional vector nearest neighbor data retrieval method and device;

[0066] C-data acquisition equipment;

[0067] D-information display device;

[0068] 1000 - second electronic device;

[0069] Ⅰ-computational unit;

[0070] II-ROM;

[0071] III-RAM;

[0072] IV-bus;

[0073] V-interface;

[0074] VI - input unit;

[0075] VII-output unit;

[0076] VIII- Storage medium;

[0077] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0078] It should be noted that the processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0079] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0080] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0081] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0082] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0083] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0084] It should also be understood that the term "and / or" in the present invention is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship, which can be understood by referring to the context.

[0085] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0086] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0090] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] When conducting research on large-scale vector similarity retrieval, the inventors found that the defects in the prior art were caused by the failure to consider that the retrieval index was difficult to update in a timely manner due to the continuous increase in data. The main problems include the following two points:

[0092] 1. The current mainstream methods have proposed solutions for the creation of query indexes, but lack support for updating the indexes after the creation is completed. As a result, when new vectors are added to the data set, either the index is not updated, at the cost of lower query accuracy as the number of new vectors increases, or the query index is regenerated every time a new vector is added. Although the query accuracy can be guaranteed, it consumes a lot of computing power.

[0093] 2. For vector data sets with very high dimensions, the process of generating and updating indexes consumes a lot of computing power. The operation time of directly executing the algorithm on the original data is too long. It is necessary to compress it to reduce the computing power consumption of subsequent query index operations.

[0094] The inventors have found through research on large-scale data retrieval analysis that the above problems can be solved by improving the existing product quantization method to support the update of the query index after the addition of new vectors. On this basis, the execution efficiency of query index establishment and update is improved by designing a compression algorithm for vector data.

[0095] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0096] The structure and operation process of the large-scale high-dimensional vector data compression retrieval technology verification system constructed in the present invention are as follows: Figure 4 As shown:

[0097] To construct a query index for a specific data set and use the index for similarity retrieval, firstly perform a preliminary compression on the original vector data according to the high-dimensional vector fusion dimension reduction method of the present invention, and then use the improved method based on product quantization proposed in the following step 2 to establish the query index of the data set. When it is necessary to perform similar content retrieval on the query vector, it is necessary to first reduce its dimension according to the same fusion compression method as before, and then use the same division as used in the previous query index to divide it from the current high-dimensional space into the Cartesian product of multiple low-dimensional subspaces, that is, the query subvector in the subsequent steps. These low-dimensional subspaces are the same as those used in the previous index establishment. Then, the distance between the query subvector and the low-dimensional subvector of each original vector is compared in each low-dimensional subspace. Since the process is highly complex and consumes a lot of computing power, the subvectors in each low-dimensional subspace are clustered using the quantization idea, and the center vector of each cluster represents all the subvectors in the cluster. When calculating the distance between the query subvector and all the subvectors of the original vector in the low-dimensional subspace, the centroid vector of the cluster where each subvector is located can be used to replace it. The query subvector only needs to calculate the distance with a limited number of cluster centroid vectors. In this way, the operation part can be reduced from the original N D-dimensional queries to k times, where k is the number of clusters in each low-dimensional subspace. The original data set is divided into m low-dimensional subspaces, so the amount of operation is now reduced to m×k D / m-dimensional operations. After calculating the distance between the query subvector and the subvector in the cluster according to the centroid, the distance between each vector in the original high-dimensional data set and the query vector can be calculated. The distance between the query vector in the high-dimensional space and the vector is represented by the sum of the distances between the subvectors of a high-dimensional vector in the original data set in each low-dimensional space and the corresponding query subvector in the space, that is, the sum of the distances between the centroid of the cluster where each subvector of the high-dimensional vector is located and the query subvector in the low-dimensional subspace where it is located. Compared with direct distance comparison, the distance value calculated by this method is not strictly the Euclidean distance, but the accuracy of its query results can reach a very high level, and the query efficiency is much higher than that of accurate search.

[0098] When a large number of new data are added, the previously determined partitioning is used to split the original high-dimensional vectors into several low-dimensional sub-vectors. Then all the low-dimensional sub-vectors of each high-dimensional vector are added to the nearest cluster in its corresponding subspace and the query index is updated. When the corresponding components of all the sub-vectors of the new vectors in the query index are updated, the query index is updated. In each low-dimensional subspace, clusters with newly added subvectors are directly added as first-level re-clustering class candidates. For each cluster in these first-level re-clustering class candidates, several clusters closest to it are found according to the cluster centroid distance. The number of clusters is determined as the re-clustering class range threshold. After finding the adjacent clusters of each first-level re-clustering class candidate cluster, the parts that are already in the first-level re-clustering class candidates are eliminated to form second-level re-clustering class candidate clusters. The clusters in the existing second-level re-clustering class candidates are sorted according to the frequency of occurrence, and the parts that appear too few times are eliminated to form the final second-level re-clustering class candidates, which are combined with the first-level re-clustering class candidates to form re-clustering class candidates. The low-dimensional subspace re-clusters the vectors of the clusters in the set, and the number of clusters remains unchanged. The query index is corrected with the result of the re-clustering to complete the final update of the query index.

[0099] The algorithm of the present invention needs to pay extra attention to the selection of some hyperparameters when processing the construction and update of large-scale high-dimensional vector query indexes. Its changes have a great impact on the construction and update efficiency of the query index and the accuracy of the query results. Therefore, it is necessary to test the influence of different combinations of values ​​on the above index performance (including construction, update, and query). In step 3, a large-scale high-dimensional vector similarity retrieval technology verification system is used to study the influence of different hyperparameter values ​​on query performance, and a cross comparison is performed. In the process of building a query index, the algorithm proposed in step 2 first divides the original vector group in the high-dimensional space into several low-dimensional subspaces. The number of divisions m determines the efficiency of the process of clustering in each low-dimensional subspace during the subsequent index construction process and splitting the distance between the original vector and the query vector into each sub-vector and the query sub-vector during the query. That is, the more low-dimensional subspaces are divided, the finer the division of the original high-dimensional space, and the better the query results, but the time complexity of the operation will also increase accordingly. At the same time, the number of clusters in the low-dimensional subspace determines the fineness of the division in the low-dimensional subspace, that is, the more quantitative divisions there are, the more accurate the query results will be, but the time complexity of the operation will also increase accordingly.

[0100] Specifically, the large-scale high-dimensional vector compression retrieval method based on improved product quantization proposed in the present invention specifically includes the following steps:

[0101] Step 1: Fusion compression step based on high-dimensional vector data distribution and covariance features:

[0102] Considering that the data to be processed is too high in dimension, which will lead to long processing time and consume a lot of computing resources, it is necessary to compress and pre-process the high-dimensional vector before creating the query index. To this end, the inventors propose a fusion method for high-dimensional vector dimensional compression, which compresses data based on two aspects: data density in a specific dimension and data correlation in a specific dimension. Without loss of generality, let the fusion compression method C=C1C2, so that for the vector group X, the compressed data X″=C(X)=C2(C1(X)), where C1 is the first part of the fusion compression method, which compresses the high-dimensional vector by removing the low-density part based on its data distribution; C2 is the second part, which merges the parts with higher correlation based on the covariance characteristics of the high-dimensional vector, thereby achieving dimensionality reduction. First, the distribution of the original vector data is examined. For a vector group v with a scale of N and a dimension of D, since this method only involves dimensionality reduction, a single vector is not considered in this step, and the data on each column is examined:

[0103] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0104] Statistics on the data distribution of each column, calculate the expected μj and standard deviation of the data in each column:

[0105]

[0106] Inspecting the distribution of σj, the lower σl on column 1 means that the data density on this column is low. It can be removed during the initial dimensionality reduction to obtain the matrix X′ after the first dimensionality reduction. Its dimension is denoted as D′. At this time, the dimension of the data set is reduced by half.

[0107] Since highly correlated data will cause information redundancy, the correlation between different columns should be considered after eliminating the information redundancy on a single column. Highly correlated columns should be merged to reduce the dimension of the vector group. This method remaps the vector group after the initial dimensionality reduction to a new dimension with lower correlation by constructing a mapping from high-dimensional to low-dimensional space.

[0108] Zero the mean in the column dimension of X′ in the previous step:

[0109]

[0110] Then perform eigendecomposition on the covariance matrix to obtain the eigenvalue λk and the corresponding eigenvector vk:

[0111]

[0112] Arrange the eigenvectors vk in the order of dimensionality reduction of the eigenvalue λk, determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0113] W=[v′1, v′2,…,v′ k ],λ 1′ >λ 2′ >…>λ k′

[0114] Use W to project the vector group X′ onto the k-dimensional space and obtain X″:

[0115] X″=X′W

[0116] Thus, we can obtain further dimensionally reduced vector cluster data X″, where the dimension of the vector set is further reduced by half to D″.

[0117] Step 2: Large-scale high-dimensional vector query index construction and dynamic update steps:

[0118] The fusion compression method solves the problems of too high vector dimension, low information density and high redundancy, and realizes the compression of high-dimensional data. However, compressing high-dimensional vectors alone is of limited help in solving the similarity retrieval problem of large-scale vector data. When the data set is large, the retrieval process still requires a lot of computing power, so it is still necessary to design an approximate retrieval algorithm to realize the retrieval of large-scale data sets. This scheme improves the product quantization method to support the re-clustering of sub-vectors in a small range of low-dimensional subspaces, and at the same time realizes the dynamic update of the retrieval index for continuous large-scale new data by limiting the re-clustering threshold. The fusion compression method based on the distribution and covariance characteristics of high-dimensional vector data is used to reduce the dimension of the vector group X of size N and dimension D to the vector group X" of dimension D", and the size of the vector group remains unchanged. For each column of the vector group X":

[0119] X″=[X″0,X″1,…,X″ D″-1 ]

[0120] At this time, the vector set X″ before the product quantization process is as follows Figure 1 As shown. The column dimension is divided equally, and there exists m that can divide D″, and the original D″-dimensional space is divided into the Cartesian product of m D″ / m-dimensional subspaces. The vector set X″ corresponding to the original high-dimensional space is divided into sub-vector sets Y in the corresponding m low-dimensional subspaces. j :

[0121]

[0122] X″=[Y0,Y1,…,Y m-1 ]

[0123] This partition is denoted as P. At this time, the partitioned vector set X″ is used as follows Figure 2 As shown:

[0124] The set of subvectors Y in each low-dimensional subspacej Perform clustering and summarize the original N sub-vectors into k clusters C jl middle:

[0125] Y j =(y 0j ,y 1j , …, y N-1,j ), j = 0, 1, ..., m-1

[0126]

[0127] From this we can get the clustering results:

[0128] Y j ={C jl |l=0,1,…,k-1}

[0129] The structure of the data set after clustering in the divided low-dimensional subspace is as follows Figure 3 As shown:

[0130] In each cluster C jl The center vector is calculated as the centroid of the cluster. The mean method is used in this study:

[0131]

[0132] Use cluster centroid to represent cluster C jl For all other vectors in , the previous expression evolves to:

[0133]

[0134] Generate a search index based on the above results and save it in the following two sets of data:

[0135] 1) centroid_all matrix, with dimension k×m×D″ / m, stores the centroid vectors of each cluster in the m subspaces, with dimension D″ / m

[0136] 2)clustered_label_result_summarized matrix, dimension is N×m, saves the cluster number of each subvector in the m subspace

[0137] When a large number of new vectors are added to the data set, each low-dimensional subspace needs to be re-clustered within a limited range to update the query index. In this study, the range is limited to the part of each low-dimensional subspace that is closest to the cluster where the newly added vector is located, denoted as diffuse_range. For the newly added vector v, after fusion and dimensionality reduction, it is split into low-dimensional subspaces using the same partition m:

[0138] v=[v0,v1,…,v m-1 ]

[0139] For the subvector v in the jth low-dimensional subspace j According to the previous update algorithm, it has been added to the cluster closest to its centroid, denoted as C jv Then find the distance cluster C in the i-th low-dimensional subspace jv The nearest clusters are found by comparing the low-dimensional subspace with the cluster C. iv All other clusters except C jw The centroid c jw With cluster C jv The centroid c jv The distance between them is 0. The shortest first diffuse_range is selected. The clusters corresponding to these centroids are the same as cluster C. jv Participate in the re-gathering class together.

[0140] Step 3: Verification method of large-scale high-dimensional vector similarity retrieval technology:

[0141] In the present invention, the previous fusion compression method and index creation and update method are integrated to form an independent system, which can effectively manage large-scale high-dimensional vector data around the query index. According to the research and improvement of the structure of existing large-scale vector management systems, such as faiss, SPFresh, HQANN, Milvus, etc., a compression retrieval verification system for large-scale high-dimensional vectors is constructed. The core of the system is to maintain the query index of large-scale high-dimensional vector data sets, including its construction and query process, and at the same time supports the processing of a large number of newly added vectors to join the data set and its low-dimensional subspace component data to join the existing retrieval index, and supports the real-time dynamic update of large quantities of newly added data. Taking faiss as an example, as a similarity retrieval method library, it contains many ANNS methods, but most of them have less support for updates. This system is mainly aimed at the dynamic update of the index of continuously added data. Whether to use compression preprocessing for data in faiss depends on which method is used, and has little to do with faiss itself. For example, the product quantization in faiss is not compressed. If the product quantization method in faiss also uses compression preprocessing, the difference from this system is the support of the index for dynamic update of new data.

[0142] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0143] The present invention also proposes Figure 6A large-scale high-dimensional vector nearest neighbor data retrieval device B is shown, characterized in that it includes:

[0144] The initial module M1 obtains the original database, takes each feature vector in the original database as a row and the index of each feature as a column to form an original vector group of the original database, and compresses the row vector of the vector group to obtain a compressed vector group;

[0145] The index building module M2 divides the compressed vector group into m low-dimensional subspaces in the column dimension, and obtains multiple clusters of each low-dimensional subspace by clustering the subvectors of each low-dimensional subspace in the row dimension; and uses the central vector of each cluster as an index to represent all subvectors in the cluster;

[0146] Retrieval module M3 obtains the query vector to be retrieved and compresses the query vector to obtain a compressed vector, divides the compressed vector into m low-dimensional sub-vectors in the column dimension, calculates the distance between the low-dimensional sub-vector in each column and the central vector of each cluster, selects the cluster with the closest distance as the candidate cluster, obtains the candidate cluster corresponding to each column, counts the row with the most appearances in all candidate clusters as the target row, and outputs the feature vector of the target row in the original vector group as the query result of the query vector.

[0147] The large-scale high-dimensional vector nearest neighbor data retrieval device shown is characterized by comprising:

[0148] The index update module obtains the newly added vector, compresses the newly added vector and divides it into m low-dimensional sub-vectors in the column dimension, which are used as the newly added sub-vectors in the m low-dimensional subspaces, and re-clustering the sub-vectors of each low-dimensional subspace in the row dimension, and updating multiple clusters in each low-dimensional subspace.

[0149] The large-scale high-dimensional vector nearest neighbor data retrieval device is characterized by comprising:

[0150] The compression includes:

[0151] The size of the compressed vector X is N and the dimension is D:

[0152] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0153] According to the expected μ of the data in the row dimension j Calculate the standard deviation σ of the data in the row dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X in the row dimension:

[0154]

[0155] The row dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain

[0156]

[0157] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0158]

[0159] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0160] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0161] Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″:

[0162] X″=X′W.

[0163] The large-scale high-dimensional vector nearest neighbor data retrieval device is characterized in that the query vector is a feature vector of image, text, video or audio data; and the multiple feature vectors included in the original database are feature vectors of image, text, video or audio data respectively.

[0164] like Figure 7 As shown, the present invention further proposes a first electronic device A in another embodiment, comprising the large-scale high-dimensional vector nearest neighbor data retrieval device.

[0165] like Figure 8 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain the feature vector of the query to be retrieved, and the information display device D is used to display the query results obtained by the analysis of the present invention.

[0166] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, such as news updates, system information, etc., so that the user can understand this information more timely without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0167] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the large-scale high-dimensional vector nearest neighbor data retrieval method. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0168] Fig. 9A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the present invention, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present invention described and / or required in the present invention. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0169] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0170] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0171] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S3. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0172] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A large-scale high-dimensional vector nearest neighbor data retrieval method, characterized by: include: In the initial step, an original database is obtained, each feature vector in the original database is used as a row, and the index of each feature is used as a column to form an original vector group of the original database, and the row vector of the vector group is compressed to obtain a compressed vector group; In the index construction step, the compressed vector group is divided into m low-dimensional subspaces in the column dimension. The subvectors of each low-dimensional subspace are clustered in the row dimension to obtain multiple clusters in each low-dimensional subspace. The center vector of each cluster is used as an index to represent all subvectors in the cluster. In the retrieval step, the query vector to be retrieved is obtained and compressed to obtain a compressed vector. The compressed vector is divided into m low-dimensional sub-vectors in the column dimension. The distance between the low-dimensional sub-vector in each column and the center vector of each cluster is calculated. The cluster with the closest distance is selected as the candidate cluster. The candidate cluster corresponding to each column is obtained. The row with the most occurrences in all candidate clusters is counted as the target row. The feature vector of the target row in the original vector group is output as the query result of the query vector.

2. The large-scale high-dimensional vector nearest neighbor data retrieval method as claimed in claim 1, characterized in that: include: In the index update step, a new vector is obtained, and after compressing the new vector, it is divided into m low-dimensional sub-vectors in the column dimension, which are respectively used as new sub-vectors in the m low-dimensional subspaces. The sub-vectors of each low-dimensional subspace are re-clustered in the row dimension, and multiple clusters in each low-dimensional subspace are updated.

3. The large-scale high-dimensional vector nearest neighbor data retrieval method according to claim 1 or 2, characterized in that: include: The compression includes: The size of the compressed vector X is N and the dimension is D: X j ={X ij |i=0,1,…,N},j=0,1,…,D According to the expected μ of the data in the row dimension j Calculate the standard deviation σ of the data in the row dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X in the row dimension: Zero-mean the row dimension of the compressed intermediate vector X′ of dimension D′, and we get Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k : According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W: W=[v′1,v′2,…,v′ k ],λ1′>λ2′>…>λ k ′ Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″: X″=X′W.

4. The large-scale high-dimensional vector nearest neighbor data retrieval method according to claim 1 or 2, characterized in that: The query vector is a feature vector of image, text, video or audio data; and the multiple feature vectors included in the original database are feature vectors of image, text, video or audio data respectively.

5. A large-scale high-dimensional vector nearest neighbor data retrieval device, characterized in that: include: The initial module obtains the original database, takes each feature vector in the original database as a row and the index of each feature as a column to form an original vector group of the original database, and compresses the row vector of the vector group to obtain a compressed vector group; The index construction module divides the compressed vector group into m low-dimensional subspaces in the column dimension, clusters the subvectors of each low-dimensional subspace in the row dimension, and obtains multiple clusters in each low-dimensional subspace. The central vector of each cluster is used as an index to represent all subvectors in the cluster. The retrieval module obtains the query vector to be retrieved and compresses the query vector to obtain a compressed vector. The compressed vector is divided into m low-dimensional sub-vectors in the column dimension. The distance between the low-dimensional sub-vector in each column and the center vector of each cluster is calculated. The cluster with the closest distance is selected as the candidate cluster, and the candidate cluster corresponding to each column is obtained. The row with the most occurrences in all candidate clusters is counted as the target row, and the feature vector of the target row in the original vector group is output as the query result of the query vector.

6. The large-scale high-dimensional vector nearest neighbor data retrieval device as claimed in claim 5, characterized in that: include: The index update module obtains the newly added vector, compresses it, and divides it into m low-dimensional subvectors in the column dimension. These subvectors are used as new subvectors in the m low-dimensional subspaces. The subvectors in each low-dimensional subspace are re-clustered in the row dimension, and multiple clusters in each low-dimensional subspace are updated.

7. The large-scale high-dimensional vector nearest neighbor data retrieval device according to claim 5 or 6, characterized in that: include: The compression includes: The size of the compressed vector X is N and the dimension is D: X j ={X ij |i=0,1,…,N},j=0,1,…,D According to the expected μ of the data in the row dimension j Calculate the standard deviation σ of the data in the row dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X in the row dimension: Zero-mean the row dimension of the compressed intermediate vector X′ of dimension D′, and we get Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k : According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W: W=[v′1v′2,…,v′ k ],λ1′>λ2′>…>λ k ′ Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″: X″=X′W.

8. The large-scale high-dimensional vector nearest neighbor data retrieval device according to claim 5 or 6, characterized in that: The query vector is a feature vector of image, text, video or audio data; and the multiple feature vectors included in the original database are feature vectors of image, text, video or audio data respectively.

9. An electronic device, characterized in that: It includes a large-scale high-dimensional vector nearest neighbor data retrieval device as described in claims 5-8; the electronic device is connected to an information display device, and the information display device is used to display the query results according to display parameters and attributes set by the user or through an artificial intelligence model.

10. A storage medium for storing a computer program for executing the large-scale high-dimensional vector nearest neighbor data retrieval method according to claims 1-4.

Citation Information

Cited By

  • Joint optimization method and system of index and representation model and readable storage medium

    CN119988386A

  • Similarity search method and system based on vector database

    CN120179872A

  • A similarity search method and system based on vector database

    CN120179872B

  • Vector query method, electronic equipment, storage medium and program product

    CN120371838A

  • Data processing method and device, storage medium and computer program product

    CN121681527A