Vector data learning type indexing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a vector data learning type indexing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence and databases, the method comprises construction operation, retrieval operation and updating operation of an index structure, and the index structure is composed of a mapping relation table from cluster numbers to vector ID sets and a classification model. According to the construction operation, reference vectors in a vector data management system are divided, a mapping relation table from cluster numbers to vector ID sets is formed, and meanwhile, a classification model is trained to be used for predicting L-nearest neighbor clusters of query vectors. In the retrieval operation, candidate vectors are collected through one-time prediction of the classification model, and k designated candidate vectors with the closest distance are returned by calculating the preset distance between the query vector and the candidate vectors. The updating operation updates the index structure in real time according to addition, deletion, modification and change of the reference vector. The invention provides a construction method and a dynamic updating mechanism of a vector data learning type index structure, and more efficient approximate nearest neighbor search is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and database technology, and in particular to a vector data learning-based indexing method, apparatus, device, and storage medium. Background Technology

[0002] Nearest neighbor retrieval for vector data is a key technology in modern artificial intelligence and big data applications, widely used in multimedia search, semantic document retrieval, and recommender systems. With the development of deep learning, various structured and unstructured data can be mapped into high-dimensional vectors through embedding techniques, and the similarity between vectors can be measured using distance or cosine similarity. However, as the scale and dimensionality of data increase, traditional exact nearest neighbor search (NNS) methods face significant challenges in terms of efficiency and scalability.

[0003] To address these challenges, Approximate Nearest Neighbor Search (ANNS) techniques have emerged. ANNS sacrifices some search accuracy in exchange for higher efficiency and better scalability. Common ANNS methods include hash-based indexing (such as LSH), graph-based indexing (such as HNSW and NSG), tree-based indexing, and product quantization. These methods perform well when handling large-scale, high-dimensional data, but they still have some limitations.

[0004] First, there are the issues of memory limitations and cost. Traditional nearest neighbor search methods typically require storing the index structure and raw data in memory to achieve fast retrieval. However, with the explosive growth of data volume, memory resources become a bottleneck. For example, memory-based algorithms such as HNSW require a large amount of memory when processing large-scale data, resulting in high costs.

[0005] Secondly, there is the efficiency bottleneck in high-dimensional data. In high-dimensional spaces, traditional nearest neighbor search methods easily degenerate into linear scans, resulting in extremely low efficiency. Although the ANNS method improves efficiency through approximate search, it still struggles to simultaneously meet the requirements of low latency and high recall in high-dimensional data scenarios. For example, hash-based indexes (such as LSH) offer fast retrieval speeds but have low recall rates.

[0006] Finally, there is the performance bottleneck of disk I / O. For disk-based retrieval methods (such as DiskANN), the efficiency of disk I / O operations directly affects retrieval performance. Large-scale adjacency lists may lead to frequent disk accesses, thereby reducing retrieval speed.

[0007] In summary, although ANNS technology has made significant progress in the field of vector data retrieval, existing retrieval methods still face limitations such as memory constraints, efficiency bottlenecks in high-dimensional data, and difficulties in index updates. The field of vector data retrieval urgently needs better solutions, further optimization of hybrid disk and memory management, improvement of dynamic update mechanisms for index structures, and exploration of more efficient approximate search algorithms. Summary of the Invention

[0008] In view of this, the present invention provides a vector data learning-based indexing method, apparatus, device, and storage medium.

[0009] The specific technical solution provided by this invention is as follows: a vector data learning indexing method, applied to a vector data management system, the method including index structure construction operation, index structure retrieval operation, and index structure update operation; the index structure construction operation includes: obtaining an index construction request submitted by a user, the index construction request including a vector two-dimensional table, each row of the vector two-dimensional table corresponding to a vector, including an ID column and a vector data column, the vectors in the vector two-dimensional table are called reference vectors; based on all the reference vectors, incrementally constructing a mapping relationship table and a cluster center vector table in memory, the mapping relationship table consisting of mapping records in the form of "cluster number -> vector ID set", the vector ID set of the mapping record is non-empty and consists of no more than N Composed of vector IDs, N The vector set corresponding to the vector ID set is called a cluster, and the cluster number is set to the upper limit of the preset cluster size. i The cluster is called the first i There are several clusters; the preset distance between each reference vector and each cluster center vector is calculated based on the cluster center vector table, and the value of each reference vector is obtained based on the preset distance. L - A list of nearest neighbor cluster numbers, L This is a preset upper limit on the number of nearest neighbor clusters to be searched, and the entirety of the number list. L Each element is the one with the smallest preset distance to the reference vector. L The cluster number corresponding to each cluster center vector; the complete list of the numbers. L The elements are arranged in ascending order according to the preset distance between the reference vector and the corresponding cluster center vector; obtain all the aforementioned reference vectors. L After compiling the nearest neighbor cluster number list, clear the cluster center vector table and release the memory space occupied by the cluster center vector table; by gradually building... L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L- A list of nearest neighbor cluster numbers; using the vector data of the reference vector as input to the training samples, and adjusting the reference vector... L - The list of nearest neighbor cluster numbers is used as the output to construct training samples; a classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with the cluster numbers. The revised loss function machine learning algorithm refers to replacing the loss function of the original machine learning algorithm with the one that has the highest predicted probability value among the training samples. L List of category tags and their corresponding L - A machine learning model is obtained by weighting the pre-defined probability error among the nearest neighbor clusters' indexes; the model calculates the probability values ​​predicted by the trained classification model for each reference vector. L A list of class labels, used as the prediction of the reference vector. L - A list of nearest neighbor cluster numbers; predictions based on each of the reference vectors. L - The nearest neighbor cluster ID list is obtained by solving the maximum weight matching algorithm in a bipartite graph, and the mapping table is updated accordingly. The updated mapping table and the trained classification model are summarized and labeled as the index structure of the vector two-dimensional table. The index structure retrieval operation includes: obtaining the vector retrieval request submitted by the user, the vector retrieval request including the vector to be retrieved and the expected number of target vectors. k The vector to be retrieved is marked as the query vector; the query vector is input into the trained classification model to obtain the top vectors with the highest predicted probabilities. m Cluster number, m yes k and L The minimum value between, the former m The cluster number is denoted as For 1 to m Each sequence number between i Obtain the first from the mapping table. The set of vector IDs of the cluster, the th cluster The vector data corresponding to the set of vector IDs of each cluster is loaded from disk into a memory buffer and marked as candidate vectors; the preset distance between the query vector and all the candidate vectors is calculated, and the vector with the smallest preset distance is selected. k The candidate vectors are set as target vectors; if the number of target vectors is less than kIf all candidate vectors are set as target vectors, the set of all target vectors is returned to the user. The index structure update operation includes: if the data in the vector two-dimensional table with the constructed index structure changes and the change type is deleting a vector, then a delete vector operation instruction is executed, the instruction content including the vector ID to be deleted; if the data in the vector two-dimensional table with the constructed index structure changes and the change type is adding a vector, then an add vector operation instruction is executed, the instruction content including the vector ID to be added and the vector data; if the data in the vector two-dimensional table with the constructed index structure changes and the change type is modifying a vector, then a modify vector operation instruction is executed, the instruction content including the vector ID to be modified and the vector data.

[0010] In the method of this invention, a mapping table and a cluster center vector table are incrementally constructed in memory based on all the reference vectors, including: initializing the mapping table and the cluster center vector table as empty tables; when adding the first reference vector, adding "1->{first reference vector ID}" to the mapping table, and adding the first reference vector as the center vector of the first cluster to the cluster center vector table; subsequently, processing the reference vectors one by one, adding the first reference vector as the center vector of the first cluster. k When referring to the nth reference vector, find the vector in the cluster center vector table that corresponds to the nth reference vector. k Each reference vector is pre-defined as the cluster center vector with the smallest distance and its corresponding cluster number. c , the first k Add the ID of the reference vector to the first reference vector. c The vector ID set of each cluster; if the number of elements in the vector ID set of the c-th cluster exceeds a preset number. N Then the first c Vector data for all vectors in each cluster is loaded from disk into a memory buffer for processing, starting from the first... c In a cluster of vectors, locate two distinct new center vectors such that the sum of the combined distances from other vectors to these two new center vectors is minimized. The combined distance from each vector to these two new center vectors is the minimum of the preset distances between that vector and these two new center vectors. Any one of these new center vectors is labeled as the [nth] […]. c The new center vector of the cluster, and another new center vector is labeled as the first. n +1 cluster center vectors, n The number of cluster center vectors in the current cluster center vector table; the number of cluster center vectors in the current table; n The vector ID set of the +1 cluster is initialized to an empty set; for the ... c Each vector in the clusters is if and only if the vector is related to the first... c The preset distance between the new center vector of the cluster and the first cluster is greater than the distance between that vector and the first cluster. n When the preset distance to the center vector of the +1 cluster is reached, the ID of the vector is changed from the first...c The vector ID set of the cluster is moved to the . n The vector ID set of +1 clusters; according to the first... c The new center vector of the cluster and the first n Update the cluster center vector table with +1 cluster center vectors, and then... n +1->No. n A mapping record for "+1 cluster vector ID set" is added to the mapping table.

[0011] In the method of this invention, if the data in the vector two-dimensional table with the constructed index structure changes, and the change type is vector deletion, a vector deletion operation instruction is executed. The instruction content includes the vector ID to be deleted, including: marking the vector ID to be deleted as a deletion ID, and determining the cluster number corresponding to the deletion ID according to the mapping relationship table. c The deletion ID will be removed from the first... c Remove from the vector ID set of the i-th cluster; if the i-th cluster is removed from the vector ID set; c If the vector ID set of each cluster is not empty, then the currently updated mapping table and the original trained classification model are marked as the index structure of the vector two-dimensional table after executing the vector deletion operation instruction; otherwise, continue to execute subsequent operations; delete the "c->empty set" mapping record from the mapping table; for each "c->empty set" in the mapping table i ->No. i The vector ID set of the cluster is mapped to the record, and the first cluster's vector ID set is mapped to the record. i The vector data corresponding to the vector ID set of each cluster is loaded from disk into the memory buffer to locate the center vector, minimizing the sum of the preset distances between other vectors and the center vector; the center vectors of each cluster in the mapping table are summarized to form a cluster center vector table; based on the cluster center vector table, the preset distance between each reference vector and the center vector of each cluster is calculated to obtain the center vector of each reference vector. L - A list of nearest neighbor cluster numbers; the entire list of numbers. L Each element is the one with the smallest preset distance to the reference vector. L The cluster number corresponding to each cluster center vector; the complete list of the numbers. L The elements are arranged in ascending order according to the preset distance between the reference vector and the corresponding cluster center vector; obtain all the aforementioned reference vectors. L After compiling the nearest neighbor cluster number list, clear the cluster center vector table and release the memory space occupied by the cluster center vector table; by gradually building... L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L- A list of nearest neighbor cluster numbers; using the vector data of the reference vector as input to the training samples, and adjusting the reference vector... L - The list of nearest neighbor cluster numbers is used as the output to construct training samples; a classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with the cluster numbers. The revised loss function machine learning algorithm refers to replacing the loss function of the original machine learning algorithm with the one that has the highest predicted probability value among the training samples. L List of category tags and their corresponding L - A machine learning model is obtained by weighting the pre-defined probability error among the nearest neighbor clusters' indexes; the model calculates the probability values ​​predicted by the trained classification model for each reference vector. L A list of class labels, used as the prediction of the reference vector. L - A list of nearest neighbor cluster numbers; predictions based on each of the reference vectors. L - The nearest neighbor cluster number list is solved using the bipartite graph maximum weight matching algorithm, and the mapping relationship table is updated; the updated mapping relationship table and the trained classification model are summarized and marked as the index structure of the vector two-dimensional table after the vector deletion operation instruction is executed.

[0012] In the method of this invention, if the data in the vector two-dimensional table with the constructed index structure changes, and the change type is adding a vector, then an add vector operation instruction is executed. The instruction content includes the vector ID to be added and the vector data, including: marking the ID of the vector to be added as an insertion ID, and marking the vector data to be added as an insertion vector; inputting the insertion vector into the trained classification model to obtain the top vector with the highest prediction probability. D Cluster number, denoted as , D To meet The preset maximum number of insertion attempts; incremented by 1 for each attempt. D between natural numbers i Obtain the first [number] according to the mapping table. c i The vector ID set of each cluster S i Add the insertion ID to S i In the middle; if S i The number of elements in the middle does not exceed N If so, no further processing will be done; otherwise, from S i Delete the inserted ID; if the inserted ID is added to a certain... S iIf no further processing is performed, the currently updated mapping table and the original trained classification model are marked as the index structure of the vector two-dimensional table after executing the vector addition operation instruction; otherwise, continue with subsequent operations; add the insertion ID to S In 1, and the first c The vector data of all vectors in a cluster are loaded from disk into a memory buffer. Two distinct new center vectors are located such that the sum of the comprehensive distances from other vectors to these two new center vectors is minimized. The comprehensive distance from each vector to these two new center vectors is the minimum of the preset distances between that vector and these two new center vectors. Any one of these new center vectors is labeled as the [number missing]. c One new center vector of a cluster, and another new center vector set is the first... n +1 cluster center vectors, n The current largest cluster number; the first... n The vector ID set of the +1 cluster is initialized to an empty set; for the ... c Each vector in a cluster is such that it is associated with the first vector. c The preset distance of the new center vector of a cluster is greater than the distance between that vector and the first cluster. n When the preset distance to the center vector of the +1 cluster is reached, the ID of the vector is changed from the first... c The vector ID set of a cluster is moved to the first... n +1 clusters of vector IDs; will " n +1->No. n A mapping record for "+1 clusters of vector ID sets" is added to the mapping table; for each of the reference vectors including the inserted vector... v Call the trained classification model to predict v The highest probability of being in the front L Cluster number, denoted as The mapping table will uniquely contain v The cluster number corresponding to the vector ID set of the corresponding ID is marked as c 0; if there exists 1 to 0; L One of the middle i Make Then Marked as v of L - A list of nearest neighbor cluster numbers, otherwise... Marked as v of L - A list of nearest neighbor cluster numbers; built step by step. L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L- A list of nearest neighbor cluster numbers; using the vector data of the reference vector as input to the training samples, and adjusting the reference vector... L - The list of nearest neighbor cluster numbers is used as the output to construct training samples; a classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with the cluster numbers. The revised loss function machine learning algorithm refers to replacing the loss function of the original machine learning algorithm with the one that has the highest predicted probability value among the training samples. L List of category tags and their corresponding L - A machine learning model is obtained by weighting the pre-defined probability error among the nearest neighbor clusters' indexes; the model calculates the probability values ​​predicted by the trained classification model for each reference vector. L A list of class labels, used as the prediction of the reference vector. L - A list of nearest neighbor cluster numbers; predictions based on each of the reference vectors. L - The nearest neighbor cluster number list is solved using the bipartite graph maximum weight matching algorithm, and the mapping relationship table is updated; the updated mapping relationship table and the trained classification model are summarized and marked as the index structure of the vector two-dimensional table after executing the vector addition operation instruction.

[0013] In the method of this invention, if the data in the vector two-dimensional table with the constructed index structure changes, and the change type is vector modification, a vector modification operation instruction is executed. The instruction content includes the vector ID to be modified and the vector data, including: marking the vector data to be modified as an insertion vector, generating a new ID that does not exist in the mapping table and marking it as an insertion ID, generating and executing an add vector operation instruction based on the insertion ID and the insertion vector; marking the vector ID to be modified as a deletion ID, generating and executing a delete vector operation instruction based on the deletion ID; replacing the insertion ID with the deletion ID in the mapping table; summarizing the updated mapping table and the trained classification model, and marking it as the index structure of the vector two-dimensional table after executing the vector modification operation instruction.

[0014] In the method of this invention, by step-by-step construction L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers, including: constructing the first bipartite graph maximum weight matching problem, wherein constructing the first bipartite graph maximum weight matching problem includes defining two node sets. V and U ,in V Composed of reference vector ID,U Cluster number N This consists of multiple copies; assuming the ID is... v The reference vector L -Nearest neighbor clusters are numbered sequentially as follows Then for 1 to L Each between i ,Give v Corresponding nodes and ci corresponding N Each copy node is weighted. The edge, give v Corresponding nodes and node sets U Add weights to the remaining nodes in the array. The edge, of which It is a return value and an argument. i A preset function of inversely proportional integer values; solving the first bipartite graph maximum weight matching problem to obtain a first matching solution, and processing each reference vector according to the first matching solution; the processing of each reference vector according to the first matching solution includes assuming the reference vector ID is... v And in the first matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then the reference vector will be... L - Replace the first cluster number in the nearest neighbor cluster number list with c and will v Corresponding node and cluster number c The edge weights of all corresponding copied nodes are changed to 0; after all the reference vectors have been processed, the resulting bipartite graph is marked as the bipartite graph of the second bipartite graph maximum weight problem; the second bipartite graph maximum weight matching problem is solved to obtain the second matching solution, and each of the reference vectors is processed according to the second matching solution; the process of processing each reference vector according to the second matching solution includes assuming the reference vector ID is... v And in the second matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then the reference vector will be... L - Replace the second cluster number in the nearest neighbor cluster number list with c and will v Corresponding node and cluster number c Set the edge weights of all corresponding copied nodes to 0; after processing all the reference vectors, mark the resulting bipartite graph as the bipartite graph for solving the third maximum weight bipartite graph problem; and so on, until the solution of the third maximum weight bipartite graph problem is obtained. L The maximum weight matching problem in a bipartite graph yields the result of the following steps. LAfter finding a matching solution to the maximum weight matching problem in a bipartite graph, each of the reference vectors is processed in the same way to finally obtain the adjusted value for each of the reference vectors. L - A list of nearest neighbor cluster numbers.

[0015] In the method of the present invention, based on the prediction of each of the reference vectors L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm for a bipartite graph, and the mapping table is updated, including: constructing the final bipartite graph maximum weight matching problem; the final bipartite graph maximum weight matching problem includes defining two node sets. V and U ,in V Composed of reference vector ID, U Cluster number N This consists of multiple copies; assuming the ID is... v The reference vector L -Nearest neighbor clusters are numbered sequentially as follows Then for 1 to L Each between i ,Give v Corresponding nodes and ci corresponding N Each copy node is weighted. The edge, give v Corresponding nodes and node sets U Add weights to the remaining nodes in the array. The edge, of which It is a return value and an argument. i A preset function with inversely proportional integer values; resetting the vector ID set of each mapping in the mapping table to an empty set, solving the final bipartite graph maximum weight matching problem to obtain the final matching solution; processing each reference vector according to the final matching solution; the process of processing each reference vector according to the final matching solution includes assuming the reference vector ID is... v And in the final matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then... v Cluster number added to the mapping table c The corresponding vector ID set.

[0016] The present invention also provides a vector data learning indexing device, comprising: an index structure construction module for performing the index structure construction operation; an index structure retrieval module for performing the index structure retrieval operation; and an index structure update module for performing the index structure update operation.

[0017] The present invention also provides a computer device, including a processor and a memory; the processor is used to execute a computer program stored in the memory to implement the above-described vector data learning indexing method.

[0018] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described vector data learning indexing method.

[0019] Implementing the embodiments of the present invention will have the following beneficial effects: This invention provides an Approximate Nearest Neighbor Search (ANNS) algorithm based on a learned index structure, which can collect data through a single prediction by a classification model. mN Given a list of candidate vectors, the query vector is calculated and a specified distance between the candidate vectors is returned, along with the closest candidate vector. k The execution time and space complexity of a candidate vector are independent of the number of reference vectors, thus exhibiting extremely high execution efficiency and space utilization.

[0020] Furthermore, this invention proposes a method for constructing a learning-based index structure suitable for ANNS, which can automatically construct all training samples for a classification model based on reference vectors, and use machine learning techniques to build a classification model for predicting query vectors. L -Nearest neighbor clusters, where the class labels of the classification model correspond one-to-one with the cluster numbers, according to... L - The nearest neighbor cluster list can be used to collect candidate vectors that have the smallest preset distance to the query vector. This ensures that the number of vectors in each cluster does not exceed [a certain threshold]. N This invention introduces cluster adjustment steps before and after the classification model training phase. By transforming the cluster adjustment problem into a bipartite graph maximum weight matching problem, it ensures that cluster adjustment can be completed in polynomial time. The time cost of constructing the index structure is equal to the number of reference vectors. M The polynomial level has a space requirement of approximately equal to N + M / N This reduces the amount of memory used by using multiple vectors.

[0021] Furthermore, the learning-based index structure update method based on the present invention can update the index structure according to the real-time changes of the reference vector, without having to build the index structure from scratch, and can avoid high-cost updates of the index structure as much as possible. Even if high-cost updates of the index structure cannot be avoided, such updates can be performed asynchronously to reduce the impact on online retrieval, thereby ensuring the continuity and stability of online retrieval services. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating the steps of a learning-based indexing method for vector data; Figure 2 This is a schematic diagram of the index structure; Figure 3 This is an example graph for the maximum weight matching problem in a bipartite graph; Figure 4 This is a schematic diagram of the structure of a vector data learning indexing device; Figure 5 A schematic block diagram of the structure of a computer device; Among them, 201 is the index structure construction module; 202 is the index structure retrieval module; and 203 is the index structure update module. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such processes, methods, products, or apparatus.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] Nearest neighbor retrieval for vector data is a key technology in modern artificial intelligence and big data applications, widely used in multimedia search, semantic document retrieval, and recommender systems. With the development of deep learning, various structured and unstructured data can be mapped into high-dimensional vectors through embedding techniques, and the similarity between vectors can be measured using distance or cosine similarity. However, as the scale and dimensionality of data increase, traditional exact nearest neighbor search (NNS) methods face significant challenges in terms of efficiency and scalability.

[0028] To address these challenges, Approximate Nearest Neighbor Search (ANNS) techniques have emerged. ANNS sacrifices some search accuracy in exchange for higher efficiency and better scalability. Common ANNS methods include hash-based indexing (such as LSH), graph-based indexing (such as HNSW and NSG), tree-based indexing, and product quantization. These methods perform well when handling large-scale, high-dimensional data, but they still have some limitations.

[0029] First, there are the issues of memory limitations and cost. Traditional nearest neighbor search methods typically require storing the index structure and raw data in memory to achieve fast retrieval. However, with the explosive growth of data volume, memory resources become a bottleneck. For example, memory-based algorithms such as HNSW require a large amount of memory when processing large-scale data, resulting in high costs.

[0030] Secondly, there is the efficiency bottleneck in high-dimensional data. In high-dimensional spaces, traditional nearest neighbor search methods easily degenerate into linear scans, resulting in extremely low efficiency. Although the ANNS method improves efficiency through approximate search, it still struggles to simultaneously meet the requirements of low latency and high recall in high-dimensional data scenarios. For example, hash-based indexes (such as LSH) offer fast retrieval speeds but have low recall rates.

[0031] Finally, there is the performance bottleneck of disk I / O. For disk-based retrieval methods (such as DiskANN), the efficiency of disk I / O operations directly affects retrieval performance. Large-scale adjacency lists may lead to frequent disk accesses, thereby reducing retrieval speed.

[0032] In summary, although ANNS technology has made significant progress in the field of vector data retrieval, existing retrieval methods still face limitations such as memory constraints, efficiency bottlenecks in high-dimensional data, and difficulties in index updates. The field of vector data retrieval urgently needs better solutions, further optimization of hybrid disk and memory management, improvement of dynamic update mechanisms for index structures, and exploration of more efficient approximate search algorithms.

[0033] This invention proposes a vector data learning-based indexing method to explore more efficient approximate search algorithms. Please refer to [link to relevant documentation]. Figure 1 This includes steps S110 to S130.

[0034] Step S110: Index structure construction operation.

[0035] In the method of this invention, the index structure construction operation includes: The system retrieves the user-submitted index building request, which includes a vector two-dimensional table. Each row of the vector two-dimensional table corresponds to a vector, containing an ID column and a vector data column. The vectors in the vector two-dimensional table are called reference vectors. In this embodiment, an index structure is added to the vector two-dimensional table to improve the approximation of the query vector. k - Nearest neighbor vector retrieval efficiency. This index structure consists of a mapping table of "cluster number -> vector ID set" and a classification model, where... L This is a preset upper limit on the number of nearest neighbors to be retrieved. Please refer to the schematic diagram of the index structure in this embodiment. Figure 2 .

[0036] Based on all reference vectors, an incremental mapping table and a cluster center vector table are constructed in memory. The mapping table consists of mapping records in the form of "cluster number -> vector ID set". The vector ID set of each mapping record is non-empty and consists of no more than [number missing] vector IDs. N Composed of vector IDs, N The vector set corresponding to the vector ID set is called a cluster, and the cluster number is set to the upper limit of the preset cluster size. i The cluster is called the first i Clusters. Specifically, they include: Initialize the "cluster number -> vector ID set" mapping table and the cluster center vector table to empty tables.

[0037] When adding the first reference vector, add "1->{first reference vector ID}" to the mapping table, and add the first reference vector as the center vector of the first cluster to the cluster center vector table.

[0038] Then, the reference vectors are processed one by one, and the first one is added. k When referring to the nth reference vector, find the vector in the cluster center vector table that is related to the nth reference vector. k Each reference vector is pre-defined as the cluster center vector with the smallest distance and its corresponding cluster number. c (If there are multiple cluster center vectors with the smallest preset distance, then) c Defined as any cluster number with the fewest elements in the corresponding vector ID set in the mapping table, the first... k Add the ID of the reference vector to the first reference vector. c The vector ID set of each cluster.

[0039] If the firstc The number of elements in the vector ID set of each cluster exceeds the preset number. N Then the first c Vector data for all vectors in each cluster is loaded from disk into a memory buffer for processing, starting from the first... c In a cluster of vectors, locate two distinct new center vectors such that the sum of the combined distances from all other vectors to these two new center vectors is minimized. The combined distance from each vector to these two new center vectors is the minimum of the predetermined distances between that vector and these two new center vectors. u 1 and u 2 is the first c Minimize in clusters Two distinct vectors, where C Indicates the first c The clusters are named dist, where dist is a preset distance function. Any new center vector is labeled as the nth cluster. c The new center vector of the cluster, and another new center vector is labeled as the first. n +1 cluster center vectors, n This represents the number of cluster center vectors in the current cluster center vector table. The [number]th [vector]... n The vector ID set of the +1 cluster is initialized to an empty set, and then for the... c Each vector in the clusters is if and only if the vector is related to the first... c The preset distance between the new center vector of the cluster and the first cluster is greater than the distance between that vector and the first cluster. n When the preset distance to the center vector of the +1 cluster is reached, the ID of the vector is changed from the first... c The vector ID set of the cluster is moved to the . n The vector ID set of +1 clusters. Finally, according to the... c The new center vector of the cluster and the first n Update the cluster center vector table with +1 cluster center vectors, and then... n +1->No. n A mapping record for "+1 cluster vector ID set" is added to the mapping table.

[0040] Based on the cluster center vector table, calculate the preset distance between each reference vector and each cluster center vector, and obtain the value of each reference vector based on the preset distance. L - A list of nearest neighbor cluster numbers, where L This is the preset upper limit for the number of nearest neighbor clusters to be searched. (All numbers in the list) L Each element is related to the reference vector. v The minimum preset distance L The cluster number corresponding to each cluster center vector. In the list of numbers, this... L Each element is based on the reference vector. v Sort all reference vectors in ascending order by their preset distances to the corresponding cluster center vectors.L - After listing the nearest neighbor clusters, clear the cluster center vector table and release the memory space occupied by the cluster center vector table.

[0041] Because of all reference vectors L -Nearest neighbor clusters are not necessarily uniformly distributed in the cluster space, which leads to certain... The first of all reference vectors i The nearest neighbor center vector may tend to appear in certain clusters, causing the number of vectors in these clusters to exceed the preset upper limit. N Therefore, in order to obtain more accurate training samples to construct the query vector for prediction, L - The classification model for the cluster containing the nearest neighbor center vector; this embodiment addresses each... Construct the first i The maximum weight matching problem in a bipartite graph is solved, and the i-th maximum weight matching problem is obtained by solving this problem. i The expected distribution of the k nearest neighbor center vector in the cluster space. This expected distribution needs to guarantee the k-th nearest neighbor center vector of all reference vectors. i The nearest neighbor center vector appears no more than a preset upper limit in each cluster. This is achieved by gradually constructing... L Solve the maximum weight matching problem in a bipartite graph by adjusting each reference vector. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers.

[0042] Specifically, a bipartite graph is a special type of graph where the set of nodes is divided into two disjoint sets. U and V This ensures that each edge in the graph is connected to a... U A node and a V In a bipartite graph, a matching is a set of edges such that no two edges share a common endpoint; that is, no two endpoints of any edge are shared with any other edge. Given a bipartite graph... ,in U and V They are two disjoint sets of nodes. It is a set of edges, each edge Each has a weight The maximum weight matching problem in a bipartite graph is... G Find a maximization Matching M ,in M yes A subset of.

[0043] The specific construction process of the first bipartite graph maximum weight matching problem includes: defining two node sets. V and U ,in V Composed of reference vector ID, U Cluster number N This consists of multiple copies; assuming the ID is... v The reference vector L -Nearest neighbor clusters are numbered sequentially as follows Then for 1 to L Each between i ,Give v Corresponding nodes and c i corresponding N Each copy node is weighted. The edge, give v Corresponding nodes and node sets U Add weights to the remaining nodes in the array. The edge, of which It is a return value and an argument. i Preset functions that are inversely proportional to integer values, such as ,in, It is a rounding function.

[0044] Specifically, please refer to Figure 3 . Figure 3 An example of the maximum weight matching problem in a bipartite graph constructed in this invention is given, wherein the node set V The node set consists of 6 reference vector IDs, namely v1, v2, ..., v6. U It consists of three copies of two clusters, c1 and c2. V and U The edge weights are determined by Give, .

[0045] Figure 3 A matching solution to the maximum weight matching problem in the bipartite graph shown is: {v1 c2_1,v2 c2_2,v3 c1_1,v4 c2_3,v5 c1_2,v6 The sum of the weights of c1_3} is 4+4+4+4+4+4=24.

[0046] Solving the first maximum weight matching problem in a bipartite graph yields the first matching solution. Each reference vector is then processed based on this first matching solution. This process includes assuming the reference vector ID is... v And in the first matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then the reference vector will be... L - Replace the first cluster number in the nearest neighbor cluster number list with c and will v Corresponding node and cluster number c The edge weights of all corresponding copied nodes are changed to 0; after all reference vectors have been processed, the resulting bipartite graph is marked as the bipartite graph of the second bipartite graph maximum weight problem.

[0047] Solving the second maximum weight matching problem in the bipartite graph yields the second matching solution. Each reference vector is then processed based on this second matching solution. This process includes assuming the reference vector ID is... v And in the second matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then the reference vector will be... L - Replace the second cluster number in the nearest neighbor cluster number list with c and will v Corresponding node and cluster number c Change the edge weights of all corresponding copied nodes to 0.

[0048] After all reference vectors have been processed, the resulting bipartite graph is marked as the bipartite graph for solving the third bipartite graph maximum weight problem; this process continues until the solution of the third bipartite graph problem is obtained. L The maximum weight matching problem in a bipartite graph yields the result of the following steps. L After finding a matching solution for the maximum weight matching problem in a bipartite graph, process each reference vector in the same way to finally obtain the adjusted value for each reference vector. L - A list of nearest neighbor cluster IDs. Specifically, assume the reference vector ID is... v And in the matching solution v With cluster number c If a copy of the vector is concatenated, then the vector will be... L -The number of the nearest neighbor clusters in the list L The cluster number is replaced with c.

[0049] Through the above-mentioned targets L - Adjustments to the nearest neighbor cluster numbering list, for different All reference vectors have been guaranteed to be of the first 100th ... i The nearest neighbor center vectors appear no more than the upper limit in each cluster. NTraining samples are constructed based on the complete set of all reference vectors, where the vector data of each reference vector serves as the input to the training samples. L - The list of nearest neighbor cluster numbers is output as the training samples. Then, a classification model is trained using a pre-defined machine learning algorithm with a revised loss function, where the class labels of the target classification model correspond one-to-one with the cluster numbers, and the loss function in the original machine learning algorithm is replaced with the cluster number of the nearest neighbor cluster with the highest predicted probability value among the training samples. L The list of class tags and their corresponding outputs are... L - A preset probability-weighted error between the nearest neighbor cluster index lists. The preset machine learning algorithm can be RandomForest, Gradient Boosting Decision Tree, Neural Network, etc. The preset probability-weighted error can be probability-weighted average absolute error, probability-weighted mean square error, etc., where the weight is the probability of the corresponding class label in the L class label lists with the highest predicted probability values.

[0050] Calculate the top-ranked reference vectors whose predicted probabilities are maximized by the trained classification model. L A list of class labels, used as a reference vector for prediction. L - A list of nearest neighbor cluster numbers.

[0051] To ensure the consistency of the index structure, it is necessary to predict the index based on the data from each reference vector. L - The list of nearest neighbor cluster IDs is used to update the "cluster ID -> vector ID set" mapping table in the index structure. However, no classification model can guarantee a perfect fit to the training samples. This means that constructing the mapping table based on the list of predicted L-nearest neighbor cluster IDs for all reference vectors cannot guarantee that the number of vectors in each cluster does not exceed a preset upper limit N. Therefore, this invention again uses the bipartite graph maximum weight matching algorithm to reconstruct the mapping table. Specifically, it includes: Constructing the final maximum weight matching problem in a bipartite graph; the final maximum weight matching problem in a bipartite graph involves defining two node sets. V and U , where the node set V The node set consists of reference vector IDs. U Cluster number N This consists of multiple copies; assuming the ID is... v Prediction of the reference vector L -Nearest neighbor clusters are numbered sequentially as follows Then for 1 to L Each between i ,Give v Corresponding nodes and ci corresponding NEach copy node is weighted. The edge, give v Corresponding nodes and node sets U Add weights to the remaining nodes in the array. The edge, of which It is a return value and an argument. i A preset function for inversely proportional integer values. Reset the vector ID set of each mapping in the "cluster number -> vector ID set" mapping table to an empty set, solve the final bipartite graph maximum weight matching problem to obtain the final matching solution; process each reference vector according to the final matching solution; process each reference vector according to the final matching solution, including assuming the reference vector ID is... v And in the final matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then... v Add cluster number to the mapping table c The corresponding vector ID set.

[0052] The updated mapping table and the trained classification model are summarized and labeled as an index structure of a two-dimensional vector table.

[0053] Step S120: Index structure retrieval operation.

[0054] In the method of the present invention, the index structure retrieval operation includes: Retrieve the vector retrieval request submitted by the user. The vector retrieval request includes the vector to be retrieved and the expected number of target vectors. k Mark the vector to be retrieved as the query vector and set... .

[0055] Input the query vector into the trained classification model and obtain the top [predicted probabilities]. m Each cluster number, will be the first m The cluster number is denoted as .

[0056] For 1 to m Each sequence number between i Retrieve the first from the mapping table The set of vector IDs for the cluster, which will be the th cluster The vector data corresponding to the set of vector IDs for each cluster is loaded from disk into a memory buffer and marked as candidate vectors. The number of candidate vectors loaded into the memory buffer does not exceed [number missing]. mN Therefore, the memory buffer only needs to be no more than 1. mN Storage space for vectors.

[0057] Calculate the preset distance between the query vector and all candidate vectors, and select the vector with the smallest preset distance. kA candidate vector is set as the target vector; if the number of target vectors is less than 1, the target vector is set as the target vector. k If so, then all candidate vectors are set as target vectors. The set of all target vectors is then returned to the user.

[0058] Step S130: Index structure update operation.

[0059] In the method of this invention, the index structure update operation includes: If the data in a vector two-dimensional table with an established index structure changes, and the change type is "delete vector", then execute the "delete vector" operation instruction, which includes the ID of the vector to be deleted.

[0060] If the data in a vector two-dimensional table with an existing index structure changes, and the change type is adding a vector, then the add vector operation instruction is executed. The instruction content includes the vector ID to be added and the vector data.

[0061] If the data in a vector two-dimensional table with an established index structure changes, and the change type is "modify vector", then the "modify vector" operation instruction is executed. The instruction content includes the vector ID to be modified and the vector data.

[0062] In the method of this invention, the basic principle of deleting vector operation instructions is to minimize high-cost updates to the index structure, that is, to reduce high-cost updates to the index structure by accumulating the effect of deleting vector IDs. Therefore, index structure reconstruction is triggered only when a set of vector IDs in the mapping table is cleared, specifically including: Mark the vector ID to be deleted as the deletion ID, and determine the cluster number corresponding to the deletion ID according to the mapping table. c That is, the first c The vector ID set of each cluster is a unique vector ID set in the mapping table that contains the deleted ID. The deleted ID is then removed from the first cluster. c Remove from the vector ID set of the i-th cluster. If the i-th cluster... c If the set of vector IDs for each cluster is not empty, then the currently updated mapping table and the original trained classification model are marked as the index structure of the two-dimensional vector table after the vector deletion operation instruction is executed; otherwise, continue to execute subsequent operations.

[0063] Delete the "c->empty set" mapping record from the mapping table.

[0064] For each " in the mapping table i ->No. i The vector ID set of the cluster is mapped to the record, and the first cluster's vector ID set is mapped to the record. i The vector data corresponding to the vector ID set of each cluster is loaded from disk into the memory buffer to locate the center vector, which minimizes the sum of the preset distances between the other vectors and the center vector, i.e., the center vector. u It is the first iMinimize in clusters The reference vector, where C i Indicates the first i There are several clusters. Finally, the center vectors of each cluster in the mapping table are summarized to form a cluster center vector table. Based on the generated cluster center vector table, the preset distance between each reference vector and the center vector of each cluster is calculated, and the value of each reference vector is obtained. L - A list of nearest neighbor cluster numbers; the entire list of numbers. L Each element is the one with the smallest preset distance to the reference vector. L The cluster number corresponding to each cluster center vector; the complete list of cluster numbers. L The elements are sorted in ascending order according to the preset distance between the reference vector and the corresponding cluster center vector; obtain all reference vectors. L - After compiling the nearest neighbor cluster number list, clear the cluster center vector table and release the memory space occupied by the cluster center vector table; this is done by gradually building... L Solve the maximum weight matching problem in a bipartite graph by adjusting each reference vector. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers; Using the reference vector's vector data as input to the training samples, the reference vector is adjusted... L - The list of nearest neighbor cluster numbers is used as the output to construct training samples; a classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with the cluster numbers. The revised loss function machine learning algorithm refers to replacing the loss function of the original machine learning algorithm with the one that has the highest predicted probability value among the training samples. L List of category tags and their corresponding L - A machine learning model is obtained by weighting the pre-defined probability errors among the nearest neighbor cluster lists; the model calculates the probability values ​​predicted by the trained classification model for each reference vector. L A list of class labels, used as a reference vector for prediction. L - A list of nearest neighbor cluster numbers; predictions based on each reference vector. L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm of bipartite graph, and the mapping table is updated.

[0065] The updated mapping table and the trained classification model are summarized and marked as the index structure of the two-dimensional vector table after the vector deletion operation instruction is executed.

[0066] In the method of this invention, the basic processing principle for adding vector operation instructions is also to avoid high-cost updates to the index structure as much as possible, that is, to reduce the high-cost updates to the index structure by accumulating the effect of adding vector IDs. To this end, the vector ID to be added is added to the set of vector IDs corresponding to the cluster numbers predicted by the classification model (hereinafter referred to as the predicted ID set). If the predicted ID set with the highest prediction probability can accommodate the vector ID to be added, then the vector ID to be added is added to that predicted ID set; otherwise, the predicted ID set with the second highest prediction probability is tried; and so on, until the vector ID to be added is added to the set with the highest prediction probability. i Predicted ID set (1≤ i ≤ D Or detect a predicted probability rank between 1 and D All predicted ID sets between D and D cannot accommodate the vector ID to be added, where D is a set of values ​​satisfying 1 ≤ D ... D ≤ L The preset maximum number of insertion attempts. Only when the predicted probability rank is between 1 and... D Only when any set of predicted IDs cannot accommodate the vector ID to be added will the vector ID to be added be added to the set of predicted IDs with the highest prediction probability, thereby splitting that set of predicted IDs and triggering index structure reconstruction. Specific steps include: Label the ID of the vector to be added as the insertion ID, and label the vector data to be added as the insertion vector. Input the insertion vector into the trained classification model to obtain the top vectors with the highest predicted probabilities. D Cluster number, denoted as , D To meet The preset maximum number of insertion attempts.

[0067] Process 1 to 1 in increments one by one D between natural numbers i Obtain the first [item] from the mapping table. c i The vector ID set of each cluster S i Add the insertion ID to S i In the middle; if S i The number of elements in the middle does not exceed N If so, no further processing will be done; otherwise, from S i Delete the inserted ID; if the inserted ID is added to a certain... S i If no further processing is performed, the currently updated mapping table and the original trained classification model will be marked as the index structure of the two-dimensional vector table after executing the vector addition operation instruction; otherwise, continue to execute subsequent operations.

[0068] Add the insertion ID to S In 1, and the first c The vector data of all vectors in a cluster is loaded from disk into a memory buffer. Two distinct new center vectors are located, such that the sum of the combined distances of all other vectors to these two new center vectors is minimized. The combined distance of each vector to these two new center vectors is the minimum of the preset distances between that vector and these two new center vectors. u 1 and u 2 is the first c Minimize in a cluster Two distinct vectors, where C 1 indicates the first c There is one cluster, and dist is a preset distance function. Then, any new center vector is labeled as the first... c One new center vector for a cluster, and another new center vector for the set. n +1 cluster center vectors, n This is the current largest cluster number. Next, the [number]th cluster... n The vector ID set of the +1 cluster is initialized to an empty set; for the ... c Each vector in a cluster is such that it is associated with the first vector. c The preset distance of the new center vector of a cluster is greater than the distance between that vector and the first cluster. n When the preset distance to the center vector of the +1 cluster is reached, the ID of the vector is changed from the first... c The vector ID set of a cluster is moved to the first... n +1 cluster vector ID set; finally, " n +1->No. n The mapping record for "+1 cluster vector ID set" is added to the mapping table. Due to the split... c The two clusters obtained from one cluster each contain at least their respective center vectors, while the vector ID sets of other clusters remain unchanged. Therefore, this step will not clear any clusters, and thus will not trigger index structure reconstruction caused by deleting empty clusters.

[0069] For each reference vector including the insertion vector v Call the trained classification model to predict v The highest probability of being in the front L Cluster number, denoted as ; The mapping table contains only one v The cluster number corresponding to the vector ID set of the corresponding ID is marked as c 0; if there exists 1 to 0; L One of the middle i Make Then Marked as v of L - A list of nearest neighbor cluster numbers, otherwise... Marked as v of L - A list of nearest neighbor cluster numbers.

[0070] By building step by step L Solve the maximum weight matching problem in a bipartite graph by adjusting each reference vector. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers; Using the reference vector's vector data as input to the training samples, the reference vector is adjusted... L - The list of nearest neighbor cluster numbers is used as the output to construct training samples; a classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with the cluster numbers. The revised loss function machine learning algorithm refers to replacing the loss function of the original machine learning algorithm with the one that has the highest predicted probability value among the training samples. L List of category tags and their corresponding L - A machine learning model is obtained by weighting the pre-defined probability errors among the nearest neighbor cluster lists; the model calculates the probability values ​​predicted by the trained classification model for each reference vector. L A list of class labels, used as a reference vector for prediction. L - A list of nearest neighbor cluster numbers; predictions based on each reference vector. L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm of bipartite graph, and the mapping table is updated.

[0071] The updated mapping table and the trained classification model are summarized and marked as the index structure of the two-dimensional vector table after executing the vector addition operation instruction.

[0072] In the method of this invention, the basic processing idea of ​​modifying vector operation instructions is to treat the vector modification operation as a combination of adding and deleting vector operations. First, the vector addition operation, which has relatively low time complexity in the worst case (does not involve rebuilding the cluster center vector table), is executed, followed by the vector deletion operation (which requires rebuilding the cluster center vector table in the worst case), in order to avoid higher-cost index structure updates as much as possible. Specific steps include: The vector data to be modified is marked as the insertion vector. A new ID that does not exist in the mapping table is generated and marked as the insertion ID. An instruction to add vector is generated based on the insertion ID and the insertion vector and executed.

[0073] Mark the vector ID to be modified as the deletion ID, generate a deletion vector operation instruction based on the deletion ID, and execute it.

[0074] Replace the inserted ID with the deleted ID in the mapping table. Since the deleted ID was removed from the mapping table in the previous step, this step will not introduce duplicate vector IDs into the mapping table.

[0075] The updated mapping table and the trained classification model are summarized and marked as the index structure of the two-dimensional vector table after executing the vector modification operation instruction.

[0076] Let's first analyze the time complexity. For a single approximation... k -Nearest neighbor vector retrieval: The retrieval method proposed in this invention only requires applying the classification model once and comparing no more than [number missing] neighbors. mN There are reference vectors, where m =min( k , L The time complexity of the retrieval method is dominated by the calculation of the preset distance between the candidate vector and all reference vectors; therefore, the time complexity of the retrieval method is... O ( mN ),in m and N Both are related to the number of reference vectors M Irrelevant. Because the traditional time complexity is determined by the number of reference vectors... M Therefore, the time complexity of the retrieval method is constant. For the construction of the index structure, the time spent in each step of the proposed construction method is equal to the number of reference vectors. M The time complexity of the index structure is polynomial. Therefore, the construction time complexity is also polynomial. The index structure can be constructed offline, so its construction time complexity does not affect the online retrieval time complexity. For index structure updates, i.e., the execution of vector operation instructions (insert, delete, modify), the update method proposed in this invention only reconstructs the index structure in the worst case. The time complexity of reconstructing the index structure is equal to the number of reference vectors. M The time complexity of the index structure is polynomial. Therefore, the update time complexity is also polynomial. The index structure can be updated asynchronously, so the update time complexity does not affect the online retrieval time complexity.

[0077] Next, we will analyze the space complexity. Since a single vector occupies a relatively large amount of memory, the space complexity of this invention's solution is determined by the number of vectors. MTo measure this. It should be noted that, in the technical solution of this invention, the training samples for building the classification model can be stored on disk and trained using in-library machine learning techniques, or they only need to be loaded into memory in small batches to complete a single iteration of training, and the memory resources occupied are cleared after training is complete. Therefore, the memory space required to build the classification model can be ignored or considered to be no more than [amount missing]. N vectors, where N This is the upper limit of the cluster size introduced when defining the mapping table. For approximation... k -Nearest neighbor vector retrieval: The retrieval method proposed in this invention only needs to store no more than [a certain number of elements] in the memory buffer. mN There are 10 candidate vectors, therefore the space complexity of the retrieval method is 1. O ( mN ).because m and N All with M Since it is irrelevant, the space complexity of the retrieval method can be considered constant. For the construction of the index structure, the proposed method requires allocating a space to hold... N A memory buffer containing +1 vectors stores the vector data of the clusters to be split, and a cluster center vector table is maintained in memory. Since the number of vectors in each cluster can approach or reach... N Therefore, the cluster center vector table roughly contains M / N Cluster center vectors. That is, the storage space required for the first step is approximately... N + M / N The second step does not increase vector storage, and in subsequent steps, the cluster center vector table is no longer used and its occupied memory resources are cleared. Therefore, the memory space occupied by the remaining steps will not exceed the memory space used in the first step. In other words, the space complexity of building the index structure is O(n log n). O ( N + M / N For index structure updates, the part that consumes the most memory is rebuilding the cluster center vector table when processing vector deletion operations. This part requires allocating a memory space to hold... N The memory buffer of each vector computes the center vector of each cluster and maintains a roughly contiguous cluster in memory. M / N The cluster center vector table of each vector. The memory space occupied by the remaining stages will not exceed the memory space used by this stage, therefore the space complexity of updating the index structure is also... O ( N + M / N ).

[0078] The technical effects are analyzed below. In the process of constructing and updating the index structure, the technical solution of this invention does not require additional manually labeled training samples to build the classification model. Instead, it automatically constructs all training samples for the classification model based on the reference vector. The technical solution of this invention can update the index structure according to the real-time changes of the reference vector, eliminating the need to build the index structure from scratch and minimizing the high-cost updates to the index structure. Even if high-cost updates to the index structure cannot be avoided, such updates can be performed asynchronously to reduce the impact on online retrieval, thereby ensuring the continuity and stability of the online retrieval service.

[0079] The scope of protection of this invention is analyzed below.

[0080] First, this invention does not limit how to solve the maximum weight matching problem of a bipartite graph, which is reduced from the cluster adjustment problem. This problem can be solved using traditional exact algorithms such as the Hungarian algorithm and network flow algorithms, or approximate algorithms such as greedy algorithms and randomized algorithms. Furthermore, this problem can also be solved using general solvers for more general problems, such as solvers based on linear programming or integer programming. The technical solution of this invention does not depend on whether the matching solution obtained by the solver is a maximum weight matching solution; any matching solution is acceptable because any matching solution guarantees that the number of matched vector IDs for each cluster does not exceed a certain limit. N That is, each cluster has no more than N A vector. The use of a specific bipartite graph maximum weight matching problem solver should not be considered beyond the scope of technical protection of this patent.

[0081] Secondly, this invention does not limit the machine learning techniques used to train the classification model. The construction method proposed in this invention can use any machine learning technique to build the classification model, such as decision trees, random forests, neural networks, etc. The construction method proposed in this invention does not depend on the accuracy of a specific machine learning technique, as long as it can build a classification model based on the training sample set, because there is a cluster adjustment step after training the classification model to ensure that each cluster in the mapping table built using the prediction results corresponds to no more than [a certain number of clusters]. N Each vector ID. The use of specific machine learning techniques should not be considered outside the scope of technical protection of this patent.

[0082] Finally, the approximation proposed in this invention k -Nearest neighbor vector retrieval method considers min( k , L Candidate vectors from 1 to 2000 clusters. Other variations may consider different numbers of clusters, but as long as the number of clusters considered remains between 1 and 2000. L The retrieval method proposed in this invention remains applicable. Therefore, changing the number of clusters considered in the retrieval method should not be considered beyond the scope of technical protection of this patent.

[0083] This invention also provides a vector data learning indexing device; please refer to [link to related document]. Figure 4 . Figure 4 This is a schematic diagram of the structure of a vector data learning indexing device, including: an index structure construction module 201 for performing index structure construction operations; an index structure retrieval module 202 for performing index structure retrieval operations; and an index structure update module 203 for performing index structure update operations.

[0084] The present invention also provides a computer device including a processor and a memory; the processor is configured to execute a computer program stored in the memory to implement the steps of the method as described in any of the foregoing embodiments. Figure 5 The diagram illustrates the internal structure of a computer device. This computer device can specifically be a terminal or a server. Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a string retrieval method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the string retrieval method. Those skilled in the art will understand that… Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0085] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the foregoing embodiments.

[0086] The above embodiments merely illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A vector data learning-based indexing method, applied to a vector data management system, characterized in that, The method includes index structure construction operations, index structure retrieval operations, and index structure update operations; The index structure construction operation includes: Obtain the index building request submitted by the user. The index building request includes a vector two-dimensional table. Each row of the vector two-dimensional table corresponds to a vector and includes an ID column and a vector data column. The vectors in the vector two-dimensional table are called reference vectors. Based on all the aforementioned reference vectors, an incremental mapping table and a cluster center vector table are constructed in memory. The mapping table consists of mapping records in the form of "cluster number -> vector ID set", and the vector ID set of each mapping record is non-empty and consists of no more than [number missing]. N Composed of vector IDs, N The vector set corresponding to the vector ID set is called a cluster, and the cluster number is set to the upper limit of the preset cluster size. i The cluster is called the first i A cluster; Calculate the preset distance between each reference vector and each cluster center vector based on the cluster center vector table, and obtain the value of each reference vector based on the preset distance. L - A list of nearest neighbor cluster numbers, L This is a preset upper limit on the number of nearest neighbor clusters to be searched, and the entirety of the number list. L Each element is the one with the smallest preset distance to the reference vector. L The cluster number corresponding to each cluster center vector; the complete list of the numbers. L The elements are arranged in ascending order according to the preset distance between the reference vector and the corresponding cluster center vector; obtain all the aforementioned reference vectors. L - After the nearest neighbor cluster number list is completed, the cluster center vector table is cleared, and the memory space occupied by the cluster center vector table is released; By building step by step L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers; The vector data of the reference vector is used as the input of the training samples, and the reference vector is adjusted... L - The list of nearest neighbor cluster numbers is used as the output of the training samples to construct the training samples; A classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with cluster numbers. The revised loss function machine learning algorithm refers to replacing the original machine learning algorithm with the loss function of the sample with the highest predicted probability value. L List of category tags and their corresponding L -A machine learning model obtained by weighting the pre-defined probability errors among the nearest neighbor clusters' index lists; Calculate the top-ranked reference vectors whose predicted probability values ​​are obtained by the trained classification model. L A list of class labels, used as the prediction of the reference vector. L - A list of nearest neighbor cluster numbers; Based on the prediction of each of the reference vectors L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm of bipartite graph, and the mapping table is updated accordingly; The updated mapping table and the trained classification model are summarized and labeled as the index structure of the vector two-dimensional table; The index structure retrieval operation includes: Obtain the vector retrieval request submitted by the user, the vector retrieval request including the vector to be retrieved and the expected number of target vectors. k Mark the vector to be retrieved as the query vector; The query vector is input into the trained classification model to obtain the top-ranked classification vectors with the highest predicted probabilities. m Cluster number, m yes k and L The minimum value between, the former m The cluster number is denoted as ; For 1 to m Each sequence number between i Obtain the first from the mapping table. The set of vector IDs of the cluster, the th cluster The vector data corresponding to the set of vector IDs of each cluster is loaded from disk into the memory buffer and marked as candidate vectors; Calculate the preset distance between the query vector and all the candidate vectors, and select the vector with the smallest preset distance. k The candidate vectors are set as target vectors; if the number of target vectors is less than k Then all the candidate vectors are set as the target vector; Provide the user with a set of all the stated target vectors; The index structure update operation includes: If the data in the vector two-dimensional table with the constructed index structure changes, and the change type is "delete vector", then execute the delete vector operation instruction, the instruction content of which includes the vector ID to be deleted; If the data in the vector two-dimensional table with the constructed index structure changes, and the change type is adding a vector, then the add vector operation instruction is executed. The instruction content includes the vector ID to be added and the vector data. If the data in the vector two-dimensional table with the constructed index structure changes, and the change type is "modify vector", then execute the "modify vector" operation instruction. The instruction content includes the vector ID to be modified and the vector data.

2. The vector data learning indexing method as described in claim 1, characterized in that, The incremental construction of the mapping table and cluster center vector table in memory based on all the reference vectors includes: Initialize the mapping table and the cluster center vector table to empty tables; When adding the first reference vector, add "1->{first reference vector ID}" to the mapping table, and add the first reference vector as the center vector of the first cluster to the cluster center vector table; Then, the reference vectors are processed one by one, and the first one is added. k When referring to the nth reference vector, find the vector in the cluster center vector table that corresponds to the nth reference vector. k Each reference vector is pre-defined as the cluster center vector with the smallest distance and its corresponding cluster number. c , the first k Add the ID of the reference vector to the first reference vector. c The vector ID set of each cluster; If the number of elements in the vector ID set of the c-th cluster exceeds the preset number N Then the first c Vector data for all vectors in each cluster is loaded from disk into a memory buffer for processing, starting from the first... c In a cluster, locate two distinct new center vectors among all vectors such that the sum of the combined distances of other vectors to these two new center vectors is minimized, where the combined distance of each vector to these two new center vectors is the minimum of the preset distances between that vector and these two new center vectors. Label any of the new center vectors as the first... c The new center vector of the cluster, and another new center vector is labeled as the first. n +1 cluster center vectors, n This represents the number of cluster center vectors in the current cluster center vector table; The first n The vector ID set of +1 clusters is initialized to an empty set; Regarding the first c Each vector in the clusters is if and only if the vector is related to the first... c The preset distance between the new center vector of the cluster and the first cluster is greater than the distance between that vector and the first cluster. n When the preset distance to the center vector of the +1 cluster is reached, the ID of the vector is changed from the first... c The vector ID set of the cluster is moved to the . n +1 clusters of vector IDs; According to the c The new center vector of the cluster and the first n Update the cluster center vector table with +1 cluster center vectors, and then... n +1->No. n A mapping record for "+1 cluster vector ID set" is added to the mapping table.

3. The vector data learning indexing method as described in claim 1, characterized in that, If the data in the vector two-dimensional table with the constructed index structure changes, and the change type is vector deletion, then a vector deletion operation instruction is executed. The instruction content includes the vector ID to be deleted, including: The vector ID to be deleted is marked as a deletion ID, and the cluster number corresponding to the deletion ID is determined according to the mapping table. c The deletion ID will be removed from the first... c Remove from the set of vector IDs of each cluster; If the first c If the vector ID set of each cluster is not empty, then the currently updated mapping table and the original trained classification model are marked as the index structure of the vector two-dimensional table after executing the vector deletion operation instruction; otherwise, continue to execute the subsequent operations. Delete the "c->empty set" mapping record from the mapping table; For each "" in the mapping table i ->No. i The vector ID set of the cluster is mapped to the record, and the first cluster's vector ID set is mapped to the record. i The vector data corresponding to the vector ID set of each cluster is loaded from the disk into the memory buffer to locate the center vector, so that the sum of the preset distances between other vectors and the center vector is minimized; The cluster center vector table is formed by summing the center vectors of each cluster in the mapping table. Based on the cluster center vector table, calculate the preset distance between each reference vector and the center vector of each cluster, and obtain the distance between each reference vector and the center vector of each cluster. L - A list of nearest neighbor cluster numbers; the entire list of numbers. L Each element is the one with the smallest preset distance to the reference vector. L The cluster number corresponding to each cluster center vector; the complete list of the numbers. L The elements are arranged in ascending order according to the preset distance between the reference vector and the corresponding cluster center vector; obtain all the aforementioned reference vectors. L - After the nearest neighbor cluster number list is completed, the cluster center vector table is cleared, and the memory space occupied by the cluster center vector table is released; By building step by step L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers; The vector data of the reference vector is used as the input of the training samples, and the reference vector is adjusted... L - The list of nearest neighbor cluster numbers is used as the output of the training samples to construct the training samples; A classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with cluster numbers. The revised loss function machine learning algorithm refers to replacing the original machine learning algorithm with the loss function of the sample with the highest predicted probability value. L List of category tags and their corresponding L -A machine learning model obtained by weighting the pre-defined probability errors among the nearest neighbor clusters' index lists; Calculate the top-ranked reference vectors whose predicted probability values ​​are obtained by the trained classification model. L A list of class labels, used as the prediction of the reference vector. L - A list of nearest neighbor cluster numbers; Based on the prediction of each of the reference vectors L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm of bipartite graph, and the mapping table is updated accordingly; The updated mapping table and the trained classification model are summarized and labeled as the index structure of the two-dimensional vector table after executing the vector deletion operation instruction.

4. The vector data learning indexing method as described in claim 1, characterized in that, If the data in the vector two-dimensional table with the constructed index structure changes, and the change type is adding a vector, then an add vector operation instruction is executed. The instruction content includes the vector ID to be added and the vector data, including: The ID of the vector to be added is marked as the insertion ID, and the vector data to be added is marked as the insertion vector; The insertion vector is input into the trained classification model to obtain the top-predicted probability vector. D Cluster number, denoted as , D To meet The preset maximum number of insertion attempts; Process 1 to 1 in increments one by one D between natural numbers i Obtain the first [number] according to the mapping table. c i The vector ID set of each cluster S i Add the insertion ID to S i In the middle; if S i The number of elements in the middle does not exceed N If so, no further processing will be done; otherwise, from S i Delete the inserted ID; If the insertion ID is added to a certain S i If no further processing is performed, the currently updated mapping table and the original trained classification model are marked as the index structure of the vector two-dimensional table after executing the vector addition operation instruction; otherwise, continue to execute subsequent operations. Add the insertion ID to S In 1, and the first c The vector data of all vectors in a cluster are loaded from disk into a memory buffer. Two different new center vectors are located to minimize the sum of the comprehensive distances of other vectors to these two new center vectors. The comprehensive distance of each vector to these two new center vectors is the minimum of the preset distances between the vector and these two new center vectors. Label any of the new center vectors as the first... c One new center vector of a cluster, and another new center vector set is the first... n +1 cluster center vectors, n The current largest cluster number; The first n The vector ID set of +1 clusters is initialized to an empty set; Regarding the first c Each vector in a cluster is such that it is associated with the first vector. c The preset distance of the new center vector of a cluster is greater than the distance between that vector and the first cluster. n When the preset distance to the center vector of the +1 cluster is reached, the ID of the vector is changed from the first... c The vector ID set of a cluster is moved to the first... n +1 clusters of vector IDs; Will" n +1->No. n Add the mapping record of "+1 cluster vector ID set" to the mapping table; For each of the reference vectors including the insertion vector v Call the trained classification model to predict v The highest probability of being in the front L Cluster number, denoted as The mapping table will uniquely contain v The cluster number corresponding to the vector ID set of the corresponding ID is marked as c 0; if there exists 1 to 0; L One of the middle i Make Then Marked as v of L - A list of nearest neighbor cluster numbers, otherwise... Marked as v of L - A list of nearest neighbor cluster numbers; By building step by step L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers; The vector data of the reference vector is used as the input of the training samples, and the reference vector is adjusted... L - The list of nearest neighbor cluster numbers is used as the output of the training samples to construct the training samples; A classification model is trained based on the training samples and a machine learning algorithm with a revised loss function. The class labels of the classification model correspond one-to-one with cluster numbers. The revised loss function machine learning algorithm refers to replacing the original machine learning algorithm with the loss function of the sample with the highest predicted probability value. L List of category tags and their corresponding L -A machine learning model obtained by weighting the pre-defined probability errors among the nearest neighbor clusters' index lists; Calculate the top-ranked reference vectors whose predicted probability values ​​are obtained by the trained classification model. L A list of class labels, used as the prediction of the reference vector. L - A list of nearest neighbor cluster numbers; Based on the prediction of each of the reference vectors L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm of bipartite graph, and the mapping table is updated accordingly; The updated mapping table and the trained classification model are summarized and labeled as the index structure of the two-dimensional vector table after executing the vector addition operation instruction.

5. The vector data learning indexing method as described in claim 1, characterized in that, If the data in the vector two-dimensional table with the constructed index structure changes, and the change type is "modify vector", then a "modify vector" operation instruction is executed. The instruction content includes the vector ID to be modified and the vector data, including: The vector data to be modified is marked as an insertion vector, a new ID that does not exist in the mapping table is generated and marked as an insertion ID, and an instruction to add a vector is generated and executed based on the insertion ID and the insertion vector; The vector ID to be modified is marked as a deletion ID, and a deletion vector operation instruction is generated and executed based on the deletion ID; Replace the insertion ID with the deletion ID in the mapping table; The updated mapping table and the trained classification model are summarized and labeled as the index structure of the two-dimensional vector table after executing the modified vector operation instruction.

6. The vector data learning indexing method as described in claims 1, 3, and 4, characterized in that, The process of building step by step L Solve the maximum weight matching problem in a bipartite graph, adjusting each of the reference vectors. L - A list of nearest neighbor cluster numbers, obtaining the adjusted value for each reference vector. L - A list of nearest neighbor cluster numbers, including: Constructing the first maximum weight matching problem in a bipartite graph involves defining two node sets. V and U ,in V Composed of reference vector ID, U Cluster number N This consists of multiple copies; assuming the ID is... v The reference vector L -Nearest neighbor clusters are numbered sequentially as follows Then for 1 to L Each between i ,Give v Corresponding nodes and ci corresponding N Each copy node is weighted. The edge, give v Corresponding nodes and node sets U Add weights to the remaining nodes in the array. The edge, of which It is a return value and an argument. i Preset functions for integer values ​​that are inversely proportional; Solving the first bipartite graph maximum weight matching problem yields a first matching solution. Each reference vector is then processed based on this first matching solution. The process of processing each reference vector based on the first matching solution includes assuming the reference vector ID is... v And in the first matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then the reference vector will be... L - Replace the first cluster number in the nearest neighbor cluster number list with c and will v Corresponding node and cluster number c Set the edge weights of all corresponding copied nodes to 0; After all the reference vectors have been processed, the resulting bipartite graph is marked as the bipartite graph of the second bipartite graph maximum weight problem; Solving the second bipartite graph maximum weight matching problem yields a second matching solution. Each reference vector is then processed based on this second matching solution. The process of processing each reference vector based on the second matching solution includes assuming the reference vector ID is... v And in the second matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then the reference vector will be... L - Replace the second cluster number in the nearest neighbor cluster number list with c and will v Corresponding node and cluster number c Set the edge weights of all corresponding copied nodes to 0; After all the reference vectors have been processed, the resulting bipartite graph is marked as the bipartite graph for solving the third bipartite graph maximum weight problem; and so on, until the solution of the third bipartite graph maximum weight problem is obtained. L The maximum weight matching problem in a bipartite graph yields the result of the following steps. L After finding a matching solution to the maximum weight matching problem in a bipartite graph, each of the reference vectors is processed in the same way to finally obtain the adjusted value for each of the reference vectors. L - A list of nearest neighbor cluster numbers.

7. The vector data learning indexing method as described in claims 1, 3, and 4, characterized in that, The prediction based on each of the reference vectors L - The list of nearest neighbor cluster numbers is obtained by solving the maximum weight matching algorithm for a bipartite graph, and the mapping table is updated accordingly, including: Constructing the final bipartite graph maximum weight matching problem; the final bipartite graph maximum weight matching problem includes defining two node sets. V and U ,in V Composed of reference vector ID, U Cluster number N This consists of multiple copies; assuming the ID is... v The reference vector L -Nearest neighbor clusters are numbered sequentially as follows Then for 1 to L Each between i ,Give v Corresponding nodes and ci corresponding N Each copy node is weighted. The edge, give v Corresponding nodes and node sets U Add weights to the remaining nodes in the array. The edge, of which It is a return value and an argument. i Preset functions for integer values ​​that are inversely proportional; Reset the vector ID set of each mapping in the mapping table to an empty set, and solve the final bipartite graph maximum weight matching problem to obtain the final matching solution; Each reference vector is processed according to the final matching solution; the process of processing each reference vector according to the final matching solution includes assuming that the reference vector ID is... v And in the final matching solution v Corresponding node and cluster number c If a corresponding copy node is connected, then... v Cluster number added to the mapping table c The corresponding vector ID set.

8. A vector data learning indexing device, characterized in that, include: An index structure construction module is used to perform the index structure construction operation; An index structure retrieval module is used to perform the index structure retrieval operation; The index structure update module is used to perform the index structure update operation.

9. A computer device, characterized in that, Including processor and memory; The processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images