Method for constructing learning index of mass high-dimensional data based on partition hierarchical graph

Abstract

The application discloses a learning index construction method for mass high-dimensional data based on a partition hierarchical graph and belongs to the technical field of learning index construction. The steps of the application comprise the following steps: determining a data set and the number of clustering partitions; based on the data set and the number of clustering partitions, using a k-means* algorithm to partition the data in the data set; when the partition of all data points in the data set is completed, using a parallel computing framework to simultaneously construct an HNSW graph in each partition to obtain a partition similarity graph index structure; and based on the partition similarity graph index structure, using a proximity query method to query mass high-dimensional data. The application divides a super large-scale data space into several subspaces, reallocates the data points at the edges of the clusters, simultaneously constructs HNSW graphs in multiple partitions by using a parallel computing framework, and effectively combines the advantages of the partition learning-based method and the graph-based method, thereby improving the efficiency and precision of index construction.

Description

Technical Field

[0001] This invention relates to the field of learning index construction technology, specifically a method for constructing learning indexes for massive high-dimensional data based on partitioned hierarchical graphs. Background Technology

[0002] Efficient data retrieval is one of the key enabling technologies for big data. Particularly influenced by the rapid development of artificial intelligence, the scale and dimensionality of data are growing rapidly, while data applications place increasingly stringent requirements on data retrieval latency. This makes indexing ultra-large-scale, high-dimensional data a significant challenge. Against this backdrop, Learn to Index (LTI) technology has emerged. By using machine learning models to learn data distribution, it replaces traditional database indexes, effectively solving the problem of efficient retrieval of ultra-large-scale data.

[0003] Current research largely focuses on designing retrieval systems based on the concept of near-nearest neighbors, such as inverted indexes, tree-based index models, and hash-based index models. However, with the increase in data scale and dimensionality, existing near-nearest neighbor index models suffer from excessive space costs and low query efficiency. With the development of machine learning technology, learning-based indexing techniques utilize machine learning to obtain data similarity relationships and achieve near-nearest neighbor search, significantly reducing the space cost of index structures and improving index efficiency.

[0004] Current indexing methods widely employ spatial partitioning based on partitioning and graph-based indexing. In real-world applications, we've found that partitioning-based methods are inaccurate in defining boundary data, leading to significant index accuracy degradation. Because spatial partitioning makes it difficult to accurately represent the similarity of data at inter-cluster edges, nodes in these edge regions are often difficult to retrieve accurately. Graph-based methods typically require constructing a nearest-neighbor graph index structure and traversing the graph to obtain the final search result. While graph-based methods offer advantages such as high accuracy and low search latency, the construction and storage costs of the nearest-neighbor graph structure increase exponentially with the growth of data volume.

[0005] Therefore, the current learning index model still has the following two problems:

[0006] (1) The partition-based learning method is inaccurate in dividing boundary data, which leads to a decrease in index accuracy;

[0007] (2) As the amount of data increases, the construction time of the nearest neighbor graph will increase exponentially, making it difficult to meet the real-time and distributed requirements. Summary of the Invention

[0008] The purpose of this invention is to provide a method for constructing learning indexes for massive high-dimensional data based on partitioned hierarchical graphs. By adopting a divide-and-conquer strategy, the massive data space is divided into several subspaces through partition learning, and the data points at the edges of each cluster are redistributed. Finally, each subspace is indexed using a hierarchical navigable small-world graph based on partition learning, thereby effectively integrating the advantages of partition-based learning methods and graph-based methods, and improving the efficiency and accuracy of index construction.

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: a method for constructing an index for massive high-dimensional data based on a partitioned hierarchical graph, comprising:

[0010] Determine the dataset and the number of cluster partitions;

[0011] Based on the dataset and the number of cluster partitions, the k-means* algorithm is used to partition the data in the dataset;

[0012] Once all data points in the dataset are partitioned, an HNSW graph is constructed simultaneously in each partition using a parallel computing framework to obtain the partition similarity graph index structure.

[0013] Based on the partition similarity graph index structure, the nearest neighbor query method is used to query massive high-dimensional data.

[0014] According to the above technical solution, the step of determining the number of cluster partitions includes:

[0015] Select the range for the number of cluster partitions;

[0016] Run the k-means* algorithm for each value within the range of cluster partition numbers and calculate the corresponding distortion level avgSSE;

[0017] The distortion degree (avgSSE) corresponding to each value is plotted as a graph. The distortion degree clustering is evaluated using the "elbow method" to determine the number of cluster partitions. The distortion degree is represented by the average sum of squared errors (SSE) within each cluster.

[0018] According to the above technical solution, the steps of the k-means* algorithm include:

[0019] Iterate through each data point x in dataset X i Calculate data point x i The minimum distance d is obtained by comparing the distances to each centroid. min and will be with data point x i The region containing the shortest centroid is denoted as region α;

[0020] Calculate data point x i Distance d to each centroidj minimum distance d min The difference is denoted as δ. j ;

[0021] If the distance δ j If the value is less than the threshold τ, then it proves that the data point x... i For the boundary data between partition α and partition j, the data point is assigned to both partition α and partition j; partition j represents the boundary data between partition x and partition j. i Distance d j The region where the centroid is located;

[0022] If the distance δ j If the value is greater than or equal to the threshold τ, then it proves that the data point x i If the data point x is not on the boundary of any partition, then x i Assign it to partition α. ​​When dealing with non-clustered data, the parameter sensitivity and boundary ambiguity of the k-means algorithm make it impossible to accurately describe the similarity of boundary data. However, the data boundary quality obtained after k-means* partitioning is higher, and it can contain as much neighbor information as possible within the partition, thus ensuring the quality of the final index.

[0023] According to the above technical solution, the steps of the nearest neighbor query method are as follows:

[0024] Input a query point q and the number of its nearest neighbors k;

[0025] Compare the distances of q to the centroids of each partition, and select the partition closest to q as the index sub-partition j;

[0026] Input the query q and the number of nearest neighbors k into the HNSW index structure in the index sub-partition j, and output the final index result in the HNSW index of the index sub-partition j.

[0027] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0028] 1. This invention utilizes the k-means* algorithm to clearly delineate the boundary data of each cluster, solving the problem of ambiguous delineation of inter-cluster boundary data points in traditional clustering methods, which leads to a decrease in query accuracy;

[0029] 2. This invention proposes a partitioned hierarchical graph index structure and utilizes parallel computing for its construction. Compared to traditional methods, it effectively integrates the advantages of spatial partitioning and graph-based index structures, improving index construction efficiency and enabling its application in distributed scenarios with real-time requirements.

[0030] 3. The core idea of ​​this invention is to design a hierarchical indexing model that adopts a divide-and-conquer strategy. The massive data space is divided into several subspaces through partition learning, and the subspaces are indexed using a graph-based method. This effectively integrates the advantages of partition learning-based and graph-based methods. Specifically, the k-means* algorithm is used to partition the original data, and the partition boundary results are optimized to improve partition quality. The parallel computing framework simultaneously constructs HNSW graphs in multiple partitions, thereby improving the efficiency and accuracy of index construction, effectively reducing index construction time, and improving the real-time performance of index construction. Attached Figure Description

[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a schematic diagram of the structure of the method for constructing a learning index for massive high-dimensional data based on a partitioned hierarchical graph, as described in this invention.

[0033] Figure 2 It is a k-means* elbow method avgSSE line chart;

[0034] Figure 3 It is a graph showing the relationship between index recall and index construction time;

[0035] Figure 4 It is a graph showing the index recall rate versus indexing time. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] This invention provides a technical solution: a method for constructing an index for massive high-dimensional data based on a partitioned hierarchical graph, comprising the following steps:

[0038] S1. Determine the dataset and the number of cluster partitions;

[0039] The steps to determine the number of cluster partitions are as follows: select a range for the number of cluster partitions; run the k-means* algorithm for each value within the range of the number of cluster partitions and calculate the corresponding distortion level avgSSE; plot the distortion level avgSSE corresponding to each value as a chart, use the "elbow method" to evaluate the distortion level clustering, and then determine the number of cluster partitions.

[0040] S2. Based on the dataset and the number of cluster partitions, partition the data in the dataset using the k-means* algorithm; specifically, iterate through each data point x in the dataset X. i Calculate data point x i The minimum distance d is obtained by comparing the distances to each centroid. min and will be with data point x i The region containing the shortest centroid is denoted as region α;

[0041] Calculate data point x i Distance d to each centroid j minimum distance d min The difference is denoted as δ. j ;

[0042] If the distance δ j If the value is less than the threshold τ, then it proves that the data point x... i For the boundary data between partition α and partition j, the data point is assigned to both partition α and partition j; partition j represents the boundary data between partition x and partition j. i Distance d j The region where the centroid is located;

[0043] If the distance δ j If the value is greater than or equal to the threshold τ, then it proves that the data point x i If the data point x is not on the boundary of any partition, then x i Assigned to partition α;

[0044] S3. After all data points in the dataset are partitioned, an HNSW graph is constructed simultaneously in each partition using a parallel computing framework, resulting in the partition similarity graph index structure as follows: Figure 1 As shown;

[0045] S4. Based on the partition similarity graph index structure, the nearest neighbor query method is used to query massive high-dimensional data. Specifically, a query point q and the number of nearest neighbors k are input, the distance between q and the centroid of each partition are compared, and the partition closest to q is selected as the index sub-partition j. The query q and the number of nearest neighbors k are input into the HNSW index structure in the index sub-partition j, and the final index result is output in the HNSW index in the index sub-partition j.

[0046] Example 1

[0047] Select a cluster partition size range of 1-10. For each cluster partition size, run the k-means* algorithm and calculate the corresponding distortion level (avgSSE). Plot the distortion level (avgSSE) for each cluster partition size value as a graph, such as... Figure 2As shown in the figure, the elbow position is reached when the number of cluster partitions is 4. However, to ensure the quality of cluster partitions, the number of cluster partitions in this paper is set to 5.

[0048] Comparative Example 1

[0049] The index building time consumed by the RP-Tree, HNSW, and KM*-HNSW algorithms to achieve the same recall rate was measured (the KM*-HNSW algorithm represents the index building method for massive high-dimensional data learning based on partitioned hierarchical graphs in this invention):

[0050] First, an index was built for one million training vectors in the SIFT dataset, and the construction time was recorded for each index. For recall, the top 100 vectors in the test vector set were selected for indexing, and the final result was evaluated using recall@10. Experimental results are as follows. Figure 3 As shown.

[0051] As shown in the graph, when the recall rate is low, the index building time for each algorithm is roughly the same. However, as the index recall rate increases, the index building time for the HNSW method and RP-Tree increases dramatically. When the recall rate reaches 0.93, building an HNSW index takes over 550 seconds, and building an RP-Tree index takes over 300 seconds, severely impacting the real-time performance of index building. In contrast, the KM*-HNSW algorithm, with a recall rate exceeding 0.93, only requires 200 seconds to build an index, which is only 36.4% of the time of the HNSW algorithm, significantly lower than both HNSW and RP-Tree algorithms. The KM*-HNSW algorithm can build indexes on millions of high-dimensional datasets in a much shorter time, greatly improving the possibility of real-time index updates.

[0052] Comparative Example 2

[0053] The indexing times of RP-Tree, HNSW, and KM*-HNSW algorithms under different recall rates were compared (KM*-HNSW algorithm represents the indexing method for massive high-dimensional data based on partitioned hierarchical graphs in this invention). The first 100 vectors of the test vector set were selected for indexing, and the results were evaluated using the recall@10 metric. The experimental results are as follows: Figure 4 As shown;

[0054] from Figure 4 As can be seen, traditional RP-Tree, due to its extensive projection dimensionality reduction and subtree construction, results in a massive number and depth of subtrees when processing large amounts of high-dimensional data, significantly reducing indexing efficiency. In contrast, HNSW and KM*-HNSW algorithms are indexing methods based on hierarchical navigable small-world graphs, enabling near-nearest neighbor indexing in massive amounts of high-dimensional data with extremely high efficiency.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0056] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a learning index for massive high-dimensional data based on a partitioned hierarchical graph, characterized in that, include: Determine the dataset and the number of cluster partitions; The steps for determining the number of cluster partitions include: Select the range for the number of cluster partitions; Run the k-means* algorithm for each value within the range of cluster partition numbers and calculate the corresponding distortion level avgSSE; The distortion level avgSSE corresponding to each value is plotted as a chart, and the distortion level clustering is evaluated using the "elbow method" to determine the number of cluster partitions. Based on the dataset and the number of cluster partitions, the k-means* algorithm is used to partition the data in the dataset; the steps of the k-means* algorithm include: Iterate through each data point in dataset X Calculate data points The minimum distance is obtained by comparing the distances to each centroid. and will be with data points The region containing the shortest centroid is denoted as a partition. ; Calculate data points Distance to each centroid minimum distance The difference is denoted as ; If distance Less than the threshold This proves the data points For partitioning If the boundary data is the same as that of partition j, then the data points will be assigned to partition j simultaneously. With partition j; If distance Greater than or equal to the threshold This proves the data points If the data points are not on the boundaries of each partition, then... Assigned to partition ; Once all data points in the dataset are partitioned, an HNSW graph is constructed simultaneously in each partition using a parallel computing framework to obtain the partition similarity graph index structure. Based on the partition similarity graph index structure, the nearest neighbor query method is used to query massive high-dimensional data.

2. The method for constructing a learning index for massive high-dimensional data based on a partitioned hierarchical graph according to claim 1, characterized in that, The steps of the nearest neighbor query method are as follows: Enter a query point Number of nearest neighbors ; Compare Calculate the distance from the centroid of each partition and select the appropriate distance. The most recent partition is used as an index subpartition. ; Query Number of nearest neighbors Input index subpartition The HNSW index structure in the index, and in the index sub-partition The final index result is output in the HNSW index.

Images