High-dimensional data retrieval method based on distributed computing and mixed indexing

Abstract

The invention discloses a high-dimensional data retrieval method based on distributed computing and mixed indexing. The method comprises the following steps: step 1, data preprocessing; step 2, cluster generation; step 3, index construction; step 4, consistent Hash: distributing each cluster to a virtual node of a Hash ring, dynamically adjusting data distribution on the virtual node according to a load strategy, and mapping the virtual node to a physical server; and step 5, hierarchical aggregation: mapping a query request to a node and adjacent nodes on a hash ring through a hash function, performing candidate set calculation in parallel, merging candidate sets sent by all the nodes by a Spark center node, and finally optimizing a result. According to the method, high-recall-rate and low-delay distributed retrieval is achieved through mixed index construction, dynamic consistency hash allocation and hierarchical aggregation strategies, meanwhile, system load balancing and index updating efficiency are guaranteed, and the method is suitable for large-scale high-dimensional data scenes.

Description

Technical Field

[0001] The present invention belongs to the field of high-dimensional data processing and distributed systems, and specifically relates to a high-dimensional data retrieval method based on distributed computing and hybrid indexing. Background Art

[0002] With the advent of the big data era, data dimensionality continues to increase. High-dimensional data is being used increasingly widely, particularly in fields like image recognition and natural language processing. High-dimensional data retrieval involves finding items similar to the query data within large, high-dimensional datasets. In traditional low-dimensional data retrieval, the data dimensionality is relatively small, and classic retrieval methods (such as linear scans and inverted indexes) can effectively accomplish this task. However, when the data dimensionality increases dramatically, the performance of these methods rapidly degrades, primarily due to the following: 1. Computational complexity. The amount of computation required for high-dimensional data increases significantly, especially for distance calculations (such as Euclidean distance and cosine similarity). This computational complexity increases exponentially with higher data dimensionality, leading to increased query time and storage requirements. 2. The curse of dimensionality: As dimensionality increases, data becomes increasingly sparse, and previously effective distance metrics gradually become ineffective, resulting in decreased retrieval accuracy. 3. Storage and memory pressure: The storage and processing requirements for high-dimensional data are far greater than those for low-dimensional data, placing significant strain on the system's memory, disk, and computing resources.

[0003] Traditional noise filtering methods are prone to misjudgment in high-dimensional scenarios due to differences in statistical distribution across dimensions. Their Gaussian distribution assumption is inconsistent with the multimodal distribution of high-dimensional features. Traditional data processing also suffers from low data accuracy and reliability, clustering affected by dimensionality, and low retrieval efficiency that fails to meet real-time requirements.

[0004] High-dimensional indexing techniques primarily fall into two categories: approximate nearest neighbor indexing and graph-structured indexing. Approximate nearest neighbor indexing, represented by LSH, relies on hash function mapping to achieve O(1) query speeds, but its recall rate is limited by the granularity of the hash buckets. Existing hybrid indexing research, such as LSH+HNSW, often employs a serial cascade strategy, which fails to resolve index consistency conflicts during incremental updates.

[0005] The consistent hashing algorithm can achieve uniform distribution of data shards, but its static virtual node mapping mechanism is difficult to cope with two dynamic scenarios: load skew (such as hotspot clustering causing local server overload) and data drift (clustering structure evolution during online learning, static sharding increases the proportion of cross-node queries).

[0006] Existing distributed retrieval systems mostly adopt the "central node full sorting" retrieval result aggregation strategy, which has two major bottlenecks: network bandwidth pressure and computational redundancy.

[0007] Therefore, how to process and store large amounts of high-dimensional data while ensuring efficient retrieval performance has become an important challenge in the current field of computer science. Summary of the Invention

[0008] This invention aims to address the distributed clustering and retrieval optimization issues in high-dimensional data processing by providing a high-dimensional data retrieval method based on distributed computing and hybrid indexing. This method utilizes a hybrid index (LSH+HNSW), dynamic consistent hashing, and a hierarchical aggregation strategy to achieve high-recall, low-latency distributed retrieval while ensuring system load balancing and efficient index updates. It is suitable for large-scale, high-dimensional data scenarios.

[0009] The technical solution for achieving the purpose of the present invention is:

[0010] A high-dimensional data retrieval method based on distributed computing and hybrid indexing includes the following steps:

[0011] Step 1. Data preprocessing: standardize and normalize the data to ensure accuracy and reliability;

[0012] Step 2: Cluster generation: Use the Spark distributed computing framework to split the preprocessed data and distribute the preprocessed data to different Spark nodes. Use the HDBSCAN algorithm to generate clusters for the data distributed to different Spark nodes and aggregate them into multiple clusters based on similarity.

[0013] Step 3: Index construction: Build LSH and HNSW indexes simultaneously on each cluster, using a hybrid indexing strategy to optimize retrieval speed and accuracy.

[0014] Step 4: Consistent Hashing: Assign each cluster to a virtual node in the hash ring, dynamically adjust data distribution on the virtual node based on the load strategy, and then map the virtual node to the physical server.

[0015] Step 5, hierarchical aggregation: Map the query request to a node and its adjacent nodes on the hash ring through a hash function, calculate the candidate set in parallel, and then the Spark central node merges the candidate sets sent by each node, and finally optimizes the results.

[0016] The process of data preprocessing in step 1 is as follows:

[0017] Assume the original data set is , improve data quality by:

[0018] 1.1) Data normalization: Convert the data in the original dataset into a distribution with a mean of 0 and a standard deviation of 1:

[0019] (1)

[0020] (2)

[0021] (3)

[0022] in It is The original eigenvalues ​​of the data, It is The mean of the features, n is the number of data in the dataset, It is The standard deviation of the features, It is The standardized value of the data points;

[0023] 1.2) Data normalization:

[0024] = (4)

[0025] in is the normalized data, is the original data point, 、 are the maximum and minimum values ​​of the feature respectively.

[0026] The process of generating clusters using the HDBSCAN algorithm in step 2 is as follows:

[0027] 2.1) Mutual reach distance calculation: define two parameters for data points: minPts and epsilon. minPts represents the minimum number of neighbor points required to form a "dense" area, and epsilon represents a distance radius. The core distance of the data point 、 Refers to data points respectively The epsilon radius neighborhood contains exactly minPts points. The mutual distance describes the distance between two data points. The minimum distance required for "mutual reach" between them, the mutual reach distance matrix is ​​described as:

[0028] (4) ;

[0029] 2.2) Minimum spanning tree construction: Use Prim's algorithm to construct a minimum spanning tree on the mutual reach distance matrix and obtain the edge weight set ;

[0030] 2.3) Cluster extraction: Split the minimum spanning tree according to the stability criterion and define the cluster λ stability:

[0031] (5)

[0032] Passing the stability threshold , screening effective clustering .

[0033] The specific steps of step 3 are:

[0034] right Each cluster in Constructing mixed index structures in parallel:

[0035] 3.1) LSH hash function: maps similar data points into the same hash bucket to accelerate similarity search;

[0036] (6)

[0037] in It is a d-dimensional space data points, is a data point The hash value is mapped to the hash bucket;

[0038] 3.2) HNSW Index: Building a Hierarchical Navigation Graph

[0039] (7)

[0040] in represents the jth cluster, Indicates based on The constructed HNSW graph, k represents the number of neighbors of each node, Indicates the distance threshold in HNSW construction;

[0041] 3.3) Hybrid index construction: define joint retrieval function:

[0042] (8)

[0043] in , dynamically adjusted based on query latency.

[0044] The specific steps of step 4 are:

[0045] 4.1) Dynamic mapping function: distributes clustering results and index data to different virtual nodes on the hash ring, and virtual nodes are further mapped to physical nodes to ensure load balancing.

[0046] (9)

[0047] in Represents the clockwise distance on the hash ring;

[0048] 4.2) Load balancing strategy: Introducing dynamic weight factors:

[0049] (10)

[0050] in is the resource weight coefficient, and load migration is achieved through virtual node replication:

[0051] (11).

[0052] The specific steps of step 5 are:

[0053] 5.1) Query request hash map:

[0054] =hash(q) (12)

[0055] Where q is the query request, For query requests Position on the hash ring;

[0056] 5.2) Adjacent node selection:

[0057] (13)

[0058] in represents the adjacent nodes of the query request q, and k represents the number of neighbors of each node;

[0059] 5.3) Parallel calculation of candidate sets:

[0060] (14)

[0061] in represents the local candidate set generated by each node, and q represents the query request;

[0062] 5.4) Candidate set and sorting: Merge and sort the local candidate sets generated by each node , and obtain the final candidate set , the merged candidate set is sorted by the distance from the query point q, and the most similar k results are selected.

[0063] This technical solution focuses on distributed clustering and retrieval optimization, improves HDBSCAN clustering, constructs a hybrid index of LSH and HNSW, adopts a dynamic consistent hashing allocation strategy, and combines it with a hierarchical aggregation strategy to achieve high recall and low latency distributed retrieval, ensuring system load balancing and index update efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic flow chart of an embodiment;

[0065] Figure 2 This is a diagram of the hybrid index construction and consistent hashing distribution architecture in the embodiment;

[0066] Figure 3 1 is a hierarchical aggregation flow chart in an embodiment. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0068] Example:

[0069] Reference Figure 1 , a high-dimensional data retrieval method based on distributed computing and hybrid indexing, comprising the following steps:

[0070] Step 1. Data preprocessing: standardize and normalize the data to ensure accuracy and reliability;

[0071] Step 2: Cluster generation: Use the Spark distributed computing framework to split the preprocessed data and distribute the preprocessed data to different Spark nodes. Use the HDBSCAN algorithm to generate clusters for the data distributed to different Spark nodes and aggregate them into multiple clusters based on similarity.

[0072] Step 3: Index construction: Build LSH and HNSW indexes simultaneously on each cluster, using a hybrid indexing strategy to optimize retrieval speed and accuracy.

[0073] Step 4: Consistent Hashing: Assign each cluster to a virtual node in the hash ring, dynamically adjust data distribution on the virtual node based on the load strategy, and then map the virtual node to the physical server.

[0074] Step 5, hierarchical aggregation: Map the query request to a node and its adjacent nodes on the hash ring through a hash function, calculate the candidate set in parallel, and then the Spark central node merges the candidate sets sent by each node, and finally optimizes the results.

[0075] The process of data preprocessing in step 1 is as follows:

[0076] Assume the original data set is , improve data quality by:

[0077] 1.1) Data normalization: Convert the data in the original dataset into a distribution with a mean of 0 and a standard deviation of 1:

[0078] (1)

[0079] (2)

[0080] (3)

[0081] in It is The original eigenvalues ​​of the data, It is The mean of the features, n is the number of data in the dataset, It is The standard deviation of the features, It is The standardized value of the data points;

[0082] 1.2) Data normalization:

[0083] = (4)

[0084] in is the normalized data, is the original data point, 、 are the maximum and minimum values ​​of the feature respectively.

[0085] The process of generating clusters using the HDBSCAN algorithm in step 2 is as follows:

[0086] 2.1) Mutual reach distance calculation: define two parameters for data points: minPts and epsilon. minPts represents the minimum number of neighbor points required to form a "dense" area, and epsilon represents a distance radius. The core distance of the data point 、 Refers to data points respectively The epsilon radius neighborhood contains exactly minPts points. The mutual distance describes the distance between two data points. The minimum distance required for "mutual reach" between them, the mutual reach distance matrix is ​​expressed as:

[0087] (4) ;

[0088] 2.2) Minimum spanning tree construction: Use Prim's algorithm to construct a minimum spanning tree on the mutual reach distance matrix and obtain the edge weight set ;

[0089] 2.3) Cluster extraction: Split the minimum spanning tree according to the stability criterion and define the cluster λ stability:

[0090] (5)

[0091] Passing the stability threshold , screening effective clustering .

[0092] The specific steps of step 3 are:

[0093] right Each cluster in Construct hybrid index structures in parallel, such as Figure 2 As shown:

[0094] 3.1) LSH hash function: maps similar data points into the same hash bucket to accelerate similarity search;

[0095] (6)

[0096] in It is a d-dimensional space data points, is a data point The hash value is mapped to the hash bucket;

[0097] 3.2) HNSW Index: Building a Hierarchical Navigation Graph

[0098] (7)

[0099] in represents the jth cluster, Indicates based on The constructed HNSW graph, k represents the number of neighbors of each node, Indicates the distance threshold in HNSW construction;

[0100] 3.3) Hybrid index construction: define joint retrieval function:

[0101] (8)

[0102] in , dynamically adjusted based on query latency.

[0103] The specific steps of step 4 are:

[0104] Set up a server cluster , build a virtual node ring;

[0105] 4.1) Dynamic mapping function: distributes clustering results and index data to different virtual nodes on the hash ring, and virtual nodes are further mapped to physical nodes to ensure load balancing.

[0106] (9)

[0107] in Represents the clockwise distance on the hash ring;

[0108] 4.2) Load balancing strategy: Introducing dynamic weight factors:

[0109] (10)

[0110] in is the resource weight coefficient, and load migration is achieved through virtual node replication:

[0111] (11).

[0112] The specific steps of step 5 are as follows: Figure 3 As shown:

[0113] 5.1) Query point hash map:

[0114] =hash(q) (12)

[0115] Where q is the query request, For query requests Position on the hash ring;

[0116] 5.2) Adjacent node selection:

[0117] (13)

[0118] in represents the adjacent nodes of the query request q, and k represents the number of neighbors of each node;

[0119] 5.3) Parallel calculation of candidate sets:

[0120] (14)

[0121] in represents the local candidate set generated by each node, and q represents the query request;

[0122] 5.4) Candidate set and sorting: Merge and sort the local candidate sets generated by each node , and obtain the final candidate set ,The merged candidate set is sorted by the distance from the query request q, and the most similar Top-k results are selected.

[0123] This example is verified by the following experiments:

[0124] 1. Experimental Procedure

[0125] This experiment mainly includes the following steps: (1) distributed implementation of the HDBSCAN clustering algorithm; (2) construction of an LSH+HNSW hybrid index structure within the cluster; (3) construction of a virtual node ring for index data distribution and load balancing; (4) query tasks are scheduled through hash mapping and parallel similarity search is performed.

[0126] 2. Distributed HDBSCAN Clustering

[0127] The core process includes mutual reach distance calculation and minimum spanning tree construction.

[0128]

[0129] 3. Parallel construction of hybrid index structure

[0130] Each cluster constructs LSH and HNSW index structures in parallel and integrates query results

[0131]

[0132] 4. Build virtual node hash ring and query scheduling

[0133]

[0134] 5. Experimental Summary

[0135] This experiment, covering the entire process from distributed clustering, index construction, virtual node scheduling, to parallel retrieval, verified the feasibility of an efficient clustering and retrieval system for high-dimensional big data. While ensuring accuracy, it effectively improved the system's query performance and resource scheduling capabilities, demonstrating excellent scalability and practical value.

Claims

1. A high-dimensional data retrieval method based on distributed computing and hybrid indexing, characterized in that: The steps include: Step 1. Data preprocessing: standardize and normalize the data to ensure accuracy and reliability; Step 2: Cluster generation: Use the Spark distributed computing framework to split the preprocessed data and distribute the preprocessed data to different Spark nodes. Use the HDBSCAN algorithm to generate clusters for the data distributed to different Spark nodes and aggregate them into multiple clusters based on similarity. Step 3: Index construction: Build LSH and HNSW indexes simultaneously on each cluster, using a hybrid indexing strategy to optimize retrieval speed and accuracy. Step 4: Consistent Hashing: Assign each cluster to a virtual node in the hash ring, dynamically adjust data distribution on the virtual node based on the load strategy, and then map the virtual node to the physical server. Step 5, hierarchical aggregation: Map the query request to a node and its adjacent nodes on the hash ring through a hash function, calculate the candidate set in parallel, and then the Spark central node merges the candidate sets sent by each node, and finally optimizes the results.

2. The high-dimensional data retrieval method based on distributed computing and hybrid indexing according to claim 1 is characterized in that: The data preprocessing process in step 1 is as follows: Assume the original data set is , improve data quality by: 1.1) Data normalization: Convert the data in the original dataset into a distribution with a mean of 0 and a standard deviation of 1: (1) ； (2) ； (3) ； in It is The original eigenvalues ​​of the data, It is The mean of the features, n is the number of data in the dataset, It is The standard deviation of the features, It is The normalized value of the data points; 1.2) Data normalization: = (4) in is the normalized data, is the original data point, 、 are the maximum and minimum values ​​of the feature respectively.

3. The high-dimensional data retrieval method based on distributed computing and hybrid indexing according to claim 1 is characterized in that: The process of generating clusters using the HDBSCAN algorithm in step 2 is as follows: 2.1) Mutual reach distance calculation: define two parameters for data points: minPts and epsilon. minPts represents the minimum number of neighbor points required to form a dense area, and epsilon represents a distance radius. The core distance of the data point 、 Refers to data points respectively The epsilon radius neighborhood contains minPts points, and the mutual distance describes the distance between two data points. The minimum distance required for mutual reachability between them is described as: (4) ； 2.2) Minimum spanning tree construction: Use Prim's algorithm to construct a minimum spanning tree on the mutual reach distance matrix and obtain the edge weight set ; 2.3) Cluster extraction: Split the minimum spanning tree according to the stability criterion and define the cluster λ stability: （5） Passing the stability threshold , screening effective clustering .

4. The high-dimensional data retrieval method based on distributed computing and hybrid indexing according to claim 3 is characterized in that: The specific steps of step 3 are: right Each cluster in Constructing mixed index structures in parallel: 3.1) LSH hash function: maps similar data points into the same hash bucket to accelerate similarity search; （6） in It is a d-dimensional space data points, is a data point The hash value is mapped to the hash bucket; 3.2) HNSW Index: Building a Hierarchical Navigation Graph （7） in represents the jth cluster, Indicates based on The constructed HNSW graph, k represents the number of neighbors of each node, Indicates the distance threshold in HNSW construction; 3.3) Hybrid index construction: define joint retrieval function: （8） in , dynamically adjusted based on query latency.

5. The high-dimensional data retrieval method based on distributed computing and hybrid indexing according to claim 1 is characterized in that: The specific steps of step 4 are: 4.1) Dynamic mapping function: distributes clustering results and index data to different virtual nodes on the hash ring, and virtual nodes are further mapped to physical nodes to ensure load balancing. （9） in Represents the clockwise distance on the hash ring; 4.2) Load balancing strategy: Introducing dynamic weight factors: （10） in is the resource weight coefficient, and load migration is achieved through virtual node replication: （11）。 6. The high-dimensional data retrieval method based on distributed computing and hybrid indexing according to claim 1 is characterized in that: The specific steps of step 5 are: 5.1) Query request hash map: =hash(q) (12) Where q is the query request, For query requests Position on the hash ring; 5.2) Adjacent node selection: (13) in represents the adjacent nodes of the query request q, and k represents the number of neighbors of each node; 5.3) Parallel calculation of candidate sets: (14) in represents the local candidate set generated by each node, and q represents the query request; 5.4) Candidate set and sorting: Merge and sort the local candidate sets generated by each node , and obtain the final candidate set ,The merged candidate set is sorted by the distance from the query request q, and the most similar k results are selected.

Images