Uncertain data probability nearest neighbor query method based on locality sensitive hashing

Abstract

The invention provides a probabilistic nearest neighbor query method on high-dimensional continuous uncertain data, and relates to a method for quickly retrieving the nearest neighbor of the uncertain data by combining a locality sensitive hashing technology and probability calculation. The method comprises the following steps: firstly, constructing locality sensitive Hash for mapping, and mapping the uncertainty of a data object into a plurality of Hash tables through sampling; in a query stage, candidate adjacent objects are extracted from a given query point by using an index, and the probability that each candidate object becomes the nearest neighbor of the query point is calculated through Monte Carlo simulation. And finally outputting the neighbor object with the maximum probability and the probability value thereof. According to the method, the efficiency of high-dimensional uncertain data nearest neighbor query is greatly improved while the query accuracy is guaranteed, and the method can be widely applied to the fields of big data analysis, uncertain information retrieval and the like.

Description

Technical Field

[0001] This invention relates to a database similarity search method. Background Technology

[0002] High-dimensional uncertain data is prevalent in modern information systems such as sensor networks, medical imaging, and autonomous driving. This type of data typically exhibits location uncertainty, manifesting as a probability distribution region in a multidimensional space. Probabilistic Nearest Neighbor Query (PNNQ) is a fundamental operation for processing this type of data. Given an uncertain dataset D, a query point q, and an Euclidean distance metric dist(·,·), the goal of PNN search is to return all points in D that could be the nearest neighbor of q and their corresponding probabilities.

[0003] The core idea of ​​the current PNNQ method is to quickly filter out uncertain objects that may become nearest neighbors through an efficient pruning strategy, forming a candidate set, and then only calculate the exact probability value of points in the candidate set whose probability is greater than zero. Common methods include: (1) R-tree based method: pruning objects that cannot become nearest neighbors by calculating the minimum / maximum distance boundary (MBR) between the query point q and the data object, quickly identifying candidate objects with non-zero probability. (2) PV-cell (Possible Voronoi Cell) method: constructing a possible Voronoi cell for each uncertain object. If the query point q is located in the PV-cell of an object, the object is considered a possible nearest neighbor candidate, without needing to explicitly calculate the distance boundary. After initially filtering out candidate objects, it is usually necessary to calculate the probability of each candidate object becoming the nearest neighbor. In the case of low dimension, this can be accurately calculated through the joint probability density function. Summary of the Invention

[0004] This invention aims to overcome the low query efficiency of existing PNNQ methods in high-dimensional spaces and proposes an indexing framework for efficient probabilistic nearest neighbor queries on high-dimensional data. The basic idea is to use Locality-Sensitive Hashing (LSH) to reduce the dimensionality of high-dimensional uncertain data into low-dimensional deterministic data, and then perform fast filtering in the low-dimensional space to reduce distance calculation operations.

[0005] The specific process of the LSH-based probabilistic nearest neighbor query method for uncertain data is as follows:

[0006] Step 1: Generate random projection vectors;

[0007] Step two: Dimensionally reduce and store uncertain data;

[0008] Step 3: Find and filter to obtain the candidate set;

[0009] Step 4: Estimate the probability and output the result.

[0010] The beneficial effects of this invention are as follows:

[0011] This invention proposes a high-efficiency probabilistic nearest neighbor query method (PNN-LSH) based on Locality-Sensitive Hashing (LSH). This method can effectively build and query data objects with interval uncertainty (continuous uncertain data). Compared with traditional algorithms based on R-trees or PV-cells, this invention has better query performance, index building efficiency, and storage overhead control in data environments with both high-dimensional space and uncertainty.

[0012] This invention designs a two-endpoint projection mechanism to map the boundaries of uncertain objects into multiple low-dimensional spaces and uses B+ trees to build an index, significantly reducing sample sampling costs and high-dimensional distance computation overhead. During the query phase, a candidate set selection strategy based on collision counting is employed, and an adaptive search space adjustment mechanism is introduced to reduce unnecessary computation while maintaining accuracy. This method can effectively capture the probability distribution of nearest neighbors without requiring high-overhead integration or large-scale Monte Carlo sampling.

[0013] The PNN-LSH method of the present invention has the following advantages:

[0014] Supports high-dimensional uncertain data processing: Through two-endpoint LSH mapping, high-dimensional uncertain problems are transformed into a series of low-dimensional deterministic problems, alleviating the curse of dimensionality; Efficient candidate point filtering mechanism: The collision number filtering strategy significantly reduces the number of objects whose probabilities need to be calculated, improving query speed; Strong theoretical interpretability: It has clear probability lower bounds and search space termination conditions to ensure the correctness and stability of the results; Strong scalability: It maintains stable performance on datasets ranging from millions to tens of millions, demonstrating excellent adaptability in dimensionality-enhancing scenarios; Suitable for real-time or large-scale query tasks: It is particularly suitable for deployment in real-world scenarios such as medical diagnosis, autonomous driving, and sensor networks that require processing uncertain and high-dimensional data.

[0015] In summary, this invention provides a high-dimensional uncertain data probabilistic nearest neighbor query method that is simple in structure, has controllable accuracy, and excellent efficiency, and has broad engineering application prospects and promotional value. Attached Figure Description

[0016] Figure 1 A diagram illustrating the process of constructing the storage structure;

[0017] Figure 2This is a diagram of the query process. Detailed Implementation

[0018] The specific process of the LSH-based probabilistic nearest neighbor query method for uncertain data in this implementation is as follows:

[0019] Previous works on probabilistic nearest neighbor queries for uncertain data have typically focused on low-dimensional space. This involves storing continuous uncertain data in a spatial structure like an R-tree and then pruning the data within that structure. However, this approach is only suitable for low-dimensional uncertain data. In high-dimensional spaces, the performance of these methods significantly degrades due to the "curse of dimensionality." Firstly, traditional index structures suffer from severe node overlap in high dimensions, drastically reducing pruning efficiency. Secondly, they struggle to effectively identify nearest neighbors, potentially misclassifying all points as candidates in the worst case. Furthermore, accurately calculating the joint probability density function becomes impractical in high dimensions, while using Monte Carlo sampling to estimate probabilities requires a large number of sampling points, leading to excessively long query times.

[0020] Therefore, for PNNQ of high-dimensional uncertain data, there is an urgent need to develop efficient processing methods to overcome the limitations of existing technologies.

[0021] To address the issues of low efficiency in probabilistic nearest neighbor queries on high-dimensional uncertain data and severe degradation of traditional index structures in high-dimensional spaces, this invention designs a structured multi-stage probabilistic nearest neighbor query method, PNN-LSH. This method is based on Locality Sensitive Hashing (LSH) and constructs a two-endpoint uncertain projection mechanism to accurately represent the boundary information of high-dimensional uncertain objects. Simultaneously, it introduces a collision counting-guided candidate filtering strategy, combined with a dynamic search window adjustment mechanism, to effectively filter low-probability objects and reduce the candidate set size. Furthermore, it incorporates Monte Carlo probability estimation to achieve accurate probability calculation for candidate objects. Finally, it constructs a high-dimensional index system with a theoretical lower bound on the search radius and proof of candidate inclusion, significantly reducing index overhead and computational costs while improving query accuracy.

[0022] To achieve efficient indexing and fast filtering of high-dimensional uncertain data, this invention proposes an index building algorithm based on dual-endpoint mapping and multi-space construction. This algorithm takes a set of uncertain objects as input, uses the LSH principle to generate a multi-dimensional projection space, encodes the uncertain region of each object into multiple one-dimensional hash values, and constructs a corresponding index structure to support efficient retrieval. Its core process is shown in pseudocode algorithm 1.

[0023] The algorithm mainly completes two stages: generating a random hash function and mapping objects into an index structure. This is achieved by setting the upper and lower bounds of each object... The projection in the space can completely cover its possible value range, thus providing a low-overhead, high-dimensional, robust support structure for subsequent probabilistic retrieval. The specific process is shown in Algorithm 1, with an overall time complexity of O(n log n). The space complexity is Where n is the data size and d is the dimension. This represents the number of hash spaces.

[0024] Algorithm 1 Index building process enter: Uncertain dataset D, number of projected spaces m Output: A set of m -trees 1 Randomly generate m hash functions using random projections 2 For each data object o ∈ D: 3 Compute and (lower and upper projections) 4 Map o to projected interval pair in the i-th space 5 Construct the -tree based on the mapped values 6 Return the index set

[0025] In Algorithm 1, the input is first an uncertain dataset D, and each object... o is a high-dimensional uncertain point, represented as a d-dimensional hyperrectangular region, where Let represent the uncertain interval in the i-th dimension.

[0026] Line 2 calls `Generate Random Projections(m)`, which generates m independent random vectors. Each vector has dimension d, and its elements follow a Gaussian distribution. .

[0027] Line 3 iterates through each object. And extract its upper and lower boundary points. The lower boundary point of o is The upper bound of o is .

[0028] In line 4, for each projection space i, calculate the projection value:

[0029]

[0030] That is, it is projected as a one-dimensional value through dot product.

[0031] Line 5 inserts the projection values ​​of the upper and lower boundary points of each object in the i-th hash space into the corresponding B+ tree index structure Bi, constructing a total of m index trees. Each object is ultimately mapped to 2m values.

[0032] Line 6 returns the structure consisting of all m index trees. For use during the query phase.

[0033] To achieve efficient probabilistic nearest neighbor retrieval for uncertain data in high-dimensional space, this invention further proposes a collision-driven candidate generation algorithm. This algorithm combines the collision characteristics of the LSH hash space, dynamically expanding the search window and using multi-space collision counting to progressively filter objects with a high probability of becoming the nearest neighbor. It then constructs a search radius termination condition based on the minimum upper bound distance, thereby controlling the computational range and improving efficiency. The algorithm takes the query point q and the index structure as input, combining the confidence parameter λ and the collision threshold l, and employs an adaptive radius adjustment strategy to significantly reduce the number of candidate objects to be evaluated without sacrificing accuracy. The final output candidate set S is used in the probability estimation stage. Theoretically, this method guarantees recall, and in practice, it demonstrates good query speed and scalability. Its specific process is shown in Algorithm 2, and the worst-case time complexity is linear.

[0034] Algorithm 2 Query process enter: Query point q, dataset D, index set , number of projections m, confidence parameter Output: Candidate result set S 1 Initialize best object , current radius , candidate set 2 Compute hash values for to 3 Set initial window width and repeat the following until convergence: 4 Expand search window: 5 For each -tree : 6 Search for or in interval 7 Update the collision count 8 If o is not yet visited and , then add o to S1 9 Find the best candidate and update 10 Select from S1 to build candidate result set S

[0035] In Algorithm 2, the input is the query point q∈ and the m index trees already constructed in Algorithm 1 , as well as the collision threshold l and the confidence probability λ.

[0036] In line 2, the query point q is first hash-projected (dimensionality reduced) to obtain its projection value in each hash space:

[0037]

[0038] The third line initializes the search window w = 0, sets the step size Δw for each expansion, and initializes the distance R = ∞ between the candidate set S and the current optimal point o* and its minimum upper bound.

[0039] Line 4 performs the main loop iterative search: expanding the search window to... ;

[0040] Line 5 performs a range query in each B+ tree. Retrieves all objects whose hash values ​​fall within this range;

[0041] Line 7: For each hit object o, if it has not been visited before, update its collision count. Line 8: If object o accumulates a total of [number] collisions in m spaces. If it is found, then add it to the candidate set S1.

[0042] Line 9 updates the current best candidate point o*, which is defined as the minimum upper bound distance of distance q in the current candidate set. Object. Update the search termination condition based on this value:

[0043]

[0044] in It is the inverse function of the chi-square distribution, used to provide a lower bound on the probability.

[0045] In line 12, if the above condition is true, break out of the loop and find the point in S1. minimum distance from q These points construct the final candidate set S, and the candidate set S is returned, completing the search.

[0046] After the candidate set is filtered, we employ a Monte Carlo sampling-based probability estimation method to evaluate the likelihood of each data object in the candidate set becoming a probabilistic nearest neighbor. Specifically, for each uncertain object, we randomly sample N precise points within its defined attribute range and calculate the Euclidean distance from these sampled points to the query point q. Finally, the proportion of times these sampled points become nearest neighbors out of the total number of samples is used as the estimated probability that the object becomes a probabilistic nearest neighbor. Since this method is a widely used and mature traditional method in probabilistic nearest neighbor query tasks, and its calculation process is clear and straightforward, this paper will not elaborate on its specific implementation steps.

[0047] This invention constructs a high-dimensional probabilistic nearest neighbor query framework that integrates multi-stage uncertainty modeling, probabilistic space reasoning, and dynamic candidate optimization, achieving both theoretical completeness and engineering efficiency. The method introduces a dual-endpoint projection mechanism to achieve a compact representation of high-dimensional uncertain objects, and combines collision-guided candidate filtering with an adaptive search radius control strategy to effectively balance query efficiency and accuracy. Simultaneously, probability estimation and confidence boundary reasoning ensure the reliability and consistency of the results. The overall framework significantly reduces index building and query computation overhead while maintaining high recall, exhibiting good scalability and interpretability. It is particularly suitable for complex data scenarios requiring both accuracy and real-time performance, such as high-dimensional uncertain data analysis, intelligent sensing, medical retrieval, and security monitoring.

[0048] The beneficial effects of the present invention are verified using the following embodiments:

[0049] Example 1:

[0050] Dataset Description

[0051] To verify the applicability and effectiveness of the method of this invention in high-dimensional uncertain data retrieval tasks, this experiment was conducted based on six publicly available real datasets, including Audio, Sun, Enron, Msong, Sift1M, and Deep1M. These datasets broadly cover different types of feature vectors, such as text, images, and audio, and have different dimensions (from 128 to 1369 dimensions) and data sizes (from 50,000 to 1 million records), constituting a typical test environment for evaluating high-dimensional uncertain query methods.

[0052] To simulate real-world uncertain data scenarios, ±5% random perturbation was added to each feature dimension of each data point in the experiment, thus constructing an input object that conforms to the uncertainty of high-dimensional intervals. The model was repeatedly tested on each dataset, and all experimental metrics were averaged to ensure robustness.

[0053] Experimental process

[0054] We implemented the indexing and querying module of PNN-LSH in C++. All experiments were conducted on Ubuntu 20.04, using a 32-core Intel Xeon 8352V CPU and 60GB of memory. During the projection vector generation stage, the LSH space dimension m was set to 64, and each hash function used a standard normal distribution. A random vector is sampled from the middle. During the hash mapping process, the upper and lower boundary points of each uncertain object are projected to construct a B+ tree structure for each dimension. In the candidate generation (query) stage, the hash value of each query point q in the m projection spaces is calculated, and the search window is gradually expanded. Collision-guided candidate selection is performed. The collision threshold l is set to 40, and the confidence parameter... Furthermore, the termination search condition is adaptively determined using theoretical formulas, avoiding unnecessary distance calculations and sampling operations. After the candidate set is generated, a Monte Carlo method is used for probability estimation. For each candidate object, 100 instance points are randomly sampled and their Euclidean distances to the query point q are calculated to estimate the probability of it becoming the probabilistic nearest neighbor.

[0055] Evaluation indicators

[0056] To comprehensively evaluate the retrieval performance and efficiency of the PNN-LSH method proposed in this invention under high-dimensional uncertain data, this paper adopts several commonly used evaluation metrics, measuring them from the dimensions of query efficiency, indexing efficiency, accuracy, and scalability. Specific evaluation metrics are described below (see a):

[0057] a) Commonly used performance evaluation metrics in probabilistic nearest neighbor query tasks were selected, namely query time, index building time, index size, and recall.

[0058] Query Time: This refers to the total time it takes for the system to complete a probabilistic nearest neighbor query task, including all processes such as hash mapping, candidate filtering, distance calculation, and probability estimation. The unit is seconds (s). This metric measures the algorithm's response speed; a lower value is better.

[0059] Indexing Time: This refers to the time required for the algorithm to process the original dataset and build the query index structure, including vector projection under multiple hash spaces. Tree construction and other processes. This metric reflects the preprocessing efficiency of the method; shorter time is better.

[0060] Index Size: This refers to the actual storage size (in MB) of the constructed index structure in memory or on disk. It is used to evaluate the deployment capability of a method in a resource-constrained environment. The smaller the value, the more compact the structure.

[0061] Recall: Represents the percentage of the probability nearest neighbors that should actually be retrieved that are successfully captured by the PNN-LSH method, measuring the system's recall capability. This metric should be maintained above 99% to ensure fair comparison with other precise algorithms.

[0062] Results and Analysis

[0063] This invention's method was systematically compared with two mainstream uncertain nearest neighbor retrieval methods: the R-tree method, which uses a traditional spatial index structure and performs pruning using distance upper and lower bounds; and the PV-cell method, which filters candidate objects by constructing uncertain bounding boxes (UBRs). For a fair comparison, all methods used the same hardware environment and Monte Carlo probability estimation strategy, and the recall rate of this invention was set to be above 99% to compare its performance with precise methods.

[0064] The query time comparison results of the PNN-LSH model of this invention and the control experiment are shown in Table 1.

[0065] The comparison of construction time between the PNN-LSH model of this invention and the control experiment is shown in Table 1.

[0066] The index size comparison results of the PNN-LSH model of this invention and the control experiment are shown in Table 1.

[0067] Table 1. Query time (s) for PNN-LSH and control experiments

[0068] Dataset PNN-LSH R-tree PV-cell Audio 1.1 8.8 12 Enron 0.3 1.9 4.5 Sun 0.2 2.2 4.2 Sift1M 0.6 3.5 7.2 Deep1M 1.4 9.2 14.2 Msong 1.5 8.6 16.5

[0069] Table 2. Construction time (s) of PNN-LSH and control experiments

[0070] Dataset PNN-LSH R-tree PV-cell Audio 1.6 1.6 1.98 Enron 17.6 24.2 26 Sun 6.6 8.15 10.7 Sift1M 68 85 86 Deep1M 82 93 93 Msong 79 83 86

[0071] Table 1. Index size (MB) of PNN-LSH and control experiments

[0072] Dataset PNN-LSH R-tree PV-cell Audio 160 192 192 Enron 301 397 397 Sun 307 763 763 Sift1M 2200 2457 2457 Deep1M 3328 5222 5222 Msong 3328 7987 7987

[0073] As shown in Table 1, the PNN-LSH method proposed in this invention significantly outperforms existing mainstream methods in terms of query time. Compared to R-tree and PV-cell methods, PNN-LSH achieves the fastest response speed on multiple high-dimensional uncertain datasets. Its average query time on the Deep1M dataset is only 1.4 seconds, which is nearly 7 times faster than R-tree and nearly 10 times faster than PV-cell, fully demonstrating the method's rapid retrieval capability when facing large-scale, high-dimensional, and complex data. This advantage is mainly due to the efficient candidate selection strategy implemented by PNN-LSH through a collision mechanism in the projection space, which avoids redundant search and complex probability estimation, and significantly reduces search costs.

[0074] As shown in Tables 2 and 3, PNN-LSH also performs exceptionally well in terms of index building time and index space usage. Compared to traditional R-trees, it has a shorter building time and occupies less space. Compared to the PV-cell method, PNN-LSH indexes are built faster on the Msong dataset, with approximately 50% less index storage space, significantly reducing the burden of system deployment and maintenance. This indicates that PNN-LSH is not only suitable for offline query systems but also for resource-constrained applications or scenarios requiring frequent index rebuilding.

[0075] In summary, the PNN-LSH method proposed in this invention outperforms existing methods in terms of overall query performance, indexing efficiency, and storage cost, demonstrating significant practical value and engineering feasibility. During the candidate set generation stage, PNN-LSH effectively controls the candidate size by introducing a collision-driven filtering mechanism, resulting in more focused and accurate subsequent probability estimations and a fundamental improvement in the overall efficiency of the query system.

[0076] In summary, PNN-LSH, as a highly efficient method for querying uncertain data, not only demonstrates leading performance in accuracy, efficiency, and resource consumption, but also possesses excellent generalization ability and adaptability in its structural design. It is particularly suitable for scenarios with extremely high real-time and accuracy requirements in the context of massive uncertain data, such as intelligent search, risk detection, and sensor network data retrieval. Compared with current mainstream methods (such as R-tree and PV-cell), PNN-LSH exhibits stronger comprehensive capabilities and stability when handling large-scale complex query tasks, demonstrating significant technical advantages and broad application prospects.

[0077] Finally, this invention is Figure 1 The flowchart for building the index is shown. Figure 2 The flowchart for performing the query is shown in the image.

[0078] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A probabilistic nearest neighbor (PNN) query method for uncertain data based on locality-sensitive hashing (LSH), characterized in that: The specific process of the method is as follows: Step 1: Generate random projection vectors; Step two: Dimensionally reduce and store uncertain data; Step 3: Find and filter to obtain the candidate set; Step 4: Estimate the probability and output the result.

2. The LSH-based PNN query method according to claim 1, characterized in that: In step one, a random projection vector is generated; the specific process is as follows: Generate m sets of locality-sensitive hashes: Each LSH is a d-dimensional random vector, and each component is independently sampled from a standard normal distribution.

3. The method according to claim 2, characterized in that: Step two involves using the LSH vector generated in step one to reduce the dimensionality of the uncertain data. The specific process is as follows: For each continuous uncertain data object Divided into left endpoints and right endpoint Suppose the dataset has n uncertain objects. Take the dot product of the two endpoints of each uncertain object with the generated LSH to obtain two points after dimensionality reduction: The points after dimensionality reduction are stored in B+ trees, generating m B+ trees. Each B+ tree stores 2n points in its leaf nodes.

4. The method according to claim 3, characterized in that: Step three, which involves searching for a candidate set of query points within the reduced-dimensional space, includes: (1) Dimensionality Reduction of Query Points. For a d-dimensional query point q, through... Obtain the dimensionality-reduced position of q within each B+ tree. (2) Set up the search box. For each B+ tree, set a size of w as the center and w as the radius. The search box. (3) Count the number of collisions. Within each leaf node of B+, when the projection values ​​of the two endpoints of any point o are... or fall into Inside, the number of collisions of o increases by one. (4) Collision filtering. Set a collision threshold l. When the number of collisions to a point exceeds l, the collision will be filtered out. It is assigned to candidate set S1. (5) Update the candidate set S. At the start of the query, a distance threshold R is set as a filtering condition for whether a candidate can enter S from S1. Initially, it is set to ∞. The farthest distance from o to q is calculated for each candidate entering S1. and closest distance If the nearest distance Then o is assigned to S, and if the farthest distance Then update R to the farthest distance. (6) Expand the search space. Let And continue to repeat (3)-(5) to update the points falling into the search space. (7) Return the candidate set S. When The search stops and the candidate set S is returned. It is the inverse function of the cumulative distribution function of a chi-square distribution with l degrees of freedom.

5. The method according to claim 4, characterized in that: Step four involves estimating the probability of the candidate set as follows: For each point in the candidate set, N definite values ​​are selected from the uncertain interval as sample points according to the probability distribution of the uncertain interval. The Euclidean distance from all sample points to q is calculated. The probability that each point o is the nearest neighbor of q is... ,in The number of times a point becomes the nearest neighbor in each round of sampling for the value of o.

Images