Efficient query method integrating entity analysis and KNN query

Abstract

The invention relates to an efficient query method integrating entity analysis and KNN query. The efficient query method comprises the following steps of S1, data set preparation and data fragmentation; s2, creating a region tree index; s3, performing entity analysis based on the region tree index; and S4, performing KNN query based on the region tree index. According to the method, the extended data set is decomposed into a plurality of small subsets, and entity analysis is performed on each small subset by applying the region tree index, so that the configuration requirement on physical equipment is correspondingly reduced, and the analysis process is quick and effective. The query results of the KNN query processing algorithm used in the method are different clusters or representatives thereof, so that efficient and accurate KNN query can be realized, the problem of result deviation caused by repeated data in traditional KNN query is avoided, and the method can be applied to complex query scenes in a big data environment.

Description

Technical Field

[0001] The present invention relates to a database retrieval method, in particular to an efficient query method integrating entity resolution and KNN query. Background Art

[0002] With the rapid development of information technology today, the frequency of data updates and the speed of data generation are far greater than in the past. In large-scale information system projects, it is sometimes necessary to integrate and merge data from multiple data sources to improve the quality of data information, achieve information complementarity of data resources, and provide a more unified, complete, and clean data set for subsequent data processing. Compared with multiple data sets before merging, the integrated data set has higher value and can provide users with more valuable information. In this process, a very important step is entity resolution (ER), which is the process of identifying tuples (or records) pointing to the same entity in the real world in the same data source or different data sources and merging them.

[0003] Generally speaking, entity resolution usually needs to consider two situations: disambiguation and deduplication. These two problems refer to: (1) which entity a given tuple describes; (2) which tuples describe the same entity in the real world.

[0004] K-nearest neighbor query (KNN query) is to find the K tuples closest to the query tuple and sort the results according to the specified sorting function. K-nearest neighbor query is also called KNN query or Top-K query.

[0005] Currently, the KNN query method has not been combined with entity resolution. Due to the existence of duplicate data, especially for multiple data sources, there may be multiple tuples describing the same entity. If entity resolution is not considered, then the KNN query may fail. Summary of the invention

[0006] The purpose of the present invention is to provide an efficient query method integrating entity resolution and KNN query to solve the problem of result deviation caused by duplicate data in traditional KNN query.

[0007] The object of the present invention is achieved in that:

[0008] An efficient query method integrating entity resolution and KNN query includes the following steps:

[0009] S1. Dataset preparation and data sharding: Divide the data in dataset R into w subsets: The amount of data in each subset is the same; in the wth subset Rw contains the remaining tuples after the previous subsets process the data, that is, |R w |≤|R 1 |=···=|R w-1 |; The first subset R 1 As a sample set, it is used to create a region tree index; the region tree index is used to parse other subsets (R 2 ,···,R w ) in the tuple.

[0010] S2. Create a region tree index: For the data set R(U) = R(tid,A 1 ,···,A n ), where tid is the tuple identifier; A 1 ,···,A n are different attributes; U is a set of tuples, and U={tid,A 1 ,···,A n}; The area contained in the data set R(U) is obtained by calculation And a i =min(A i )-δ, b i =max(A i ); where min(A i ) represents attribute A i The minimum value, max(A i ) represents attribute A i The maximum value of , δ is a constant; for all t∈R, let A∈U be the attribute with the largest number of non-repeated values ​​in t[A]. If t[A] is unique for all t∈R, then attribute A is a candidate key for the dataset R; this attribute A is used to generate a non-decreasing sorted list in the region tree index, which is called a list attribute; the region tree index is obtained by It is created by partitioning, that is, by recursively calling the algorithm PRC.

[0011] S3, Entity resolution based on region tree index: After finding the query tuple Leaf Node Then, we get the leaf node Points to the list L k ; If tuple t is in list L k If there is a tuple r in r.Ctid that is equal to it, then tuple t is a duplicate tuple, and then the identifier t[tid] of tuple t is appended to the end of r.Ctid; otherwise, tuple t is a new tuple, and the tuple t is inserted into the list L. k middle.

[0012] S4. KNN query based on region tree index: For any given query object, first quickly find the location of the query point, calculate the distance between the query point and the neighboring points, and take the minimum distance as the region radius, call the region query algorithm, find all tuples in the region, and calculate the distance to the query point in turn, and find the K data objects closest to it; if there are less than K, expand the region radius and continue to call the region query until K data objects are found; if the distances of the first K data objects are all less than or equal to the query radius, return the query result; if not satisfied, continue to search with the maximum distance among the K data objects as the query area until the distances of the first K data objects are all less than or equal to the query radius.

[0013] Furthermore, the query method of the present invention further comprises the following steps:

[0014] S5. Output of KNN result set: Return the KNN result set obtained by the query to the user.

[0015] The query method of the present invention is an accurate KNN query based on data integration and deduplication. The region of the data set is dynamically divided by the spatial partitioning algorithm PRC, and a region is divided into several non-overlapping and complete sub-regions. A region tree index is established during the partitioning process. This divide-and-conquer mechanism can effectively perform real-time entity resolution. The present invention decomposes the extended data set into several smaller subsets, and applies the region tree index to perform entity resolution on each smaller subset, which correspondingly reduces the configuration requirements for physical devices and makes the resolution process fast and effective. The KNN query processing algorithm used in the present invention has query results that are different clusters or their representatives, so that efficient and accurate KNN queries can be achieved, avoiding the result deviation problem caused by duplicate data in traditional KNN queries, and can be applied to complex query scenarios in big data environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the query method of the present invention.

[0017] Figure 2 is an example graph of a two-dimensional spatial region query; Figure 2 (a) is an inclusion relationship, Figure 3 (b) It is an intersection relationship.

[0018] Figure 3 It is a two-dimensional space to find K candidate tuple example graphs; among them, Figure 3 (a) is r≤d m relation, Figure 3 (b) r ≥ d m relation.

[0019] Figure 4 This is a schematic diagram of the region tree index structure.

[0020] Figure 5 It is a time diagram for partitioning a low-dimensional data set.

[0021] Figure 6 It is a time diagram for dividing medium and high dimensional data sets.

[0022] Figure 7 It is a schematic diagram of the number of query area blocks in the low-dimensional data set.

[0023] Figure 8 This is a schematic diagram of the number of query area blocks in the SIFT dataset.

[0024] Fig. 9 This is a schematic diagram of the number of query area blocks for the HW dataset.

[0025] Fig.10 It is a schematic diagram of the number of candidate tuples of a low-dimensional data set.

[0026] Fig.11 This is a schematic diagram of the number of candidate tuples in the SIFT dataset.

[0027] Fig.12 It is a schematic diagram of the number of candidate tuples of HW dataset.

[0028] Fig.13 It is a statistical graph of query time consumption for low-dimensional data sets.

[0029] Fig.14 This is a statistical graph of the time consumption for querying SIFT datasets.

[0030] Fig.15 This is a statistical chart of the time consumption for querying HW data sets. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, the efficient query method integrating entity resolution and KNN query of the present invention comprises the following steps:

[0033] Step 1: Data set preparation and data sharding.

[0034] S1-1 Dataset preparation: There are 15 sample datasets in total, including 5 low-dimensional datasets and 10 medium- and high-dimensional datasets. The corresponding clean datasets and dirty datasets are obtained from the original datasets. "Clean dataset" refers to a dataset without duplicate data. "Dirty dataset" refers to a dataset containing duplicate data. Low-dimensional datasets (2D, 3D, 4D, and 10D) include artificially synthesized datasets and real datasets. Among them, the real datasets include Census2D, Census3D (two-dimensional and three-dimensional projections of U.S. Census Bureau data fragments), Cover4D, and Cover10D (four-dimensional and ten-dimensional projections of CovType datasets, used to predict forest cover types based on mapping variables); the synthetic dataset is Gauss3D. These datasets are abbreviated as C2D, C3D, G3D, C4D, and C10D, respectively. The 10 medium- and high-dimensional (20D-784D) datasets are SIFT128D and HW784D or some of their own projections or themselves (referred to as SIFT-like and HW-like datasets, respectively). SIFT128D comes from a real dataset, and a part of it is taken and synthesized, which reduces its data quality and increases the amount of repeated data. HW784D comes from an artificially synthesized dataset. The suffix "nD" indicates that the dataset is n-dimensional. For example, n-dimensional means n attributes (A1, ···, An) in addition to the identifier tid.

[0035] S1-2 Data Sharding: For the dataset R(tid,A 1 ,···,A n ), divide the data into w subsets: The amount of data in each subset is the same. w contains the remaining tuples after the previous subsets process the data, that is, |R w |≤|R 1 |=···=|R w-1 |. The first subset R 1 As a sample set, it is used to create a region tree index. This region tree index is used to parse other subsets (R 2 ,···,R w ) in a distributed database system. 1 ,R 2 ,···,R w} can be w different data sets under the same mode.

[0036] Dataset R(tid,A 1 ,···,A n ) can be easily divided. For example, assuming that there are s segments in total, the data segment R in the relational database system can be divided into R i =σ Fi(R) Given, F i = ((i - 1)·s < tid ≤ i·s) is the selection condition for obtaining the fragment R i , where 1 ≤ i ≤ w. Therefore, each fragment R i is a subset of the data set R.

[0037] Step 2: Create a region tree index.

[0038] For the data set R(U) = R(tid, A 1 , ···, A n ), where tid is the tuple identifier; A 1 , ···, A n are different attributes; U is a set of tuples, and U = {tid, A 1 , ···, A n}. By calculation, the regions included in the data set R(U) are obtained, and a and a i = min(A i ) - δ, b i = max(A i ); where min(A i ) represents the minimum value of the attribute A i , max(A i ) represents the maximum value of the attribute A i , and δ is a constant; if the attribute A i is a decimal attribute, then δ = 0.000001; if the attribute A i is an integer attribute, then δ = 1. For all t ∈ R, let A ∈ U be the attribute with the largest number of non-repeating values in t[A]. If for all t ∈ R, t[A] is unique, then the attribute A is a candidate key of the data set R. This attribute A is used to generate a non-decreasing sorted list in the region tree index, which is called the list attribute.

[0039] The created region tree index consists of two parts. One is a tree, and each node (including leaf nodes and non-leaf nodes) in this tree is a cell (a cell is a region ), and its form is: The other is a set of sorted lists. Each leaf node in the tree points to a list, and the data in the list is sorted according to the list attribute A.

[0040] The region tree index is created by dividing the region , that is, by recursively calling the algorithm PRC. In addition, different structure trees can be constructed, for example, a balanced tree or an unbalanced tree. In the present invention, a structure such as Figure 4In the balanced tree shown, each node has twelve child nodes, forming a dodecagonal tree structure. Each bottom-level leaf node points to a list sorted by list attribute A, and the data in the list are all in the area contained by the leaf node.

[0041] The region tree index is a hierarchical index structure based on data partitioning, which aims to simplify and accelerate the query process by partitioning the data space. The region tree index is built on the dirty data to be parsed, and the nodes are divided according to the data characteristics to form a multi-level tree structure. This structure is used to record and manage the geographical or feature distribution of data, and can effectively support KNN queries. For each data set, two sample sets with different data volumes are used to create the ER index, one sample set contains 10,000 tuples, and the other sample set contains 100,000 tuples, which are randomly selected from the data set.

[0042] The time required to partition the low-dimensional data set when the number of samples extracted is 10,000 and 100,000 respectively is as follows: Figure 5 As shown in the figure, when the number of samples is the same, the time consumption will also increase when the dimension of the data set increases, such as C2D to C3D and C4D to C10D. The smaller the sample data volume, the less time consumption. When the sample is 10,000, the time consumption ranges from 7.92ms to 25.39ms, and when the sample is 100,000, the time consumption ranges from 70.80ms to 170.47ms.

[0043] The time required to partition medium and high dimensional data sets when the sample size is 10,000 and 100,000 is as follows: Figure 6 As shown in the figure. When the number of samples is 10,000, the time consumption of the "HW" dataset ranges from 47.36ms to 186.30ms; the time consumption of the "SIFT" dataset ranges from 39.41ms to 207.71ms. When the number of samples is 100,000, the time consumption of the "HW" dataset ranges from 359.10ms to 1958.72ms; the time consumption of the "SIFT" dataset ranges from 361.84ms to 1977.68ms. In the same dataset, the higher the data dimension, the higher the time consumption.

[0044] Experiments were conducted on 15 datasets, including low-dimensional datasets, "SIFT" datasets, and "HW" datasets. From the perspective of the time required to partition a dataset, for the same dataset, the larger the sample size, the longer the partition time; for different datasets, when the sample size is roughly the same, the higher the dimension, the longer the partition time.

[0045] Step 3: Entity resolution based on region tree index.

[0046] Find the query tuple Leaf Node Then, we get the leaf node Points to the list L k If tuple t is in list L k If there is a tuple r in r.Ctid that is equal to it, then tuple t is a duplicate tuple, and then the identifier t[tid] of tuple t is appended to the end of r.Ctid; otherwise, tuple t is a new tuple, and the tuple t is inserted into the list L. k where t is the representative of this new tuple.

[0047] In order to list L k In order to quickly determine whether the query tuple is duplicate data, the present invention proposes to use a binary search algorithm combined with a region tree to perform real-time entity resolution. k The way to determine whether the query tuple is duplicate data is to use the region tree index to find the leaf node Leaf Node Points to the list L k There is clean data, and the clean data has attribute A 1 Incrementally, at this time, the data to be processed q={tid,A 1 ,···,A n}'s property A 1 Use the binary search method to find its location. If the list attribute A 1 If they are the same, compare other attributes A 1+i (i=0,1,2···n-1); if other attributes are the same, the data q to be processed is duplicate data, record its ID, end the current processing process, and proceed to the next data processing. If other attributes are different, continue to search and compare downward until the binary search is completed. At this time, the data is clean data, record the current position, insert data q into the current position, end the current processing process, and proceed to the next data processing.

[0048] After the region tree index is established, the sample data is cleaned, and duplicate data is removed by entity resolution during the cleaning process. The entity resolution (ER) process involves identifying and merging identical or similar tuples to eliminate duplicate entities in the data set. By performing deduplication and clustering between different shards of the data set, a more accurate data representation is obtained. The cleaned data is organized by the new region tree index so that when performing KNN queries, the deduplication data can be directly used, and the data can be inserted in batches for entity resolution operations, and data shards other than the sample shards can be processed.

[0049] The time consumption of sample entity resolution is not only affected by the number of samples, but also by the data distribution. The more evenly the data set is distributed, the less time is consumed for entity resolution. When the distribution is roughly the same, the higher the data dimension, the longer it takes, and the larger the data volume, the longer it takes.

[0050] Step 4: KNN query based on region tree index.

[0051] For any given query object, first quickly find the location of the query point, calculate the distance between the query point and the neighboring points, and take the minimum distance as the region radius. Call the region query algorithm to find all tuples in the region, and calculate the distance to the query point in turn to find the K data objects closest to it; if there are less than K, expand the region radius and continue to call the region query until K data objects are found; if the distances of the first K data objects are all less than or equal to the query radius, return the query result; if not satisfied, continue to search with the maximum distance among the K data objects as the query area until the distances of the first K data objects are all less than or equal to the query radius.

[0052] When receiving a query request Q, the system uses the cleaned region tree index to perform a KNN query. First, quickly find the location of the query point, calculate the distance between the query point and its neighboring points, and take the minimum distance as the region radius. Then call the region query algorithm to find all tuples in the region, and calculate the distance to the query point in turn to find the K data objects closest to it. If there are not enough K, expand the region radius and continue to call the region query (such as Figure 2 ), until K data objects are found. If the distances of the first K data objects are all less than or equal to the query radius, the query result is returned; if not, the search continues with the maximum distance among the K data objects as the query area until the distances of the first K data objects are all less than or equal to the query radius (as shown in Figure 3 Finally, the K tuples closest to the query Q are extracted from the leaf nodes to form the KNN result set. For the three types of data sets, namely, low-dimensional, “HW” and “SIFT”, Figure 7-Figure 15 The average number of query area blocks, the average number of candidate tuples, and the average query time for each data item are shown for 1000 queries when K = 10, 20, 50, 100, and 200, respectively.

[0053] In terms of query processing, the smaller the data volume and the more evenly distributed the data set is, the less time it takes to perform point queries, area queries, and KNN queries. Conversely, the more time it takes.

[0054] Step 5: Output of KNN result set.

[0055] The KNN result set obtained by the query is returned to the user. Since entity resolution is performed before the KNN query, each tuple in the KNN result set is an independent entity, avoiding the interference of duplicate data, thereby improving the accuracy of the results and user experience.

[0056] The present invention uses 15 data sets of different dimensions for experimental verification, wherein the dimension n ranges from 2≤n≤784. The region tree index and algorithm used in the present invention have good performance in real-time entity resolution and KNN query processing.

Claims

1. An efficient query method integrating entity resolution and KNN query, characterized in that: The following steps are involved: S1. Dataset preparation and data sharding: Divide the data in dataset R into w subsets: The amount of data in each subset is the same; in the wth subset R w contains the remaining tuples after the previous subsets process the data, that is, |R w |≤|R1|=···=|R w-1 |; Use the first subset R1 as the sample set to create a region tree index; use the region tree index to parse other subsets (R2, ···, R w ) in tuples; S2. Create a region tree index: For the data set R(U) = R(tid,A1,···,A n ), where tid is the tuple identifier; A1, ···, A n are different attributes; U is a set of tuples, and U={tid,A1,···,A n }; The area contained in the data set R(U) is obtained by calculation And a i =min(A i )-δ, b i =max(A i ); where min(A i ) represents attribute A i The minimum value, max(A i ) represents attribute A i The maximum value of , δ is a constant; for all t∈R, let A∈U be the attribute with the largest number of non-repeated values ​​in t[A]. If t[A] is unique for all t∈R, then attribute A is a candidate key for the dataset R; this attribute A is used to generate a non-decreasing sorted list in the region tree index, which is called a list attribute; the region tree index is obtained by It is created by partitioning, that is, by recursively calling the algorithm PRC; S3, Entity resolution based on region tree index: After finding the query tuple Leaf Node Then, we get the leaf node Points to the list L k ; If tuple t is in list L k If there is a tuple r in r.Ctid that is equal to it, then tuple t is a duplicate tuple, and then the identifier t[tid] of tuple t is appended to the end of r.Ctid; otherwise, tuple t is a new tuple, and the tuple t is inserted into the list L. k middle; S4. KNN query based on region tree index: For any given query object, first quickly find the location of the query point, calculate the distance between the query point and the neighboring points, and take the minimum distance as the region radius, call the region query algorithm, find all tuples in the region, and calculate the distance to the query point in turn, and find the K data objects closest to it; if there are less than K, expand the region radius and continue to call the region query until K data objects are found; if the distances of the first K data objects are all less than or equal to the query radius, return the query result; if not satisfied, continue to search with the maximum distance among the K data objects as the query area until the distances of the first K data objects are all less than or equal to the query radius.

2. The efficient query method integrating entity resolution and KNN query according to claim 1 is characterized in that: The following steps are also included: S5. Output of KNN result set: Return the KNN result set obtained by the query to the user.

3. The efficient query method integrating entity resolution and KNN query according to claim 1 is characterized in that: The region tree index created in step S2 consists of two parts, one of which is a tree. Each node in the tree is a cell in the form of The other is a set of sorted lists. Each leaf node in the tree points to a list, and the data in the list is sorted according to the list attribute A.

4. The efficient query method integrating entity resolution and KNN query according to claim 1 is characterized in that: In step S3, in list L k The way to determine whether the query tuple is duplicate data is to use the region tree index to find the leaf node Leaf Node Points to the list L k There is clean data, and the clean data increases according to attribute A1. At this time, the data to be processed q={tid,A1,···,A n } attribute A1 uses the binary search method to find its location. If the list attribute A1 is the same, compare other attributes A 1+i (i=0,1,2···n-1); if other attributes are the same, the data to be processed q is duplicate data, and its ID is recorded, the current processing process ends, and the next data processing is carried out; If other attributes are different, continue to search and compare downward until the binary search is completed; at this time, the data is clean data, record the current position, insert data q into the current position, end the current processing process, and proceed to the next data processing.

Images