An efficient approximate query system and method based on a multi-dimensional index tree

Abstract

The application discloses a kind of high-efficiency approximate query system and method based on multidimensional index tree, approximate query system includes: query request receiving module, for receiving query request;Exact query module, for according to the query request received by query request receiving module, from data set, the information corresponding to query request is matched, output query result, and form historical query information;The data set stores data according to the set approximate aggregation query framework;Historical query module, for storing historical query information according to the set historical information index structure, and according to the query request received by query request receiving module, from historical query information, the information corresponding to query request is matched, output query result.The application has the characteristics such as high-efficiency aggregation approximate query precision of multidimensional data, small index memory consumption, short query time etc..

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of database approximate query, and particularly relates to an efficient approximate query system and method based on a multi-dimensional index tree. BACKGROUND

[0002] In real life, many related query systems in many fields need not completely accurate aggregate query results. For example, in the meteorological field, for meteorological indexes (such as air pressure, temperature, humidity, etc.) in a certain area, only approximate statistical information such as mean value and variance is needed to know the approximate situation of the meteorological conditions in the area. If a complete data set is traversed to obtain completely accurate aggregate results, a large amount of computing resources will be consumed for the case of small collection granularity and too many meteorological stations in the area. Therefore, the approximate query system is used to deal with similar application scenarios. Traditional approximate query techniques based on sampling mainly have problems of low query accuracy, large index memory consumption, and long query time.

[0003] Data cube technology based on data pre-computation is often used for approximate calculation of aggregate queries. This method is different from real-time sampling technology. The data cube of the entire data set is established offline after data loading is completed. The cube records a subset of the aggregate results of the data set in a block manner. If the division is more detailed, the pre-aggregated results are more detailed, and the user's query request can be better covered, but the space consumption is larger, and the time consumption for establishing the data cube is larger. When the user's query request can be completely covered by the pre-computed data range in the cube, the query result is completely accurate without error. However, in reality, the diversity of query requests leads to the problem of missed judgment of the data cube.

[0004] Therefore, it is difficult to meet the approximate query processing in real situations by using a single approximate query technique, especially for high-dimensional data. The space required by the data cube is more huge, and a long time is needed to pre-establish the cube. SUMMARY

[0005] To solve the problems in the prior art, the application provides an efficient approximate query system and method based on a multi-dimensional index tree, which has the characteristics of high efficiency, high accuracy of multi-dimensional data aggregation approximate query, small index memory consumption, and short query time.

[0006] To achieve the above object, the technical scheme adopted by the application is as follows:

[0007] In a first aspect, an approximate query system is provided, comprising: a query request receiving module configured to receive a query request; an exact query module configured to match information corresponding to the query request from a data set according to the query request received by the query request receiving module, output a query result, and form historical query information; the data set stores data according to a set approximate aggregate query framework; and a historical query module configured to store the historical query information according to a set historical information index structure, and match information corresponding to the query request from the historical query information according to the query request received by the query request receiving module, and output a query result.

[0008] Further, the query request receiving module pre-processes the received query request, and then abstracts the query area in the query request into a rectangular form using a minimum bounding rectangle, to form a query rectangle; wherein the minimum bounding rectangle of an arbitrary polygon refers to the minimum outer bounding rectangle that can contain the polygon.

[0009] Further, storing data according to the set approximate aggregate query framework comprises: establishing a tree model by combining a data cube and a sampling-based query structure, using an exact data cube query when located at a non-leaf node, and using a sampling technique to approximately estimate a query result when the query falls in a leaf node.

[0010] Further, the set historical information index structure uses a space-filling curve for dimension reduction, i.e., mapping a two-dimensional point set to a one-dimensional coordinate system using a space-filling curve, for establishing a mapping relationship between coordinate points and actual storage locations using a hierarchical index model, and using a learning index structure to establish an index, i.e., for each piece of historical query information, only saving the position information Z LT of the upper-left node, and then learning the probability density function of the position information Z LT and the id value using a machine learning method, and finally generating a hierarchical index model.

[0011] Further, the historical query module comprises a historical query record table and a predicate mapping table; the historical query information defines a predicate space in the form of a bounding rectangle, and the z-order curve values of the left upper and right lower vertices of the rectangular area are stored in the historical query record table; meanwhile, the query result is recorded in an aggregate value column; the historical query record table serves as the basis of subsequent data structures, and only addition operations are allowed in the subsequent query process, and modification and deletion of existing records are not allowed; the predicate mapping table is used to establish the corresponding relationship between the space-filling curve values and the real latitude and longitude values; when the query rectangle obtains the first piece of historical data having an intersection or containing relationship with it through the hierarchical index model, the real data value corresponding to the range is found through a pointer for intersection area calculation, and meanwhile, the data in the predicate mapping table is arranged in sequence, for sequentially searching for subsequent intersection areas.

[0012] Further, in the historical query module, the two-dimensional predicate coordinates of the historical query information are placed in the divided spatial grid and correspond to the values on the space-filling curve applied to the spatial grid.

[0013] Further, the historical query module queries the first historical rectangle intersecting the query rectangle through the established hierarchical index model; through the pointer of the upper-left node of the intersecting rectangle and the predicate mapping table, the corresponding item in the predicate index table is found, and all intersecting areas can be found by sequentially scanning the subsequent historical records.

[0014] Further, after the historical query module finds all intersecting rectangles, the effective intersection area of the rectangles and the query rectangle is calculated, and then it is judged whether the query is an effective hit of the historical query record; given the query area Q, the candidate historical record set S = {Q h1 ,Q h2 …Q hn}, and a threshold τ, if the intersection area of Q and S is greater than τ, the historical query is an effective hit; when it is judged to be an effective hit, the historical rectangles intersecting the query rectangle are traversed, and their intersection areas are calculated through the real latitude and longitude coordinates; at this time, for different aggregation function values, the calculation methods include: (1) AVG average, according to the size of each intersecting area, the weighted average of all averages is calculated as the average of the query window rectangle; (2) SUM summation, the total area of all intersecting areas is calculated as the SUM value of the query rectangle by equal proportion scaling according to the proportion of each historical rectangle intersection area in the total area of the historical rectangle; when the historical query rectangle has an overlapping area and the area is located within the query window rectangle, the aggregated value of the area needs to be reevaluated; given the set S overlap = {Q h1 ,Q h2 …Q hk} intersecting a certain area Ω within the query window rectangle Q, the aggregated value of the area is equal to the average of the values of each area.

[0015] Further, the accurate query module uses a tree-like approximate query structure for querying; through the shard index tree structure, the complete data set is divided, and the accurate aggregation results of the non-leaf nodes are stored; the data cube is pre-established and saved; when the range represented by the non-leaf node reached by the query is completely contained by the query rectangle, the accurate pre-aggregated value stored in the leaf node is directly used; when there is partial coverage, i.e., the range of the next bottom node partially intersects the query rectangle, the sampling aggregation estimation phase represented by the leaf node is entered to sample the points in the range in real time, and the probability distribution of the whole is estimated through statistical methods, and finally the estimated aggregation function result is obtained.

[0016] In a second aspect, a method for approximate query is provided, which adopts the approximate query system in the first aspect. The method comprises: preprocessing an input query request, abstracting a query region in the query request into a rectangular form using a minimum bounding rectangle to form a query rectangle, and using the query rectangle to query information through a historical query module and / or an accurate query module; when the historical query module exists, matching information corresponding to the query request from historical query information stored in the historical query module according to the query request, and outputting a query result; when the historical query module does not exist or information corresponding to the query request is not matched in the historical query module, matching information corresponding to the query request from a data set through the accurate query module according to the query request, outputting a query result, and storing historical query information in the historical query module.

[0017] Compared with the prior art, the present application has the following beneficial effects: the present application matches information corresponding to the query request from the data set through the accurate query module, outputs the query result, and forms the historical query information; the data set stores data according to the set approximate aggregation query framework; the historical query module stores the historical query information according to the set historical information index structure, and matches information corresponding to the query request from the historical query information according to the query request received by the query request receiving module, and outputs the query result. The present application has the characteristics of high efficient aggregation approximate query of multi-dimensional data, high accuracy, small index memory consumption, short query time, etc. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 Fig. 1 is a main flow diagram of an efficient approximate query method based on a multi-dimensional index tree provided by an embodiment of the present application;

[0019] Figure 2 Fig. 3 is a main flow diagram of constructing an accurate query module in an embodiment of the present application;

[0020] Figure 3 Fig. 4 is a detailed flow diagram of an efficient approximate query method based on a multi-dimensional index tree provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application will be further described below in conjunction with the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0022] Embodiment one:

[0023] The application discloses a high-efficiency approximate query system based on a multi-dimensional index tree, which comprises a query request receiving module, an accurate query module, and a historical query module.

[0024] The query request receiving module pre-processes the received query request, and then abstracts the query area in the query request into a rectangular form by using a minimum bounding rectangle to form a query rectangle.

[0025] The data is stored according to a set approximate aggregate query framework, which comprises the following steps: a tree model is established by combining a data cube and a sampling-based query structure; when located at a non-leaf node, an accurate data cube query is adopted; and when the query falls in a leaf node, a sampling technology is used to approximately estimate the query result.

[0026] The set historical information index structure uses a space-filling curve to reduce dimensions, that is, a two-dimensional point set is mapped into a one-dimensional coordinate system by using the space-filling curve, which is used to establish a mapping relationship between coordinate points and actual storage positions by using a hierarchical index model, and a learning index structure is used to establish an index, that is, for each historical query information, only the position information Z LT of the left upper node is saved. LT Then, the probability density function of the position information Z

[0027] The application improves the multi-dimensional data query system by using the historical query module, thereby improving the performance of the query system. Figure 1 As shown in the figure, the method comprises the following steps:

[0028] Step 10: after pre-processing the input data (the query request received by the query request receiving module), the query area is abstracted into a rectangular form by using a minimum bounding rectangle to form a query rectangle, and after selecting an aggregate function, the query request is input into the query system. The geographic area information under the actual coordinates is often irregular in shape, and it is difficult to directly analyze and calculate it, so the concept of minimum bounding rectangle (MBR) is introduced. The minimum bounding rectangle (MBR) of an arbitrary polygon refers to the minimum outer bounding rectangle that can contain the polygon. In subsequent queries, the concept of MBR is used for data processing, which is convenient for analysis and calculation.

[0029] Step 20, if there is historical query information, go to step 30, otherwise go to step 60.

[0030] Step 30, enter the historical query module, and query the first historical rectangle intersecting with the query rectangle through the established learning index. For the historical query module, mainly used for storing historical query information, as a cache part of the exact query module, to speed up the query process. The two-dimensional predicate coordinates of the historical query information are placed in the divided spatial grid, and are one-to-one corresponding to the values applied to the space-filling curve on the grid. The module mainly consists of the following two parts:

[0031] 1. Historical query record table

[0032] The historical query information defines a predicate space in the form of a boundary rectangle. For easy storage, the present application stores the z-order curve values of the left upper (Z LT ) and right lower (Z RB ) vertices of the rectangular region in the record table. At the same time, according to the previous query results, record in the aggregate value column. The historical record table serves as the basis for subsequent data structures, and only allows addition operations in the subsequent query process, and cannot modify and delete the existing records.

[0033] 2. Predicate mapping table

[0034] The table mainly establishes the correspondence between the space-filling curve values and the real latitude and longitude values. When the query obtains the first historical data with intersection or inclusion relationship through the index model, the real data value corresponding to the range will be found through the pointer for intersection area calculation. At the same time, since the data in the table is arranged in order, the subsequent intersection area can be sequentially searched in the table, thereby accelerating the calculation.

[0035] By applying machine learning model to the field of index establishment, the time and space overhead of query index can be greatly reduced. The present application maps the two-dimensional point set to one-dimensional coordinate system through space-filling curve, so that it can establish the mapping relationship between coordinate points and actual storage location through hierarchical index model. In order to facilitate retrieval and reduce unnecessary overhead, only the position information Z LT of the left upper node of each historical query record is saved. Then the probability density function of Z LT and id value is learned by machine learning method, and finally the hierarchical index model is generated.

[0036] Step 40, through the pointer of the left upper node of the intersection rectangle and the predicate mapping table, find the corresponding item in the predicate index table. Since the historical records intersecting with the same area must exist continuously in the corresponding storage space, only the subsequent historical records need to be scanned to quickly find all the intersection areas.

[0037] Step 50, after finding all intersecting rectangles, the effective intersection area of these rectangles and the query rectangle is calculated, and then it is judged whether the query is an effective hit of the historical query record. Given the query region Q, the candidate historical record set S = {Q h1 ,Q h2 …Q hn}, set threshold τ, if the intersection area of Q and S is greater than v, then the historical query is an effective hit.

[0038] If the historical query is an invalid hit, the system should go to the precise query phase of step 60 to perform a complete data set traversal query.

[0039] When it is judged to be an effective hit, the system will traverse the historical rectangles intersecting with the query window rectangle, and calculate their intersection area through real latitude and longitude coordinates. At this time, for different aggregation function values, the calculation method is also different: (1) AVG (average), according to the size of each intersection area, the weighted average of all average values is calculated as the average value of the query window rectangle. (2) SUM (sum), according to the proportion of each historical rectangle intersection area to the total area of the historical rectangle, the total sum of all intersection areas is calculated as the SUM value of the query rectangle.

[0040] When the historical query rectangle has overlapping regions, and the region is within the query window rectangle, the aggregation value of the region needs to be reevaluated. Set S overlap = {Q h1 ,Q h2 …Q hk} intersect in a region Ω within the query window rectangle Q, the aggregation value of the region is equal to the average of each region value. Through the above calculation, the historical record value of each part can be referred to as much as possible to make a more accurate evaluation under the condition that the intersection area is large enough.

[0041] Step 60, place the query rectangle into the complete data set for precise query. This step uses a tree-like approximate query structure for efficient query. Through the tree-like structure of the sharded index, the complete data set is divided, and the precise aggregation results of the non-leaf nodes are stored. The pre-established data cube is saved. When the range represented by the non-leaf node is completely contained by the query rectangle, the precise pre-aggregation value stored in the leaf node can be directly used. When there is partial coverage, that is, the range of the next bottom node intersects with the query rectangle, it will enter the sampling aggregation estimation phase represented by the leaf node. This phase samples the points in the range in real time, and estimates the overall probability distribution through statistical methods, and finally obtains the estimated aggregation function result.

[0042] Step 70, the calculation results obtained in the above steps are taken as the final query results.

[0043] Step 80, add the result to the history query module as history information for accelerating the following query process.

[0044] Step 90, query completed, exit the query system.

[0045] As shown in Figure 2 the flow of constructing the exact query module is as follows:

[0046] Step 601, use the traditional space partitioning method such as KD tree to divide the complete data set into regions, and recursively perform the process until the data volume contained in the region reaches the threshold.

[0047] Step 602, establish a hierarchical index structure for the divided region. Based on the index tree of a data set, the following conditions need to be met: (1) the tree is multi-branch, that is, the same dimension can be divided into multiple sub-regions in the same round. (2) The sub-trees divided by the tree correspond to non-intersecting space regions. (3) All sub-trees completely divide the space range represented by the parent node, that is, there is no space region that does not belong to any node.

[0048] Step 603, for non-minimal regions, pre-calculate the values of each aggregate function, and save them in the form of data cubes in the database. When the query region completely contains the non-leaf node, the saved data cube data can be directly used to obtain the exact aggregate result.

[0049] Step 604, correspond the above pre-computed results to the non-leaf nodes of the tree structure.

[0050] Step 605, select a sampler for future online aggregation sampling estimation for leaf nodes. The online aggregation method can be directly used for the estimation of the aggregate function value of the leaf node. The uniform sampling method can sample each sample with equal probability to obtain a sample set for estimating the overall aggregate function value. When the estimation is completed, the corresponding confidence interval is also obtained to describe the reliability of the estimated value.

[0051] The complete query process is mainly performed in the form of tree traversal. When performing a depth-first traversal of the index tree, if the range represented by the current node p is completely contained by the query range Q, the current recursion is terminated directly, and the pre-aggregated value of the node is returned. If the current node is independent of Q, 0 is directly returned as the result of the failed search. When partially overlapping, if the current p is a leaf node, the online aggregation result of the range is returned, otherwise all sub-trees are recursively traversed.

[0052] The application accelerates the query speed and reduces the space memory consumption by introducing a learning data structure. The application first constructs an approximate aggregation query framework for a complete data set, establishes a tree model by combining a data cube and a query structure based on sampling, uses an accurate data cube query when located at a non-leaf node, and uses a sampling technology to approximately estimate the query result when the query falls on a leaf node. Next, the historical query information is saved to a database, and an index structure about the historical information is established. The structure uses a space-filling curve for dimension reduction and uses a learning index structure to establish an index, thereby improving the query efficiency. Next, it is determined whether the newly arrived query information hits the historical information. If a hit is successfully made, the aggregation result is directly obtained. Otherwise, a complete data set is traversed for a query. The application greatly improves the hit efficiency through the historical query module when there is a large amount of historical information, so that a relatively accurate query result can be obtained without traversing the complete data set. At the same time, because the tree index structure is introduced, the query for the complete data set still has better query efficiency than the traditional method.

[0053] In the application, the historical query module records the spatial position of the top-left vertex of the historical rectangle in a one-dimensional coordinate through a space-filling curve, and introduces a learning index to record the correspondence between the identifier and the storage position of the vertex, thereby accelerating the index process. At the same time, all the recorded information is recorded in a predicate mapping table for subsequent acceleration of the retrieval process. The accurate query module divides the complete data space using a KD tree to establish a hierarchical tree index structure, divides the query region into completely covered and partially covered parts, the completely covered part corresponds to a non-leaf node of the tree structure, uses a data cube to store the accurate result of the aggregation result, and can be directly used to constitute the final query result. The partially covered part corresponds to a leaf node, and needs to estimate the aggregation result of the part through a sampling method. The index tree can greatly reduce the approximate query error and realize fast approximate query.

[0054] Embodiment two

[0055] Based on the high-efficiency approximate query system based on a multi-dimensional index tree in embodiment one, the embodiment provides a high-efficiency approximate query method based on a multi-dimensional index tree. The method comprises the following steps:

[0056] The input query request is preprocessed, the query region in the query request is abstracted into a rectangular form using a minimum bounding rectangle, a query rectangle is formed, and information query is performed through the historical query module and / or the accurate query module;

[0057] When the historical query module exists, the information corresponding to the query request is matched from the historical query information stored in the historical query module according to the query request, and the query result is output;

[0058] When the historical query module does not exist or no information corresponding to the query request is matched in the historical query module, information corresponding to the query request is matched from the data set by the exact query module according to the query request, a query result is output, and historical query information is formed and stored in the historical query module.

[0059] The above merely describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. An approximate query system, characterized in that, include: The query request receiving module is used to receive query requests; The precise query module is used to match the information corresponding to the query request from the dataset according to the query request received by the query request receiving module, output the query results, and form historical query information; the dataset stores data according to a set approximate aggregation query framework. The historical query module is used to store historical query information according to the set historical information index structure, and to match the information corresponding to the query request from the historical query information according to the query request received by the query request receiving module, and output the query results. The query request receiving module preprocesses the received query request, and then uses the minimum bounding rectangle to abstract the query area in the query request and unify it into a rectangular form to form a query rectangle; wherein, the minimum bounding rectangle of an arbitrary polygon refers to the minimum outer bounding rectangle that can contain the polygon. Data is stored according to the set approximate aggregation query framework, including: building a tree model by combining data cubes and sampling-based query structures; using precise data cube queries when the query is located in a non-leaf node; and using sampling techniques to approximate the query result when the query falls in a leaf node. The established historical information index structure utilizes space-filling curves for dimensionality reduction, mapping a two-dimensional point set to a one-dimensional coordinate system. This is used to establish the mapping relationship between coordinate points and actual storage locations through a hierarchical indexing model. Furthermore, a learned index structure is used to build the index, meaning that for each historical query record, only the position information of its top-left node is saved. Then, location information was learned using machine learning methods. The probability density function of the ID value is used to generate a hierarchical index model.

2. The approximate query system according to claim 1, characterized in that, The historical query module includes a historical query record table and a predicate mapping table; Historical query information defines a predicate space in the form of a bounding rectangle. The z-order curve values ​​of the top left and bottom right vertices of the rectangular region are stored in the historical query record table. At the same time, the query results are recorded in the aggregate value column. The historical query record table serves as the basis for subsequent data structures. During subsequent queries, only addition operations are allowed, and modification or deletion of existing records is not permitted. The predicate mapping table is used to establish the correspondence between space-filling curve values ​​and actual latitude and longitude values; Once the query rectangle obtains the first historical data that intersects or contains the rectangle through the hierarchical index model, the pointer finds the actual data value corresponding to that range for calculating the intersection area. At the same time, the data in the predicate mapping table is arranged in order for sequentially finding subsequent intersection areas.

3. The approximate query system according to claim 2, characterized in that, In the historical query module, the two-dimensional predicate coordinates of the historical query information are placed in the divided spatial grid and correspond one-to-one with the values ​​on the spatial filling curve applied to the spatial grid.

4. The approximate query system according to claim 2, characterized in that, The historical query module uses the established hierarchical index model to find the first historical rectangle that intersects with the query rectangle. By using the pointer between the top left node of the intersecting rectangle and the predicate mapping table, it finds the corresponding item in the predicate index table and scans subsequent historical records in sequence to find all intersecting areas.

5. The approximate query system according to claim 4, characterized in that, After the historical query module finds all intersecting rectangles, it calculates the effective intersection area between these rectangles and the query rectangle, and then determines whether the query effectively hits the historical query record. Known query range Alternative historical record set Set a threshold ,like and Intersecting area greater than If so, the historical query has been successfully completed. When a valid match is determined, the historical rectangles that intersect with the query rectangle are traversed, and their intersection area is calculated using the actual latitude and longitude coordinates. The calculation method varies depending on the aggregation function value: (1) AVG mean: The weighted average of all means is calculated according to the size of each intersecting region and used as the mean of the query window rectangle. (2) SUM summation: The SUM value of the query rectangle is calculated by scaling the intersection area of ​​each historical rectangle proportionally to the total area of ​​the historical rectangle, and then calculating the sum of all intersection areas. When historical query rectangles overlap and the area falls within the query window rectangle, the aggregated value of that area needs to be re-evaluated; let the set be... Intersecting the query window rectangle A certain area within The aggregate value of this region is equal to the average value of all regions.

6. The approximate query system according to claim 5, characterized in that, The precise query module uses a tree-like approximate query structure for querying. It divides the complete dataset using a sharded index tree structure and stores the precise aggregation results of non-leaf nodes. A pre-built data cube is used for storage. When the range represented by a non-leaf node is completely contained within the query rectangle, the precise pre-aggregated value stored in that leaf node is directly used. When partial coverage occurs, i.e., the range of the next lower-level node partially intersects with the query rectangle, the sampling aggregation estimation stage represented by the leaf node is entered. Points within the range are sampled in real time, and the overall probability distribution is estimated using statistical methods, ultimately yielding the estimated aggregation function result.

7. An approximate query method, characterized in that, The approximate query system according to any one of claims 1 to 6, the method comprising: The input query request is preprocessed, and the query area in the query request is abstracted using the minimum bounding rectangle and unified into a rectangular form to form a query rectangle, which is used to query information through the historical query module and / or the precise query module. When a historical query module exists, the system matches the information corresponding to the query request from the historical query information stored in the historical query module and outputs the query results. When the historical query module does not exist or no information corresponding to the query request is found in the historical query module, the precise query module matches the information corresponding to the query request from the dataset, outputs the query results, and stores the historical query information in the historical query module.

Images