Method for Constructing Multidimensional Learning Index for Partitioning Spatial Data Based on Frequently Queried Items

Abstract

The present invention provides a multidimensional learning index construction method for partitioning spatial data based on query frequent items, belonging to the field of learning index technology. The method mines the top-k query frequent items in the historical query data stream and evenly divides the grids according to the estimated query frequency in each grid. The probability of each grid being queried and read is basically similar. Therefore, when a grid has many query frequent items, the total number of data points in the grid is smaller. Therefore, when reading the disk page where the grid is located, fewer data points are traversed, thereby improving the query speed of spatial data under query-skewed workloads; a reasonably designed mapping function ensures that the mapping value of points located in small-numbered unit grids is smaller than the mapping value of points located in large-numbered unit grids, and assigns monotonically increasing mapping values to multidimensional data that is difficult to sort according to the grid number. Therefore, when processing query problems, the unit grids that need to be read can be easily obtained, reducing the reading of grids outside the query range, and improving the query speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of learning indexes, and in particular relates to a method for constructing a multidimensional learning index based on partitioning spatial data into frequent query items. Background Art

[0002] To support efficient queries, databases have relied on powerful indexing structures for decades. An index is a storage structure that sorts the values of one or more columns in a database table and serves as a list of logical pointers to the physically identified data pages within the table. An index that can only retrieve a single dimension of data is called a one-dimensional index, while an index that can retrieve multiple dimensions of data is called a multidimensional index. With the advancement of the information age, the number of data attributes is gradually increasing, and the amount of data is exploding. As data sets grow larger, multidimensional indexes consume increasing storage space and slow down search speeds. Therefore, building efficient multidimensional indexes has become a critical issue.

[0003] Multidimensional learning indexes are considered a solution to these problems and have become a research hotspot in the database field. Multidimensional learning indexes utilize machine learning models to construct indexes, replacing the tree structure used in traditional multidimensional indexes. By training the mapping relationship between data and storage locations, a learned model is generated. This model captures the distribution characteristics and patterns between data and storage locations, enabling improvements and optimization of multidimensional indexes.

[0004] Recent research has found that in many real-world applications, query requirements primarily revolve around a small portion of valuable data within the overall dataset. Frequently queried data items in a dataset are called frequent query items, and this phenomenon of frequent query items is known as query skew. For example, in map applications, iconic locations such as city centers, train stations, commercial districts, schools, and tourist attractions are often frequent query items. However, current multidimensional learning indexes have the following two issues when handling query-skewed workloads:

[0005] 1. Improper data layout design leads to reduced query speed. Data layout refers to the division of a data set according to a certain rule. The result of the division is called data layout. For example, Li et al. proposed a data layout based on data distribution density, which is divided into a grid. Combined with a machine learning model, they generated a searchable data layout for any multidimensional data set on disk pages. The data layout based on data distribution density cannot effectively adapt to query-skewed workloads. Because the amount of data allocated to each disk page by the index is basically the same, the query time for all data is basically the same. Therefore, under query-skewed workloads, there is not much difference in query time between frequent and non-frequent items, and the query speed of frequent items is not optimized.

[0006] 2. Improper mapping strategy design leads to slower query speeds. For example, Wang et al. proposed combining the Z-order curve and RMI (Recursive Model Index) to index multidimensional data. Using the Z-order curve as a mapping function can result in two points that are very close in multidimensional space having very different mapping values; conversely, two points with similar mapping values can be far apart in multidimensional space. This results in scanning many irrelevant points during the query process, which in turn reduces query speed. Summary of the Invention

[0007] In response to the problems existing in the prior art, the present invention provides a multidimensional learning index construction method for partitioning spatial data based on query frequent items, aiming to improve the query speed of spatial data under query tilt workload.

[0008] The technical solution of the present invention is:

[0009] A method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent items, the method comprising the following steps:

[0010] Step 1: Divide the d-dimensional spatial data V into grids evenly based on the estimated query frequencies of the Top-k query frequent items. Each small grid is called a unit grid, and each unit grid contains data points that exist within the longitude and latitude range of the unit grid.

[0011] The Top-k query frequent items refer to a given historical query data stream The k data items with the largest estimated query frequency;

[0012] The historical query data stream S={ω1, ω2, ..., ω p Each data item ω in * It consists of data points in spatial data V that answer the query question;

[0013] The data points in the d-dimensional space data V are represented by key=(x0, ..., x α ,...,x d-1 ), where x d-1 Represents the value of the data point on the d-1th dimension attribute;

[0014] Step 2: Calculate the mapping value of each data point in the spatial data V based on the distance between each data point and the starting point of the unit grid, and the distance between the starting point and the end point of the unit grid;

[0015] Step 3: Divide the data points in each unit grid into fragments with different numbers. Then point each fragment to a disk page and build a local model of the fragment to complete the construction of the multidimensional learning index. This index is named the frequent item learning index TKFI-LI.

[0016] The fragment is a logical address, represented by S id , representing the fragment numbered id; data points on the same unit grid will be pointed to multiple fragments, and the same fragment will be pointed to one or more disk pages; the mapping relationship between the fragments and disk pages is recorded in the local model of the fragment In; the local model of the fragment A local model consisting of a series of local fragments Composition, of which It is the fragment S with the ID id Local model, each local fragment has its own local model;

[0017] Step 4: Build a range query algorithm for use with TKFI-LI.

[0018] Furthermore, according to the multidimensional learning index construction method based on partitioning spatial data with frequent query items, the estimated query frequency refers to the data item ω * Querying data streams in history An estimate of the frequency of occurrence in .

[0019] Furthermore, according to the multidimensional learning index construction method based on partitioning spatial data based on query frequent terms, step 1 includes the following steps:

[0020] Step 1-1: mining the top-k frequent query items in the historical query data stream S, and determining the estimated query frequency of the top-k frequent query items;

[0021] Step 1-2: The d-dimensional spatial data V is evenly divided into grids based on the estimated query frequencies of the Top-k frequent items.

[0022] Furthermore, according to the multidimensional learning index construction method based on the partitioning of spatial data by query frequent items, the Sketch BF algorithm is used to mine the historical query data stream. and determining the estimated query frequencies of the Top-k query frequent items.

[0023] Furthermore, according to the multidimensional learning index construction method for partitioning spatial data based on query frequent items, the method for uniformly partitioning the d-dimensional spatial data V into grids based on the estimated query frequencies of Top-k query frequent items in step 1-2 includes:

[0024] First, the spatial data V is preprocessed according to the estimated query frequency of the Top-k query frequent items: different weights are assigned to the data points belonging to the Top-k query frequent items in the spatial data V. The data points with higher estimated query frequency have higher weights, while the data points with lower estimated query frequency have lower weights.

[0025] Then, traverse each dimension of the spatial data V and perform data partitioning operations on each dimension, thereby generating a grid data layout for the entire d-dimensional spatial data V;

[0026] The data partitioning operation includes:

[0027] Construct the cumulative distribution function along each dimension;

[0028] Divides each dimension into series based on the cumulative distribution function value of the data points in that dimension.

[0029] Furthermore, according to the multidimensional learning index construction method based on partitioning spatial data based on query frequent terms, step 3 includes the following steps:

[0030] Step 3-1: Calculate the fragment number based on the mapping value of the data point;

[0031] Step 3-2: Build a local model of the fragment

[0032] Step 3-2-1: Set the maximum number of data points stored in the disk page to Ω;

[0033] Step 3-2-2: Determine the data point set I in fragment Sid id and I id The mapping value of each data point in;

[0034] Step 3-2-3: Initialize the local model of fragment Sid And set and Empty; Is a list used to record storage I id The disk page address of the data point in When the number of disk pages recorded in is greater than 1, the mapping values of the data points stored in each disk page are recorded in sequence. middle;

[0035] Step 3-2-4: Determine the relationship between the number of data points Γ in the fragment Sid and the maximum number of data points Ω stored in the disk page. If Γ≤Ω, then I id stored in the same disk page and Add the address of the page to Set to empty; if Γ>Ω, then I id Divide the data into several parts according to the mapping value, and make each part contain less than Ω data points. The data points in each part are stored in the same disk page, and then all the storage I id The disk page address of the data point in is added to In the , the mapping values of the data on the same disk page are stored in order middle;

[0036] Step 3-2-5: Finally, local models of all local fragments Local models that are merged together to form fragments

[0037] Furthermore, according to the multidimensional learning index construction method based on partitioning spatial data by query frequent items, the shard prediction function is used to calculate the shard number, that is, the mapping values of all data points are input to the shard prediction function, and then the data points contained in each shard are output.

[0038] Furthermore, according to the multidimensional learning index construction method based on partitioning spatial data based on query frequent terms, step 4 includes the following steps:

[0039] Step 4-1: Get the cell grid that overlaps with the range query rectangle qr and decompose qr into smaller query rectangles qr′ j , each small rectangular box qr′ j Intersects only one unit grid;

[0040] For d-dimensional spatial data, enter the query range starting point q start =(l0, l1, ..., l d-1 ) and the end point q end =(u0, u1, ..., u d-1 ), generate a range query rectangle qr = [l0, u0) × ... × [l d-1 ,u d-1 );

[0041] Each of the small rectangular frames qr′ j Expressed as The starting point of the small rectangular box is represented by The end point is represented by For the small rectangular box qr′ j Overlapping cell grids;

[0042] Step 4-2: Move the small rectangular box qr′ j Divided into and Two categories: When the small rectangular box qr′ j and its unit grid When completely overlapped, this type of small rectangular frame qr′ j Defined as All data points in the unit grid where the small rectangular box is located are the answers that need to be output; when the small rectangular box qr′ j and its unit grid When the overlap is not complete, this type of small rectangular frame qr′ j Defined as Need to Filter the data points in the unit grid where the small rectangular box is located, and select the data points that meet the given query range to be added to the query results;

[0043] Step 4-3: Calculate the mapping value of the starting point and the mapping value of the ending point of each unit grid that overlaps with the range query rectangle qr, and use the fragment prediction function to predict the fragment where the calculated mapping value is located, and obtain the fragment number of the starting point of the unit grid and the fragment number of the ending point of the unit grid, thereby determining the fragment number interval corresponding to the unit grid;

[0044] Step 4-4: For each cell grid that overlaps with the range query rectangle qr, first obtain the numbers of all fragments in the cell grid according to the fragment number interval corresponding to the cell grid, and then use the local model of the fragment Get the disk pages corresponding to these numbered fragments;

[0045] Step 4-5: Filter the data points that fall within the range of qr from the disk pages obtained in step 4-4 and add them to the result R range Specifically:

[0046] for If the small rectangular box is included, all data points on the disk page corresponding to the cell grid where it is located are directly added to R range middle;

[0047] for If a small rectangular box is included, all the data points on the disk page corresponding to the unit grid where it is located will be judged and the data points belonging to Add the data points in R range middle.

[0048] Compared with the prior art, the technical solution proposed by the present invention has the following beneficial effects:

[0049] 1. To efficiently handle query-skewed workloads, the present invention designs a data layout that partitions spatial data based on frequent query terms. The top-k frequent query terms in the historical query data stream are mined, their estimated query frequencies are counted, and the grids are partitioned based on the uniformity of the estimated query frequencies within each grid. The uniformity of the estimated query frequencies across each grid means that the probability of each grid being queried and read is roughly the same. Therefore, when a grid has many frequent query terms, the total number of data points within that grid is smaller. Consequently, when reading the disk page containing that grid, fewer data points are traversed, thereby improving the query speed of spatial data under query-skewed workloads.

[0050] 2. The present invention designs an efficient mapping function for the aforementioned data layout, achieving faster query speeds. By designing a rational mapping function, the present invention ensures that the mapping values of points in smaller-numbered grid cells are always smaller than those in larger-numbered grid cells. This monotonically increasing property assigns monotonically increasing mapping values to multidimensional data, which is difficult to sort, according to the grid cell number. This allows queries to easily retrieve the desired grid cell from a large number of grid cells, reducing the need to read grid cells outside the query range and thus improving query speed.

[0051] 3. Compared with traditional multidimensional indexes, the multidimensional learning index constructed by the present invention takes into account the use of machine learning models instead of traditional tree structures, which can provide lower space overhead and faster query speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific methods in the embodiments of the present invention, the relevant drawings involved in the embodiments will be briefly described below. The following drawings are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative changes.

[0053] Figure 1 Schematic diagram of the process of constructing a multidimensional learning index based on the partitioning space of query frequent items in this embodiment;

[0054] Figure 2 This is a schematic diagram of the implementation process of the multi-dimensional learning index construction method based on the query frequent item partitioning space;

[0055] Figure 3 A graph showing the I / O ratio comparison test results for range queries using the index TKFI-LI constructed by the present invention and other existing indexes;

[0056] Figure 4 A graph showing the average time comparison test results for range queries between the index TKFI-LI constructed by the present invention and other existing indexes;

[0057] Figure 5 (a) Comparison test results of the space overhead of the index TKFI-LI constructed by the present invention and other existing indexes on the Uniform dataset; (b) Comparison test results of the space overhead of the index TKFI-LI constructed by the present invention and other existing indexes on the Zipf dataset;

[0058] Figure 6 The ablation test results of the query time overhead of the index TKFI-LI constructed by the present invention and other existing indexes. DETAILED DESCRIPTION

[0059] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0060] Figure 1 4 is a flowchart of a method for constructing a multidimensional learning index based on partitioning spatial data into query frequent items in this embodiment. Figure 2 This is a schematic diagram of the implementation process of the multi-dimensional learning index construction method based on query frequent item partitioning spatial data. Figure 1 and Figure 2 Describe the method in detail, such as Figure 1 and Figure 2 As shown, the method includes the following steps:

[0061] Step 1: Divide the d-dimensional spatial data V into grids evenly based on the estimated query frequencies of Top-k frequent items;

[0062] The spatial data V is information about the physical locations and shapes of geometric objects. These objects may be point locations or more complex objects such as countries / regions, roads or lakes, and are usually presented in a multidimensional form. Each piece of information in the d-dimensional spatial data V is called a data point key = (x0, ..., x α ,...,x d-1 ), where x d-1 Indicates the value of the data point on the d-1th dimension attribute. The implementation of this step is to divide the d-dimensional spatial data V into grids evenly based on the estimated query frequency of the Top-k query frequent items.

[0063] The Top-k query frequent items are defined as: given a historical query data stream and an integer k, which will be used to query the historical data stream The k data items with the highest estimated query frequency are called Top-k query frequent items. The k most frequently appearing data items in the query are named Top-k query frequent items.

[0064] The historical query data stream is considered as a set of query data document sequences with time order, where each data item ω * It is composed of data points in spatial data V that answer the query question. Among them, data items can be repeated, indicating multiple queries. To avoid confusion, it is specially emphasized that this embodiment defines the data point key as the data in spatial data V, and the data item ω * Is the historical query data stream The data in the dataset is a data point, but not all data points are data items.

[0065] The estimated query frequency refers to the data item ω * Querying data streams in history An estimate of the frequency of occurrence in .

[0066] Step 1-1: Mining historical query data streams The Top-k query frequent items in , and estimate the query frequency of the Top-k query frequent items to obtain the estimated query frequency of the Top-k query frequent items;

[0067] This implementation uses the Sketch BF algorithm to mine historical query data streams The Sketch BF algorithm (Lu J, Chen H, Zhang Z, et al. Sketch-BF: A fast algorithm for finding top-k flows [J]. Electronics Letters, 2022, 58 (11): 429-431.) is used to estimate the query frequency of the Top-k query frequent items in the top-k query frequent items.

[0068] In this embodiment, the counter in the Sketch BF algorithm is used to store the estimated query frequencies of all data items, and a Bloom filter is used to determine whether each data item is stored in the minimum heap.

[0069] The min-heap has k buckets H(1), H(2), ..., H(k). The min-heap stores the data items of the k top-k frequent query items and their estimated query frequencies in the k buckets in ascending order of estimated query frequencies. That is, H(1) stores the frequent item with the smallest estimated query frequency and its estimated query frequency in the min-heap, and H(k) stores the frequent item with the largest estimated query frequency and its estimated query frequency in the min-heap. In this way, the top-k frequent query items and their estimated query frequencies are stored in order.

[0070] Step 1-2: Preprocess the spatial data V according to the estimated query frequency of the Top-k query frequent items, then evenly divide the spatial data V and finally generate a grid.

[0071] The term "uniform" means that the cumulative sum of the estimated query frequencies of the data points contained in each column in each dimension is roughly the same. The specific steps are as follows:

[0072] Step 1-2-1: Preprocess the spatial data V according to the estimated query frequency of the Top-k query frequent items: assign different weights to the data points belonging to the Top-k query frequent items in the spatial data V. The data points with higher estimated query frequency have higher weights, while the data points with lower estimated query frequency have lower weights.

[0073] Step 1-2-2: After preprocessing the spatial data V, construct the cumulative distribution function F(x α ), where 0≤α≤d-1;

[0074] F(x α )=P(X≤x α ) (1)

[0075] In the above formula, x α Is the data point key = (x0, ..., x α ,...,x d-1 ) on the dimension α attribute, X is a random variable. A random variable is an uncertain value obtained by randomness and is a basic concept in probability.

[0076] The above cumulative distribution function (CDF), also called distribution function, is the integral of the probability density function and can fully describe the probability distribution of a random variable X. α ) is used to estimate P(X≤x α ) probability. That is, F(x α ) is the random variable X less than x α The probability of an event occurring.

[0077] Step 1-2-3: Based on the cumulative distribution function F(x α ) Divide the dimension α into T α List.

[0078] The number of columns T α is a given parameter that aims to divide the dimension α in the spatial data V into T α part.

[0079] The cumulative distribution function F(x α ) is divided into two equal parts: α Part, calculate the x corresponding to the boundary α value and record it as a boundary point.

[0080] For example, since F(x α ) represents the probability of an event, so the range is [0, 1]. When 5 columns are divided on dimension 1, the range is divided into five equal parts, so the boundaries are 0.2, 0.4, 0.6, and 0.8 respectively. According to the cumulative distribution function F(x α ) Find the 4 x corresponding to these 4 values α , which are the four boundary points on dimension 1.

[0081] Step 1-2-4: Record the boundary point set on dimension α as Θ α :

[0082]

[0083] In the above formula, θ is the symbol of the boundary point, The upper right corner (α) is the αth dimension, and the overall meaning is the Tth dimension on the αth dimension. α -1 boundary point, where T α is the number of columns in dimension α.

[0084] Step 1-2-5: Execute steps 1-2-2, 1-2-3, and 1-2-4 for each dimension until the data in all dimensions are evenly divided; after dividing the data in each dimension, complete the data layout of the grid generated for the entire space V.

[0085] For example, suppose spatial data V has two dimensions: longitude and latitude. The longitude dimension is divided into four columns, and the latitude dimension is divided into four columns. Then, the spatial data V is divided into a 4×4 grid, where each small grid is called a cell grid. Each cell grid contains data points that fall within the longitude and latitude range of the cell grid.

[0086] The entire space V can be represented by the union of a series of non-intersecting unit grids as shown in formula f3), where C uis the unit grid as shown in formula (4), and u is the number of the unit grid as shown in formula (5):

[0087]

[0088]

[0089] u=(((j0×T1+j1)×T2+j2)×…)×T d-1 +j d-1 (5)

[0090] In the above formula (3) and formula (5), T d-1 is the number of columns defined on the d-1th dimension.

[0091] In the above formulas (4) and (5), j d-1 is the boundary point number of the d-1th dimension.

[0092] In the above formula (4) is the starting point of the unit grid C u _start, Unit grid end point C u _end; is the jth on the d-1th dimension d-1 +1 border point.

[0093] After the grid is divided, any data point can determine which unit grid it is in through the starting point and end point of the unit grid.

[0094] Step 2: Based on the distance h(H) between each data point and the starting point of the unit grid u ), the distance h between the starting point and the end point of the unit grid (C u ), calculate the mapping value M(Key) of each data point in the spatial data V. The distance can be measured by various methods, such as Hausdorff measure, Lebesgue measure, etc.

[0095] This step constructs a monotonically increasing mapping function to assign mapping values to the originally unordered multidimensional data, preparing for subsequent model training. The specific calculation steps are as follows:

[0096] Step 2-1: Calculate the distance h (H) between the data point and the starting point of the unit grid u );

[0097] In this embodiment, the distance h (H) between a data point and the starting point of the unit grid where it is located is calculated according to formula f6). u );

[0098]

[0099] In the above formula, h(H u ) represents the data point key and the starting point C of the unit grid where it is located u _start, where data point key = (x0, ..., x d-1 ), the cell grid C where the data point key is located u Starting point u is the unit grid C where the data point key is located u Number;

[0100] Step 2-2: Calculate the distance h (C u );

[0101] In this embodiment, the distance h (C) between the starting point of the unit grid and the ending point of the unit grid is calculated according to formula (7). u );

[0102]

[0103] In the above formula, h(C u ) represents the starting point C of the unit grid u _start and cell grid end point C u _end, where the unit grid starts and ends Unit grid end point u is the unit grid C where the data point key is located u Number;

[0104] Step 2-3: Calculate the mapping value M(key) for each data point in the spatial data V

[0105] When the data point key=(x0,...,x d-1 ),and The mapping function formula is:

[0106]

[0107] Step 2-4: Traverse all data points and calculate their mapping values according to the methods of steps 2-1 to 2-3.

[0108] To intuitively understand why this function is monotonically increasing, we will give an example. Hausdorff measure is an important measure in geometric measure theory, which can be simplified as "distance". Then, This means the distance between the data point and the starting point of the grid cell it resides in is divided by the distance between the starting point and the ending point of the grid cell, resulting in a value range of [0, 1), plus the grid cell number u. For example, if the data point's grid cell number is 5, the value range of M(key) is [5, 6]. Therefore, each data point can calculate its own mapping value, and conversely, if a mapping value is available, the data point's grid cell can be directly located.

[0109] Step 3: Divide the data points in each unit grid into fragments with different numbers, point each fragment to a disk page, and build a local model of the fragment, thereby completing the construction of the multidimensional learning index. The index is named TKFI-LI (Top-k Frequent Item Learned Index).

[0110] The fragments are logical addresses, not physical addresses or disk pages. Symbolically represented by S id , which means the shard numbered by id. Data points on the same unit grid will be directed to multiple shards, and the same shard will be directed to one or more disk pages.

[0111] The mapping relationship between the above fragments and disk pages is recorded in the local model of the fragment In. Local model of fragments A local model consisting of a series of local fragments Composition, of which It is the fragment S with the ID id Each fragment has its own local model, which together constitutes the local model of the fragment.

[0112] Step 3-1: Calculate the fragment number based on the mapping value of the data point;

[0113] After obtaining the mapping values of all data points, this embodiment uses SP (Shard Prediction Function) to calculate the shard number, that is, input the mapping values of all data points, and then output the data points contained in each shard through SP.

[0114] Step 3-2: Build a local model of the fragment

[0115] Step 3-2-1: Set the maximum number of data points stored in the disk page to Ω.

[0116] If the number of data points contained in a fragment exceeds Ω, it is necessary to pass the local model of the local fragment Determine how to fragment Sid All data points in the disk are stored in multiple disk pages, ensuring that the maximum number of data points stored in each disk page does not exceed Ω.

[0117] Step 3-2-2: Determine S id The data point set I id and I id The mapping value of each data point in .

[0118] Above I id =(k0, ..., k r-1 ) is the fragment S obtained by the fragment prediction function in step 3-1 id The set of data points in , Γ is the fragment S id The number of data points in the array is arranged in the order of mapping values of the data points from small to large or from large to small. The mapping values are calculated by formula f8).

[0119] Step 3—2-3: Initialize the local model And set and Empty.

[0120] described Is a list used to record storage I id The disk page address of the data point in When the number of disk pages recorded in is greater than 1, the mapping value of the data point stored in each disk page is recorded in the order from small to large or from large to small. middle.

[0121] Step 3-2-4: Determine fragment S id The relationship between the number of data points Γ and the maximum number of data points Ω stored in the disk page, if Γ≤Ω, then I id can be stored in the same disk page, just Add the address of the page and Set to empty; if Γ>Ω, then I id Divide the data into several parts according to the mapping value, and make each part contain less than Ω data points. The data points in each part are stored in the same disk page. Then all the stored data are id The disk page address of the data point in is added to In the same disk page, the mapping values of the data are stored in the order from small to large or from large to small. In the above, I id Divided into several parts, here Several sets of mapping values will be stored in it.

[0122] Step 3-2-5: Local model of the final series of local fragments Local models that are merged together to form fragments

[0123] At this point, the step of constructing a multidimensional learning index in this embodiment is completed. The index is named TKFI-LI (Top-k Frequent Item Learned Index). The overall steps are summarized as follows: Given an initial data set, the data is evenly divided along each dimension V based on the query density, and a boundary point set Θ is generated. i ; Based on all Θ i Generate a cell grid and calculate the mapping values of all data points; use the mapping values as input to build a fragment prediction function, which is used to divide V and generate a series of fragments; finally, build a local model of the fragment, pointing to the physical address.

[0124] Next, we tested the query performance of the TKFI-LI index. The query performance test involves performing range queries directly on three real datasets to determine whether the query algorithm can efficiently perform range queries on different datasets.

[0125] Step 4: Build a range query algorithm for use with TKFI-LI;

[0126] The range query is a common database query used to query tuples whose attribute values are within a specified range. That is, for d-dimensional spatial data, enter the query range starting point q start =(l0, l1, ..., l d-1 ) and the end point q end =(u0, u1, ..., u d-1 ), generate the range query rectangle qr:

[0127] qr=[l0,u0)×…×[l d-1 ,u d-1 ) (9)

[0128] The range query output is the set R of all data points in the range query rectangle qr range The specific steps are as follows:

[0129] Step 4-1: Get the cell grid that overlaps with qr;

[0130] Decompose qr into a union of smaller query rectangles Each small rectangular box qr′ j It intersects only one unit grid, represented by

[0131]

[0132] In the above formula, j is the number of the small rectangular box, Traverse from 0 to Q' for j, Q'+1 is the number of unit grids that the small rectangular box intersects with the entire grid layout. End point of the small rectangle For the small rectangular box qr′ j Overlapping cell grids.

[0133] Step 4-2: Move the small rectangular box qr′ j Divided into and Two categories;

[0134] When the small rectangular box qr′ j and its unit grid When completely overlapped, This type of small rectangular frame qr′ j Defined as The characteristic of this type of small rectangular box is that all data points within the unit grid are the answers to be output.

[0135] When the small rectangular box qr′ j and its unit grid When the overlap is not complete, this type of small rectangular frame qr′ j Defined as The characteristics of this type of small rectangular box are different from the former. It is necessary to filter the data points in the unit grid where the small rectangular box is located, and select the data points that meet the given query range to be added to the query results.

[0136] The query results are expressed as

[0137]

[0138] In the above formula, j c for The number of the small rectangular box, for j c Start from 0 and traverse to that is The number of grid cells that the small rectangle-like box intersects with the entire grid layout.

[0139] In the above formula, j l for The number of the small rectangular box, for j l Start from 0 and traverse to that is The number of grid cells that the small rectangle-like box intersects with the entire grid layout.

[0140] Step 4-3: Calculate the mapping values of the starting points and ending points of all unit grids that overlap with the range query rectangle qr according to formulas (6), (7), and (8), and predict the fragment where the calculated mapping value is located using the fragment prediction function SP described in step 3-1 to obtain the fragment number.

[0141] Step 4-4: Obtain all numbers between the unit grid start point fragment number and the unit grid end point fragment number, and use the local model of the fragment to obtain the disk pages storing the data points in the fragments with these numbers;

[0142] Steps 4-5: Read the disk pages storing the data points within these numbered fragments and add the data points that fall within the range of qr to the result R range middle;

[0143] This step is for and The two types of grids are treated differently;

[0144] for Small rectangular box, which will store Add all data points in the disk page of data points in R range middle;

[0145] for Small rectangular box for storage All data points in the disk page of the data point are judged and the data points belonging to Add the data points in R range middle;

[0146] This embodiment uses three spatial data sets for experiments. The specific information of the data sets is shown in Table 1.

[0147] Table 1

[0148]

[0149] The index TKFI-LI constructed by the present invention divides the spatial data into the same number of parts along each dimension (axis) when generating the unit grid, that is, T0=T1=…=T d-1 =T. When d is 2, 3, 4, 5, and 6, T is set to 240, 90, 32, 18, and 12 respectively. Each experiment generates 10,000 range queries to calculate the average query time. Figure 3As shown, the index TKFI-LI constructed by the present invention clearly achieves better performance in terms of the IO ratio of range queries. Here is an evaluation metric for the IO ratio, which is based on the ZM (Z-order Mode) index and is the ratio of the IO cost of each method to the IO cost of the ZM. The IO cost is the average number of disk pages to be loaded for a range query. In addition, the index TKFI-LI constructed by the present invention is compared with other indexes in terms of the average query time (the sum of CPU time and IO time) for range queries. Figure 4 The query response time of the index TKFI-LI constructed by the present invention is compared with four existing indexes, namely ZM (Z-order Mode, Z-order model index), KD tree (k-dimensional, K-dimensional tree), R tree (R-tree), and R* tree (R*-tree). ZM and KD trees need to access a large number of pages when processing range queries, resulting in a long IO time, which in turn leads to a reduction in overall query performance. Compared with R tree and R* tree, TKFI-LI achieves higher query performance by mining frequent items in historical queries. Experiments show that the index TKFI-LI constructed by the present invention saves nearly half of the query time compared with R tree and R* tree. After analyzing the possible reasons, the index TKFI-LI constructed by the present invention divides the data evenly based on the frequency of data queries, so that the unit grid where the data of the frequently queried part is located contains fewer data points, which is directly related to the improvement in the speed of searching on the disk.

[0150] Test the space overhead of the index TKFI-LI constructed by the present invention, Figure 5 (a) and (b) respectively show the space overhead after indexing data of different dimensions on the Uniform and Zipf datasets. The space overhead increases step by step with the increase of the dimension. The index TKFI-LI constructed by the present invention has better query performance than the R-tree and R* tree, and the required disk storage space is 5% to 10% less than that of the R-tree and R* tree on average. Compared with the KD tree and ZM, although the space overhead of the index TKFI-LI constructed by the present invention is larger than theirs, the execution efficiency of the KD tree and ZM in range queries is greatly reduced. Combined with the query performance test content, they need to read dozens of times more pages than the index TKFI-LI constructed by the present invention. Therefore, it is believed that sacrificing a certain amount of space overhead in exchange for better query performance is effective. After analyzing the possible reasons, compared with the R-tree and R* tree, the present invention uses a machine learning model instead of the traditional tree structure, which is directly related to the reduction of space overhead.

[0151] Ablation test, in order to verify the effectiveness of the index TKFI-LI constructed by the present invention in processing biased query workload representation, the experimental part also conducted an ablation test, and designed the following experiment to compare the benchmark algorithm LISA based on data density grid division. The index TKFI-LI and LISA (Learned Index structure forSpatial dAta, learned index for spatial data) constructed by the present invention were tested in biased query workload and unbiased query workload respectively, and the average query time of the two indexes was observed. The range query performance comparison test of the index TKFI-LI and LISA constructed by the present invention was carried out on the 3d-Uniform, imis-3month and 3d-Zipf datasets, as shown in the figure. Figure 6 Display. For workloads with biased queries, the query time of the index TKFI-LI constructed by the present invention is reduced by 40%-60% compared with LISA. For workloads with unbiased queries, the query time of the index TKFI-LI constructed by the present invention is slightly improved compared with LISA. After analyzing the possible reasons, the present invention evenly divides the grid based on the frequency of data queries. Compared with LISA that evenly divides the grid based on the density of data distribution, the present invention can make the spatial grid where the data of the frequently read part is located contain fewer data points, which is directly related to the improvement in speed when searching on the disk. On the contrary, the spatial grid where the non-frequent part is located contains slightly more data points, but because this part of the data is only read occasionally, it has little effect on the overall average query time. For workloads with unbiased queries, the query time is still improved, but the magnitude is small. Because the query workload cannot be completely unbiased, the present invention still makes appropriate optimizations to the data layout to slightly improve the query performance.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. Therefore, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multidimensional learning index construction method based on query frequent item partitioning spatial data, characterized in that: The method comprises the following steps: Step 1: Divide the d-dimensional spatial data V into grids evenly based on the estimated query frequencies of the Top-k query frequent items. Each small grid is called a unit grid, and each unit grid contains data points that exist within the longitude and latitude range of the unit grid. The Top-k query frequent items refer to a given historical query data stream The k data items with the largest estimated query frequency; The historical query data stream Each data item ω in * It consists of data points in spatial data V that answer the query question; The data points in the d-dimensional space data V are represented by key=(x0,…,x α ,…,x d-1 ), where x d-1 Represents the value of the data point on the d-1th dimension attribute; Step 2: Calculate the mapping value of each data point in the spatial data V based on the distance between each data point and the starting point of the unit grid, and the distance between the starting point and the end point of the unit grid; Step 3: Divide the data points in each unit grid into fragments with different numbers. Then point each fragment to a disk page and build a local model of the fragment to complete the construction of the multidimensional learning index. This index is named the frequent item learning index TKFI-LI. The fragment is a logical address, represented by S id , representing the fragment numbered id; data points on the same unit grid will be pointed to multiple fragments, and the same fragment will be pointed to one or more disk pages; the mapping relationship between the fragments and disk pages is recorded in the local model of the fragment In; the local model of the fragment A local model consisting of a series of local fragments Composition, of which It is the fragment S with the ID id Local model, each local fragment has its own local model; Step 4: Build a range query algorithm for use with TKFI-LI.

2. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 1, characterized in that: The estimated query frequency refers to the data item ω * Querying data streams in history An estimate of the frequency of occurrence in .

3. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1-1: Mining historical query data streams Top-k query frequent items in , and determining the estimated query frequency of the Top-k query frequent items; Step 1-2: The d-dimensional spatial data V is evenly divided into grids based on the estimated query frequencies of the Top-k frequent items.

4. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 3, characterized in that: Using Sketch BF algorithm to mine historical query data stream and determining the estimated query frequencies of the Top-k query frequent items.

5. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 3, characterized in that: The method of evenly dividing the d-dimensional spatial data V into grids based on the estimated query frequencies of Top-k query frequent items in step 1-2 includes: First, the spatial data V is preprocessed according to the estimated query frequency of the Top-k query frequent items: different weights are assigned to the data points belonging to the Top-k query frequent items in the spatial data V. The data points with higher estimated query frequency have higher weights, while the data points with lower estimated query frequency have lower weights. Then, traverse each dimension of the spatial data V and perform data partitioning operations on each dimension, thereby generating a grid data layout for the entire d-dimensional spatial data V; The data partitioning operation includes: Construct the cumulative distribution function along each dimension; Divide each dimension into series based on the cumulative distribution function value of the data points in the dimension.

6. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3-1: Calculate the fragment number based on the mapping value of the data point; Step 3-2: Build a local model of the fragment Step 3-2-1: Set the maximum number of data points stored in the disk page to Ω; Step 3-2-2: Determine the fragment S id The data point set I id and I id The mapping value of each data point in; Step 3-2-3: Initialize fragment S id Local model And set and Empty; Is a list used to record storage I id The disk page address of the data point in When the number of disk pages recorded in is greater than 1, the mapping values of the data points stored in each disk page are recorded in sequence. middle; Step 3-2-4: Determine fragment S id The relationship between the number of data points Γ in the disk and the maximum number of data points Ω stored in the disk page, if Γ≤Ω, then I id stored in the same disk page and Add the address of the page to Set to empty; if Γ>Ω, then I id Divide the data into several parts according to the mapping value, and make each part contain less than Ω data points. The data points in each part are stored in the same disk page, and then all the storage I id The disk page address of the data point in is added to In the , the mapping values of the data on the same disk page are stored in order middle; Step 3-2-5: Finally, local models of all local fragments Local models that are merged together to form fragments 7. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 6, characterized in that: The Shard Prediction Function is used to calculate the shard number, that is, the mapping values of all data points are input to the shard prediction function, and then the data points contained in each shard are output.

8. The method for constructing a multidimensional learning index based on partitioning spatial data based on query frequent terms according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4-1: Get the cell grid that overlaps with the range query rectangle qr and decompose qr into smaller query rectangles qr′ j , each small rectangular box qr′ j Intersects only one unit grid; For d-dimensional spatial data, enter the query range starting point q start =(l0,l1,…,l d-1 ) and the end point q end =(u0,u1,…,u d-1 ), generate a range query rectangle qr=[l0,u0)×…×[l d-1 ,u d-1 ); Each of the small rectangular frames qr′ j Expressed as The starting point of the small rectangular box is represented by The end point is represented by For the small rectangular box qr′ j Overlapping cell grids; Step 4-2: Move the small rectangular box qr′ j Divided into and Two categories: When the small rectangular box qr′ j and its unit grid When completely overlapped, this type of small rectangular frame qr′ j Defined as All data points in the unit grid where the small rectangular box is located are the answers that need to be output; when the small rectangular box qr′ j and its unit grid When the overlap is not complete, this type of small rectangular frame qr′ j Defined as Need to Filter the data points in the unit grid where the small rectangular box is located, and select the data points that meet the given query range to be added to the query results; Step 4-3: Calculate the mapping value of the starting point and the mapping value of the ending point of each unit grid that overlaps with the range query rectangle qr, and use the fragment prediction function to predict the fragment where the calculated mapping value is located. Obtain the fragment number of the starting point of the unit grid and the fragment number of the ending point of the unit grid, thereby determining the fragment number interval corresponding to the unit grid; Step 4-4: For each cell grid that overlaps with the range query rectangle qr, first obtain the numbers of all fragments in the cell grid according to the fragment number interval corresponding to the cell grid, and then use the local model of the fragment Get the disk pages corresponding to these numbered fragments; Step 4-5: Filter the data points that fall within the range of qr from the disk pages obtained in step 4-4 and add them to the result R range Specifically: for If the small rectangular box is included, all data points on the disk page corresponding to the cell grid where it is located are directly added to R range middle; for If a small rectangular box is included, all the data points on the disk page corresponding to the unit grid where it is located will be judged and the data points belonging to Add the data points in R range middle.

Images