Spatial index optimization method and system for dynamic query load

Abstract

The application provides a kind of spatial index optimization method and system for dynamic query load, comprising: step S1: using KL divergence obtains load fluctuation quantitative index, for quantifying the degree of change of dynamic query load;Step S2: based on the load fluctuation quantitative index and the double rules of window query time are constructed trigger condition;Step S3: when meeting trigger condition, then the dimensionality reduction index structure of query load is adjusted, realizes spatial index optimization.The application proposes a quantitative index to ensure the effectiveness of dynamic adjustment mechanism, which is used to evaluate the degree of change of query load. Only when the load change exceeds the set threshold, the adjustment is triggered;By analyzing the characteristics of query load, part of the structure in BMTree dimensionality reduction index structure is optimized, filtering, rewriting, partial construction and merging operation of data and query load, so as to flexibly adapt to the change of query load, and avoid the performance bottleneck of global reconstruction.

Description

Technical Field

[0001] This invention relates to the field of spatial indexing technology, and more specifically, to a spatial indexing optimization method and system for dynamic query loads. Background Technology

[0002] Existing SFC-based dimensionality reduction index structures only consider optimizing the accuracy of dimensionality reduction mapping or improving query performance by combining learning algorithms under a fixed query workload. They do not optimize the dimensionality reduction index structure in real-time, i.e., under dynamically changing query workloads. Changes in query workload can lead to a decrease in the query performance of an existing, excellent dimensionality reduction index structure. There are two solutions: rebuilding or partial reconstruction. On the one hand, rebuilding the dimensionality reduction index structure is a cost-effective approach for some simple and efficient structures, but it is not a good solution for time-consuming structures like BMTree. On the other hand, partially reconstructing the dimensionality reduction index structure allows for a trade-off between which part and how much of the index structure to reconstruct, optimizing the dimensionality reduction index structure with minimal time cost.

[0003] Therefore, choosing a solution is a factor to consider. However, a crucial prerequisite for implementing a solution is determining when to optimize the dimensional index structure under dynamically changing query workloads.

[0004] Therefore, this invention proposes a spatial index optimization method and system for dynamic query workloads. It is a real-time index structure optimization method based on KL divergence (Dya-BMTree), which optimizes the dimensionality reduction index structure through partial reconstruction to cope with dynamically changing query workloads. The system only makes local adjustments to the affected parts when the load changes, avoiding the high time cost of full reconstruction and improving query efficiency. Furthermore, the system uses quantitative indicators to evaluate load changes, thereby triggering a dynamic adjustment mechanism and avoiding performance bottlenecks caused by global reconstruction. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a spatial index optimization method and system for dynamic query loads.

[0006] A spatial index optimization method for dynamic query loads provided by the present invention includes: Step S1: Use KL divergence to obtain a load fluctuation quantification index, which is used to quantify the degree of change in dynamically queried loads; Step S2: Construct triggering conditions based on the dual rules of the load fluctuation quantification index and window query time; Step S3: When the triggering condition is met, the dimensionality reduction index structure of the query load is adjusted to achieve spatial index optimization.

[0007] Preferably, step S1 includes: quantifying the query operations within two adjacent time windows into two probability distributions, and using KL as an indicator to calculate the degree of difference between the two distributions; the larger the KL value, the greater the degree of difference between the two.

[0008] in, This is the previous query load window; This is the current query load window; for The probability distribution of query load access within the data space partitioned by the reduced-dimensional index structure; for The probability distribution of query load access within the data space partitioned by the reduced-dimensional index structure; The number of queries in the window; in, ; The number of data spaces to be customized; To indicate A query workload in An array of distribution sizes in different partitioned spaces; To indicate the first The number of queries in each data space block.

[0009] Preferably, the data space divided by the dimensionality reduction index structure is a division of the range of query workload data; When using a single SFC as the dimensionality reduction index structure, the data space is directly and evenly divided into ranges for query load. When using segmented SFC as a dimensionality reduction index structure, the data space is divided based on the distribution of the query load among the smallest range nodes in the BMTree tree.

[0010] Preferably, step S2 includes: If the query time change does not meet the preset requirements and the KL value does not reach the threshold, no adjustment will be made; When the query time change does not meet the preset requirements but the KL value has reached the threshold, no adjustment will be made without affecting performance; If the query time change meets the preset requirements but the KL value does not reach the threshold, and the performance change is not due to load change, no adjustment will be made. When the time change meets the preset requirements and the KL value has reached the threshold, the adjustment is triggered.

[0011] Preferably, step S3 includes: Adjusting the dimensionality reduction index structure for a single SFC includes: analyzing the impact of different space filling curves on page access counts using QUILTS, generating an optimal SFC as the dimensionality reduction index structure based on QUILTS analysis, and completing the dynamic optimization of the index structure. The adjustment of the dimensionality reduction index structure for segmented SFC includes: identifying the Top K nodes with the largest differences in load quantification metrics between the two distributions as the root nodes for adjustment; filtering and correcting the dataset and the current query load window based on the adjusted root nodes to generate a new dataset and a new query load set that fit the adjustment range; constructing a partial BMTree tree within the adjustment range based on the new dataset and the new query load set using the Monte Carlo tree search algorithm; and merging the constructed partial BMTree tree with the original BMTree tree to complete the dynamic optimization of the index structure.

[0012] Preferably, determining the Top K nodes with the largest differences in load quantization metrics between the two distributions as the root nodes for adjustment includes: Based on the previous query load window With the current query load window Distribution in the BMTree tree and And denote the complete BMTree tree as ;get and The difference array arr; the difference array arr is used to reflect the minimum range of node distribution changes of each node in the BMTree tree between two windows; Select the node with the largest value from the difference array arr, sort them by value, and initialize the selection array S; Traverse the sorted nodes. If the current node has a subtree relationship with a node in the selection array S, select the node with higher value and add it to S; otherwise, add the current node directly to S. Select a node from array S as the adjusted root node of the BMTree tree, denoted as . Let k be the root node of the BMTree tree that needs to be adjusted.

[0013] Preferably, the step of filtering and correcting the dataset and the current query load window based on the adjusted root node to generate a new dataset and a new query load set adapted to the adjustment range includes: When dataset D or the current window If the data in the dataset does not follow the traversal path from the root node of the BMTree to the root node k of the tree structure that needs to be adjusted, delete the data to obtain the initially filtered dataset D' and query load window wc'; delete the binary bits of the corresponding dimension of the data in the above traversal path to complete the data correction and obtain the final dataset D'' and query load window wc''.

[0014] Preferably, the construction of a partial BMTree tree within the adjustment range based on the new dataset and the new query load set using the Monte Carlo tree search algorithm includes: constructing a partial BMTree structure T2 according to the tree hierarchy using the Monte Carlo tree search algorithm based on the dataset D'' and the query load window wc''.

[0015] Preferably, merging the constructed partial BMTree tree with the original BMTree tree includes: using a level-order traversal algorithm to find the subtree T3 to be replaced with the adjusted root node in the original BMTree, replacing the dimension selection of T3 with the dimension selection of the partial BMTree structure T2, thereby merging the partial BMTree with the original BMTree.

[0016] A spatial index optimization system for dynamic query loads provided by the present invention includes: Module M1: Uses KL divergence to obtain a quantitative indicator of load fluctuation, which is used to quantify the degree of change in dynamically queried load. Module M2: Constructs triggering conditions based on the dual rules of the load fluctuation quantification index and window query time; Module M3: When the triggering conditions are met, the dimensionality reduction index structure of the query load is adjusted to achieve spatial index optimization.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. The index optimization of this invention for dynamic changes in query load is a trigger-adjustment mechanism. In practical applications, as the query workload continues to arrive, the difference between the two windows is used to determine whether the requirements for triggering adjustment have been met, thereby optimizing the dimensionality reduction index structure. 2. This invention provides a probability distribution generation method for different dimensionality reduction index structures to calculate KL values. When calculating KL values ​​for the KL-TREE index, the query workload distribution of the BMTree is used as a probability distribution array. Compared with the KL-SFC method, which directly uses the distribution of query workload in the data space for statistical analysis, KL-TREE can comprehensively consider the impact of changes in the BMTree structure on subsequent query workloads, rather than relying solely on the distribution of query workloads. 3. This invention aims to improve the performance of dimensionality reduction index structure under dynamic query load. By introducing a local adjustment mechanism, only the affected part is adjusted, avoiding the high time cost of full reconstruction, thereby significantly improving query efficiency. 4. To ensure the effectiveness of the dynamic adjustment mechanism, this invention proposes a quantitative indicator to evaluate the degree of query load change. Adjustment is triggered only when the load change exceeds a set threshold. By analyzing the query load characteristics, the invention optimizes part of the structure in the BMTree dimensionality reduction index structure, and performs data and query load filtering, rewriting, partial construction and merging operations to flexibly adapt to query load changes and avoid the performance bottleneck of global reconstruction. 5. Experiments of this invention show that Dya-BMTree can dynamically adjust the dimensionality reduction index structure when the load changes. Compared with the benchmark experiment, it optimizes the time and I / O cost by an average of about 50%. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram illustrating the spatial region partitioning based on different dimensionality reduction index structures.

[0019] Figure 2 This diagram illustrates the distribution of queries within a given window using different index structures.

[0020] Figure 3 This is a schematic diagram showing the distribution of query workload in the statistics window.

[0021] Figure 4 This is a schematic diagram showing the distribution of query workloads in the BMTree statistics window.

[0022] Figure 5 This is a diagram illustrating how the KL value changes with the number of query workload sets in the real-time window.

[0023] Figure 6 A schematic diagram of adjusting the BMTree structure based on distribution differences.

[0024] Figure 7 This is the MCTS flowchart.

[0025] Figure 8 This is a diagram illustrating the window changes.

[0026] Figure 9 This is a complete example diagram.

[0027] Figure 10 This is a schematic diagram showing the comparative experimental results of Dya-BMTree in terms of query time.

[0028] Figure 11 This is a schematic diagram showing the comparative experimental results of Dya-BMTree in terms of I / O cost. Detailed Implementation

[0029] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0030] Example 1 A spatial index optimization method for dynamic query loads provided by the present invention includes: Step S1: Use KL divergence to obtain a load fluctuation quantification index, which is used to quantify the degree of change in dynamically queried loads; Step S2: Construct triggering conditions based on the dual rules of the load fluctuation quantification index and window query time; Step S3: When the triggering condition is met, the dimensionality reduction index structure of the query load is adjusted to achieve spatial index optimization.

[0031] In this embodiment, the dimensions of the dataset and query workload are defined in a two-dimensional space during the data indexing process. In this context, its range is represented as (220, 220); the dataset consists of multiple two-dimensional data points, denoted by the symbol [symbol missing]. Among them, two-dimensional data points This represents a combination of values ​​across two data dimensions, where the values ​​satisfy the range requirement. A query workload is represented as a range query across two dimensions, denoted by the symbol [symbol missing]. Among them, in The query range on the axis is The query range on the y-axis is A query workload set consists of multiple query workloads, denoted by the symbol . Each query workload is confined to the same, defined value range. A window consists of multiple query workloads, with the number of workloads being customizable. The set of is represented by the symbol . .

[0032] Specifically, step S1 includes: The essence of dimensionality reduction indexes is to simplify data storage and query operations by mapping multidimensional data, such as multidimensional vectors or data points in a high-dimensional space, to a low-dimensional space. Because different mapping methods, i.e., subspace partitioning, result in data points being stored sequentially in different locations within the storage structure, the maximum and minimum values ​​within the query range will fall within different ranges, leading to varying query efficiency. Changes in the query range can cause previously high-performing mapping methods to become less efficient. Therefore, a quantitative indicator of query load variation is needed to allow the dimensionality reduction index to adjust its structure when the query range changes to a certain extent.

[0033] In this embodiment, KL divergence is used to quantify the degree of change in dynamic query load. Its formalization is as follows: query operations within two adjacent time windows are quantified into two probability distributions, and KL is used as an indicator to calculate the degree of difference between the two distributions. A larger KL value indicates a greater degree of difference between the two distributions, and vice versa. Specifically, through a window mechanism, the probability distribution of query load access within the data space partitioned based on the dimensionality-reduced index structure is calculated for each of the two adjacent windows, and the degree of dynamic change in query load is calculated using the following formula.

[0034]

[0035] in, This is the previous query load window. This is the current query load window; for Similarly, the probability distribution of query load accessed within the data space partitioned by the dimensionality-reduced index structure, for The probability distribution of query load access within the data space partitioned by the dimensionality-reduced index structure. This represents the number of queries in the window.

[0036] Because the optimized dimensionality reduction index structure may differ, the computation... The difference lies in the data space partitioning method. In this embodiment, the partitioning of the data space in the dimensionality reduction index structure is defined as the partitioning of the query workload data range. When using a single SFC as the dimensionality reduction index structure, the data space is directly and uniformly partitioned to represent the range of the query workload, and the final change index is represented as KL-SFC. However, in the segmented SFC design, i.e., the BMTree structure, the partitioning of the data space takes into account the BMTree tree structure, and its partitioning method is based on the distribution of the query workload in the smallest range of nodes in the tree, and the final change index is represented as KL-TREE.

[0037] like Figure 1As shown, when using a single SFC as the dimensionality reduction index structure, the data space is evenly divided into 16 regions, each representing a specific query range. When a query is given... At that time, the query range is marked with a red rectangle. The query will fall into blocks 10, 11, 14 and 15 (red blocks), and the number of queries will be increased in these areas respectively.

[0038] For the segmented SFC design, the data space is partitioned based on the position of the minimum range node in the tree. The minimum range node is defined as a tree node that can contain both the maximum and minimum values ​​in the query workload. For example, given a query... Its maximum value is (79, 117) and its minimum value is (42, 82). Converting it to binary representation, we get... Starting from the root node of the tree, query the data in descending order of digits. If the dimension selected for the current node is... According to The maximum and minimum values ​​are in The highest bit of each dimension is 1 (red), determining the next node as the right child; subsequently, based on the dimension selected for the current node... By comparison maximum value The highest bit of the dimension is 1 (blue) and the minimum value. When the highest bit of the dimension is 0 (green), the current node is determined to be the node with the smallest range. At this time, the query count data for this node is incremented by one (red node).

[0039] Based on the above rules, the workload distribution for each query in a window is statistically analyzed. Figure 2 This displays the distribution of query workloads across two dimensionality reduction index structures, given a window-sized set of query workloads. In a single SFC dimensionality reduction index structure, the window consists of 10 queries. The query workload distribution across the 16 partitioned blocks, for example, falling into block 10. In the BMTree-based dimensionality reduction index structure, there exists a BMTree with a height of 3, and a window consisting of 10 queries. Distribution information obtained through BMTree. For example, the node labeled 2 in the diagram indicates that 3 query workloads are distributed across the second node of the BMTree.

[0040] Based on the above rules, the workload distribution of each query in the window is statistically analyzed. In a single SFC dimensionality reduction index structure, a window contains 10 queries. Query load distribution across 16 partitioned regions. For example, two queries. It falls into block 10. A dimensionality-reduced index structure based on a BMTree exists, containing a BMTree of height 3. Window The 10 queries are distributed through the structure of this BMTree. For example, in node number 2 of the graph, This indicates that there are 3 query workloads distributed across the 2nd node of the BMTree.

[0041] By statistically analyzing the data space partitioning and window query distribution of the dimensionality-reduced index structure, and normalizing the query distribution results, the corresponding probability distribution is obtained. The calculation formula is as follows:

[0042] in, This indicates the number of custom-defined data spaces. [d] indicates A query workload in An array of distribution sizes in different partitioned spaces. Indicates the first The number of queries in each data space block. Note that to satisfy the condition that the division by zero is not when calculating the KL value, initialization will be performed. All values ​​in the array are 0.01.

[0043] In summary, this embodiment achieves this through... Figure 3 Algorithm 4-1 is used to implement the probability distribution of a query workload set of window size in a dimension-reduced index structure. The input is a set of queries containing... A window for querying workloads The output is a probability distribution. and statistical distribution The specific steps are as follows: First, different data space partitioning methods are used for different dimensionality reduction index structures, and the distribution array is initialized. Secondly, statistical analysis is performed on the distribution of each query workload within the partitioned data space; finally, the resulting distribution array is... Normalization yields the probability distribution And return a value.

[0044] Because different dimensionality reduction index structures have different data partitioning methods and distribution statistics, this embodiment uses... Figure 4 Algorithm 4-2 is used to describe in detail the probability distribution of a query workload set of a given window size within a BMTree. The specific steps are as follows: First, determine the data space partitioning and initialize the distribution based on the number of nodes in the BMTree. Secondly, iterate through each query workload in the query workload set. During this process, The process involves binary encoding and finding the root node of the BMTree. Then, it iterates through the BMTree to find the node with the smallest range of query workloads. If found, the node is marked and the loop exits; otherwise, the loop continues to search for child nodes according to the BMTree indexing rules. Finally, the distribution array is used to... Normalize to obtain the probability distribution and return the value.

[0045] In general, this embodiment uses a window mechanism to calculate the KL value of an upcoming query workload. The window size can be preset, and the calculation is performed on adjacent windows using the window as the basic unit. , probability distribution and In this process, for different dimensionality reduction index structures, a custom data space partitioning method is used to statistically analyze the distribution of the two windows. and And calculate the corresponding probability distribution, thereby calculating the KL value.

[0046] Specifically, step S2 includes: Considering that both query range and query time can reflect changes in query efficiency of the dimensionality reduction index structure, this embodiment proposes a dual-rule triggering adjustment mechanism combining KL value and window query time. Based on segmented SFC, this embodiment uses a real dataset to analyze the changing trend of KL value between the baseline window and the real-time window, such as... Figure 5 As shown, in the baseline window, the number of query workloads is set to 1000. As the number of query workloads in the real-time window increases, the KL value gradually increases. When the number of query workloads reaches 1000, it indicates that the two windows have query workload sets of the same size, and the KL value approaches 0.025. Therefore, the KL threshold set in this embodiment should be greater than 0.025, and a threshold with a certain quantitative relationship to similar distributions is preferred. To adjust this parameter, different window sizes will be used for comparative experiments.

[0047] When determining the KL value threshold, it's also necessary to consider changes in window query time. Combining query time and KL value, four possible scenarios emerge: First, if the query time change is insignificant and the KL value hasn't reached the threshold, no adjustment is needed. Second, if the query time change is insignificant but the KL value has reached the threshold, no adjustment is needed if performance isn't affected. Third, if the query time change is significant but the KL value hasn't reached the threshold, and the performance change isn't due to load changes, no adjustment is needed. Fourth, if the time change is significant and the KL value has reached the threshold, adjustment is triggered. Therefore, when the time change is significant and the KL value reaches the set threshold, the dimensionality reduction index structure will be adjusted accordingly to adapt to load changes.

[0048] Specifically, step S3 includes: The adjustment of the dimensionality reduction index structure for a single SFC is determined based on the distribution of query workload. Therefore, this problem can be interpreted as selecting an optimal curve as the SFC to minimize the number of page visits. To address this, QUILTS can be used to analyze the impact of different space fill curves on the number of page visits. Each triggered adjustment will regenerate an optimal SFC as the dimensionality reduction index structure based on QUILTS, i.e., a reconstruction approach is used to cope with dynamic changes in query workload.

[0049] The construction process of the partial BMTree structure proposed in this embodiment mainly includes four steps: First, the top K nodes with the largest differences in load quantization metrics between the two distributions are identified as the root nodes to be adjusted; second, queries that have not visited the nodes to be adjusted in the current window are filtered to generate a new training dataset; then, a new subtree is constructed based on the new dataset; finally, the new subtree is merged with the original tree. Part of the BMTree algorithm for this process is as follows: Figure 6 As shown in Algorithm 4-3.

[0050] More specifically, the scope of the BMTree adjustment is defined, including: To determine the optimal adjustment portion of the BMTree structure after the model triggers an adjustment, it is necessary to refer to historical windows. With real-time window Distribution in BMTree and And record the complete BMTree as .get and difference array arr (lines 1) represent the statistical changes in the distribution of each node in the BMTree within the minimum range between two windows. , represents a BMTree with a height of two, where This indicates that node 0 has added 3 minimum range nodes in the comparison between the two windows. This indicates that node 1 has reduced by 2 minimum range nodes in the comparison between the two windows.

[0051] Next, from the array The top K nodes with the largest differences are selected as the root nodes for adjustment. During this process, the impact of node depth on the adjustment effect needs to be considered: deeper nodes have lower adjustment costs but less significant performance improvements, and vice versa. Therefore, the rules for selecting the top K nodes are as follows: First, from... Select the largest value from the array Count the nodes and sort them by value, initializing the selection array. Starting from the first node, if that node is related to the selection array... If a node in the selection array has a subtree relationship, then the value of the two nodes needs to be weighed, and the node with the higher value is added to the selection array. If the node is not in the selection array. If any node's subtree is selected, then that node is directly added to the selection array. The value of a node is determined by the statistical sum of the small-scale changes in the node distribution across all nodes in the subtree rooted at that node.

[0052] Select array The node in the table is used as the root node for adjustment, denoted as . Then, they began to investigate this. The following adjustment steps are performed on each node. Let the root node of the BMTree structure that needs to be adjusted be k.

[0053] Data filtering and reconstruction algorithms include: Because the adjusted portion is a partial tree structure of BMTree, the dataset D and the current window... The data in the dataset needs to be corrected within the adjusted data range. Therefore, when dataset D or the current window... If a data point in dataset D does not follow a traversal path from the root node of BMTree to the root node k of the tree structure that needs adjustment, delete that data; otherwise, add it to dataset D. ， or current window ， Then, by following the above traversal path, the binary bits in different dimensions obtained from traversing the data are deleted to correct the data, denoted as dataset D. ” or current window ” This allows for partial BMTree reconstruction within a given data space.

[0054] Part of the BMTree build process includes: The partial BMTree structure is constructed using the Monte Carlo Tree Search (MCTS) algorithm. A partial BMTree tree is generated based on the reconstructed dataset and query workload set, denoted as […]. BMTree is constructed according to a tree hierarchy, such as... Figure 7 As shown, the input is a partially constructed BMTree. After passing through the Monte Carlo Tree Search algorithm, it selects the path to the next level of tree nodes, as shown by the red ellipse in the figure, and finally outputs a deeper BMTree structure. MCTS is an algorithm that evaluates the value of decisions by simulating random samples. It constructs a search tree where each node represents a game state and each edge represents a possible action. The algorithm optimizes the search tree by iteratively selecting, expanding, simulating, and updating nodes, ultimately choosing the optimal action strategy.

[0055] Tree merging algorithms include: The replacement portion of the BMTree structure was generated. Afterwards, through Find the BMTree structure that needs to be replaced. That is, using a level-order traversal algorithm to find the element to be replaced. ,Will The selection of the middle dimension is replaced with Dimension selection.

[0056] The present invention also provides a spatial index optimization system for dynamic query loads. The spatial index optimization system for dynamic query loads can be implemented by executing the process steps of the spatial index optimization method for dynamic query loads. That is, those skilled in the art can understand the spatial index optimization method for dynamic query loads as a preferred embodiment of the spatial index optimization system for dynamic query loads.

[0057] Example 2 Example 2 is a preferred example of Example 1. The spatial index optimization method for dynamic query load provided by the present invention includes: a complete index structure adjustment process based on real-time dynamic load is implemented using a sliding window, mainly including four steps: initialization, monitoring, adjustment, and updating. Figure 8 The image shows an example of using a sliding window to implement the adjustment process; the blue rectangle represents the history window. The red rectangle represents the real-time monitoring window. And fix its window size to 100 query workloads. First, in Figure 8 In step a, initialize the window. (0-100) and (100-200), and calculate the KL value and window query time.

[0058] The system determines whether an index structure adjustment has been triggered based on certain conditions. If no adjustment is triggered, the window continues to slide backward, and the real-time window... Become a window to history ,but Change to 100-200 and regenerate a new live window. (200-300), such as Figure 8 As shown in b; in the sliding window (300-400) and (400-500) Slide to as shown Figure 8 When the range shown in c is met, the adjustment process is triggered after the adjustment mechanism is satisfied; after adjustment, the sliding window... and Swipe back to begin a new round of monitoring and adjustment.

[0059] Specifically, the constructed BMTree is set as For real-time query workloads, a window size of 8 is used to calculate the KL value. For example... Figure 9 As shown, the table in the lower left corner displays the node statistics for the workload queried in the minimum range of BMTree nodes in two adjacent windows, and triggers adjustments when the calculated KL value reaches a preset threshold. After adjustment is triggered, three nodes are selected for evaluation based on the difference array of nodes in the minimum range to determine if adjustment is necessary. The three nodes marked with red rectangles in the table represent... Initialize the selection array as First, we take node 1 and measure it, and get... Then, node 2 is taken out and measured to obtain... Next, we take node 3, which is a subtree of node 1. We calculate the values ​​of node 1 and node 3 respectively. The value of node 1 is 2, and the value of node 3 is 2. However, since node 3 has a greater depth, we update the selection array. After selecting the Top K nodes, as shown in the upper left figure, these nodes are marked in blue and adjusted accordingly.

[0060] Using the blue node in the top left diagram as the root node, trigger the query load adjustment mechanism to build a new BMTree structure. As shown in the upper right image. Real-time window filtering. have to ' Thus, a partial BMTree structure is constructed. Finally, within the dashed rectangle... Replace with The reconstructed BMTree structure is shown in the lower right image. The red nodes represent the replacement parts. In summary, by repeatedly executing the steps described in the diagram, query optimization of the BMTree structure can be achieved in real-time.

[0061] In this embodiment, the main time overhead involves calculating the KL-TREE metric, selecting K child nodes, and reconstructing the BMTree structure. Calculating the probability distribution of the query workload set within the KL-TREE metric adds time complexity. The time complexity of calculating the BMTree-based probability distribution using a single window is [formula missing]. Where k is the window size, i.e., the number of query workloads in a window, and N is the number of nodes in the BMTree. The time complexity is the number of nodes traversed in the BMTree for each query workload. In real-time, if historical time intervals are passed... If there are 1 window, then the overall time complexity is O(n). .

[0062] choose The number of child nodes is based on the number of comparisons. The decision is made based on the minimum time spent in each comparison. The maximum time spent is , The number of nodes in subtree 1. Let be the number of nodes in subtree 2. Subtree 1 and subtree 2 are compared as a partial BMTree structure. The total time cost is then... .

[0063] The reconstruction of the BMTree structure introduces time complexity for both SFC value computation and MCTS-based BMTree construction. Data points are computed using the constructed BMTree. The time complexity of the SFC value is ,in for The length. By inputting... BMTree iterates through the path from the root node to the leaf node. T(x) To calculate The time complexity of constructing a BMTree is O(n log n). )),in It is the number of nodes in the BMTree. Constructing a BMTreee requires at most... Operation. In each step of selecting an action, the selected step is sized by child node. The simulation time corresponds to the calculation of the scan range, using... Perform SFC value calculation. Sort the data. Calculate the scan range for each query.

[0064] Example 3 Example 3 is a preferred example of Example 1. According to the spatial index optimization method and system for dynamic query loads provided by the present invention, experiments were conducted on both synthetic datasets and two real datasets. For the synthetic dataset, the experiments generated granularity of 2 in a two-dimensional data space. 20 ×2 20 The data points follow a uniform distribution, denoted as UNI. For the real-world datasets, one branch is a stock dataset, denoted as Stocks, which includes daily historical stock prices for over 6000 stocks from 1970 to 2018; it includes fields capturing daily prices (open, close, adjusted close, low, and high), trading volume, and date. The other is a taxi dataset, denoted as Taxi, with data from records of yellow taxi trips in New York City in 2018 and 2019; it includes records of pick-up / drop-off date / time, pick-up / drop-off location, trip distance, itemized fare, and driver-reported passenger counts. Two dimensions are extracted from each of these two real-world datasets, and their value ranges are normalized and mapped to (0, 2). 20 )between.

[0065] To simulate real-world query scenarios, the query workload set Q generated in this embodiment is uniformly generated within a certain value range, wherein the value range of the query workload set is randomly generated, and the query does not specify a fixed aspect ratio; each query workload set consists of different combinations of value ranges and query ratios.

[0066] This embodiment selects the dimension reduction indexing method as the baseline for the experiment. (1) Z-curve; (2) C-curve; (3) Hilbert curve; (4) QUILTS; (5) BMTree without reconstruction (BMTree-0); (6) BMTree with full reconstruction (BMTree-C). Among them, BMTree-C requires the use of a trigger adjustment mechanism to adjust the BMTree index structure, while the other dimension reduction indexing methods do not change their structure when the load changes dynamically.

[0067] Considering dynamically changing query workloads, to evaluate the performance of the proposed partially reconstructed BMTree (Dya-BMTree) compared to existing SFCs and fully reconstructed BMTree, this embodiment integrates segmented SFCs and baseline SFCs into a traditional index structure. First, the BMTree index structure (and baseline SFCs) are integrated into the PostgreSQL database system, using a built-in B+ tree variant in PostgreSQL, with SFC values ​​as keys. Here, the PostgreSQL B+ tree is a disk-based index. These are chosen to evaluate the performance of segmented SFCs under different scenarios. In cases where the load dynamically changes and the BMTree structure needs adjustment, the data in the database is rebuilt.

[0068] For experiments using PostgreSQL, optimization metrics were evaluated using I / O cost (I / O) and query time (QT) recorded by the PostgreSQL system. For dimensionality reduction indexing methods that require adjustments triggered by a mechanism, the refactoring time (RT) was increased as an evaluation metric.

[0069] Table 1 lists the parameters used in the experiments of this embodiment; default settings are shown in bold. The trigger adjustment mechanism uses... and The magnitude of the two parameters determines whether an adjustment is triggered. This represents the number of BMTree nodes in the local adjustment. It determines the size of the sliding window in the adjustment mechanism.

[0070] Table 1

[0071] This example experiment uses PyTorch 1.9 and Python 3.8 to train BMTree. The experiment was conducted on an 80-core server, using an Intel(R) Xeon(R) Gold 6248 CPU@2.50GHz and 64.0GB RAM, without utilizing GPU resources for model training.

[0072] The experiment compared the effectiveness of the restructured BMTree (Dya-BMTree) in query processing with other baselines in a back-end SQL environment. To simulate the variation of query workload in real-world applications, the experiment used four query workload sets, each containing 1000 window queries. Each query workload set was randomly generated following its own distribution and value range, and then merged into a single dataset. To verify the effectiveness of Dya-BMTree, experiments were conducted on three different datasets, with query time and I / O cost as the evaluation metrics. The results are as follows. Figure 10 , 11 As shown.

[0073] This experiment demonstrates that, in terms of I / O and QT, the proposed Dya-BMTree consistently outperforms the baselines across all dataset and query distribution combinations. BMTree-C was not considered in this experiment because while it outperforms Dya-BMTree in I / O and QT, its reconstruction time is significantly longer. Among these baselines, BMTree-0 performs worse than the standard SFC approach on the UNI dataset. This is because its dimensionality-reduced index structure better suits the previous query workload, leading to performance degradation when the query workload changes. However, BMTree-0 shows a slight advantage over the unchanged SFC approach on the Stocks and Taxi datasets. This is because the data in these two real-world datasets is not uniformly distributed and closely reflects the query range of subsequent dynamic query workloads. Compared to other methods across three datasets, Dya-BMTree achieves at least a 20% improvement in query time (compared to BMTree-0 on the Taxi dataset) and at most a 100% improvement (compared to BMTree-0 on the UNI dataset); it also achieves at least a 6% improvement in I / O cost (compared to BMTree-0 on the Taxi dataset) and at most an 80% improvement (compared to BMTree-0 on the Taxi dataset).

[0074] Among the many benchmarks in this experiment, the triggered adjustment mechanism can monitor changes in dynamic query workload and thus reconstruct the BMTree. Table 2 below shows the time consumption of partially and fully reconstructed BMTrees on different datasets. The table shows that the partially reconstructed BMTree reduced the time consumption by more than 2 times.

[0075] Table 2

[0076] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0077] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A spatial index optimization method for dynamic query load, characterized in that, include: Step S1: Use KL divergence to obtain a load fluctuation quantification index, which is used to quantify the degree of change in dynamically queried loads; Step S2: Construct triggering conditions based on the dual rules of the load fluctuation quantification index and window query time; Step S3: When the triggering condition is met, the dimensionality reduction index structure of the query load is adjusted to achieve spatial index optimization; Step S1 includes: quantifying the query operations within two adjacent time windows into two probability distributions, and using KL as an indicator to calculate the degree of difference between the two distributions; the larger the KL value, the greater the degree of difference between the two. in, This is the previous query load window; This is the current query load window; for The probability distribution of query load access within the data space partitioned by the reduced-dimensional index structure; for The probability distribution of query load access within the data space partitioned by the reduced-dimensional index structure; The number of queries in the window; in, ; The number of data spaces to be customized; To indicate A query workload in An array of distribution sizes in different partitioned spaces; To indicate the first The number of queries in each data space block; The data space divided by the dimensionality reduction index structure is a division of the range of query workload data; When using a single SFC as the dimensionality reduction index structure, the data space is directly and evenly divided into ranges for query load. When using segmented SFC as a dimensionality reduction index structure, the data space is divided based on the distribution of the query load in the smallest range of nodes in the BMTree tree. Step S2 includes: If the query time change does not meet the preset requirements and the KL value does not reach the threshold, no adjustment will be made; When the query time change does not meet the preset requirements but the KL value has reached the threshold, no adjustment will be made without affecting performance; If the query time change meets the preset requirements but the KL value does not reach the threshold, and the performance change is not due to load change, no adjustment will be made. When the time change meets the preset requirements and the KL value has reached the threshold, the adjustment is triggered.

2. The spatial index optimization method for dynamic query load according to claim 1, characterized in that, Step S3 includes: Adjusting the dimensionality reduction index structure for a single SFC includes: analyzing the impact of different space filling curves on page access counts using QUILTS, generating an optimal SFC as the dimensionality reduction index structure based on QUILTS analysis, and completing the dynamic optimization of the index structure. The adjustment of the dimensionality reduction index structure for segmented SFC includes: identifying the Top K nodes with the largest differences in load quantification metrics between the two distributions as the root nodes for adjustment; filtering and correcting the dataset and the current query load window based on the adjusted root nodes to generate a new dataset and a new query load set that fit the adjustment range; constructing a partial BMTree tree within the adjustment range based on the new dataset and the new query load set using the Monte Carlo tree search algorithm; and merging the constructed partial BMTree tree with the original BMTree tree to complete the dynamic optimization of the index structure.

3. The spatial index optimization method for dynamic query load according to claim 2, characterized in that, The step of determining the Top K nodes with the largest differences in load quantization metrics between the two distributions as the root nodes for adjustment includes: Based on the previous query load window With the current query load window Distribution in the BMTree tree and And denote the complete BMTree tree as ;get and The difference array arr; the difference array arr is used to reflect the minimum range of node distribution changes of each node in the BMTree tree between two windows; Select the values ​​with the largest values ​​from the difference array arr. Count the nodes and sort them by value, then initialize the selection array S; Traverse the sorted nodes. If the current node has a subtree relationship with a node in the selection array S, select the node with higher value and add it to S; otherwise, add the current node directly to S. Select a node from array S as the adjusted root node of the BMTree tree, denoted as . Let k be the root node of the BMTree tree that needs to be adjusted.

4. The spatial index optimization method for dynamic query load according to claim 2, characterized in that, The process of filtering and correcting the dataset and the current query load window based on the adjusted root node to generate a new dataset and a new query load set that fits the adjustment range includes: When dataset D or the current window If the data in the dataset does not follow the traversal path from the root node of the BMTree to the root node k of the tree structure that needs to be adjusted, delete the data to obtain the initially filtered dataset D' and query load window wc'; delete the binary bits of the corresponding dimension of the data in the above traversal path to complete the data correction and obtain the final dataset D'' and query load window wc''.

5. The spatial index optimization method for dynamic query load according to claim 2, characterized in that, The construction of a partial BMTree tree within the adjustment range based on the new dataset and the new query load set using the Monte Carlo tree search algorithm includes: constructing a partial BMTree structure T2 according to the tree hierarchy using the Monte Carlo tree search algorithm based on the dataset D'' and the query load window wc''.

6. The spatial index optimization method for dynamic query load according to claim 2, characterized in that, The process of merging the constructed partial BMTree with the original BMTree includes: using a level-order traversal algorithm to find the subtree T3 to be replaced in the original BMTree with the adjusted root node as the root, replacing the dimension selection of T3 with the dimension selection of the partial BMTree structure T2, thereby merging the partial BMTree with the original BMTree.

7. A spatial index optimization system for dynamic query loads, characterized in that, include: Module M1: Uses KL divergence to obtain a quantitative indicator of load fluctuation, which is used to quantify the degree of change in dynamically queried load. Module M2: Constructs triggering conditions based on the dual rules of the load fluctuation quantification index and window query time; Module M3: When the triggering conditions are met, the dimensionality reduction index structure of the query load is adjusted to achieve spatial index optimization; The module M1 includes: quantifying query operations within two adjacent time windows into two probability distributions, and using KL as an indicator to calculate the degree of difference between the two distributions; the larger the KL value, the greater the degree of difference between the two. in, This is the previous query load window; This is the current query load window; for The probability distribution of query load access within the data space partitioned by the reduced-dimensional index structure; for The probability distribution of query load access within the data space partitioned by the reduced-dimensional index structure; The number of queries in the window; in, ; The number of data spaces to be customized; To indicate A query workload in An array of distribution sizes in different partitioned spaces; To indicate the first The number of queries in each data space block; The data space divided by the dimensionality reduction index structure is a division of the range of query workload data; When using a single SFC as the dimensionality reduction index structure, the data space is directly and evenly divided into ranges for query load. When using segmented SFC as a dimensionality reduction index structure, the data space is divided based on the distribution of the query load in the smallest range of nodes in the BMTree tree. The module M2 includes: If the query time change does not meet the preset requirements and the KL value does not reach the threshold, no adjustment will be made; When the query time change does not meet the preset requirements but the KL value has reached the threshold, no adjustment will be made without affecting performance; If the query time change meets the preset requirements but the KL value does not reach the threshold, and the performance change is not due to load change, no adjustment will be made. When the time change meets the preset requirements and the KL value has reached the threshold, the adjustment is triggered.

Images