A method for concurrent memory database split
By optimizing the database splitting algorithm using IART indexes and the IARTC algorithm, the problems of low caching efficiency and query efficiency in in-memory databases are solved, achieving efficient concurrent splitting and index updates, and improving query performance and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2023-06-26
- Publication Date
- 2026-05-29
Smart Images
Figure CN116775662B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, and in particular to a method for concurrent in-memory database splitting. Background Technology
[0002] With the rapid development of emerging information technologies such as the Internet of Things, cloud computing, and big data, the amount of data is growing exponentially, exceeding 40 ZB by 2020. Quickly and accurately retrieving query data from such massive amounts of data has always been a focus of research in the database field. Creating high-performance database indexes is one important technique. The emergence of adaptive indexing technology, especially database partitioning technology, overcomes the shortcomings of traditional indexing techniques. Its index construction does not require advance planning, decision-making, or construction for all data; instead, it relies on dynamically changing workloads, maintaining the changes in the physical design of the index on demand in an adaptive, continuous, incremental, and partial manner.
[0003] Database splitting is one of the earliest adaptive indexing techniques. Its main concept is to adaptively and incrementally create and maintain indexes during query execution, treating each incoming request as a suggestion to optimize the index structure. The splitting process is very similar to quicksort, where each query is executed on a partition, and tuples are reorganized based on the query predicates. In this way, the index creation process is distributed across multiple subsequent queries, each benefiting from the preceding queries.
[0004] The standard database splitting algorithm was first proposed by S. Idreos et al. in their paper "Database Cracking" presented at CIDR 2007. The data structure used to implement the splitting index in this algorithm is the AVL-Tree, which suffers from the following performance bottlenecks: Although the AVL-Tree resides in memory and drives database splitting during queries, it is not memory-optimized and has low caching efficiency for range queries; each query in the database splitting algorithm requires refining and updating the index, and the numerous balancing operations of the AVL-Tree incur significant overhead; as the dataset increases or the query workload changes, the query efficiency of this method is not high, which can significantly impact the overall performance of database splitting.
[0005] In their paper "Concurrency Control for Adaptive Indexing" presented at PVLDB 2012, Goetz Graefe et al. provided a detailed introduction to the Parallel Standard Cracking (PSC) concurrency algorithm for standard database splitting. PSC primarily achieves concurrency control by latching and unlocking the splitting columns and AVL indexes used in the standard database splitting algorithm. While PSC achieves concurrency by latching and unlocking or concurrently partitioning the data structures related to database splitting, this approach introduces significant lock contention and latching / unlocking overhead, becoming a major bottleneck in database splitting concurrency algorithms.
[0006] Furthermore, database splitting algorithms need to keep pace with the development of in-memory indexing technology. With the emergence of increasingly superior in-memory database indexes, such as ART indexes, the query efficiency of database splitting algorithms still needs to be improved compared to these indexes. Summary of the Invention
[0007] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a method for concurrent in-memory database splitting, a method for driving database splitting with an in-memory database index that has better caching and query efficiency, and a method for optimizing the database splitting algorithm by combining the in-memory database index with the database splitting technology. At the same time, based on the optimized algorithm, a better concurrent strategy for database splitting is proposed.
[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0009] A concurrent in-memory database splitting method is proposed, which improves upon the ART index to obtain a new incremental adaptive radix tree index structure that supports database splitting technology, namely the IART index. Based on the IART index, a splitting algorithm based on the incremental adaptive radix tree, namely the IARTC algorithm, is proposed. The IARTC algorithm uses the IART index as the splitting index to drive database splitting. At the same time, based on the IARTC algorithm, a new concurrent database splitting algorithm, namely the PIARTC algorithm, is derived. The PIARTC algorithm uses an optimistic locking coupling concurrency strategy to control the locking and unlocking of nodes on the IART index to achieve concurrent splitting and index updates, maintaining a small locking granularity during concurrent execution.
[0010] The IART index adds parent, minKey, maxKey and multi-level support based on the ART index according to different internal node types, so that the upper and lower bounds of the split pivot partition can be obtained by querying the IART index, and it can be used as the split index in the algorithm to drive the database split.
[0011] The IARTC algorithm includes four steps: index pivot partitioning, reorganizing split columns, updating the index, and querying the results. Among them, index pivot partitioning and updating the index are improvements on the original standard database splitting algorithm based on the ART index.
[0012] Furthermore, the data structure of the Node4 type node within the IART index is as follows:
[0013] First, add a parent pointer to the Node4 type internal node of the IART index, which represents a pointer to its parent node; if the key being searched does not exist, it is necessary to continue searching for the upper and lower bounds of the key in its parent or ancestor nodes, which is also necessary in other types of internal nodes of IART; then add multi-level support to the Node4 type node.
[0014] Furthermore, the data structure of the Node16 type node within the IART index is as follows:
[0015] Add a pointer to the parent node and multi-level support for Node16 type nodes.
[0016] Furthermore, the data structure of the Node48 type node within the IART index is as follows:
[0017] Add a pointer to the parent node for the Node48 type node, and add two additional keys, maxkey and minkey; where maxkey represents the maximum value of the keys stored in the current Node48, and minkey represents the minimum value of the keys stored in the current Node48.
[0018] Furthermore, the data structure of the Node256 type node within the IART index is as follows:
[0019] Add a pointer to the parent node to Node256, the maximum value of the key stored internally in Node256 (maxkey), and the minimum value of the key stored internally in Node256 (minkey).
[0020] Furthermore, the specific implementation of the index pivot partition in the IARTC algorithm is as follows:
[0021] There are two scenarios: successful query and unsuccessful query.
[0022] If the value of the range query already exists in the IART index, the position of the current value on the split column is returned directly, and no other split operation is required;
[0023] If the value of the range query is not in the IART index, then the upper and lower bounds of the pivot partition to be searched for the current value need to be given, which consists of the following three aspects:
[0024] (a) When searching the boundary values of a range query on the IART index and failing to find a match at a leaf node, if the current query value is less than the current leaf node value, the upper bound of the pivot partition of the current query value is the value of that leaf node, and its lower bound is the maximum value among all leaf nodes to the left of that node. If the current query value is greater than the value of the current leaf node, the lower bound of the pivot partition of the current query value is the value of that leaf node, and its upper bound is the minimum value among all leaf nodes to the right of that node.
[0025] (b) When using range queries to search the boundary values on the IART index, if the traversal encounters a mismatch with the prefix of the internal node, different handling is required depending on the size of the prefix and the current value.
[0026] If the value of the current query is less than the value of the prefix of the current internal node, then the upper bound of the pivot partition of the current query value is the minimum value of the current internal node, and its lower bound is the maximum value among all leaf nodes to the left of this node.
[0027] If the value of the current query is greater than the value of the prefix of the current internal node, then the lower bound of the pivot partition of the current query value is the maximum value of the current internal node, and its upper bound is the minimum value among all leaf nodes to the right of that node.
[0028] (c) When searching for a value on the IART index using a range query, if the retrieved value is null, different handling is required depending on the type of the parent node, as follows:
[0029] If the parent node is a Node4 type node, first compare the current queried value with the value stored in the parent node. If the current queried value is less than the minimum value stored in the parent node, then the lower bound of the pivot partition of the current queried value is the minimum value of the current leaf node, and the upper bound is the maximum value among all leaf nodes to the left of this node. If the current queried value is greater than the maximum value stored in the parent node, then the lower bound of the pivot partition of the current queried value is the maximum value of the current leaf node, and the upper bound is the minimum value among all leaf nodes to the right of this node. If the value is exactly between the maximum and minimum values of the parent node, then the lower bound of the pivot partition of the current queried value is the maximum value among all leaf nodes to the left of this node, and the upper bound is the minimum value among all leaf nodes to the right of this node.
[0030] If the parent node is a Node16 type node, the search method is the same as that for internal nodes of the Node4 type.
[0031] If the parent node is an internal node of type Node48, due to the special structure of Node48 nodes, it is first necessary to obtain the maximum and minimum values stored in the Node48 type node. Then, the current query value is compared with the found maximum and minimum values. If the current query value is greater than the maximum value of the Node48 type node, the lower bound of the pivot partition of the current query value is the maximum value among all leaf nodes in the current layer, and the upper bound is the minimum value among all leaf nodes to the right of this node. If the current query value is less than the maximum value of the Node48 type node, the lower bound of the pivot partition of the current query value is the minimum value among all nodes in the current layer, and the upper bound is the maximum value among all leaf nodes to the left of this node. Unlike Node4 type nodes, when the current query value is between the maximum and minimum values, it is still necessary to traverse to find the positions of the upper and lower bounds of the pivot partition.
[0032] If the parent node is an internal node of type Node256, the pivot partition lookup is the same as for a node of type Node48.
[0033] Furthermore, the execution process of the IARTC algorithm is as follows: For a range query [low, high], firstly, it is checked whether this is the first query. That is, when the IART index is empty, the two most important data structures in the IARTC algorithm need to be initialized, completing the creation of the IART index and the copying of the split column. The corresponding pivot partitions are searched on the IART index according to the lower bound (low) and upper bound (high) of the range query, with each pivot partition identified by a numerical pair. Different splitting algorithms are executed on the split column based on the query results. When the upper and lower bounds of the index query belong to the same pivot partition, a three-split algorithm is executed. When the upper and lower bounds of the index query belong to two different pivot partitions, two two-split algorithms are executed respectively. Then, the elements in the split column are reorganized and the position of the pivot partition is returned. Afterwards, the IART index is updated, inserting the upper and lower bounds of the query and the position of the corresponding pivot partition into the IART index. Finally, a scan is performed on the split column to return the result of the range query.
[0034] Furthermore, the PIATC algorithm first performs concurrent queries on the IART index to obtain the pivot partition to be split. Similar to the single-threaded IARTC algorithm, it is divided into three parts: searching for the case where the matching prefix fails, searching for the case where NULL is found, and searching for the case where a leaf node is found. Then, the same operation is performed in each part, that is, after concurrently searching for the corresponding pivot partition on the IART index, the pivot partition is concurrently split. After that, the IART index is adaptively and concurrently constructed to accelerate the execution of concurrent queries.
[0035] Furthermore, the basic process of concurrent range query execution using the PIATC algorithm is as follows:
[0036] First, when a range query performs a concurrent search on a pivot, an optimistic locking-coupled concurrency strategy (OLC) is used to acquire a read lock on the root node. During the concurrent search of the pivot partition using the IART index, different processing is required based on the size of the prefix and the pivot. Furthermore, an optimistic locking-coupled concurrency strategy is used to traverse the nodes to find the minimum, maximum, maximum, and minimum values of all leaf nodes to the left and right of the current node, ensuring the correct unlocked pivot partition is obtained. Finally, the pivot partition is split, and the IART index is updated.
[0037] When a prefix mismatch occurs, it affects not only the current node but also the pointer of the parent node. Therefore, it is necessary to simultaneously update the read locks on both the current node and the parent node to write locks, and then unlock the corresponding node after the IART index has been updated.
[0038] When the pivot and the prefix of the current IART index internal node are successfully matched, it is necessary to obtain the lower-level node of the IART index, while avoiding reading an invalid pointer that could cause the program to crash.
[0039] The corresponding pivot partition is found using an optimistic locking coupled concurrency strategy based on the type of the parent node. If it is the first query, only one thread is currently performing the split operation on the entire split column. Since there are no leaf nodes in the current IART index, it is only necessary to update the lock of the root node to a write lock to prevent other threads from locking the partition and performing the split.
[0040] Afterwards, the IART index is updated and unlocked. When it is not the first query, the corresponding node needs to be locked to complete the IART index update, depending on whether the node of the IART index needs to be expanded. When the parent node is safe, that is, the parent node will not be modified, the lock of the parent node is released and the next level of nodes is traversed.
[0041] If the currently queried pivot matches the currently queried leaf node, no operation is required and an empty value can be returned. Otherwise, it is still necessary to find the correct pivot partition based on the size relationship between the currently queried pivot and the current leaf node.
[0042] The split column is executed, and an optimistic locking coupling concurrency strategy is used to ensure the vertical split of the current IART index node in order to complete the index update.
[0043] The beneficial effects of adopting the above technical solution are as follows: The concurrent in-memory database splitting method provided by this invention first proposes a new IART index structure for optimizing internal ART nodes, making it usable as a splitting index for database splitting algorithms. Based on this, a new database splitting algorithm, namely the IATRC algorithm, is proposed. This algorithm primarily uses the IART index as the splitting index to drive database splitting and selects different splitting algorithms based on the different index pivot partitions during the splitting process. In a multi-core parallel environment, a new concurrent database splitting algorithm, namely the PIATC algorithm, is proposed based on the IARTC algorithm. This algorithm uses an optimistic locking coupling concurrency strategy to control the locking and unlocking of nodes on the IART index, achieving concurrent splitting and index updates. This maintains a small locking granularity during concurrent execution, significantly reducing the probability of conflicts between threads.
[0044] Compared to existing algorithms, the two adaptive indexes proposed in this invention demonstrate excellent query performance in both single-core and multi-core systems when facing dynamic environments and large-scale dataset queries, while maintaining the lightweight characteristics of the original database splitting algorithm. Compared to the standard database splitting algorithm, the IARTC algorithm improves query response performance by 30.8% and reduces cache misses by 37.1% in a single thread under random workload mode. In a multi-threaded environment, the IARTC algorithm improves query response performance by 44%, achieving a speedup of 2.86 with 8 threads. The robustness of both algorithms is also improved. Attached Figure Description
[0045] Figure 1 This is a diagram of the IART index structure provided in an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of the execution flow of the IARTC algorithm provided in an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram illustrating the execution flow of the three range PARTC algorithms provided in this embodiment of the invention. Detailed Implementation
[0048] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0049] The method in this embodiment improves upon the ART index by proposing an IART index that supports database splitting technology. Based on the IART index, the IARTC algorithm is proposed, which uses the IART index to drive database splitting.
[0050] The native ART index is not suitable as a split index. This embodiment optimizes the different internal node structures of ART differently, proposing a new incremental adaptive radix tree index structure, namely the IART index. Figure 1 As shown, compared to the ART index, the IART index adds parent, minKey, maxKey, and multi-level support based on different internal node types. This allows the upper and lower bounds of the split pivot partition to be obtained by querying the IART index, and it can be used as the split index in the algorithm to drive database splitting, thus supporting efficient range queries. The data structure of different internal nodes of IART is described in detail below.
[0051] For Node4 type nodes: First, a parent pointer is added to the internal Node4 type of the IART index, representing a pointer to its parent node. If the currently searched key does not exist, it is necessary to continue searching for the upper and lower bounds of the key in its parent or ancestor nodes. This is also necessary for other types of internal nodes in IART. At the same time, it is also obvious that when there are two keys in the path, ART's path compression technology will not work. Therefore, in addition to adding a parent pointer to the Node4 type node, multi-level support is also added.
[0052] Node16 type nodes: such as Figure 1 As shown, a pointer to the parent node and multi-level support are also added to the Node16 type node.
[0053] For Node48 type nodes: First, a pointer to the parent node is added to the Node48 type node. Two additional keys are also added: `maxkey` represents the maximum value of the keys stored within the current Node48, and `minkey` represents the minimum value of the keys stored within the current Node48. Because the internal storage structure of Node48 nodes differs from that of Node4 and Node16 nodes, where the keys are stored contiguously and ordered in an array, it's easy to obtain the maximum and minimum values. The first key in the array represents the minimum value within Node4 and Node16, and its corresponding child pointer points to the minimum node; the last key represents the maximum value within Node4 and Node16, and its corresponding child pointer points to the maximum node. While the keys within Node48 are also ordered, they are stored discretely. Only when the number of nodes stored within Node48 reaches the maximum of 48 can the maximum and minimum values of the current node be obtained as quickly as with Node4 and Node16 types. This is crucial for index pivot partitioning. Therefore, two additional keys are added to Node48 type nodes.
[0054] Node256 type node: The storage method of Node256 node is similar to that of Node48. Therefore, a pointer to the parent node, maxkey, the maximum value of the key stored in Node256, and minkey, are added to Node256 node.
[0055] This embodiment proposes a new database splitting algorithm based on the IART index mentioned above, namely the Incremental Adaptive Radix Tree-based splitting algorithm (IARTC algorithm). The IARTC algorithm includes four steps: index pivot partitioning, reorganizing the split columns, updating the index, and querying the results. Among them, index pivot partitioning and updating the index are the main improvements made to the original standard database splitting algorithm based on the ART index.
[0056] The execution process of the IARTC algorithm is as follows: For a range query [low, high], firstly, it checks if this is the first query. If the IART index is empty, the two most important data structures in the IARTC algorithm need to be initialized: creating the IART index and copying the split columns. Then, based on the lower bound (low) and upper bound (high) of the range query, the corresponding pivot partitions are searched on the IART index. Each pivot partition is identified by a pair of values. Simultaneously, compared to standard database splitting algorithms, the IARTC algorithm improves the logic of executing splitting algorithms. After weighing the performance of the three-split and two-split algorithms, when the upper and lower bounds after the index query belong to the same pivot partition, the three-split algorithm is executed; when the upper and lower bounds belong to two different pivot partitions, two two-split algorithms are executed respectively. Then, after the split columns in the pivot partitions are reorganized, the IART index is updated, inserting the upper and lower bounds of the query and the corresponding pivot partition positions into the IART index. Finally, a scan is performed on the split columns to return the results of the range query. The basic execution flow of the IARTC algorithm is as follows: Figure 2 As shown.
[0057] The specific implementation of the indexed pivot partitioning in the IARTC algorithm is as follows:
[0058] There are two scenarios: successful query and failed query. We will focus on the failed query scenario.
[0059] (1) If the value of the range query already exists on the IART index, the query algorithm is basically similar to the original ART query algorithm. It can directly return the position of the pivot partition of the current value on the split column, and no other split operation is required.
[0060] (2) If the value of the range query is not in the IART index, the pivot to be searched for the current value needs to be given.
[0061] The upper and lower bounds of a partition are mainly divided into the following three aspects:
[0062] (a) When searching the boundary values of a range query on the IART index and failing to find a match at a leaf node, if the current query value is less than the current leaf node value, the upper bound of the pivot partition of the current query value is the value of that leaf node, and its lower bound is the maximum value among all leaf nodes to the left of that node. If the current query value is greater than the value of the current leaf node, the lower bound of the pivot partition of the current query value is the value of that leaf node, and its upper bound is the minimum value among all leaf nodes to the right of that node.
[0063] (b) When using range queries to search the boundary values on the IART index, if the traversal encounters a mismatch with the prefix of the internal node, different handling is required depending on the size of the prefix and the current value.
[0064] If the value of the current query is less than the value of the prefix of the current internal node, then the upper bound of the pivot partition of the current query value is the minimum value of the current internal node, and its lower bound is the maximum value among all leaf nodes to the left of this node.
[0065] If the value of the current query is greater than the value of the prefix of the current internal node, then the lower bound of the pivot partition of the current query value is the maximum value of the current internal node, and its upper bound is the minimum value among all leaf nodes to the right of that node.
[0066] (c) When searching for a value on the IART index using a range query, if the retrieved value is null, different handling is required depending on the type of the parent node, as follows:
[0067] If the parent node is a Node4 type node, the current queried value is first compared with the value stored in the parent node. If the current queried value is less than the minimum value stored in the parent node, the lower bound of the pivot partition of the current queried value is the minimum value of the current layer's leaf nodes, and the upper bound is the maximum value among all leaf nodes to the left of this node. If the current queried value is greater than the maximum value stored in the parent node, the lower bound of the pivot partition of the current queried value is the maximum value of the current layer's leaf nodes, and the upper bound is the minimum value among all leaf nodes to the right of this node. If the value is exactly between the maximum and minimum values of the parent node, the lower bound of the pivot partition of the current queried value is the maximum value among all leaf nodes to the left of this node, and the upper bound is the minimum value among all leaf nodes to the right of this node.
[0068] If the parent node is a Node16 type node, the search method is the same as that for Node4 type internal nodes because the internal nodes of Node16 type and Node4 type have similar storage structures.
[0069] If the parent node is an internal node of type Node48, due to the special structure of Node48 nodes, it is first necessary to obtain the maximum and minimum values stored in the Node48 type node. Then, the current query value is compared with the found maximum and minimum values. If the current query value is greater than the maximum value of the Node48 type node, the lower bound of the pivot partition of the current query value is the maximum value among all leaf nodes in the current layer, and the upper bound is the minimum value among all leaf nodes to the right of this node. If the current query value is less than the maximum value of the Node48 type node, the lower bound of the pivot partition of the current query value is the minimum value among all nodes in the current layer, and the upper bound is the maximum value among all leaf nodes to the left of this node. Unlike Node4 type nodes, when the current query value is between the maximum and minimum values, it is still necessary to traverse to find the positions of the upper and lower bounds of the pivot partition.
[0070] If the parent node is an internal node of type Node256, the pivot partition lookup is the same as for a node of type Node48.
[0071] To improve the query performance of algorithms under multi-core parallelism, a concurrent algorithm for database splitting based on adaptive ART indexes, namely the PIATC algorithm, is proposed.
[0072] The PIARTC algorithm describes the process of concurrently executing range queries using the select operator, with its core ensuring that locks on the correct data structure are acquired. The PIARTC algorithm first performs concurrent queries of the IART index to obtain the pivot partition to be split. Similar to the single-threaded IARTC algorithm, it can be divided into three main parts: searching the IART index for cases where a matching prefix fails, searching for NULL values, and searching for leaf nodes. Then, the same operation is performed in each part: concurrently searching the corresponding pivot partition on the IART index, performing concurrent splitting of the pivot partition, and then adaptively and concurrently building the IART index to accelerate the execution of concurrent queries. The PIARTC algorithm combines the idea of optimistic locking coupling concurrency strategy and leverages the advantages of the IART index structure itself, making significant improvements to the original data splitting concurrency algorithm. The PIARTC algorithm is described in detail below.
[0073] First, when a concurrent search is performed on a pivot in a range query, an optimistic locking-coupled concurrency strategy (OLC) is used to acquire a read lock on the root node. During the concurrent search of the pivot partition using the IART index, different processing is required based on the size of the prefix and the pivot. Furthermore, the optimistic locking-coupled concurrency strategy is used to traverse the nodes pointing to by the current node (minimum, maximum, left, and right) to ensure the correct unlocked pivot partition is obtained. Afterward, the algorithm performs pivot partition splitting and IART index updates.
[0074] Note that when a prefix mismatch occurs, it will affect not only the current node but also the pointer of the parent node. Therefore, it is necessary to update the read locks on both the current node and the parent node to write locks simultaneously, and then unlock the corresponding node after the IART index has been updated.
[0075] Subsequently, when the pivot and the prefix of the current IART index internal node are successfully matched, it is necessary to obtain the lower-level node of the IART index, while avoiding reading invalid pointers that could cause the program to crash.
[0076] Next, based on the type of the parent node, an optimistic locking-coupled concurrency strategy is used to find the corresponding pivot partition, which is basically similar to the search in a single thread. It's important to note that if this is the first query, only one thread can currently perform the split operation on the entire split column. Since there are currently no leaf nodes in the IART index, it's only necessary to update the lock on the root node to a write lock to prevent other threads from locking the partition and performing the split.
[0077] After updating and unlocking the IART index, if it's not the first query, the corresponding node needs to be locked to complete the IART index update, depending on whether the node needs to be expanded. When the parent node is safe (i.e., no modification will be made to the parent node), the lock on the parent node is released, and the process continues traversing the next level of nodes.
[0078] If the currently queried pivot matches the currently queried leaf node, then no operation needs to be performed and an empty value can be returned. Otherwise, it is still necessary to find the correct pivot partition based on the size relationship between the currently queried pivot and the current leaf node.
[0079] Next, the split column is split and an optimistic locking coupling concurrency strategy is used to ensure the vertical split of the current IART index node, so as to complete the index update.
[0080] At this point, the entire PIARTC algorithm execution is complete.
[0081] like Figure 3As shown, the basic process of the PIATC algorithm concurrently executing three range queries, Q1, Q2, and Q3, is as follows: Figure 3 The top third represents the state after executing range queries [100, 130] and [220, 290]. When... Figure 3 When the three range queries Q1, Q2, and Q3 arrive simultaneously, they can concurrently query the IART index to obtain the pivot partitions to be split. Q1's lower bound of 40 retrieves the pivot partition P1 in the range [0, 100], Q2's lower bound of 180 retrieves the pivot partition P3 in the range [130, 220], Q3's lower bound of 270 retrieves the pivot partition P4 in the range [220, 290], and Q3's upper bound of 350 retrieves the pivot partition P5 in the range [290, N], where N is the maximum value in the split column. Due to the improvement in the locking / unlocking strategy in the PIATC algorithm, it is not necessary to lock each retrieved pivot partition; concurrent splitting of the pivot partitions can be completed simply by locking the leaf nodes of the IART index that identify each pivot partition. Figure 3 As shown, after write locks are acquired on nodes 110, 130, 220, and 290 using a concurrent strategy coupled with optimistic locking, the four pivot partitions P1, P3, P4, and P5 can be concurrently split, and the tuples of each partition can be reassembled. It is worth noting that the upper bound of Q1 (150) and the lower bound of Q2 (180) query the same pivot partition P3, and the upper bound of Q2 (240) and the lower bound of Q3 (270) query the same pivot partition P4. Since the pivot partitions identified by P3 and P4 have already been locked, the split cannot be performed until the locks on the leaf nodes of the pivot partitions identified by P3 and P4 are released. After the concurrent split is completed, the IART index is updated using optimistic locking. During the first concurrent split, the Q3 query completes the entire execution process and returns its query result. Figure 3 The middle third).
[0082] After the first split is completed, the locks on the corresponding data structures are released. Through concurrent queries of the IART index, the upper bound of Q1 (150) finds pivot partition P4, and the upper bound of Q2 (240) finds pivot partition P6. Since there is no conflict between them, similar to the first split, the concurrent splits of pivot partitions P4 and P6 can be completed concurrently. Afterwards, the updates to the IART index are completed concurrently. Figure 3 (The next third). Finally, return the results of the range query for Q1 and Q2.
[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.
Claims
1. A method for concurrent in-memory database splitting, characterized in that: An improvement to the ART index was made, resulting in a new incremental adaptive radix tree index structure that supports database splitting technology, namely the IART index. Based on the IART index, a splitting algorithm based on incremental adaptive radix tree, namely the IARTC algorithm, is proposed. The IARTC algorithm uses the IART index as the splitting index to drive database splitting. At the same time, based on the IARTC algorithm, a new database splitting concurrency algorithm, namely the PIATC algorithm, is obtained. The PIATC algorithm uses an optimistic locking coupling concurrency strategy to control the locking and unlocking of nodes on the IART index to achieve concurrent splitting and index updates, and maintains a small locking granularity during concurrent execution. The IART index adds parent, minKey, maxKey and multi-level support based on the ART index according to different internal node types, so that the upper and lower bounds of the split pivot partition can be obtained by querying the IART index, and it can be used as the split index in the algorithm to drive the database split. The data structure of the Node4 type nodes inside the IART index is as follows: First, add a parent pointer to the Node4 type internal node of the IART index, which represents a pointer to its parent node; if the key being searched does not exist, it is necessary to continue searching for the upper and lower bounds of the key in its parent or ancestor nodes, which is also necessary in other types of internal nodes of IART; then add multi-level support to the Node4 type node. The data structure of the Node16 type node inside the IART index is as follows: Add a pointer to the parent node and multi-level support for Node16 type nodes; The data structure of the Node48 type node inside the IART index is as follows: Add a pointer to the parent node for the Node48 type node, and add two additional keys, maxkey and minkey; where maxkey represents the maximum value of the keys stored in the current Node48, and minkey represents the minimum value of the keys stored in the current Node48. The data structure of the Node256 type node inside the IART index is as follows: Add a pointer to the parent node to Node256, the maximum value of the key stored internally in Node256 (maxkey), and the minimum value of the key stored internally in Node256 (minkey). The IARTC algorithm includes four steps: index pivot partitioning, reorganizing split columns, updating the index, and querying results. Among them, index pivot partitioning and updating the index are improvements on the original standard database splitting algorithm based on the ART index. The specific implementation of the indexed pivot partitioning in the IARTC algorithm is as follows: There are two scenarios: successful query and unsuccessful query. If the value of the range query already exists in the IART index, the position of the current value on the split column is returned directly, and no other split operation is required; If the value of the range query is not in the IART index, then the upper and lower bounds of the pivot partition to be searched for the current value need to be given, which consists of the following three aspects: (a) When searching the boundary values of a range query on the IART index and failing to find a match at a leaf node, if the value of the current query is less than the value of the current leaf node, the upper bound of the pivot partition of the current query value is the value of the leaf node, and the lower bound is the maximum value among all leaf nodes to the left of the current node; if the value of the current query is greater than the value of the current leaf node, the lower bound of the pivot partition of the current query value is the value of the leaf node, and the upper bound is the minimum value among all leaf nodes to the right of the current node. (b) When using a range query to search the boundary value on the IART index, if the traversal encounters a mismatch with the prefix of the internal node, different handling is required depending on the size of the prefix and the current value. If the value of the current query is less than the value of the prefix of the current internal node, then the upper bound of the pivot partition of the current query value is the minimum value of the current internal node, and its lower bound is the maximum value among all leaf nodes to the left of that node. If the value of the current query is greater than the value of the prefix of the current internal node, then the lower bound of the pivot partition of the current query value is the maximum value of the current internal node, and its upper bound is the minimum value among all leaf nodes to the right of that node. (c) When searching for a value on the IART index using a range query, if the retrieved value is null, different handling is required depending on the type of the parent node, as follows: If the parent node is a Node4 type node, first compare the current queried value with the value stored in the parent node; if the current queried value is less than the minimum value stored in the parent node, then the lower bound of the pivot partition of the current queried value is the minimum value of the current layer leaf node, and the upper bound is the maximum value among all leaf nodes to the left of this node; if the current queried value is greater than the maximum value stored in the parent node, then the lower bound of the pivot partition of the current queried value is the maximum value of the current layer leaf node, and the upper bound is the minimum value among all leaf nodes to the right of this node. If the value is exactly between the maximum and minimum values of the parent node, then the lower bound of the pivot partition of the current queried value is the maximum value among all leaf nodes to the left of the node, and the upper bound is the minimum value among all leaf nodes to the right of the node. If the parent node is a Node16 type node, the search method is the same as that for internal nodes of the Node4 type. If the parent node is an internal node of type Node48, due to the special structure of Node48 type nodes, it is first necessary to obtain the maximum and minimum values stored in the Node48 type node. Then, the current query value is compared with the maximum and minimum values found. If the current query value is greater than the maximum value of the Node48 type node, the lower bound of the pivot partition of the current query value is the maximum value among all leaf nodes in the current layer, and the upper bound is the minimum value among all leaf nodes to the right of the current node. If the current query value is less than the maximum value of a Node48 type node, then the lower bound of the pivot partition of the current query value is the minimum value among all nodes in the current layer, and the upper bound is the maximum value among all leaf nodes to the left of this node. Unlike Node4 type nodes, when the current query value is between the maximum and minimum values, it is still necessary to traverse to find the positions of the upper and lower bounds of the pivot partition. If the parent node is an internal node of type Node256, the pivot partition lookup is the same as that for a node of type Node48. The execution process of the IARTC algorithm is as follows: range query First, it checks whether it is the first query. That is, when the IART index is empty, it is necessary to initialize the two most important data structures in the IARTC algorithm, complete the creation of the IART index and the copying of the split column. Search the IART index for the corresponding pivot partition based on the lower bound (low) and upper bound (high) of the range query, with each pivot partition identified by a pair of values. Based on the query results, different splitting algorithms are executed on the splitting column; when the upper and lower bounds of the index query belong to the same pivot partition, a three-split algorithm is executed; when the upper and lower bounds of the index query belong to two different pivot partitions, two two-split algorithms are executed respectively; then the elements in the splitting column are reorganized and the position of the pivot partition is returned; Next, update the IART index, inserting the upper and lower bounds of the query and the corresponding pivot partition positions into the IART index; finally, scan the split column and return the results of the range query.
2. The method for concurrent in-memory database splitting according to claim 1, characterized in that: The PIATC algorithm first performs concurrent queries on the IART index to obtain the pivot partition to be split. Similar to the single-threaded IARTC algorithm, it consists of three parts: searching the IART index for cases where the matching prefix fails, searching for NULL values, and searching for leaf nodes. Then, the same operation is performed in each part, that is, after concurrently searching the corresponding pivot partition on the IART index, the pivot partition is concurrently split, and then the IART index is adaptively and concurrently built to accelerate the execution of concurrent queries.
3. The method for concurrent in-memory database splitting according to claim 2, characterized in that: The basic process of concurrent range query execution using the PIARTC algorithm is as follows: First, when a concurrent search is performed on a pivot of a range query, an optimistic locking concurrency strategy OLC is used to acquire a read lock on the root node. During the concurrent search of the pivot partition using the IART index, different processing is still required depending on the size of the prefix and the pivot. At the same time, optimistic locking coupled concurrency strategy is used to traverse the current node in the processes of finding the smallest leaf node, finding the largest leaf node, finding the maximum value among all leaf nodes to the left of the current node, and finding the minimum value among all leaf nodes to the right of the current node to ensure that the correct unlocked pivot partition is obtained. After that, the pivot partition is split and the IART index is updated. When a prefix mismatch occurs, the read locks on the current node and the parent node need to be updated to write locks simultaneously, and the corresponding node needs to be unlocked after the IART index is updated. When the pivot and the prefix of the current IART index internal node are successfully matched, it is necessary to obtain the lower-level node of the IART index, while avoiding reading an invalid pointer that could cause the program to crash. The corresponding pivot partition is found using an optimistic locking coupling concurrency strategy based on the type of the parent node; If this is the first query, only one thread is currently performing the split operation on the entire split column. Since there are no leaf nodes in the current IART index, it is only necessary to update the lock of the root node to a write lock to prevent other threads from locking the partition and performing the split. Afterwards, the IART index is updated and unlocked. If it is not the first query, the corresponding node needs to be locked to complete the IART index update, depending on whether the node of the IART index needs to be expanded. When the parent node is safe, meaning no modifications will be made to the parent node, the lock on the parent node is released, and the process continues to traverse the next level of nodes. If the current pivot node matches the current leaf node, no operation is required and an empty value is returned. Otherwise, the correct pivot partition still needs to be found based on the size relationship between the current pivot node and the current leaf node. The split column is executed, and an optimistic locking coupling concurrency strategy is used to ensure the vertical split of the current IART index node in order to complete the index update.