System and method for supporting scalability of distributed online transactional database

Abstract

Systems and methods for data partitioning are provided. The data on the data nodes are organized according to a range interval (named as a partition) with a fixed size. The index table and the log table of the data node are not partitioned. A process is disclosed that does not affect read operations by applying MVCC visibility rules, splits partitions online and moves partitions between nodes, and maximizes data availability during partition operations such as splitting and moving. A new process is also disclosed that maximizes availability in the splitting process by enabling certain write operations to be performed in the partition currently being split.

Description

Technical Field

[0001] This invention relates to the field of database systems, and more specifically, to a system and method for partitioning a distributed database to support online transaction processing. Background Technology

[0002] In environments with geographically distributed storage systems (including but not limited to cloud-native databases), dynamic cluster scaling promises to support dynamic workloads. To build scalable storage systems, data can be partitioned and stored on individual data nodes. When the workload of a given application changes, more compute and storage resources may be required. In this case, the system needs to adapt to the new workload and prevent customers from overpaying for resources that are no longer needed.

[0003] In scalable systems, changes in required storage resources can be addressed by altering the number of servers in the storage system. By adding or removing servers allocated for data storage, the system can adapt to different types of workloads. A challenge is designing a system with multiple member nodes that supports dynamic membership changes and transparent data rebalancing. System membership changes and the data rebalancing performed to support these changes should be completed without significantly impacting the current workload. When a new server is added to the system, some data on the existing servers may need to be moved to the new server to achieve a degree of load balancing.

[0004] This may require the system to be able to efficiently identify portions of data that can be migrated to new resources within the system. Ideally, the movement of this identified data should be transparent to users, meaning that such data transfer should not negatively impact the current workload. The system's dynamic membership and transparent data rebalancing capabilities alleviate workload bottlenecks by distributing so-called "hot data" across different servers.

[0005] In database management systems, data can be partitioned across multiple different servers to improve system scalability. This approach is used in shared-nothing architectures, where data can be partitioned into static scopes. For example, Figure 1 This example illustrates a database cluster (or table) with three data nodes. The data in this cluster is partitioned and distributed across these three data nodes. Each record in the database is associated with an index value (also known as an index). id (Value) is associated. Data node 1 contains 2 partitions. The first partition stores values ​​from negative infinity to 250. (-inf, 250) Within the range id The record associated with the value. The second partition stores records related to [ 250，500 Within the range idThe record associated with the value. Similarly, data node 2 has two data partitions. The first partition of data node 2 stores records associated with... [500，750） Within range id The record associated with the value. The second partition of data node 2 stores and [750，1000） Within range id The record associated with the value. Data node 3 has a data partition that stores records in the range from 1000 to infinity. id The records associated with the value. The partition boundaries of static partitions are typically defined when the database comes online. With a static partitioning model, the boundaries cannot be modified throughout the database's lifetime. This assumes the workload (which may involve an existing database) is static. When creating a new database (or new tables) with the required number of partitions and boundaries, and copying all data from the original database to the new database, the current workload operations may slow down, and the final cost of the current workload operations may be negatively impacted: copying data within a sufficiently large database can take a considerable amount of time, during which system performance may significantly degrade.

[0006] Some other traditional database management systems can avoid this problem by over-partitioning the data. Similar to the database systems with static range partitioning discussed earlier, the partition boundaries of an over-partitioned database system are determined at table creation time. While the partition boundaries are fixed, creating a large number of small tables makes it easier to move the entire table or set of tables to a new data node, thus helping to reduce the processing load on each data node. For example, Figure 2 An example of a database with numerous partitions distributed across data nodes 1 through 3 is shown. These partitions can be easily moved to accommodate a new cluster size if the number of data nodes (servers) in the database cluster changes. Excessive partitioning simplifies the process of moving data between different servers in a database cluster. However, this increases the overhead of table management when the amount of data in the database is small. This increases the cost of running this type of database.

[0007] Some databases are designed to store records across a set of partitions, which can be hosted on different data nodes. Data is stored across partitions, and if any given partition exceeds a specified size, it can be split. The resulting partitions can be moved to different data nodes. This move operation is achieved through replication. A target data node is added to the replication group to accumulate data. Once the target data node has captured all the data, it becomes the leader node of the replication group, and the original leader node can be removed from the replication group. Each partition can have, for example, ... Figure 3The diagram shows a separate Write-Ahead Logging (WAL) log table. If a transaction modifies data across two or more partitions, it can use a two-phase commit protocol to ensure the correctness of the committed data in the event of a failure. The two-phase commit protocol is recorded in two separate WAL log tables, which can be used to ensure or verify the accurate recording of data. Separate WAL log tables ensure that log table records are not merged, thus avoiding erroneous I / O records. Even if two partitions are physically located on the same data node, a two-phase commit protocol may still be necessary.

[0008] Hash partitioning is similar to range partitioning. Instead of using the data values ​​in a record as an index, it applies a hash function to the data within the record to generate a hash index. This hash index can then be used to set partition boundaries. This allows for a uniform distribution of data across data nodes without prior knowledge of the data distribution. However, the partition assignment of data nodes is fixed. Combining hash partitioning with consistent hashing enables automatic load balancing. Data records are assigned to data nodes (servers) based on the hash partition key. The hash ring consists of all values ​​in the hash space. A set of virtual nodes is also hashed to a position on the hash ring using a unique ID (e.g., an IP address). Data is distributed to virtual nodes counter-clockwise, where the hash value of the partition key is closest to the hash value of the virtual node's unique ID. This method statistically guarantees a balanced data distribution. Even when new data nodes are added to the ring, only a small portion of the total data needs to be re-migrated. If n is the number of nodes, approximately 1 / n of the total data will be allocated to each virtual node. Hash partitioning solves the load balancing problem during cluster member changes, but it does not necessarily address any of the other shortcomings of the techniques mentioned above. Adding data nodes or splitting partitions is still a process that typically increases data access latency.

[0009] Therefore, improvements are needed to support online transaction processing in distributed databases. Summary of the Invention

[0010] According to an embodiment of the present invention, a method for supporting database scalability is provided. The method is performed by a data node (DN) of the database. The method includes: determining that an existing partition should be split into two partitions, the existing partition comprising a recordset, at least one record in the recordset having an associated record lock, the record lock preventing an entity other than the entity associated with the record lock from performing at least one operation, and at least one record in the recordset not having an associated record lock. The method further includes: copying each record without an associated record lock to a left partition or a right partition; copying each record that has not been previously copied to the left partition or the right partition; before copying each record without the associated record lock, applying a shared partition lock to the existing partition to prevent at least one of a partition move operation and a partition split operation from being performed on the existing partition; after copying each record without an associated record lock, replacing the shared partition lock with an exclusive partition lock to prevent an entity other than the entity associated with the exclusive partition lock from performing at least one operation in the existing partition; and releasing the exclusive partition lock. In some embodiments of the method, the shared partition lock prevents entities other than those associated with the shared partition lock from performing partition move or partition split operations; after each record without the associated record lock is copied to the left or right partition, at least one copied record may be associated with a corresponding record transfer link, an identification number (ID) for the corresponding record copy operation, and a record transfer mark (RTM). In some embodiments, the at least one copied record is an active record; after the shared partition lock is applied, each record without the associated record lock in the existing partition is available for record read operations. In some embodiments, a record read operation of a first record with a snapshot prior to a first record copy operation reads a first record value of the existing partition; a record read operation of a second record with a snapshot prior to a second record copy operation reads a second record value at a location indicated by a second record transfer link; after copying each previously uncopyable record to the left partition or the right partition, at least one previously uncopyable record may be associated with a corresponding record transfer link, the ID of the corresponding record copy operation, and a record transfer mark (RTM); the at least one previously uncopyable record may be an active record. In some embodiments, determining that the existing partition should be split into two partitions includes comparing the amount of unoccupied space within the existing partition with a threshold. In some embodiments, the method includes defining the left partition and the right partition. In some embodiments, the existing partition has unoccupied free space; the existing partition can be used for record insertion operations.The method may include: determining a split pivot value (SPV) having a value between the lowest and highest record value in the sorting of the recordset within the existing partitions. In some embodiments, the left partition has a range defined by a lower limit value of the existing partitions and an upper limit value determined according to the SPV; the right partition has a range defined by a lower limit value determined according to the SPV and an upper limit value of the existing partitions. The method may include: maintaining a log table, wherein the log table is a single log table of the DN. In some embodiments, the method includes: maintaining an index table, wherein the index table is a single index table of the DN. The method may further include: maintaining a block range map (BRM), wherein the BRM associates each partition of the DN with a corresponding logical block range. In some embodiments, each record in the recordset is associated with an identification number (ID) of a corresponding record insertion operation. In some embodiments, after defining the left and right partitions, the BRM may be updated by associating the left and right partitions with corresponding logical block ranges. In some embodiments, at least one record in the recordset is associated with the ID of a corresponding logical delete operation. In other embodiments, the method may further include: receiving a request to provide data for the existing partition; initiating a discovery operation to discover active records in the recordset; when the discovery operation is successful: defining the active recordset; and sending a response including information indicating the active recordset. Each active record in the active recordset may be associated with the ID of a corresponding logical insert operation that has been committed; each active record in the active recordset may not be associated with the ID of a corresponding logical delete operation, or the corresponding logical delete operation may have been aborted. Some embodiments may include one of the following steps: waiting for the corresponding logical insert operation to be committed; waiting for the corresponding logical delete operation to be committed; or including at least one of the following steps: waiting for the corresponding logical delete operation to be aborted; aborting the corresponding logical delete operation. In other embodiments, the request to provide data is received from the coordinator node (CN) of the database; the response is sent to the CN.

[0011] In some embodiments, the CN includes a mapping that associates the existing partition with the DN. In other embodiments, the discovery operation includes at least one of the following steps: waiting for the corresponding record logical insertion operation to abort; aborting the corresponding record logical insertion operation. Each active record in the active recordset of the existing partition may be associated with a record moved mark (RMM). In some embodiments, after receiving the request to provide data, each record in the existing partition without the associated record lock is available for a record read operation. In other embodiments of the method, a record read operation of a third record with a snapshot prior to the discovery operation reads a third record value on the DN; a record read operation of a fourth record with a snapshot after the discovery operation reads a fourth record value on the DN or reads a fifth record value, wherein the position of the fifth record value is indicated by the mapping. In some embodiments, the recordset is divided into two or more consecutive pages; each page on the existing partition has a corresponding offset; copying each record without the associated record lock and copying each previously uncopyable record includes a page-based record copy operation. In some embodiments, each page on the existing partition can be copied to a corresponding target page on the target partition, wherein the corresponding target page has the corresponding offset; the target partition is either the first partition or the second partition. The method may include associating each page with a page split marker and an identification number (ID) of a corresponding page-based record copy operation. In some embodiments, each page in the recordset is associated with an identification number (ID) of a corresponding page insertion operation.

[0012] Another aspect of the present invention provides a method for supporting database scalability. The database has a coordinator node (CN) operatively coupled to a data node (DN) and a second DN. The method includes: the CN receiving information indicating left and right partitions from the DN; the CN updating its mapping by associating the left and right partitions with the DN. The method may further include: sending a request to the DN to provide data for existing partitions on the DN; receiving a response from the DN to the request to provide the data, the response including information indicating a recordset. In some embodiments, the mapping of the CN associates the existing partitions with the DN. In other embodiments, the method further includes: sending a request to the second DN to store data, the storage request including information indicating the recordset; receiving a response from the second DN to the request to store the data. In other embodiments of the method, the response includes confirmation of successful receipt of the information indicating the recordset and a range of new partitions on the second DN storing the information indicating the recordset. The method may include: updating the mapping of the CN by associating the new partitions with the second DN.

[0013] Another aspect of the present invention provides a method for supporting database scalability, wherein the database has a second data node (DN) operatively coupled to a coordinator node (CN). The method includes the second DN of the database performing the following operations: receiving a request to store data from the coordinator node (CN), the request including information indicating a recordset; storing the information indicating the recordset; and sending an acknowledgment message to the CN, the acknowledgment message confirming successful receipt of the information indicating the recordset and the range of a new partition on the second DN storing the information indicating the recordset. In some embodiments, the method may include updating a log table and an index table of the second DN. In other embodiments of the method, the log table of the second DN is a single log table of the second DN, and the index table of the second DN is a single index table of the second DN.

[0014] Another aspect of the present invention provides a data node (DN) apparatus for supporting database scalability. The DN apparatus is integrated into the database infrastructure, includes processing electronics, and is configured to: determine that an existing partition should be split into two partitions, the existing partition comprising a record set, at least one record in the record set having an associated record lock, the record lock preventing an entity other than the entity associated with the record lock from performing at least one operation, and at least one record in the record set not having an associated record lock; copy each record without an associated record lock to a left partition or a right partition; copy each record that has not been previously copied to the left partition or the right partition; before copying each record without the associated record lock, apply a shared partition lock to the existing partition to prevent at least one of a partition move operation and a partition split operation from being performed on the existing partition; after copying each record without an associated record lock, replace the shared partition lock with an exclusive partition lock to prevent an entity other than the entity associated with the exclusive partition lock from performing at least one operation in the existing partition; and release the exclusive partition lock. In some embodiments, the DN device can be used to: receive a request to provide data for the existing partition; initiate a discovery operation to discover active records in the record set. When the discovery operation is successful: define the active record set; send a response including information indicating the active record set. Each active record in the active record set is associated with the ID of a corresponding logical insertion operation that has been committed; each active record in the active record set is not associated with the ID of a corresponding logical deletion operation, or the corresponding logical deletion operation has been aborted.

[0015] In some other embodiments of the DN device, the record set is divided into two or more consecutive pages; each page on the existing partition has a corresponding offset; the copying of each record without the associated record lock and the copying of each previously uncopyable record include page-based record copying operations.

[0016] Another aspect of the present invention provides a coordinator node (CN) device for supporting database scalability. The CN device is operatively coupled to a data node (DN) device and a second DN device. The CN device is integrated into the database infrastructure, includes processing electronics, and is configured to: receive information indicating left and right partitions from the DN devices; and update the mapping of the CN device by associating the left and right partitions with the DN devices. In some embodiments, the CN device is further configured to: send a request to the DN to provide data for existing partitions on the DN device; and receive a response from the DN device to the request to provide the data, the response including information indicating a recordset.

[0017] Another aspect of the present invention provides a second data node (DN) device for supporting database scalability. The second DN device is operatively coupled to a coordinator node (CN) device. The second DN device is integrated into the database infrastructure, includes processing electronics, and is configured to: receive a request to store data from the CN device, the request including information indicating a recordset; store the information indicating the recordset; and send an acknowledgment message to the CN device. The acknowledgment message confirms successful receipt of the information indicating the recordset and the range of a new partition on the second DN device storing the information indicating the recordset.

[0018] Another aspect of the present invention provides a system for supporting database scalability. The system includes a data node (DN) device, a second DN device, and a coordinator node (CN) device. The DN device, the second DN device, and the CN device are integrated into the database infrastructure. In the system, the DN device is configured to: determine that an existing partition should be split into two partitions, the existing partition comprising a record set, at least one record in the record set having an associated record lock, the record lock preventing an entity other than the entity associated with the record lock from performing at least one operation, and at least one record in the record set not having an associated record lock; copy each record without an associated record lock to the left partition or the right partition; copy each record that has not been copied previously to the left partition or the right partition; before copying each record without the associated record lock, apply a shared partition lock to the existing partition to prevent at least one of a partition move operation and a partition split operation from being performed on the existing partition; after copying each record without an associated record lock, replace the shared partition lock with an exclusive partition lock to prevent an entity other than the entity associated with the exclusive partition lock from performing at least one operation in the existing partition; and release the exclusive partition lock.

[0019] The CN device of the system is configured to: receive information indicating a left partition and a right partition from the DN device; and update the mapping of the CN device by associating the left partition and the right partition with the DN device. The second DN device of the system is configured to: receive a request from the CN device to store data, the request including information indicating a recordset; store the information indicating the recordset; and send an acknowledgment message to the CN device to confirm successful receipt of the information indicating the recordset and the range of new partitions on the second DN device storing the information indicating the recordset. In some embodiments of the system, the CN device is further configured to: send a request to the DN device to provide data for a certain partition in the DN device; and receive a response from the DN device to the request to provide data, the response including information indicating the recordset.

[0020] The embodiments described above in conjunction with various aspects of the present invention can be implemented based on these aspects. Those skilled in the art will understand that embodiments can be implemented in conjunction with the aspects described therein, but also in conjunction with other embodiments of that aspect. It will be apparent to those skilled in the art that embodiments are mutually exclusive or contradictory. Some embodiments may be described in conjunction with one aspect, but may also be applicable to other aspects, as will be apparent to those skilled in the art. Attached Figure Description

[0021] Other features and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which: Figure 1 An example of partitioning data to a static range is shown; Figure 2 An example of excessive data partitioning is shown; Figure 3 An exemplary database cluster is shown, where each partition has a separate WAL log table; Figure 4 An exemplary database employing micro-partitioning is shown; Figure 5 An example of a micro-partitioning system component is shown; Figure 6 An exemplary DistMap of the coordinating node is shown; Figure 7 An example of a block range map (BRM) is shown; Figure 8A and Figure 8B An exemplary method for performing record splitting transmission is shown; Figure 9 An exemplary partition movement flowchart is shown; Figure 10 An exemplary partition movement between two data nodes is shown; Figure 11 An exemplary page transfer process is shown; Figure 12 An exemplary schematic diagram of an electronic device for performing any or all operations according to an embodiment of the present invention is shown.

[0022] It should be noted that in all the accompanying drawings, similar features are identified by similar reference numerals. Detailed Implementation

[0023] In this invention, the embodiments discussed below provide a system and method for supporting the scalability of distributed databases (e.g., distributed online transaction processing (OLTP) databases). The system and method of this invention employ micro-partitioning to ensure the scalability of the distributed database. A micro-partition is a small partition of the overall data, which may contain data with partition keys within certain partition boundaries. The partition size may be small, for example, 64MB. Considering the size limitations of micro-partitions, the size of the data stored in the partition may eventually grow to exceed 64MB. The transparent online partition splitting procedure of this invention can automatically split data partitions on a data node into two new partitions. The partition splitting operation can keep the partitions within size limits and can support online operations with minimal negative impact on the current workload. Embodiments of this invention can transparently migrate micro-partitions to different servers with minimal negative impact on the current workload.

[0024] Micro-partitioning may partition only the data layer, without partitioning the index table or log table. Figure 4 A brief overview of a database employing micro-partitioning is shown. This database includes DN 401, DN 402, and DN 403. Data is partitioned according to each partition boundary. For example, DN 401 has partitions 431 and 441. Each partition on a data node uses the same index table and the same Write-Ahead Logging (WAL) table. For example, on DN 401, partitions 431 and 441 share WAL log table 411. Partitions 431 and 441 also share index table 421. Setting up a single WAL log table on a DN means that when data in different partitions on that DN is modified in the same transaction, and the partitions involved are located on the same physical server, a two-phase commit protocol is not required. A single index table on the DN (e.g., index table 421 on data node 401) enables efficient index access and improves index range scans. For read queries, a single index table allows index access operations to bypass the micro-partitioning layer and directly access the required data without requesting and acquiring any locks at the partitioning layer. However, allowing all partitions of a DN to share the same index table can lead to a significant increase in the size of the index table.

[0025] As mentioned earlier, as data volume increases, the size limits set for data partitions may necessitate partition splitting. In an OLTP system environment, splitting must be performed while the partitions remain online and accessible. When a partition is identified as needing to be split, the record constituting the split point needs to be identified. The identifier of the record constituting the split point is called the pivot value. Each partition (sometimes called a subpartition) created by the partition splitting operation may contain approximately half the data of the parent partition. This data can be divided according to the median of the parent partition's partition key. When a partition is detected to be full, a background thread automatically completes the partition splitting process.

[0026] During data migration in a partition split, other ongoing transactions may append intermediate data (i.e., data added to the partition after the partition split begins) to the migrated data. The visibility of the intermediate data may depend on the outcome of the transaction. The systems, apparatus, and methods of this invention enable the resolution of "uncertain" transactions rather than forcibly aborting them. Data associated with the resolved partition is accessible in the sub-partitions created by the split operation. Concurrent write transactions initiated before the partition split process of a data partition are prevented from rolling back. Although during the blocking split phase, subsequent write transactions to the partition that begin after the partition split process has started may be blocked or buffered. Any transaction requesting access to data within a partition where a split operation is underway may be blocked until the entire split process is complete. For example, a transaction requiring the latest version of the data may be blocked, but a transaction with a snapshot from before the split operation may still have read access to those older versions of data. Isolating data partitions during the blocking split phase to prevent them from being modified may impair the performance of the current workload. Early splitting can reduce the blocking time required for the split operation. Early splitting can begin the split process before the data partition is completely filled (i.e., at a certain threshold, such as 80% full). Furthermore, early partitioning processes can be performed using only shared access permissions, rather than acquiring exclusive access to the data partition. For example, when a data partition is partitioned, that partition may still be available for certain update, delete, and insert queries.

[0027] Partition splitting operations can have two phases: an early splitting phase and a blocking splitting phase. The early splitting phase migrates all "cold" data from the original partition to the new partition, while deferring "hot" data to the blocking splitting phase. Cold data can be considered records that no external entity has attempted to modify or add, while hot data can be considered records locked to allow external modification or records being inserted into the table. In the blocking splitting phase, a small amount of data may remain in the partition (e.g., records added during transfer), so exclusive access to the partition may be required to prevent external modification or record addition, thus completing the splitting process.

[0028] Depending on the size of the records (tuples), the overhead of transferring records one by one during a partition split operation can be significant when selecting data for a new partition. As mentioned earlier, the blocking split phase of a partition split operation can prevent clients from modifying data within a partition. A page-based transfer flow can reduce the time required to transfer records to new data partitions, thereby reducing the total time required to perform the partition split operation. In one example, the size of each data partition can be limited to no more than 64MB, which can reduce the time spent migrating data partitions from one data node to another.

[0029] In some embodiments, the micro-partitioning scheme disclosed can be implemented using a database management system called GaussDB, which is based on the open-source Postgres-XC system (which itself was originally based on PostgreSQL). GaussDB employs a "shared-nothing" architecture, consisting of compute nodes that act as coordinator nodes (CNs) and data nodes (DNs). The CN's responsibility is to handle SQL requests from clients, parse these requests, generate data access plans for incoming queries, and communicate with the corresponding DN to actually execute the queries. The DN is responsible for physically storing the data on storage devices and responding to data access requests from incoming query requests.

[0030] As mentioned earlier, micro-partitioning may be suitable when the workload of the underlying application is dynamic, or when cluster members need to adapt to the workload. In both cases, data migration may be required to rebalance the data load across each server in the database cluster. The micro-partitioning approach can provide a way to migrate data between DNs (servers) while minimizing the negative impact on the current transaction processing workload.

[0031] To enable data movement between different data nodes (servers) within a cluster, a naming service may be needed to map partition keys to their respective data nodes. In some embodiments, two mapping services can be used to map partition keys within the database cluster. A coarse-grained mapping service can reside on the coordinator node (CN). The coarse-grained mapping service is represented as... distmap . distmap It can contain a mapping from each partition range to each data node. When the CN receives a request or query from a client, the CN... distmapThese can be used for general data access plans, such as query plans. In some embodiments, these requests can be provided as Structured Query Language (SQL) requests. Naming service (represented as...) Block range mapping This can be located on a data node (DN). Block range mapping maps partition ranges to the logical block numbers where the data within those partitions resides. Block range mapping ensures that data with a given partition key is placed in the correct location for future access. Naming services can support data migration between different DNs (servers) and ensure that migrated data can be found when needed later. Figure 5 The database system shown, in addition to DNs 401 to DN 403, includes two coordinator nodes (CNs): coordinator node 501 and coordinator node 502. CN 501 has a distmap table 511, and CN 502 has a distmap table 522. A distmap is a many-to-one mapping between a range of data partitions and the corresponding DNs storing those data partitions. The contents of different distmap tables may differ, usually only slightly, and these contents may be updated over time to eliminate these differences. Each block range mapping is specific to the particular data node in which it resides.

[0032] Figure 6 The diagram shows DistMap 511 on CN 501. On data nodes, there is a structure called a block range map, for example, Block Range Map (BRM) 451 on DN 401. BRM 451 contains... Block range A list is a structure that contains metadata information for a corresponding data partition. Given a partition key, a block range mapping is used to locate the corresponding block range containing that partition key. Once the block range is located, the metadata information (such as the logical blocks storing the partition data) can be used to locate and modify the actual data.

[0033] When migrating partitions or performing partition splitting operations, the block range mapping may change. Partition migration may add a new active partition to the block range mapping of the target server and remove an active partition from the source server. A partition splitting operation may add two new active partitions to the block range mapping of the server containing the partition and remove an active partition that is being split. Figure 7BRM 701 is shown. After successfully performing the split operation 711 on partition 721, the ranges of new partition 1 and new partition 2 are added to BRM 701. BRM can be stored as an unsorted linear array. Looking up a BRM involves finding the partition containing a specific value. A binary search tree (BST) structure can use partition ranges (i.e., [250, 500)) as keys for fast lookup of specific partitions.

[0034] When an online e-commerce company anticipates a significant increase in traffic and sales in the near future, it may decide to add additional computing resources to its transaction processing or storage systems to support the increased load. As new data nodes (servers) are added to the database system, the data stored within those data nodes may need to be rebalanced across the database cluster. By partitioning data into relatively small 64MB partitions, micropartitioning allows for rapid data migration during partitioning or movement, reducing the likelihood of significant overhead during partition splitting or moving, without significantly negatively impacting the database system's performance.

[0035] When application users travel abroad, data access latency can be severely impacted if the data doesn't move with them. For example, if a user travels from North America to Europe, but the data remains in North America, any data access requests will be redirected to a server geographically located in North America. These requests will retrieve the data from the North American server and send it to Europe. In this scenario, higher data access latency can occur because both command retrieval and data transfer must use higher-latency connections. Moving the data closer to the geographical location initiating the data access request can significantly reduce access latency.

[0036] In addition to moving data partitions to different locations for better load balancing and access latency, database systems can also be designed to perform partition migration operations quickly and efficiently. Utilizing distmap The efficient partition splitting operation of the block range mapping structure can ensure fast and efficient partition migration.

[0037] The partitioning method of this invention ensures that write requests involving data on a partition in an ongoing transaction can be completed without rollback. The visibility and validity of data modifications in these ongoing transactions may depend on the outcome of the transaction.

[0038] To determine whether a data element is available for any particular query, a concurrency control protocol can be used, such as a modified version of the multi-version concurrency control (MVCC) protocol in the PostgreSQL structured query language. To maintain data consistency, each query can be associated with a "snapshot" of the current database state. This snapshot can describe a list of transactions (indicated by transaction identification number (ID) or... xid The query should be able to view the modifications made to these transactions (in the form of a snapshot). By keeping all deleted data available, multiple concurrent queries (with snapshots of different timeframes) can be supported: write operations generate new versions of the data while storing older versions. Queries can select the relevant data version based on the corresponding snapshot.

[0039] Tables can be organized by rows. Each row of data (called a tuple or record) can be paired with two... Xid Related: Xmin and Xmax . Xmin It is a transaction that creates records (i.e., inserts). Xid , Xmax It is a transaction that updates or deletes records. Xid For records that have not been updated or deleted during the query, XmaxThis may be empty. Records that have not been updated or deleted previously can be described as active records. Transactions involving a record that are still receiving client input may be aborted. Aborting may be limited to transactions that are currently receiving or are expected to receive client input. There may be a period of waiting for a transaction involving a record (i.e., the transaction is ready to commit and no longer needs client input) to commit. Records that have been updated or deleted can be described as inactive records. When a record is copied from a parent partition to a child partition, the Xmax of the active record in the parent partition may change, and its record location link (tuple location link) may also be modified. A transaction that accesses a specific record in the parent partition (that specific record has a transfer flag, such as TUPLE_TRANSFERRED) can trigger the fetching of the next record. The fetching of the next record can be triggered even if the corresponding Xmin is not committed, for example, when an ongoing transaction is aborted. The fetching of the next record may be conditional on the successful result of a page split. If an ongoing transaction is aborted, the fetched record may have a transfer flag and a record location link pointing to the child partition. The fetching of the next record via the transfer flag may depend on the page's Xmax and page flags. If page Xmax is aborted, it indicates the operation is invalid, and record transactions in the parent partition can continue as if no transfer (copy) has been made to the child partition. Records transferred (copied) from the parent partition to the child partition can trigger the retrieval of the next record even if their corresponding Xmin is not committed (e.g., when an ongoing transaction is aborted). If the record... Xmin Visible to the query (based on a snapshot of the query), while Xmax If a record is not visible to a query, it may be visible to a query. Xmin and Xmax It can be associated with a data page (where a page is a collection of records). This is due to partition splitting or partition migration operations. Xmin and Xmax It can indicate whether the entire page should be visible.

[0040] The following discloses a blocking splitting method applicable to existing partitions on a data node. In this embodiment, the existing partition may include a recordset. Each record in the recordset may have a corresponding partition key. The recordset may be divided into two or more contiguous pages. The process may include applying an exclusive lock to the existing partition and defining a left partition and a right partition. The left partition may have a range defined by a lower limit and an upper limit of the existing partition. The upper limit of the left partition may be determined based on a partition split pivot (PSP) value. The range of the right partition is defined by a lower limit and an upper limit. The upper limit of the right partition may be set to the upper limit of the existing partition. The lower limit may be determined based on the PSP value, typically as a function of the determined upper limit of the left partition. The blocking splitting method may further include: applying an exclusive lock to each page in the recordset and copying each record in the page to either the left or right partition. Each record in the page copied to the left partition may have a corresponding partition key belonging to the range of the left partition. Each record in the page copied to the right partition may have a corresponding partition key belonging to the range of the right partition. After each record in a page is copied to the left or right partition, one or more records in the page can be associated with the corresponding new record location, the identifier (Xmax) of the corresponding record copy transaction, and the record transfer mark (RTM). The one or more records in the page can be identified as the active record in the page. After each record in a page is copied to the left or right partition, the Xmax of each page can be associated with the ID of the corresponding current transaction and the page split mark. The blocking split method may also include: releasing the exclusive lock on each page; releasing the exclusive lock on an existing partition; and associating the existing partition with the partition split mark. The blocking split operation can traverse each page in a data partition and transfer each record to a new partition. Which particular partition a record is transferred to depends on the record's position relative to the split pivot. The split pivot can be the median of the partition. Each transferred record can have a flag (e.g., TUPLE_TRANSFERRED) whose value can be used to indicate whether the record has been transferred to a new partition. A record can be migrated regardless of whether it has recently been modified by an ongoing transaction. When subsequent read queries find a record marked as transmitted by its transmission flag value, these read queries can trace the link to find the next record, regardless of the transaction result on that record. Figure 8A The diagram illustrates a partition split transfer from a parent partition 801 with a range of (negative infinity, positive infinity) to a child partition 802 with a range of (negative infinity, 2) and a child partition 803 with a range of [2, positive infinity), where each record in the parent partition 801 is moved to one of the child partitions 802 or 803. Figure 8BA flowchart of the partition splitting process is shown. In action 810, the data node maintains a partition, such as parent partition 801. Parent partition 801 includes a recordset. At least one record in the recordset has an associated record lock, which prevents entities other than the entity associated with the record lock from performing at least one operation. At least one record in the recordset on parent partition 801 does not have an associated record lock. In action 820 of this process, a decision is made to split parent partition 801. In action 830, a shared partition lock is applied to parent partition 801. The shared lock prevents partition move operations and / or partition splitting operations from being performed on parent partition 801. In action 840, each record in parent partition 801 without an associated record lock is copied to child partition 802 or child partition 803. In action 850, the shared partition lock is replaced with an exclusive partition lock. The exclusive partition lock prevents entities other than the entity associated with the exclusive partition lock from performing at least one operation in parent partition 801. In action 860, each record that has not been copied before in parent partition 801 is copied to child partition 802 or child partition 803. In action 870, the exclusive partition lock of parent partition 801 is released.

[0041] The early splitting method of this invention can employ a two-phase approach to reduce the blocking time in partition splitting operations. The early splitting phase can acquire only a shared lock on the split data partition (referred to as the parent partition in the following discussion). The lock level may not block other threads from modifying data in the parent partition. This means that data may be modified while data is being transferred to the new partition. The early splitting phase may attempt to move cold data to the new partition. Cold data may be the majority of data in the parent partition. Records in the parent partition that are not associated with any locks indicating possible editing or updating are called cold data or cold records. When a cold record is transferred to the new partition, its corresponding... xmax The transfer flag values ​​may be updated to indicate that these cold records were transferred due to a partition split operation. Records transferred from the parent partition to the new partition may have their corresponding Xmax and transfer flag values ​​marked as unavailable and / or set to zero.

[0042] Since the early split method may be invoked before a partition is filled with data, the partition size limit may not be significant. If a partition is filled with data during its early split phase, transactions attempting to add data to the partition may need to wait until the partition split process is complete.

[0043] Hot data is defined by the ability to acquire locks on specific records during the partitioning process. Records marked as "hot" are skipped in the initial phase because they may be modified before the partitioning process can lock them. Skipping hot records reduces blocking and contention. At the end of the early partitioning phase, a small number of tuples may be skipped due to contention. Another phase of the partitioning process can complete the partitioning by addressing these hot data. Exclusive locks can be acquired on partitions to prevent subsequent modifications. Partitions can be scanned again to find any records that have not been copied or moved previously. These identified records can be transferred to the new partitions through a blocking partitioning process.

[0044] In some embodiments, a partition migration operation may be initiated by a coordinating node. When a partition move operation is invoked, data may be ready to be migrated to the new data node. Records involved in an ongoing transaction may not be migrated. In some embodiments, an ongoing transaction may be forcibly aborted. In other embodiments, the partition move operation may be paused, allowing the ongoing transaction to be committed or aborted. For example, the coordinating node may request the source data node to acquire all data that needs to be transferred to the new data node. Figure 9 This is a flowchart illustrating the call to partition movement, specifically a two-phase commit partition movement. In action 901, CN 501 sends a request message to DN 401 to retrieve partition data (e.g., partition data of partition 442). In response to the request message from CN 501, DN 401 retrieves the partition data in action 902. In action 903, DN 401 sends an acknowledgment message back to CN 501. The acknowledgment message includes the partition data. In action 904, CN 501 sends a storage request to DN 402. The storage request includes the partition data. After receiving the storage request, in action 905, DN 402 stores the partition data in one of its partitions, for example, partition 442. In action 906, DN 402 may send a storage acknowledgment message to CN 501.

[0045] Partition migration operations can be viewed as logical delete instructions on the source data node and logical insert instructions on the target data node. To ensure the validity of data moved between data nodes, only active and visible tuples can be migrated. An active record is a record not associated with a valid Xmax value, i.e., the latest version of a record (tuple) that has not been deleted. Since transaction results are stored locally on each data node, moving data via network communication can be very costly. When only active records are moved to the new data node, all records can be treated as new logical inserts.

[0046] Both the source and target data nodes can have replicas of the partitions. Therefore, MVCC read transactions may not be blocked while the partition migration process is initiated and in progress. For read transactions that only require older versions of records, a snapshot of the older version can guide the read transaction to access data on the source data node. Transactions that require newer versions of records may need to use the target data node. Figure 10 This illustrates a partition move between a source data node (DN 401) and a target data node (DN 402). Partition 441, containing blocks 5 through 8, is moved from DN 401 to DN 402. On DN 402, blocks 5 through 8 are merged into partition 442. After the move is complete, once it is determined that the data in partition 441 is no longer visible to everyone, DN 401 can finally clear partition 441.

[0047] To reduce the time required to transfer tuples during partitioning operations, some embodiments of the present invention may employ a page transfer process, transferring data within a partition at the page level rather than the tuple level. The page transfer process eliminates the overhead of adding records one by one to a new partition. All record offsets on the page can be preserved. For example, the page of the source partition... I The records on the target partition remain in the target partition even after being transmitted during page transfer. i Pages, and maintain the same byte offset on that page. If two records are located on different sides of the split pivot (split median), the only difference between the two records is the target partition number.

[0048] The page transfer process copies all pages from the old partition and stores the data pages in the new partition. Figure 11 This illustrates page transfers between a source partition (partition 1101) and two target partitions (partitions 1102 and 1103). Records -42, 3, 0, 11, 2, 7, 5, 1, and -1 on partition 1101 are moved to partitions 1102 and 1103 based on the split pivot value (median). If a tuple does not belong to a partition, that partition releases the tuple slot. For example, records -1 and -42 from partition 1101 are copied to partition 1102. Records 5 and 11 from partition 1101 are copied to partition 1103. However, record (tuple) links still need to be set, as these links are crucial for record (tuple) visibility resolution.

[0049] Figure 12Schematic diagrams of an electronic device 1200 according to various embodiments of the present invention are shown, which can perform any or all of the operations of the methods and features described herein, either explicitly or implicitly. For example, a computer equipped with network capabilities can be configured as electronic device 1200. Such electronic devices can be used as part of one or more of a controller, edge server, processing device, boundary area module, AV, RSU, etc.

[0050] As shown, the device includes a processor 1210 (e.g., a Central Processing Unit (CPU) or a dedicated processor, such as a Graphics Processing Unit (GPU), or other such processor units), a memory 1220, a non-transient mass storage 1230, an I / O interface 1240, a network interface 1250, and a transceiver 1260, all of which are communicatively coupled via a bidirectional bus 1270. According to some embodiments, any or all of the elements shown may be utilized, or only a subset of the elements may be used. Furthermore, the device 1200 may include multiple instances of certain elements, such as multiple processors, multiple memories, or multiple transceivers. Additionally, elements in the hardware device may be directly coupled to other elements without a bidirectional bus. Other electronic devices, such as integrated circuits, may be employed to perform the required logical operations as a supplement to or alternative to the processor and memory.

[0051] Memory 1220 may include any type of non-transitory memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or any combination thereof. Mass storage 1230 may include any type of non-transitory storage device, such as a solid-state drive, hard disk drive, disk drive, optical disk drive, USB flash drive, or any computer program product for storing data and machine-executable program code. According to some embodiments, memory 1220 or mass storage 1230 may contain statements and instructions executable by processor 1210 for performing any of the above-described methods.

[0052] It should be understood that while specific embodiments of the technology have been described herein for illustrative purposes, various modifications can be made without departing from the scope of the technology. Therefore, the specification and drawings are to be considered merely as a description of the invention as defined in the appended claims, and are intended to cover any and all modifications, variations, combinations, or equivalents falling within the scope of the invention. Specifically, computer program products or program elements for storing machine-readable signals, or program memories or storage devices such as magnetic wires, optical fibers, magnetic tapes, or disks, are provided within the scope of this technology for controlling the operation of a computer according to the method of this technology and / or constructing some or all of its components according to the system of this technology.

[0053] The actions associated with the method described herein can be implemented as coded instructions in a computer program product. In other words, a computer program product is a computer-readable medium on which software code is recorded, which executes the method when the computer program product is loaded into memory and executed on the microprocessor of a wireless communication device.

[0054] Furthermore, each operation of this method can be performed on any computing device such as a personal computer, server, or PDA, based on one or more program units, modules, or objects, or a portion thereof, generated from any programming language such as C++ or Java. Additionally, each operation, or the file or object implementing each operation, can be performed by dedicated hardware or a circuit module designed for this purpose.

[0055] Based on the description of the above embodiments, the present invention can be implemented using only hardware, or using software and necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile or non-transitory storage medium, such as a compact disk read-only memory (CD-ROM), a USB flash drive, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to perform the methods provided in the embodiments of the present invention. For example, such performance may correspond to the simulation of the logical operations described herein. The software product may additionally or alternatively include multiple instructions that enable a computer device to perform operations for configuring or programming a digital logic device according to embodiments of the present invention.

[0056] Although the invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof can be made without departing from the invention. Therefore, the specification and drawings are to be regarded merely as illustrative of the invention as defined by the appended claims and are intended to cover any and all modifications, variations, combinations, or equivalents falling within the scope of the invention.

Claims

1. A method for supporting database scalability, characterized by, The method is executed by the data node (DN) of the database, and the method includes: The existing partition is determined to be split into two partitions, the existing partition comprising a record set, at least one record in the record set having an associated record lock, the record lock preventing entities other than the entity associated with the record lock from performing at least one operation, and at least one record in the record set not having an associated record lock; Copy each record that has no associated record lock to the left or right partition; Copy each record that was not previously copied to the left partition or the right partition; Before copying each record that does not have the associated record lock, apply a shared partition lock to the existing partition to prevent at least one of the partition move operation and partition split operation from being performed on the existing partition; After copying each record that has no associated record lock, the shared partition lock is replaced with an exclusive partition lock to prevent entities other than the entity associated with the exclusive partition lock from performing at least one operation in the existing partition. Release the exclusive partition lock.

2. The method of claim 1, wherein, The shared partition lock prevents entities other than those associated with the shared partition lock from performing partition move or partition split operations.

3. The method according to claim 1 or 2, characterized in that, Also includes: After copying each record that does not have the associated record lock to the left partition or the right partition, at least one copied record is associated with the corresponding record transfer link, the identification number (ID) of the corresponding record copy operation, and the record transfer mark (RTM).

4. The method of claim 3, wherein, The at least one copied record is an active record.

5. The method according to claim 3, characterized in that, After the shared partition lock is applied, each record in the existing partition that does not have the associated record lock is available for record read operations.

6. The method according to claim 5, characterized in that, A record read operation that reads the first record value of the existing partition before the first record copy operation has a snapshot of the first record; The record read operation of the second record with a snapshot after the second record copy operation reads the second record value at the location indicated by the second record transfer link.

7. The method according to claim 1 or 2, characterized in that, Also includes: After copying each previously uncopyable record to the left partition or the right partition, at least one previously uncopyable record is associated with the corresponding record transfer link, the ID of the corresponding record copy operation, and the record transfer mark (RTM).

8. The method according to claim 7, characterized in that, The at least one previously uncopied record is the active record.

9. The method according to claim 1, characterized in that, Determining that the existing partition should be split into two partitions includes comparing the amount of unused space within the existing partition with a threshold.

10. The method according to claim 9, characterized in that, The existing partition has unused free space; The existing partitions can be used for record insertion operations.

11. The method according to claim 1, characterized in that, Also includes: The left partition and the right partition are defined.

12. The method according to claim 1 or 2, characterized in that, Also includes: Determine a split pivot value (SPV) that has a value between the lowest and highest record values ​​in the sorting of the record set within the existing partition.

13. The method according to claim 12, characterized in that, The left partition has a range defined by the lower limit of the existing partition and the upper limit determined according to the SPV; The right partition has a range defined by the lower limit determined according to the SPV and the upper limit of the existing partition.

14. The method according to claim 1, characterized in that, Also includes: Maintain the log table, in which, The log table is a single log table of the DN.

15. The method according to claim 14, characterized in that, Also includes: Maintain the index table, where, The index table is a single index table of the DN.

16. The method according to claim 1, characterized in that, Also includes: Maintaining a block range map (BRM), where, The BRM associates each partition of the DN with a corresponding logical block range.

17. The method according to claim 16, characterized in that, Also includes: After defining the left and right partitions, the BRM is updated by associating the left and right partitions with the corresponding logical block ranges.

18. The method according to claim 1, characterized in that, Each record in the record set is associated with an identification number (ID) for the corresponding record insertion operation.

19. The method according to claim 18, characterized in that, At least one record in the record set is associated with the ID of the corresponding record logical deletion operation.

20. The method according to claim 1, characterized in that, Also includes: Receive a request to provide data for the existing partition; Initiate a discovery operation to discover activity records in the record set; When the discovery operation is successful: Limited activity record collection; Send a response, the response including information indicating the activity record set, wherein... Each activity record in the activity record set is associated with the ID of the corresponding record logical insertion operation, which has been submitted. Each activity record in the activity record set is not associated with the ID of the corresponding record logical deletion operation, or the corresponding record logical deletion operation has been aborted.

21. The method according to claim 20, characterized in that, The discovery operation includes one of the following steps: Waiting for the corresponding record insertion operation to be submitted; waiting for the corresponding record deletion operation to be submitted.

22. The method according to claim 20, characterized in that, The discovery operation includes at least one of the following steps: Wait for the corresponding record logical deletion operation to stop; stop the corresponding record logical deletion operation.

23. The method according to claim 20, characterized in that, The request to provide data is received from the coordinator node (CN) of the database; The response is sent to the CN.

24. The method according to claim 23, characterized in that, The CN includes a mapping; The mapping of the CN associates the existing partition with the DN.

25. The method according to claim 20, characterized in that, The discovery operation includes at least one of the following steps: Wait for the corresponding record logical insertion operation to stop; stop the corresponding record logical insertion operation.

26. The method according to claim 20, characterized in that, Also includes: Associate each active record in the active record set of the existing partition with a record-moved mark (RMM).

27. The method according to claim 24, characterized in that, Upon receiving the request to provide data, in the existing partition, each record without the associated record lock is available for record read operations.

28. The method according to claim 27, characterized in that, A record read operation that takes a snapshot of the third record prior to the discovery operation reads the third record value on the DN; Following the discovery operation, a record read operation of a fourth record with a snapshot reads the fourth record value on the DN or reads the fifth record value, wherein the position of the fifth record value is indicated by the mapping.

29. The method according to claim 1, characterized in that, The record set is divided into two or more consecutive pages; Each page on the existing partition has a corresponding offset; The copying of each record without the associated record lock and the copying of each record that has not been copied before include page-based record copying operations.

30. The method according to claim 29, characterized in that, Each page on the existing partition is copied to a corresponding target page on the target partition, the corresponding target page having the corresponding offset; The target partition is either the first partition or the second partition.

31. The method according to claim 30, characterized in that, Also includes: Each page is associated with a page splitting marker and a corresponding identification number (ID) for the page-based record copying operation.

32. The method according to claim 31, characterized in that, Each page in the record set is associated with an identification number (ID) for the corresponding page insertion operation.

33. A method for supporting database scalability, characterized in that, The database has a coordinating node (CN) operatively coupled to a data node (DN) and a second DN, and the method includes: The CN performs the following operations: Receive information indicating the left and right partitions from the DN; The mapping of the CN is updated by associating the left and right partitions with the DN.

34. The method according to claim 33, characterized in that, Also includes: Send a request to the DN to provide data on existing partitions on the DN; Receive a response from the DN to the request for the provided data, the response including information indicating the record set.

35. The method according to claim 34, characterized in that, The mapping of the CN associates the existing partition with the DN.

36. The method according to claim 34, characterized in that, Also includes: Send a request to store data to the second DN, the storage request including information indicating the record set; Receive a response from the second DN to the request for the stored data.

37. The method according to claim 36, characterized in that, The response includes confirmation that the information indicating the recordset has been successfully received, as well as the range of a new partition on the second DN that stores the information indicating the recordset.

38. The method according to claim 36, characterized in that, Also includes: The mapping of the CN is updated by associating the new partition with the second DN.

39. A method for supporting database scalability, characterized in that, The database has a second data node (DN) operatively coupled to the coordinator node (CN), and the method includes: The second DN of the database performs the following operations: Receive a request to store data from the coordinator node (CN) of the database, the request including information indicating the record set; Store the information indicating the record set; A confirmation message is sent to the CN, the confirmation message being used to confirm successful receipt of the information indicating the record set and the range of the new partition storing the information indicating the record set on the second DN.

40. The method according to claim 39, characterized in that, Also includes: Update the log table and index table of the second DN.

41. The method according to claim 40, characterized in that, The log table of the second DN is a single log table of the second DN, and the index table of the second DN is a single index table of the second DN.

42. A data node (DN) device for supporting database scalability, characterized in that, The DN device is integrated into the database infrastructure, including processing electronics, and is used for: The existing partition is determined to be split into two partitions, the existing partition comprising a record set, at least one record in the record set having an associated record lock, the record lock preventing entities other than the entity associated with the record lock from performing at least one operation, and at least one record in the record set not having an associated record lock; Copy each record that has no associated record lock to the left or right partition; Copy each record that was not previously copied to the left partition or the right partition; Before copying each record that does not have the associated record lock, apply a shared partition lock to the existing partition to prevent at least one of the partition move operation and partition split operation from being performed on the existing partition; After copying each record that has no associated record lock, the shared partition lock is replaced with an exclusive partition lock to prevent entities other than the entity associated with the exclusive partition lock from performing at least one operation in the existing partition. Release the exclusive partition lock.

43. The DN device according to claim 42, characterized in that, Also used for: Receive a request to provide data for the existing partition; Initiate a discovery operation to discover activity records in the record set; When the discovery operation is successful: Limited activity record collection; Send a response, the response including information indicating the activity record set, wherein... Each activity record in the activity record set is associated with the ID of the corresponding record logical insertion operation, which has been submitted. Each activity record in the activity record set is not associated with the ID of the corresponding record logical deletion operation, or the corresponding record logical deletion operation has been aborted.

44. The DN device according to claim 42 or 43, characterized in that, The record set is divided into two or more consecutive pages; Each page on the existing partition has a corresponding offset; The copying of each record without the associated record lock and the copying of each record that has not been copied before include page-based record copying operations.

45. A coordinator node (CN) device for supporting database scalability, characterized in that, The CN device is operatively coupled to a data node (DN) device and a second DN device, the CN device being integrated into the database infrastructure, including processing electronics, and used for: Receive information indicating the left and right partitions from the DN device; The mapping of the CN device is updated by associating the left and right partitions with the DN device.

46. ​​The CN device according to claim 45, characterized in that, Also used for: Send a request to the DN to provide data on existing partitions on the DN device; The DN device receives a response to the request for the provision of data, the response including information indicating the record set.

47. A second data node (DN) device for supporting database scalability, characterized in that, The second DN device is operatively coupled to a coordinator node (CN) device, which is integrated into the database infrastructure, including processing electronics, and is used for: Receive a request to store data from the CN device, the request including information indicating a set of records; Store the information indicating the record set; A confirmation message is sent to the CN device to confirm successful receipt of the information indicating the record set and the range of the new partition storing the information indicating the record set on the second DN device.

48. A system for supporting database scalability, characterized in that, The system includes a data node (DN) device, a second DN device, and a coordinator node (CN) device, wherein the DN device, the second DN device, and the CN device are integrated into the database infrastructure. In the system, The DN device is used for: The existing partition is determined to be split into two partitions, the existing partition comprising a record set, at least one record in the record set having an associated record lock, the record lock preventing entities other than the entity associated with the record lock from performing at least one operation, and at least one record in the record set not having an associated record lock; Copy each record that has no associated record lock to the left or right partition; Copy each record that was not previously copied to the left partition or the right partition; Before copying each record that does not have the associated record lock, apply a shared partition lock to the existing partition to prevent at least one of the partition move operation and partition split operation from being performed on the existing partition; After copying each record that has no associated record lock, the shared partition lock is replaced with an exclusive partition lock to prevent entities other than the entity associated with the exclusive partition lock from performing at least one operation in the existing partition. Release the exclusive partition lock; The CN device is used for: Receive information indicating the left and right partitions from the DN device; The mapping of the CN device is updated by associating the left and right partitions with the DN device; The second DN device is used for: Receive a request to store data from the CN device, the request including information indicating a set of records; Store the information indicating the record set; A confirmation message is sent to the CN device to confirm successful receipt of the information indicating the record set and the range of the new partition storing the information indicating the record set on the second DN device.

49. The system according to claim 48, characterized in that, The CN device is also used for: Send a request to the DN device to provide data for a specific partition within the DN device; The DN device receives a response to the request for the provision of data, the response including information indicating the record set.

Images