Configuration and management of replication units for asynchronous database transaction replication

Abstract

The invention provides a replication method based on a consensus protocol. The blocks are grouped into replication units (RUs) to optimize replication efficiency. The block may be assigned to the RU based on the load and the replication throughput. The splitting and merging RUs do not interrupt concurrent user workloads or require routing changes. Transactions across blocks within the RU do not require distributed transaction processing. Each replication unit has a replication factor (RF), which refers to the number of copies / replicas of the replication unit, and an associated distribution factor (DF), which refers to the number of servers that take over the workload from the failed leader server. The RU may be placed in a ring of servers, where the number of servers in the ring is equal to the replication factor, and the quiescent workload can be limited to the ring of servers rather than the entire database.

Description

[0001] Cross - reference to related applications; claim of rights

[0002] This application claims the benefit of Provisional Application No. 63 / 415,466, filed on October 12, 2022, the entire content of which is incorporated herein by reference in its entirety as if fully set forth herein, under 35 U.S.C.§119(e). Technical field

[0003] The present invention relates to asynchronous database transaction replication using a consensus protocol with fast automatic failover, zero data loss, strong consistency, full SQL support, and horizontal scalability. Background art

[0004] Consensus protocols allow a collection of machines to work as a coherent group that can continue to operate in the presence of some member failures. For this reason, various consensus protocols play a key role in large - scale software systems such as replicated database systems. Raft is a consensus protocol designed to be easy to understand and easy to implement. Raft provides a general way to distribute state machines across a cluster of computing nodes (referred to herein simply as "nodes" or "participant nodes") such that each node in the cluster agrees on the same series of state transitions. Replicated state machines are typically implemented using replicated logs. Each node stores a log replica containing a series of commands, and the state machine of that log replica executes these commands in order; thus, each state machine processes the same sequence of commands. Since state machines are deterministic, each state machine computes the same state and the same sequence of outputs.

[0005] Sharding is a database scaling technique based on horizontal partitioning of data across multiple independent physical databases. Each physical database in such a configuration is referred to as a "shard".

[0006] Sharding relies on replication for availability. Database sharding customers often require high - performance, low - overhead replication that provides strong consistency, supports fast failover with zero data loss, and full Structured Query Language (SQL) and relational transactions. Replication must support nearly infinite horizontal scalability and symmetric sharding, and balance utilization across each shard. There are Raft implementations for database replication that attempt to address the above requirements, such as Database system, Cloud Spanner TM Cloud software, CockRoachDB TM Database system, YugabyteDB, and TiDB.

[0007] Current replication solutions for sharding meet many of these requirements. However, none of them meet all the requirements. For example, LogicalStandby can support failover in 1-2 seconds; however, to balance the load, multiple databases must be configured on each physical server, and more shards must be used to keep up with the workload of the primary shard. Active Data Guard and Logical Standby are active / passive replication strategies with idle hardware at the shard level. GoldenGate TM does not support automatic fast failover in sharded databases.

[0008] Typical NoSQL databases (such as Cassandra TM , DynamoDB TM ) meet many of the above requirements, such as horizontal scalability, simplicity, and symmetric sharding; however, they lack SQL support, ACID (atomicity, consistency, isolation, and durability) transactions, and strong consistency. Some NewSQL databases (such as CloudSpanner TM , CockRoachDB TM , YugabyteDB TM , TiDB TM ) provide SQL support and implement consensus-based replication (Paxos or Raft) that supports strong consistency. They typically implement synchronous database replication, which increases the user transaction response time. YugabyteDB TM claims that for some cases (e.g., single-key DML) it applies changes asynchronously to followers. However, YugabyteDB TM may still require synchronization to present a global time for transactions.

[0009] Kafka TM is a well-known messaging system and meets many of the above requirements, but Kafka TM is non-relational. Raft-based replication (RR) does not require persistent memory, RR adds logical logging on top of the physical redo log, RR supports full SQL, and RR can more easily tolerate replicas that are geographically remote.

[0010] The methods described in this section are methods that can be adopted, but not necessarily methods that have been previously envisioned or adopted. Therefore, unless otherwise specified, it should not be assumed that any method described in this section is considered prior art merely because they are included in this section. Additionally, it should not be considered that they are readily understandable, routine, or conventional merely because any method described in this section is included in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings:

[0012] Figure 1 is a block diagram of a distributed computing system having a state machine and a log replicated across multiple computing nodes according to a consensus protocol, in which aspects of the illustrative embodiments can be implemented.

[0013] Figure 2 is a block diagram depicting grouping of blocks for replication according to an illustrative embodiment.

[0014] Figure 3 illustrates a replication unit in a sharded database management system according to an illustrative embodiment.

[0015] Figure 4 is a diagram illustrating a replication user request process based on a consensus protocol according to an illustrative embodiment.

[0016] Figure 5 is a flowchart illustrating operations for consensus protocol-based replication for a sharded database management system according to an illustrative embodiment.

[0017] Figure 6 is a block diagram illustrating an architecture for consensus protocol-based replication in a sharded database according to an illustrative embodiment.

[0018] Figure 7 is a block diagram depicting log persistence optimization in a leader shard server according to an illustrative embodiment.

[0019] Figure 8 is a block diagram depicting log persistence optimization in a follower shard server according to an illustrative embodiment.

[0020] Figure 9 depicts a replicated log having interleaved transactions according to an illustrative embodiment.

[0021] Figure 10 depicts progress tracking in a replicated log according to an exemplary embodiment.

[0022] Figure 11 Illustrates an example of the multiple ring placement of a replication unit according to an illustrative embodiment.

[0023] Figure 12 Is a data flow diagram illustrating the splitting of an example replication unit according to an illustrative embodiment.

[0024] Figure 13 Is a flow chart illustrating the operation of a new leader shard server taking over a replication unit when there is a commit initiated in the replication log according to an illustrative embodiment.

[0025] Figure 14 Is a flow chart illustrating the operation of a shard server performing replication log recovery according to an illustrative embodiment.

[0026] Figure 15 Is a block diagram of a computer system on which embodiments of the present invention can be implemented.

[0027] Figure 16 Is a block diagram of a basic software system that can be used to control the operation of a computer system. Detailed Description

[0028] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be clear that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0029] General Overview

[0030] Illustrative embodiments provide asynchronous database transaction replication based on log replication.

[0031] According to an illustrative embodiment, a leader server receives a command to perform a change operation on a row of a table in a database. The table is replicated across a replication group of servers such that each server within the replication group of the server stores a corresponding copy of that row of the table. The replication group of the server includes the leader server and one or more follower servers. The leader server is configured to perform a data manipulation language (DML) operation on the row of the table and replicate the DML operation to one or more follower servers. The leader server performs the change operation on the copy of the row of the table stored at the leader server. The leader server then creates a replication log record for the change operation to be replicated to one or more follower servers in a replication pipeline and returns the result of the change operation to the client. The replication pipeline includes components and mechanisms for storing the log record at the leader and propagating the log record to the followers (from generating the log record on the leader to persisting the log record to disk on the followers). The leader server performs the change operation and returns the result of the change operation to the client without waiting for consensus from one or more follower servers for replicating the log record. Thus, according to the illustrative embodiment, the result of the change operation (DML operation) is returned to the client immediately after the change is made and the log record is created at the leader, without waiting for consensus from the follower servers. After being returned to the client, the propagation of the log record to the followers occurs asynchronously.

[0032] In some embodiments, the leader server receives a database transaction commit command to perform a database transaction commit operation for a particular transaction, creates a replication log record in a replication pipeline for the database transaction commit operation, and performs the database transaction commit operation on the copy of the row of the table at the leader server in response to receiving an acknowledgement that the replication log record for the database transaction commit operation has been appended to the replication logs of a quorum of one or more follower servers. The leader server then returns the result of the database transaction commit operation to the client. For the database transaction commit operation, the leader server waits for consensus from the follower servers to perform the database transaction commit operation and return the result to the client. Thus, according to the illustrative embodiment, the database transaction commit operation log record is replicated synchronously. The leader server performs DML operations and database transaction commit operations asynchronously, replicates DML log records asynchronously, and replicates commit log records synchronously, thereby enabling fast automatic shard failover with zero data loss and strong consistency.

[0033] According to some illustrative embodiments, to ensure consistency between leader transaction commit and synchronous log replication, the leader server prepares a local transaction for commit and marks the transaction as "pending", and then sends a commit log record to the follower servers. After receiving consensus on this commit log record, the leader server can commit the local transaction. After a leader server failure, if the old leader becomes a follower and there is consensus on the commit log record based on the replicated log, then the "pending" state of the local transaction in the old leader can be committed; otherwise, the transaction is rolled back in the old leader.

[0034] In an alternative embodiment, before committing the local transaction, the leader server sends a pre-commit log record to the follower servers. After receiving consensus on the pre-commit record, the leader server can commit the local transaction and send a post-commit log record to the follower servers. After a leader shard server failure and the leader becomes a follower, if there is consensus on the pre-commit log record based on the replicated log, then the local transaction in the old leader can be committed. If the local transaction in the old leader has been rolled back by the database, then the transaction is replayed to the old leader based on the replicated log.

[0035] According to illustrative embodiments, to optimize replication efficiency, chunks are grouped into replication units (RUs). A chunk is a unit of distribution in a sharded database system. A sharded database system includes multiple shard servers. In the sharded database of the sharded database system, each sharded table is divided across multiple chunks; a chunk can contain portions of multiple tables. There are typically a large number (thousands or tens of thousands) of chunks, which minimizes data movement during re-sharding and minimizes dynamic chunk splitting. According to some illustrative embodiments, chunks are assigned to RUs based on load and replication throughput. Splitting and merging RUs does not interrupt concurrent user workloads and does not require routing changes since the associated chunks remain in the same set of shard servers. Additionally, transactions that span chunks within an RU do not require distributed transaction processing.

[0036] According to an illustrative embodiment, each replication unit has a replication factor (RF) and an associated distribution factor (DF). The replication factor refers to the number of replicas / copies of the replication unit, including the primary replica at the leader. The distribution factor refers to the number of shard servers that take over the workload from a failed leader shard server. The replication factor must be odd to determine a majority. The higher the DF, the more balanced the workload distribution after failover. In some embodiments, the RUs are placed in a ring of shard servers, where the number of shard servers in the ring is equal to the replication factor. This placement helps with schema upgrades. In the case of barrier DDL, the quiescing of the workload can be restricted to the ring of shard servers rather than the entire sharded database. Alternatively, the placement can be in the form of a single ring. For example, the leader RU on shard 1 is replicated on shards 2 and 3, the leader RU on shard 2 is replicated on shards 3 and 4, and so on. For a single-ring structure, barrier DDL results in the quiescing of the entire sharded database; however, when new shards are added, it is possible to limit the number of times the RUs can be split, moved, and merged to a fixed number, thereby reducing data movement during incremental deployment and making incremental deployment a more deterministic operation. In a multi-ring or single-ring arrangement, the removal of a shard server (e.g., due to failure or scaling down) can be performed similarly.

[0037] According to an illustrative embodiment, lead-sync logging is used to synchronize the replication logs of follower shards to the leader shard. In response to the inability to determine the existence of a consensus for a database transaction commit operation after a shard server becomes the new leader, the new leader shard uses lead-sync logging to perform a synchronization operation to synchronize the replication logs of follower shards to the replication log of the new leader. Lead-sync logging requires consensus. Therefore, the new leader can determine whether there is a consensus for a database transaction commit operation based on the result of the synchronization operation. The new leader performs the synchronization operation by sending lead-sync logging to the follower shard servers.

[0038] In an embodiment, when recovering from a database failure, the shard server identifies a first transaction in the replication log that has a first log record but does not have a post-commit log record, defines a recovery window in the replication log (the recovery window starts at the first log record of the identified first transaction and ends at a lead-sync log record), identifies a set of transactions to recover, and performs recovery actions on the set of transactions to recover. Any transaction that has a log record outside the recovery window is not included in the set of transactions to recover, any transaction that has a post-commit log record within the recovery window is not included in the set of transactions to recover, and any transaction that has a rollback log record in the recovery window is not included in the set of transactions to recover. This allows the shard server to match the state of the replication log to the state of the database transactions by re-creating missing log entries and replaying or rolling back transactions.

[0039] The following describes the implementation of illustrative embodiments with reference to a sharded database management system; however, aspects of the illustrative embodiments can be applied to other types of database management systems. In some embodiments, a non-sharded database can be represented as a sharded database with only one shard (where all blocks are in that one database, which can be a single server instance or a Real Application Cluster (RAC)). In other embodiments, a non-sharded database can be a mirrored database implementation (where the non-sharded database can be represented as a sharded database with one replication unit that includes all blocks), the replication group includes all servers, and the replication factor is equal to the number of servers. That is, the replication group can include all servers or a subset of the servers, and the replication unit can include all blocks of the database or a subset of the blocks.

[0040] RAFT protocol

[0041] Raft is a consensus protocol for managing replicated logs. For enhanced understandability, Raft separates the key elements of consensus, such as leader election, log replication, and safety, and enforces a stronger degree of coherency to reduce the number of states that must be considered. Figure 1 is a block diagram of a distributed computing system having a state machine and a log replicated across multiple computing nodes according to a consensus protocol, in which aspects of the illustrative embodiments can be implemented. In Figure 1 the example shown, there is a leader node 110 and two follower nodes 120, 130; however, the distributed computing system can include other numbers of nodes depending on the configuration or workload. For example, the number of nodes in the group of participating nodes can be scaled up or down depending on the workload or other factors that affect resource usage. The consensus protocol typically appears in the context of replicated state machines. As Figure 1As shown, state machines 112, 122, and 132 are replicated across groups of compute nodes 110, 120, and 130, respectively. State machines 112, 122, and 132 operate to compute the same state, and continue to operate even if one or more of compute nodes 110, 120, and 130 fail.

[0042] The replicated state machines are implemented using replicated logs. Each node 110, 120, and 130 stores a log 115, 125, and 135, respectively, which contains a series of commands executed in order by the node's state machine 112, 122, and 132. Each log should contain the same commands in the same order, so each state machine will process the same sequence of commands. Since state machines 112, 122, and 132 are deterministic, each state machine computes the same state and the same output sequence.

[0043] Keeping the replicated logs consistent is the purpose of the consensus protocol. The consensus module 111 on the leader node 110 receives commands from clients (such as client 105) and adds the command to its log 115. The consensus module 111 of the leader node 110 communicates with the consensus modules 121 and 131 of the follower nodes 120 and 130 to ensure that the logs 125 and 135 of these nodes ultimately contain the same requests or commands in the same order even in the event of one or more node failures. When the commands are correctly replicated, the state machines of each node process them in log order and return the output to the client 105. Thus, nodes 110, 120, and 130 appear to form a single, highly reliable state machine.

[0044] A Raft cluster or group (also referred to herein as a replication group) contains several nodes, such as servers. For example, a typical Raft group can include five nodes, which allows the system to tolerate two failures. At any given time, each server is in one of three states: leader, follower, or candidate. In normal operation, there is exactly one leader, and all other participating nodes are followers. Followers are passive and do not initiate requests on their own; followers simply respond to requests from the leader and candidates. The leader handles all client requests. If a client contacts a follower, the follower redirects the client to the leader. The third state (candidate) is used to elect a new leader.

[0045] After a leader is elected, the leader begins serving client requests. Each client request contains a command to be executed by the replicated state machine. The leader node 110 appends the command as a new entry to its log 115 and then issues AppendEntries RPCs in parallel to the other nodes 120 and 130 to replicate the entry. Each log entry stores the state machine command and the term number when the leader received the entry. The term number in the log entry is used to detect inconsistencies between logs and to ensure some of the properties in the nature. Each log entry also has an integer log index that identifies the position of the log entry in the log.

[0046] Raft guarantees that "committed" entries are durable and will eventually be executed by all available state machines. A log entry is committed after the leader that created the entry has replicated the entry on a majority of servers (consensus). This also commits all previous entries in the leader's log, including entries created by previous leaders. The leader keeps track of the highest index known to the leader to be committed and includes that index in future AppendEntries RPCs (including heartbeats) so that other servers will eventually learn of it.

[0047] The RAFT Protocol in a Replicated DBMS

[0048] This document describes the Raft consensus protocol for a cluster or group of computing nodes such as servers. In the context of a replicated DBMS, the Raft consensus protocol is applied to a log of replicated commands that are to be executed by the state machines of database servers to apply changes to the database. The changes to be applied to the database by the leader database server (e.g., the leader shard in a sharded database system) are logged at the leader database server and replicated to one or more follower database servers. In turn, each follower database server receives the commands in its log and applies the changes to the corresponding replica of the database in order using the server's state machine.

[0049] In a replicated DBMS implementation, the leader node intercepts changes (e.g., Data Manipulation Language (DML) commands, segmented Large Object (LOB) updates, JavaScript TM Object Notation (JSON) inserts and updates) as Logical Change Records (LCRs). The leader node constructs a Raft log record based on the LCR, which is replicated to the follower database servers.

[0050] As an example of a specific implementation scheme of a replicated DBMS, sharding distributes fragments of a dataset across many database servers on different computers (nodes). Sharding is a data layer architecture where data is horizontally partitioned across independent database servers. Each database server is hosted on a dedicated computing node with its own local resources. Each database server in such a configuration is referred to as a "shard server" or a "shard". All shards together constitute a single logical database system, which is referred to as a sharded database management system (SDBMS). In some embodiments, horizontal partitioning involves splitting a database table across shards so that each shard contains a table with the same columns but a different subset of rows. A table split in this way is also referred to as a sharded table. In an SDBMS, each participating node can be a leader for one subset of the data and a follower for other subsets of the data.

[0051] In the context of a replicated DBMS such as an SDBMS, the Raft consensus protocol disposes of leader election, log replication, and replicated group membership changes, with modifications that will be described below. The modifications of the illustrative embodiments help to ensure asynchronous database transaction replication for fast, automatic shard failover with zero data loss, strong consistency, full SQL support, and horizontal scalability.

[0052] Grouping of blocks for replication

[0053] A replication unit (RU) consists of a collection of blocks. Figure 2 is a block diagram depicting the grouping of blocks for replication according to an illustrative embodiment. Shard 210 may have multiple block collections (RUs) 220. Each RU 220 has a collection of blocks 230. Smaller replication units have lower instantiation overheads after a shard failure. Too many RUs will have higher runtime overheads (e.g., processes). Thus, there is a trade-off between the lower overhead associated with smaller RUs (i.e., smaller collections of blocks) and the lower runtime overhead associated with larger RUs (i.e., fewer RUs).

[0054] Each RU consists of a collection of blocks. All transactions within an RU are replicated in the same replication pipeline, which consists of a set of processes, in-memory data structures, and associated replication logs. To minimize replication overhead, the illustrative embodiments configure the size of the replication unit to maximize throughput. Large replication units increase the data movement time during re-sharding.

[0055] Figure 3Illustrates a replication unit in a sharded database management system according to an illustrative embodiment. Each RU has a set of a leader shard server 310 and follower shard servers 320, 330, and the leader shard and all follower shards have the same set of blocks. All DML operations for a particular row are performed in the leader and replicated to the followers of that leader. This is primary replica replication. A shard can be the leader of one replication unit and a follower of other replication units. This results in better hardware utilization. All reads are routed to the leader, unless the application explicitly requests to read from a specified follower shard and tolerate stale data (which can be beneficial if the application is geographically closer to the follower).

[0056] According to an illustrative embodiment, each replication unit has a replication factor (RF), which refers to the number of replicas / copies of the replication unit (including the primary replica at the leader). The Raft protocol requires a majority of replicas to be available for writes; however, a read quorum is not required. This is in contrast to NoSQL databases. When RF = 3, one replica failure can be tolerated; when RF = 5, two replica failures can be tolerated. The illustrative embodiment maintains the replication factor in the event of a shard failure, assuming there is capacity in other available shards. Each replication unit also has an associated distribution factor (DF), which refers to the number of shard servers that take over the workload from a failed leader shard server. The higher the DF, the more balanced the workload distribution after a failover.

[0057] Asynchronous database transaction replication based on synchronizing the Raft log

[0058] User Request Process

[0059] The illustrative embodiment uses a consensus protocol (such as the Raft protocol) to synchronously replicate the LCR and perform leader elections after a failure or on demand. The synchronous replication of the LCR does not mean the synchronous replication of transactions at the follower shard servers. Figure 4 Is a schematic diagram illustrating a replication user request process for change operations based on a consensus protocol according to an illustrative embodiment. For a given replication unit, there is a leader 410 and multiple followers 420, 430. Each node or server 410, 420, 430 has a corresponding shard catalog 412, 422, 432, which is a dedicated database that supports automatic shard deployment, centralized management of the sharded database, and multi-shard queries. For a given replication unit (a set of blocks), the shard catalogs 412, 422, 432 maintain data describing which server is the leader and which servers are the followers. In an alternative implementation, each node or server can share the shard catalog.

[0060] A logical change record (LCR) encapsulates row changes (e.g., insert, update, LOB operations, JSON operations, old / new values) and transaction directives (e.g., commit, rollback, partial rollback). A replicated log record (LR) is an LCR with a valid log index. Log records in a replication unit have a strictly increasing log index. A replicated log contains log records for in-flight, uncommitted transactions (like a redo log), but does not contain undo or index. This is in contrast to other solutions that only contain committed transactions. The terms "logical change record (LCR)" and "log record (LR)" are used interchangeably herein.

[0061] According to an illustrative embodiment, a read-only (R / O) routing state is maintained for blocks in a follower RU in a routing map (not shown), and a read-write (R / W) routing state is maintained for blocks in a leader RU in the routing map. When there is a change in the leader of a RU, the R / O and R / W states change accordingly. According to an illustrative embodiment, when there is a leadership change, the routing map cached outside the database (e.g., in a client-side driver) is invalidated and reloaded via a notification.

[0062] In the existing Raft protocol, a client sends a command to the leader of a Raft group, and the leader appends the command to the leader's log, sends an AppendEntries RPC to all followers, and after the new entry is committed (stored in the replicated logs of a quorum of followers), the leader executes the command and returns the result to the client. The leader notifies the followers of the committed entries in subsequent AppendEntries RPCs, and the followers execute the committed commands in their state machines. In contrast, the sharded database replication method of the illustrative embodiment first executes the DML in the database before appending it to the replicated log.

[0063] In the replication method based on a consensus protocol of the illustrative embodiment, a logical change record (LCR) encapsulates row changes (e.g., insert, update, delete, LOB operations, JSON operations, old / new values) and database transaction directives (e.g., database transaction commit, database transaction rollback, partial rollback). A log record is an LCR with a valid log index and a term. Log records in a replication group have a strictly increasing log index. Log records contain in-flight, uncommitted transactions, as will be described in further detail below.

[0064] In the consensus protocol - based replication method of the illustrative embodiment, the user / client 401 sends a DML to the leader 410, which executes the DML on the replicas of the leader's replication unit before appending the log record (LR) for the DML to the leader's replication log 415. The leader also returns a result (e.g., and an acknowledgement) to the user / client 401 in response to the LR being created in the leader's replication log 415. The result is returned without verifying that the followers have stored the LR in the followers' replication logs, and thus without the followers' confirmation of storing the LR. The leader can return to the client before writing the log record to the leader's replication log. The persistence of the log record is done asynchronously.

[0065] The replication log 415 is propagated to the followers 420, 430, which respectively store the LR for the DML in the replication logs 425, 435. In response to the follower 420 appending the LR to the follower's replication log 425, the follower 420 returns an acknowledgement (ACK). If a quorum of followers returns ACKs to confirm that the LR has been stored in the replication logs, then the leader 410 considers the DML LR to be committed. Similarly, the followers 420, 430 eagerly execute the DML while concurrently appending the DML to the followers' replication logs 425, 435. This minimizes the impact on the user transaction response time and improves replication efficiency.

[0066] To maintain the ACID properties among the replicas, the leader commits the relevant database transaction only when the committed LCR is a committed log record (appended to a quorum of follower replication logs), which means that any LCR generated previously for the database transaction is also a committed log record. Thus, in the sharded database replication method of the illustrative embodiment, the user / client 401 sends a database transaction commit command to the leader 410, which creates a commit LR in the leader's replication log 415 and propagates the replication log 415 to the followers 420, 430. The leader commits the database transaction in the replicas of the leader's replication unit only when it receives confirmations for the commit LR from a quorum of followers. The leader generates a post - commit LR and then returns the result (e.g., and an acknowledgement) of the database transaction commit to the user / client 401.

[0067] Figure 5FIG. 0 is a flowchart illustrating operations of consensus - protocol - based replication for a sharded database management system. The operation starts (block 500), and the leader shard server receives a command from a client (block 501). The leader shard server determines whether the command is a database transaction commit command (block 502). If the command is not a database transaction commit command (block 502: No), then the command is a change operation (e.g., insert, update, delete, etc.), and the leader shard server executes the change operation (block 503). Then, the leader shard server creates a replication log record in the replication log of the leader shard server (block 504) and returns the result of the change operation to the client (block 505). After that, the operation returns to block 501 to receive the next command from the client.

[0068] If the command is a database transaction commit command (block 502: Yes), then the leader shard server creates a replication log record for the database transaction commit in the replication log of the leader shard server (block 506). The leader shard server determines whether the leader shard server has received consensus for the commit log record (block 507). If consensus is received (block 507: Yes), then the leader shard server executes the database transaction commit operation (block 508), advances the commit index (block 509), and writes the log record to disk (block 510). After that, the operation returns to block 501 to receive the next command from the client.

[0069] If consensus is not received (block 507: No), then the leader shard server rolls back the database transaction (block 511). After that, the operation returns to block 501 to receive the next command from the client.

[0070] System Architecture

[0071] Figure 6 FIG. 13 is a block diagram illustrating the architecture for consensus - protocol - based replication in a sharded database according to an illustrative embodiment. User 601 sends DML and transaction indications to leader 610 for a given replication unit. As Figure 6 shown, leader 610 includes capture component 611, System Global Area (SGA) 612, in - memory replication log queue 613, network senders 614A…614B, consensus module 615, and a persistent replication log in disk 616. Capture component 611 intercepts DML execution and captures DML, piecemeal LOB updates, JSON inserts, and updates (JSON_Transform) as LCR. Capture component 611 also intercepts transaction execution to capture database transaction commit, database transaction rollback, and rollback to savepoint.

[0072] The SGA is a set of shared memory structures that contain data and control information for a database instance, called SGA components. The SGA is shared by all server and background processes. In the depicted example, the capture component 611 stores LCRs to be inserted into the in-memory replication log queue 613.

[0073] In some embodiments, there is a commit queue (not shown) between the SGA 612 and the consensus module 615. This commit queue contains commit records for each transaction. When the consensus module 615 receives an acknowledgement from a follower, the consensus module reads the commit queue to find a matching transaction and checks whether this transaction has achieved consensus. If consensus is achieved, then the consensus module releases the user session, which allows the transaction to commit, generates a post-commit LCR, and returns control to the user.

[0074] The network senders 614A, 614B distribute the replication logs to the followers 620, 630 via the network 605. The leader can have a network sender for each follower in the replication group. The consensus module 615 communicates with consensus modules on other servers, such as the consensus modules 625, 635 on the followers 620, 630, to ensure that each log ultimately contains the same log records in the same order even if some servers fail.

[0075] The follower 620 includes a network receiver 621, an in-memory replication log queue 622, an SQL application server 623, a consensus module 625, and a persistent replication log in the disk 626. Similarly, the follower 630 includes a network receiver 631, an in-memory replication log queue 632, an SQL application server 633, a consensus module 635, and a persistent replication log in the disk 636. The network receivers 621, 631 receive the replication logs from the leader 610 and transfer the LCRs to the SQL application servers 623, 633.

[0076] The consensus modules 615, 625, 635 have LCR persistencer processes at each replica (including the leader) to durably persist the LCRs into the persistent replication logs in the disks 616, 626, 636. In the followers, the consensus modules 625, 635 send the highest persisted log index back to the leader 610 of the followers to confirm that the log records up to the persisted log index have been persisted.

[0077] Only the leader can process the user's DML requests. Followers can automatically redirect DMLs to the leader. In the DML execution path, the leader constructs a log record to encapsulate the DML change, enqueues the log record into the SGA circular buffer, and immediately returns to the user. The illustrative embodiments decouple replication from the original DML and asynchronously pipeline DML log record, propagation, and SQL application at the followers with minimal latency. For multi-DML transactions, replication mostly overlaps with the user transaction and the latency overhead from commit consensus is significantly smaller.

[0078] DML Replication Process

[0079] Reference Figure 6 , user 601 submits a DML. The capture component 611 constructs an LCR in image format or object format and enqueues the LCR into the in-memory queue 613. In one embodiment, the in-memory queue is a contention-free multi-writer, single-reader queue. The LCR in image format in the queue minimizes multiple downstream serializations (e.g., persisting and propagating the LCR). The leader 610 returns to the user 601 immediately after constructing the required log record in the DML execution path.

[0080] The in-memory queue 613 is a single-writer and multi-reader queue. The LCRs in the SGA 612 are stored in a different queue which is a contention-free multi-writer and single-reader queue. Each capture process representing a user is a writer to this queue. The LCR producer process is a single-reader of this queue, dequeues the LCR from the SGA 612, assigns a unique and strictly increasing log index to the LCR, and enqueues the log into the in-memory queue 613. The strictly increasing log index is an important property of the Raft log.

[0081] Asynchronously, the consensus module 615 scans through the queue 613 and constructs replication log records based on the LCRs dequeued from the queue. The consensus module 615 (LCR persister) persists the in-memory log records to the disk 626. The consensus module 615 (ACK receiver) counts the acknowledgments from the followers 620, 630 and appropriately advances the log commitIndex. The network senders 614A, 614B call the AppendEntries RPC to propagate the log records to all followers via the network 605. For each log record propagation, the network senders 614A, 614B also include the current committed log index.

[0082] The network receivers 621, 631 are automatically generated due to the connections from the network transmitters 614A, 614B at the leader. If the AppendEntries RPC passes its verification, then the network receivers 621, 631 enqueue the log records from the wire to the queues 622, 632 via the consensus protocol application programming interface (API). The consensus modules 625, 635 at the followers read the in-memory queues 622, 632 containing the log records from the leader, persist the log records to the disks 626, 636, and send acknowledgments to the leader via separate network connections.

[0083] The SQL application servers 623, 633 read the LCRs from the in-memory queues 622, 632, assemble the transactions from the cross-cutting LCRs from different transactions, and apply the transactions to the database. This is referred to herein as "eager apply". If the SQL application servers 623, 633 are slow or are catching up, then the SQL application servers may need to retrieve the relevant log records from the disks 626, 636.

[0084] As mentioned above, the consensus-protocol-based replication of the illustrative embodiments does not require explicit consensus on DML. Once the LCR for the row change is pushed into the replication pipeline, the leader immediately returns control to the user or allows subsequent processing of the transaction. This minimizes the impact on the user's response time for DML. The replication (propagation, persistence) of DML is done asynchronously in a streaming fashion. If there is consensus for the commit, then the replication method maintains the transaction ACID properties among the replicas.

[0085] Submit Replication Process

[0086] When receiving a database transaction commit for a user transaction from the user 601, the network transmitters 614A, 614B send the log records containing the transaction commit to all the followers 620, 630. In parallel, the consensus module 615 (LCR persister) writes the log records to the disk 616 and keeps track of the persisted log indexes. In other words, the leader 610 does not need to wait for the persistence of the log records containing the transaction commit before sending its log records to its followers.

[0087] When the leader 610 receives consensus from a quorum of followers, the leader advances the leader's log commitIndex as far as possible. The committing and advancing of the commitIndex for user transactions need not be atomic. The leader 610 need not verify that the log records have been persisted locally before sending the next set of log records to followers 620, 630. The leader needs to ensure that the committed log records have been persisted locally before committing the transaction locally. In practice, when the leader receives consensus, the log records will have been persisted anyway. One way is to verify that the persisted log index is equal to or greater than the log index for which the log record was committed for the transaction. Then, the leader commits the transaction and returns success to the user.

[0088] If the leader 610 crashes before communicating the consensus of the log records to its followers, then the new leader can complete the replication of any last set of committed log records regardless of whether the previous leader has started. Because the application of the transaction commit command and the return to the user are not atomic, the user may resubmit the transaction, potentially to a different leader. This resubmission of the transaction may encounter an error and exist outside the sharded database.

[0089] Submitted Batch Processing Confirmation

[0090] According to an illustrative embodiment, all log records (including commits) are sent continuously and without interruption in the same stream. Acknowledgments are sent back via a separate network connection. There is only one network round-trip for acknowledgments from each fast follower. Even when the leader is waiting for acknowledgments of outstanding commits for a user transaction, the leader can continue to stream log records with higher log indexes, including other commits from other user transactions. In some embodiments, all log records cannot have inter-transaction dependencies. This maximizes concurrency for independent user transactions. This also enables natural batching of acknowledgments for commits: one acknowledgment from a follower can include multiple transactions. Therefore, this improves replication efficiency. Obtaining consensus for committed log records requires only one round-trip.

[0091] Replication Log Persistence

[0092] A set of the replication log is maintained for each replica in each replication group of the server (a copy of the RU on the server). There will be no interference between replication units when reading and writing to the replication log. There are numerous possible embodiments for persisting the replication log across all replicas (including the leader), including the following examples:

[0093] 1. Since database redo persistence has been highly optimized, it would be efficient to write replication logs as database redo records. The replication log records are interleaved with database change records in the redo log. There is no indexing ability when reading the redo log. Therefore, it is inefficient to find a specific replication log record from the redo.

[0094] 2. Write the replication log to an external file or queue. It will be necessary to deploy and monitor external components.

[0095] 3. Write the replication log to a database table. The overhead for each user row change insert is very high.

[0096] 4. Implement custom file management.

[0097] According to an illustrative embodiment, a custom file is implemented so that replication logs can be read faster during recovery (role transition) and when catching up with a new replica. In addition, the custom file allows replication logs to be transported between replicas (e.g., for catching up), moving a follower from one shard server to another, and re-instantiating a new follower. The illustrative embodiment employs asynchronous I / O during writing and asynchronous I / O during prefetching and reading.

[0098] Figure 7 is a block diagram depicting log persistence optimization in a leader shard server according to an illustrative embodiment. The log persistence process group 710 in the leader shard receives the LCR 702 from the SGA. The LCR producer process 711 dequeues the LCR 702 and enqueues the LCR into the circular queue 713. The queue in the LCR 702 is a multi-writer and single-reader queue. The network senders 714A, 714B and the LCR persister process 715 are subscribers to the circular queue 713. The network senders 714A, 714B browse the circular queue 713 and distribute the replication logs to the followers.

[0099] The LCR persister process 715 maintains a set 716 of I / O buffers in the SGA. The log records are dequeued from the circular queue 713 and placed into the I / O buffers 716 as they appear in the on-disk log file 717. On the leader, since the LCR persister process 715 is also a subscriber to the circular queue 713, this persister process must ensure that the records are dequeued quickly to ensure that the queue 713 does not fill up. Therefore, the I / O is asynchronous; once the I / O buffer is full, an asynchronous I / O is issued. This I / O is reclaimed when the buffer is used again or if a commit record is encountered (when all outstanding I / O has been reclaimed).

[0100] The LCR persister process 715 performs the following operations:

[0101] 1. Dequeue one or more log records. A log record is a wrapper that contains an LCR with a term and a log index.

[0102] 2. Place the replicated log into one or more IO buffers. When the IO buffer is full, issue IO asynchronously.

[0103] 3. Recover the issued IO when the buffer is to be reused.

[0104] 4. At commit, recover all outstanding IO.

[0105] Log Persistence Optimization in Followers

[0106] Just like log persistence at the leader, each follower uses a replicated log file for durable persistence of LCRs. Figure 8 is a block diagram depicting log persistence optimization in a follower shard server according to an illustrative embodiment. The log persistence process group 820 in the follower shard receives LCRs, a commitIndex, a minimum persisted log index, and a minimum oldest log index from the leader. The network receiver 821 receives log records from the wire, places the LCRs into a circular queue 823 and an IO buffer 825 in the SGAS, and notifies the LCR persister process 826 if there is no free IO buffer.

[0107] The LCR persister process 826 persists the LCRs from the IO buffer 825 to the disk log file 828. The LCR persister process 826 monitors the IO buffer 825 and issues a block of the buffer as a synchronous IO when the utilization of the buffer reaches a predetermined threshold (e.g., 60%) or when a commit is received. The LCR persister process 826 also notifies the acknowledgement (ACK) sender 827 when the IO is complete.

[0108] The ACK sender 827 maintains a record of the last log index 829 that has been acknowledged. Whenever the LCR persister process 826 persists a commit index that is higher than the last acknowledged index, the ACK sender 827 sends this information to the leader.

[0109] The network receiver 821 sends the commitIndex 822 to the SQL application process 824. The SQL application process 824 reads LCRs from the circular queue 823, assembles transactions from the cross-cutting LCRs from different transactions, and performs DML operations on the database.

[0110] Cross Logging

[0111] A log record contains cross-cutting, uncommitted transactions in the replicated log. Figure 9Depicts a replication log with cross transactions according to an illustrative embodiment. Figure 9 The replication log shown in Figure 9 includes a Raft log, and each log in the Raft log has a log index. T1L1 means the first LCR in transaction T1. T1C means the commit LCR for transaction T1. T2R means the rollback LCR for transaction T2. As can be seen in the depicted example, T1L1 has a log index of 100, T2L1 has a log index of 101, T1C has a log index of 102, and T2R has a log index of 103. Thus, transactions T1 and T2 are crossed because transaction T2 has a log entry between the first LCR and the commit LCR of transaction T1.

[0112] Application Progress Tracking

[0113] There is a group of SQL application processes in the follower to replicate transactions. In one embodiment, the SQL application server consists of an LCR reader process, an application reader process, a coordinator process, and multiple applicator processes. The application LCR reader process dequeues the LCR, possibly reads the LCR from the persistence layer, and calculates the hash value for the relevant key columns. The application reader process assembles the LCR into a complete transaction, calculates the dependencies between transactions, and passes the transaction to the coordinator. The coordinator process assigns the transaction to an available applicator based on transaction dependencies and commit ordering. Each applicator process executes the entire transaction at the replica database and then requests another transaction. The applicators process independent transactions concurrently to achieve better throughput. To minimize application latency, the applicators start applying DML for the transaction before receiving a commit or rollback. This is called "eager application". However, even if an applicator sees the commit LCR for this transaction, it cannot commit the transaction unless the commit LCR has reached a consensus. Each transaction is applied exactly once.

[0114] Figure 10Depicts the application progress tracking in the replication log according to an exemplary embodiment. The SQL application maintains two key values regarding its progress: the Low Water Mark Log Index (LWMLI) and the oldest log index. All transactions with a commit index less than or equal to the LWMLI have been applied and committed. The oldest log index is the log index of the earliest log record that the application may need. For each replicated transaction, the SQL application inserts a row into a system table (e.g., appliedTxns) that contains the source transaction ID, the log index of the first DML, and the log index of the txCommit (which references the commit for the user transaction). During recovery, the recovery process or the application process can start reading from the oldest log index, skip the transactions that have already been applied based on appliedTxns, and complete the replication of any open transactions. As an optimization, multiple transactions can be batched and applied as one transaction in the follower for better performance.

[0115] Replication Unit Placement

[0116] There are multiple ways to place Replication Units (RUs) across shards in a cross-shard database, and these ways come with trade-offs.

[0117] Multi-ring Placement

[0118] During initial placement, in a balanced scenario, the RUs are placed in a ring where the size of the ring is equal to the replication factor. This placement helps with schema upgrades. In the case of barrier DDL, the quiescing workload can now be restricted to the ring of the shard servers instead of the entire shard database management system. For example, assume there are 6 shards, 12 RUs, and RF = 3, and each RU contains 60 blocks. Then the initial placement balances all the RUs across the 6 shards to create two rings. Figure 11 Depicts an example of the multi-ring placement of replication units according to an illustrative embodiment. In this example, RU1 and RU2 are replicated from shard 1 to shards 2 and 3, RU3 and RU4 are replicated from shard 2 to shards 1 and 3, and RU5 and RU6 are replicated from shard 3 to shards 1 and 2. Similarly, RU7 and RU8 are replicated from shard 4 to shards 5 and 6, RU9 and RU10 are replicated from shard 5 to shards 4 and 6, and RU11 and RU12 are replicated from shard 6 to shards 4 and 5.

[0119] All replicas in RU1 through RU6 are contained within shards 1, 2, and 3. A similar structure is observed for the second ring, where all replicas in RU7 through RU12 are contained within shards 4, 5, and 6. With this placement, any quiescing for the barrier DDL can be restricted to one ring without affecting the entire sharded database management system. Application schema upgrades can be performed in a way that only affects a small subset of the shard servers.

[0120] In the depicted example, RF = 3, and each ring is sized at three. If RF = 5, then each ring would be sized at five shard servers.

[0121] For incremental deployment, it is ideal to add the number of shard servers equal to RF simultaneously. In Figure 11 the example shown, if three shard servers are added, then an equal number of blocks will be pulled from all RUs to create six new RUs, which will be deployed on these three new shard servers, thus creating the shard servers for the third ring.

[0122] However, if the user adds one shard server at a time, then the following steps occur:

[0123] Step 1: For one ring and three shard servers, the topology is as shown in Table 1 below:

[0124] Fragment 1 (RU) Fragment 2 (RU) Fragment 3 (RU) Leader 1、2 3、4 5、6 Follower 3、4、5、6 1、2、5、6 1、2、3、4

[0125] Table 1

[0126] Each RU has 60 blocks. In this case, the barrier DDL must be synchronized across three shard servers.

[0127] Step 2: Add shard 4. Even though a shard server is added, since there are not enough shard servers to form two smaller rings, one ring remains. Blocks for the new RUs are pulled equally from the other RUs. Now each RU has 45 blocks. After adding the fourth shard server, for any barrier DDL propagation, synchronization across all four shard servers is required. The topology is as shown in Table 2 below:

[0128] Fragment 1 (RU) Fragment 2 (RU) Fragment 3 (RU) Fragment 4 (RU) Leader 1、2 3、4 5、6 7、8 Follower 5、6、7、8 1、2、7、8 1、2、3、4 3、4、5、6

[0129] Table 2

[0130] Step 3: Add shard 5. Again, for the same reason, no new smaller ring is created. The new shard server pulls blocks from all shard servers. Now each RU has 36 blocks. After adding the fifth shard server, for any barrier DDL propagation, synchronization across all five shard servers is required. The topology is as shown in Table 3 below:

[0131]

[0132] Table 3

[0133] Step 4: Add Shard 6. This time, the followers are moved apart to form two smaller rings. Now each RU has 30 chunks. After adding the sixth shard server, for any barrier DDL propagation, only synchronization across three shard servers is required. The topology is as follows in

[0134] shown in Table 4:

[0135]

[0136] Table 4

[0137] As seen in Table 4 above, Shard 1, Shard 2, and Shard 3 form one ring, and Shard 4, Shard 5, and Shard 6 form a second ring. Thus, a barrier DDL can be applied to Shard 1, Shard 2, and Shard 3 (only quiescing the workload for RU1 - 6), and then to Shard 4, Shard 5, and Shard 6 (quiescing the workload for RU7 - 12).

[0138] Step 5: Thereafter, when new shards are added, there is no longer a need to modify the first ring (Shard 1, Shard 2, Shard 3). Any new shard server will pull chunks from all shards, but the RU composition of the first ring will not change. When Shards 7 and 8 are added, the second ring (Shard 4, Shard 5, Shard 6) will expand to five shard servers, and by Shard 9, Shards 4 through 9 will split to form two smaller rings, thus forming a total of three smaller rings. This layout allows a barrier DDL to be applied to a single ring of shard servers without affecting the entire sharded database management system.

[0139] Single-ring Placement

[0140] Replica unit placement can also be done in the form of a single ring. The set of leader chunks on Shard 1 is replicated on Shards 2 and 3, the leader chunks on Shard 2 are replicated on Shards 3 and 4, and so on. For example, with four initial shards, RF = 3, DF = 2, and each RU having sixty chunks, there will be eight RUs, numbered from 1 to 8, for a total of 480 chunks. The set of leader chunks is marked "L", and the set of follower chunks is marked "F". The numbers indicate the number of chunks in the set of chunks. The chunk distribution is as shown in Table 5 below:

[0141]

[0142]

[0143] Table 5

[0144] In the RF = 3, DF = 2 configuration, adding a new shard server results in two leader RUs and four follower RUs on the new shard server. This can be achieved in two ways:

[0145] 1. Obtain an equal number of chunks from all shard servers to populate the new shard server. This approach maintains a balanced system at the cost of a lot of data movement between followers. Since all leaders on all shard servers are affected by this approach, all followers on all shard servers are also affected - that is, if there are n shard servers in the system, then the cost of adding the (n + 1)th shard depends on n. For example, consider 4 shards, each with 2 leader RUs and 4 follower RUs. When pulling chunks from each RU, the 8 leader RUs shrink to create the 9th and 10th RUs on the 5th shard. The followers for each affected RU must also shrink to match that leader - so adding a shard server ultimately touches every RU on every shard server.

[0146] 2. When populating the new shard server, limit the "donor" shard servers to a small fixed number. This approach does not attempt to achieve a balanced system when adding a shard server, but this trade-off allows the system to reach a stable state more quickly after a fixed amount of data movement. In this approach, adding a shard server to the ring involves pulling chunks from the four closest shard servers (two on each side). This is done as follows:

[0147] a. Determine the insertion point: Find 4 adjacent shard servers that together have the highest number of chunks compared to any other adjacent group of 4.

[0148] The new shard server should be inserted in the middle of these 4 donor shard servers.

[0149] As a special case, when changing from 3 to 4, insert anywhere and pull from all 3 shard servers.

[0150] b. Split the set of leader chunks on the 4 donor shard servers (for the DF = 2 case, 8 leaders are split). The split should be done in such a way that all 5 shard servers (4 donor shards and the new shard) retain approximately the same number of chunks for each set of leader chunks.

[0151] 3. Split the corresponding set of follower chunks in the same way (for RF = 3, 16 followers are split).

[0152] 4. Move 2 leaders from each donor shard server to the new shard server (8 leaders moved).

[0153] 5. On the new shard server, merge in groups of 2 RUs so that there are 4 leader RUs (down from 8). Then merge the corresponding follower RUs. This can be done without any data movement.

[0154] 6. At this point, all the leader block sets on the new shard server are in the correct position except for 4 follower block sets. Move these 4 follower block sets for the final round of merging.

[0155] 7. Merge the RUs on the new shard server so that 2 RUs remain, with each RU having 4 block sets. Merge the corresponding follower RUs.

[0156] 8. Move the followers to the new shard server (4 followers moved).

[0157] 9. Summary: 8 leader splits, 16 follower splits, 8 leader moves, 4 leader merges, 8 follower merges, 4 follower merges, 2 leader merges, 4 follower merges, 4 follower moves.

[0158] The single-ring structure has a drawback: Barrier DDL causes the entire sharded database management system to become static. However, when adding a new shard server, it is possible to limit the number of RU splits, moves, and merges to a fixed amount, thereby reducing the data moved during incremental deployment and making incremental deployment a more deterministic operation. Removing a shard server (e.g., due to failure or downsizing) can be done similarly in two options.

[0159] Resharding

[0160] Split and Merge Replication Units

[0161] To add a new shard server or remove a shard server (e.g., shard failure), it may be necessary to move RUs from one shard server to another, or split and / or merge RUs. According to an illustrative embodiment, the shard coordinator ensures that in-progress transactions during those operations are not affected. There are several methods for splitting or merging RUs. Figure 12 FIG. is a data flow diagram illustrating an example replication unit split according to an illustrative embodiment. For simplicity, assume no leader change during RU split. Transactions T1, T2, T3, T4 are executed on a given shard server (leader or follower) within the sharded database management system. The RU state 1201 in the shard starts with RU1 = {C1, C2}, where C is a block. That is, RU1 includes two blocks, C1 and C2.

[0162] There is in-progress DML 1211 from transaction T1 to insert row 1 into C2 and in-progress DML 1231 from transaction T3 to insert row 3 into C2. Then, the fragmentation coordinator 1250 initiates a split operation 1251 to split RU1 into RU1 and RU2. As a result, the RU state 1202 in the fragmentation servers becomes RU1 = {C1}, RU2 = {C2}. That is, RU1 is split into RU1 containing block C1 and RU2 containing block C2.

[0163] The fragmentation coordinator 1250 creates a new RU ( Figure 12 RU2 in it), which has the same set of fragmentation servers for replicas of this RU. The leader of the old RU is the leader of the new RU. The fragmentation coordinator 1250 creates a new RU and enqueues a split RU begin marker into both the old RU and the new RU. The fragmentation coordinator 1250 associates the set of blocks ( Figure 12 C2 in it) with the new RU. The fragmentation coordinator 1250 establishes a replication process for the new RU and suspends the SQL apply process for the new RU.

[0164] The fragmentation coordinator 1250 waits for the transactions running concurrently with the RU split to complete. In the example shown in Figure 12 , when the split is initiated, transactions T1 and T3 are in progress. Transaction T3 depends on transaction T2, and transaction T2 completes during the split process. As a defensive measure, the fragmentation coordinator 1250 enqueues the old values of all scalar columns for update and deletion during the RU split process. The fragmentation coordinator 1250 performs the same operation for the RU merge process.

[0165] The remaining changes of the in-progress transactions (e.g., T1 and T3) after the RU split start will be enqueued to the existing RU. Thus, DML 1212 and commit 1213 for T1 and DML 1232 and commit 1233 are enqueued to the existing RU1. Transactions started after the split (e.g., T2 and T4) will be enqueued to both the new RU and the old RU. Thus, DML 1221 and commit for T2 and DML 1241 and commit for T4 are enqueued to both RU1 and RU2. After all in-progress transactions are completed, the fragmentation coordinator 1250 enqueues a split RU end marker into both the old RU and the new RU, thus performing an end split RU 1252.

[0166] If a transaction (e.g., T1, T3) starts before the split command is initiated, then the SQL application for the old RU executes this transaction. These transactions are considered in progress. For transactions that start after the RU split operation is initiated (e.g., T2, T4), if a transaction (e.g., T2) ends before the split command is completed, then the SQL application for the old RU executes this transaction, and the SQL application for the new RU ignores this transaction. If a transaction ends after the RU split operation is completed, then the SQL application for the new RU executes this transaction, and the SQL application for the old RU ignores this transaction.

[0167] When the SQL application in the follower for the old RU receives the split RU end marker enqueued above, it suspends the SQL application process for the new RU. Long-running transactions will cause the SQL application process to be suspended for a longer time.

[0168] After the RU split process, the shard coordinator enables periodic leadership changes based on a consensus protocol.

[0169] As referenced above Figure 12 The RU split process described above can be generalized to split one RU into multiple RUs, not just two.

[0170] Add New Fragment

[0171] When adding a new shard server, if the user requests block rebalancing, then new RUs can be created on the new shard via the relocate block command and move RU command. There are many rebalancing options, such as the following:

[0172] 1. Fill the new shard server with an average number of blocks per shard server in each replication group. This will result in a balanced system at the cost of a one-time large data movement among the followers.

[0173] 2. When filling the new shard server, limit the "donor" shards to a small fixed number. This is to minimize data movement. However, this will result in a more unbalanced system.

[0174] Move Replication Unit to New Follower

[0175] Moving an RU from one follower to another can be performed as follows:

[0176] 1. Stop the SQL application at the old follower and record the application metadata (low watermark and the oldest log index).

[0177] 2. Execute the following two tasks in parallel:

[0178] a. Instantiate a new follower. Copy relevant data (application metadata, user data) using the old follower, e.g., TTS, export / import. Instantiate user data and create a new application in the new follower with the low watermark and the oldest log index.

[0179] b. Copy the replication log at the old follower (starting from the oldest log index) and transport it to the new follower.

[0180] 3. Make the new follower a non-voting member. Start applying the replicated log starting from the old log index.

[0181] 4. When the new follower has almost caught up, make the new follower a voting member.

[0182] 5. Remove the old follower from the replication unit.

[0183] Restore Replication Unit

[0184] Reasons for restoring an RU include creating additional replicas on a new shard server to increase the replication factor, replacing the replication unit on a "failed" shard server with a replication unit on another shard server, rebuilding the replication unit on a shard server to recover from data / log divergence, and rebuilding the replication unit after a long downtime of an outdated replication unit to allow it to catch up.

[0185] The high-level steps for restoring an RU are as follows:

[0186] 1. If it exists, then remove the RU from the target shard server.

[0187] 2. Copy the RU.

[0188] 3. Restore the RU on the source database to R / W state.

[0189] 4. Update the peers and the directory (only needed when adding / replacing).

[0190] 5. Start the RU process group on the target shard server.

[0191] Recovery

[0192] According to an illustrative embodiment, if a shard server fails and there is a new standby shard, then select the target shard (usually hosting a follower), and copy all the blocks and the replication log for the relevant replication unit to the new shard. If there is no new standby shard server and there is available capacity in the existing shard servers, then redistribute all the blocks for the relevant replication unit to those shards, thus maintaining the replication factor. For simplicity, new transactions are not allowed to span epochs; however, commit and rollback records can be generated in the new leader with a different epoch.

[0193] Consistency of Leader Transaction Commit and Replication Log Synchronous Replication

[0194] To ensure consistency between leader transaction commits and replicated log synchronous replication, two embodiments are described below.

[0195] In one embodiment, the leader prepares a local transaction for commit and first marks the transaction as "pending", then sends a commit LCR to the followers of the leader. After receiving consensus on the commit LCR, the leader commits the local transaction. After a leader shard server failure, if the failed leader becomes a follower and there is consensus on this commit LCR based on the replicated log, then the "pending" state of the local transaction in the failed leader can be committed. Otherwise, the transaction must be rolled back in the failed leader. Each follower receives the commit LCR but does not perform a commit on the replicas of the follower's replication unit until the follower receives a commit log index from the leader that is equal to or greater than the log index of the commit LCR, indicating that the commit LCR has been persisted by other followers.

[0196] In an alternative embodiment, before committing the local transaction, the leader sends a pre-commit LCR to the followers of the leader. After receiving consensus on the pre-commit LCR, the leader commits the local transaction and sends a post-commit LCR to the followers of the leader. After a leader shard server failure and the resulting leader becomes a follower, if there is already consensus on the pre-commit LCR based on the replicated log, then the local transaction can be committed in the failed leader. If the local transaction in the failed leader is rolled back, then the transaction can be replayed to the failed leader based on the replicated log. If the transaction fails after sending the pre-commit LCR (e.g., the user session crashes), if the shard server is still the leader, then the process monitor in the database generates a rollback record and sends the rollback record to its followers.

[0197] Transaction Recovery

[0198] The LCR sequence for transaction T1 is as follows:

[0199]

[0200] Where T D is a DML LCR, T F is the first LCR of the transaction, this T F can be a DML LCR, T pre is a pre-commit LCR (awaiting consensus), and T post is a post-commit LCR. At any point in time before the post-commit LCR, the transaction can be rolled back, resulting in T RBK which TRBK is the rollback LCR. The LCR before submission is intended to allow the leader to receive consensus before locally committing the transaction.

[0201] The leader does not wait for consensus on the post-submission LCR before returning to the client. The commitment of the transaction at the leader occurs between the pre-submission LCR and the post-submission LCR. On each follower, the commitment occurs after receiving the post-submission LCR. The follower does not commit the transaction to the copy of the replica unit of that follower after receiving the pre-submission LCR because the follower does not know if there is already a consensus for the pre-submission LCR. In fact, the follower may receive a rollback LCR after the pre-submission LCR. If the follower receives the post-submission LCR, then the follower knows that there is already a consensus for the pre-submission LCR and the leader has committed the transaction.

[0202] If the leader fails before sending the post-submission LCR and an election for leadership occurs, then the new leader will be the follower with the longest replicated log. Several scenarios can occur. One scenario is that the new leader has an LCR for the transaction that does not include the pre-submission LCR or the post-submission LCR. In this case, the new leader does not know if there are other DML LCRs for the changes made by the old leader and confirmed to the user / client. Therefore, the new leader will roll back the transaction and issue a rollback LCR.

[0203] Another scenario is that the new leader has an LCR for the transaction that includes the pre-submission LCR and the post-submission LCR. This transaction must be committed. Therefore, the new leader commits the transaction.

[0204] Another scenario is that the new leader has a pre-commit LCR for a transaction, including the pre-commit LCR but not the post-commit LCR. The new leader cannot assume that the pre-commit LCR has received consensus. According to the illustrative embodiment, the new leader issues a lead-sync LCR, which is a special LCR sent at the start of a term to synchronize the replicated logs of the followers. The new leader waits for consensus on the lead-sync LCR, which indicates that the number of followers in consensus have persisted all LCRs up to the lead-sync LCR. In other words, a follower acknowledges the lead-sync LCR only when the follower has persisted all LCRs up to the lead-sync LCR. Each LCR has a log index; thus, each follower will know whether all LCRs up to and including the lead-sync LCR have been persisted. A follower cannot acknowledge that the lead-sync LCR has been persisted without acknowledging that every LCR with a log index less than the lead-sync LCR has been persisted. This ensures that for transaction T1, all followers will have the pre-commit LCR and all LCRs before that pre-commit LCR. Then, the new leader can commit transaction T1 to the replicas of the leader's replication unit because the new leader now knows that there is consensus on the pre-commit LCR. Then, the new leader will send a post-commit LCR to the leader's followers.

[0205] Consider the following sequence of LCRs in the replicated log of the new leader:

[0206]

[0207] If the new leader receives consensus on the lead-sync LCR with a log index of 501, then the new leader knows that the number of followers in consensus have persisted LCRs with log indices up to 500. The new leader can commit transactions T1 and T2. The new leader will generate the following LCRs: and

[0208] Figure 13FIG. is a flowchart illustrating the operation of a new leader shard server taking over a replication unit when a commit initiated in a replication log exists. The operation begins (block 1301) when a commit for a transaction exists in the replication log. The leader shard server determines whether there is a consensus for the transaction commit (block 1301). If there is a post-commit LCR in the replication log for the transaction, then the leader shard server determines that there is a consensus for the transaction. If there is no consensus for the transaction (block 1301: NO), then the leader shard server performs a lead-sync operation by generating a lead-sync LCR and propagating the lead-sync LCR to the followers of the leader (block 1302).

[0209] In response to receiving the lead-sync LCR, the follower will request each LCR in the replication log up to the log index of the lead-sync LCR. If the follower successfully receives each LCR in the replication log up to the log index of the lead-sync LCR and persists these LCRs into the follower's replication log, then the follower returns an acknowledgment specifying the log index of the lead-sync LCR. Thereby, the leader shard server knows that the follower has acknowledged each LCR in the replication log up to and including the lead-sync LCR.

[0210] Then, the leader shard server determines whether it has received a consensus for the lead-sync LCR (block 1303). If a consensus number of followers have acknowledged the LCRs with the log index up to and including the lead-sync LCR, thereby including the pre-commit LCR for the transaction, then the leader shard server determines that there is a consensus for the transaction. If there is a consensus for the lead-sync LCR (block 1301: YES or block 1303: YES), then the leader shard server completes the commit operation (block 1304), and the operation ends (block 1305). In one embodiment, completing the commit operation includes generating a post-commit LCR and propagating the post-commit LCR to its followers.

[0211] If a consensus for the lead-sync LCR is not received, then the system does not initiate recovery and the leader does not generate any log records. The system only initiates recovery after a consensus on the lead-sync LCR is reached, effectively ensuring a consensus on each LCR before the lead-sync LCR. If there is no consensus for the lead-sync LCR (block 1303: NO), then the leader shard server rolls back the transaction (block 1306). Thereafter, the operation ends (block 1305).

[0212] Replication Log Recovery

[0213] Replication log recovery is the process of making the state of the replication log match the state of the database transactions by re - creating the missing entries in the replication log and replaying or rolling back the transactions.

[0214] According to an illustrative embodiment, the shard server determines a recovery window that defines the transactions that must be recovered. The recovery window starts from the earliest transaction that has a first LCR in the replication log but does not have a post - commit LCR. Consider the following LCR sequence in the follower's replication log:

[0215]

[0216] Transaction T4 has reached post - commit in the replication log; thus, T4 does not need to be recovered because the post - commit LCR for T4 indicates that the commit of T4 has received consensus. The earliest transaction that has a first LCR but does not have a post - commit LCR is transaction T1 at log index 100. Thus, the follower must recover the replication log starting from log index 100 up to log index 1001 (lead - sync LCR). The recovery window is shown in bold. If a shard fails after receiving consensus for a pre - commit LCR but before generating a post - commit LCR, then there is not enough information in the replication log to infer the fate of the transactions in the failed shard. The method of the illustrative embodiment is to let the database resolve this issue. If the transaction has been rolled back, then replay the transaction; otherwise, do not replay the transaction against the database.

[0217] Transaction T1 requests publish / replay / commit. Transaction T2 does not need to be processed because transaction T2 is already local. Transaction T3 requests rollback and rollback LCR because T3 has not reached the pre - commit LCR in the replication log. Transaction T4 does not need to be processed because the first LCR of T4 is outside the recovery window. Transaction T5 does not need to be processed because T5 has been rolled back. Thus, the follower can ignore transactions T2, T4, and T5 during recovery.

[0218] If the database fails in the middle of a commit, then at database restart, an election for leadership will be held. If the old leader becomes a follower because at least one follower has a pre - commit LCR, then there is a question about what happened to the transactions in the old leader. Since the database failed during the commit, the user will receive an error. According to an embodiment, the transaction will be replayed.

[0219] Figure 14FIG. is a flow chart illustrating the operation of a shard server performing replicated log recovery according to an illustrative embodiment. The operation begins at the shard server when recovering from a failure (block 1400). The shard server identifies the first transaction in the replicated log that has a first LCR but no post-commit LCR (block 1401). The shard server defines a recovery window as described above (block 1402). The shard server then identifies the set of transactions to be recovered (block 1403). As described above, the shard server identifies transactions that have an LCR within the recovery window that can be ignored during recovery. The shard server then performs recovery on the identified set of transactions to be recovered (block 1404). Thereafter, the operation ends (block 1405).

[0220] DDL Execution Without Global Barrier

[0221] In the consensus protocol-based replication method for a sharded database of the illustrative embodiment, data definition language (DDL) operations are performed outside of the consensus protocol, i.e., DDL is not part of the DML replication stream. This is in contrast to typical database replication.

[0222] For a given table family in a sharded database management system (SDBMS), each shard contains the same schema definition. DDL operations in the SDBMS are issued on the global data service (GDS) catalog, which propagates the DDL to each shard. The consensus protocol-based replication method of the illustrative embodiment only replicates DML asynchronously. Thus, there can be in-progress transactions in the replication pipeline that can: a) lag the schema definition in the follower when DDL is first executed for a replication unit in the follower; or b) advance the schema definition in the follower when DDL is first executed for a replication unit in the leader.

[0223] DDL is classified as barrier DDL or non-barrier DDL. Barrier DDL does not allow in-progress DML, e.g., ALTER TABLE RENAME COLUMN. The SDB is quiesced, the DDL is executed synchronously for all available shards, and DML is only allowed after the DDL execution. Non-barrier DDL allows in-progress DML, e.g., CREATE SYNONYM. There are two types of non-barrier DDL: DDL that does not require any synchronization with in-progress DML (e.g., CREATE SYNONYM), and DDL that requires synchronization of delayed user DML and DDL execution in the replication stream. Some in-progress DML is still allowed, e.g., ALTER TABLE ADD COLUMN with a default value.

[0224] To handle non-barrier DDLs that require synchronization, the start and end of the non-barrier DDL must be demarcated so that SQL applications and inlined triggers enter a special mode. In the easier case, if the DDL has been executed at the leader but not at the followers, then the SQL application attempts to replicate the DML as safely as possible. For example, for a drop column DDL where the dropped column has a default value, the SQL application can ignore this column. Alternatively, the leader filters out this column. The SQL application stops when it encounters an error. The SQL application can be restarted when the DDL is executed at the followers. If the DML can stop the SQL application (e.g., alter table add column with a default value and insert a non-default value into this column), then the leader can raise an error.

[0225] In the more complex case, the DDL has been executed at the followers but not at the leader. In this resilient handling mode, the SQL application continues to apply. For example, for a drop column, the SQL application can ignore this column in the LCR. For adding a column with a default value, since the leader does not yet have this new column, the SQL application will not include this column in all replicated DML.

[0226] When a shard fails during DDL execution, it is not necessary to re-execute the DDL when the shard becomes available. The DDL execution must be coordinated with the replication of in-progress transactions in the replication log.

[0227] Disaster Recovery

[0228] Users can place replicas in different geographical regions. For example, two replicas are placed in one region and another replica is placed in a different region. This allows the two local replicas to act as the majority for stable workloads and allows the remote replica to act as a backup for disaster recovery. When a disaster occurs, the user may need to reconfigure the consensus protocol-based replication in the remote region.

[0229] Cross-Replication Unit Transaction Support

[0230] Any transaction that spans blocks will be initiated from the shard coordinator, which acts as a distributed transaction coordinator (DTC). The DTC executes the typical two-phase commit protocol among the leaders of all relevant replica units. The leader of the relevant replica unit must propagate sufficient information about this distributed transaction (e.g., global transaction ID, local transaction ID) to the followers of that leader and obtain consensus from the followers of that leader in the prepare-to-commit phase before informing the DTC that the leader is ready to commit. The DTC can only commit after receiving the prepare-to-commit messages from all relevant RU leaders. The DTC is not aware of the existence of followers. If there is a failover in a replica unit and the new leader has a prepare-to-commit record but does not have the final commit or rollback for the transaction, then the new leader of that replica unit must contact the DTC to find out the fate of such a distributed transaction. Using the two-phase commit protocol, distributed transactions introduce a greater latency in transaction response time: one more network round-trip is required. Note that the replication of distributed transaction branches is asynchronous and not part of the original distributed transaction. In a typical sharded deployment, it is assumed that most transactions are within the same block, thus avoiding the two-phase commit protocol.

[0231] DBMS Overview

[0232] A database management system (DBMS) manages databases. The DBMS can include one or more database servers. A database includes database data and a database dictionary stored on a persistent storage mechanism (such as a collection of hard disks). The database data can be stored in one or more collections of records. The data within each record is organized into one or more attributes. In a relational DBMS, the collections are called tables (or data frames), the records are called tuples, and the attributes are called columns. In a document DBMS (“DOCS”), the collection of records is a collection of documents, and each document can be a data object marked up with a hierarchical markup language, such as a JSON object or an XML document. The attributes are called JSON fields or XML elements. A relational DBMS can also store hierarchically marked data objects; however, the hierarchically marked data objects are contained within the attributes of a record, such as an attribute of JSON type.

[0233] Users interact with the database servers of the DBMS by submitting commands to the database servers, which cause the database servers to perform operations on the data stored in the database. The users can be one or more applications running on a client computer that interact with the database servers. Multiple users can also be collectively referred to as users in this document.

[0234] A database command can be in the form of a database statement conforming to a database language. The database language used to express database commands is Structured Query Language (SQL). There are many different versions of SQL; some versions are standard, some are proprietary, and there are also various extensions. Data Definition Language ("DDL") commands are issued to a database server to create or configure data objects, called database objects in this document, such as tables, views, or complex data types. SQL / XML is a common extension of SQL that is used when manipulating XML data in an object-relational database. Another database language used to express database commands is Spark TM SQL, which uses a syntax based on function or method calls.

[0235] In DOCS, a database command can be in the form of a function or object method call that invokes CRUD (Create, Read, Update, Delete) operations. An example of an API for such function and method calls is MQL (MondoDB TM Query Language). In DOCS, database objects include fields defined by a JSON schema for a collection, a collection of documents, documents, or views. A view can be created by calling a function provided by the DBMS for creating a view in the database.

[0236] Changes to the database in a DBMS are made through transaction processing. A database transaction is a collection of operations that change the database data. In a DBMS, a database transaction is initiated in response to a database command requesting a change (such as a DML command requesting an update, insert record, or delete record, or a CRUD object method call requesting the creation, update, or deletion of a document). DML commands and DDL specify changes to the data, such as INSERT and UPDATE statements. A DML statement or command does not refer to a statement or command that only queries the database data. Committing a transaction means making the changes of the transaction permanent.

[0237] In transaction processing, all changes to a transaction are made atomically. When a transaction is committed, either all changes are committed or the transaction is rolled back. These changes are recorded in a change log, which can include redo records and undo records (undo log). Redo records can be used to reapply the changes made to data blocks. Undo records are used to reverse or undo the changes made to data blocks by a transaction.

[0238] Examples of such transaction metadata include a change log that records the changes made to the database data by a transaction. Another example of transaction metadata is embedded transaction metadata stored in the database data that describes the transaction that changed the database data.

[0239] Undo records are used to provide transaction consistency by performing operations referred to herein as consistency operations. Each undo record is associated with a logical time. An example of logical time is the System Change Number (SCN). For example, the Lamporting mechanism can be used to maintain the SCN. For data blocks that are read to compute a database command, the DBMS applies the required undo records to a copy of the data block so that the copy is in a state consistent with the snapshot time of the query. The DBMS determines which undo records to apply to the data block based on the corresponding logical time associated with the undo records.

[0240] In a distributed transaction, multiple DBMSs use the two-phase commit method to commit the distributed transaction. Each DBMS executes a local transaction in a branch transaction of the distributed transaction. One DBMS (the coordinating DBMS) is responsible for coordinating the commit of the transaction on one or more other database systems. The other DBMSs are referred to herein as participating DBMSs.

[0241] The two-phase commit involves two phases: the prepare-to-commit phase and the commit phase. In the prepare-to-commit phase, the branch transaction is prepared in each participating database system. When a branch transaction is prepared on a DBMS, the database is in a "ready state" such that the database can guarantee that the modifications made to the database data as part of the branch transaction can be committed. This guarantee may require the persistent storage of change records for the branch transaction. The participating DBMS confirms when it has completed the prepare-to-commit phase and has entered the ready state for the corresponding branch transaction.

[0242] In the commit phase, the coordinating database system commits the transaction on the coordinating database system and on the participating database systems. Specifically, the coordinating database system sends a message to the participants requesting that the participants commit the modifications to the data on the participating database systems specified by the transaction. Then, the participating database systems and the coordinating database system commit the transaction.

[0243] On the other hand, if a participating database system cannot prepare or the coordinating database system cannot commit, then at least one of the database systems cannot make the changes specified by the transaction. In this case, all modifications at each of the participants and the coordinating database system are undone, thus resetting each database system to the state before the database changes.

[0244] The client can issue a series of requests to the DBMS by establishing a database session, such as requests for executing queries. A database session includes a specific connection established for the client to the database server through which the client can issue a series of requests. The database session process executes within the database session and processes the requests issued by the client through the database session. The database session can generate an execution plan for a query issued by the database session client and marshal subordinate processes for the execution of that execution plan.

[0245] The database server can maintain session state data about the database session. The session state data reflects the current state of the session and can include the identity of the user for whom the session was established, the services used by the user, instances of object types, language and character set data, statistical data about resource usage for the session, values of temporary variables generated by processes of the execution software within the session, storage for pointers, variables, and other information.

[0246] The database server includes multiple database processes. The database processes run under the control of the database server (i.e., can be created or terminated by the database server) and perform various database server functions. The database processes include processes that run within a database session established for a client.

[0247] A database process is a unit of execution. A database process can be a computer system process or thread or a user-defined execution context, such as a user thread or fiber. A database process can also include a "database server system" process that provides services and / or performs functions on behalf of the entire database server. Such database server system processes include listeners, garbage collectors, log writers, and recovery processes.

[0248] A multi-node database management system consists of interconnected computing nodes ("nodes"), each running a database server that shares access to the same database. Typically, the nodes are interconnected via a network and share access to a shared storage device to varying degrees, e.g., shared access to a set of disk drives and the data blocks stored thereon. The nodes in a multi-node database system can be in the form of a group of computers (e.g., workstations, personal computers) interconnected via a network. Alternatively, the nodes can be nodes of a grid consisting of server blades interconnected with other server blades on a rack.

[0249] Each node in a multi-node database system hosts a database server. A server such as a database server is a combined allocation of integrated software components and computing resources (such as memory, the node, and the processes on the node for executing the integrated software components), the combination of software and computing resources dedicated to performing specific functions on behalf of one or more clients.

[0250] Resources from multiple nodes in a multi-node database system can be allocated to run the software of a specific database server. Each combination of the software and the allocation of resources from the nodes is a server referred to as a "server instance" or "instance" herein. A database server can include multiple database instances, and some or all of these database instances run on separate computers (including separate server blades).

[0251] A database dictionary can include multiple data structures that store database metadata. For example, a database dictionary can include multiple files and tables. Portions of the data structures can be cached in the main memory of the database server.

[0252] When a database object is said to be defined by a database dictionary, the database dictionary contains metadata that defines the nature of the database object. For example, the metadata in the database dictionary that defines a database table can specify the attribute names and the data types of the attributes, as well as one or more files or portions thereof for storing data for the table. The metadata in the database dictionary that defines a procedure can specify the name of the procedure, the arguments of the procedure, and the return data type and the data types of the arguments, and can include the source code and its compiled version.

[0253] A database object can be defined by a database dictionary, but the metadata in the database dictionary itself may only partially specify the nature of the database object. Other natures can be defined by data structures that are not considered part of the database dictionary. For example, a user-defined function implemented in a JAVA class can be partially defined by the database dictionary by specifying the name of the user-defined function and by specifying references to the file containing the source code of the Java class (i.e., the.java file) and the compiled version of the class (i.e., the.class file).

[0254] A database object can have an attribute that is a primary key. The primary key contains a primary key value. The primary key value uniquely identifies a record among the records in the database object. For example, a database table can include a column that is a primary key. Each row in the database table holds a primary key value that uniquely identifies the row among the rows in the database table.

[0255] A database object can have an attribute that is a foreign key to the primary key of another database object. The foreign key of the primary key contains the primary key value of the primary key. Thus, the foreign key value in the foreign key uniquely identifies a record in the corresponding database object of the primary key.

[0256] A foreign key constraint based on the primary key can be defined for the foreign key. The DBMS ensures that any value in the foreign key exists in the primary key. It is not necessary to define a foreign key for the foreign key. Instead, a foreign key relationship can be defined for the foreign key. The application that populates the foreign key is configured to ensure that the foreign key values in the foreign key exist in the corresponding primary key. Even when no foreign key relationship is defined for the foreign key, the application can maintain the foreign key in this way.

[0257] Hardware Overview

[0258] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the techniques, or can include digital electronic devices (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques), or can include one or more general-purpose hardware processors programmed to execute the techniques according to program instructions in firmware, memory, other storage devices, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to implement the techniques. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the techniques.

[0259] For example, Figure 15 is a block diagram illustrating a computer system 1500 on which embodiments of the present invention can be implemented. The computer system 1500 includes a bus 1502 or other communication mechanism for conveying information, and a hardware processor 1504 coupled to the bus 1502 for processing information. The hardware processor 1504 can be, for example, a general-purpose microprocessor.

[0260] The computer system 1500 also includes a main memory 1506 (such as random access memory (RAM) or other dynamic storage device) coupled to the bus 1502 for storing instructions and information to be executed by the processor 1504. The main memory 1506 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1504. When such instructions are stored in a non-transitory storage medium accessible by the processor 1504, the computer system 1500 becomes a special-purpose machine customized to perform the operations specified in the instructions.

[0261] The computer system 1500 also includes a read-only memory (ROM) 1508 or other static storage device coupled to the bus 1502 for storing static information and instructions for the processor 1504. A storage device 1510 (such as a magnetic disk, an optical disk, or a solid-state drive) is provided and coupled to the bus 1502 for storing information and instructions.

[0262] The computer system 1500 can be coupled via a bus 1502 to a display 1512, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 1514, including alphanumeric keys and other keys, is coupled to the bus 1502 for transmitting information and command selections to the processor 1504. Another type of user input device is a cursor control 1516, such as a mouse, trackball, or cursor direction keys, for transmitting direction information and command selections to the processor 1504 and for controlling cursor movement on the display 1512. Such input devices typically have two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), which allows the device to specify a position in a plane.

[0263] The computer system 1500 can implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that in combination with the computer system causes or programs the computer system 1500 to be a special-purpose machine. According to one embodiment, the computer system 1500 performs the techniques herein in response to the processor 1504 executing one or more sequences of one or more instructions contained in the main memory 1506. These instructions can be read into the main memory 1506 from another storage medium, such as the storage device 1510. Executing the instruction sequence contained in the main memory 1506 causes the processor 1504 to perform the processing steps described herein. In an alternative embodiment, hardwired circuitry may be used in place of or in combination with software instructions.

[0264] As used herein, the term "storage medium" refers to any non-transitory medium that stores instructions and / or data that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical discs, magnetic disks, or solid state drives, such as the storage device 1510. Volatile media includes dynamic memory, such as the main memory 1506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with hole patterns, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0265] Storage media is different from transmission media but can be used in combination with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and optical fibers, including the wires that comprise the bus 1502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0266] Various forms of media can participate in carrying one or more sequences of one or more instructions to the processor 1504 for execution. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into the dynamic memory of that computer and send the instructions via a modem over a telephone line. A modem local to the computer system 1500 can receive the data on the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector can receive the data carried in the infrared signal, and appropriate circuitry can place the data on the bus 1502. The bus 1502 passes the data to the main memory 1506, from which the processor 1504 retrieves and executes the instructions. The instructions received by the main memory 1506 can optionally be stored on the storage device 1510 before or after being executed by the processor 1504.

[0267] The computer system 1500 also includes a communication interface 1518 coupled to the bus 1502. The communication interface 1518 provides two-way data communication coupled to a network link 1520 that connects to a local network 1522. For example, the communication interface 1518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 1518 can be a local area network (LAN) card that provides a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 1518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0268] The network link 1520 typically provides data communication through one or more networks to other data devices. For example, the network link 1520 can provide a connection through the local network 1522 to a main computer 1524 or to a data device operated by an Internet service provider (ISP) 1526. The ISP 1526 in turn provides data communication services through the global packet data communication network (now commonly referred to as the “Internet” 1528). Both the local network 1522 and the Internet 1528 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through the various networks and signals on the network link 1520 and through the communication interface 1518 that carry digital data to and from the computer system 1500 are example forms of transmission media.

[0269] The computer system 1500 can send messages and receive data, including program code, via one or more networks, network link 1520, and communication interface 1518. In an Internet example, the server 1530 can transmit the requested code for an application program via the Internet 1528, ISP 1526, local network 1522, and communication interface 1518.

[0270] The received code can be executed by the processor 1504 when received, and / or stored in the storage device 1510 or other non-volatile memory for later execution.

[0271] Software Overview

[0272] Figure 16 is a block diagram of a basic software system 1600 that can be used to control the operation of a computer system 1600. The software system 1600 and its components, including their connections, relationships, and functions, are merely exemplary and are not meant to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0273] The software system 1600 is provided to bootstrap the operation of the computer system 1500. The software system 1600, which can be stored in the system memory (RAM) 1506 and the fixed storage device (e.g., hard disk or flash memory) 1510, includes a kernel or operating system (OS) 1610.

[0274] The OS 1610 manages the low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more applications represented as 1602A, 1602B, 1602C... 1602N can be "loaded" (e.g., transferred from the fixed storage device 1510 to the memory 1506) for execution by the system 1600. Applications or other software intended to be used on the computer system 1500 can also be stored as a set of downloadable computer-executable instructions, e.g., for downloading and installation from an Internet location (e.g., a web server, app store, or other online service).

[0275] The software system 1600 includes a graphical user interface (GUI) 1615 for receiving user commands and data in a graphical (e.g., "point-and-click" or "touch gesture") manner. Further, these inputs can be operated on by the system 1600 according to instructions from the operating system 1610 and / or one or more applications 1602. The GUI 1615 is also used to display the operation results from the OS 1610 and one or more applications 1602, based on which the user can provide additional inputs or terminate the session (e.g., log off).

[0276] The OS 1610 can execute directly on the bare hardware 1620 (e.g., one or more processors 1504) of the computer system 1500. Alternatively, a hypervisor or virtual machine monitor (VMM) 1630 can be inserted between the bare hardware 1620 and the OS 1610. In this configuration, the VMM 1630 acts as a software "buffer" or virtualization layer between the bare hardware 1620 of the computer system 1500 and the OS 1610.

[0277] The VMM 1630 instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine includes a "guest" operating system (such as the OS 1610), and one or more applications (such as one or more applications 1602) designed to execute on the guest operating system. The VMM 1630 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0278] In some cases, the VMM 1630 can allow the guest operating system to run as if it were running directly on the bare hardware 1620 of the computer system 1500. In these instances, the same version of the guest operating system configured to execute directly on the bare hardware 1620 can also execute on the VMM 1630 without modification or reconfiguration. In other words, the VMM 1630 can provide full hardware and CPU virtualization to the guest operating system in some cases.

[0279] In other cases, the guest operating system can be specifically designed or configured to execute on the VMM 1630 for increased efficiency. In these cases, the guest operating system "is aware" that the system is executing on the virtual machine monitor system. In other words, the VMM 1630 can provide paravirtualization to the guest operating system in certain cases.

[0280] A computer system process includes the allocation of hardware processor time, as well as the allocation of memory (physical and / or virtual) for storing instructions executed by the hardware processor, for storing data generated by the execution of instructions by the hardware processor, and / or for storing the hardware processor state (e.g., the contents of registers) between allocations of hardware processor time when the computer system process is not running. A computer system process runs under the control of an operating system and can also run under the control of other programs executable on the computer system.

[0281] Cloud computing

[0282] In general, the term "cloud computing" is used herein to describe a computing model that enables on-demand access to a shared pool of computing resources (such as computer networks, servers, software applications, and services), and that allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0283] Cloud computing environments (sometimes referred to as cloud environments or the cloud) can be implemented in a variety of different ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes the organization's cloud services available to other organizations or the public. In contrast, a private cloud environment is generally intended for use by or within a single organization only. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private, community, or public) bound together through data and application portability.

[0284] In general, cloud computing models enable some of those responsibilities that might previously have been provided by an organization's own information technology department to instead be delivered as service layers within a cloud environment for use by consumers (either internally within the organization or externally, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where the consumer uses software applications running on cloud infrastructure, and the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), where the consumer can use software programming languages and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, and the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), where the consumer can deploy and run any software applications, and / or provision processing, storage devices, networks, and other basic computing resources, and the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything under the operating system layer). Database as a Service (DBaaS), where the consumer uses a database server or database management system running on cloud infrastructure, and the DbaaS provider manages or controls the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0285] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the invention and what the applicant intends to be the scope of the invention is the literal and equivalent scope of the set of claims presented by this application in the specific form of the presented claims, including any subsequent corrections.

Claims

1. A computer-implemented method, comprising: Assigning multiple blocks of a database to multiple database shard servers, where: The database is partitioned into the multiple blocks; Grouping the multiple blocks into multiple replication units; Each database shard server among the multiple database shard servers is a leader shard for a first subset of the multiple replication units and a follower shard for a second subset of the multiple replication units, Each replication unit among the multiple replication units has an associated distribution factor DF, where the DF is equal to the number of database shard servers configured to take over the user workload from a failed database shard server, and The multiple database shard servers are organized in one or more rings such that the follower shards for a particular replication unit are the DF database shard servers adjacent to the leader shard of the particular replication unit in the ring of the follower shards; and In response to a leader shard of a given replication unit performing a data manipulation language DML operation on the given replication unit, copying the DML operation to the follower shards of the given replication unit; where the method is executed by one or more computing devices.

2. The method according to claim 1, where: Each replication unit among the multiple replication units has an associated replication factor RF, where the RF is equal to the number of replicas of the replication unit stored on the multiple database shard servers, and Each of the one or more rings has a size equal to the RF.

3. The method according to claim 2, where: The one or more rings include multiple rings, The method includes: In response to a barrier mode change operation affecting a given replication unit, quiescing the workload of the ring of database shard servers serving the given replication unit without quiescing the workloads for other rings.

4. The method according to claim 2, further comprising: Adding one or more new database shard servers to the multiple database shard servers; and While maintaining the replication factor and the distribution factor, regrouping and reassigning the multiple blocks to the multiple database shard servers.

5. The method according to claim 4, where: The number of the one or more new database shard servers is equal to the replication factor, and Regrouping and reassigning the multiple blocks includes: Reassigning blocks from the multiple replication units to create multiple new replication units; and Assigning the new replication units to the one or more new database shard servers.

6. The method according to claim 4, where regrouping and reassigning the multiple blocks includes: In response to adding the one or more new database shard servers to the multiple database shard servers resulting in the number of the multiple database shard servers being a multiple of the replication factor, adding rings to the one or more rings.

7. The method according to claim 2, further comprising: Removing one or more new database shard servers from the multiple database shard servers; and While maintaining the replication factor and the distribution factor, regroup and reassign the plurality of blocks to the plurality of database shard servers.

8. The method according to claim 1, further comprising: In response to the failure of a database shard server within a given ring, adding a new database shard server to the plurality of database shard servers and assigning the new database shard server to the given ring.

9. The method according to claim 1, further comprising: Initiating splitting an existing replication unit containing a set of blocks into a first new replication unit containing a first subset of the set of blocks and a second new replication unit containing a second subset of the set of blocks; Queueing an in-progress transaction for the existing replication unit into the existing replication unit; Queueing a transaction that starts after initiating splitting the existing replication unit into the first new replication unit and the second new replication unit; And In response to completion of the in-progress transaction, queueing a split end marker into the first new replication unit and the second new replication unit.

10. The method according to claim 1, further comprising: Initiating merging a first existing replication unit containing a first set of blocks and a second existing replication unit containing a second set of blocks into a new replication unit containing the first set of blocks and the second set of blocks; Queueing an in-progress transaction for the first existing replication unit into the first existing replication unit; Queueing an in-progress transaction for the second existing replication unit into the second existing replication unit; Queueing a transaction that starts after initiating merging the first existing replication unit and the second existing replication unit into the new replication unit; And In response to completion of the in-progress transactions for the first existing replication unit and the second existing replication unit, queueing a merge end marker into the new replication unit.

11. The method according to claim 1, further comprising: Storing a routing map for each given replication unit, where the routing map maintains a read-only routing state for one or more follower shards and a read-write routing state for one or more leader shards.

12. The method according to claim 11, wherein: The routing map is cached outside the database, and The method further comprises invalidating and reloading the cached routing map for the specific replication unit in response to a leadership change for the specific replication unit.

13. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause the execution of a method, the method comprising: Dividing a database into a plurality of blocks; Grouping the set of the plurality of blocks into a plurality of replication units; Assigning the plurality of blocks to a plurality of database shard servers, wherein: Each database shard server within the plurality of database shard servers is a leader shard for a first subset of the plurality of replication units and a follower shard for a second subset of the plurality of replication units. Each replication unit within the plurality of replication units has an associated distribution factor DF, where the DF is equal to the number of database shard servers configured to take over the user workload from a failed database shard server, and the plurality of database shard servers are organized in one or more rings such that the follower shards for a particular replication unit are the DF database shard servers adjacent to the leader shard of the particular replication unit in the ring of the follower shards; and in response to a leader shard of a given replication unit performing a data manipulation language DML operation on the given replication unit, copying the DML operation to the follower shards of the given replication unit.

14. The one or more non-transitory storage media according to claim 13, wherein: each replication unit within the plurality of replication units has an associated replication factor RF, where the RF is equal to the number of replicas of the replication unit stored on the plurality of database shard servers, and each of the one or more rings has a size equal to the RF.

15. The one or more non-transitory storage media according to claim 14, wherein: the one or more rings include a plurality of rings, the method includes: in response to a barrier mode change operation affecting a given replication unit, quiescing the workload of the ring of database shard servers serving the given replication unit without quiescing the workloads for other rings.

16. The one or more non-transitory storage media according to claim 14, the method further includes: adding one or more new database shard servers to the plurality of database shard servers; and while maintaining the replication factor and the distribution factor, regrouping and reassigning the plurality of blocks to the plurality of database shard servers.

17. The one or more non-transitory storage media according to claim 16, wherein: the number of the one or more new database shard servers is equal to the replication factor, and regrouping and reassigning the plurality of blocks includes: reassigning blocks from the plurality of replication units to create a plurality of new replication units; and assigning the new replication units to the one or more new database shard servers.

18. The one or more non-transitory storage media according to claim 16, wherein regrouping and reassigning the plurality of blocks includes: in response to adding the one or more new database shard servers to the plurality of database shard servers resulting in the number of the plurality of database shard servers being a multiple of the replication factor, adding a ring to the one or more rings.

19. The one or more non-transitory storage media according to claim 14, the method further includes: removing one or more new database shard servers from the plurality of database shard servers; and while maintaining the replication factor and the distribution factor, regrouping and reassigning the plurality of blocks to the plurality of database shard servers.

20. The one or more non-transitory storage media according to claim 13, the method further includes: In response to a failure of a database shard server within a given ring, add a new database shard server to the plurality of database shard servers and assign the new database shard server to the given ring.

Images