Consensus protocol for asynchronous database transaction replication with fast automatic failover, zero data loss, strong consistency, complete SQL support and horizontal scalability

Abstract

The invention provides a replication method based on a consensus protocol. For each change operation performed by the leader server on the copy of the database, the leader server creates a copy log record and returns the result to the client. The leader does not wait for a consensus for the change operation from the follower. For submission, the leader creates a submission log record and waits for a consensus. Thus, the leader asynchronously performs database transactions, asynchronously performs replication of change operations, and synchronously performs replication of transaction submissions.

Description

[0001] Cross - Reference to Related Applications; Claim of Priority

[0002] This application claims the benefit of Provisional Application 63 / 415,466, filed on October 12, 2022, the entire content of which is incorporated herein by reference in its entirety as if fully set forth herein, under 35 U.S.C.§119(e). Field of the Invention

[0003] The present invention relates to asynchronous database transaction replication using a consensus protocol with fast automatic failover, zero data loss, strong consistency, full SQL support, and horizontal scalability. Background Art

[0004] Consensus protocols allow a collection of machines to work as a coherent group that can continue to operate in the presence of some member failures. As such, various consensus protocols play a key role in large - scale software systems such as replicated database systems. Raft is a consensus protocol designed to be easy to understand and easy to implement. Raft provides a general way to distribute state machines across a cluster of computing nodes (referred to herein simply as "nodes" or "participant nodes") such that each node in the cluster agrees on the same series of state transitions. Replicated state machines are typically implemented using replicated logs. Each node stores a log replica containing a series of commands, and the state machine of that log replica executes these commands in order; thus, each state machine processes the same sequence of commands. Since state machines are deterministic, each state machine computes the same state and the same sequence of outputs.

[0005] Sharding is a database scaling technique based on horizontal partitioning of data across multiple independent physical databases. Each physical database in such a configuration is referred to as a "shard".

[0006] Sharding relies on replication for availability. Database sharding customers often require high - performance, low - overhead replication that provides strong consistency, supports fast failover with zero data loss, and full Structured Query Language (SQL) and relational transactions. Replication must support nearly unlimited horizontal scalability and symmetric sharding, and balance utilization across each shard. There are Raft implementations for database replication that attempt to address the above requirements, such as Database systems, Cloud Spanner TM Cloud software, CockRoachDB TM Database systems, YugabyteDB, and TiDB.

[0007] Current replication solutions for sharding meet many of these requirements. However, none of them meet all the requirements. For example, Logical Standby can support failover in 1 - 2 seconds; however, in order to balance the load, multiple databases must be configured on each physical server, and more shards must be used to keep up with the workload of the primary shard. Active Data Guard and Logical Standby are active / passive replication strategies with idle hardware at the shard level. GoldenGate TM does not support automatic fast failover in sharded databases.

[0008] Typical NoSQL databases (such as Cassandra TM , DynamoDB TM ) meet many of the above requirements, such as horizontal scalability, simplicity, symmetric sharding; however, they lack SQL support, ACID (atomicity, consistency, isolation, and durability) transactions, and strong consistency. Some NewSQL databases (such as CloudSpanner TM , CockRoachDB TM , YugabyteDB TM , TiDB TM ) provide SQL support and implement consensus-based replication (Paxos or Raft) that supports strong consistency. They typically implement synchronous database replication, which increases the user transaction response time. YugabyteDB TM claims that for some cases (e.g., single-key DML) it applies changes asynchronously to followers. However, YugabyteDB TM may still require synchronization to present a global time for transactions.

[0009] Kafka TM is a well-known messaging system and meets many of the above requirements, but Kafka TM is non-relational. Raft-based replication (RR) does not require persistent storage, RR adds logical logging on top of the physical redo log, RR supports full SQL, and RR can more easily tolerate replicas that are geographically remote.

[0010] The methods described in this section are methods that can be adopted, but not necessarily methods that have been previously envisioned or adopted. Therefore, unless otherwise specified, it should not be assumed that any of the methods described in this section are considered prior art merely because they are included in this section. Additionally, it should not be considered that any of the methods described in this section are readily understandable, routine, or conventional merely because they are included in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings:

[0012] Figure 1 is a block diagram of a distributed computing system having a state machine and a log replicated across multiple computing nodes in accordance with a consensus protocol, illustrating aspects of an illustrative embodiment that can be implemented therein.

[0013] Figure 2 is a block diagram depicting the grouping of blocks for replication in accordance with an illustrative embodiment.

[0014] Figure 3 illustrates a replication unit in a sharded database management system in accordance with an illustrative embodiment.

[0015] Figure 4 is a diagram illustrating a replication user request flow based on a consensus protocol in accordance with an illustrative embodiment.

[0016] Figure 5 is a flowchart illustrating operations for replication based on a consensus protocol for a sharded database management system in accordance with an illustrative embodiment.

[0017] Figure 6 is a block diagram illustrating the architecture for replication based on a consensus protocol in a sharded database in accordance with an illustrative embodiment.

[0018] Figure 7 is a block diagram depicting log persistence optimization in a leader shard server in accordance with an illustrative embodiment.

[0019] Figure 8 is a block diagram depicting log persistence optimization in a follower shard server in accordance with an illustrative embodiment.

[0020] Figure 9 depicts a replicated log having interleaved transactions in accordance with an illustrative embodiment.

[0021] Figure 10 depicts progress tracking in a replicated log in accordance with an exemplary embodiment.

[0022] Figure 11 Illustrates an example of the multiple ring placement of a replication unit according to an illustrative embodiment.

[0023] Figure 12 Is a data flow diagram illustrating the splitting of an example replication unit according to an illustrative embodiment.

[0024] Figure 13 Is a flowchart illustrating the operation of a new leader shard server taking over a replication unit when there is a commit initiated in the replication log according to an illustrative embodiment.

[0025] Figure 14 Is a flowchart illustrating the operation of a shard server performing replication log recovery according to an illustrative embodiment.

[0026] Figure 15 Is a block diagram of a computer system on which embodiments of the present invention can be implemented.

[0027] Figure 16 Is a block diagram of a basic software system that can be used to control the operation of a computer system. Detailed Description

[0028] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be clear that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0029] General Overview

[0030] Illustrative embodiments provide asynchronous database transaction replication based on log replication.

[0031] According to an illustrative embodiment, a leader server receives a command to perform a change operation on a row of a table in a database. The table is replicated across a replication group of servers such that each server within the replication group of the servers stores a corresponding copy of that row of the table. The replication group of the servers includes the leader server and one or more follower servers. The leader server is configured to perform a data manipulation language (DML) operation on the row of the table and replicate the DML operation to one or more follower servers. The leader server performs the change operation on the copy of the row of the table stored at the leader server. The leader server then creates a replication log record for the change operation to be replicated to one or more follower servers in a replication pipeline and returns the result of the change operation to the client. The replication pipeline includes components and mechanisms for storing the log record in the leader and propagating the log record to the followers (from generating the log record on the leader to persisting the log record to disk on the followers). The leader server performs the change operation and returns the result of the change operation to the client without waiting for consensus from one or more follower servers for replicating the log record. Thus, according to the illustrative embodiment, the result of the change operation (DML operation) is returned to the client immediately after the change is made and the log record is created in the leader, without waiting for consensus from the follower servers. After being returned to the client, the propagation of the log record to the followers occurs asynchronously.

[0032] In some embodiments, the leader server receives a database transaction commit command to perform a database transaction commit operation for a particular transaction, creates a replication log record for the database transaction commit operation in a replication pipeline, and performs the database transaction commit operation for the particular transaction on a copy of the row of the table at the leader server in response to receiving an acknowledgement that the replication log record for the database transaction commit operation has been appended to the replication logs of a quorum of one or more follower servers. The leader server then returns the result of the database transaction commit operation to the client. For the database transaction commit operation, the leader server waits for consensus from the follower servers to perform the database transaction commit operation and return the result to the client. Thus, according to the illustrative embodiment, the database transaction commit operation log records are replicated synchronously. The leader server executes DML operations and database transaction commit operations asynchronously, replicates DML log records asynchronously, and replicates commit log records synchronously, thereby enabling fast automatic shard failover with zero data loss and strong consistency.

[0033] According to some illustrative embodiments, to ensure consistency between leader transaction commit and synchronous log replication, the leader server prepares a local transaction for commit and marks the transaction as "pending", and then sends a commit log record to the follower servers. After receiving the consensus on this commit log record, the leader server can commit the local transaction. After a leader server failure, if the old leader becomes a follower and there is a consensus on the commit log record based on the replicated log, then the "pending" state of the local transaction in the old leader can be committed; otherwise, the transaction is rolled back in the old leader.

[0034] In an alternative embodiment, before committing the local transaction, the leader server sends a pre-commit log record to the follower servers. After receiving the consensus on the pre-commit record, the leader server can commit the local transaction and send a post-commit log record to the follower servers. After a leader shard server failure and the leader becomes a follower, if there is a consensus on the pre-commit log record based on the replicated log, then the local transaction in the old leader can be committed. If the local transaction in the old leader has been rolled back by the database, then the transaction is replayed to the old leader based on the replicated log.

[0035] According to illustrative embodiments, to optimize replication efficiency, chunks are grouped into replication units (RUs). A chunk is a unit of distribution in a sharded database system. The sharded database system includes multiple shard servers. In the sharded database of the sharded database system, each sharded table is divided across multiple chunks; a chunk can contain portions of multiple tables. There are typically a large number (thousands or tens of thousands) of chunks, which minimizes data movement during re-sharding and minimizes dynamic chunk splitting. According to some illustrative embodiments, chunks are assigned to RUs based on load and replication throughput. Splitting and merging RUs does not interrupt concurrent user workloads and does not require routing changes because the associated chunks remain in the same set of shard servers. Additionally, transactions spanning chunks within an RU do not require distributed transaction processing.

[0036] According to an illustrative embodiment, each replication unit has a replication factor (RF) and an associated distribution factor (DF). The replication factor refers to the number of replicas / copies of the replication unit, including the primary copy at the leader. The distribution factor refers to the number of shard servers that take over the workload from a failed leader shard server. The replication factor must be odd to determine a majority. The higher the DF, the more balanced the workload distribution after a failover. In some embodiments, the RUs are placed in a ring of shard servers, where the number of shard servers in the ring is equal to the replication factor. This placement helps with schema upgrades. In the case of barrier DDL, the quiescing workload can be restricted to the ring of shard servers rather than the entire sharded database. Alternatively, the placement can be in the form of a single ring. For example, the leader RU on shard 1 is replicated on shards 2 and 3, the leader RU on shard 2 is replicated on shards 3 and 4, and so on. For a single-ring structure, barrier DDL results in quiescing the entire sharded database; however, when new shards are added, it is possible to limit the number of times the RUs can be split, moved, and merged to a fixed number, thereby reducing data movement during incremental deployments and making incremental deployments a more deterministic operation. In a multi-ring arrangement or a single-ring arrangement, the removal of a shard server (e.g., due to a failure or a scale-down) can be done similarly.

[0037] According to an illustrative embodiment, lead-sync logging is used to synchronize the replication logs of follower shards to the leader shard. In response to being unable to determine the existence of a consensus for a database transaction commit operation after a shard server becomes the new leader, the new leader shard uses lead-sync logging to perform a synchronization operation to synchronize the replication logs of the follower shards to the replication log of the new leader. Lead-sync logging requires consensus. Thus, the new leader can determine whether there is a consensus for a database transaction commit operation based on the result of the synchronization operation. The new leader performs the synchronization operation by sending the lead-sync logging to the follower shard servers.

[0038] In an embodiment, when recovering from a database failure, the shard server identifies a first transaction in the replication log that has a first log record but does not have a post-commit log record, defines a recovery window in the replication log (the recovery window starts at the first log record of the identified first transaction and ends at a lead-sync log record), identifies a set of transactions to be recovered, and performs recovery actions on the set of transactions to be recovered. Any transaction that has a log record outside the recovery window is not included in the set of transactions to be recovered, any transaction that has a post-commit log record within the recovery window is not included in the set of transactions to be recovered, and any transaction that has a rollback log record in the recovery window is not included in the set of transactions to be recovered. This allows the shard server to match the state of the replication log to the state of the database transactions by re-creating missing log entries and replaying or rolling back the transactions.

[0039] The following describes the implementation of illustrative embodiments with reference to a sharded database management system; however, aspects of the illustrative embodiments may be applied to other types of database management systems. In some embodiments, a non-sharded database may be represented as a sharded database with only one shard (where all blocks are located in that one database, which may be a single server instance or a Real Application Cluster (RAC)). In other embodiments, a non-sharded database may be a mirrored database implementation (where the non-sharded database may be represented as a sharded database with one replication unit that includes all blocks), the replication group includes all servers, and the replication factor is equal to the number of servers. That is, the replication group may include all servers or a subset of the servers, and the replication unit may include all blocks or a subset of the blocks of the database.

[0040] RAFT protocol

[0041] Raft is a consensus protocol for managing replicated logs. For enhanced understandability, Raft separates the key elements of consensus, such as leader election, log replication, and safety, and enforces a stronger degree of coherency to reduce the number of states that must be considered. Figure 1 is a block diagram of a distributed computing system having a state machine and a log replicated across multiple computing nodes according to a consensus protocol, in which aspects of the illustrative embodiments may be implemented. In Figure 1 the example shown, there is a leader node 110 and two follower nodes 120, 130; however, the distributed computing system may include other numbers of nodes depending on the configuration or workload. For example, the number of nodes in the group of participating nodes may be scaled up or down depending on the workload or other factors that affect resource usage. The consensus protocol typically appears in the context of replicated state machines. As Figure 1As shown, state machines 112, 122, and 132 are replicated across groups of compute nodes 110, 120, and 130, respectively. State machines 112, 122, and 132 operate to compute the same state, and continue to operate even if one or more of compute nodes 110, 120, and 130 fails.

[0042] The replicated state machines are implemented using replicated logs. Each node 110, 120, 130 stores a log 115, 125, 135, respectively, which contains a sequence of commands executed in order by the node's state machine 112, 122, 132. Each log should contain the same commands in the same order, so that each state machine will process the same sequence of commands. Since state machines 112, 122, 132 are deterministic, each state machine computes the same state and the same output sequence.

[0043] Keeping the replicated logs consistent is the purpose of the consensus protocol. The consensus module 111 on the leader node 110 receives commands from clients (such as client 105) and adds the command to its log 115. The consensus module 111 of the leader node 110 communicates with the consensus modules 121, 131 of the follower nodes 120, 130 to ensure that their logs 125, 135 eventually contain the same requests or commands in the same order even in the event of one or more node failures. When the commands are correctly replicated, the state machines of each node process them in log order and return the output to the client 105. Thus, nodes 110, 120, 130 appear to form a single, highly reliable state machine.

[0044] A Raft cluster or group (also referred to herein as a replication group) contains several nodes, such as servers. For example, a typical Raft group can include five nodes, which allows the system to tolerate two failures. At any given time, each server is in one of three states: leader, follower, or candidate. In normal operation, there is exactly one leader, and all other participating nodes are followers. Followers are passive and do not issue requests on their own; followers simply respond to requests from the leader and candidates. The leader handles all client requests. If a client contacts a follower, the follower redirects the client to the leader. The third state (candidate) is used to elect a new leader.

[0045] After a leader is elected, the leader begins serving client requests. Each client request contains a command to be executed by the replicated state machine. The leader node 110 appends the command as a new entry to its log 115 and then issues AppendEntries RPCs in parallel to other nodes 120 and 130 to replicate the entry. Each log entry stores the state machine command and the term number at which the leader received the entry. The term number in the log entry is used to detect inconsistencies between logs and to ensure some of the properties in the nature. Each log entry also has an integer log index that identifies the position of the log entry in the log.

[0046] Raft guarantees that "committed" entries are durable and will eventually be executed by all available state machines. A log entry is committed after the leader that created the entry has replicated the entry on a majority of servers (consensus). This also commits all previous entries in the leader's log, including entries created by previous leaders. The leader keeps track of the highest index known to the leader to be committed and includes that index in future AppendEntries RPCs (including heartbeats) so that other servers will eventually learn it.

[0047] The RAFT Protocol in a Replicated DBMS

[0048] This document describes the Raft consensus protocol for a cluster or group of computing nodes such as servers. In the context of a replicated DBMS, the Raft consensus protocol is applied to a log of replicated commands that are to be executed by the state machines of database servers to apply changes to the database. The changes to be applied to the database by the leader database server (e.g., the leader shard in a sharded database system) are logged at the leader database server and replicated to one or more follower database servers. In turn, each follower database server receives the commands in its log and applies the changes to the corresponding replica of the database in order using the server's state machine.

[0049] In a replicated DBMS implementation, the leader node intercepts changes (e.g., Data Manipulation Language (DML) commands, segmented Large Object (LOB) updates, JavaScript TM Object Notation (JSON) inserts and updates) as Logical Change Records (LCRs). The leader node constructs a Raft log record based on the LCR, which is replicated to follower database servers.

[0050] As an example of a specific implementation scenario of a replicated DBMS, sharding distributes fragments of a dataset across many database servers on different computers (nodes). Sharding is a data layer architecture where data is horizontally partitioned across independent database servers. Each database server is hosted on a dedicated computing node with its own local resources. Each database server in this configuration is referred to as a "shard server" or "shard". All shards together form a single logical database system, which is referred to as a sharded database management system (SDBMS). In some embodiments, horizontal partitioning involves splitting a database table across shards so that each shard contains a table with the same columns but a different subset of rows. A table split in this way is also referred to as a sharded table. In an SDBMS, each participating node can be a leader for one subset of data and a follower for other subsets of data.

[0051] In the context of a replicated DBMS (such as an SDBMS), the Raft consensus protocol handles leader election, log replication, and replicated group membership changes, with modifications that will be described below. The modifications of the illustrative embodiments help ensure asynchronous database transaction replication for fast, automatic shard failover with zero data loss, strong consistency, full SQL support, and horizontal scalability.

[0052] Grouping of blocks for replication

[0053] A replication unit (RU) consists of a collection of blocks. Figure 2 is a block diagram depicting the grouping of blocks for replication according to an illustrative embodiment. Shard 210 can have multiple collections of blocks (RUs) 220. Each RU 220 has a collection of blocks 230. Smaller replication units have lower instantiation overheads after a shard failure. Too many RUs can have higher runtime overheads (e.g., processes). Thus, there is a trade-off between the lower overheads associated with smaller RUs (i.e., smaller collections of blocks) and the lower runtime overheads associated with larger RUs (i.e., fewer RUs).

[0054] Each RU consists of a collection of blocks. All transactions within an RU are replicated in the same replication pipeline, which consists of a set of processes, in-memory data structures, and associated replication logs. To minimize replication overhead, the illustrative embodiments configure the size of the replication unit to maximize throughput. Large replication units can increase the data movement time during resharding.

[0055] Figure 3Illustrates a replication unit in a sharded database management system according to an illustrative embodiment. Each RU has a set of a leader shard server 310 and follower shard servers 320, 330, and the leader shard and all follower shards have the same set of blocks. All DML operations for a particular row are performed in the leader and replicated to the followers of that leader. This is primary replica replication. A shard can be the leader of one replication unit and a follower of other replication units. This results in better hardware utilization. All reads are routed to the leader, unless the application explicitly requests to read from a specified follower shard and tolerate stale data (which can be beneficial if the application is geographically closer to the follower).

[0056] According to an illustrative embodiment, each replication unit has a replication factor (RF), which refers to the number of replicas / copies of the replication unit (including the primary replica at the leader). The Raft protocol requires a majority of replicas to be available for writes; however, a read quorum is not required. This is in contrast to NoSQL databases. When RF = 3, one replica failure can be tolerated; when RF = 5, two replica failures can be tolerated. The illustrative embodiment maintains the replication factor in case of a shard failure, assuming there is capacity in other available shards. Each replication unit also has an associated distribution factor (DF), which refers to the number of shard servers that take over the workload from a failed leader shard server. The higher the DF, the more balanced the workload distribution after a failover.

[0057] Asynchronous database transaction replication based on synchronizing the Raft log

[0058] User Request Process

[0059] The illustrative embodiment uses a consensus protocol (such as the Raft protocol) to synchronously replicate the LCR and perform leader election after a failure or on demand. The synchronous replication of the LCR does not mean the synchronous replication of transactions at the follower shard servers. Figure 4 Is a schematic diagram illustrating a replication user request process for change operations based on a consensus protocol according to an illustrative embodiment. For a given replication unit, there is a leader 410 and multiple followers 420, 430. Each node or server 410, 420, 430 has a corresponding shard catalog 412, 422, 432, which is a dedicated database that supports automatic shard deployment, centralized management of the sharded database, and multi-shard queries. For a given replication unit (a set of blocks), the shard catalogs 412, 422, 432 maintain data describing which server is the leader and which servers are the followers. In an alternative implementation, each node or server can share the shard catalog.

[0060] A logical change record (LCR) encapsulates row changes (e.g., insert, update, LOB operations, JSON operations, old / new values) and transaction directives (e.g., commit, rollback, partial rollback). A replicated log record (LR) is an LCR with a valid log index. Log records in a replication unit have a strictly increasing log index. The replicated log contains log records for in-flight, uncommitted transactions (like a redo log), but does not contain undo or index. This is in contrast to other solutions that only contain committed transactions. The terms "logical change record (LCR)" and "log record (LR)" are used interchangeably herein.

[0061] According to an illustrative embodiment, a read-only (R / O) routing state is maintained for blocks in a follower RU in a routing map (not shown), and a read-write (R / W) routing state is maintained for blocks in a leader RU in the routing map. When there is a change in the leader of an RU, the R / O and R / W states change accordingly. According to an illustrative embodiment, when there is a leadership change, the routing map cached outside the database (e.g., in a client-side driver) is invalidated and reloaded via a notification.

[0062] In an existing Raft protocol, a client sends a command to the leader of a Raft group, and the leader appends the command to the leader's log, sends an AppendEntries RPC to all followers, and after the new entry is committed (stored in the replicated logs of a quorum of followers), the leader executes the command and returns the result to the client. The leader notifies the followers of the committed entries in subsequent AppendEntries RPCs, and the followers execute the committed commands in their state machines. In contrast, the sharded database replication method of an illustrative embodiment first executes the DML in the database before appending it to the replicated log.

[0063] In a replication method based on a consensus protocol of an illustrative embodiment, a logical change record (LCR) encapsulates row changes (e.g., insert, update, delete, LOB operations, JSON operations, old / new values) and database transaction directives (e.g., database transaction commit, database transaction rollback, partial rollback). A log record is an LCR with a valid log index and a term. Log records in a replication group have a strictly increasing log index. Log records contain in-flight, uncommitted transactions, as will be described in further detail below.

[0064] In the consensus-protocol-based replication method of the illustrative embodiment, the user / client 401 sends a DML to the leader 410, which executes the DML on the replicas of the leader's replication unit before appending a log record (LR) for the DML to the leader's replication log 415. The leader also returns a result (e.g., and an acknowledgement) to the user / client 401 in response to the LR being created in the leader's replication log 415. The result is returned without verifying that the followers have stored the LR in the followers' replication logs, and thus without followers acknowledging the storage of the LR. The leader can return to the client before writing the log record to the leader's replication log. The persistence of the log record is done asynchronously.

[0065] The replication log 415 is propagated to the followers 420, 430, which respectively store the LR for the DML in the replication logs 425, 435. In response to the follower 420 appending the LR to the follower's replication log 425, the follower 420 returns an acknowledgement (ACK). If a quorum of followers returns an ACK to confirm that the LR has been stored in the replication log, then the leader 410 considers the DML LR to be committed. Similarly, the followers 420, 430 eagerly execute the DML while concurrently appending the DML to the followers' replication logs 425, 435. This minimizes the impact on the user transaction response time and improves replication efficiency.

[0066] To maintain the ACID properties among the replicas, the leader commits the relevant database transaction only if the committed LCR is a committed log record (appended to a quorum of follower replication logs), which means that any LCR generated previously for the database transaction is also a committed log record. Thus, in the sharded database replication method of the illustrative embodiment, the user / client 401 sends a database transaction commit command to the leader 410, which creates a commit LR in the leader's replication log 415 and propagates the replication log 415 to the followers 420, 430. The leader commits the database transaction in the replicas of the leader's replication unit only when it receives an acknowledgement from a quorum of followers for the commit LR. The leader generates a post-commit LR and then returns the result (e.g., and an acknowledgement) of the database transaction commit to the user / client 401.

[0067] Figure 5FIG. 0 is a flowchart illustrating operations of consensus - protocol - based replication for a sharded database management system. The operation starts (block 500), and the leader shard server receives a command from a client (block 501). The leader shard server determines whether the command is a database transaction commit command (block 502). If the command is not a database transaction commit command (block 502: No), then the command is a change operation (e.g., insert, update, delete, etc.), and the leader shard server executes the change operation (block 503). Then, the leader shard server creates a replication log record in the replication log of the leader shard server (block 504) and returns the result of the change operation to the client (block 505). After that, the operation returns to block 501 to receive the next command from the client.

[0068] If the command is a database transaction commit command (block 502: Yes), then the leader shard server creates a replication log record in the replication log of the leader shard server for the database transaction commit (block 506). The leader shard server determines whether the leader shard server has received consensus for the commit log record (block 507). If consensus is received (block 507: Yes), then the leader shard server executes the database transaction commit operation (block 508), advances the commit index (block 509), and writes the log record to disk (block 510). After that, the operation returns to block 501 to receive the next command from the client.

[0069] If consensus is not received (block 507: No), then the leader shard server rolls back the database transaction (block 511). After that, the operation returns to block 501 to receive the next command from the client.

[0070] System Architecture

[0071] Figure 6 FIG. 13 is a block diagram illustrating the architecture for consensus - protocol - based replication in a sharded database according to an illustrative embodiment. User 601 sends DML and transaction indications to leader 610 for a given replication unit. As Figure 6 shown, leader 610 includes capture component 611, System Global Area (SGA) 612, in - memory replication log queue 613, network senders 614A…614B, consensus module 615, and a persistent replication log in disk 616. Capture component 611 intercepts DML execution and captures DML, piece - wise LOB updates, JSON inserts and updates (JSON_Transform) as LCRs. Capture component 611 also intercepts transaction execution to capture database transaction commits, database transaction rollbacks, and rollbacks to save points.

[0072] The SGA is a set of shared memory structures that contain data and control information for a database instance, called SGA components. The SGA is shared by all server and background processes. In the depicted example, the capture component 611 stores the LCRs to be inserted into the in-memory replication log queue 613.

[0073] In some embodiments, there is a commit queue (not shown) between the SGA 612 and the consensus module 615. This commit queue contains commit records for each transaction. When the consensus module 615 receives an acknowledgement from a follower, the consensus module reads the commit queue to find a matching transaction and checks if this transaction has achieved consensus. If consensus is achieved, then the consensus module releases the user session, which allows the transaction to commit, generates a post-commit LCR, and returns control to the user.

[0074] The network senders 614A, 614B distribute the replication logs to the followers 620, 630 via the network 605. The leader can have a network sender for each follower in the replication group. The consensus module 615 communicates with the consensus modules on other servers (such as the consensus modules 625, 635 on the followers 620, 630) to ensure that each log ultimately contains the same log records in the same order even if some servers fail.

[0075] The follower 620 includes a network receiver 621, an in-memory replication log queue 622, an SQL application server 623, a consensus module 625, and a persistent replication log on the disk 626. Similarly, the follower 630 includes a network receiver 631, an in-memory replication log queue 632, an SQL application server 633, a consensus module 635, and a persistent replication log on the disk 636. The network receivers 621, 631 receive the replication logs from the leader 610 and transfer the LCRs to the SQL application servers 623, 633.

[0076] The consensus modules 615, 625, 635 have an LCR persister process at each replica (including the leader) to durably persist the LCRs into the persistent replication logs on the disks 616, 626, 636. In the followers, the consensus modules 625, 635 send the highest persisted log index back to the leader 610 of the follower to confirm that the log records up to the highest persisted log index have been persisted.

[0077] Only the leader can process user DML requests. Followers can automatically redirect DMLs to the leader. In the DML execution path, the leader constructs a log record to encapsulate the DML change, enqueues the log record into the SGA circular buffer, and immediately returns to the user. The illustrative embodiment decouples replication from the original DML and asynchronously pipelines DML log record, propagation, and SQL application at the follower with minimal latency. For multi-DML transactions, replication overlaps mostly with the user transaction, and the latency overhead from commit consensus is significantly smaller.

[0078] DML Replication Process

[0079] Reference Figure 6 , user 601 submits a DML. The capture component 611 constructs an LCR in image format or object format and enqueues the LCR into the in-memory queue 613. In one embodiment, the in-memory queue is a contention-free multi-writer, single-reader queue. The LCR in image format in the queue minimizes multiple downstream picklings (e.g., persisting and propagating the LCR). The leader 610 returns to the user 601 immediately after constructing the required log record in the DML execution path.

[0080] The in-memory queue 613 is a single-writer and multi-reader queue. The LCRs in the SGA 612 are stored in a different queue which is a contention-free multi-writer and single-reader queue. Each capture process representing a user is a writer to this queue. The LCR producer process is a single-reader of this queue and dequeues the LCR from the SGA 612, assigns a unique and strictly increasing log index to the LCR, and enqueues the log into the in-memory queue 613. The strictly increasing log index is an important property of the Raft log.

[0081] Asynchronously, the consensus module 615 scans through the queue 613 and constructs replication log records based on the LCRs dequeued from the queue. The consensus module 615 (LCR persister) persists the in-memory log records to disk 626. The consensus module 615 (ACK receiver) counts the acknowledgments from the followers 620, 630 and appropriately advances the log commitIndex. The network senders 614A, 614B call the AppendEntries RPC to propagate the log records to all followers over the network 605. For each log record propagation, the network senders 614A, 614B also include the current committed log index.

[0082] The network receivers 621, 631 are automatically generated due to the connections from the network transmitters 614A, 614B at the leader. If the AppendEntries RPC passes its verification, then the network receivers 621, 631 enqueue the log records from the wire to the queues 622, 632 via the consensus protocol application programming interface (API). The consensus modules 625, 635 at the followers read the in-memory queues 622, 632 containing the log records from the leader, persist the log records to the disks 626, 636, and send acknowledgments to the leader via separate network connections.

[0083] The SQL application servers 623, 633 read the LCRs from the in-memory queues 622, 632, assemble the transactions from the cross-cutting LCRs from different transactions, and apply the transactions to the database. This is referred to herein as "eager apply". If the SQL application servers 623, 633 are slow or catching up, then the SQL application servers may need to retrieve the relevant log records from the disks 626, 636.

[0084] As mentioned above, the consensus-protocol-based replication of the illustrative embodiments does not require explicit consensus on DML. Once the LCRs for the row changes are pushed into the replication pipeline, the leader immediately returns control to the user or allows subsequent processing of the transaction. This minimizes the impact on the user's response time for DML. The replication (propagation, persistence) of DML is done asynchronously in a streaming fashion. If there is consensus for the commit, then the replication method maintains the transaction ACID properties among the replicas.

[0085] Submit Replication Process

[0086] When a database transaction commit for a user transaction is received from the user 601, the network transmitters 614A, 614B send the log records containing the transaction commit to all the followers 620, 630. In parallel, the consensus module 615 (LCR persister) writes the log records to the disk 616 and keeps track of the persisted log indexes. In other words, the leader 610 does not need to wait for the persistence of the log records containing the transaction commit before sending its log records to its followers.

[0087] When the leader 610 receives consensus from a quorum of followers, the leader advances the leader's log commitIndex as far as possible. The committing and advancing of the commitIndex for user transactions need not be atomic. The leader 610 does not need to verify that the log records have been persisted locally before sending the next set of log records to followers 620, 630. The leader needs to ensure that the committed log records have been persisted locally before committing the transaction locally. In practice, the log records will have been persisted anyway when the leader receives consensus. One way is to verify that the persisted log index is equal to or greater than the log index for which the log record was committed for the transaction. The leader then commits the transaction and returns success to the user.

[0088] If the leader 610 crashes before communicating the consensus of the log records to its followers, then the new leader can complete the replication of any last set of committed log records regardless of whether the previous leader started. Because the application of the transaction commit command and the return to the user are not atomic, the user may resubmit the transaction, potentially to a different leader. This resubmission of the transaction may encounter an error and exist outside of the sharded database.

[0089] Submitted Batch Processing Confirmation

[0090] According to an illustrative embodiment, all log records (including commits) are sent continuously without interruption in the same stream. Acknowledgments are sent back via a separate network connection. There is only one network round-trip for acknowledgments from each fast follower. Even when the leader is waiting for an acknowledgment of a pending commit for one user transaction, the leader can continue streaming log records with higher log indexes, including other commits from other user transactions. In some embodiments, all log records cannot have intertransaction dependencies. This maximizes concurrency for independent user transactions. This also enables natural batching of acknowledgments for commits: one acknowledgment from a follower can include multiple transactions. This thus improves replication efficiency. Obtaining consensus for committed log records only requires one round-trip.

[0091] Replication Log Persistence

[0092] A set of the replication log is maintained for each replica in each replication group of the server (a copy of the RU on the server). There will be no interference between replication units when reading and writing to the replication log. There are numerous possible embodiments for persisting the replication log in all replicas (including the leader), including the following examples:

[0093] 1. Since database redo persistence has been highly optimized, it would be efficient to write replication logs as database redo records. Replication log records are interleaved with database change records in the redo log. There is no indexing ability when reading the redo log. Therefore, it is inefficient to find a specific replication log record from the redo.

[0094] 2. Write replication logs to an external file or queue. It will be necessary to deploy and monitor external components.

[0095] 3. Write replication logs to a database table. The overhead for inserting for each user row change is very high.

[0096] 4. Implement custom file management.

[0097] According to an illustrative embodiment, a custom file is implemented so that replication logs can be read faster during recovery (role transition) and when catching up with a new replica. In addition, the custom file allows replication logs to be transported between replicas (e.g., for catching up), moving a follower from one shard server to another, and re-instantiating a new follower. The illustrative embodiment employs asynchronous I / O during writing and asynchronous I / O during prefetching and reading.

[0098] Figure 7 is a block diagram depicting log persistence optimization in a leader shard server according to an illustrative embodiment. The log persistence process group 710 in the leader shard receives the LCR 702 from the SGA. The LCR producer process 711 dequeues the LCR 702 and enqueues the LCR into the circular queue 713. The queue in the LCR 702 is a multi-writer and single-reader queue. The network senders 714A, 714B and the LCR persister process 715 are subscribers to the circular queue 713. The network senders 714A, 714B browse the circular queue 713 and distribute the replication logs to the followers.

[0099] The LCR persister process 715 maintains a set 716 of I / O buffers in the SGA. Log records are dequeued from the circular queue 713 and placed into the I / O buffers 716 as they appear in the on-disk log file 717. On the leader, since the LCR persister process 715 is also a subscriber to the circular queue 713, this persister process must ensure that records are dequeued quickly to ensure that the queue 713 does not fill up. Therefore, I / O is asynchronous; once the I / O buffer is full, asynchronous I / O is issued. This I / O is reclaimed when the buffer is used again or if a commit record is encountered (when all outstanding I / O has been reclaimed).

[0100] The LCR persister process 715 performs the following operations:

[0101] 1. Dequeue one or more log records. A log record is a wrapper that contains an LCR with a term and a log index.

[0102] 2. Place the replicated log into one or more IO buffers. When the IO buffer is full, issue the IO asynchronously.

[0103] 3. Recover the issued IO when the buffer is to be reused.

[0104] 4. At commit time, recover all outstanding IO.

[0105] Log Persistence Optimization in Followers

[0106] Just like log persistence at the leader, each follower uses the replicated log file for durably persisting the LCR. Figure 8 is a block diagram depicting log persistence optimization in a follower shard server according to an illustrative embodiment. The log persistence process group 820 in the follower shard receives the LCR, commitIndex, the minimum persisted log index, and the minimum oldest log index from the leader. The network receiver 821 receives log records from the wire, places the LCR into the IO buffer 825 in the circular queue 823 and the SGAS, and notifies the LCR persister process 826 if there is no free IO buffer.

[0107] The LCR persister process 826 persists the LCR from the IO buffer 825 to the disk log file 828. The LCR persister process 826 monitors the IO buffer 825 and issues a block of the buffer as a synchronous IO when the utilization of the buffer reaches a predetermined threshold (e.g., 60%) or when a commit is received. The LCR persister process 826 also notifies the acknowledgment (ACK) sender 827 when the IO is completed.

[0108] The ACK sender 827 maintains a record of the last log index 829 that has been acknowledged. Whenever the LCR persister process 826 persists a commit index that is higher than the last acknowledged index, the ACK sender 827 sends this information to the leader.

[0109] The network receiver 821 sends the commitIndex 822 to the SQL application process 824. The SQL application process 824 reads the LCR from the circular queue 823, assembles transactions from the LCRs that cross different transactions, and performs DML operations on the database.

[0110] Cross Logging

[0111] A log record contains uncommitted transactions that cross in the replicated log. Figure 9Depicts a replication log with crossed transactions according to an illustrative embodiment. Figure 9 The replication log shown in Figure 9 includes a Raft log, and each log in the Raft log has a log index. T1L1 means the first LCR in transaction T1. T1C means the commit LCR for transaction T1. T2R means the rollback LCR for transaction T2. As can be seen in the depicted example, T1L1 has a log index of 100, T2L1 has a log index of 101, T1C has a log index of 102, and T2R has a log index of 103. Thus, transactions T1 and T2 are crossed because transaction T2 has a log entry between the first LCR and the commit LCR of transaction T1.

[0112] Application Progress Tracking

[0113] There is a group of SQL application processes in the follower to replicate transactions. In one embodiment, the SQL application server consists of an LCR reader process, an application reader process, a coordinator process, and multiple applicator processes. The application LCR reader process dequeues the LCR, possibly reads the LCR from the persistence layer, and calculates the hash value for the relevant key columns. The application reader process assembles the LCR into a complete transaction, calculates the dependencies between transactions, and passes the transaction to the coordinator. The coordinator process assigns the transaction to an available applicator based on transaction dependencies and commit ordering. Each applicator process executes the entire transaction at the replica database and then requests another transaction. The applicators process independent transactions concurrently to achieve better throughput. To minimize application latency, the applicators start applying DML for the transaction before receiving a commit or rollback. This is called "eager application". However, even if the applicator sees the commit LCR for this transaction, it cannot commit the transaction unless the commit LCR has reached consensus. Each transaction is applied exactly once.

[0114] Figure 10Depicts the application progress tracking in the replication log according to an exemplary embodiment. The SQL application maintains two key values regarding its progress: the Low Water Mark Log Index (LWMLI) and the oldest log index. All transactions with a commit index less than or equal to the LWMLI have been applied and committed. The oldest log index is the log index of the earliest log record that the application may need. For each replicated transaction, the SQL application inserts a row into a system table (e.g., appliedTxns) that contains the source transaction ID, the log index of the first DML, and the log index of the txCommit (which references the commit for the user transaction). During recovery, the recovery process or the application process can start reading from the oldest log index, skip the transactions that have already been applied based on appliedTxns, and complete the replication of any open transactions. As an optimization, multiple transactions can be batched and applied as one transaction in the follower for better performance.

[0115] Replication Unit Placement

[0116] There are multiple ways to place replication units (RUs) across shards in a cross-shard database, and these ways come with trade-offs.

[0117] Multi-ring Placement

[0118] During initial placement, in a balanced scenario, the RUs are placed in a ring where the size of the ring is equal to the replication factor. This placement helps with schema upgrades. In the case of barrier DDL, the quiescence workload can now be restricted to the ring of shard servers instead of the entire shard database management system. For example, assume there are 6 shards, 12 RUs, and RF = 3, and each RU contains 60 blocks. Then the initial placement balances all the RUs across the 6 shards to create two rings. Figure 11 Depicts an example of the multi-ring placement of replication units according to an illustrative embodiment. In this example, RU1 and RU2 are replicated from shard 1 to shards 2 and 3, RU3 and RU4 are replicated from shard 2 to shards 1 and 3, and RU5 and RU6 are replicated from shard 3 to shards 1 and 2. Similarly, RU7 and RU8 are replicated from shard 4 to shards 5 and 6, RU9 and RU10 are replicated from shard 5 to shards 4 and 6, and RU11 and RU12 are replicated from shard 6 to shards 4 and 5.

[0119] All replicas in RU1 through RU6 are contained within shards 1, 2, and 3. A similar structure is observed for the second ring, where all replicas in RU7 through RU12 are contained within shards 4, 5, and 6. With this placement, any quiescence for the barrier DDL can be restricted to one ring without affecting the entire sharded database management system. The application schema upgrade can be performed in a way that only affects a small subset of the shard servers.

[0120] In the depicted example, RF = 3, and each ring is sized three. If RF = 5, then each ring would be sized five shard servers.

[0121] For incremental deployment, it is ideal to add the RF number of shard servers simultaneously. In the example shown in Figure 11 , if three shard servers are added, then an equal number of blocks are pulled from all RUs to create six new RUs, which will be deployed on these three new shard servers, thus creating the shard servers for the third ring.

[0122] However, if the user adds one shard server at a time, then the following steps occur:

[0123] Step 1: For one ring and three shard servers, the topology is shown in Table 1 below:

[0124] Fragment 1 (RU) Fragment 2 (RU) Fragment 3 (RU) Leader 1、2 3、4 5、6 Follower 3、4、5、6 1、2、5、6 1、2、3、4

[0125] Table 1

[0126] Each RU has 60 blocks. In this case, the barrier DDL must be synchronized across three shard servers.

[0127] Step 2: Add shard 4. Even though a shard server is added, since there are not enough shard servers to form two smaller rings, one ring remains. Blocks for the new RUs are pulled equally from the other RUs. Now each RU has 45 blocks. After adding the fourth shard server, for any barrier DDL propagation, synchronization across all four shard servers is required. The topology is shown in Table 2 below:

[0128] Fragment 1 (RU) Fragment 2 (RU) Fragment 3 (RU) Fragment 4 (RU) Leader 1、2 3、4 5、6 7、8 Follower 5、6、7、8 1、2、7、8 1、2、3、4 3、4、5、6

[0129] Table 2

[0130] Step 3: Add shard 5. Again, for the same reason, no new smaller ring is created. The new shard server pulls blocks from all shard servers. Now each RU has 36 blocks. After adding the fifth shard server, for any barrier DDL propagation, synchronization across all five shard servers is required. The topology is shown in Table 3 below:

[0131]

[0132] Table 3

[0133] Step 4: Add Shard 6. This time, the followers are moved apart to form two smaller rings. Now each RU has 30 blocks. After adding the sixth shard server, for any barrier DDL propagation, synchronization across only three shard servers is required. The topology is shown in Table 4 below:

[0134]

[0135] Table 4

[0136] As seen in Table 4 above, Shard 1, Shard 2, and Shard 3 form one ring, and Shard 4, Shard 5, and Shard 6 form a second ring. Thus, a barrier DDL can be applied to Shard 1, Shard 2, and Shard 3 (only quiescing the workloads for RU1 - 6), and then to Shard 4, Shard 5, and Shard 6 (quiescing the workloads for RU7 - 12).

[0137] Step 5: Thereafter, when new shards are added, there is no longer a need to modify the first ring (Shard 1, Shard 2, Shard 3). Any new shard server will pull blocks from all shards, but the RU composition of the first ring will not change. When Shards 7 and 8 are added, the second ring (Shard 4, Shard 5, Shard 6) will expand to five shard servers, and by Shard 9, Shards 4 through 9 will split to form two smaller rings, thus forming a total of three smaller rings. This layout allows a barrier DDL to be applied to a single ring of shard servers without affecting the entire sharded database management system.

[0138] Single-ring Placement

[0139] Replica unit placement can also be done in the form of a single ring. The set of leader blocks on Shard 1 is replicated on Shards 2 and 3, the leader blocks on Shard 2 are replicated on Shards 3 and 4, and so on. For example, if there are four initial shards, RF = 3, DF = 2, and each RU has sixty blocks, then there will be eight RUs, numbered from 1 to 8, for a total of 480 blocks. The set of leader blocks is labeled "L", and the set of follower blocks is labeled "F". The numbers indicate the number of blocks in the block sets. The block distribution is shown in Table 5 below:

[0140]

[0141]

[0142] Table 5

[0143] In the RF = 3, DF = 2 configuration, adding a new shard server results in two leader RUs and four follower RUs on the new shard server. This can be achieved in two ways:

[0144] 1. Obtain an equal number of chunks from all shard servers to fill the new shard server. This method maintains a balanced system at the cost of a lot of data movement between followers. Since all leaders on all shard servers are affected by this method, all followers on all shard servers are also affected - that is, if there are n shard servers in the system, then the cost of adding the (n + 1)th shard depends on n. For example, consider 4 shards, each with 2 leader RUs and 4 follower RUs. When pulling chunks from each RU, the 8 leader RUs shrink to create the 9th and 10th RUs on the 5th shard. The followers for each affected RU must also shrink to match that leader - so adding a shard server ultimately touches every RU on every shard server.

[0145] 2. When filling the new shard server, limit the "donor" shard servers to a smaller fixed number. This method does not attempt to achieve a balanced system when adding a shard server, but this trade-off allows the system to reach a stable state faster after a fixed amount of data movement. In this method, adding a shard server to the ring involves pulling chunks from the four closest shard servers (two on each side). This is done as follows:

[0146] a. Determine the insertion point: Find 4 adjacent shard servers that together have the highest number of chunks compared to any other adjacent group of 4.

[0147] The new shard server should be inserted in the middle of these 4 donor shard servers.

[0148] As a special case, when changing from 3 to 4, insert anywhere and pull from all 3 shard servers.

[0149] b. Split the set of leader chunks on the 4 donor shard servers (for the DF = 2 case, 8 leaders are split). The split should be done in such a way that all 5 shard servers (4 donor shards and the new shard) retain approximately the same number of chunks for each set of leader chunks.

[0150] 3. Split the corresponding set of follower chunks in the same way (for RF = 3, 16 followers are split).

[0151] 4. Move 2 leaders from each donor shard server to the new shard server (8 leaders are moved).

[0152] 5. On the new shard server, merge in groups of 2 RUs, so that there are 4 leader RUs (reduced from 8). Then merge the corresponding follower RUs. This can be done without any data movement.

[0153] 6. At this point, all the leader block sets on the new shard server are in the correct position except for 4 follower block sets. Move these 4 follower block sets for the final round of merging.

[0154] 7. Merge the RUs on the new shard server so that 2 RUs remain, with each RU having 4 block sets. Merge the corresponding follower RUs.

[0155] 8. Move the followers to the new shard server (4 followers are moved).

[0156] 9. Summary: 8 leaders split, 16 followers split, 8 leaders moved, 4 leaders merged, 8 followers merged, 4 followers merged, 2 leaders merged, 4 followers merged, 4 followers moved.

[0157] The single-ring structure has a drawback: barrier DDL causes the entire sharded database management system to become static. However, when adding a new shard server, it is possible to limit the number of times RUs are split, moved, and merged to a fixed amount, thereby reducing the data moved during incremental deployment and making incremental deployment a more deterministic operation. Removing a shard server (e.g., due to failure or downsizing) can be done similarly in two options.

[0158] Resharding

[0159] Split and Merge Replication Units

[0160] To add a new shard server or remove a shard server (e.g., shard failure), it may be necessary to move RUs from one shard server to another, or split and / or merge RUs. According to an illustrative embodiment, the shard coordinator ensures that in-progress transactions during those operations are not affected. There are several methods for splitting or merging RUs. Figure 12 FIG. is a data flow diagram illustrating an example replication unit split according to an illustrative embodiment. For simplicity, assume there is no leader change during RU split. Transactions T1, T2, T3, T4 are executed on a given shard server (leader or follower) within the sharded database management system. The RU state 1201 in the shard starts with RU1 = {C1, C2}, where C is a block. That is, RU1 includes two blocks, C1 and C2.

[0161] There is in - progress DML 1211 from transaction T1 to insert row 1 into C2 and in - progress DML 1231 from transaction T3 to insert row 3 into C2. Then, the fragmentation coordinator 1250 initiates a split operation 1251 to split RU1 into RU1 and RU2. As a result, the RU state 1202 in the fragmentation server becomes RU1 = {C1}, RU2 = {C2}. That is, RU1 is split into RU1 containing block C1 and RU2 containing block C2.

[0162] The fragmentation coordinator 1250 creates a new RU ( Figure 12 RU2 in), which has the same set of fragmentation servers for replicas of this RU. The leader of the old RU is the leader of the new RU. The fragmentation coordinator 1250 creates a new RU and enqueues a split RU begin marker into both the old RU and the new RU. The fragmentation coordinator 1250 associates the set of blocks ( Figure 12 C2 in) with the new RU. The fragmentation coordinator 1250 establishes a replication process for the new RU and suspends the SQL apply process for the new RU.

[0163] The fragmentation coordinator 1250 waits for the transactions running concurrently with the RU split to complete. In the Figure 12 example shown, when the split is initiated, transactions T1 and T3 are in progress. Transaction T3 depends on transaction T2, and transaction T2 completes during the split process. As a defensive measure, the fragmentation coordinator 1250 enqueues the old values of all scalar columns for update and deletion during the RU split process. The fragmentation coordinator 1250 performs the same operation for the RU merge process.

[0164] The remaining changes of the in - progress transactions (e.g., T1 and T3) after the RU split start will be enqueued to the existing RU. Thus, DML 1212 and commit 1213 for T1 and DML 1232 and commit 1233 are enqueued to the existing RU1. Transactions started after the split (e.g., T2 and T4) will be enqueued to both the new RU and the old RU. Thus, DML 1221 and commit for T2 and DML 1241 and commit for T4 are enqueued to both RU1 and RU2. After all in - progress transactions are completed, the fragmentation coordinator 1250 enqueues a split RU end marker into both the old RU and the new RU, thus performing the end split RU 1252.

[0165] If a transaction (e.g., T1, T3) starts before the split command is initiated, then the SQL application for the old RU executes this transaction. These transactions are considered in progress. For transactions that start after the RU split operation is initiated (e.g., T2, T4), if a transaction (e.g., T2) ends before the split command is completed, then the SQL application for the old RU executes this transaction, and the SQL application for the new RU ignores this transaction. If a transaction ends after the RU split operation is completed, then the SQL application for the new RU executes this transaction, and the SQL application for the old RU ignores this transaction.

[0166] When the SQL application in a follower for the old RU receives the split RU end marker enqueued above, it suspends the SQL application process for the new RU. Long-running transactions will cause the SQL application process to be suspended for a longer time.

[0167] After the RU split process, the shard coordinator enables periodic leadership changes based on a consensus protocol.

[0168] The above reference Figure 12 The RU split process described above can be generalized to split one RU into multiple RUs, not just two.

[0169] Add New Fragment

[0170] When adding a new shard server, if the user requests block rebalancing, then new RUs can be created on the new shard via relocate block commands and move RU commands. There are many rebalancing options, such as the following:

[0171] 1. Fill the new shard server with an average number of blocks per shard server in each replication group. This will result in a balanced system at the cost of a one-time large data movement among followers.

[0172] 2. When filling the new shard server, limit the "donor" shards to a small fixed number. This is to minimize data movement. However, this will result in a more unbalanced system.

[0173] Move Replication Unit to New Follower

[0174] Moving an RU from one follower to another can be performed as follows:

[0175] 1. Stop the SQL application at the old follower and record the application metadata (low watermark and the oldest log index).

[0176] 2. Execute the following two tasks in parallel:

[0177] a. Instantiate a new follower. Copy relevant data (application metadata, user data) using the old follower, e.g., TTS, export / import. Instantiate user data and create a new application in the new follower with the low watermark and the oldest log index.

[0178] b. Copy the replicated log at the old follower (starting from the oldest log index) and transport it to the new follower.

[0179] 3. Make the new follower a non-voting member. Start applying the replicated log starting from the old log index.

[0180] 4. When the new follower has almost caught up, make the new follower a voting member.

[0181] 5. Remove the old follower from the replication unit.

[0182] Restore Replication Unit

[0183] Reasons for restoring an RU include creating additional replicas on a new shard server to increase the replication factor, replacing the replication unit on a "failed" shard server with a replication unit on another shard server, rebuilding the replication unit on a shard server to recover from data / log divergence, and rebuilding the replication unit after a long downtime of an outdated replication unit to allow it to catch up.

[0184] The high-level steps for restoring an RU are as follows:

[0185] 1. If it exists, then remove the RU from the target shard server.

[0186] 2. Replicate the RU.

[0187] 3. Restore the RU on the source database to the R / W state.

[0188] 4. Update the peers and the directory (only needed when adding / replacing).

[0189] 5. Start the RU process group on the target shard server.

[0190] Recovery

[0191] According to an illustrative embodiment, if a shard server fails and there is a new standby shard, then select the target shard (usually hosting a follower) and copy all the blocks and the replicated log for the relevant replication unit to the new shard. If there is no new standby shard server and there is available capacity in the existing shard servers, then redistribute all the blocks for the relevant replication unit to those shards, thereby maintaining the replication factor. For simplicity, new transactions are not allowed to span epochs; however, commit and rollback records can be generated in the new leader with a different epoch.

[0192] Consistency of Leader Transaction Commit and Replication Log Synchronous Replication

[0193] To ensure consistency between leader transaction commits and replicated log synchronous replication, two embodiments are described below.

[0194] In one embodiment, the leader prepares a local transaction for commit and first marks the transaction as "pending", then sends a commit LCR to the followers of the leader. After receiving the consensus on the commit LCR, the leader commits the local transaction. After a leader shard server failure, if the failed leader becomes a follower and there is a consensus on this commit LCR based on the replicated log, then the "pending" state of the local transaction in the failed leader can be committed. Otherwise, the transaction must be rolled back in the failed leader. Each follower receives the commit LCR but does not perform a commit on the replicas of the follower's replication unit until the follower receives a commit log index from the leader that is equal to or greater than the log index of the commit LCR, indicating that the commit LCR has been persisted by other followers.

[0195] In an alternative embodiment, before committing the local transaction, the leader sends a pre-commit LCR to the followers of the leader. After receiving the consensus on the pre-commit LCR, the leader commits the local transaction and sends a post-commit LCR to the followers of the leader. After a leader shard server failure and the resulting leader becomes a follower, if there is a consensus on the pre-commit LCR based on the replicated log, then the local transaction can be committed in the failed leader. If the local transaction in the failed leader is rolled back, then the transaction can be replayed to the failed leader based on the replicated log. If the transaction fails after sending the pre-commit LCR (e.g., the user session crashes), if the shard server is still the leader, then the process monitor in the database generates a rollback record and sends the rollback record to its followers.

[0196] Transaction Recovery

[0197] The LCR sequence for transaction T1 is as follows:

[0198]

[0199] where T D is the DML LCR, T F is the first LCR of the transaction, this T F can be the DML LCR, T pre is the pre-commit LCR (awaiting consensus), and T post is the post-commit LCR. At any point in time before the post-commit LCR, the transaction can be rolled back, resulting in T RBK which is the TRBK is the rollback LCR. The LCR before submission is designed to allow the leader to receive consensus before locally committing the transaction.

[0200] The leader does not wait for consensus on the post-submission LCR before returning to the client. The consensus on the post-submission LCR. The commitment of the transaction at the leader occurs between the pre-submission LCR and the post-submission LCR. On each follower, the commitment occurs after receiving the post-submission LCR. The follower does not commit the transaction to the replica of the follower's replication unit after receiving the pre-submission LCR because the follower does not know if there is already a consensus for the pre-submission LCR. In fact, the follower may receive a rollback LCR after the pre-submission LCR. If the follower receives the post-submission LCR, then the follower knows that there is already a consensus for the pre-submission LCR and the leader has committed the transaction.

[0201] If the leader fails before sending the post-submission LCR and an election for leadership occurs, then the new leader will be the follower with the longest replication log. Several scenarios may occur. One scenario is that the new leader has the LCR for the transaction, excluding the pre-submission LCR or the post-submission LCR. In this case, the new leader does not know if there are other DML LCRs for the changes made by the old leader and already confirmed to the user / client. Therefore, the new leader will roll back the transaction and issue a rollback LCR.

[0202] Another scenario is that the new leader has the LCR for the transaction, including the pre-submission LCR and the post-submission LCR. This transaction must be committed. Therefore, the new leader commits the transaction.

[0203] Another scenario is where the new leader has LCRs for a transaction, including pre-commit LCRs but not including post-commit LCRs. The new leader cannot assume that the pre-commit LCRs have received consensus. According to the illustrative embodiment, the new leader issues a lead-sync LCR, which is a special LCR sent at the start of a term to synchronize the replicated logs of the followers. The new leader waits for consensus on the lead-sync LCR, which indicates that the number of followers in consensus have persisted all LCRs up to the lead-sync LCR. In other words, a follower only acknowledges the lead-sync LCR when the follower has persisted all LCRs up to the lead-sync LCR. Each LCR has a log index; thus, each follower will know whether all LCRs up to and including the lead-sync LCR have been persisted. A follower cannot acknowledge that the lead-sync LCR has been persisted without acknowledging that every LCR with a log index less than the lead-sync LCR has been persisted. This will ensure that for transaction T1, all followers will have the pre-commit LCR and all LCRs before that pre-commit LCR. Then, the new leader can commit transaction T1 to the replicas of the leader's replication unit because the new leader now knows that there is consensus on the pre-commit LCR. Then, the new leader will send a post-commit LCR to the leader's followers.

[0204] Consider the following sequence of LCRs in the replicated log of the new leader:

[0205]

[0206] If the new leader receives consensus on the lead-sync LCR with a log index of 501, then the new leader knows that the number of followers in consensus have persisted LCRs with log indices up to 500. The new leader can commit transactions T1 and T2. The new leader will generate the following LCRs: and

[0207] Figure 13FIG. is a flowchart illustrating the operation of a new leader shard server taking over a replication unit when a commit initiated in the replication log exists. The operation begins (block 1301) when a commit for a transaction exists in the replication log. The leader shard server determines whether there is a consensus for the transaction commit (block 1301). If there is a post-commit LCR in the replication log for the transaction, then the leader shard server determines that there is a consensus for the transaction. If there is no consensus for the transaction (block 1301: No), then the leader shard server performs a lead-sync operation by generating a lead-sync LCR and propagating the lead-sync LCR to the followers of the leader (block 1302).

[0208] In response to a follower receiving the lead-sync LCR, the follower will request each LCR in the replication log up to the log index of the lead-sync LCR. If the follower successfully receives each LCR in the replication log up to the log index of the lead-sync LCR and persists these LCRs into the follower's replication log, then the follower returns an acknowledgment specifying the log index of the lead-sync LCR. Thereby, the leader shard server knows that the follower has acknowledged each LCR in the replication log up to and including the lead-sync LCR.

[0209] Then, the leader shard server determines whether it has received a consensus for the lead-sync LCR (block 1303). If a consensus number of followers have acknowledged the LCRs with the log index up to and including the lead-sync LCR, thereby including the pre-commit LCR for the transaction, then the leader shard server determines that there is a consensus for the transaction. If there is a consensus for the lead-sync LCR (block 1301: Yes or block 1303: Yes), then the leader shard server completes the commit operation (block 1304), and the operation ends (block 1305). In one embodiment, completing the commit operation includes generating a post-commit LCR and propagating the post-commit LCR to its followers.

[0210] If a consensus for the lead-sync LCR is not received, then the system does not initiate recovery and the leader does not generate any log records. The system only initiates recovery after a consensus on the lead-sync LCR is reached, effectively guaranteeing a consensus on each LCR before the lead-sync LCR. If there is no consensus for the lead-sync LCR (block 1303: No), then the leader shard server rolls back the transaction (block 1306). Thereafter, the operation ends (block 1305).

[0211] Replication Log Recovery

[0212] Replication log recovery is the process of making the state of the replication log match the state of the database transactions by recreating missing entries in the replication log and replaying or rolling back transactions.

[0213] According to an illustrative embodiment, a shard server determines a recovery window that defines the transactions that must be recovered. The recovery window starts from the earliest transaction that has a first LCR in the replication log but does not have a post-commit LCR. Consider the following sequence of LCRs in the follower's replication log:

[0214]

[0215] Transaction T4 has reached post-commit in the replication log; thus, T4 does not need to be recovered because the post-commit LCR for T4 indicates that the commit of T4 has received consensus. The earliest transaction that has a first LCR but does not have a post-commit LCR is transaction T1 at log index 100. Thus, the follower must recover the replication log starting from log index 100 until log index 1001 (lead-sync LCR). The recovery window is shown in bold. If a shard fails after receiving consensus for a pre-commit LCR but before generating a post-commit LCR, then there is not enough information in the replication log to infer the fate of the transactions in the failed shard. The method of the illustrative embodiment is to let the database solve this problem. If the transaction has been rolled back, then replay the transaction; otherwise, do not replay the transaction against the database.

[0216] Transaction T1 requests publish / replay / commit. Transaction T2 does not need to be processed because transaction T2 is already local. Transaction T3 requests rollback and rollback LCR because T3 has not reached the pre-commit LCR in the replication log. Transaction T4 does not need to be processed because the first LCR of T4 is outside the recovery window. Transaction T5 does not need to be processed because T5 has been rolled back. Thus, the follower can ignore transactions T2, T4, and T5 during recovery.

[0217] If the database fails in the middle of a commit, then at database restart, an election for leadership will occur. If the old leader becomes a follower because at least one follower has a pre-commit LCR, then there is a question of what happened to the transactions in the old leader. Since the database failed during the commit, the user will receive an error. According to an embodiment, the transaction will be replayed.

[0218] Figure 14FIG. is a flow chart illustrating the operation of a shard server performing replicated log recovery according to an illustrative embodiment. The operation begins at the shard server when recovering from a failure (block 1400). The shard server identifies the first transaction in the replicated log that has a first LCR but does not have a post-commit LCR (block 1401). The shard server defines a recovery window as described above (block 1402). Then, the shard server identifies the set of transactions to be recovered (block 1403). As described above, the shard server identifies transactions that have an LCR within the recovery window that can be ignored during recovery. Then, the shard server performs recovery on the identified set of transactions to be recovered (block 1404). Thereafter, the operation ends (block 1405).

[0219] DDL Execution Without Global Barrier

[0220] In a consensus-protocol-based replication method for a sharded database according to an illustrative embodiment, data definition language (DDL) operations are performed outside of the consensus protocol, i.e., DDL is not part of the DML replication stream. This is in contrast to typical database replication.

[0221] For a given table family in a sharded database management system (SDBMS), each shard contains the same schema definition. DDL operations in the SDBMS are issued on the global data service (GDS) catalog, which propagates the DDL to each shard. The consensus-protocol-based replication method of the illustrative embodiment only replicates DML asynchronously. Thus, there can be in-flight transactions in the replication pipeline that can: a) lag the schema definition in the follower when DDL is first executed for a replication unit in the follower; or b) advance the schema definition in the follower when DDL is first executed for a replication unit in the leader.

[0222] DDL is classified as barrier DDL or non-barrier DDL. Barrier DDL does not allow in-flight DML, e.g., ALTER TABLE RENAME COLUMN. The SDB is quiesced, the DDL is executed synchronously for all available shards, and DML is only allowed after the DDL execution. Non-barrier DDL allows in-flight DML, e.g., CREATE SYNONYM. Non-barrier DDL has two types: DDL that does not require any synchronization with in-flight DML (e.g., CREATE SYNONYM), and DDL that requires synchronization of delayed user DML and DDL execution in the replication stream. Some in-flight DML is still allowed, e.g., ALTER TABLE ADD COLUMN with a default value.

[0223] To handle non-barrier DDLs that require synchronization, the start and end of non-barrier DDLs must be demarcated so that SQL applications and inlined triggers enter a special mode. In the easier case, if the DDL has been executed at the leader but not at the follower, then the SQL application tries to replicate DML as safely as possible. For example, for a drop column DDL where the dropped column has a default value, the SQL application can ignore this column. Alternatively, the leader filters out this column. The SQL application stops when it encounters an error. The SQL application can be restarted when the DDL is executed at the follower. If the DML can stop the SQL application (e.g., alter table add column with a default value and insert a non-default value into this column), then the leader can raise an error.

[0224] In the more complex case, the DDL has been executed at the follower but not at the leader. In this resilient handling mode, the SQL application continues to apply. For example, for a drop column, the SQL application can ignore this column in the LCR. For adding a column with a default value, since the leader does not yet have this new column, the SQL application will not include this column in all replicated DML.

[0225] When a shard fails during DDL execution, it is not necessary to re-execute the DDL when the shard becomes available. DDL execution must be coordinated with the replication of ongoing transactions in the replication log.

[0226] Disaster Recovery

[0227] Users can place replicas in different geographical regions. For example, two replicas are placed in one region and another replica is placed in a different region. This allows the two local replicas to act as the majority for stable workloads and allows the remote replica to act as a backup for disaster recovery. When a disaster occurs, the user may need to reconfigure consensus protocol-based replication in the remote region.

[0228] Cross-Replication Unit Transaction Support

[0229] Any transaction that spans blocks will be initiated from the shard coordinator, which acts as a distributed transaction coordinator (DTC). The DTC executes the typical two-phase commit protocol among the leaders of all relevant replication units. The leader of the relevant replication unit must propagate sufficient information about this distributed transaction (e.g., global transaction ID, local transaction ID) to the followers of that leader and obtain consensus from the followers of that leader during the prepare-to-commit phase before informing the DTC that the leader is ready to commit. The DTC can only commit after receiving the prepare-to-commit messages from all relevant RU leaders. The DTC is not aware of the existence of followers. If there is a failover in a replication unit and the new leader has a prepare-to-commit record but does not have the final commit or rollback for the transaction, then the new leader of that replication unit must contact the DTC to learn the fate of such a distributed transaction. Using the two-phase commit protocol, distributed transactions introduce a greater latency in transaction response time: one more network round-trip is required. Note that replication of distributed transaction branches is asynchronous and not part of the original distributed transaction. In a typical sharded deployment, it is assumed that most transactions are within the same block, thus avoiding the two-phase commit protocol.

[0230] DBMS Overview

[0231] A database management system (DBMS) manages databases. The DBMS can include one or more database servers. A database includes database data and a database dictionary stored on a persistent memory mechanism (such as a collection of hard disks). The database data can be stored in one or more collections of records. The data within each record is organized into one or more attributes. In a relational DBMS, the collections are called tables (or data frames), the records are called tuples, and the attributes are called columns. In a document DBMS ("DOCS"), the collection of records is a collection of documents, and each document can be a data object marked up with a hierarchical markup language, such as a JSON object or an XML document. The attributes are called JSON fields or XML elements. A relational DBMS can also store hierarchically marked data objects; however, the hierarchically marked data objects are contained within the attributes of a record, such as an attribute of type JSON.

[0232] Users interact with the database servers of the DBMS by submitting commands to the database servers, which cause the database servers to perform operations on the data stored in the database. The users can be one or more applications running on a client computer that interact with the database servers. Multiple users can also be collectively referred to as users in this document.

[0233] A database command can be in the form of a database statement conforming to a database language. The database language used to express database commands is Structured Query Language (SQL). There are many different versions of SQL; some versions are standard, some are proprietary, and there are also various extensions. Data Definition Language ("DDL") commands are issued to a database server to create or configure data objects, called database objects in this document, such as tables, views, or complex data types. SQL / XML is a common extension of SQL that is used when manipulating XML data in an object-relational database. Another database language used to express database commands is Spark TM SQL, which uses a syntax based on function or method calls.

[0234] In DOCS, a database command can be in the form of a function or object method call that invokes CRUD (Create, Read, Update, Delete) operations. An example of an API for such function and method calls is MQL (MondoDB TM Query Language). In DOCS, database objects include fields defined by a JSON schema for a collection, a collection of documents, documents, or views. A view can be created by calling a function provided by the DBMS for creating a view in the database.

[0235] Changes to the database in a DBMS are made through transaction processing. A database transaction is a collection of operations that change the database data. In a DBMS, a database transaction is initiated in response to a database command requesting a change (such as a DML command requesting an update, insert record, or delete record, or a CRUD object method call requesting the creation, update, or deletion of a document). DML commands and DDL specify changes to the data, such as INSERT and UPDATE statements. DML statements or commands do not refer to statements or commands that only query the database data. Committing a transaction means making the changes of the transaction permanent.

[0236] In transaction processing, all changes to a transaction are made atomically. When a transaction is committed, either all changes are committed or the transaction is rolled back. These changes are recorded in a change log, which can include redo records and undo records (undolog). Redo records can be used to reapply the changes made to a data block. Undo records are used to reverse or undo the changes made to a data block by a transaction.

[0237] Examples of such transaction metadata include a change log that records the changes made to the database data by a transaction. Another example of transaction metadata is embedded transaction metadata stored in the database data that describes the transaction that changed the database data.

[0238] Undo records are used to provide transaction consistency by performing an operation referred to herein as a consistency operation. Each undo record is associated with a logical time. An example of logical time is the System Change Number (SCN). For example, the Lamporting mechanism can be used to maintain the SCN. For data blocks that are read to evaluate database commands, the DBMS applies the required undo records to a copy of the data block so that the copy is in a state consistent with the snapshot time of the query. The DBMS determines which undo records to apply to the data block based on the corresponding logical time associated with the undo records.

[0239] In a distributed transaction, multiple DBMSs use the two-phase commit method to commit the distributed transaction. Each DBMS executes a local transaction in a branch transaction of the distributed transaction. One DBMS (the coordinating DBMS) is responsible for coordinating the commit of transactions on one or more other database systems. The other DBMSs are referred to herein as participating DBMSs.

[0240] The two-phase commit involves two phases: the prepare-to-commit phase and the commit phase. In the prepare-to-commit phase, the branch transaction is prepared in each participating database system. When the branch transaction is prepared on a DBMS, the database is in a "ready state" such that the database can guarantee that the modifications made to the database data as part of the branch transaction can be committed. This guarantee may require the persistent storage of the change records for the branch transaction. The participating DBMS confirms when the participating DBMS has completed the prepare-to-commit phase and has entered the ready state for the corresponding branch transaction.

[0241] In the commit phase, the coordinating database system commits the transaction on the coordinating database system and on the participating database systems. Specifically, the coordinating database system sends a message to the participants requesting the participants to commit the modifications to the data on the participating database systems specified by the transaction. Then, the participating database systems and the coordinating database system commit the transaction.

[0242] On the other hand, if a participating database system cannot prepare or the coordinating database system cannot commit, then at least one of the database systems cannot make the changes specified by the transaction. In this case, all modifications at each of the participants and the coordinating database system are undone, thereby resetting each database system to the state prior to the database change.

[0243] A client can issue a series of requests to a DBMS by establishing a database session, such as requests for query execution. A database session includes a specific connection established for the client to the database server through which the client can issue a series of requests. The database session process executes within the database session and processes the requests issued by the client through the database session. The database session can generate an execution plan for a query issued by the database session client and marshal subordinate processes for the execution of that execution plan.

[0244] The database server can maintain session state data regarding the database session. The session state data reflects the current state of the session and can include the identity of the user for whom the session was established, the services used by the user, instances of object types, language and character set data, statistical data regarding resource usage for the session, values of temporary variables generated by processes executing software within the session, storage for pointers, variables, and other information.

[0245] The database server includes multiple database processes. The database processes run under the control of the database server (i.e., can be created or terminated by the database server) and perform various database server functions. The database processes include processes that run within a database session established for a client.

[0246] A database process is a unit of execution. A database process can be a computer system process or thread or a user-defined execution context, such as a user thread or fiber. A database process can also include "database server system" processes that provide services and / or perform functions on behalf of the entire database server. Such database server system processes include listeners, garbage collectors, log writers, and recovery processes.

[0247] A multi-node database management system consists of interconnected computing nodes ("nodes"), each running a database server that shares access to the same database. Typically, the nodes are interconnected via a network and share access to a shared storage device to varying degrees, e.g., shared access to a set of disk drives and the data blocks stored thereon. The nodes in a multi-node database system can be in the form of a group of computers (e.g., workstations, personal computers) interconnected via a network. Alternatively, the nodes can be nodes of a grid consisting of server blades interconnected with other server blades on a rack.

[0248] Each node in a multi-node database system hosts a database server. A server, such as a database server, is a combined allocation of integrated software components and computing resources (such as memory, nodes, and processes on the nodes for executing the integrated software components on a processor), the combination of software and computing resources dedicated to performing specific functions on behalf of one or more clients.

[0249] Resources from multiple nodes in a multi-node database system can be allocated to run the software of a specific database server. Each combination of the software and the allocation of resources from the nodes is a server referred to as a "server instance" or "instance" herein. A database server can include multiple database instances, and some or all of these database instances run on separate computers (including separate server blades).

[0250] A database dictionary can include multiple data structures that store database metadata. For example, a database dictionary can include multiple files and tables. Parts of the data structures can be cached in the main memory of the database server.

[0251] When a database object is said to be defined by a database dictionary, the database dictionary contains metadata that defines the nature of the database object. For example, the metadata in the database dictionary that defines a database table can specify the attribute names and the data types of the attributes, as well as one or more files or parts thereof that store data for the table. The metadata in the database dictionary that defines a procedure can specify the name of the procedure, the arguments of the procedure, and the return data type and the data types of the arguments, and can include the source code and its compiled version.

[0252] A database object can be defined by a database dictionary, but the metadata in the database dictionary itself may only partially specify the nature of the database object. Other natures can be defined by data structures that are not considered part of the database dictionary. For example, a user-defined function implemented in a JAVA class can be partially defined by the database dictionary by specifying the name of the user-defined function and by specifying references to the file containing the source code of the JAVA class (i.e., the.java file) and the compiled version of the class (i.e., the.class file).

[0253] A database object can have an attribute that is a primary key. The primary key contains primary key values. The primary key values uniquely identify records among the records in the database object. For example, a database table can include a column that is a primary key. Each row in the database table holds a primary key value that uniquely identifies the row among the rows in the database table.

[0254] A database object can have an attribute that is a foreign key to the primary key of another database object. The foreign key of the primary key contains the primary key value of that primary key. Therefore, the foreign key value in the foreign key uniquely identifies a record in the corresponding database object of the primary key.

[0255] A foreign key constraint based on the primary key can be defined for the foreign key. The DBMS ensures that any value in the foreign key exists in the primary key. It is not necessary to define a foreign key for the foreign key. Instead, a foreign key relationship can be defined for the foreign key. The application that populates the foreign key is configured to ensure that the foreign key values in the foreign key exist in the corresponding primary key. Even when no foreign key relationship is defined for the foreign key, the application can maintain the foreign key in this way.

[0256] Hardware Overview

[0257] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the techniques, or can include digital electronic devices (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques), or can include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage devices, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to implement the techniques. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the techniques.

[0258] For example, Figure 15 is a block diagram illustrating a computer system 1500 on which embodiments of the present invention can be implemented. The computer system 1500 includes a bus 1502 or other communication mechanism for conveying information, and a hardware processor 1504 coupled to the bus 1502 for processing information. The hardware processor 1504 can be, for example, a general-purpose microprocessor.

[0259] The computer system 1500 also includes a main memory 1506 (such as random access memory (RAM) or other dynamic storage device) coupled to the bus 1502 for storing instructions and information to be executed by the processor 1504. The main memory 1506 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1504. When such instructions are stored in a non-transitory storage medium accessible to the processor 1504, the computer system 1500 becomes a special-purpose machine customized to perform the operations specified in the instructions.

[0260] The computer system 1500 also includes a read-only memory (ROM) 1508 or other static storage device coupled to the bus 1502 for storing static information and instructions for the processor 1504. A storage device 1510 (such as a magnetic disk, an optical disk, or a solid-state drive) is provided and coupled to the bus 1502 for storing information and instructions.

[0261] The computer system 1500 may be coupled via a bus 1502 to a display 1512, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 1514, including alphanumeric keys and other keys, is coupled to the bus 1502 for transmitting information and command selections to the processor 1504. Another type of user input device is a cursor control 1516, such as a mouse, trackball, or cursor direction keys, for transmitting direction information and command selections to the processor 1504 and for controlling cursor movement on the display 1512. Such input devices typically have two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), which allows the device to specify a position in a plane.

[0262] The computer system 1500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that in combination with the computer system causes the computer system 1500 to be or program the computer system 1500 to be a special-purpose machine. According to one embodiment, the computer system 1500 performs the techniques herein in response to the processor 1504 executing one or more sequences of one or more instructions contained in the main memory 1506. These instructions may be read into the main memory 1506 from another storage medium, such as the storage device 1510. Executing the instruction sequence contained in the main memory 1506 causes the processor 1504 to perform the processing steps described herein. In an alternative embodiment, hardwired circuitry may be used in place of or in combination with software instructions.

[0263] As used herein, the term "storage medium" refers to any non-transitory medium that stores instructions and / or data that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical discs, magnetic disks, or solid state drives, such as the storage device 1510. Volatile media includes dynamic memory, such as the main memory 1506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with hole patterns, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0264] Storage media is different from transmission media but may be used in combination with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise the bus 1502. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0265] Various forms of media can participate in carrying one or more sequences of one or more instructions to the processor 1504 for execution. For example, the instructions can initially be carried on a disk or solid state drive of a remote computer. The remote computer can load the instructions into the dynamic memory of that computer and send the instructions over a telephone line using a modem. A modem local to the computer system 1500 can receive the data on the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector can receive the data carried in the infrared signal, and appropriate circuitry can place the data on the bus 1502. The bus 1502 transfers the data to the main memory 1506, from which the processor 1504 retrieves and executes the instructions. The instructions received by the main memory 1506 can optionally be stored on the storage device 1510 before or after being executed by the processor 1504.

[0266] The computer system 1500 also includes a communication interface 1518 coupled to the bus 1502. The communication interface 1518 provides two-way data communication coupled to a network link 1520 that connects to a local network 1522. For example, the communication interface 1518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 1518 can be a local area network (LAN) card that provides a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 1518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0267] The network link 1520 generally provides data communication through one or more networks to other data devices. For example, the network link 1520 can provide a connection through the local network 1522 to a main computer 1524 or to a data device operated by an Internet service provider (ISP) 1526. The ISP 1526 in turn provides data communication services through the global packet data communication network (now commonly referred to as the “Internet” 1528). Both the local network 1522 and the Internet 1528 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through the various networks and signals on the network link 1520 and through the communication interface 1518 that carry digital data to and from the computer system 1500 are example forms of transmission media.

[0268] The computer system 1500 can send messages and receive data, including program code, via one or more networks, network links 1520, and communication interface 1518. In an Internet example, the server 1530 can transmit the requested code for an application program via the Internet 1528, ISP 1526, local network 1522, and communication interface 1518.

[0269] The received code can be executed by the processor 1504 when received, and / or stored in the storage device 1510 or other non-volatile memory for later execution.

[0270] Software Overview

[0271] Figure 16 is a block diagram of a basic software system 1600 that can be used to control the operation of a computer system 1600. The software system 1600 and its components, including their connections, relationships, and functions, are merely exemplary and are not meant to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments can have different components, including components with different connections, relationships, and functions.

[0272] The software system 1600 is provided to bootstrap the operation of the computer system 1500. The software system 1600, which can be stored in the system memory (RAM) 1506 and on a fixed storage device (e.g., hard disk or flash memory) 1510, includes a kernel or operating system (OS) 1610.

[0273] The OS 1610 manages the low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more applications, represented as 1602A, 1602B, 1602C... 1602N, can be "loaded" (e.g., transferred from the fixed storage device 1510 to the memory 1506) for execution by the system 1600. Applications or other software intended to be used on the computer system 1500 can also be stored as a set of downloadable computer-executable instructions, e.g., for downloading and installation from an Internet location (e.g., a web server, app store, or other online service).

[0274] The software system 1600 includes a graphical user interface (GUI) 1615 for receiving user commands and data in a graphical (e.g., "point-and-click" or "touch gesture") manner. In turn, these inputs can be operated on by the system 1600 according to instructions from the operating system 1610 and / or one or more applications 1602. The GUI 1615 is also used to display the operation results from the OS 1610 and one or more applications 1602, based on which the user can provide additional inputs or terminate the session (e.g., log off).

[0275] The OS 1610 can execute directly on the bare hardware 1620 (e.g., one or more processors 1504) of the computer system 1500. Alternatively, a hypervisor or virtual machine monitor (VMM) 1630 can be inserted between the bare hardware 1620 and the OS 1610. In this configuration, the VMM 1630 acts as a software "buffer" or virtualization layer between the bare hardware 1620 of the computer system 1500 and the OS 1610.

[0276] The VMM 1630 instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine includes a "guest" operating system (such as the OS 1610), and one or more applications (such as one or more applications 1602) designed to execute on the guest operating system. The VMM 1630 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0277] In some cases, the VMM 1630 can allow the guest operating system to run as if it were executing directly on the bare hardware 1620 of the computer system 1500. In these instances, the same version of the guest operating system configured to execute directly on the bare hardware 1620 can also execute on the VMM 1630 without modification or reconfiguration. In other words, the VMM 1630 can provide full hardware and CPU virtualization to the guest operating system in some cases.

[0278] In other cases, the guest operating system can be specifically designed or configured to execute on the VMM 1630 for improved efficiency. In these cases, the guest operating system "is aware" that the system is executing on the virtual machine monitor. In other words, the VMM 1630 can provide para-virtualization to the guest operating system in certain cases.

[0279] A computer system process includes the allocation of hardware processor time, as well as the allocation of memory (physical and / or virtual) for storing instructions executed by the hardware processor, for storing data generated by the execution of instructions by the hardware processor, and / or for storing the hardware processor state (e.g., the contents of registers) between allocations of hardware processor time when the computer system process is not running. A computer system process runs under the control of an operating system and can also run under the control of other programs executable on the computer system.

[0280] Cloud computing

[0281] In general, the term "cloud computing" is used herein to describe a computing model that enables on-demand access to a shared pool of computing resources (such as computer networks, servers, software applications, and services), and that allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0282] A cloud computing environment (sometimes referred to as a cloud environment or simply a cloud) can be implemented in a variety of different ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes the organization's cloud services available to other organizations or the public. In contrast, a private cloud environment is generally intended for use by or within a single organization only. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private, community, or public) bound together through data and application portability.

[0283] Generally speaking, cloud computing models enable some of those responsibilities that may previously have been provided by an organization's own information technology department to instead be delivered as service layers within a cloud environment for use by consumers (either within the organization or externally, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer may vary, but common examples include: Software as a Service (SaaS), where the consumer uses software applications running on cloud infrastructure, and the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), where the consumer can use software programming languages and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, and the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), where the consumer can deploy and run any software applications, and / or provision processing, storage devices, networks, and other basic computing resources, and the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything under the operating system layer). Database as a Service (DBaaS), where the consumer uses a database server or database management system running on cloud infrastructure, and the DbaaS provider manages or controls the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0284] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the invention and what the applicant intends to claim as the scope of the invention is the literal and equivalent scope of the set of claims presented by this application in the specific form of the presented claims, including any subsequent corrections.

Claims

1. A computer-implemented method, comprising: Receiving, at a leader server, a command from a client to perform a change operation on a row of a table in a database, wherein: The row of the table is replicated across a replication group of servers such that each server within the replication group of servers stores a corresponding copy of the row of the table, The replication group of servers includes the leader server and one or more follower servers, and The leader server is configured to perform a Data Manipulation Language (DML) operation on the row of the table and replicate the DML operation to the one or more follower servers; Performing, by the leader server, the change operation on the copy of the row of the table stored at the leader server; In response to the leader server performing the change operation, creating, by the leader server, a replication log record for the change operation to be asynchronously replicated to the one or more follower servers in a replication pipeline; and In response to the leader server creating the replication log record for the change operation in the replication pipeline, returning the result of the change operation from the leader server to the client; Wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein: The command is associated with a specific transaction, The method further comprises: Receiving, at the leader server, a commit command from the client to perform a commit operation on the specific transaction; Creating, by the leader server, a replication log record for the commit operation in the replication pipeline; In response to the leader server receiving an acknowledgement that the replication log record for the commit operation has been appended to the replication logs of the consensus number of the one or more follower servers, performing, at the leader server, the commit operation on the specific transaction on the copy of the row of the table; and Returning the result of the commit operation to the client.

3. The method of claim 2, wherein a given acknowledgement received from a given follower server includes multiple transaction commits.

4. The method of claim 2, the method further comprising: In response to the leader server receiving an acknowledgement that the replication log record for the commit operation has been appended to the replication logs of the consensus number of the one or more follower servers, advancing, by the leader server, a commit index.

5. The method of claim 2, wherein performing the commit operation includes writing a log record for the specific transaction to disk.

6. The method of claim 1, wherein: The command is associated with a specific transaction, The method further comprises: Receiving, at the leader server, a commit command from the client to perform a commit operation on the specific transaction; Preparing, at the leader server, the specific transaction for commit; Marking the specific transaction as pending; Creating, by the leader server, a replication log record for the commit operation in the replication pipeline; In response to the leader server failing and becoming a new follower server, determine whether there is a consensus that the replication log record for the commit operation has been appended to the replication logs of one or more follower servers of the quorum; In response to determining that there is a consensus, perform the commit operation on the copy of the row of the table at the new follower server; and In response to determining that there is no consensus, roll back the specific transaction at the new follower server.

7. The method according to claim 1, wherein: The command is associated with a specific transaction, The method includes: Receiving, at the leader server, a commit command from the client to perform a commit operation on the specific transaction; Creating, by the leader server, a replication log record for the pre-commit operation in the replication pipeline; In response to determining that there is a consensus for the replication log record for the pre-commit operation, perform the commit operation on the specific transaction on the copy of the row of the table at the leader server and create a replication log record for the commit operation in the replication pipeline.

8. The method according to claim 7, the method includes: In response to the leader server failing and becoming a new follower server, determine whether there is a consensus for the replication log record for the pre-commit operation; In response to determining that there is a consensus, perform the commit operation on the specific transaction on the copy of the row of the table at the new follower server; And In response to determining that there is no consensus, roll back the specific transaction at the new follower server.

9. The method according to claim 1, wherein: The command is associated with a specific transaction, The method further includes: Receiving, at the leader server, a rollback command from the client to perform a rollback operation on the specific transaction; and Creating, by the leader server, a replication log record for the rollback operation in the replication pipeline.

10. The method according to claim 1, wherein: The replication log of each follower contains crossed replication log records for uncommitted transactions, and The replication log records in the replication log of each follower server have a strictly increasing log index.

11. The method according to claim 1, wherein each given follower server within the one or more follower servers eagerly performs a change operation on the corresponding copy of the row of the table of the given follower server and appends the replication log record to the corresponding replication log of the given follower server in parallel.

12. The method according to claim 1, wherein each given follower server within the one or more follower servers is configured to redirect commands from the client to the leader server.

13. The method according to claim 1, wherein the replication log record includes a logical change record for the change operation, a valid log index, and a term.

14. The method according to claim 1, further comprising invoking an additional entry remote procedure call (RPC) to propagate the replication log records in the replication log to the one or more follower servers.

15. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause: At a leader server, receive from a client a command to perform a change operation on a row of a table in a database, where: The row of the table is replicated across a replication group of servers such that each server within the replication group of the server stores a corresponding copy of the row of the table, The replication group of the server includes the leader server and one or more follower servers, and The leader server is configured to perform a data manipulation language (DML) operation on the row of the table and replicate the DML operation to the one or more follower shard servers; Perform the change operation on the copy of the row of the table stored at the leader server; In response to the leader server performing the change operation, create, by the leader server, a replication log record for the change operation to be asynchronously replicated to the one or more follower servers in a replication pipeline; And In response to the leader server creating the replication log record for the change operation in the replication pipeline, return the result of the change operation from the leader server to the client.

16. A database shard server, comprising: At least one processor; Persistent storage; And At least one memory storing instructions that, when executed by the at least one processor, cause to implement: An in-memory queue; A capture component configured to intercept operations performed by the database server in response to commands received from a client and create, in the in-memory queue, a replication log record for a change operation performed by the database server on a row of a table in a database, where: The row of the table is replicated across a replication group of servers such that each server within the replication group of the server stores a corresponding copy of the row of the table, The replication group of the server includes a leader server and one or more follower servers, The database server is the leader server for the row of the table, The database server is configured to perform a data manipulation language (DML) operation on the row of the table and replicate the DML operation to the one or more follower servers, and The database server returns the result of the change operation to the client in response to the capture component creating a replication log record for the change operation in the in-memory queue; A network sender component for each of the one or more follower servers, where each network sender component is configured to distribute the replication log record to a corresponding follower server among the one or more follower servers; and A consensus server configured to: Count the acknowledgments received from the one or more follower servers; Determine whether there is consensus for the replication log record; and In response to determining that there is consensus for the replication log record, persist the replication log record to the replication log in the persistent storage device.

17. The database shard server according to claim 16, wherein: The change operation is associated with a specific transaction, The capture component is configured to create a replication log record in the in-memory queue for the commit command received from the client for the specific transaction, The network sender component for each of the one or more follower servers is configured to distribute the replication log record for the commit command to the corresponding follower server among the one or more follower servers, The consensus server is configured to receive from the one or more follower servers an acknowledgment that the replication log record for the commit command has been persisted to the persistent storage device, and The database server is configured to perform a commit operation on the copy of the row of the table at the database server in response to the consensus server receiving acknowledgments from a consensus number of the one or more follower servers and return the result of the commit operation to the client.

18. The database shard server according to claim 16, wherein The database server is a follower server for a second row of the table, The database server includes: A network receiver configured to receive a second replication log record from the leader server of the second row of the table and add the second replication log record to a second in-memory queue, The consensus server is configured to persist the second log record to the second replication log in the persistent storage device and send an acknowledgment to the leader server of the second row of the table.

19. The database shard server according to claim 18, comprising an application server configured to read the second replication log record from the in-memory queue and apply a second change operation specified by the second replication log record to the copy of the second row of the table stored at the database server.

20. The database shard server according to claim 16, wherein the network sender component invokes an AppendEntries Remote Procedure Call (RPC) to propagate the replication log record to the one or more follower servers.

Images