Distributed serializable concurrency control scheme

Abstract

A distributed serial safety net (SSN) includes an orderer that receives a multi- shard transaction from a transaction client, assigns a commit timestamp (CTS) to the multi-shard transaction, and assigns the multi-shard transaction to at least two database shards for execution. A validator instance in each of the at least two database shards determines an order of the multi-shard transaction according to the assigned CTS and cooperates with each other to exchange local transaction validation values (e.g., eta and pi values of a serial safety net validation process) of the assigned multi-shard transaction. The validator instance in each of the at least two database shards each further performs max() and min() operations, such as the serial safety net validation process, on the received transaction validation values and local transaction validation values to determine a final transaction validation value and commit or abort the assigned multi-shard transaction based on the final transaction validation value.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 928,135, filed on October 30, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to database systems and transaction processing, and more specifically to concurrent control of databases distributed over communication networks, in order to achieve more serializable scheduling of multi-sharded transactions and single-sharded transactions. Background Technology

[0004] In database systems and transaction processing, concurrency control (CC) schemes interweave read / write requests from multiple clients, creating the illusion that each transaction has exclusive access to the data. Distributed concurrency control refers to the concurrency control of databases distributed across a communication network. Serializability ensures that scheduling concurrent transactions is equivalent to scheduling transactions to be executed serially in some order. Serializability is considered the highest level of isolation between concurrent transactions. The ideal goal of distributed databases is distributed serializability, that is, the serializability of concurrent transaction scheduling on a distributed database.

[0005] The most common distributed concurrency control schemes include two-phase locking (2PL), snapshot isolation (SI), multi-version optimistic concurrency control (MVOCC), and read committed (RC). These are also common centralized concurrency control schemes. 2PL can achieve a serializable isolation level, but RC and SI cannot. To enable non-serializable concurrency control schemes to provide a serializable isolation level, a serializable authenticator can be used. One example is serializable snapshot isolation (SSI). In their August 2017 paper, "Efficient Implementation of Serializability for (Almost) Any Concurrency Control Mechanism," Wang et al. introduced an authenticator called "Serial Safety Net" (SSN), a serializable authenticator that enables other non-serializable RC and SI concurrency control schemes to achieve a serializable isolation level while maintaining higher concurrency than the SSI scheme.

[0006] SSN assumes shared memory, where a timestamp is updated instantaneously for each version. This assumption is not valid in a distributed environment and can result in lower throughput in a distributed environment. Moreover, SSN can abort many transactions when there is a dispute over a transaction, thereby limiting the rate of valid transactions. Thus, while Wang et al. provide a fully parallel, multi-threaded, lock-free, shared-memory implementation for a multi-versioned database management system on a single multi-processor server, Wang et al. do not provide a fully distributed multi-versioned database management system. SUMMARY

[0007] Various examples are now described with reference to the drawings. Various examples are now described with reference to the drawings. The following description is provided in the context of illustrative examples, and is not intended to limit the scope of the application to the specific examples described. The description is intended only to provide an overview of certain examples of methods, apparatuses, and systems consistent with the present disclosure. The description is not intended to be comprehensive or to provide all details of the methods, apparatuses, and systems consistent with the present disclosure. The description is intended to provide a sufficient understanding of certain examples of methods, apparatuses, and systems consistent with the present disclosure to enable others to practice the methods, apparatuses, and systems consistent with the present disclosure. The description is not intended to be complete or exhaustive, and is not intended to limit the scope of the methods, apparatuses, and systems consistent with the present disclosure to the specific examples described.

[0008] In an example embodiment, a distributed SSN is provided, including an orderer that maintains an execution order and validator instances that exchange local transaction validation values with other validator instances. The orderer and the validator instances combine to achieve serializability guarantees for serializable scheduling of multi-shard transactions and single-shard transactions in a distributed database system. A coordinator is also connected with transaction clients and the distributed SSN.

[0009] In some example embodiments, a best effort hot key throttling mechanism is performed before the distributed SSN validation in each shard in the distributed database to track abort rates associated with database keys to identify hot keys. Each data item can be accessed by a key. For example, in a relational database, each row in each table can have a unique key. As another example, in an object-oriented database, each object can have a unique key. The hot key throttling mechanism tracks abort rates associated with keys of the database. When a key abort rate exceeds a threshold, such key is considered a "hot key." When there is a read operation or a write operation of a pending transaction on a hot key, the transaction will attempt to own the hot key (e.g., acquire a lock on the hot key) until the transaction aborts or commits. Read operations or write operations of any other pending transactions on the owned hot key are delayed until the owned hot key is released.

[0010] According to a first aspect of the present application, there is provided a distributed concurrency control system serializing transactions of at least one transaction client in a distributed database system having respective database shards. The system comprises at least one sequencer receiving a multi-shard transaction from the at least one transaction client, assigning a commit timestamp (CTS) to the multi-shard transaction, and assigning the multi-shard transaction to at least two database shards for execution. A validator instance in each of the at least two database shards determines an order of the multi-shard transaction according to the CTS assigned to the multi-shard transaction, and cooperates with each other to exchange local transaction validation values of the multi-shard transaction. The validator instance in each of the at least two database shards determines a final transaction validation value by performing an operation on a received transaction validation value and a local transaction validation value, and commits or aborts the multi-shard transaction based on the final transaction validation value.

[0011] According to a second aspect of the present application, there is provided a distributed concurrency control method serializing transactions of at least one transaction client in a distributed database system having respective database shards. The method comprises receiving a multi-shard transaction from the at least one transaction client, assigning a commit timestamp (CTS) to the multi-shard transaction, and assigning the multi-shard transaction to at least two database shards for execution. A validator instance in each of the at least two database shards determines an order of the multi-shard transaction according to the CTS assigned to the multi-shard transaction, and cooperates with each other to exchange local transaction validation values of the multi-shard transaction. The validator instance in each of the at least two database shards determines a final transaction validation value by performing an operation on a received transaction validation value and a local transaction validation value, and commits or aborts the multi-shard transaction based on the final transaction validation value.

[0012] In a first implementation form of any of the above aspects, the local transaction validation value comprises an eta value and a pi value of a serializable safety network validation process. The validator instances in the at least two database shards use an associative property and a commutative property of min() and max() operations of the serializable safety network validation process to update the eta value and the pi value of the multi-shard transaction.

[0013] In a second implementation form of any of the preceding aspects, the at least one sequencer comprises a sequencer in each of the at least one transaction client, the at least one sequencer time-stamps each received transaction with a CTS representing an execution order and sends the received transaction to the verifier instance in each of the at least two database shards for exchanging the local transaction verification value, the order of the received transaction being determined based on the CTS.

[0014] In a third implementation form of any of the preceding aspects, the at least one sequencer comprises a centralized sequencer in a computing unit of the distributed database system, the centralized sequencer assigns a sequence number and time-stamps each received transaction with a globally unique CTS representing an execution order and sends the received transaction to the verifier instance in the at least two database shards for exchanging the local transaction verification value, the order of the received transaction being determined based on the CTS.

[0015] In a fourth implementation form of any of the preceding aspects, a coordinator is provided, the coordinator being connected to the at least one transaction client and the at least one sequencer, the coordinator initiating transactions and processing responses to the transactions.

[0016] In a fifth implementation form of any of the preceding aspects, a transaction is identified by a unique transaction identifier (ID) and comprises at least one of a read operation, a write operation and a commit operation, wherein the read operation comprises the transaction ID and at least one key; the write operation comprises the transaction ID, at least one key and a corresponding value; the commit operation requests committing the transaction.

[0017] In a sixth implementation form of any of the preceding aspects, the coordinator determines to which database shard a data item in a single-shard transaction is sharded and sends a read-write operation request to the determined database shard bypassing the at least one sequencer, a verifier instance of the determined database shard locally assigning a CTS to the single-shard transaction.

[0018] In a seventh implementation form of any of the preceding aspects, the verifier instance of the determined database shard locally assigns the CTS to the single-shard transaction based on an interpolation of CTSs of multi-shard transactions immediately preceding and succeeding the single-shard transaction.

[0019] In an eighth implementation form of any of the preceding aspects, the verifier instance in each of the at least two database shards comprises a delay element for delaying processing of out-of-order commit operations requesting committing respective transaction operations so as to execute the commit operations in order.

[0020] In a ninth implementation form of any of the preceding aspects, when two commit requests received by a first validator instance have the same CTS, the first validator instance orders the two commit requests according to identifiers of sequencer instances from which each of the two commit requests is received.

[0021] In a tenth implementation form of any of the preceding aspects, the at least one validator instance interleaves multi-shard transactions with single-shard transactions concurrently without having overlapping read-write key sets.

[0022] In an eleventh implementation form of any of the preceding aspects, the at least one validator instance comprises an approximate membership query (AMQ) data structure that identifies when a transaction has an overlapping read-write key set with another transaction and freezes the overlapping read-write key set until all previous multi-shard transactions are processed.

[0023] In a twelfth implementation form of any of the preceding aspects, the system further comprises a first queue for validating multi-shard transactions and a second queue for validating single-shard transactions, and the at least one validator instance concurrently executes one transaction from the first queue and one transaction from the second queue.

[0024] In a thirteenth implementation form of any of the preceding aspects, when a transaction from the second queue has an overlapping read-write key set with a transaction from the first queue, the transaction from the second queue is re-queued in the second queue so that a next transaction in the second queue can be considered for execution concurrently with the transaction from the first queue.

[0025] In a fourteenth implementation form of any of the preceding aspects, the at least one validator instance processes a batch of sequential single-shard transactions in the second queue and uses one communication message for transaction validation values for the batch of sequential single-shard transactions that have non-overlapping read-write key sets.

[0026] In a fifteenth implementation form of any of the above aspects, read-write operations are individually and independently validated at each validator instance related to the transaction according to a commit protocol. An overall independent validation result is recorded at each validator instance taking into account the independent validation results of all read-write operations. The validator instances dependently validate the transaction by exchanging the local transaction validation values and dependently record the dependently validation results using the associated properties of the validation operations and the exchange properties to invalidate the recorded overall independent validation result.

[0027] Apparatuses can perform the methods, and can process instructions on a computer-readable medium, further features of which are derived from the functionality of the apparatuses. Furthermore, the explanations of the individual aspects and implementation forms apply analogously to the other aspects and the corresponding implementation forms. The different embodiments can be realized by hardware, software or any combination thereof. Furthermore, any of the aforementioned examples can be combined with one or more of the other aforementioned examples to create new embodiments, without departing from the scope of the application. BRIEF DESCRIPTION OF DRAWINGS

[0028] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. The drawings typically show the various embodiments discussed herein in example, not limitation.

[0029] Figure 1A An implementation of a fully distributed database system in an example embodiment is shown;

[0030] Figure 1B Another implementation of a fully distributed database system with one (centralized) instance of a sequencer is shown;

[0031] Figure 2A Routines in a validator instance handling commit requests in an example embodiment are shown;

[0032] Figure 2B Routines in a validator instance handling read operations initiated by a coordinator in an example embodiment are shown;

[0033] Figure 2C Routines in a validator instance handling write operations initiated by a coordinator in an example embodiment are shown;

[0034] Figure 3 A flowchart of a method of an overall commit protocol between a coordinator, a sequencer and validator instances for determining whether to abort or commit a transaction is shown;

[0035] Figure 4A A method for a distributed sequencer is shown; Figure 1ACommunication message flow of the coordinator, sequencer, and validator instance modules in the illustrated distributed database system architecture;

[0036] Figure 4B Operation of the probabilistic dependency checker for regulating contentious transactions for higher effective transaction commit rate in the example embodiment is illustrated; Figure 1B Communication message flow of the coordinator, sequencer, and validator instance modules in the illustrated distributed database system architecture;

[0037] Figure 5 Operation of the distributed SSN authenticator in the example embodiment is illustrated;

[0038] Figure 6 Operation of the probabilistic dependency checker for regulating contentious transactions for higher effective transaction commit rate in the example embodiment is illustrated;

[0039] Figure 7 A distributed concurrency control method for managing transactions in at least one transaction client in a distributed database system in the example embodiment is illustrated;

[0040] Figure 8 A block diagram of circuitry for performing the method provided by the example embodiment is illustrated. DETAILED DESCRIPTION

[0041] In the following description, reference is made to the accompanying drawings which form a part hereof, and which are shown by way of illustration of specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the application, and it is to be understood that other embodiments can be utilized and that structural, logical, and electrical changes can be made without departing from the scope of the present application. The following description of exemplary embodiments is not to be taken in a limiting sense, but is made merely for the purpose of providing an enabling teaching of the application.

[0042] In one embodiment, the functions or algorithms described herein can be implemented using software. The software comprises computer-executable instructions stored on a computer-readable medium, which can be one or more non-transitory memory devices or other types of hardware storage devices. Further, the functions can correspond to modules, which can be software, hardware, firmware, or any combination thereof. The functions described can be performed by one or more modules, and the described embodiments are merely examples. The software can be executed on a digital signal processor, an application specific integrated circuit (ASIC), a microprocessor, or other processing device, which can be local or remote from the rest of the system. For example, the software can be executed on a personal computer, server, or other computer system, which transforms the computer system into a specifically programmed machine.

[0043] There is a need to provide a fully distributed database that supports high-throughput concurrent transactions while maintaining Atomicity, Consistency, Isolation, and Durability (ACID) properties. The concurrent control scheme utilizing the SSN can ensure serializable isolation levels while achieving higher concurrency. However, as described by Wang et al., SSN is not a good distributed concurrency control scheme because the SSN verification would be a central access point, limiting the scalability of a fully distributed system. The systems and methods described herein propose a fully distributed concurrency control scheme that enables the SSN verification to be distributed to work in a geographically distributed environment.

[0044] Furthermore, the systems and methods described herein provide a best-effort hot key throttling mechanism that can be performed prior to the distributed SSN verification in respective database shards in the distributed database. The hot key throttling mechanism enables the distributed database to perform well in the presence of highly contentious concurrent transactions, where SSN would cause aborts in these cases. The systems and methods described herein can greatly improve the performance of ACID transactional database systems, including multi-core databases, clustered databases, and distributed databases, especially geographically distributed databases.

[0045] As described by Wang et al., SSN uses min() and max() functions on the timestamps of potentially conflicting transactions to detect and prevent a dependency cycle of executing transactions on a non-distributed database. Given the associative and commutative properties of the min() and max() operations used to determine the minimum and maximum values in the given values, respectively, in the SSN verification, a method is provided to subdivide the operations based on distributed local information about transactions in the database and each shard. The method ensures transaction atomicity by ordering the transactions processed in each shard, while the verification modules involved in the same transaction exchange their respective local information in a locking step. As a result, each of the verification modules obtains the same SSN verification result about the transaction, and the transaction is committed or aborted as a whole.

[0046] The systems and methods described herein also provide a hot key throttling mechanism that complements the distributed SSN verification in the presence of highly contentious concurrent transactions and improves the performance of the concurrency control scheme. SSN aborts those highly contentious transactions because they will use data that is stale due to later committed transactions. Aborts cause long delays for those transactions to be retried. The hot key throttling mechanism tracks the abort rate associated with keys of the database. When the key abort rate exceeds a threshold, such keys are considered hot keys. When there is a read or write operation of a pending transaction on a hot key, the transaction will try to own the hot key until the transaction aborts or commits. Any other read or write operation of a pending transaction on the owned hot key will be delayed until the owned hot key is released. The delay, which can be a short delay, will be preferred over abort because the delay will save the retry.

[0047] The systems and methods described herein focus on the functional components of a distributed database. In exemplary embodiments, a database generally refers to a data store that can store structured or unstructured data. Each data item can be accessed by a key, and the schema of the data item is irrelevant. For example, in a relational database, each row in each table can have a unique key. As another example, in an object-oriented database, each object can have a unique key. A transaction refers to a set of one or more operations on one or more data items that are executed as a logical unit. The data items of a distributed database are stored and distributed in one or more database shards. Each shard is treated as a separate computing and storage unit. A shard (e.g., a storage node) provides at least the following functions: serving transaction operation requests that have been dynamically mapped from other components; providing values for read operation requests according to keys; storing write values provided by write operation requests after a validation end operation (i.e., commit operation) request. Thus, a distributed transaction can operate on one or more data items that are single-shard data items (i.e., located on the same shard) or multi-shard data items (i.e., located on multiple shards). The systems and methods described herein support concurrent distributed transactions that preserve ACID properties.

[0048] The systems and methods described herein specify three related functional modules: a coordinator, a sequencer, and a verifier instance. Depending on the overall system architecture, one or more instances of the functional modules can be located in one or more components of the system. Figure 1A An implementation of the overall database system 100 is shown, in which the instances of the functional modules are highlighted. Figure 1AThe database system 100 shown in the middle is a fully distributed database system and includes a distributed sequencer configuration. One instance of the coordinator 110 is located in each transaction client 120 that requests transaction service. One instance of the sequencer 130 is present with the instance of the coordinator 110, which has the advantage of minimizing communication latency between the two functional modules. The sequencer 130 implements a management rule to dynamically map transactions to one or more validator instances 140. One validator instance 140 is located inside each storage node 150. The transaction clients 120 and the storage nodes 150 are distributed over a network 160.

[0049] Figure 1B Another implementation of the overall database system 100' is shown with a centralized sequencer configuration. One instance of the coordinator 110 is located in each transaction client 125; one instance of the sequencer 135 is located in a separate computing unit; and one instance of the validator instance 140 is located inside each storage node 150. Having one (centralized) instance of the sequencer 135 over the network 160 reduces the complexity of ensuring the order of execution and logging of the transactions, although this can extend the communication latency and reduce the system scalability.

[0050] In an exemplary embodiment, implementing the distributed SSN validation requires the presence of the sequencer 130, while requiring the collaboration of the validator instances 140, which can implement a modified version of the SSN algorithm. The presence of the coordinator 110 can complete the concurrency control scheme.

[0051] In an example embodiment, the coordinator module 110 is responsible for initiating transaction operations such as read operation requests, write operation requests, and end operation requests, and processing responses to the requests. Each transaction is identified by a unique transaction identifier (ID). A read operation should contain the transaction ID and at least one key, while a write operation should contain the transaction ID and at least one key and corresponding value. An end operation request commits the transaction. Each response from other components of the database system 100 should indicate either acceptance or rejection of the operation of the request. Rejection should result in aborting the transaction. Acceptance of a read operation or a write operation indicates that the coordinator 110 can proceed to perform the next operation. Acceptance of an end operation indicates that the transaction has been validated and serialized, and all written data items have been stored. In one embodiment, the coordinator 110 knows or understands how data items in a transaction are sharded, and can bypass the sequencer 130 / 135 as an optimization to send the read / write operation requests to the appropriate shard. For the end operation request, the coordinator 110 sends the end operation request through the sequencer 130 / 135 so that the sequencer 130 / 135 can ensure ordering of the concurrent transactions.

[0052] The sequencer 130 / 135 orders the concurrent transactions that request validation and commit in order to facilitate the functionality of the validator instances 140. The implementation of the sequencer 130 / 135 varies slightly in different system architectures, such as the architectures shown in Figure 1A and Figure 1B The sequencer 130 / 135 knows or understands how data items in a transaction are sharded and sends the commit request to the validator instance 140 in the appropriate shard.

[0053] In an example embodiment, the hot key throttling mechanism includes hot key detection and hot key ownership. The hot key detection and hot key ownership are implemented at the validator instances 140. Since there can be multiple validator instances 140, one per shard, the hot key throttling mechanism can be a fully distributed mechanism.

[0054] To implement the hot key throttling mechanism, the validator instances 140 maintain a key abort rate. The key abort rate is calculated by the number of aborted transactions using the key per duration of time (e.g., number of aborts per minute > 1, or number of keys with the most number of aborts in the last hour). When the key abort rate exceeds a specified threshold, such keys are considered hot keys, which indicates that the key has caused a large number of contentious transactions to abort.

[0055] The validator instance 140 also monitors the read-write operation requests. If a read request or a write request associated with a pending transaction involves a hot key, and the hot key is or is not owned by the transaction, the validator instance 140 assigns ownership of the hot key to the transaction and proceeds to respond to the request. If the request involves a hot key and the hot key is owned by another transaction, the validator instance 140 delays responding to the request, e.g., by re-queuing the request, until the key is no longer owned by the other transaction. At this point, the read operation request will read the latest committed key value, thereby reducing the likelihood of aborting the transaction in the SSN validation; or the write operation request will prepare the latest key value to be written, thereby reducing the likelihood of causing another transaction to abort in the SSN validation.

[0056] The validator instance 140 also monitors the status of all pending transactions. When a pending transaction's status changes to aborted or committed, the validator instance 140 releases the hot keys owned by the transaction. There can be various reasons for a transaction to abort. For example, the SSN validation can cause the transaction to fail. As another example, the transaction's read or write operation can fail.

[0057] The coordinator 110 can also use the hot key throttling mechanism to improve the performance of retrying aborted transactions. The coordinator 110 can track the abort rate of a particular transaction as well as the read-write key set of the particular transaction. In an example embodiment, a transaction's read key set includes all the read key-value tuples of the transaction, and a transaction's write key set includes all the write key-value tuples of the transaction. The read key set and the write key set can be tracked separately. When the read-write key set is identified as overlapping, then it means that the write key set overlaps or the read key set and the write key set overlap. The read key set and the read key set are generally allowed to overlap. When the abort rate of a particular transaction as well as the read-write key set of the particular transaction exceeds a specified threshold, the coordinator 110 can consider the transaction as a hot transaction, and can infer or learn that the transaction's read keys and write keys are also hot keys from the response. If the transaction is a hot transaction, the coordinator 110 can send a special read operation request for all the read keys of the transaction as well as the write keys. The special read operation request will trigger hot key throttling for all the keys to be reserved (i.e., owned) at the associated validator instance 140 before retrying the original operation of the transaction. If the special read operation request gets a positive response, the retry transaction is likely to succeed without wasting system bandwidth on non-productive work.

[0058] The sequencers 130 / 135 arrange the concurrent transactions requesting verification and submission in sequence to facilitate the execution of SSN verification at the validator instance 140. The implementation of the sequencers 130 / 135 varies slightly in different system architectures, such as... Figure 1A and Figure 1B The system architecture shown is illustrated.

[0059] Distributed sequencer 130

[0060] The distributed sequencer instance 130 is very suitable for Figure 1A The distributed database system architecture is illustrated in the example. In this embodiment, the sequencer instance 130 serves one or more coordinator instances 110. The sequencer instances 130 exchange clock synchronization messages among themselves, ensuring that their local clocks are synchronized within a specified precision.

[0061] Centralized sequencer 135

[0062] The centralized sequencer 135 is very suitable for Figure 1B The distributed database system architecture is illustrated exemplarily. In this embodiment, the sequencer instance 135 serves all coordinator instances 110.

[0063] Distributed SSN algorithm

[0064] The distributed version of the SSN algorithm is as follows: Figure 2A , Figure 2B and Figure 2C As shown. Figure 2A to Figure 2B The distributed version of the SSN algorithm shown is described with reference to the SSN algorithm described in the paper published by Wang et al. cited above. Further details of the SSN algorithm can be found in the aforementioned paper by Wang et al. In an exemplary embodiment, the coordinator 110 receives transactions with transaction IDs (T) and associated with key-value pairs (K, V) through a client application programming interface (API) of the following type:

[0065] (T) = BEGIN_TXN()

[0066] Read(T,K x V x )#T will K x The value of the identifier is read into V x middle

[0067] Read(T,K y V y )

[0068] Write(T,Kl l l l

[0069] Write(T, K m , V m )

[0070] Read(T, K z , V z )

[0071] Write(T, K n , V n )

[0072] COMMIT_TXN(T)

[0073] Figure 2A The SSN routine 200 in the validator instance 140 is shown, which handles a commit request for a part of a transaction T, denoted as T[I], executed on shard I, the transaction having a timestamp denoted as cts. The SSN routine 200 includes blocks 210, 220, 230, 235, 240, and 250. The validator instance 140 of all shards involved in the transaction will either abort the transaction or commit the part being validated. Thus, the distributed SSN routine 200 allows for distributed validation of transactions.

[0074] To handle the commit request, in block 210, the validator instance 140 first checks whether the current transaction part T[I] executed on shard I should be committed immediately or delayed until a previous transaction T' has a commit or abort result. Since the sequencer 130 / 135 has determined the order of transactions, the validator instance 140 can distinguish between previous transactions T', the current transaction T, and subsequent transactions T". The validator instance 140 first completes processing of all previous transactions T, determining their commit and abort results. The validator instance 140 also delays processing of all subsequent transactions T". If the previous and current transactions are not completed within a specified time, these transactions will be aborted.

[0075] Second, in block 220, the validator instance 140 updates the transaction validation values, i.e., pi and eta, using its shard's local metadata. As described by Wang et al., a transaction U can depend on a transaction T if U reads or overwrites a version created by T. In addition, T can anti-depend on a transaction U if U overwrites a version read by T. The dependency relationships can be represented by a directed graph G, whose vertices are committed transactions, and whose edges indicate serialization ordering relationships.

[0076] ​​​In the example embodiment, Wang et al. define pi as the commit time of the earliest successor transaction U of a transaction T that reaches through the post-edge path, where:

[0077] In this equation, c(U) and c(T) represent the commit times of transactions U and T. The b* relationship between U and T indicates that the min() function in the first version of the equation is applied to the commit times of all transactions U in the successor set at any location in the graph G. The b relationship between U and T indicates that the min() function in the second version of the equation is applied to the pi times of all transactions U that immediately succeed T in the graph G. Thus, since the second version of the equation restricts the evaluation to immediate successors rather than all successors, and the two versions of the equation are mathematically equivalent, the second version can involve a simplified computation.

[0078] Wang et al. define eta as the commit time of the most recent predecessor U of a transaction T that reaches through the pre-edge path, where:

[0079]

[0080] If pi < eta (i.e., π(T) < η(T)), an exclusion window violation occurs that leads to an abort decision. In this equation, the f relationship between U and T indicates that T depends on U. In the paper published by Wang et al., the computation of pi and eta is for the entire transaction T. However, in block 210 and block 220, the computation is distributed and performed with respect to the portion of the transaction T, the graph G, and the transactions U known to the validator instance 140 on the single shard.

[0081] Third, in block 230, the validator instance 140 multicasts its transaction validation value to all other validator instances 140 of the shards involved in the current transaction and waits to receive the transaction validation values from the other validator instances 140 of the shards involved in the current transaction. For single-shard transactions, this step is meaningless because no other shards are involved. If the validator instance 140 does not receive from all expected validator instances 140, the current transaction will time out and abort. The received data is stored in a data structure denoted as T[J], where each shard in the multi-shard transaction has a different value J.

[0082] Fourth, in block 235, the validator instance 140 updates its local transaction validation values (e.g., pi(T[i]) and eta(T[i])) using the transaction validation values received from other validator instances. Due to the associative and commutative properties of the min() and max() operations used to determine the minimum and maximum values in the given values respectively in the SSN validation, all validator instances 140 associated with the current transaction will arrive at the same final transaction validation values.

[0083] Finally, in block 240, the validator instance 140 obtains the abort or commit result and updates the local data appropriately. In block 250, any hot keys held by the validator instance 140 are released.

[0084] It can be appreciated that single-shard transactions make the exchange of local transaction validation values meaningless. Multi-shard transactions require the exchange of relevant local transaction validation values of the validator instances 140.

[0085] In the process of exchanging the local transaction validation values, the associated validator instances 140 need to wait for each other to continue. If a first validator instance 140 executes a transaction T while a second validator instance 140 executes its previous transaction T', the first validator instance 140 does not execute its subsequent transaction T" and will wait for the second validator instance 140 to execute the transaction T so that the local validation values of the transaction T, which should have been frozen and gossiped to the second validator instance 140, are not changed by the transaction T". However, if the transaction T" is a single-shard transaction, the first validator instance 140 can execute the transaction T" with confidence upon determining that the transaction T" will not change the local validation values of the transaction T. Therefore, the validator instances 140 can interleave their partial single-shard transactions between their multi-shard transactions to achieve higher concurrency.

[0086] Since the SSN validation of multi-shard transactions requires waiting for the exchange of communication messages and can stop the next multi-shard transaction in sequence, all relevant validator instances 140 of a transaction need to exchange communication messages at the same time to minimize the waiting time. The sequencer 130 / 135 can help the validator instances 140 to schedule the SSN validation of the transaction at the same time by providing a timestamp in the commit request message sent to the validator instances 140. The commit timestamp (CTS) can serve the purpose of this timestamp.

[0087] In one embodiment of the validator instance module 140, the validator instance 140 receives transaction requests from the sequencer 130 / 135. The validator instance 140 tracks the transaction requests in an input queue. Upon receiving a transaction commit request, the validator instance 140 looks at the transaction in the input queue. If the transaction is a single-shard transaction, as indicated by the lack of an associated validator instance 140 in the transaction's metadata, the validator instance 140 moves the transaction to the fast lane queue; otherwise, the validator instance 140 moves the transaction to the slow lane queue. The validator instance 140 can concurrently execute one transaction from the slow lane queue and one transaction from the fast lane queue. If the current fast lane transaction has a read-write key set that overlaps with the current slow lane transaction, the current fast lane transaction is re-queued into the fast lane queue to support the next transaction in the fast lane queue. In this regard, the execution order of the single-shard transactions can be rearranged in real time. This is acceptable because the modified SSN validation will guarantee serializability and abort pending transactions that use stale values due to the validated and committed transactions. Furthermore, the validator instance 140 can query the CTS of the next transaction in the slow lane queue to determine when to execute that transaction.

[0088] In another embodiment of the validator instance, the validator instance 140 maintains the input queue, the fast lane queue, and the slow lane queue in the same manner as described above. However, in this embodiment, the validator instance 140 can process a batch of sequential transactions in the slow lane queue. The batch processing of such sequential transactions improves concurrency because the SSN validation needs to wait for a communication message exchange to complete one transaction. Furthermore, the validator instance 140 can choose to use one communication message for the batch of transaction validation values instead of one communication message per transaction. The use of one communication message improves system throughput because communication messages can suffer from relatively high latency. The batch of sequential transactions must have read-write key sets that do not overlap with each other. If the next transaction in the slow lane queue has a read-write key set that overlaps with any transaction in the batch, the batch should be terminated and marked to exclude the next transaction.

[0089] To efficiently identify whether a transaction has a read-write key set that overlaps with another transaction or a batch of transactions, an approximate membership query (AMQ) data structure can be used. For example, a Bloom filter can be used to hold the keys of a reference transaction or a batch of transactions. The keys of a candidate transaction are then tested against the Bloom filter. A hit indicates that the candidate transaction can overlap with the read-write key set of the reference transaction or batch of transactions. Any overlapping read-write key sets are frozen until all previous multi-shard transactions have completed. It is understood that a Bloom filter can produce false positives, but not false negatives.

[0090] Figure 2B An SSN read routine 260 in the validator instance 140 is shown, which handles read operations initiated by the transaction portion T[I] and the coordinator 110 of the read or written key-value tuples' version V. The V version is locked, and the system provides the latest version V for read requests. In Figure 2B In the shown embodiment, the SSN read routine described by Wang et al. is modified to include a block 270. In operation, the validator instance 140 first checks in block 270 whether the relevant key is a hot key. If so, the validator instance 140 further checks whether the hot key is owned by other pending transactions. If none of the hot keys are owned by other pending transactions, the validator instance 140 owns such hot keys and proceeds by validating whether they are invalid, and then responding to the read operation. If one of the hot keys is owned by other pending transactions, the validator instance 140 delays responding to the read operation until a timeout or all hot keys are released.

[0091] In addition to the read transaction portion T[I], Figure 2B The SSN read routine 260 shown in the middle also receives as a parameter a reference to the appropriate version V returned by the underlying concurrency control algorithm. In operation 272, the transaction portion T[I] can record in T[I].pstamp the maximum v.cstamp reflecting the transaction portion T[I]'s dependency on the version creator. In operation 274, T[I] records in t.sstamp the minimum v.sstamp to record anti-dependencies read from transactions that override V, if any. If the version has not been overridden, as shown in operation 274, the version is added to the transaction portion T[I]'s read set and checked for delayed overrides during pre-commit. The transaction portion T[I] then validates the exclusion window in operation 276 and aborts upon detecting a violation. The transaction portion T[I] can then transition to an aborted state.

[0092] Figure 2CAn SSN write routine 280 in the validator instance 140 is shown, which handles a write operation initiated by the coordinator 110 for transaction portion T[I] and version V, where V refers to a new version generated by the transaction portion T[I]. In operation 290, the validator instance 140 first checks if the relevant key is a hot key. If so, the validator instance 140 further checks if the hot key is owned by other pending transactions. If none of the hot keys are owned by other pending transactions, the validator instance 140 owns such hot keys and proceeds to validate them for invalidity before responding to the write operation. If one of the hot keys is owned by other pending transactions, the validator instance 140 delays responding to the write operation until a timeout or all such hot keys are released.

[0093] In Figure 2C When updating version V, the transaction portion T[I] updates its predecessor timestamp t.pstamp with v.prev.pstamp in operation 292, and then uses v.prev.pstamp instead of v.prev.cstamp. Then, in operation 294, the transaction portion T[I] can record V in its write set for final validation at pre-commit. If more reads are received later, the transaction portion T[I] can update t.stamp with v.prev.pstamp, which is updated by a read operation received after T[I], but installs new version V before the transaction portion T[I] enters pre-commit. In addition, V will also be removed from the read key set of the transaction portion T[I] (if there is one), because using edge update pi(T[I]) will violate the exclusion window of the transaction portion T[I] and trigger unnecessary abort. As described in the paper by Wang et al., V can be removed from the read key set of a transaction by skipping the processing of V when checking the read key set, without making the read key set searchable.

[0094] Data item shards can be implemented in a variety of ways. For example, the keys can be sorted and partitioned into ranges, and a subset of the ranges can be deterministically mapped to a shard. The coordinator 110, the sequencer 130 / 135, and the validator instance 140 can evaluate the mapping from key to shard without coordination. Alternatively, there can be a shard manager that centrally determines the mapping, and the coordinator 110, the sequencer 130 / 135, and the validator instance 140 can query the mapping of the shard manager and learn of changes, which are expected to happen infrequently.

[0095] The commit log is a collection of records of committed transactions. The sequencer 130 / 135 can use the commit log to keep track of history and help discover failures. Typically, all committed writes are written to the commit log before being assigned, so that a transaction that was running at the time a shard storage node was shut down can be recovered and reassigned by examining the commit log. With a centralized sequencer 135, the commit log can be centralized at one node, such as the sequencer 135. Alternatively, the commit log can be composed of fragments that are dispersed across multiple nodes, such as the coordinator instances 110 or the verifier instances 140. Each record in the commit log should contain temporal or order information about the committed transaction so that the reassignment order of the transaction can be determined correctly.

[0096] The commit log can also contain transactions that have approved their read-write operations and are waiting for verification. Recording pending transactions that have passed all their read-write operations can help the verifier instances 140 recover from failures, thus quickly recovering verification of the pending transactions. The log messages in the commit log are called commit intent messages.

[0097] Figure 3 A flowchart of a method 300 of the overall commit protocol between the coordinator 110, the sequencer 130, and the verifier instances 140 for determining whether to abort or commit a transaction is shown. The method 300 includes operations 310, 320, 330, 340, 350, and 360.

[0098] First, in operation 310, each read-write operation of a transaction initiated by the coordinator instance 110 will be separately and independently approved at each of the verifier instances 140 related to the transaction. That is, each verifier instance 140 uses its local metadata without relying on the other verifier instances 140 to determine whether to abort the transaction or approve the operation. The coordinator 110 collects the results, and can abort the transaction if one of the results is to abort.

[0099] Second, in operation 320, the overall result of the previous step is recorded. In an exemplary embodiment, the log message can be a commit intent message stored in the commit log of the sequencer instance 130 when the coordinator 110 determines to proceed to send an end operation request to the sequencer instance 130.

[0100] Third, in operation 330, the sequencer instance 130 / 135 can request those verifier instances 140 to verify the transaction mutually dependently by exchanging local verification parameters about the transaction. The verifier instances 140 should reach the same verification result.

[0101] Fourth, in operation 340, the verification result is recorded. In an exemplary embodiment, the sequencer instances 130 / 135 record the submission message, thus invalidating the submission intent message.

[0102] In operation 350, the write data of the pending transaction is stored in the storage node 150 associated with the validator instance 140. The storage node 150 makes the write data invisible to other concurrent transactions. Once the validator instance 140 changes the transaction state to committed, the storage node 150 makes the write data visible to other concurrent transactions. If the transaction with the invisible write data is aborted, garbage collection of the invisible write data is performed in operation 360.

[0103] Figure 4A It shows Figure 1A The diagram illustrates the communication message flow of the coordinator, sequencer, and validator instance modules in the distributed database system architecture. The coordinator instance 110 can bypass the sequencer 130 and generate read / write operation requests 400 to the validator instance 140 associated with the keys in the transaction. If any read / write operation request is not satisfied, the coordinator instance 110 aborts the transaction. Otherwise, the coordinator 110 generates a termination operation request 410 to the sequencer instance 130.

[0104] The sequencer instance 130 concurrently receives end-of-operation requests from one or more coordinator instances 110. When the sequencer instance 130 appends a Commit Request (CTS) to each end-of-operation request and forwards it as a commit request to the relevant validator instance 140, the sequencer instance 130 assigns the CTS at 420 based on its local clock to the end-of-operation request representing the expected execution order at the relevant validator instance 140. The CTS also helps the validator instance 140 maintain multiple versions of data items.

[0105] The validator instance 140 may receive out-of-order submit requests from one or more sequencer instances 130. This could be because the clocks of the sequencer instances 130 may not be fully synchronized, or because communication messages from the sequencer instances 130 may arrive at the validator instance 140 asynchronously. Therefore, the validator instance 140 will not immediately perform the SSN verification after receiving the submit request. Instead, the validator instance 140 may delay for a specified time interval at 430, anticipating that the submit request with the lower CTS may arrive late and be executed in the correct order. When two submit requests have the same CTS, they should come from two different sequencer instances 130, and they are ordered using the identifier of the sequencer instance 130 as a connection interruptor.

[0106] The validator instance 140 must abort a multi-shard transaction with a commit request to be validated whose CTS is lower than the CTS of the current multi-shard transaction that is in progress for the SSN validation. Aborting the transaction at the validator instance 140 will cause the transaction to be aborted at the other validator instances 140 associated with the transaction because the latter will not receive a transaction validation value from the validator instance 140 and will cause the transaction to time out.

[0107] The sequencer instance 130 receives one or more responses from the one or more related validator instances 140 of the transaction. All responses should be consistent, indicating that all responses are either commit or abort. Thus, one positive response is sufficient to trigger the sequencer instance 130 to append the transaction to its commit log at 440.

[0108] The advantage of having the distributed sequencer 130 is database scalability. As the number of transaction clients 120 increases, more sequencer instances 130 can be added since the number of coordinator instances 110 increases. The disadvantage is clock synchronization, whose accuracy affects the amount of latency, which in turn affects out-of-order commit requests.

[0109] Figure 4B A communication message flow is shown for the centralized sequencer 135 Figure 1B A communication message flow is shown for the centralized sequencer 135

[0110] The sequencer 135 concurrently receives end operation requests 460 from one or more coordinator instances 110. When the sequencer 135 appends a sequence number and a CTS to the end operation requests 460 and forwards the requests as commit requests to the related validator instances 140, the sequencer 135 assigns the sequence number and the CTS to each request at 470 that represents the expected execution order at the related validator instances 140.

[0111] The sequence number helps the validator instances 140 detect any missing communication messages in case the communication channel is unreliable. The CTS also helps the validator instances 140 maintain multiple versions of data items. Since the centralized sequencer 135 is the single source of the sequence number and the CTS, using sequence numbers or using CTS is sufficient to identify the execution order of commit requests and support the SSN validation.

[0112] The sequencer 135 receives one or more responses from the one or more related validator instances 140 of the transaction. All responses should be consistent, indicating that all responses are either commit or abort. Thus, one positive response is sufficient to trigger the sequencer 135 to append the transaction to its commit log at 480.

[0113] An advantage of this embodiment is that having only one sequencer instance 135 makes it easier to ensure that all validator instances 140 have the same order of concurrent transactions, and that each validator instance 140 will not receive out-of-order commit requests assuming the communication channel is reliable. Furthermore, the sequencer 135 can easily reorder concurrent transactions as it knows all concurrent transactions in the distributed database system 100 to optimize the throughput of the validator instances 140. Moreover, the sequencer 135 can implement a centralized hot key throttling mechanism as it knows the metadata of all transactions.

[0114] However, a disadvantage of having a centralized sequencer 135 is that the database scalability is limited. As the number of coordinators 110 associated with transaction clients 125 in the database increases, the load on the centralized sequencer 135 can increase.

[0115] It is contemplated that the stress on the centralized sequencer 135 can be mitigated by having only multi-shard transactions go through the sequencer 135. Single-shard transactions can be offloaded by having the coordinator instance 110 send commit requests directly to the associated validator instance 140. In this case, the validator instance 140 can locally assign the CTS to the single-shard transaction based on the interpolation of the CTS of the multi-shard transactions immediately before and after it.

[0116] By having one sequencer instance 135 per database, overall system scalability can be achieved. When the distributed database system 100 hosts multiple databases or multiple tenants, there can still be many sequencer instances 135.

[0117] Furthermore, it is understood that the systems and methods described herein can greatly improve the performance of database transaction systems, including multi-core databases, cluster databases, and distributed databases, especially geographically distributed databases. Databases as used herein include all database systems that require ACID properties, such as storage systems, data stores, etc.

[0118] Figure 5The operation of a distributed Serial Safety Net (SSN) authenticator in an exemplary embodiment is illustrated. As described above, sequencers 130 / 135 are used to determine the order of cross-sharded transactions, including, for example, transactions T1, T2, T3, ..., where T1 is the first in time. The sequencers 130 / 135 record the cross-sharded transactions T1, T2, T3, ... in a log 510 for replay in case of failure. The log 510 may be physically located in the authenticator instance 140 or in the sequencers 130 / 135. The sequencers 130 / 135 distribute the corresponding transactions to the corresponding shards (e.g., shard S with keys a, b, c). abc S fragments with keys l, m, n lmn A partition S with keys x, y, z xyz Each shard includes a validator instance 140. In an exemplary embodiment, the sequencers 130 / 135 locally record the commit order of cross-shard transactions and single-shard transactions in each shard, enabling each participating shard to independently make the same abort / commit decision without further synchronization. Furthermore, multiple cross-shard transactions can be grouped to minimize cross-shard communication of the following metadata: pi (commit time of the earliest successor transaction of a transaction arriving via the back edge path) and eta (commit time of the most recent predecessor transaction of a transaction arriving via the front edge path). As described above, single-shard transactions can be scheduled independently when batching cross-shard transactions. In this case, a batch of cross-shard transactions can be scheduled as one entity within a participating shard.

[0119] Figure 5 Steps (1) to (4) for processing the cross-sharded transactions T1, T2, and T3 are shown.

[0120] In step (1), as Figure 2A As shown in code block 220, the validator instance 140 uses the local metadata of each shard to update the validation values ​​pi and eta of transaction T1. As illustrated, each distributed SSN of transaction T1 computes pi and eta of its own shard via the access key {a,l,z} (e.g., eta{T}). sabc For a slice S including keys a, b, and c abc Transactions within; pi{T sxyz For a partition S containing keys x, y, and z xyz Transaction T in the middle.

[0121] In step (2), transaction T2 reads key {b,c,y,z} and writes it to key {c,y}. For example... Figure 2AAs shown in code block 230, the validator instance 140 multicasts its transaction validation value to all other validator instances 140 involved in the current transaction. This message exchange is performed in a single round in step (2).

[0122] In step (3), the validator instance 140 waits to receive the transaction verification values ​​from other validator instances 140 involved in the current transaction. The validator instance 140 uses the received transaction verification values ​​to update its local transaction verification value. Due to the association and commutative properties of the min() and max() operations used to determine the minimum and maximum values ​​of the given values ​​in the SSN verification, all validator instances 140 associated with the current transaction will obtain the same final transaction verification value. In this case, π = 40, η = 32. Therefore, step (3) corresponds to Figure 2A The code in block 235 is "#finalize pi(T)and eta(T)".

[0123] Finally, in step (4), the verifier instance 140 obtains the abort or commit result and updates the local data appropriately, which corresponds to Figure 2A Block 240 is shown. Transaction T3 accesses the key {a,c,l} that includes the updated key {c}.

[0124] For fairness reasons, a rate limiting mechanism can be introduced to balance the throughput of single-shard and cross-shard. Figure 6 The operation of the probabilistic dependency checker of validator instance 140, which adjusts disputed transactions to obtain a higher efficient transaction commit rate, is illustrated in an exemplary embodiment.

[0125] like Figure 6 As shown, transactions T1, T2, T3, ... are examined at validator instance 140 to determine whether they are controversial. For example, transaction T1 is examined at 610, 620, and 630 to determine whether it is controversial by, for example, determining whether transaction T1 has a key that overlaps with another transaction. If not, transaction T1 is forwarded to shard S. abc S lmn and S xyz Transaction T1 has been assigned to the shard for processing. On the other hand, when transaction T1 is determined to be controversial at 610, 620, and / or 630, transaction T1 is also distributed to the hotkey throttling mechanism 640 to throttle transactions T2 and T3 in the correct order for assignment to the corresponding shards. For example, by the corresponding shard S lmn and S xyzThe transactions T2 and T3 that are processed can be held by the throttle 640 to allow the transactions Tl to be processed in the correct time order.

[0126] As described above, the hot key throttling mechanism tracks the abort rate associated with keys of the database to identify hot keys. When there are read or write operations of pending transactions on a hot key, the transactions will attempt to own the hot key until the transaction aborts or commits. Any other read or write operations of pending transactions on the owned hot key will be delayed until the owned hot key is released. The transactions can then be processed in the correct order.

[0127] Figure 7 A distributed concurrency control method 700 for managing transactions in at least one transaction client in a distributed database system having respective database shards is shown in an example embodiment. As shown, the distributed concurrency control method 700 includes receiving, in operation 710, multi-shard transactions and single-shard transactions from at least one transaction client. In operation 720, the multi-shard transactions and single-shard transactions are assigned to one or more database shards for execution while maintaining an execution order of the multi-shard transactions and single-shard transactions. In operation 730, a key related to a contentious multi-shard transaction is frozen such that the contentious multi-shard transaction cannot execute. In operation 740, validator instances 140 in the one or more database shards to which the transactions are assigned cooperate with each other to exchange local transaction validation values of the assigned multi-shard transactions and single-shard transactions by multicasting the frozen key and local transaction validation values to each shard processing a multi-shard transaction. In operation 750, execution of the contentious multi-shard transaction is delayed until all prior multi-shard transactions complete execution. In operation 760, the key of the contentious multi-shard transaction is released after all prior transactions complete execution, and the contentious multi-shard transaction executes in the execution order determined by the commit timestamp on the respective transaction. Thus, the at least one sequencer and validator instances 140 work in concert to enable serializable scheduling of the multi-shard transactions and single-shard transactions in the distributed database system. In an example embodiment, the validator instances 140 concurrently interleave multi-shard transactions with single-shard transactions without overlapping read-write key sets for processing. Figure 1A The distributed sequencer or Figure 1B The centralized sequencer of the illustrated embodiment assigns the multi-shard transactions and single-shard transactions to one or more database shards for execution while maintaining an execution order of the multi-shard transactions and single-shard transactions. In operation 730, a key related to a contentious multi-shard transaction is frozen such that the contentious multi-shard transaction cannot execute. In operation 740, validator instances 140 in the one or more database shards to which the transactions are assigned cooperate with each other to exchange local transaction validation values of the assigned multi-shard transactions and single-shard transactions by multicasting the frozen key and local transaction validation values to each shard processing a multi-shard transaction. In operation 750, execution of the contentious multi-shard transaction is delayed until all prior multi-shard transactions complete execution. In operation 760, the key of the contentious multi-shard transaction is released after all prior transactions complete execution, and the contentious multi-shard transaction executes in the execution order determined by the commit timestamp on the respective transaction. Thus, the at least one sequencer and validator instances 140 work in concert to enable serializable scheduling of the multi-shard transactions and single-shard transactions in the distributed database system. In an example embodiment, the validator instances 140 concurrently interleave multi-shard transactions with single-shard transactions without overlapping read-write key sets for processing.

[0128] The distributed concurrency control method can further include individually and independently validating, at each validator instance 140, read-write operations related to a transaction according to a commit protocol, and recording, at each validator instance 140, an overall independent validation result upon considering all independent validation results of all read-write operations. The validator instances 140 can dependently validate the transaction by exchanging local validation parameters. The validator instances 140 can also record the dependently validated result to invalidate the recorded independent validation result.

[0129] The concurrency control techniques described herein can enable more serializable scheduling of multi-shard transactions and single-shard transactions in a distributed database system. The serializable scheduling is provided by recording the cross-shard transaction order to enable recoverable deterministic serializable validator computation. This means that each shard can reach the same conclusion about the transaction result if no cross-shard communication is used to negotiate commit or abort decisions. Multiple cross-shard transactions can be grouped to minimize cross-shard communication between these transactions because the inputs need to be synchronized only once and all further computations are deterministic. Controversial cross-shard transactions are detected locally proactively and scheduled by a scheduler to improve effective transaction throughput with minimal overhead.

[0130] Thus, the distributed concurrency control techniques described herein improve serializable validators such as SSN by providing efficient concurrency even in a distributed environment. Concurrency control systems such as the system described by Thomson et al., “Calvin: Fast Distributed Transactions for Partitioned Database Systems,” in Proceedings of the ACM SIGMOD International Conference on Management of Data, Pages 1-12, May 2012, and the system described by Ren et al., “SLOG: Serializable, Low-Latency, Geo-Replicated Transactions,” in Proceedings of the VLDB Endowment, Volume 12, Issue 11, use deterministic sequencers and deterministic execution but require read operations / write operations to be known in advance. The concurrency control systems described herein do not require read operations / write operations to be known in advance and can support dynamic execution. Thus, the concurrency control systems described herein combine the advantages of SSN with the advantages of deterministic validation for use in a distributed environment.

[0131] Figure 8A block diagram showing circuitry for performing the method provided by the example embodiments is shown. In various embodiments, not all components need be used.

[0132] Figure 8 An example computing device in the form of a computer 800 is shown, which can include a processing unit 802, a memory 804, a removable storage device 806, and a non-removable storage device 808. While the example computing device is illustrated and described as computer 800, the computing device can have different forms in different embodiments. For example, the computing device can be a smartphone, a tablet, a smartwatch, or other computing device including the same or similar elements as those shown and described. Smartphones, tablets, and smartwatches, among other devices, are commonly referred to as mobile devices or user devices. Furthermore, while various data storage elements are illustrated as part of the computer 800, the storage devices can also or alternatively include cloud-based storage devices or server-based storage devices accessible via a network, such as the Internet. Figure 8

[0133] The memory 804 can include volatile memory 810 and nonvolatile memory 812. The computer 800 can also include or access a computing environment. The computing environment includes various computer-readable media, such as the volatile memory 810 and non-volatile memory 812, removable storage device 806, and non-removable storage device 808. Computer storage includes random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0134] ​The computer 800 can also include or access a computing environment that includes an input interface 814, an output interface 816, and a communication interface 818. The output interface 816 can include a display device that can also function as an input device, such as a touchscreen. The input interface 814 can include one or more of a touchscreen, a touchpad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated within or coupled to the computer 800 through a wired or wireless data connection, and other input devices.

[0135] The computer 800 can operate in a networked environment using the communication interface 818 to connect to one or more remote computers, such as a database server. The remote computer can include a personal computer (PC), a server, a router, a network PC, a peer device or other common network node, and so on. The communication connection(s) through the communication interface 818 can include a local area network (LAN), a wide area network (WAN), cell phone network, WiFi, Bluetooth, or other networks. According to an embodiment, various components of the computer 800 are connected by a system bus 820.

[0136] Computer-readable instructions stored on a computer-readable medium that can be executed by the processing unit(s) 802 of a computer 800, such as a program 822. In some embodiments, the program 822 includes software that, when executed by the processing unit(s) 802, performs operations in accordance with any of the embodiments contained herein. Hard disks, CD-ROMs, and RAM are some examples of non-transitory computer-readable media that include components of a storage device, among others. The terms “computer-readable medium” and “storage device” do not include a carrier wave that is deemed to be too transitory. Storage devices can also include networked storage devices, such as a storage area network (SAN). The computer program 822 can be used to cause the processing unit(s) 802 to perform one or more methods or algorithms described herein.

[0137] It should also be understood that software, including one or more computer-executable instructions, can be installed on and / or sold with one or more servers or databases and can facilitate the processes and operations described above with reference to any or all of the steps of the present application. Alternatively, the software can be obtained and loaded into one or more servers or one or more databases in a manner consistent with the present application, including by obtaining the software from a server that does not own or use the software, e.g., from a server that owns the software creator or from a server that does not own the software creator but uses the software. For example, the software can be stored on a server for distribution over the Internet.

[0138] Moreover, those skilled in the art will appreciate that the present application is practiced with other embodiments or in various ways. Furthermore, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," or "having" and variations thereof herein is intended to be equivalent to the term "consisting of" to mean the inclusion of the recited item or items and nothing more. "Including" is not limited to the recited items or groups and can include items or groups not explicitly recited.

[0139] Components of the illustrative devices, systems and methods employed in accordance with the illustrated embodiments can be implemented, at least partially, in digital electronic circuitry, analog electronic circuitry, or computer hardware, firmware, software, or combinations thereof. These components can also be implemented as a computer program product, such as a computer program tangibly embodied in an information carrier or machine-readable storage device, for execution by a programmable processor, a computer or multiple computers, or the like.

[0140] The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be run on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. Also, functional programs, codes, and code segments for accomplishing the functions described herein can be easily understood by programmers skilled in the art to which the present application pertains. The steps of a method associated with the illustrative embodiments can be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. The steps can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, e.g., an FPGA or an ASIC, without departing from the scope of the present application.

[0141] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0142] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes, or is operatively coupled to receive data from or transfer data to, or in

[0143] Those skilled in the art will appreciate that information and signals can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0144] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention. Software modules can reside in random access memory (RAM), flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. Sample storage media are coupled to a processor such that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. In other words, the processor and the storage medium can reside in an integrated circuit or be implemented as discrete devices.

[0145] As used herein, "machine-readable medium" means a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., EEPROM), and any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media capable of storing processor instructions (e.g., centralized or distributed databases, or associated caches and servers). The term "machine-readable medium" should also be understood to include any medium (or combination of media) capable of storing instructions executable by one or more processors, such that, when executed by the one or more processors, the instructions cause the one or more processors to perform any one or more methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" as used herein does not include signals by itself.

[0146] Although a few embodiments have been described in detail above, other modifications are possible. For example, the logic flows described above can not require the particular order described or sequential order to achieve desirable results. Other steps can be provided, or steps can be eliminated, from the described flows, and other components can be added to, or removed from, the described systems. Other embodiments can be within the scope of the following claims.

Claims

1. A distributed concurrency control system, characterized by, Serializing transactions for at least one transaction client in a distributed database system having corresponding database shards, the distributed concurrency control system comprising: at least one sequencer receiving a multi-shard transaction from the at least one transaction client, assigning a commit timestamp (CTS) to the multi-shard transaction, and assigning the multi-shard transaction to at least two database shards for execution; a validator instance in each of the at least two database shards for performing operations comprising: determining an order of the multi-shard transaction according to the CTS assigned to the multi-shard transaction; cooperating with other database shards of the at least two database shards to exchange local transaction validation values of the multi-shard transaction; determining a final transaction validation value by performing operations on the local transaction validation values and received transaction validation values; the transaction validation values comprising eta and pi values of a serializable network validation process; committing or aborting the multi-shard transaction based on the final transaction validation value.

2. The system of claim 1, wherein, the local transaction validation values comprising eta and pi values of a serializable network validation process, the validator instances of the at least two database shards using associativity and commutativity properties of min() and max() operations of the serializable network validation process to update the eta and pi values of the multi-shard transaction.

3. The system of claim 1, wherein, the at least one sequencer comprising a sequencer in each of the at least one transaction client, the at least one sequencer time-stamping each received transaction with a CTS representing an execution order, and sending the received transaction to the validator instance in each of the at least two database shards for exchanging the local transaction validation values, the order of the received transaction being determined based on the CTS.

4. The system of claim 1, wherein, the at least one sequencer comprising a centralized sequencer in a computing unit of the distributed database system, the centralized sequencer assigning sequence numbers, time-stamping each received transaction with a globally unique CTS representing an execution order, and sending the received transaction to the validator instance in each of the at least two database shards for exchanging the local transaction validation values, the order of the received transaction being determined based on the CTS.

5. The system of claim 1, wherein, further comprising a coordinator connected to the at least one transaction client and the at least one sequencer, the coordinator initiating transactions and processing responses to the transactions.

6. The system of claim 5, wherein, transactions are identified by a unique transaction identifier (ID) and comprise at least one of a read operation, a write operation, and a commit operation, wherein the read operation comprises the transaction ID and at least one key; the write operation comprises the transaction ID, at least one key, and a corresponding value; the commit operation requests committing the transaction.

7. The system of claim 6, wherein, The coordinator determines to which database shard to shard a data item in a single-shard transaction, and sends read-write operation requests to the determined database shard bypassing the at least one sequencer, a validator instance of the determined database shard locally assigns a CTS to the single-shard transaction.

8. The system of claim 7, wherein, The validator instance of the determined database shard locally assigns the CTS to the single-shard transaction based on interpolation of CTSs of multi-shard transactions immediately before and after.

9. The system of claim 6, wherein, The validator instance in each of the at least two database shards comprises a delay element for delaying processing of out-of-order end operations that request committing respective transaction operations, so as to execute the end operations in order.

10. The system of claim 9, wherein, When two commit requests received by a first validator instance have the same CTS, the first validator instance orders the two commit requests according to identifiers of sequencer instances from which each of the two commit requests is received.

11. The system of claim 1, wherein, At least one validator instance concurrently interleaves multi-shard transactions with single-shard transactions without overlapping read-write key sets; The system further comprises a first queue for validating multi-shard transactions and a second queue for validating single-shard transactions, the at least one validator instance concurrently executes one transaction from the first queue and one transaction from the second queue, the at least one validator instance processes a batch of sequential single-shard transactions in the second queue, the batch of sequential single-shard transactions having non-overlapping read-write key sets.

12. The system of claim 11, wherein, The at least one validator instance comprises an approximate membership query (AMQ) data structure that identifies when a transaction has an overlapping read-write key set with another transaction and freezes the overlapping read-write key set until all previous multi-shard transactions are processed.

13. The system of claim 11, wherein, When a transaction from the second queue has an overlapping read-write key set with a transaction from the first queue, the transaction from the second queue is re-queued in the second queue so that the next transaction in the second queue can be considered concurrently with the transaction from the first queue.

14. The system of claim 11, wherein, The at least one validator instance processes a batch of sequential single-shard transactions in the second queue and uses one communication message for transaction validation values of the batch of sequential single-shard transactions.

15. The system of claim 1, wherein: read-write operations are individually and independently validated at each validator instance associated with a transaction according to a commit protocol; overall independent validation results are recorded at each validator instance considering independent validation results of all read-write operations; the validator instances dependently validate the transaction by exchanging the local transaction validation values; the validator instances use associative and commutative properties of validation operations to dependently record the dependent validation results, invalidating the recorded overall independent validation results.

16. A method of distributed concurrency control, the method comprising: Serializing transactions for at least one transaction client in a distributed database system having respective database shards, the distributed concurrency control method comprising: receiving a multi-shard transaction from the at least one transaction client; assigning a commit timestamp (CTS) to the multi-shard transaction; assigning the multi-shard transaction to at least two database shards for execution; a validator instance in each of the at least two database shards determining an order of the multi-shard transaction according to the CTS assigned to the multi-shard transaction; the validator instance in each of the at least two database shards cooperating with other database shards in the at least two database shards to exchange local transaction validation values of the multi-shard transaction; the validator instance in each of the at least two database shards determining a final transaction validation value by performing operations on the local transaction validation values and received transaction validation values; the transaction validation values comprising eta and pi values of a serializable snapshot validation process; the validator instance in each of the at least two database shards committing or aborting the multi-shard transaction based on the final transaction validation value.

17. The method of claim 16, wherein, the local transaction validation values comprising eta and pi values of a serializable snapshot validation process, the method further comprising the validator instance in the at least two database shards using associativity and commutativity properties of min() and max() operations of the serializable snapshot validation process to update the eta and pi values of the multi-shard transaction.

18. The method of claim 16, wherein, Further comprising: stamping each received transaction with a CTS representing an order of execution and sending the received transaction to the validator instance in each of the at least two database shards to exchange the local transaction validation values, the order of the received transaction determined based on the CTS.

19. The method of claim 16, wherein, Further comprising: using an approximate membership query (AMQ) data structure to identify when a transaction has overlapping read-write key sets with another transaction, freezing overlapping read-write key sets until all previous multi-shard transactions are processed, and at least one validator instance concurrently interleaving multi-shard transactions with single-shard transactions without overlapping read-write key sets; the system further comprising a first queue for validating multi-shard transactions and a second queue for validating single-shard transactions, the at least one validator instance concurrently interleaving multi-shard transactions with single-shard transactions comprising: the at least one validator instance concurrently executing one transaction from the first queue and one transaction from the second queue, the at least one validator instance processing a batch of sequential single-shard transactions in the second queue, the batch of sequential single-shard transactions having non-overlapping read-write key sets.

Images