Scalable database system based on quorum quantity

Abstract

Techniques relating to a database system are disclosed herein. The database system includes a plurality of coordinator nodes that store copies of partitions. Each partition describes the state of the lock and transaction of the keys covered by the keys of the partition. Each partition is in turn replicated. A plurality of coordinator nodes receive requests for a key grant lock from a plurality of worker nodes to allow the worker nodes to write records for the key as part of executing a transaction. A given coordinator node of the plurality of coordinator nodes sends an approval response to the lock to at most one of the worker nodes. The single worker node obtains the lock in response to receiving an approval response from a majority of the plurality of coordinator nodes, and the plurality of worker nodes do not obtain the lock in response to neither receiving the approval response from the majority of the plurality of coordinator nodes.

Description

Technical Field

[0001] The public text as a whole relates to database systems, and more specifically to the various mechanisms used to implement scalable database systems based on a quorum. Background Technology

[0002] Modern database systems typically manage systems that allow users to store collections of information in an organized manner that can be efficiently accessed and manipulated. In some cases, these management systems maintain a log-structured merge tree (LSM tree), which consists of multiple levels, each storing information from database records as key-value pairs. A database system may include persistent storage housing the LSM tree and database nodes with in-memory buffers. During operation, the database nodes initially write records to the in-memory buffer and then flush the records to persistent storage. As part of flushing records, the database nodes write them to a new file stored at one of the multiple levels of the LSM tree. Over time, as records move down the LSM tree, they are rewritten to new files stored at lower levels. Attached Figure Description

[0003] Figure 1 This is a block diagram illustrating exemplary elements of a system with database, worker nodes, and transaction coordinator services according to some implementation schemes.

[0004] Figure 2A and Figure 2B This is a block diagram illustrating an example of how, according to some implementation schemes, requests affecting partitions are processed within the time provided by the worker node.

[0005] Figure 3 This is a block diagram illustrating an example where, according to some implementation schemes, multiple transactions attempt to acquire a lock but none of them win the conflict.

[0006] Figures 4A-4B This is a block diagram illustrating exemplary elements of partitioning operations that can be performed by a coordinator node according to some implementation schemes.

[0007] Figure 5 This is a block diagram illustrating an example of a worker node interacting with a quorum of multiple partitions during the relocation of partition copies according to some implementation schemes.

[0008] Figure 6 This is a flowchart illustrating an exemplary method, according to some implementation schemes, related to a coordinator node ensuring the transaction consistency of transactions executed by worker nodes.

[0009] Figure 7This is a block diagram illustrating the elements of a computer system for implementing the various systems described in the public text, according to some implementation schemes. Detailed Implementation

[0010] Many database systems handle transactions based on specific guarantees that ensure transaction consistency. Read Committed Snapshot Isolation (RCSI) is one example. RCSI is a guarantee that all reads performed within a transaction will see a consistent snapshot of the database system, and the transaction will only successfully commit if its updates do not conflict with any concurrent updates performed since the snapshot. Specifically, a transaction can contain one or more database statements, and in RCSI, a transaction can be executed one statement at a time. Each statement acquires the snapshot time to use as it reads, locks, and updates records; the transaction can read records committed up to the snapshot time. If a statement performs an update, the database system ensures that the record being updated has not been modified since the statement's snapshot time—the transaction can acquire the snapshot time at the transaction level, and therefore the database system can ensure that when the transaction is committed, the record being updated has not been modified since the transaction's snapshot time.

[0011] Adherence to RCSI imposes scalability limitations on database systems, limitations inherent in RCSI characteristics. For example, if multiple transactions attempt to write records for the same database key at relatively equal times, these transactions may block each other. As an example, one of these transactions might acquire a lock on the key (allowing it to write records for the key), preventing other transactions from acquiring locks on the key. As another example, in the event of conflicting updates, transactions can undergo rollback, where all changes made within the transaction's statements are undone, restoring the state to the beginning of the statements. Thus, in certain situations, transactions can be stopped and / or at least partially rolled back. If these situations occur infrequently, even at large scales, the database system can scale considerably—that is, if transactions do not contend to update the same records, RCSI may not impose any limitations on database scalability.

[0012] However, while RCSI may impose database scalability limitations on database systems, modern database system implementations that support RCSI have artificial scalability limitations due to design choices. Implementations of database systems that support RCSI (Multi-Version Concurrency Control (MVCC)) today can be categorized into three prototypes: single-writer, shared-writer, and partitioned-writer. MVCC is a method in which a database system stores multiple versions of records. In a typical popular implementation of MVCC, the current version has a "home" location that holds the most recently committed version of the record, or possibly an uncommitted version. The home of a record can be a server, a data structure (e.g., a B+ tree), and / or a block. Updating a record occurs by first moving any older versions of the record elsewhere and recording where the record can be found. Then, a new version of the record, now committed (or pending), is stored at home. In a single-writer database system, record changes originate from a single master server, typically supported by secondary servers performing read-only transactions. In a shared-writer database system, multiple master servers are involved in generating record changes. In a partitioned writer database system, the database is divided into partitions, and each partition can only be updated by one master server.

[0013] These prototypes have limitations on database scalability. Modern single-writer database systems scale poorly because they rely on a single master server to update records. Modern shared-writer database systems scale poorly because multiple servers must compete to coordinate data structures to make changes to records. That is, to update a specific record, a transaction must acquire exclusive access to the record's home part (e.g., a B+ tree). This leads to infighting, competition, and coordination between update and read transactions. Furthermore, read transactions often have to search for the location of older versions of records, which requires coordinated access. Modern partitioned-writer database systems scale poorly because write operations to the database system can be mismatched with the partitioning of the keys being written to—partitions become "hot" and cause performance issues. Moving record keys from one partition to another is very complex and impacts application availability. Therefore, partitioned-writer database systems cannot quickly adapt to changes in write operations. Additionally, a single transaction can update records across partitions, so these database systems implement two-phase commit to address this situation. However, two-phase commit can slow down these systems, especially when it is executed during repartitioning. Many of the limitations of these database prototypes stem from in-situ update methods, where updating records at the record's own location involves complex coordination between transactions. Therefore, these implementations result in cross-transaction coordination, which further restricts scalability on top of those inherent to RCSI. Among other things, the published text addresses the technical problem of how to implement a database system that overcomes one or more of the aforementioned drawbacks.

[0014] In the various implementations described below, the system includes a database, worker nodes, and a transaction coordinator service, which includes coordinator nodes that ensure the transactional consistency of database transactions executed by those worker nodes. Uncommitted records, most recently committed records, and locks may be stored in partitions divided by key ranges (e.g., one of these partitions may store data for the key range "AC"). A given partition may be managed by a quorum of coordinator nodes, each of which may store a copy of the given partition. In various implementations, when a worker node executes a database transaction, it interacts with one or more quorums of coordinator nodes to store work and obtain approval for actions to execute the database transaction (e.g., acquiring a lock for a key to write a record for that key). If the database transaction involves writing records for key ranges distributed across different partitions, then if these partitions are stored by different quorums, the worker node may interact with the corresponding quorum of coordinator nodes corresponding to those different partitions. Interactions between worker nodes and coordinator nodes may include requesting a record lock that allows the worker node to write a record for a specific key, requesting permission to commit the transaction, requesting a snapshot of the transaction, etc.

[0015] In various implementations, coordinator nodes process requests from worker nodes based on data stored in their replicas. For example, a worker node might request a record lock for a key from a quorum of coordinator nodes corresponding to a partition associated with a key range. Each coordinator node might check its replica for the relevant partition to determine if any existing record locks conflict with the requested record lock and respond to the worker node based on its findings (e.g., granting the lock if no conflict exists). In various implementations, to acquire a record lock, snapshot time, and / or commit a record, a worker node must obtain approval from at least a majority of the quorum of coordinator nodes (in some cases, approval may need to be obtained from a majority of multiple quorums). By requiring majority agreement from a quorum of coordinator nodes, this quorum-based approach ensures that at most one transaction wins. However, when multiple transactions are competing for the same objective, it is not necessarily necessary to determine a winner. Consider an example where there is a quorum of three coordinator nodes, and two transactions are competing to acquire a lock for a specific key. One of these transactions may acquire a lock from one coordinator node, another transaction may acquire a lock from another coordinator node, and a third coordinator node may not respond to these transactions. Since neither of these two transactions has acquired a lock from a majority of the three coordinator nodes, in various implementations, both transactions are rolled back (e.g., restored to the state at the beginning of the statement) because neither transaction has won.

[0016] When a worker node seeks to commit a database transaction, in various implementations, it issues a commit request to the relevant quorum corresponding to the partitions touched by the database transaction. The commit request may specify a future time when the quorum of coordinating nodes will process the commit request. When the given quorum of coordinating nodes arrives at its time (relative to its local clock), it may prepare uncommitted work for commit and provide a response to the worker node. As part of the preparation, the coordinating node may check for conflicts between the database transaction and other database transactions committed between the start of the database transaction (which may be limited by a snapshot time) and the processing of the commit request. If a conflict exists, the coordinating node may reject the request to commit the database transaction. However, if no conflict exists, the coordinating node may approve the commit. In various implementations, before committing the transaction in its own worker log, the worker node waits for at least a majority of the relevant quorum of coordinating nodes to approve the commit and confirms the result with those coordinating nodes that triggered the database transaction and / or the client application. However, if insufficient approval is obtained, the worker node may roll back at least a portion of the database transaction and may attempt to commit again.

[0017] In various implementations, recently committed work is stored in a partition managed by the transaction coordinator service and at the worker nodes. However, over time, committed work can become stored in the system's LSM tree, where it is merged down the LSM tree over time. Therefore, versions of records can circulate within the database system over time. Thus, in various implementations, records do not have a "headquarters" location (e.g., a page or B+ tree) storing multiple versions of the record. Therefore, updates to records do not involve locking the headquarters location, and transactions updating records can avoid conflicts with transactions attempting to obtain the latest version of the record committed before the transaction's snapshot. Due to this non-in-place update method and the characteristic that at most one transaction can win a majority of approvals for actions (e.g., acquiring a lock on a key) from a quorum of partitions, partitions can be repartitioned without halting transactions.

[0018] Due to load skew and key range skew, partitions may have to be modified. As mentioned above, in various implementations, uncommitted / committed records and record locks are divided into key range partitions by key, which are replicated across a quorum of coordinator nodes. Over time, some partitions may become too large, for example, due to a large number of writes received by a particular key range. To handle load skew and key range skew, coordinator nodes can perform three different operations relative to their replicas: split, merge, and reposition.

[0019] • For split operations, in various implementations, the key range of the replica is split into multiple smaller key ranges. Records and / or locks can be copied to a new location on the coordinator node, but removed from the coordinator node. The operation can produce multiple partitions and thus multiple replicas.

[0020] • For merge operations, in various implementations, two or more subranges are merged into a key range. Records and / or locks for those merged partitions can be stored in a common location on the coordinator node—multiple partitions can become one partition.

[0021] For relocation operations, in various implementations, a new replica is created on another coordinator node, and the data from the old replica (being relocated) is copied to the new replica. While data is being copied, the coordinator node of the new replica can process new incoming traffic and use the results (e.g., the allocation of new record locks) to update the new replica. Once the data has been copied, the old replica can be removed. In some cases, a coordinator node can split a partition into multiple partitions and relocate one or more of these partitions to another coordinator. In this way, the coordinator node can dynamically discard storage and traffic.

[0022] • When failure conditions are unlikely to cause competing transactions to fail simultaneously, repartitioning and quorum are easier. When a failure occurs, it is not necessary to perfectly determine which transaction won the contention for the lock. This significantly reduces implementation complexity and recovery robustness. It also makes repartitioning easier.

[0023] Relocation can add complexity because worker nodes may have to check multiple locations to find the answer, especially when determining whether another worker node has acquired a lock for the key. Relocation can also be considered in quorum architectures, where a given partition resides as a replica across multiple coordinator nodes that collectively act as the quorum.

[0024] These techniques may outperform existing methods because they enable scalable database systems without one or more of the limitations found in the database implementations of the prototypes discussed. For example, these techniques provide database systems with better scalability than single-writer databases because the database system can include multiple worker nodes capable of performing update transactions, and therefore each of those worker nodes can be a writer. As discussed, shared-writer database systems include multiple writers, and records have a home location where multiple versions of the record can be found. Updating a record involves acquiring exclusive access to the record's home location, which can lead to infighting, competition, and coordination between update and read transactions. However, records in the disclosed database system do not have a home location, so update and read transactions do not conflict with each other because update transactions do not have to lock the location where the current or previous version of the record exists, and vice versa. Previously committed values ​​of records can be read from the coordinator (most recent) or the LSM (older). By avoiding a single home location for records, reading past content is separated from new updates.

[0025] For partitioned writer database systems, in-situ updates are also implemented (records have a home location). Therefore, moving data from one partition to another is very complex and impacts application availability. Specifically, it is necessary to prevent key updates in the old partition, move the data from the old partition to the new partition, and then enable key-range updates in the new partition. This approach requires strict control over the transition. In contrast, because records in the disclosed database system do not have a home location, and a quorum majority is used to ensure that at most one transaction wins in any conflict, partitions can be easily moved between different coordinator nodes (and can also be split and merged) without problems, as the quorum approach ensures that worker nodes interact with both locations and will observe any work that may exist in either location. This approach ensures that at most one transaction wins in a conflict. Therefore, the disclosed technical solution provides a database system that does not have one or more of the artificial scalability limitations found in the prototype database implementation. Therefore, the disclosed database system can scale to a larger size than other database implementations while still meeting the requirements of the database system.

[0026] Now go to Figure 1 A block diagram of system 100 is shown. System 100 includes a set of components that can be implemented via hardware or a combination of hardware and software routines. In the illustrated embodiment, system 100 includes a database 110, worker nodes 120, and a transaction coordinator service 130. As further shown, database 110 includes a merge tree (LSM tree) 115 of a log structure, and transaction coordinator service 130 includes transaction coordinator partitions 135A-C. The illustrated embodiment can be implemented in a different manner than shown. For example, system 100 may include a directory service that maintains metadata (e.g., topology information) that can be used by worker nodes 120 to access data within system 100.

[0027] In various implementations, system 100 implements platform services (e.g., customer relationship management (CRM) platform services) that allow users of the services to develop, run, and manage applications. System 100 may be a multi-tenant system providing various functionalities to users / tenants hosted by a multi-tenant system. Therefore, system 100 can execute software routines from various different users (e.g., the provider and tenants of system 100), and provide code, web pages, and other data to users, databases, and other entities associated with system 100. In various implementations, system 100 is implemented using cloud infrastructure provided by a cloud provider. Therefore, database 110, worker node 120, and / or transaction coordinator service 130 can leverage the available cloud resources (e.g., storage resources, network resources, etc.) of the cloud infrastructure to facilitate their operation. For example, the software used to implement worker node 120 may be stored on non-transitory computer-readable media on server-based hardware included in the cloud provider's data center and executed in virtual machines hosted on the server-based hardware. Components 110, 120, and 130 may be implemented without the aid of virtual machines or other deployment technologies such as containerization. In some implementations, system 100 is implemented using local or private infrastructure, as opposed to a public cloud.

[0028] In various implementations, database 110 is a collection of information organized in a manner that allows access, storage, and manipulation of information. Therefore, database 110 may include supporting software (e.g., storage nodes) that allows worker nodes 120 to perform operations (e.g., access, storage, etc.) on the information stored at database 110. In various implementations, database 110 is implemented using one or more storage devices connected together on a network (e.g., a storage attached network (SAN)) and configured to redundantly store information to prevent data loss. The storage devices can persistently store data, and therefore database 110 can be used as persistent storage for system 100. Records written to database 110 by worker nodes 120 can be accessed by other worker nodes 120. In various implementations, records may be stored in an LSM file as part of an LSM tree 115 implemented at database 110.

[0029] In various implementations, a record is a key-value pair that includes data and a key that can be used to look up the record. For example, a record may correspond to a row in a database table and specify the value of one or more fields of the database table. In various implementations, records are immutable, so a new record is written to update the underlying database construct (e.g., a row in a database table). Therefore, a database construct can be associated with multiple records, each a different version of the database construct. These records can be referred to as “record versions” of the database construct. As an example, a first record (first record version) may initially be written to store specific values ​​of rows in a database table, and a second record (second record version) may subsequently be written to update one or more of those values ​​of rows in the database table. Both records may be accessible using the same key. Unless otherwise specified, the term “record version” may be used interchangeably with the term “record”.

[0030] In various implementations, the LSM tree 115 is a data structure that stores files (with records) in an organized manner using a hierarchical scheme. A hierarchy is a storage area in which a set of records is stored. In some implementations, the hierarchy corresponds to different types of storage devices (e.g., solid-state drives (SSDs), hard disk drives (HDDs), etc.), where lower hierarchies may correspond to slower devices with higher storage capacities. For example, the top hierarchy of the LSM tree 115 may be implemented using random access memory, the next set of lower hierarchies may be implemented using SSDs, and the remaining lower hierarchies may be implemented using HDDs. In various implementations, during operation, records are flushed from worker nodes 120 to the LSM tree 115, where records are “merged” down the LSM tree 115 over time by copying records from higher hierarchies to lower hierarchies and then removing records from those higher hierarchies. Because records are merged down the hierarchy, newer records in higher hierarchies replace records in lower hierarchies. For example, two records can be written for the same underlying data construct (e.g., rows in a database table), where one of these records is written later to overwrite one or more values ​​included in the other record. Because the former record is written later, it resides at a higher level than the latter record. When performing a single-key search to locate the latest record version of the key, worker node 120 can traverse the levels of LSM tree 115 starting from the top level until it encounters the record version of the key, and then return the record version, which is the latest record version due to the properties discussed above.

[0031] In various implementations, LSM tree 115 includes older committed work, while most recently committed work may reside in transaction coordinator partition 135 and / or worker node 120. Therefore, when records are committed by worker node 120, there can be time periods in which those records are not located in LSM tree 115 because the storage nodes have not yet consumed worker log 125 of worker node 120 to make those records available for storage in LSM tree 115. As a record "ages out," it can circulate within system 100 (e.g., copied to a new location and deleted from an old location)—for example, a record may first appear at worker node 120, then at transaction coordinator partition 135, and finally at LSM tree 115, where it can continue to move down the hierarchy of LSM tree 115 by merging. Therefore, in various implementations, the location of records is time-dependent, so multiple versions of records with the same database construct (e.g., rows in a table) can reside throughout system 100 (e.g., the latest version may reside in transaction coordinator partition 135, while another version may reside in LSM tree 115). Thus, there may not be a single home location (e.g., a memory page or B+ tree) where multiple versions of records with the same database construct reside.

[0032] In various implementations, worker node 120 provides various database services, such as data storage, data retrieval, and / or data manipulation. In various implementations, worker node 120 is a set of software routines capable of executing on hardware, while in some implementations, worker node 120 encompasses both hardware and software routines. Database services can be provided to other components within system 100 or to components outside system 100. For example, a database connection (e.g., a Java Database Connectivity (JDBC) connection) can be established between worker node 120 and an application node (not depicted) to perform database transactions. Application nodes can provide various services to users (e.g., CRM services) and communicate with worker node 120 to implement the results of interactions with those users (e.g., to update records). Therefore, application nodes can interact with worker node 120 to perform database transactions. In various implementations, a database transaction is a logical unit of work to be performed with respect to a database (e.g., a specified set of database operations / statements). For example, processing a database transaction may include executing an SQL SELECT statement to select one or more rows from one or more database tables. The content of a row can be specified in a record, and therefore worker node 120 can return one or more records corresponding to one or more rows. In various implementations, a transaction is executed by a single worker node 120; the transaction may not be distributed (where multiple worker nodes 120 participate in the execution of a single transaction).

[0033] Executing a database transaction may involve worker node 120 writing one or more records to database 110. These records may have two main types: data records and log records. Data records may include data and keys that can be used to look up the data records—for example, a data record may include data for rows of a table. Log records may describe one or more database operations (e.g., record insertion) performed as a result of executing the database transaction. Thus, in various implementations, worker node 120 writes data records to update values ​​in database objects (e.g., tables) of database 110 and records changes to log records stored in worker log 125. Worker node 120 may initially write records (e.g., data records and / or log records) to its local memory cache and one or more transaction coordinator partitions. When the cache becomes full or fills up at periodic intervals, worker node 120 may flush committed records to database 110. As part of the record refresh process, in various implementations, worker node 120 writes records to a new file at the top level of LSM tree 115, and as time goes on, when newer files are written to the top level, the records of these files are merged down the hierarchy.

[0034] Before committing and refreshing records, in various implementations, worker node 120 communicates with transaction coordinator service 130 to ensure that these records do not conflict with record writes performed by other worker nodes 120. As used herein, the phrase "commit transaction" or "record" is used according to its well-known meaning and refers to the process that makes changes made during a transaction visible outside the entity that performed the transaction. When a transaction is committed, its records may be associated with a transaction commit number indicating a time period during the database operation. As discussed below, the transaction commit number can be used to determine when these records were written, and therefore can be used to determine which records were written before or after these records. (Relative to...) Figure 2A and Figure 2B An example of worker node 120 interacting with transaction coordinator service 130 to commit records is discussed in more detail.

[0035] Worker node 120 may further communicate with transaction coordinator service 130 to obtain the snapshot time of the database transaction when a database transaction begins. Worker node 120 may also obtain the snapshot time of database statements within the database transaction. In various embodiments, the snapshot time is a value indicating which records can be read by worker node 120 for use in a database transaction or database statement. In various embodiments, system 100 increments the transaction commit number over time (i.e., as progress is made within the database system), and the snapshot time may correspond to one of the values ​​of the transaction commit number. In some embodiments, the snapshot time is a time value (e.g., a time window covering milliseconds) or a numerical value indicating a time period. Records with transaction commit numbers less than (or in some cases equal to) the snapshot time may be read by worker node 120. For example, a snapshot time "445" may be assigned to a transaction, and therefore worker node 120 may read records with transaction commit numbers less than or equal to 445 for the transaction. The snapshot time may also be used to ensure that a transaction does not commit records for a key that conflict with another record for that key that was written or committed after the snapshot time.

[0036] In various implementations, System 100 implements read committed isolation, where a transaction (or more precisely, a database statement) only sees records committed before it begins, and not uncommitted data or changes committed during the execution of concurrent transactions. Each statement within a transaction can acquire a snapshot time for use when it reads, locks, and updates records. When a statement performs an update, System 100 ensures that the key written to it by the statement has not been written to by other transactions since the statement's snapshot time. In the event of a conflicting update, all changes made by the statement are undone, and the statement is rolled back. A new snapshot time later than the conflicting update can then be acquired, and the statement's work can be retried. For example, suppose transaction T2 acquires a snapshot time S2 and attempts to write a record for the key. If, after time S2, another transaction T1 has committed a record for the key, transaction T2 experiences a statement rollback to roll back the relevant statement. After restarting the statement, transaction T2 can acquire a new snapshot time that includes the updates from transaction T1.

[0037] In various implementations, the transaction coordinator service 130 ensures that changes made by worker nodes 120 do not violate read committed isolation (as discussed above) when database transactions are executed concurrently on worker nodes—the transaction coordinator service 130 can facilitate the execution of database transactions and help ensure transactional consistency. In various implementations, the transaction coordinator service 130 includes multiple coordinator nodes forming one or more coordinator quorums / clusters (also referred to as “partition quorums”). Coordinator nodes can be software executing on hardware, or they can encompass both hardware and software. The number of coordinator nodes in the quorum can vary between implementations—for example, the quorum can include three coordinator nodes, five coordinator nodes, seven coordinator nodes, etc. In various implementations, the coordinator nodes maintain transaction coordinator partitions 135.

[0038] In various implementations, transaction coordinator partition 135 includes information that can be used to ensure that concurrent changes made by worker node 120 do not violate read committed isolation when the worker node executes a database transaction. In various implementations, transaction coordinator partition 135 includes information about uncommitted updates (e.g., records of uncommitted transactions), most recently committed updates, and record locks granted to transactions. As shown, transaction coordinator partition 135 can correspond to different key ranges and include the aforementioned information for their respective key ranges. As an example, transaction coordinator partition 135A can be used to store record locks for records whose keys fall within key range AC (i.e., records whose keys begin with A, B, or C). In addition to key ranges, partition 135 can also be divided by time. Therefore, a coordinator node can store multiple partitions 135 for the same key range, but each partition 135 belongs to a different time frame (e.g., a different range of transaction commit numbers).

[0039] To ensure robustness and enable dynamic relocation for system 100 scaling, multiple instances of the transaction coordinator partition 135 are maintained in various implementation schemes (in...). Figure 1 (Depicted as replica 137). Different replicas 137 of partition 135 can be stored at different, respective coordinator nodes of the transaction coordinator service 130, such that if a coordinator node becomes unresponsive or otherwise unavailable, information within the transaction coordinator partition 135 can be accessed from another replica 137 on another coordinator node. The coordinator node can also manage replicas 137 from different partitions 135 (e.g., a coordinator node can manage replicas 137 of partition 135A and replicas 137 of partition 135C), and partition 135 can be distributed across different partition quorums. (As relative to...) Figure 4A and Figure 4BIn more detail, the coordinator node can perform operations to split, merge, and / or relocate partitions 135 based on different conditions experienced by system 100. The coordinator node can perform these operations to handle load balancing and key range skew, and therefore can dynamically discard storage and traffic. By being able to split, merge, and relocate partition replicas, the transaction coordinator service 130 can be easily scaled, as new coordinator nodes can be added to service 130 and replicas from existing coordinator nodes can be provisioned, thereby reducing the workload on these existing coordinator nodes. Relative to Figure 5 An example in which replicas are relocated to different coordinator nodes is discussed in more detail.

[0040] While a group of coordinator nodes may be part of the same quorum or corresponding copies 137 of coordinator partition 135 processing the same transaction, in various implementations, these coordinator nodes may not always communicate with each other when performing operations on their respective copies 137. Therefore, a coordinator node may process a request based on its own knowledge and perspective, independent of the knowledge and perspective of the other coordinator nodes in its quorum. Thus, a copy 137 of partition 135 may not be an exact copy; instead, each copy 137 may represent the knowledge of its corresponding coordinator node, derived from the request observed and processed by the coordinator node. As an example, two coordinator nodes may receive a request from worker node 120 to acquire a lock for a key. One of these coordinator nodes may successfully process the request and store information about the existence of the lock in its copy 137 for that particular partition 135, while the other coordinator node may fail to process the request, and therefore its copy 137 for partition 135 may not store information about the existence of the lock. Therefore, the coordinator node may store an incomplete view of partition 135.

[0041] However, in various implementations, worker node 120 must obtain approval from at least a majority of the coordinator nodes in the quorum of the partition before proceeding. (In some cases, worker node 120 may have to obtain approval from multiple quorums associated with the same partition 135, particularly when the partition is being relocated). Therefore, at least a majority of the coordinator nodes in the quorum of the partition will store information related to the worker node's request in their copy 137, such as information about locks that have been assigned. Although coordinator nodes may not communicate with each other while processing requests, in various implementations, coordinator nodes periodically communicate with each other to learn about the information stored in another coordinator node's partition copy 137. For example, if a majority of the quorum has granted a lock to worker node 120, but a particular coordinator node in the quorum is unaware of the lock, it can learn about the lock from other coordinator nodes and thus update its copy 137 to include information about the lock. As another example, if a coordinator node crashes and a new coordinator node is enabled, the new coordinator node can communicate with the other coordinator nodes in its quorum to recreate a copy 137 of the specific partition 135.

[0042] Throughout the execution of a transaction, in various implementations, worker node 120 interacts with coordinator nodes to ensure that its actions do not interfere with the actions of other worker nodes 120. When interacting with a group of coordinator nodes, worker node 120 may send requests (e.g., to acquire a record lock or commit the transaction) to these coordinator nodes (e.g., some or all of them), and then wait for an approved response from at least a majority of the coordinator nodes in this group before proceeding. For example, worker node 120 seeking to acquire a record lock for a key falling within the key range DE may send a request to the group of coordinator nodes managing copy 137 of partition 135B to acquire the record lock. In various implementations, the coordinator node checks for conflicts between the request submitted by worker node 120 and information in the relevant copy 137 managed by the coordinator node (e.g., approved locks, committed records, uncommitted records, etc.). Identified conflicts may lead the coordinator node to provide an objection response to worker node 120. If worker node 120 does not receive a sufficient number of approved responses (e.g., approved responses from a majority of the coordinators in a relevant set of coordinators), worker node 120 may continue along a different path (e.g., roll back the statement and retry). Relative to Figure 3 An example is discussed in more detail where multiple worker nodes 120 attempt to acquire a lock on the same key for their respective transactions. In this example, no worker node 120 is able to obtain a majority approval, and therefore no transaction wins the lock, which could result in both transactions being partially rolled back or aborted.

[0043] Now go to Figure 2A The figure is a block diagram illustrating an example of a coordinator node of partition 135 processing a request at a specific time. During the execution of a database transaction, worker node 120 may send a request to a specific coordinator node of the transaction coordinator service 130 to approve a set of requested actions (e.g., obtaining approval to commit a database transaction, obtaining a snapshot time, obtaining a lock for the primary key, etc.). In the illustrated embodiment, worker node 120 sends a request to the coordinator nodes (not shown) of partitions 135A-C to approve, for example, the commit of a database transaction involving each partition 135 during its execution. Worker node 120 proposes a commit time “T-09”. In various embodiments, the proposal time (the time when the coordinator node can process the request) is selected by worker node 120 based on its knowledge of the local time identified by the coordinator node's local clock.

[0044] In various implementations, the local clock identifies the local time observed by a node (e.g., worker node 120 or coordinator node). In various implementations, a node's local clock may indicate a transaction commit number or another value indicating the state of the database system—that is, the time value may not be the actual time (e.g., 7 pm), but rather a logical construct that can be used to represent or identify the forward progress in the database system. In various implementations, nodes perform specific database operations based on the time of their local clocks. As an example, coordinator nodes may process requests from worker node 120 at specific times (i.e., when their local clocks reach these times, which may be the times requested by these worker nodes). In various implementations, worker node 120 and coordinator node each have their own view of the current time within system 100 based on their local clocks. However, the local clocks of worker node 120 and / or coordinator node may become out of sync with each other over time. Therefore, worker node 120 may choose a future time further in the future than the local time of most coordinator nodes for each partition 135 involved in the request.

[0045] When the coordinator node arrives Figure 2AAt the proposal time "T-09", in various implementations, the coordinator node determines whether the requested action conflicts with any information in the relevant replica 137 up to the proposal time. For example, a coordinator node with a replica 137 for transaction coordinator partition 135A can determine whether a database transaction executed at worker node 120 has produced a record that conflicts with a record approved for commit between the transaction's snapshot time (or statement's snapshot time) and the proposal commit time. If a conflict exists, the coordinator node rejects the request, which may cause the worker node to roll back part or all of the database transaction if worker node 120 does not receive a majority of approvals from partition 135 (e.g., from the coordinator node with replica 137 for partition 135A). However, if there is no conflict from the coordinator node's perspective, it approves the request and can update its replica 137 to reflect the approval. In response to a majority of approvals from each partition 135, worker node 120 can then proceed to commit transactions within its worker log 125.

[0046] Now go to Figure 2B The figure is a block diagram illustrating an example of processing a request at a specific time. As shown, partitions 135A-C each include a set of replicas 137 that process requests from worker node 120 at time "T-09". In some cases, a given replica 137 may be further away in time than T-09, and therefore worker node 120 may not receive an approval response from the coordinator node of the replica. In some cases, the coordinator node may slow down significantly, making it arrive at T-09 in an untimely manner. In some cases, replica 137 may have an incomplete view and return a response that does not reflect the majority of the actual state of the database system. In all these cases, in various implementations, worker node 120 continues its operation in response to receiving approval from at least a majority of the replicas 137 of each partition 135 involved in the request. For example, worker node 120 may request a lock for a key falling within the key range of partition 135A, and therefore, if worker node 120 receives an approval response from a majority of the coordinator nodes of replicas 137 of partition 135A, it can continue with the lock.

[0047] Now go to Figure 3The diagram illustrates an example of multiple transactions attempting to acquire a lock. In the illustrated implementation, worker nodes 120A and 120B and partition quorum 310 including coordinator nodes 320A-C are present. Furthermore, as shown, coordinator nodes 320A-C each include replicas 137A-C of partition 135, and worker nodes 120A and 120B execute transactions 305A and 305B. The illustrated implementation can be implemented differently. For example, multiple partition quorum 310s may exist for the same partition 135, allowing worker nodes 120A and 120B to interact with multiple partition quorum 310s to acquire locks for keys to write records for transactions.

[0048] In various implementations, the partition quorum 310 is a group of multiple coordinator nodes 320, each storing a copy 137 of a specific partition 135. When an action affecting partition 135 is performed (e.g., acquiring a lock on a key to write a record for that key), in various implementations, worker nodes 120 must obtain majority approval from the coordinator nodes 320 within the partition quorum 310. As shown, worker nodes 120A and 120B execute transactions 305A and 305B. For the following discussion, for the example, assume that both transactions 305 seek to write a record for the same key that falls within the key range of the partition 135 shown. To write a record for the key, worker nodes 120A and 120B must acquire a lock 330 on the key for their transaction 305. In various implementations, worker nodes 120A and 120B must obtain majority approval from the partition quorum 310 to acquire the lock 330.

[0049] In various implementations, lock 330 is a construct used to protect a database resource (e.g., a database object) from manipulation by an entity other than the holder of lock 330. Lock 330 may take the form of a record stored in partition 135. In various implementations, lock 330 is acquired on a specific database resource (e.g., a key, table, index, etc.) and has a lock mode (e.g., shared access). Lock 330 can be acquired on a per-statement basis (e.g., for each database statement) and on a per-transaction basis. In various implementations, an exclusive lock 330 is granted at most one transaction at a time for a database object (e.g., a key). Specifically, an exclusive lock 330 can be acquired for the key of a record if no other transaction acquires lock 330 first. In various implementations, lock contention causes a statement-level rollback and advances the statement snapshot time. By acquiring an exclusive lock 330 for a key, a transaction ensures its priority over competing transactions and will likely commit its changes. In various cases, transactions make incremental progress, thereby acquiring lock 330 within and across statements. Similar to the record, in various implementations, lock 330 is defined by the key range, but in some implementations, lock 330 may not be defined by the key range.

[0050] As shown in the figure, coordinator nodes 120A and 120B issue requests for lock 330 to coordinator nodes 320A-C. Coordinator nodes 320A-C can process the requests based on information stored in their respective replicas 137. In the illustrated embodiment, coordinator node 320A processes worker node 120A's request for lock 330 before worker node 120B's request. Coordinator node 320A does not detect a conflict (i.e., no other conflicting lock 330) and therefore returns an approval response to worker node 120A. Coordinator node 320A can record in its replica 137A that lock 330 has been approved for worker node 120A. When processing worker node 120B's request, coordinator node 320A detects a conflict because it assigned lock 330 to worker node 120A and therefore returns a rejection response to worker node 120B. In the illustrated implementation, coordinator node 320C processes worker node 120B's request for lock 330 before worker node 120A's request for lock 330. Coordinator node 320C does not detect a conflict and therefore returns an approval response to worker node 120B. Coordinator node 320C may record in its copy 137C that lock 330 has been approved by worker node 120B. When processing worker node 120A's request, coordinator node 320C detects a conflict and therefore returns a rejection response to worker node 120A.

[0051] In the illustrated implementation, coordinator node 320B does not respond to worker nodes 120A or 120B. Coordinator node 320B may not respond for any reason, such as a crash. Since neither worker nodes 120A nor 120B has obtained a majority of approvals (i.e., they each require two approvals but only have one), neither has acquired lock 330, and therefore, in this example, there is no winning transaction. After a period of time, worker nodes 120A and 120B can roll back a portion of their transaction 305 and then attempt to acquire lock 330 again. Worker nodes 120A and 120B can notify coordinator nodes 320A and 320B, respectively, that they have not acquired lock 330, allowing coordinator nodes 320A and 320B to update this information in their respective replicas 137. If coordinator node 320B is responsive and first (for example) processes worker node 120A's request and issues an approval response to worker node 120A, then worker node 120A acquires lock 330 because it has already obtained a majority of approvals. Therefore, since there may be no winning transactions 305 or there may be one winning transaction 305, there may be at most one winning transaction 305 (e.g., for lock 330, for commit, etc.).

[0052] Now go to Figure 4A A block diagram illustrating an exemplary split operation and an exemplary merge performed on a set of transaction coordinator partitions 135 is shown. In the illustrated embodiment, prior to the split operation, transaction coordinator partition 135A logically groups committed records 410, uncommitted records 420, and locks 330 into a single key range AC. However, after the split operation, committed records 410, uncommitted records 420, and locks 330 are split into two key ranges: key range AB and key range C. In some embodiments, the split operation may result in multiple transaction coordinator partitions 135 and / or more than two key ranges. Furthermore, in some embodiments, committed records 410, uncommitted records 420, and locks 330 may be their own partition 135 or part of their own copy 137 for partition 135.

[0053] As described above, in various implementations, the transaction coordinator partition 135 includes information that can be used to ensure that concurrent changes made by worker node 120 do not violate read committed isolation. In various implementations, committed records 410 are records that were recently committed by worker node 120 but may not yet have been pushed to database 110 (e.g., written to the top level of LSM tree 115). Specifically, after worker node 120 commits a database transaction in its worker log 125, it can then inform the relevant coordinator node that the transaction has been committed. The transaction updates are then visible at the coordinator node for any reads later than the transaction's snapshot time. Therefore, worker node 120 can access committed records 410 from the coordinator node rather than the LSM tree 115, especially where those records are not stored in the LSM tree 115.

[0054] In various implementations, uncommitted record 420 is a record that has not yet been committed by the corresponding worker node 120. Knowledge of uncommitted record 420 can be used by a given coordinator node to ensure that conflicting records are not committed. For example, transaction T1 may commit a record (listed in committed record 410) that is not visible to transaction T2 whose snapshot time is before the record's commit. Transaction T2 may write a record for the same database key and request permission to commit the record. A given coordinator node may observe that the commit of the previous record occurred after T2's snapshot time and conflicts with the subsequent record. Therefore, the coordinator node may refuse to allow transaction T2 to commit. In various implementations, in response to worker node 120 committing the transaction that produced uncommitted record 420, these uncommitted records 420 become committed records 410 (within partition 135) — that is, the coordinator node managing the replica 137 of the relevant partition 135 converts the uncommitted update into the record version when the transaction is committed.

[0055] In some cases, high-volume read traffic may be aligned with narrow data key ranges, and therefore it may be desirable to dynamically split replicas 137 of the transaction coordinator partition 135 and / or create new replicas 137. Thus, a coordinator node may perform a split operation on its replicas 137 of the relevant transaction coordinator partition 135. In some implementations, splitting replica 137 involves logically dividing the data within replica 137 without moving data from the coordinator node. Splitting replica 137 may generate two or more replicas 137 that can be stored on the same coordinator node or different coordinator nodes. Splitting replica 137 may be a local operation performed at each coordinator node. In some cases, the splitting of replicas 137 of the transaction coordinator partition 135 occurs at least partially in parallel. However, in other cases, a coordinator node may decide to split its replica 137 and subsequently notify coordinator nodes with other replicas 137 to split their replicas 137.

[0056] like Figure 4A As shown, the split operation results in the data in partition 135A being split into two partitions, 135B and 135C, and thus the coordinator node can store two replicas 137, one replica for each partition. In various implementations, since the replicas 173 of partitions 135B and 135C are stored separately after the split operation but reside on the same coordinator node, traffic directed to one partition 135B may not conflict with or otherwise compete with traffic directed to the other partition 135C. Furthermore, after the split operation, the coordinator node can then relocate one or more of the resulting replicas 137 to another coordinator node, thereby allowing the previous coordinator node to reclaim storage and reduce its workload.

[0057] In various implementations, since the split operation can produce multiple partitions 135, the coordinator node notifies other coordinator nodes associated with the original partition (i.e., partition 135A in the illustrated implementation) of the split, enabling them to split the original partition 135 into multiple partitions 135 to ensure consistency among quorum numbers—that is, they can split their copies 137 for the original partition 135 into copies 137 corresponding to the multiple partitions 135. In some cases, while other coordinator nodes may eventually achieve the same split, they may split their copies 137 for the original partition 135 in a different manner. Over time, the coordinator nodes may split and merge copies 137 of partition 135 until the coordinator nodes achieve the same number of partitions 135 as the original partition 135.

[0058] In various implementations, the coordinator node can also perform a merge operation to combine replicas 137. When two adjacent key ranges have corresponding replicas 137 located on the same coordinator node, their key ranges can be merged into a larger key range, which can produce a new replica 137 that includes the data of the combined replica 137. Figure 4A As shown, partitions 135D and 135E are combined into partition 135F. Therefore, the coordinator node can initially include two replicas 137 (one for partition 135D and one for partition 135E) and merge them into a single replica 137 corresponding to partition 135. In various cases, merging multiple replicas 137 involves logically combining the data within replicas 137 without moving data on the coordinator node. Furthermore, the merge can initially be local to a particular coordinator node, and knowledge of the merge can be "lazily" propagated through system 100, as discussed above in relation to the split operation.

[0059] Now go to Figure 4BA block diagram of an exemplary relocation operation performed for replica 137 is shown. In the illustrated embodiment, there are two coordinator nodes 320A-B, which are part of the transaction coordinator service 130. Furthermore, as shown, coordinator node 320A includes replica 137A of transaction coordinator partition 135, and coordinator node 320B includes replica 137B (derived from replica 137A as part of the relocation operation). The illustrated embodiment can be implemented in a different manner than shown. For example, the relocation operation could produce multiple new replicas 137.

[0060] As discussed, each partition 135 may include multiple replicas 137 that can be updated and read by the coordinator node 320. To cope with evolving system load, replicas 137 may need to be relocated from one coordinator node 320 to another without interrupting ongoing transactions and their snapshot reads, locking, updates, and subsequent transaction commits. In various implementations, relocation involves creating a new replica 137 and loading it with a pre-existing state (e.g., lock 330, uncommitted record 420, etc.), the pre-existing state coming from one or more of the other replicas 137 forming the transaction coordinator partition 135. When a new replica 137 is loaded with a state, its coordinator node 320 can process ongoing business (e.g., lock requests, commit requests, etc.) from worker nodes 120 and insert the results (e.g., a new lock 330) into the new replica 137. Thus, a new replica 137 can be loaded with a pre-existing state and a new state from new incoming business.

[0061] Furthermore, as discussed, in various implementations, worker node 120 waits for approval from at least a majority of the coordinating nodes 320 in transaction coordinator partition 135 before continuing a database transaction—this ensures that the result of processing the worker node's request (e.g., a new lock 330) is persisted in a majority of the replicas 137 in transaction coordinator partition 135. However, adding a new replica 137 can temporarily change how many approval responses worker node 120 waits for before continuing. Consider the following example where there are seven replicas 137, and therefore worker node 120 waits for approval from four of these replicas (i.e., from their coordinating nodes 320). In the case of adding a new replica 137, worker node 120 then waits for approval from five of the eight replicas 137. In various implementations, after the new replica 137 is loaded with the state from the old replica 137, the old replica 137 is decommissioned. When the number of replicas 137 in a specific transaction coordinator partition 135 decreases, the retirement of older replicas 137 can change the quorum rules (e.g., obtaining approval from a majority). For example, relative to... Figure 5In more detail, in some implementations, the relocation operation temporarily generates multiple quorums of coordinator nodes 320 for partition 135.

[0062] As part of the relocation process, each committed record 410 and uncommitted record 420 can be copied from one or more old replicas 137 to the new replica 137. Reading up to the snapshot time may only require reading enough replicas 137 of partition 135 (e.g., a majority) to ensure that all committed updates are located and combined by worker node 120. Similarly, lock 330 can be copied from old replicas 137 to the new replica 137. In various implementations, a database transaction is considered correct as long as an exclusive lock (and update) is granted to at most one database transaction at a given point in time. Worker node 120 can acquire lock 330 if it receives approval from a majority of replicas 137 (of the locked key). Copying lock 330 from old replicas 137 to the new replica 137 and a temporary increase in the number of replicas 137 in transaction coordinator partition 135 will not result in any exclusive lock 330 being granted to more than one database transaction. In some cases, if the old copy 137 does not contain the record locked by 330, the transaction may lose permission to lock the key, and adding a new copy 137 may cause the transaction to lose most of its permissions.

[0063] Therefore, replicas 137 of the transaction coordinator partition 135 can be automatically split, merged, and relocated as needed to cope with scaling pressure. Additionally, new coordinator nodes 320 can be added to handle the pressure.

[0064] Now go to Figure 5 A block diagram is shown illustrating an example in which a worker node interacts with multiple partition quorums during partition relocation. In the illustrated embodiment, worker node 120 and partition quorums 310A and 310B are present. Furthermore, as shown, partition quorum 310A includes three coordinator nodes 320A-C, and partition quorum 310B includes three coordinator nodes 320A, 320B, and 320D. The illustrated embodiment can be implemented in a different manner than shown. As an example, partition quorums 310A and 310B may include more than three coordinator nodes 320.

[0065] As discussed, coordinator node 320 can relocate copy 137 to another coordinator node 320. In various embodiments, the relocation operation can generate multiple quorums 310 when the contents of copy 137 are copied to the target coordinator node 320. In the illustrated embodiment, coordinator node 320 relocates copy 137C to coordinator node 320D, and when copy 137C is relocated, there are two partition quorums 310 for the same transaction coordinator partition 135. In various embodiments, in order to acquire lock 330 and / or perform another action (e.g., commit transaction), worker node 120 must obtain majority approval from at least one of the partition quorums 310 associated with the transaction coordinator partition 135 affected by the action.

[0066] Therefore, in the illustrated implementation, when seeking to acquire, for example, a lock 330 for a key of a transaction, worker node 120 sends a request to partition quorum 310A and 310B—that is, coordinator nodes 320A-D can each receive the request from worker node 120. Coordinator nodes 320 in quorum 310A and 310B can each check their respective copies 137 for any conflicts associated with worker node 120 acquiring lock 330 and provide worker node 120 with an approval response or a rejection response (or in some cases, no response). In various implementations, worker node 120 acquires lock 330 upon receiving an approval response from a majority of coordinator nodes 320 in at least one quorum 310 and then proceeds with its transaction 305. However, in some implementations, worker node 120 must obtain an approval response from a majority of coordinator nodes 320 in all quorum 310 associated with the relevant partition 135. For example, worker node 120 may need to obtain approval responses from a majority of coordinator nodes 320A-C of the quorum 310A, and also from a majority of coordinator nodes 320A, 320B, and 320D of the quorum 310B. In various implementations, if worker node 120 does not obtain a majority of approval from the relevant quorum 310, it may roll back at least a portion of the transaction and then try again.

[0067] In various implementations, potential conflicts associated with requests to worker node 120 can be observed through interaction with both quorums 310A and 310B, as lock 330 and committed record 310 cannot exist without majority approval from at least one quorum 310. Therefore, copying data from replica 137 to another coordinator node 320 does not result in errors or conflicts between transactions, because worker node 120 checks both the old and new locations of replica 137—that is, worker node 120 can check both the old and new quorums 310 to accommodate the relocation and / or update of lock 330. In various implementations, the old quorum 310 ceases to exist after replica 137 has been relocated to the new quorum 310. For example, partition quorum 310A can cease to exist after replica 137C has been copied to coordinator node 320D. In addition, space can be reclaimed in old copy 137 (for example, lock 330 in copy 137C at coordinator node 320C can be removed and updated to reclaim space at coordinator node 320C).

[0068] Now go to Figure 6 A flowchart of method 600 is shown. Method 600 is an implementation of a method executed by a coordinator node (e.g., coordinator node 320) to ensure transactional consistency of transactions executed by worker nodes (e.g., worker node 120). Method 600 can be executed by executing a set of program instructions stored on a non-transitory computer-readable medium. Method 600 may include more or fewer steps than shown. For example, method 600 may include a step in which the coordinator node merges multiple replicas into a single replica.

[0069] Method 600 begins at step 610, where multiple coordinator nodes of a database system (e.g., system 100) store a copy (e.g., copy 137) of a specific partition among multiple partitions (e.g., partition 135) divided by a key range. In various embodiments, a given copy of these copies includes information about granted locks (e.g., lock 330) and records (e.g., committed record 410 and uncommitted record 420) generated by a set of multiple worker nodes operable to perform database transactions. The multiple coordinator nodes are operable to ensure transactional consistency of the database transactions. In step 620, the multiple coordinator nodes receive requests from multiple worker nodes to grant locks for a key, allowing the worker nodes to write records for that key as part of performing the database transaction. In step 630, a given coordinator node among the multiple coordinator nodes sends an approval response for the lock to at most one of the multiple worker nodes. In various embodiments, a single worker node among the multiple worker nodes acquires the lock in response to receiving an approval response from a majority of the multiple coordinator nodes. In various implementations, multiple worker nodes fail to acquire the lock in response to the fact that none of the worker nodes have received an approval response from a majority of the coordinators among the multiple coordinators.

[0070] A first coordinator node among a plurality of coordinator nodes may determine to relocate at least a portion of a copy of a specific partition of the first coordinator node to a second coordinator node in the database system. Therefore, the first coordinator node may replicate at least a portion of its copy to the second coordinator node. A given worker node among a plurality of worker nodes is operable to request, during replication, both the first and second coordinator nodes to allow the execution of a specified action associated with the specific partition. The plurality of coordinator nodes form a first quorum (e.g., partition quorum 310A) for the specific partition, and the second coordinator node and a plurality of coordinator nodes not having a first coordinator node form a second quorum (e.g., partition quorum 310B) for the specific partition. In various embodiments, to execute the specified action, the given worker node must obtain an approval response from a majority of the coordinator nodes in at least one of the first and second quorums. The given worker node is operable to request a lock from the second coordinator node and store new uncommitted work for the specific partition in response to the first coordinator node determining to relocate at least a portion of its copy of the first coordinator node.

[0071] A first coordinator node among a plurality of coordinator nodes may determine, based on a set of characteristics (e.g., size) of a replica of a specific partition of the first coordinator node, to locally split the replica of that specific partition into multiple replicas corresponding to multiple sub-partitions representing the split of the specific partition. In response to this determination, the first coordinator node splits the replica of the specific partition of the first coordinator node into multiple replicas. The first coordinator node may notify one or more of the remaining coordinator nodes among the plurality of coordinator nodes of the split, such that one or more coordinator nodes split their replicas of the specific partition. The multiple replicas at the first coordinator node may include fewer replicas than the number of replicas into which a second coordinator node among the plurality of coordinator nodes has split the replica of the specific partition of the second coordinator node.

[0072] The first coordinator node in a plurality of coordinator nodes can determine, based on a set of characteristics of replicas of a specific partition of the first coordinator node, to locally merge two or more replicas into a single replica corresponding to a single partition representing the merge of two or more partitions. In response to this determination, the first coordinator node can merge two or more replicas (whose key ranges may be adjacent) into a single replica. After merging two or more replicas into a single replica, the first coordinator node can split the single replica into a number of replicas different from the number of replicas of the two or more replicas.

[0073] The first coordinator node among multiple coordinator nodes can remove committed records from its replica for a specific partition based on committed records associated with a replica persistently stored in a persistent repository (e.g., database 110). The first coordinator node can receive different replicas from different coordinator nodes in the database system. While receiving information about different replicas, the first coordinator node can process requests from worker nodes among multiple worker nodes. Processing at least one of these requests may include storing locks granted in association with at least one request in different replicas.

[0074] Exemplary computer system

[0075] Now go to Figure 7 A block diagram depicts an exemplary computer system 700 that can implement system 100, database 110, worker node 120, transaction coordinator service 130, and / or coordinator node 230. Computer system 700 includes a processor subsystem 780 connected to system memory 720 and I / O interface 740 via interconnect 760 (e.g., system bus). I / O interface 740 is connected to one or more I / O devices 750. Although for convenience, in... Figure 7 A single computer system 700 is shown, but system 700 can also be implemented as two or more computer systems operating together.

[0076] Processor subsystem 780 may include one or more processors or processing units. In various embodiments of computer system 700, multiple instances of processor subsystem 780 may be coupled to interconnect 760. In various embodiments, processor subsystem 780 (or each processor unit within 780) may include cache or other forms of onboard memory.

[0077] System memory 720 can be used to store program instructions that can be executed by processor subsystem 780 to cause system 700 to perform the various operations described herein. System memory 720 can be implemented using different physical memory media, such as hard disk storage devices, floppy disk storage devices, removable disk storage devices, flash memory, random access memory (RAM—SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.), and so on. The memory in computer system 700 is not limited to main storage devices such as memory 720. Instead, computer system 700 may also include other forms of storage devices, such as cache memory in processor subsystem 780 and auxiliary storage devices (e.g., hard disk drives, storage arrays, etc.) on I / O devices 750. In some embodiments, these other forms of storage devices may also store program instructions that can be executed by processor subsystem 780. In some embodiments, program instructions that implement worker node 120 and / or coordinator node 230 when executed may be included / stored in system memory 720.

[0078] According to various embodiments, I / O interface 740 can be any interface of various types configured to connect to and communicate with other devices. In one embodiment, I / O interface 740 is a bridge chip (e.g., a southbridge) from a front end to one or more back end buses. I / O interface 740 can be connected to one or more I / O devices 750 via one or more corresponding buses or other interfaces. Examples of I / O devices 750 include storage devices (hard disk drives, optical disk drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local area network or wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system 700 is connected to a network (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.) via network interface device 750.

[0079] Public texts include references to an “implementation” or a group of “implementations” (e.g., “some implementations” or “various implementations”). An implementation is a different way of realizing or being implemented of the disclosed concept. References to “implementation,” “an implementation,” “a specific implementation,” etc., do not necessarily refer to the same implementation. A large number of possible implementations are envisioned, including those specifically disclosed, as well as modifications or alternatives that fall within the spirit or scope of the public text.

[0080] The disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all implementations of these embodiments will necessarily exhibit any or all of these potential advantages. Whether an advantage is realized for a particular implementation depends on many factors, some of which are outside the scope of the disclosure. In fact, there are multiple reasons why an implementation falling within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuitry outside the scope of the disclosure, which, in combination with an embodiment of the disclosed embodiments, negates or diminishes one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular implementation (e.g., the implementation technique or tool) may also negate or diminish the disclosed advantages. Even assuming a skilled implementation, the realization of an advantage can still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, the inputs provided to a particular implementation may prevent one or more problems addressed in the disclosure from occurring in a particular context, resulting in the benefits of its solution potentially not being realized. Given the existence of possible factors outside the disclosure, this invention expressly aims that any potential advantages described herein should not be construed as requiring the satisfaction of claims limitations to prove infringement. Rather, the identification of such potential advantages is intended to indicate the types of improvements available to the designer that benefit from the disclosure. The permissive description of such advantages (e.g., stating that a particular advantage "may occur") is not intended to express doubt about whether such advantages can actually be realized, but rather to recognize the technological reality that the realization of such advantages often depends on additional factors.

[0081] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims drafted based on the disclosure, even where only a single example is described with respect to a particular feature. The disclosed embodiments are intended to be illustrative rather than restrictive unless there is any statement to the contrary in the disclosure. Therefore, this application is intended to allow the claims to cover the disclosed embodiments, as well as such alternatives, modifications, and equivalents of the disclosure that would be apparent to those skilled in the art and would benefit from.

[0082] For example, features in this application can be combined in any suitable manner. Therefore, during the examination of this application (or an application claiming priority thereto), new claims can be formulated for any such combination of features. Specifically, referring to the appended claims, features from dependent claims can be combined with features from other dependent claims (including claims dependent on other independent claims) where appropriate. Similarly, features from individual independent claims can be combined where appropriate.

[0083] Therefore, while the appended dependent claims can be drafted such that each dependent claim depends on a single other claim, additional dependent relationships are also contemplated. Any combination of features in the dependent claims consistent with the published text is contemplated, and such combinations can be claimed in this application or another application. In short, the combinations are not limited to those specifically enumerated in the appended claims.

[0084] Where appropriate, it is also envisioned that claims drafted in one format or statutory type (e.g., apparatus) are intended to support corresponding claims in another format or statutory type (e.g., method).

[0085] Because the published text is a legal document, various terms and phrases may be subject to administrative and judicial interpretations. Therefore, it is hereby announced that the following paragraphs, along with the definitions provided throughout the published text, will be used to determine how to interpret claims drafted based on the published text.

[0086] Unless the context clearly specifies otherwise, references to singular terms (i.e., nouns or noun phrases beginning with "an," "a," or "the") are intended to mean "one or more." Therefore, without accompanying context, a reference to "an / a term" in a claim does not exclude additional instances of that term. "A plurality of" terms refer to a collection of two or more terms.

[0087] The word “can” is used in this text in a permissive sense (i.e., having the possibility, being able) rather than in a mandatory sense (i.e., having to).

[0088] The terms “include” and “including” and their forms are open-ended, meaning “including but not limited to”.

[0089] When the term “or” is used in public text in relation to a list of options, it is generally understood to be used in an inclusive sense, unless the context otherwise specifies. Therefore, a statement of “x or y” is equivalent to “x or y, or both,” and thus encompasses: 1) x but not y; 2) y but not x; and 3) both x and y. On the other hand, phrases such as “either x or y, but not both” clearly indicate that “or” is used in an exclusive sense.

[0090] The statements “w, x, y, or z or any combination thereof” or “...at least one of w, x, y, and z” are intended to cover all possibilities involving a single element up to all elements in the set. For example, given the set [w, x, y, z], these terms cover any single element of the set (e.g., w, but not x, y, or z), any two elements (e.g., w and x, but not y or z), any three elements (e.g., w, x, and y, but not z), and all four elements. Therefore, the phrase “...at least one of w, x, y, and z” refers to at least one element in the set [w, x, y, z], thus covering all possible combinations in the list of elements. The phrase should not be interpreted as requiring the existence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0091] In public text, various “labels” may precede nouns or noun phrases. Unless the context otherwise specifies, different labels used for features (e.g., “first circuit,” “second circuit,” “specific circuit,” “given circuit,” etc.) refer to different instances of the feature. Furthermore, unless otherwise stated, the labels “first,” “second,” and “third” do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) when applied to features.

[0092] The phrase “based on” is used to describe one or more factors that influence a determination. The term does not exclude the possibility that additional factors may influence the determination. That is, a determination may be based solely on the specified factor or on the specified factor along with other unspecified factors. Consider the phrase “A is determined based on B.” The phrase specifies that B is a factor used to determine A or that influences the determination of A. The phrase does not exclude the possibility that the determination of A may also be based on another factor, such as C. The phrase is also intended to cover implementations in which A is determined solely based on B. As used herein, the phrase “based on” is synonymous with the phrase “at least partially based on.”

[0093] The phrases “in response to” and “as a response to” describe one or more factors that trigger an effect. The phrases do not exclude the possibility that additional factors may influence or otherwise trigger an effect, either jointly with or independently of the specified factor. That is, an effect may respond only to those factors, or it may respond to the specified factor as well as other unspecified factors. Consider the phrase “A is executed in response to B.” The phrase specifies that B is a factor that triggers the execution of A or triggers a specific result of A. The phrase does not exclude that the execution of A may also be in response to another factor, such as C. The phrase also does not exclude that the execution of A may jointly respond to B and C. The phrases are also intended to cover implementations in which A is executed only in response to B. As used herein, the phrase “as a response to” is synonymous with the phrase “at least partially as a response to.” Similarly, the phrase “in response to” is synonymous with the phrase “at least partially in response to.”

[0094] Within the public text, different entities (which may be referred to differently as “units,” “circuits,” other components, etc.) may be described or claimed to be “configured” to perform one or more tasks or operations. The expression—[entity] configured to [perform one or more tasks]—is used herein to refer to a structure (i.e., a physical thing). More specifically, the expression is used to indicate that a structure is arranged to perform one or more tasks during operation. A structure may be referred to as being “configured” to perform a task even if the structure is not currently being operated. Therefore, an entity described or stated as being “configured” to perform a task refers to physical things such as devices, circuits, systems with processor units and memory storing program instructions that can be executed to perform the task, etc. The phrase is not used herein to refer to intangible things.

[0095] In some cases, various units / circuits / components may be described herein as performing a set of tasks or operations. It should be understood that, even if not specifically stated otherwise, these entities are "configured" to perform these tasks / operations.

[0096] The term "configured as" is not intended to mean "configurable as". For example, an unprogrammed FPGA would not be considered "configured as" to perform a specific function. However, an unprogrammed FPGA can be "configurable as" to perform a function. After proper programming, an FPGA can then be referred to as "configured as" to perform a specific function.

[0097] For the purposes of a U.S. patent application based on a published text, a statement structure in the claims that is “configured” to perform one or more tasks is expressly intended not to invoke 35 USC § 112(f) for any claim element. If an applicant wishes to invoke 112(f) during the examination of a U.S. patent application based on a published text, it will use the “means for [performing a function]” structure to state the claim element.

Claims

1. A method comprising: A database system stores copies of a specific partition from multiple partitions divided by key ranges by multiple coordinator nodes, wherein a given copy of the copies includes information about granted locks and records generated by a set of multiple worker nodes operable to execute database transactions, wherein the multiple coordinator nodes are operable to ensure transactional consistency of the database transactions. The multiple coordinator nodes receive requests for granting locks on keys from multiple worker nodes, allowing worker nodes to write records on the keys as part of executing database transactions; as well as An approval response for the lock is sent by a given coordinator node from the plurality of coordinator nodes to at most one of the plurality of worker nodes, wherein a single worker node acquires the lock in response to receiving an approval response from a majority of the plurality of coordinator nodes, and wherein none of the plurality of worker nodes acquires the lock in response to none of the plurality of worker nodes receiving an approval response from a majority of the plurality of coordinator nodes.

2. The method according to claim 1, further comprising: The first coordinator node among the plurality of coordinator nodes determines to relocate at least a portion of the copy of the specific partition of the first coordinator node to the second coordinator node of the database system. as well as The first coordinator node copies at least a portion of its copy to the second coordinator node, wherein a given worker node among the plurality of worker nodes is operable to make a request to both the first and second coordinator nodes during the copying process to allow the execution of a specified action associated with the particular partition.

3. The method of claim 2, wherein the plurality of coordinator nodes form a first quorum for the specific partition, and the second coordinator node and the plurality of coordinator nodes not having the first coordinator node form a second quorum for the specific partition, wherein in order to perform the specified action, the given worker node must obtain an approval response from a majority of the coordinator nodes in at least one of the first quorum and the second quorum.

4. The method of claim 2, wherein the given worker node is operable to request a lock from the second coordinator node and store new uncommitted work for the particular partition in response to the first coordinator node determining that at least a portion of a copy of the first coordinator node has been relocated.

5. The method according to claim 1, further comprising: The first coordinator node among the plurality of coordinator nodes determines, based on a set of characteristics of a copy of the specific partition of the first coordinator node, to locally split the copy of the specific partition of the first coordinator node into a plurality of copies corresponding to a plurality of sub-partitions representing the split of the specific partition. as well as In response to the determination, the first coordinator node splits the copy of the specific partition of the first coordinator node into the plurality of copies.

6. The method according to claim 5, further comprising: The first coordinator node notifies one or more of the remaining coordinator nodes among the plurality of coordinator nodes of the split, so that the one or more coordinator nodes split their copies of the specific partition.

7. The method of claim 5, wherein the plurality of replicas includes fewer replicas than the number of replicas into which the second coordinator node has split the replicas of the particular partition of the second coordinator node among the plurality of coordinator nodes.

8. The method according to claim 1, further comprising: The first coordinator node among the plurality of coordinator nodes determines, based on a set of characteristics of a copy of the specific partition of the first coordinator node, to locally merge two or more copies of adjacent partitions into a single copy corresponding to a single partition representing the merger of two or more partitions. as well as In response to the determination, the first coordinator node merges the two or more replicas of the adjacent partitions into the single replica corresponding to the single partition.

9. The method according to claim 8, further comprising: After the merge, the first coordinator node splits the single partition into a different number of partitions than the adjacent partitions.

10. The method of claim 8, wherein the key ranges of the two or more copies are adjacent.

11. A computer-readable medium having program instructions stored thereon, the program instructions enabling a computer system to perform operations including: A copy of a specific partition is stored among multiple partitions divided by a key range, wherein the copy includes information about granted locks and records generated by a set of multiple worker nodes operable to execute database transactions of the database system, and wherein the coordinator node is a coordinator node among multiple coordinator nodes operable to ensure transactional consistency of database transactions. Receive requests for granting locks on a key from multiple worker nodes to allow worker nodes to write records on that key as part of executing a database transaction; as well as An approval response for the lock is sent to at most one of the plurality of worker nodes, wherein a single worker node acquires the lock in response to receiving an approval response from a majority of the plurality of coordinator nodes, and wherein none of the plurality of worker nodes acquires the lock in response to none of the plurality of worker nodes receiving an approval response from a majority of the plurality of coordinator nodes.

12. The computer-readable medium of claim 11, wherein the operation further comprises: Determine to relocate at least a portion of the replicas of the specific partition to different coordinator nodes of the database system; as well as The copy is replicated to the different coordinator nodes, wherein the coordinator nodes and the different coordinator nodes are associated with different quorums of coordinator nodes, and wherein a given worker node among the plurality of worker nodes is operable to request, during the replication, a majority of approvals from the different quorums of coordinators to perform a specified action associated with the particular partition.

13. The computer-readable medium of claim 11, wherein the operation further comprises: Perform a split operation on the replica to logically split the replica into two or more replicas corresponding to two or more sub-partitions representing the split of the particular partition; as well as A merge operation is performed on the replica to logically merge the replica and another replica into a single replica corresponding to a single partition representing the merge of two partitions, wherein the split and merge operations are performed independently of the other coordinators among the plurality of coordinators.

14. The computer-readable medium of claim 11, wherein the operation further comprises: Based on the committed records that are persistently stored in the persistent repository associated with the replica, the committed records are removed from the replica.

15. The computer-readable medium of claim 11, wherein the operation further comprises: Receive different replicas from different coordinator nodes of the database system; as well as While receiving information from the different replicas, requests from worker nodes among the plurality of worker nodes are processed, wherein processing at least one of the requests includes storing a lock granted in association with the at least one request in a different replica.

16. A system comprising: At least one processor; and A memory storing program instructions executable by the at least one processor to enable the system to perform operations on a coordinating node, including: A copy of a specific partition is stored among multiple partitions divided by a key range, wherein the copy includes information about granted locks and records generated by a set of multiple worker nodes operable to execute database transactions of the database system, and wherein the coordinator node is a coordinator node among multiple coordinator nodes operable to ensure transactional consistency of database transactions. Receive requests for granting locks on a key from multiple worker nodes to allow worker nodes to write records for that key as part of executing a database transaction; and An approval response for the lock is sent to at most one of the plurality of worker nodes, wherein a single worker node acquires the lock in response to receiving an approval response from a majority of the plurality of coordinator nodes, and wherein none of the plurality of worker nodes acquires the lock in response to none of the plurality of worker nodes receiving an approval response from a majority of the plurality of coordinator nodes.

17. The system of claim 16, wherein the operation further comprises: Perform a split operation on the replica to logically split the replica into two or more replicas corresponding to two or more sub-partitions representing the split of the particular partition; as well as At least a portion of a particular copy of the two or more copies is copied to a different coordinator node, wherein a given worker node of the plurality of worker nodes is operable to make a request to both the coordinator node and the different coordinator nodes during the copying process to allow the execution of a specified action associated with a specific partition corresponding to the copy.

18. The system of claim 16, wherein the operation further comprises: Receive different replicas from different coordinator nodes of the database system; as well as Perform a merge operation to logically merge a copy of the specific partition and a copy of another adjacent partition into a single copy corresponding to the single partition.

19. The system of claim 16, wherein the operation further comprises: Receive different replicas from different coordinator nodes of the database system; as well as While receiving information from the different replicas, requests from worker nodes among the plurality of worker nodes are processed, wherein processing at least one of the requests includes storing a lock granted in association with the at least one request in a different replica.

20. The system of claim 16, wherein the operation further comprises: Based on the committed records that are persistently stored in the persistent repository associated with the replica, the committed records are removed from the replica.

Images