A director-based database system for transaction integrity.

Abstract

Techniques are disclosed that relate to a database system including a worker node operable to execute a transaction and a director node operable to ensure transactional integrity of the transaction. The worker node may receive a request to execute a transaction including writing a record. The worker node may then issue a request to a director node of the database system for information facilitating execution of operations for the transaction. The director node may determine whether to approve the request based on whether the operation may cause transactional inconsistency in the database system. The worker node may proceed to execute the operations for the transaction in response to receiving approval responses from a majority of the director nodes, with none of the received responses indicating disapproval of the transaction.

Description

[Technical field]

[0001] The present disclosure relates generally to database systems, and more particularly to addressing slowdowns in components of database systems. [Background technology]

[0002] Enterprises routinely employ modern database management systems to organize and store collections of information so that users can efficiently access and manipulate them. Historically, these systems have been deployed on high-end, expensive hardware maintained in data centers owned by these enterprises. In most cases, the high-end hardware behaves predictably, allowing the systems to achieve fast and responsive latency. However, over time, more and more enterprises are moving their systems to the cloud, and thus relying on hardware supplied by cloud providers. However, cloud providers are beginning to shift toward using cheaper hardware in their cloud infrastructure that behaves less predictably than high-end hardware. [Brief description of the drawings]

[0003] [Figure 1] FIG. 2 is a block diagram illustrating example elements of a system including worker nodes and director nodes, according to some embodiments. [Diagram 2] FIG. 2 is a block diagram illustrating example elements of a worker node, according to some embodiments. [Diagram 3] FIG. 2 is a block diagram illustrating example elements of a director node, according to some embodiments. [Figure 4] FIG. 2 is a block diagram illustrating example elements of a history record data structure according to some embodiments. [Diagram 5]FIG. 2 is a block diagram illustrating example elements of a snapshot-related interaction involving one worker node and multiple director nodes, according to some embodiments. [Figure 6] 4 is a block diagram illustrating an example layout of historical information on different director nodes according to some embodiments. [Figure 7] FIG. 2 is a block diagram illustrating example elements of a commit-related interaction involving one worker node and multiple director nodes, according to some embodiments. [Figure 8] FIG. 2 is a block diagram illustrating another example layout of historical information on different director nodes, according to some embodiments. [Figure 9] FIG. 2 is a block diagram illustrating example elements related to time aligning components of a system, according to some embodiments. [Figure 10] FIG. 2 is a block diagram illustrating example elements of an assistant node, according to some embodiments. [Figure 11] FIG. 1 is a block diagram illustrating example elements of a flush operation in which a worker node and an assistant node flush the same records to a database, according to some embodiments. [Figure 12] FIG. 2 is a flow diagram illustrating an example method for a worker node interacting with a director node to facilitate execution of operations for a transaction, according to some embodiments. [Figure 13] FIG. 2 is a flow diagram illustrating an example method for a director node to ensure transaction consistency in a system, according to some embodiments. [Figure 14] FIG. 11 is a flow diagram illustrating an example method for a worker node interacting with a director node to obtain history information describing the history of an approved record, according to some embodiments. [Figure 15] FIG. 2 is a flow diagram illustrating an example method for a worker node to select a proposed time associated with performing an operation for a transaction, according to some embodiments. [Figure 16] FIG. 2 is a flow diagram illustrating an example method for a worker node to select a proposed snapshot time for a transaction, according to some embodiments. [Figure 17] FIG. 2 is a flow diagram illustrating an example method for a director node to attempt to align the time observed by the director node with the time observed by other director nodes, according to some embodiments. [Figure 18] FIG. 2 is a flow diagram illustrating an example method for a director node to attempt to align times observed by the director node, according to some embodiments. [Figure 19] FIG. 1 is a flow diagram illustrating an example method for an assistant node to observe work performed by a worker node, according to some embodiments. [Figure 20] FIG. 11 is a flow diagram illustrating another example method for an assistant node to observe work performed by a worker node, according to some embodiments. [Figure 21] FIG. 1 is a block diagram illustrating elements of a multi-tenant system, according to some embodiments. [Figure 22] FIG. 1 is a block diagram illustrating elements of a computer system for implementing various systems described in this disclosure, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] As explained, cloud providers are shifting towards using cheaper, and therefore less reliable, hardware in their cloud infrastructure. The use of cheaper hardware has resulted in an increasing number of reports of servers randomly slowing down. This random slowdown of a server (or other component) causes both gray failures (where the server suddenly experiences a significant drop in its performance (e.g., 99% or more)) and stragglers (where the server experiences a less noticeable slowdown, such as a 25% drop in performance). When the slowdown occurs, the server does the "right" thing, but slowly enough to circumvent the failure detectors that would take the server down, while also causing problems for the whole system. For example, a server may run at half its normal speed, but still be able to output work and respond to requests, albeit slowly, so that the server's slower performance may go unsuspected by the rest of the system.

[0005] Slowdowns can affect transactional and non-transactional systems, but techniques used to address slowdowns in non-transactional systems are not applicable to transactional systems. A transaction is a set of commands that are logically grouped together as a single unit of work. A transactional system is a type of system that processes transactions according to a set of guidelines that ensure transactional integrity. ACID (Atomicity, Consistency, Isolation, and Durability) is an example of a guideline that tries to ensure that either all or none of the transactions succeed, ensure that data is consistent, ensure that transactions occur in isolation, and ensure that once a transaction is committed, it stays in the system. Non-transactional systems are not bound by a set of guidelines that ensure transactional integrity. One technique for addressing slowdowns in non-transactional systems is to retry some of the computational work when a time limit is reached, i.e., the work is retried on another server after a period of time has passed without the slow server being able to produce a result. In some cases, computational work is pushed to multiple servers so that if one of those servers experiences a slowdown, the other servers can still return the results of the work. This is an acceptable approach for non-transactional systems because their work is idempotent, meaning that performing the same work multiple times does not cause unintended states in the system. As an example, fetching and returning a static web page is an idempotent operation. This contrasts with non-idempotent operations, such as incrementing a counter, where multiple retries may cause the counter to increment more than intended. Because transactional database systems adhere to principles that ensure transactional integrity, their work is generally non-idempotent and it is usually guaranteed that only one transaction appears to be occurring at a time.Therefore, attempting the same work on multiple servers is not feasible for a transactional database system. This disclosure addresses, among other things, the problem of how to protect a transactional database system from slowdowns that can slow down individual components of the system.

[0006] Slowdowns also cause problems when trying to scale a transactional database system to have more database nodes (alternatively, "worker" nodes) that execute transactions. As the number of database nodes in a system increases, the possibility of transaction inconsistency also increases. In one embodiment, a director node is implemented that ensures transaction consistency in a transactional database system by checking for conflicts between transactions executed by the worker nodes. However, a system that uses one director node is susceptible to a significant performance degradation if the director node suffers from a slowdown, since transactions cannot be completed without the consent of the director node. The present disclosure further addresses the problem of how to ensure transaction consistency between worker nodes, especially in light of the slowdowns that occur in transactional database systems.

[0007] This disclosure describes techniques for implementing mechanisms to mitigate some or all of the effects of slowdowns on a system. In various embodiments described below, the system includes a database and a set of "worker" nodes that execute transactions for the system, including writing records to the database. The system further includes a group of "director" nodes that help ensure transactional consistency between transactions executed by the worker nodes. Throughout the execution of a transaction, a worker node may interact with the director node to ensure that the actions of that worker node do not interfere with the actions of other worker nodes in the system. When interacting with the group of director nodes, the worker node sends a request to the set of director nodes (e.g., some or all of them) and waits for an approval response from at least a majority of the group of director nodes before proceeding. In some cases, the director node checks for conflicts between records submitted by the worker node and the history of approved records (i.e., records approved to be committed) known to the individual director node. Identification of a conflict may result in the director node sending a disapproval response to the worker node. In other cases, the director node may check for problems when initiating a transaction and return an approval response that includes information about previously approved records up to a certain time. If a worker node does not receive an approval response from a majority of the group of director nodes, or if it receives a disapproval response, the worker node may abort the transaction.

[0008] These techniques can be advantageous because they can shift the impact of slowdowns on a system while ensuring transactional integrity in that system. In particular, as described in more detail below, a system can be structured such that ensuring transactional integrity does not require all director nodes to respond, but only a majority. Thus, one or more director nodes can become unexpectedly slow as a result of a slowdown without slowing down the progress of transactions in the system. Consider an example in which a system includes a group of three director nodes, where an acknowledgment response from at least two director nodes (the majority) together provides a guarantee of transactional integrity. A worker node can send a commit request to all three director nodes for permission to commit the transaction. In this example, because the worker node sent a request to three director nodes and only two acknowledgment responses are needed to ensure transactional integrity, one of the director nodes may suffer a slowdown and become slow while the other two director nodes are responding. As a result, that slowdown does not slow down the entire system.

[0009] The present disclosure further describes the use of assistant nodes to mitigate some or all of the effects of slowdowns on worker nodes. In various embodiments, a worker node is associated with one or more assistant nodes that observe the work performed by the worker node. The assistant node may follow the transaction log of the worker node and store records that match records written at the worker node. In operation, a first worker node (the "read" node) may attempt to communicate with a second worker node (the "write" node whose work is being observed by the assistant node) to obtain a record from the second worker node. In some cases, the write worker node may experience a slowdown and be unable to provide a response in a timely manner. The read worker node may then communicate with the assistant node of the write worker node in an attempt to obtain the record. Because the assistant node observes the work of the write worker node, the assistant node may provide a response on behalf of the write worker node. The response may include the record. These techniques can be advantageous because they can prevent a read worker node from being slowed down (e.g., waiting for a response) by another worker node (e.g., a write worker node) that is experiencing a slowdown. This disclosure also describes concepts related to time alignment between the worker nodes and the director node of the system. An exemplary application of the techniques of this disclosure will now be described with reference to FIG. 1.

[0010] Referring now to FIG. 1, a block diagram of a system 100 is shown. The system 100 includes a set of components that may be implemented via hardware or a combination of hardware and software. In the illustrated embodiment, the system 100 includes a database 110, a worker cluster 119 having worker nodes 120, and a director cluster 130 having a director node 135. Also as shown, the database 110 and the worker nodes 120 include records 115. In some embodiments, the system 100 is implemented differently than shown. For example, the system 100 may include an assistant node that observes the work of the worker nodes, although in some cases, a worker node may act as an assistant node for another worker node.

[0011] System 100, in various embodiments, implements a platform service (e.g., a customer relationship management (CRM) platform service) that enables users of the service to develop, execute, and manage applications. System 100 may be a multi-tenant system that provides various functions to users / tenants hosted by the multi-tenant system. Thus, system 100 may execute software routines from various different users (e.g., provider and tenants of system 100) as well as provide code, web pages, and other data to users, databases (e.g., database 110), and other entities of system 100. In various embodiments, system 100 is implemented using a cloud infrastructure provided by a cloud provider. Thus, database 110, worker nodes 120, and director node 135 may run on and utilize available cloud resources (e.g., computing resources, storage resources, network resources, etc.) of the cloud infrastructure to facilitate their operation. As an example, worker nodes 120 may run within respective virtual environments hosted on server-based hardware contained within the cloud provider's data center. However, in some embodiments, the system 100 is implemented using local or private infrastructure, as opposed to a public cloud.

[0012] Database 110, in various embodiments, is a collection of information organized to allow access, storage, and manipulation of that information. Thus, database 110 may include support software (e.g., storage nodes) that allow worker nodes 120 to perform operations (e.g., access, store, etc.) on the information stored in database 110. In various embodiments, database 110 is implemented using a single or multiple storage devices connected to each other over a network (e.g., a storage attached network (SAN)) and configured to store information redundantly to prevent data loss. Because the storage devices may store data persistently, database 110 may serve as persistent storage for system 100. Information (e.g., records 115) written to database 110 by worker nodes 120 may be accessible to other worker nodes 120. As described in more detail with respect to FIG. 11 , records 115 may be stored in a log-structured merge tree (LSM tree) file as part of an LSM tree implemented in database 110.

[0013] As shown, database 110 stores records 115, which in various embodiments are key-value pairs that include data and a corresponding key that can be used to look up the record 115. As an example, a record 115 may correspond to a row of data in a database table and specify values ​​for one or more attributes / fields of the database table. In various embodiments, records 115 are immutable, so that to update an underlying database construct (e.g., a row in a table), a new record 115 is written. Thus, a database construct may be associated with multiple records 115, each of which is a different version of the database construct. These records 115 may be referred to as "record versions" of the database construct. As an example, a first record 115 (first record version) that stores a particular value of a database row may be written first, and then a second record 115 (second subsequent record version) that updates one or more of the values ​​of the database row may be written. The two records 115 may be accessible using the same key.

[0014] Worker nodes 120, in various embodiments, provide various database services such as data storage, data retrieval, and / or data manipulation. In various embodiments, worker nodes 120 are sets of software routines executable on hardware, although in some embodiments, worker nodes 120 encompass both hardware and software routines. Database services may be provided to other components within system 100 or components external to system 100. For example, worker nodes 120 may receive transaction requests 125 from application nodes (not shown) to perform database transactions. A database transaction, in various embodiments, is a logical unit of work (e.g., a specified set of database operations) to be performed in connection with database 110. For example, processing a database transaction may include executing an SQL SELECT command to select one or more rows from one or more database tables. The contents of the rows may be specified in records 115, and thus worker nodes 120 may return one or more records 115 corresponding to one or more table rows. Performing a database transaction may include a worker node 120 writing one or more records 115 to the database 110. In various embodiments, the worker node 120 first writes the records 115 to a local in-memory cache and then flushes them to the database 110. Before committing and flushing those records 115, the worker node 120 may communicate with the director node 135 to ensure that the records 115 do not conflict with record writes performed by other worker nodes 120. As used herein, the expression "committing a transaction" (including committing a record) is used in accordance with its well-understood meaning to refer to the process of causing changes made during a transaction to be saved and visible outside the entity performing the transaction.Worker nodes 120 may also communicate with director node 135 when initiating transactions to obtain information regarding the state of records stored within system 100 (e.g., the history of records 115 that have potentially been committed by other worker nodes 120 but have not been flushed to be available in database 110). The various components of worker nodes 120 are described in more detail with respect to FIG.

[0015] Director nodes 135, in various embodiments, facilitate the execution of transactions and help ensure transactional integrity of those transactions. In various embodiments, director nodes 135 are sets of software routines executable on hardware, but in some embodiments, director nodes 135 encompass hardware and software routines. As shown, director nodes 135 form director cluster 130. The number of director nodes 135 in director cluster 130 may vary depending on the embodiment. For example, director cluster 130 may include three director nodes 135, five director nodes 135, eight director nodes 135, etc. Although director nodes 135 are part of director cluster 130, in various embodiments, director nodes 135 do not communicate with each other. Specifically, when worker node 120 sends a request to two or more director nodes 135, those director nodes 135 may not communicate information about the request among each other. Thus, a director node 135 may process a request from its own knowledge and perspective, independent of the knowledge and perspective of other director nodes 135. In some embodiments, however, director nodes 135 communicate with each other to share certain information (e.g., their respective histories of approved records) that can be used to complete a request. The various components of director node 135 are described in more detail with respect to Figures 3 and 4.

[0016] In various embodiments, to process requests from the worker nodes 120, the director node 135 maintains information about the state of the records 115 stored in the worker nodes 120. When initiating a transaction, the worker node 120 may send a snapshot request to a set of director nodes 120 to obtain a snapshot time. In various embodiments, the snapshot time may be used by the owner of the snapshot time (e.g., a transaction) to identify the state of the system 100 at a particular point in time and to access the records available at that time. The concept of snapshot time is explained in more detail with respect to FIG. 2. The director node 135 may then acknowledge the snapshot time and then return a response to the worker node 120. In many cases, the response includes information about the state of the records 115 so that the worker node 120 can locate and access the records 115 that may be relevant to the processing of the transaction. However, the history information from the director node 135 may be an incomplete history of the acknowledged records. As a result, in some cases, a worker node 120 must obtain responses from a majority of director clusters 130 to obtain a complete history. An exemplary layout of history information on a director node 135 and a description of the snapshot request process are provided in more detail with respect to FIGS.

[0017] When a worker node 120 desires to commit a transaction, it may send a commit request to a set of director nodes 135 for approval to commit. When determining whether to approve the request, the director node 135 may utilize its historical information to determine whether the requested action(s) (e.g., permission to commit) would result in an inconsistent or undesirable state in the system 100. If there is a conflict, the director node 135 may reject the request, causing the worker node 120 to abort some or all of the database transaction. If there is no conflict from the director node's perspective, the director node 135 may approve the request and update its historical information regarding the state of the record 115 to reflect the approval. A description of an exemplary update of historical information and commit request process is provided in more detail with respect to FIGS. 7 and 8.

[0018] 2, a block diagram of exemplary elements of a worker node 120 is shown. In the illustrated embodiment, the worker node 120 includes a database application 200, an in-memory cache 210, and historical information 220. As further shown, the database application 200 maintains a local clock 205, the in-memory cache 210 stores records 115 associated with keys 215, and the historical information 220 includes a set of historical records 225. In some embodiments, the worker node 120 is implemented differently than is shown. For example, the worker node 120 may store a transaction log having log records. However, in some embodiments, the transaction log is stored in storage shared among the worker nodes 120, as described with respect to FIG. 10.

[0019] The database application 200, in various embodiments, is a set of program instructions executable to manage the database 110, which may include managing an LSM tree built around the database 110. Thus, the database application 200 may receive requests to perform database transactions involving reading and / or writing records 115 for the system 100. Upon receiving a transaction request, the database application 200 may initiate a database transaction based on the request, including obtaining a snapshot time for the database transaction. The snapshot time, in various embodiments, is a value indicating which records 115 can be read by the worker node 120. The value may be a numeric value indicating a time (e.g., a time window covering 1 millisecond) or a period of time. That is, the system 100 may increment an integer commit value over time, and the snapshot time may correspond to one of the commit values. Records 115 whose commit value is less than (or, in various cases, equal to) the snapshot time may be read by the worker node 120. For example, the transaction may be assigned a snapshot time of "445" (or a time of "10:11:20"). As a result, database application 220 may read records 115 for that transaction that have a commit value less than or equal to 445 (or 10:11:20). As described further below, worker node 120 may communicate with director node 135 to obtain acknowledgement of the transaction's snapshot time.

[0020] The local clock 205, in various embodiments, identifies the local time observed by the worker node 120. In some cases, the local clock 205 is provided by an operating system running on the worker node 120, and in some cases, the database application 200 maintains its own local clock 205, independent of how other applications running on the worker node 120 observe time. In various embodiments, the database application 200 performs certain database operations based on the time of the local clock 205. For example, as described further below, the database application 200 may use the clock 205 to select and propose a snapshot time (or commit time, both of which are described in more detail below) to the director node 135 for approval. In various embodiments, the worker node 120 and the director node 135 each have their own opinion of the current time in the system 100 based on their local clocks 205. However, the local clocks 205 of the worker node 120 and the director node 135 may become out of sync with each other over time. That is, their local clocks 205 may identify different times. As a result, the nodes may observe a significantly different time than other nodes in the system 100. This may cause the undesirable effect of a node (e.g., the director node 135) processing requests much later than other nodes (e.g., other director nodes 135). As will be described in more detail with respect to FIG. 9, the nodes in the system 100 may attempt to gradually align their times so that the nodes do not get too far out of sync.

[0021] In-memory cache 210, in various embodiments, is a buffer that stores data in memory (e.g., random access memory) of worker node 120. HBase™ memstore is one example of an in-memory cache 210. After getting sufficient acknowledgements for the transaction's snapshot time, worker node 120 may begin reading and writing records 115 to process the transaction. In various embodiments, worker node 120 first writes record 115 to its in-memory cache 210 and then flushes record 115 to database 110 after being committed by worker node 120. In various cases, the latest / newest record version of a row in a database table is found in record 115 stored in in-memory cache 210. However, in some embodiments, records 115 written to in-memory cache 210 of worker node 120 are not visible to other worker nodes 120. That is, other worker nodes 120 do not know what information is stored in in-memory cache 210 unless asked or told. Thus, in various embodiments, a worker node 120 obtains historical information 220 from the director node 135 that identifies where a particular record 115 (e.g., the latest record version of a database row) can potentially be found. Thus, a worker node 120 may use the historical information 220 to identify which records 115 are stored in the in-memory caches 130 of other worker nodes 120.

[0022] The historical information 220, in various embodiments, includes a collection of historical records 225. The historical records 225, in various embodiments, include information identifying the location of the records 225 in the system 100 for a key 215 that can be used to look up the record 115 corresponding to that historical record 115. In various embodiments, the historical information 220 captures different records (having different keys) and different versions of the same database construct (e.g., a data row) identified with the same key. Thus, the historical information 220 may include multiple historical records 225 for multiple versions of the same database construct that were committed by different transactions at different times using the same key 215. When looking for a particular record version of a database construct (e.g., the latest record 115 of a database row), the worker node 120 may identify a set of historical records 225 based on the key 215 that corresponds to the database structure. The worker node 120 may then identify which historical record 225 corresponds to the record version from the timestamp, and then attempt to access the record 115 using that historical record 225. In some embodiments, a single history record 225 may store the locations of multiple records 115 for the same key 215.

[0023] When attempting to access a record 115 from a location outside of the worker node 120, in various embodiments, the worker node 120 sends a record request 204 to the location. For example, as shown, the worker node 120 sends the record request 204 to the database 110. The record request 204 may specify a key 215 corresponding to the record 115 that the recipient can use to attempt to locate the record 115. The worker node 120 may then receive a record response 206. In some cases, the record response 206 does not include the requested record 115. However, in some cases, the record response 206 does not include the requested record 115, even though the history record 225 indicated that the record 115 was at that location. For example, the director node 135 may create a history record 225 indicating that the record 115 may have been committed at the worker node 120. However, after being granted permission to commit the transaction associated with that record 115, that worker node 120 may abort the transaction and delete the record 115. As a result, upon receiving a record request 204 for the record 115, that worker node 120 returns a record response 206 indicating that the worker node 120 did not write the record 115 at the time specified in the received record request 204.

[0024] After processing a transaction (e.g., after writing all requested records 115 for the transaction), in various embodiments, the worker node 120 sends a commit request to a majority of the director nodes 135 of the director cluster 130. The commit request may include the records 115 written for the transaction so that the director nodes 135 can check for record conflicts. In response to receiving approval from the majority of the director nodes 135 without disapproval, the worker node 120 may commit the records 115. In some embodiments, disapproval is tolerated as long as a majority of the director clusters 130 approve the commit request. As part of the commit, in various embodiments, the worker node 120 stamps each record 115 with a value (e.g., a time, a number indicating a duration, etc.) indicating when the record 115 was committed. Committed records 115 may remain in the in-memory cache 210 until a triggering event (e.g., the in-memory cache 210 is nearly full or is full) causes those records 115 to be flushed to the database 110. Records 115 stored in the in-memory cache 210 are accessible to and may be used in other transactions whose snapshot times / values ​​are greater than the commit time / value of the records 115 being accessed.

[0025] 3, a block diagram of exemplary elements of a director node 135 is shown. In the illustrated embodiment, the director node 135 includes an orchestration application 300 and history information 220. As further shown, the orchestration application 300 maintains a local clock 205, and the history information 220 includes history records 225. In some embodiments, the director node 135 is implemented differently than is shown. For example, the director node 135 may interface with other director nodes 135 in the director cluster 130.

[0026] In various embodiments, the orchestration application 300 is a set of program instructions executable to facilitate the execution of transactions and ensure the transactional integrity of those transactions. To facilitate the execution of transactions, in various embodiments, the orchestration application 300 provides some or all of its history information 220 to the worker nodes 120 to enable the worker nodes 120 to learn about the work performed by other worker nodes 120. As described, in various embodiments, committing a record 115 includes communicating with a director node 135 to obtain approval to commit the record 115. Because a director node 135 approving the commit knows that the record 115 is potentially to be committed, the orchestration application 300 can store an indication of that approval as part of the history information 220. In some cases, the orchestration application 300 may miss a commit request (or otherwise fail to process a request) and therefore may not store a complete history of all recent records 115 that have been approved for commitment by the director cluster 130. As a result, in various embodiments, the history information 220 identifies only those records 115 for which a commitment has been approved by that orchestration application 300.

[0027] Because the orchestration application 300 tracks records 115 for which commitment has been approved, the orchestration application 300 can facilitate execution of a transaction by enabling a worker node 120 to potentially locate records 115 relevant to executing that transaction, including records 115 stored in the in-memory cache 210. Also, because the orchestration application 300 tracks records 115 that have been approved at its director node 135, in various embodiments, it can help ensure transaction integrity between transactions by ensuring that an approval request record 115 does not conflict with an approved record 115 identified in the director node's history information 220. Thus, upon receiving a commit request for a record 115, the orchestration application 300 can determine whether another record 115 having the same key 215 (i.e., associated with the same database reconstruction) has already been approved within a particular time frame. In various cases, the time frame corresponds to the time between the worker node's snapshot time for the transaction and the current time observed by the director node 135. If a conflicting approved record 115 exists, the orchestration application 300 may return a disapproval response to the requesting worker node 120 so that the requested record 115 is not committed and transaction integrity is ensured. If a conflicting record 115 does not exist, the orchestration application 300 may return an approval response and add a history record 225 corresponding to the new approved record 115 to the history information 220. Note that in some cases, a director node 135 may actually detect a conflict with a previous transaction when that previous transaction was not approved by a majority of the director nodes 135. In this case, if enough other director nodes 135 approve the transaction, the transaction may be committed and the conflict perceived at that director node 135 was not a true conflict.

[0028] As shown, the orchestration application 300 maintains a local clock 205 that identifies the current time as observed at the director node 135 (or at least as observed by the orchestration application 300, which may be independent of the time observed by other applications executing on the director node 135). Like the worker nodes 120, the director nodes 135 may perform certain operations at a specified time according to their respective local clocks 205. As an example, a commit request may be processed at a commit time proposed in the commit request. As described above, the clocks 205 of the director node 135 and the worker nodes 120 may become out of sync, and thus a director node 135 may process a request at a logical time that is substantially slower than another director node 135. Thus, the nodes of the system 100 may attempt to gradually align their times so as not to get too out of sync, as will be described in more detail with respect to FIG. 9.

[0029] 4, a block diagram of an example history record 225 is shown. In the illustrated embodiment, the history record 225 specifies a record identifier 410, a position 420, and a commit time / epoch 430. In some embodiments, the history record 225 is implemented differently than is shown. As an example, the history record 225 may specify a director identifier for the director node 135 that wrote the history record 225.

[0030] The record identifier 410, in various embodiments, is information that identifies the record(s) 115 that correspond to the history record 225. In various cases, the record identifier 410 is a key 215 that can be used to look up the corresponding record(s) 115. The record identifier 410 may be specified in a commit request sent from a worker node 120 to a director node 135 to obtain approval to commit a set of corresponding records 115. In response to deciding to approve the commit request, the director node 135 may generate, for each record 115 in the set, a history record 225 whose record identifier 410 corresponds to that record and is obtained from the commit request.

[0031] Location 420, in various embodiments, identifies the location within system 100 of record 115 that corresponds to historical record 225. For example, location 420 may specify a worker node 120 or database 110. In various cases, location 420 identifies the worker node 120 that initially obtained permission to commit the corresponding record 115. When that record 115 is committed and flushed to database 110, location 420 may be updated by director node 135 (which created the corresponding historical record 225) to identify the location within database 110 where that record 115 was stored. Thus, over time, location 420 may be adjusted as the corresponding record 115 moves within system 100.

[0032] The commit time 430, in various embodiments, identifies a logical time (or time window) at which the associated record 115 is identified as being committed. In various embodiments, the worker node 120 may commit the record 115 of the transaction at the commit time 430, or may choose not to commit it. However, the worker node 120 cannot commit the transaction at any other commit time 430 without re-requesting approval at the new commit time 430. In some cases, the record 115 is actually committed at a physical time that is later than the logical time of the commit time 430. In particular, the commit request may specify a proposed commit time. The director node 135 may process the request at the proposed commit time, and if the request is approved, may generate a history record 225 whose commit time 430 is the proposed commit time. The requesting worker node 120 may receive the approval response and may commit the record 115 of the commit request at the proposed commit time 430 (if it chooses to commit).

[0033] 5, a block diagram of an exemplary snapshot-related interaction is shown involving one worker node 120 and three director nodes 135A-C. In some embodiments, there may be more director nodes 135 than shown (e.g., five director nodes 135), and a worker node 120 may issue snapshot requests 510 to fewer than all of the director nodes 135 in the director cluster 130 (e.g., four of the five).

[0034] As described above, a worker node 120 may receive a request to perform a transaction, which may include reading and writing records 115. The request may be received from an application node, not shown. As part of processing a transaction, in various embodiments, the worker node 120 obtains a snapshot time for that transaction, which corresponds to the state of the system 100 at a particular point in time. Records 115 committed before the snapshot time may be accessed by that worker node 120 for the transaction, and when committing a transaction, the snapshot time may serve as a reference point for checking for record conflicts.

[0035] To ensure that a transaction is assigned a single snapshot time, in various embodiments, the worker node 120 selects the proposed snapshot time 505 on behalf of the director node 135. In various embodiments, the director nodes 135 may not communicate with each other, or at least may not communicate snapshot information, so the worker node 120 may select the snapshot time on behalf of the director node 135. In various embodiments, the director nodes 135 may not communicate with each other, so a situation may arise where a director node 135 selects a different snapshot time than another director node 135. Thus, a transaction may be associated with multiple snapshot times, which may violate transaction guarantees. However, in some embodiments, the director node 135 may assign a transaction a snapshot time 505 that corresponds to the time when the snapshot request 510 from the worker node 120 was seen by the director cluster 130.

[0036] As described above, the director node 135 may process requests received at a specified time. As a result, when selecting a proposed snapshot time 505, in various embodiments, the worker node 120 selects the proposed snapshot time 505 such that a corresponding snapshot request 510 is estimated to arrive at the director node 135 before the proposed snapshot time 505 according to the respective local clock 205 of the director node 135. If the snapshot request 510 arrives at the director node 135 early, the director node 135 may store the request and delay its processing until the proposed snapshot time 505 identified in the snapshot request 510 is reached according to the local clock 205 of the director node 135. In various cases, the director node 135 processes requests received at the same local time but at different times from a global perspective. If a snapshot request 510 arrives at a director node 135 after the proposed snapshot time 505, the director node 135 may issue a snapshot response 520 indicating that the proposed snapshot time 505 was rejected. In some embodiments, worker nodes 120 send snapshot requests 510 to director nodes 135 to allow them to select snapshot times 505 for their corresponding transactions.

[0037] In some embodiments, in response to receiving a rejection for the proposed snapshot time 505, the worker node 120 selects another proposed snapshot time 505 that is further away from the worker node's 120 current time than the rejected snapshot time 505 was from the time selected by the worker node 120. As described in more detail with respect to FIG. 9, the worker node 120 may track delays in communicating with the director node 135 and use those delays to adjust how far into the future the proposed snapshot time 505 is selected. In some embodiments, in response to receiving a rejection for the proposed snapshot time 505 but also receiving an approval from at least a majority of the director nodes 135 of the cluster 130, the worker node 120 proceeds to execute the transaction taking into account the proposed snapshot time 505. Thus, in some cases, if a particular director node 135 is moderately behind in time than other director nodes 135, the worker node 120 may select a proposed snapshot time 505 such that the corresponding snapshot request 510 is estimated to arrive at the other director nodes 135 before the proposed snapshot time 505, even if the corresponding snapshot request 510 is estimated to arrive at the particular director node 135 after the proposed snapshot time 505. Thus, when selecting a proposed snapshot time 505, the worker node 120 may consider a trade-off between reaching more director nodes 135 (which increases the chances of receiving responses from the majority) and selecting a proposed snapshot time 505 that is not too far in the future (which may delay transactions).

[0038] In response to receiving the snapshot request 510 with the proposed snapshot time 505, in various embodiments, the director node 135 determines whether to reject the proposed snapshot time 505. If the proposed snapshot time 505 is after the current time at the director node 135 or would cause a conflict with another transaction, the director node 135 may reject it. In response to approving the proposed snapshot time 505, in various embodiments, the director node 135 returns a snapshot response 520 with the director node's history information 220 up to the proposed snapshot time 505. As shown, the worker node 120 receives the snapshot response 520 from the director nodes 135A-B, but not from the director node 135C. Assuming that director cluster 130 includes only director nodes 135A-C and the received snapshot response 520 approves the proposed snapshot time 505, worker node 120 may proceed to process the transaction because approval was received from a majority of director cluster 130 with no disapproval.

[0039] 6, a block diagram of an example layout of history information on director nodes 135A-C is shown. In the illustrated embodiment, there is a complete history 600 that includes four history records 225A-D. As shown, director node 135A includes history records 225A and 225C-D, director node 135B includes history records 225A-B, and director node 135C includes history records 225A-D. In some cases, complete history 600 may include more or fewer history records 225 than shown, and the layout of the history information on director nodes 135 may differ.

[0040] 5, worker node 120 receives snapshot responses 520 from director nodes 135A-B, but not from director node 135C. As explained, snapshot response 520 may include history information 220 including history records 225 up to proposed snapshot time 505. Assuming proposed snapshot time 505 is after commit time 430 of history record 225D, in this example worker node 120 receives history records 225A and 225C-D from director node 135A, and history records 225A-B from director node 135B. In various embodiments, if worker node 120 receives approval from a majority of director nodes 135, worker node 120 is guaranteed to obtain a complete history 600, since a majority of director nodes 135 must approve the commit, and thus have history records 225 of approved records 115. Continuing with the example, the worker node 120 obtains the complete history 600 from the director nodes 135A-B (which are the majority in this example), even though none of these director nodes store the complete history 600. To obtain the complete history 600, the worker node 120 may union the history records 225 received from the director nodes 135A-B. If the worker node 120 receives snapshot responses 520 from less than the majority of the director nodes 135, there is no guarantee that the worker node 120 will obtain the complete history 600. As an example, if the worker node 120 receives snapshot responses 520 only from the director node 135B, the worker node 120 obtains an incomplete history 600.

[0041] 7, a block diagram of an exemplary commit-related interaction is shown involving a worker node 120 and three director nodes 135A-C. In some embodiments, there may be more director nodes 135 than shown (e.g., five director nodes 135), and a worker node 120 may issue a commit request 710 to fewer than all of the director nodes 135 in the director cluster 130 (e.g., four of the five).

[0042] After processing the transaction, the worker node 120 may decide to commit the transaction, including one or more records 115 written to the worker node's in-memory cache 210 for that transaction. To ensure that the records 115 do not conflict with records 115 written by other worker nodes 120, the worker node 120 issues a commit request 710 to at least a majority of the director nodes 135. As shown, the worker node 120 issues the commit request 710 to director nodes 135A-C, which in the illustrated embodiment are all of the director nodes 135. The commit request 710, in various embodiments, identifies the records 115 written for the transaction (e.g., via record identifiers 410), a proposed commit time 705 to be applied to those records 115, and an approved snapshot time for the transaction. Similar to the proposed snapshot time 505, the proposed commit time 705 may identify a numerical value indicating a time (or time window) or period.

[0043] In various embodiments, similar to selecting a snapshot time, the worker node 120 may select a commit time on behalf of the director node 135 since the director nodes 135 may not communicate with each other, or at least may not communicate about commits. When selecting a proposed commit time 705, in various embodiments, the worker node 120 selects the proposed commit time 705 such that a corresponding commit request 710 is estimated to arrive at the director node 135 before the proposed commit time 705 according to the local clock 205 of each of the director nodes 135. Similar to the snapshot request 510, if the commit request 710 arrives early at the director node 135, the director node 135 may store the request and delay its processing until the proposed commit time 705 identified in the commit request 710 is reached according to the local clock 205 of that director node 135. If the commit request 710 arrives after the proposed commit time 705, the director node 135 may issue a commit response 720 indicating that the commit request 710 is rejected. As described with respect to the proposed snapshot time 505, in various embodiments, the worker node 120 may also adjust how far into the future the proposed commit time 705 is selected. For example, in response to receiving a rejection of the commit request 710 based on its proposed commit time 705, the worker node 120 may select another proposed commit time 705 that is further away from the worker node's 120 current time than the selected time is from the rejected commit time 705.

[0044] In response to receiving the commit request 710, in various embodiments, the director node 135 determines whether to reject the commit request 710. As part of determining whether to reject the commit request 710, the director node 135 may determine whether any of the records 115 identified in the request conflict with records 115 previously approved for commit by that director node 135. In various embodiments, the director node 135 records the approved records 115 in the form of history records 225, so that the director node 135 can examine the history records 225 in its history information 220. In various embodiments, a first record 115 conflicts with a second record 115 if they are associated with the same record identifier, and the second record 115 is identified as having been committed after the snapshot time associated with the transaction of the first record 115. Thus, the director node 135 may search its history records 225 for ones whose record identifier 410 matches the record identifier of the record 115 being considered for approval. If a history record 225 is found, the director node 135 may determine whether its commit time 430 occurs after the snapshot time specified in the commit request 710. If a conflicting record 115 exists, the director node 135 sends a disapproved commit response 720 to the worker node 120 indicating that a record conflict exists. If a conflicting record 115 does not exist, the director node 135 sends an approved commit response 720. The director node 135 may also create a history record 225 for the record 115 of the approved transaction.

[0045] As shown, worker node 120 receives commit responses 720 from director nodes 135A and 135C, but not from director node 135B. Assuming that director cluster 130 includes only director nodes 135A-C and the received commit responses 720 approve the commit, worker node 120 may proceed to commit the transaction because approval was received from a majority of director cluster 130 without any disapproval. In response to receiving a disapproval of the proposed commit time 705, worker node 120 may select a new proposed commit time 705 and send a commit request 710 with the proposed commit time 705 to director nodes 135A-C, or worker node 120 may abort the transaction. In response to receiving a commit response 720 that does not approve the commit due to one or more record conflicts, the worker node 120 may or may not abort the transaction even if the worker node 120 received approval from a majority of the director cluster 130. In some embodiments, the director node 135 that approves the commit records (e.g., via history record 225) that the commit occurred even if the transaction was aborted by the worker node 120.

[0046] 8, a block diagram of an example layout of history information on director nodes 135A-C is shown. In the illustrated embodiment, there is a complete history 600 that includes five history records 225A-E. As shown, director node 135A includes history records 225A and 225C-E, director node 135B includes history records 225A-B, and director node 135C includes history records 225A-E. In some cases, complete history 600 may include more or fewer history records 225 than shown, and the layout of the history information on director nodes 135 may differ.

[0047] To facilitate the following discussion, it is assumed that the exemplary layout of history information shown in FIG. 6 occurs before the exemplary commit-related interaction between the worker node 120 and the three director nodes 135A-C shown in FIG. 7. Also, the exemplary layout of history information shown in FIG. 8 occurs after the exemplary commit-related interaction. As described above, in various embodiments, when the director node 135 decides to approve a commit, the director node 135 stores a history record 225 that identifies the records 115 to be committed and where they are assumed to be committed. In FIG. 7, the illustrated worker node 120 receives a commit response 720 from director nodes 135A and 135C, but not from director node 135B. Assuming that a transaction including a particular record 115 is approved to commit, the director nodes 135A and 135C, in this example, individually store a history record 225E for that particular record 115. 7. However, director node 135B does not store history record 225E because it may be experiencing a slowdown and may not be able to process the commit request 710 it received in FIG. 7. As a result, the history of approved records known to one director node 135 may differ from the history of approved records known to another director node 135. Because the transaction was approved by a majority and history record 225E was stored in the majority of director nodes 135, worker nodes 120 that receive acknowledgments from a majority of director nodes for snapshot request 510 will receive history record 225E, making them aware of the particular record 115.

[0048] In various embodiments, a director node 135 deletes historical records 225 from its historical information 220 over time. In various cases, a new record version of a database construct (e.g., a database row) is approved for a commitment to replace an earlier record version. Thus, the director node 135 may replace the earlier record version of historical record 225 with a new historical record 225 of the new record version. In some cases, the director node 135 deletes a historical record 115 in response to the record 225 becoming older than a certain threshold. For example, a record 115 associated with historical record 225A may be written out to the LSM tree of the database 110 and merged down a level of the LSM tree over time. After that record 115 reaches a certain level, the director node 135 may delete the historical record 225A from its historical information 220 (if that director node 135 has approved the transaction for that record 115 and created the historical record 225A). As part of deleting the history record 225 from its history information 220 , in some embodiments, the director node 135 sends the history record 225 to the database 110 so that it becomes accessible to the worker node 120 from the database 110 .

[0049] 9, a block diagram of exemplary elements related to aligning time among components of system 100 is shown. In the illustrated embodiment, there is a worker node 120 and three director nodes 135A-C, all having their own local clocks 205. As further illustrated, the local clocks 205A and 205B of director nodes 135A and 135B, respectively, identify the time "10:11:20", the local clock 205C of director node 135C identifies the time "10:11:23", and the local clock 205D of worker node 120 identifies the time "10:11:19". Although only three director nodes 135 and one worker node 120 are shown, in various embodiments there may be more director nodes 135 and / or worker nodes 120. Also, in some embodiments, the times are represented differently than shown. For example, the times may be numerical values ​​indicating a duration, may be in minutes, etc.

[0050] As described, in various embodiments, a node of the system 100 includes a local clock 205 that identifies the time observed by that node for the system 100. That time may differ from the time observed by another node. For example, the worker node 120 observes the time "10:11:19" and the director node 135 observes the time "10:11:23." As further described, a node may perform operations at a specified time that may affect other nodes. A node that observes a time that is reasonably different from the average time observed in the system 100 may cause undesirable delays as one or more operations are performed later than expected by other nodes. It may therefore be desirable to continually attempt to align the time in the system 100 so that the nodes do not get too far out of sync with one another.

[0051] In some embodiments, the worker nodes 120 and the director node 135 communicate their local times directly to each other. However, in various embodiments, the worker nodes 120 and the director node 135 do not communicate their local times to each other. Instead, time delays in communication between the worker nodes 120 and the director node 135 are tracked and may be used by the nodes to shift their local clocks 205 in an attempt to align their time with the time observed by other nodes. In various embodiments, when a worker node 120 issues a request (e.g., a snapshot request 510 or a commit request 710) to a director node 135, the worker node 120 stores a send timestamp 910 that specifies the time the request was issued according to its local clock 205. When the request is received at the director node 135, it may store an arrival timestamp 920 that specifies the time the request was received according to its local clock 205. When that director node 135 returns a response (e.g., snapshot response 520 or commit response 720) to the worker node 120, the response may include the arrival timestamp 920. The worker node 120 may calculate a time delay that is the difference between the send timestamp 910 and the arrival timestamp 920. When the worker node 120 issues another request to the director node 135, it may include the time delay in the request. As a result, the director node 135 may learn about the time delays associated with the worker node 120 communicating with the director node 135 and other director nodes 135 in the director cluster 130.

[0052] For example, worker node 120 may issue a commit request 710 to director nodes 135A-C and store a send timestamp 910 that specifies the time "10:11:17." In response to receiving that commit request 710, director node 135A may store an arrival timestamp 920A that specifies the time "10:11:20" according to local clock 205A, director node 135B may store an arrival timestamp 920B that specifies the time "10:11:20" according to local clock 205B, and director node 135C may store an arrival timestamp 920C that specifies the time "10:11:23" according to local clock 205C. Those arrival timestamps 920 may be provided to worker node 120 in a commit response 720 to the commit request 710. Based on send timestamp 910 and arrive timestamps 920A-C, worker node 120 determines that there was a three second time delay when communicating with director nodes 135A and 135B, and a six second time delay when communicating with director node 135C. When worker node 120 communicates again with director node 135B, worker node 120 can provide those time delays.

[0053] Based on those time delays, in various embodiments, director node 135 determines whether its local clock 205 should speed up or slow down. For example, director node 135C may learn that commit request 710 took 6 seconds to arrive, compared to 3 seconds for director nodes 135A and 135B. From this, director node 135C may determine that its local clock 205C identifies a time that is approximately 3 seconds ahead of the time observed by director nodes 135A and 135B. Often, director node 135 attempts to consider factors that affect the time delay between director node 135 and worker node 120. Such factors may include, for example, the network distance between worker node 120 and director node 135, the number of devices a request passes through on the way from one node to another, the delay between worker node 120 sampling the time and sending the request, and the delay between director node 135 receiving the request and sampling the time. Based on one or more of these factors, the director node 135 may estimate the actual clock skew based on the time delay. In various embodiments, the worker nodes 120 maintain an average of the time delays (which may be weighted for recent traffic) in communicating with each director node 135. The director node 135 may receive the average time delays from multiple worker nodes 120 and then aggregate those time delays to generate an aggregated average time delay. Thus, in various embodiments, the director node 135 learns the average time delay across multiple worker nodes 120 in communicating with a given director node 135. The aggregated average time delay may allow the director node 135 to better estimate the actual clock skew, taking into account factors that may cause delays in communication.

[0054] Thus, director node 135 may align their times very gradually. By comparing the perceived delay in receiving requests from each worker-director pair, director node 135 may determine if its time is out of sync with other nodes based on a weighted average of the time delays of the other director nodes 135. In response to deciding to speed up its local clock 205, in various embodiments, director node 135 shifts the time of its local clock 205 forward and increases its own execution speed (e.g., by running a processor core more frequently) to increase the number of requests it can process so that its local clock 205 can be shifted at a faster rate. In some cases, director node 135 may shift the time of its local clock 205 forward and skip work (e.g., requests) whose specified processing time falls within the time being shifted. In response to deciding to slow down its local clock 205, in various embodiments, director node 135 decreases the rate at which its local clock 205 time is shifted forward. In some cases, a director node 135 may freeze the time on its local clock 205 until the director node 135 considers the time aligned.

[0055] In various embodiments, the worker node 120 shifts the time of its local clock 205 based on the observed time delay. As an example, the worker node 120 may observe that its requests are taking longer and longer to reach the director node 135. Based on this observation, the worker node 120 may speed up its local clock 205. Alternatively, the worker node 120 may observe that its requests are reaching the director node 135 sooner than expected, and therefore may slow down its local clock 205. In various embodiments, the worker node 120 also uses the observed time delay to determine how far into the future the proposed snapshot time 505 or commit time 705 should be selected to arrive before their proposed time at the director node 135 according to their respective local clock 205. As the time delay changes over time, the worker node 120 may adjust how far into the future the proposed snapshot time 505 or commit time 705 is selected.

[0056] 10, a block diagram of exemplary elements related to an assistant node 1030 is shown. In the illustrated embodiment, there are worker nodes 120A-B, an assistant node 1030, and a shared storage 1040. As shown, the worker node 120B and the assistant node 1030 include records 115, and the shared storage 1040 includes logs 1045. The illustrated embodiment may be implemented differently than shown. As an example, there may be multiple assistant nodes 1030 that observe the work performed by the worker nodes 120.

[0057] As described, processing a transaction may include accessing one or more records 115, which may be stored in the database 110 or in-memory cache 210 of a worker node 120. If a record 115 is identified to be in the in-memory cache 210 (e.g., via historical records 225), the worker node 120 may send a record request 204 for that record 115 to a corresponding worker node 120. For example, as shown, worker node 120A issues a record request 204 to worker node 120B for a record 115 stored in worker node 120B. As with other components of the system 100, a worker node 120 may experience slowdowns that cause the worker node 120 to be slow or unresponsive to record requests 204. As a result, the worker node 120 may delay the processing of transactions for other worker nodes 120. For example, worker node 120B may become unresponsive, such that worker node 120A may not receive record responses 206 to its record requests 204. To help alleviate this problem, in various embodiments, a worker node 120 has one or more assistant nodes 1030 that can answer record requests 204 on behalf of the worker node 120 when it experiences a slowdown.

[0058] In various embodiments, the assistant node 1030 observes the work performed by the worker node 120B and locally stores information corresponding to the work. In various embodiments, the assistant node 1030 is a set of software routines executable on hardware, and in some embodiments, the assistant node 1030 includes both hardware and software routines. In some embodiments, the assistant node 1030 is another worker node 120. As a result, the worker nodes 120 can be assigned to each other such that each worker node 120 is an assistant node 1030 for another worker node 120. In some embodiments, the assistant node 1030 runs with the other assistant nodes 1030 on a single virtual machine, while in other embodiments, they each run on a separate virtual machine.

[0059] To observe the work performed by worker node 120B, in various embodiments, assistant node 1030 follows worker node 120B's log 1045, reads all record changes, and places them in its own memory (e.g., memory cache 210). As shown, log 1045 is kept in a shared storage 1040 shared between worker node 120 and assistant node 1030. In some embodiments, a worker node's log 1045 is stored locally at worker node 120 rather than in shared storage 1040, and thus, that assistant node(s) 1030 may access log 1045 from that worker node 120 rather than from shared storage 1040. Shared storage 1040 may be part of database 110 or a separate storage mechanism. Log 1045, in various embodiments, is a collection of log records that identify changes (e.g., record changes) to information stored in a database (e.g., database 110). A log record may be a record that identifies a new data value for a database construct (e.g., a database row), a commit record that identifies the commit of a transaction, or a rollback record that allows some or all of the database to be rolled back. Thus, the log 1045 may identify pending and committed transactions, including the ordering in which they were committed. A worker node 120 that writes a transaction's record 115 may update its log 1045 by appending a log record that identifies its work. Thus, the log 1045 may grow over time as more log records are appended by the worker node 120.

[0060] Thus, in various embodiments, assistant node 1030 reads log records written to worker node 120B's log 1045 and builds an in-memory image of recent transactions performed by worker node 120B. Thus, as worker node 120B writes records 115 to its cache 210 and log records to its log 1045, assistant node 1030 can learn about those records 115 from its log 1045 and recreate them in its own memory. As a result, assistant node 1030 can store one or more of the same records 115 stored in worker node 120B. Because assistant node 1030 follows worker node 120B's log 1045 and stores one or more of the same records 115, assistant node 1030 can process record requests 204 from worker node 120A as if it were worker node 120B.

[0061] In some embodiments, a worker node 120A seeking a record 115 first sends a record request 204 to a worker node 120B that has been identified as having committed that record 115 (although it may not have been committed if the transaction was aborted after the commit was approved). If that record request 204 times out and there is no record response 206, the worker node 120A may send the record request 204 to the assistant node 1030 of the worker node 120B. In some embodiments, the worker node 120A may send the record request 204 to the worker node 120B and its assistant node(s) 1030 in parallel. This may save time at the expense of sending more requests than may be needed.

[0062] In response to receiving the record request 204, in various embodiments, the assistant node 1030 determines whether the worker node 120B committed the requested record 115. In some cases, the assistant node 1030 determines that the requested record 115 has been committed (e.g., via a commit record written to the log 1045 of the worker node 120B). The assistant node 1030 may access that record 115 from its own memory and return it to the worker node 120A in the record response 206. In some cases, the assistant node 1030 determines that the requested record 115 does not exist and that the worker node 120B logged work past the time associated with the transaction that had the requested record 115. For example, the worker node 120B may abort the transaction rather than commit it and then commit another transaction associated with a later commit time, and thus the assistant node 1030 may determine that work has been logged past the transaction that has the requested record 115. As a result, the assistant node 1030 may return a record response 206 to the worker node 120A indicating that the requested record 115 does not exist. In some cases, the assistant node 1030 has not progressed far enough through the log 1045 of the worker node 120B to know whether the requested record 115 has been committed. The assistant node 1030 may delay the response until it has fully read the log 1045 of the worker node 120B, or may provide the worker node 120A with a record response 206 instructing the worker node 120A to retry its record request 204 later. In the former case, the assistant node 1030 may ensure that a record in the log 1045 of the worker node 120B having a timestamp indicating a time prior to the snapshot time specified in the record request 204 has been read.

[0063] In some cases, the assistant node 1030 determines that the log 1045 of the worker node 120B has not reached the point of committing the transaction of the requested record 115, and that the worker node 120B is not writing to its log 1045 or is writing at a very slow rate according to a defined threshold. Based on the fact that the worker node 120B has not updated its log 1045 and the requested record 115 has not been committed, in various embodiments, the assistant node 1030 fences the log 1045 of the worker node 120B and then returns a record response 206 to the worker node 120A indicating that the requested record 115 does not exist. Since the assistant node 1030 received the record request 204 from the worker node 120A, it is implied that the worker node 120A has obtained the history record 225 linked to the requested record 115. Since worker node 120A has its history record 225, it is suggested that worker node 120B has received permission to commit the transaction for requested record 115. In this case, to provide a record response 206 indicating that requested record 115 has not been committed, assistant node 1030 fences worker node 120B's log 1045, preventing the requested record 115 from being committed after assistant node 1030 tells worker node 120A that the requested record 115 has not been committed. To fence worker node 120B's log 1045, in various embodiments, assistant node 1030 sends a command to shared storage 1040 to cause shared storage 1040 to disable appending to worker node 120B's log 1045. As a result, worker node 120B cannot write log records and therefore cannot perform work for system 100. However, the contents of the log 1045 of the worker node 120B remain intact and can be read by other nodes (e.g., the assistant node 1030).After worker node 120B's log 1045 is fenced, assistant node 1030 may take a final pass in log 1045 to determine whether a commit has been added for the transaction of the requested record 115. If the transaction has been committed, assistant node 1030 may return the requested record 115 to worker node 120A, otherwise assistant node 1030 may indicate to worker node 120A that the requested record 115 has not been committed.

[0064] 11, a block diagram of a flush operation performed by a worker node 120 and an assistant node 1030 is shown. In the illustrated embodiment, there is a database 110, a worker node 120, and an assistant node 1030. As shown, both the worker node 120 and the assistant node 1030 contain the same record 115A. That is, they each have a record 115 that stores the same information (e.g., value) for the same database construct (e.g., database row). Also as shown, the database 110 contains a log-structured merge tree (LSM tree) 1100 that receives the same identical record 115 from the worker node 120B and the assistant node 1030. The illustrated embodiment may be implemented differently than shown. As an example, there may be multiple assistant nodes 1030 that write the same record 115 as the worker node 120 to the database 110.

[0065] The log-structured merge tree 1100 is a data structure that, in various embodiments, stores files in an organized manner using a level-based scheme. The LSM tree 1100 may include two high-level components: an in-memory component implemented in the in-memory caches 210 of the worker nodes 120, and an on-disk component implemented in the database 110. In some cases, the in-memory component is considered separate from the LSM tree 1100. The worker nodes 120 may initially write records 115 to their in-memory caches 210. As their caches 210 become full and / or at some point, the worker nodes 120 may flush the records 115 to the database 110. As part of flushing the records 115, in various embodiments, the worker nodes 120 write them to a new file at the top level of the LSM tree 1100. Over time, the records 115 of these files are merged downward through the levels of the LSM tree 1100 as newer files are written to the top level. Although the various techniques described with respect to FIG. 11 relate to LSM trees, such techniques can be applied to other types of database implementations where there are worker nodes that can write and commit records to the database.

[0066] In the following description, it is assumed that the worker node 120 commits the record 115A. In some cases, before the worker node 120 flushes the record 115A to the LSM tree 1100, the worker node 120 may experience a slowdown that makes the worker node 120 catatonic and unable to flush the record 115A in a timely manner. The efficiency of the system 100 may depend on the speed at which the committed records 115 are flushed to the LSM tree 1100. Thus, the catatonicity of the worker node 120 may result in an undesirable slowdown of the system 100. Thus, in various embodiments, both the worker node 120 and the assistant node 1030 flush the same record 115 to the LSM tree 1100. As a result, while the worker node 120 is unable to flush the record 115A in a timely manner due to a slowdown, the assistant node 1030 may independently flush the record 115A to the LSM tree 1100.

[0067] In various cases, the LSM tree 1100 stores a copy of the same record 115 as a result of both the worker node 120 and the assistant node 1030 flushing that record 115. As described above, a merge operation may be performed that moves the record 115 through a level of the LSM tree 1100. In various embodiments, the merge operation removes the copy so that only one copy of the record 115 is moved to the next level of the LSM tree 1100. Thus, the LSM tree 1100 may temporarily store a copy of the record 115 that is ultimately removed by a merge operation performed by the system 100.

[0068] 12, a flow diagram of a method 1200 is shown. Method 1200 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1200 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1200 includes more or fewer steps than shown. For example, method 1200 may include a step in which a worker node sends a request (e.g., record request 204) to another worker node to retrieve a record (e.g., record 115).

[0069] Method 1200 begins with a worker node receiving a request to execute a transaction that includes writing a record, at step 1210. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions in the database system and multiple director nodes (e.g., director node 135) operable to ensure transactional integrity of the transactions. The worker nodes may write the records to an in-memory cache (e.g., in-memory cache 210) of the worker nodes.

[0070] In step 1220, the worker node issues a request (e.g., snapshot request 510, commit request 710, etc.) to some of the director nodes (e.g., a majority subset or all) for information (e.g., history records 225) that facilitates execution of an action for the transaction. In some cases, the request for information may specify at least a record, and the action may correspond to committing the transaction. A given one of the director nodes may decide whether to approve the transaction to be committed based on whether the record conflicts with a history of approved records (e.g., history information 220) known to the given director node. The request to the director node may further specify a snapshot time of the transaction, and the given director node may check for conflicts between the record and only records in the history of approved records that were approved after the snapshot time. The history of approved records may include one or more records that have not been flushed from the multiple worker nodes to persistent storage (e.g., database 110) shared among the multiple worker nodes. A given director node may include the record in its history of approved records (e.g., by creating a history record 225 for the record) in response to sending an approval response to the worker node. In various embodiments, a given director node determines whether to approve a transaction to commit without communicating with other director nodes of the multiple director nodes. A request to a director node may indicate a commit time (e.g., proposed commit time 705), and the given director node may process requests from worker nodes at that commit time.

[0071] In step 1230, the worker node proceeds to perform operations for the transaction in response to receiving an approval response (e.g., approval snapshot response 520, approval commit response 720, etc.) from a majority of the multiple director nodes, where none of the received responses indicates disapproval of the transaction. In some cases, after determining that permission to commit the transaction has been granted, the worker node may commit the transaction and flush the record to persistent storage shared among the multiple worker nodes. In some cases, after determining that permission has been granted, the worker node may abort the transaction. The worker node may abort the transaction based on a determination that an approval response has not been received from a majority of the multiple director nodes. The worker node may abort the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on a conflict between the record and another record known to the at least one director node. The worker node may further abort the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on the arrival of the commit request after the commit time.

[0072] 13, a flow diagram of a method 1300 is shown. Method 1300 is one embodiment of a method performed by a director node (e.g., director node 135) of a database system (e.g., system 100) as part of ensuring transactional consistency within the database system. Method 1300 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1300 includes more or fewer steps than shown. For example, method 1200 may include a step in which a director node processes a snapshot request (e.g., snapshot request 510) received from a worker node (e.g., worker node 120).

[0073] Method 1300 begins at step 1310 with a director node receiving a request (e.g., commit request 710) from a worker node for permission to commit a transaction that includes a record (e.g., record 115). The worker node may be part of a database system that may include multiple worker nodes operable to execute transactions and multiple director nodes operable to ensure transactional integrity of those transactions. The received request may specify a record of the transaction.

[0074] In step 1320, the director node determines whether the record conflicts with a set of records (e.g., identified in history information 220) provided by another one of the plurality of worker nodes (e.g., as part of a commit request 710). The received request may specify a snapshot time (e.g., previously approved snapshot time 505) and a commit time (e.g., proposed commit time 705) associated with the transaction. As such, the determining may include the director node determining whether there is a conflicting record in the set of records identified as having been committed between the snapshot time and the commit time. Additionally, the determining may be performed after a local clock (e.g., local clock 205) of the director node indicates a time equal to the commit time. In some cases, the set of records includes at least one record that is not included in another set of records stored by another one of the plurality of director nodes.

[0075] In step 1330, based on the determination, the director node sends a response (e.g., commit response 720) to the worker node indicating whether permission to commit the transaction has been granted. In some embodiments, the multiple director nodes are not operable to communicate with each other regarding whether permission to commit the transaction has been granted. In some cases, the response indicates that permission to commit the transaction has been granted, and the director node stores historical information (e.g., historical record 225) that identifies the record as committed, regardless of whether the worker node commits the transaction. In some cases, the response indicates that permission to commit the transaction has not been granted, and the worker node aborts the transaction, regardless of receiving approval to commit the transaction from other director nodes of the multiple director nodes. The response may provide the worker node with an indication (e.g., approval of proposed commit time 705) indicating when the worker node is authorized to commit the transaction.

[0076] 14, a flow diagram of a method 1400 is shown. Method 1400 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1400 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1400 includes more or fewer steps than shown. For example, method 1400 may include a step of a worker node sending a request (e.g., commit request 710) to a director node (e.g., director node 135) for permission to commit the transaction.

[0077] Method 1400 begins at step 1410 with a worker node receiving a request to execute a transaction that includes writing a first record (e.g., record 115). In some cases, the worker node receives a request to execute a read-only transaction that does not include writing a record. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions in the database system and multiple director nodes (e.g., director node 135) operable to ensure transactional integrity of the transactions.

[0078] In step 1420, the worker node issues a snapshot request (e.g., snapshot request 510) to some of the director nodes (e.g., a majority subset or all) to request history information (e.g., history information 220) describing the history of the approved record up to a specified time (e.g., proposed snapshot time 505). The specified time may be selected by the worker node, and the director node may delay processing the snapshot request until the specified time. The history information may specify, for a record in the history of the approved record, the position of the record (e.g., position 24) and the time at which the record was marked committed (e.g., commit time).

[0079] In step 1430, the worker node proceeds to execute the transaction in response to receiving a response (e.g., snapshot response 520) from a majority of the plurality of director nodes. The combination of the information in the response constitutes the requested history information. The processing may include determining whether a second record exists that conflicts with the first record based on the requested history information, and aborting the transaction in response to determining that a conflicting second record exists. The history of approved records known to a first one of the director nodes may differ from the history of approved records known to a second one of the director nodes. In some cases, none of the responses individually provides the requested history information. The director nodes may not be operable to communicate with each other to share history information.

[0080] The transaction may include a second record. The worker node may determine a location of the second record based on the requested historical information and may attempt to access the second record from the location. In some cases, the location corresponds to a central storage shared among multiple worker nodes. In some cases, the location corresponds to another worker node. In attempting to access the second record, the worker node may send a request for the second record (e.g., record request 204) to the other worker node and may receive a response (e.g., record response 206) indicating that the second record was not committed.

[0081] 15, a flow diagram of a method 1500 is shown. Method 1500 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1500 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1500 includes more or fewer steps than shown. For example, method 1500 may include a step in which a worker node sends a request (e.g., record request 204) to another worker node to retrieve a record (e.g., record 115).

[0082] Method 1500 begins with a worker node receiving a request to execute a transaction that includes writing a record, at step 1510. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions in the database system, and multiple director nodes (e.g., director node 135) operable to ensure transactional integrity of the transactions.

[0083] In step 1520, the worker node selects a proposed time (e.g., proposed snapshot time 505, proposed commit time 705, etc.) associated with performing an operation for the transaction (e.g., starting the transaction, committing the transaction, etc.). In various embodiments, the worker node determines a time delay in communicating with the director node. A given one of the time delays may be a delay between sending a given request from the worker node to a given one of the director nodes and receiving the given request at the given director node. Thus, the worker node may record a first time that the given request was sent to the given director node (e.g., send timestamp 910) and receive a response to the given request that identifies a second time that the given request was received at the given director node (e.g., arrival timestamp 920). The given time delay may be determined based on the first time and the second time, and the proposed time may be selected based on the time delay such that the authorization request is estimated to arrive at the director node before the proposed time according to the director node's respective clock (e.g., local clock 205). The proposed time may be selected based on a time identified by a worker node's clock. In various cases, the worker node's clock identifies a different time than the director node's clock, which identifies a different time than the clock of at least one other director node of the multiple director nodes.

[0084] In step 1530, the worker node issues an approval request to perform the operation to some of the director nodes (e.g., a majority subset or all). In various embodiments, a director node of these director nodes is operable to process the approval request when the proposed time is reached according to the director node's clock that identifies the time observed by the director node. In some cases, the operation relates to initiating a transaction and the director node is operable to return history information (e.g., history information 220) describing a history of approved records up to the proposed time. In some cases, the operation relates to committing a transaction and the director node is operable to check for conflicts between the records of the transaction and a set of approved records known to the director node that occurred before the proposed time. The worker node may include delay information in the approval request to allow a first one of the director nodes to attempt to align the time observed by the first director node with the time observed by a second one of the director nodes.

[0085] The worker node may receive a disapproval response from the director node indicating disapproval of the proposed time based on the request arriving at the director node after the proposed time according to the director node's clock. The proposed time may be selected to be a first amount of time after a current time identified by the worker node's clock. Then, based on receiving the disapproval response, the worker node may select another proposed time such that the another proposed time is a second amount of time after a current time identified by the worker node's clock, the second amount of time being greater than the first amount of time.

[0086] 16, a flow diagram of a method 1600 is shown. Method 1600 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1600 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1600 includes more or fewer steps than shown. For example, method 1600 may include a step in which a worker node sends a request (e.g., record request 204) to another worker node to retrieve a record (e.g., record 115).

[0087] The method 1600 begins at step 1610 with a worker node accessing delay information specifying a time delay in communicating with some of a plurality of director nodes of a database system operable to ensure transaction integrity of transactions of the database system. At step 1620, the worker node receives a request to execute a transaction.

[0088] In step 1630, the worker node selects a proposed snapshot time (e.g., proposed snapshot time 505) for the transaction based on the delay information and the worker node's clock (e.g., local clock 205). The proposed snapshot time may be selected to be a time determined according to the director node's clock to allow the authorization request to arrive at the director node before the proposed snapshot time.

[0089] In step 1640, the worker node issues a first approval request (e.g., snapshot request 510) for the proposed snapshot time to two or more of the multiple director nodes. In various embodiments, one director node of the two or more director nodes is operable to process the first approval request upon arrival of the proposed snapshot time according to the director node's clock. Prior to issuing the first approval request, the worker node may issue a second approval request to the director node for the proposed time associated with another transaction. The worker node may record a first time that the second approval request was issued to the director node according to the worker node's clock (e.g., send timestamp 910) and receive a response to the second approval request from the director node identifying a second time that the second approval request was received at the director node according to the director node's clock (e.g., arrival timestamp 920). The worker node's clock may identify a different time than the director node's clock, and the proposed snapshot time may be selected based on the first time and the second time.

[0090] 17, a flow diagram of a method 1700 is shown. Method 1700 is one embodiment of a method performed by a director node (e.g., director node 135) of a database system (e.g., system 100) to attempt to align the time observed by the director node with the time observed by other director nodes. Method 1700 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1700 includes more or fewer steps than shown. For example, method 1700 may include a step in which the director node provides a response to a worker node (e.g., worker node 120) that includes the history information (e.g., history information 220).

[0091] The method 1700 begins with a director node maintaining a clock (e.g., local clock 205) that identifies a time observed by the director node, at step 1710. The director node may perform database operations at designated times.

[0092] In step 1720, the director node receives a request (e.g., snapshot request 510) from one of the plurality of worker nodes operable to process a transaction for information (e.g., historical information 220) facilitating execution of a transaction associated with the worker node. The request may provide a first indication of a time observed by other director nodes of the plurality of director nodes. In various embodiments, the request specifies a time delay observed by the worker node in communicating with the plurality of director nodes. The director node may aggregate this time delay with time delays provided by other worker nodes and shift its clock based on the aggregated time delay. A given one of the time delays may be a delay between sending a given request from the worker node to a given director node of the plurality of director nodes and receiving the given request by the given director node. The time delay may be derived based on a clock of the worker node identifying the time observed by the worker node. In some cases, the request specifies a proposed time associated with the transaction (e.g., proposed snapshot time 505). The director node may process the request when the proposed time is reached according to the director node's clock. In various cases, the director node's clock identifies a different time than the clock of another one of the director nodes. Thus, the request may be processed at the director node before the request is processed at the other director node.

[0093] In step 1730, the director, based on the first indication, attempts to shift the clock to align the time observed by the director node with the time observed by the other director nodes. Based on the time delay, the director node may determine that there is a larger time delay when the worker node communicates with the director node than when the worker node communicates with a majority of the other director nodes. Thus, shifting may include slowing down the clock. Based on the time delay, the director node may determine that there is a smaller time delay when the worker node communicates with the director node than when the worker node communicates with a majority of the other director nodes. Thus, shifting includes speeding up the clock. In some cases, the director node may identify the arrival time of the request according to the clock (e.g., arrival timestamp 920) and send a response to the worker node specifying the arrival time. The worker node may be operable to identify a time delay between sending the request and the director node receiving the request. In some cases, shifting the clock causes the director node to skip processing one or more requests received at the director node. Shifting the clock can be performed without directly communicating with other director nodes.

[0094] 18, a flow diagram of a method 1800 is shown. Method 1800 is one embodiment of a method performed by a director node (e.g., director node 135) of a database system (e.g., system 100) to attempt to align times observed by the director node. Method 1800 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1800 includes more or fewer steps than shown. For example, method 1800 may include a step in which the director node provides a response to a worker node (e.g., worker node 120) that includes the history information (e.g., history information 220).

[0095] Method 1800 begins at step 1810 with the director nodes maintaining respective clocks (e.g., local clock 205) that identify respective times observed by the director nodes. A given director node may perform a database operation at a specified time. At step 1820, the director node receives timing information from a worker node of the database system. The timing information may indicate communication time delays between the worker node and the director node.

[0096] In step 1830, the director nodes attempt to shift their respective clocks based on the timing information to align the respective times observed by the director nodes. In various embodiments, the shift is performed without some of the director nodes communicating with some of the other director nodes. The director nodes may receive requests from the worker nodes for permission to commit a transaction (e.g., commit request 710), and the request may specify a proposed commit time (e.g., proposed commit time 705). A given director node may process the request if the given director node's respective clock specifies a time that is not earlier than the proposed commit time. A given director node may send responses (e.g., commit response 720) to the worker nodes indicating the arrival time of the request at the given director node (e.g., arrival timestamp 920) according to the given director node's respective clock. A worker node may determine a time delay for the worker node to communicate with a given director node based on the arrival time. In various cases, a request received from a worker node is processed by a given director node independent of the processing of the request by another one of the director nodes.

[0097] Referring now to FIG. 19, a flow diagram of a method 1900 is shown. The method 1900 is one embodiment of a method performed by an assistant node (e.g., assistant node 1030) of a database system (e.g., system 100) that observes work performed by a worker node (e.g., worker node 120). The method 1900 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, the method 1900 includes more or fewer steps than shown. For example, the method 1900 may include a step in which the assistant node flushes a record (e.g., record 115) to a database (e.g., database 110) that was also flushed by the worker node.

[0098] The method 1900 begins at step 1910 with the worker node observing work performed by a first worker node of a database system including a plurality of worker nodes operable to execute transactions for the database system. The observing may include storing, at an assistant node, a record associated with the first worker node. The assistant node may read a transaction log (e.g., log 1045) of the first worker node stored in a data store (e.g., shared storage 1040) shared between the first worker node and the assistant node. The record stored at the assistant node may be determined from the transaction log. In some cases, the assistant node is one of a plurality of worker nodes operable to execute transactions for the database system. In various embodiments, the assistant node and the first worker node are executed using different virtual machines.

[0099] In step 1920, the assistant node receives a request for a record associated with the first worker node from a second worker node of the plurality of worker nodes (e.g., record request 204). The request may specify a snapshot time corresponding to a transaction of the second worker node. Before making a determination as to whether the first worker node committed the record, the assistant node may ensure that a record of the transaction log having a timestamp indicating a time prior to the snapshot time has been read by the assistant node. In various embodiments, the second worker node is operable to send a request for the record to the assistant node in response to not receiving a response from the first worker node to the request sent to the first worker node for the record.

[0100] In step 1930, the assistant node makes a determination as to whether the first worker node committed the record. Making the determination may include determining that the record is not committed based on the transaction log and that the first worker node does not respond. Based on the determination, in various embodiments, the assistant node fences the transaction log to prevent the first worker node from modifying the transaction log to commit the record, and then sends a response (e.g., record response 206) to the second worker node indicating that the record is not committed. Making the determination may include determining that the first worker node committed a transaction including the record based on the transaction log. Thus, the assistant node may access the record from a record stored in the assistant node, and the response to the second worker node may include the accessed record. Making the determination may include determining that the record is not committed based on the transaction log indicating that the first worker node logged work after a commit time associated with the transaction including the record and the transaction log not indicating the record. Therefore, the response to the second worker node may indicate that the record has not been committed.

[0101] In step 1940, the assistant node sends a response to the second worker node indicating whether the record was committed by the first worker node. Based on a determination that the record was committed, the assistant node may flush the record to a log-structured merge tree (LSM tree) of the database system (e.g., LSM1100). The first worker node may also flush the record to the LSM tree such that there are two instances of the record in the LSM tree over a period of time.

[0102] Referring now to FIG. 20, a flow diagram of a method 2000 is shown. The method 2000 is one embodiment of a method performed by an assistant node (e.g., assistant node 1030) of a database system (e.g., system 100) that observes work performed by a worker node (e.g., worker node 120). The method 2000 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, the method 2000 includes more or fewer steps than shown. For example, the method 2000 may include a step in which the assistant node flushes a record (e.g., record 115) to a database (e.g., database 110) that was also flushed by the worker node.

[0103] Method 2000 begins at step 2010 with the assistant reading a transaction log (e.g., log 1045) of a first worker node. In various embodiments, the transaction log specifies a set of records committed by the first worker node. Based on reading the transaction log, the assistant node may cache one or more of the set of records at the assistant node. At step 2020, the assistant node receives a request (e.g., record request 204) for a record associated with the first worker node from a second worker node.

[0104] In step 2030, based on reading the transaction log, the assistant node determines whether the record has been committed. The request received from the second worker node may identify a snapshot timestamp. The assistant node may determine that the first worker node has not updated the transaction log, and that the first worker node has not written a record to the transaction log having a timestamp indicating a time later than the time of the snapshot timestamp. Based on this determination, the assistant node may prevent the first worker node from updating the transaction log. A response to the second worker node may indicate that the record has not been committed. In response to determining that the record has been committed, the assistant node may access the record from one or more cached records. The response may include the record.

[0105] In step 2040, the assistant node sends a response to the second worker node indicating whether the record has been committed. In response to determining that the record has been committed, the assistant node may flush the record to a database of the database system (e.g., database 110) regardless of whether the first worker node flushes the record to the database. Exemplary Multitenant Database System

[0106] Referring now to FIG. 21, an exemplary multi-tenant database system (MTS) 2100 is shown in which various techniques of the present disclosure can be implemented, for example, the system 100 can be the MTS 2100. In FIG. 21, the MTS 2100 includes a database platform 2110, an application platform 2120, and a network interface 2130 connected to a network 2140. Also as shown, the database platform 2110 includes a data storage 2112 and a set of database servers 2114A-N that interact with the data storage 2112, and the application platform 2120 includes a set of application servers 2122A-N having respective environments 2124. In the illustrated embodiment, the MTS 2100 is connected to various user systems 2150A-N via the network 2140. The disclosed multi-tenant system is included for illustrative purposes and is not intended to limit the scope of the present disclosure. In other embodiments, the techniques of the present disclosure are implemented in a non-multi-tenant environment, such as a client / server environment, a cloud computing environment, a clustered computer, etc.

[0107] The MTS 2100, in various embodiments, is a set of computer systems that together provide various services to users (alternatively referred to as "tenants") that interact with the MTS 2100. In some embodiments, the MTS 2100 implements a customer relationship management (CRM) system that provides a mechanism for tenants (e.g., businesses, government agencies, etc.) to manage relationships and interactions with customers and potential customers. For example, the MTS 2100 may enable tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), identify sales opportunities, record service issues, and manage marketing campaigns. Additionally, the MTS 2100 may enable these tenants to identify how they communicated with customers, what customers purchased, when they last purchased an item, and how much they paid. To provide the services of the CRM system and / or other services, the MTS 2100 includes a database platform 2110 and an application platform 2120, as shown.

[0108] The database platform 2110, in various embodiments, is a combination of software routines and hardware elements that implement database services for storing and managing data for the MTS 2100, including tenant data. As shown, the database platform 2110 includes data storage 2112, which in various embodiments includes a set of storage devices (e.g., solid state drives, hard disk drives, etc.) connected to each other over a network (e.g., a storage attached network (SAN)) and configured to store data redundantly to prevent data loss. In various embodiments, the data storage 2112 is used to implement a database (e.g., database 110) that includes a collection of information organized in a manner that allows the information to be accessed, stored, and manipulated. The data storage 2112 may implement a single database, a distributed database, a collection of distributed databases, a database with redundant online or offline backups or other redundancies, etc. As part of implementing a database, data storage 2112 may store files containing one or more database records with their respective data payloads (e.g., values ​​of fields in a database table) and metadata (e.g., key values, timestamps, table identifiers for tables associated with the record, tenant identifiers for tenants associated with the record, etc.).

[0109] In various embodiments, a database record may correspond to a row in a table. A table generally includes one or more data categories logically arranged as columns or fields in a viewable schema. Thus, each record in a table may include an instance of data in each category defined by a field. For example, a database may include a table that describes customers with fields of basic contact information such as name, address, phone number, fax number, etc. Thus, a record in that table may include a value for each of the fields in the table (e.g., name for a name field). Another table may describe a purchase order that includes fields of information such as customer, product, sale price, date, etc. In various embodiments, standard entity tables, such as tables for account, contact, lead, and opportunity data, are provided for use by all tenants, each including predefined fields. The MTS 2100 may store database records for one or more tenants in the same table; i.e., tenants may share tables. Thus, in various embodiments, a database record includes a tenant identifier that indicates the owner of the database record. As a result, one tenant's data is kept secure and isolated from other tenants' data, so one tenant cannot access another tenant's data unless the data is explicitly shared.

[0110] In some embodiments, data stored in data storage 2112 is organized as part of a log-structured merge tree (LSM tree, e.g., LSM tree 1100). An LSM tree typically includes two high-level components: an in-memory buffer and a persistent storage. During operation, database server 2114 may first write database records to a local in-memory buffer and then flush these records to persistent storage (e.g., data storage 2112). As part of flushing the database records, database server 2114 may write the database records to a new file contained in the “top” level of the LSM tree. Over time, the database records may be rewritten by database server 2114 to new files contained in lower levels as the database records move down the levels of the LSM tree. In various implementations, as database records age and are moved down the LSM tree, they are moved to slower and slower storage devices in data storage 2112 (e.g., from solid state drives to hard disk drives).

[0111] When the database server 2114 wants to access a database record for a particular key, it may traverse different levels of the LSM tree for files that potentially contain a database record for that particular key. If the database server 2114 determines that a file may contain a related database record, it may fetch the file from the data storage 2112 into the memory of the database server 2114. The database server 2114 may then check the fetched file for a database record with the particular key. In various embodiments, a database record is immutable once it is written to the data storage 2112. Thus, if the database server 2114 wants to modify a value in a row of a table (which may be identified from an accessed database record), the database server 2114 writes out a new database record to the top level of the LSM tree. Over time, that database record is merged down the levels of the LSM tree. Thus, an LSM tree may store various database records for a database key, with older database records for that key being located at a lower level in the LSM tree than newer database records.

[0112] The database server 2114, in various embodiments, is a hardware element, software routine, or combination thereof capable of providing database services such as data storage, data retrieval, and / or data manipulation. The database server 2114 may correspond to a worker node 120. Such database services may be provided by the database server 2114 to components within the MTS 2100 (e.g., application server 2122) and components external to the MTS 2100. As an example, the database server 2114 may receive a database transaction request from the application server 2122 requesting to write or read data to the data storage 2112. The database transaction request may specify an SQL SELECT command to select one or more rows from one or more database tables. The contents of the row may be defined in a database record, and thus the database server 2114 may locate and return one or more database records corresponding to the selected table row or rows. In various cases, a database transaction request may instruct database server 2114 to write one or more database records for an LSM tree, which database server 2114 maintains implemented on database platform 2110. In some embodiments, database server 2114 implements a relational database management system (RDMS) or an object-oriented database management system (OODBMS) that facilitates storage and retrieval of information from data storage 2112. In various cases, database servers 2114 may communicate with each other to facilitate processing of transactions. For example, database server 2114A communicates with database server 2114N to determine whether database server 2114N has written a database record for a particular key to its in-memory buffer.

[0113] The application platform 2120, in various embodiments, is a combination of software routines and hardware elements that implements and executes the CRM software application, provides associated data, code, forms, web pages, and other information to and from the user systems 2150, and stores associated data, objects, web page content, and other tenant information via the database platform 2110. To facilitate these services, in various embodiments, the application platform 2120 communicates with the database platform 2110 to store, access, and manipulate data. In some cases, the application platform 2120 may communicate with the database platform 2110 via different network connections. For example, one application server 2122 may be coupled via a local area network and another application server 2122 may be coupled via a direct network link. TCP / IP (Transfer Control Protocol and Internet Protocol) is an exemplary protocol for communicating between the application platform 2120 and the database platform 2110, although it will be apparent to one skilled in the art that other transport protocols may be used depending on the network interconnect used.

[0114] The application server 2122 is a hardware element, software routine, or combination thereof capable of providing the services of the application platform 2120, including, in various embodiments, processing requests received from tenants of the MTS 2100. The application server 2122 may, in various embodiments, spawn environments 2124 that can be used for various purposes, such as providing functionality for developers to develop, execute, and manage applications (e.g., business logic). Data may be transferred to an environment 2124 from another environment 2124 and / or the database platform 2110. In some cases, an environment 2124 may not have access to such data from other environments 2124 unless such data is explicitly shared. In some embodiments, multiple environments 2124 may be associated with a single tenant.

[0115] The application platform 2120 may provide access to multiple different hosted (standard and / or custom) applications to the user systems 2150, including a CRM application and / or applications developed by the tenant. In various embodiments, the application platform 2120 may manage the creation of applications, the testing of applications, the storage of applications in database objects in the data storage 2112, the execution of applications in the environment 2124 (e.g., virtual machines in the process space), or any combination thereof. In some embodiments, the application platform 2120 may add and remove application servers 2122 from the server pool at any time for any reason, and there may be no server affinity of a user and / or organization to a particular application server 2122. In some embodiments, an interface system (not shown) that implements load balancing functionality (e.g., an F5 Big-IP load balancer) is located between the application servers 2122 and the user systems 2150 and is configured to allocate requests to the application servers 2122. In some embodiments, the load balancer routes user requests to the application servers 2122 using a least connections algorithm. Other examples of load balancing algorithms such as round robin and observed response time can also be used. For example, in a particular embodiment, three consecutive requests from the same user may hit three different servers 2122, and three requests from different users may hit the same server 2122.

[0116] In some embodiments, the MTS 2100 provides security mechanisms, such as encryption, to keep each tenant's data separate unless the data is shared. If more than one server 2114 or 2122 is used, they may be located in close proximity to one another (e.g., in a server farm located in a single building or campus) or distributed at locations far from one another (e.g., one or more servers 2114 located in city A and one or more servers 2122 located in city B). Thus, the MTS 2100 may include one or more logically and / or physically connected servers, either locally or distributed across one or more geographic locations.

[0117] One or more users (e.g., via user systems 2150) may interact with the MTS 2100 over the network 2140. The user systems 2150 may correspond to, for example, a tenant of the MTS 2100, a provider (e.g., an administrator) of the MTS 2100, or a third party. Each user system 2150 may be a desktop personal computer, a workstation, a laptop, a PDA, a mobile phone, or any wireless access protocol (WAP)-enabled device, or any other computing device capable of interfacing directly or indirectly to the Internet or other network connection. The user systems 2150 may include dedicated hardware configured to interface with the MTS 2100 over the network 2140. The user systems 2150 may execute a graphical user interface (GUI) corresponding to the MTS 2100, an HTTP client (e.g., a browsing program such as Microsoft's Internet Explorer browser, Netscape's Navigator browser, Opera browser, or a WAP-enabled browser in the case of a mobile phone, PDA, or other wireless device), or both, to enable users of the user systems 2150 (e.g., subscribers to the CRM system) to access, process, and view information and pages available from the MTS 2100 over the network 2140. Each user system 2150 may include one or more user interface devices, such as a keyboard, mouse, touch screen, pen, etc., for interacting with a graphical user interface (GUI) provided by a browser on a display monitor screen, LCD display, etc., along with pages, forms, and other information provided by the MTS 2100 or other systems or servers. As discussed above, the disclosed embodiments are suitable for use with the Internet, which refers to a particular global internetwork of networks.However, it should be understood that other networks may be used in place of the Internet, such as an intranet, an extranet, a virtual private network (VPN), a non-TCP / IP based network, any LAN or WAN, etc.

[0118] Because users of the user system 2150 may be users of different capabilities, the capabilities of a particular user system 2150 may be determined by one or more permission levels associated with the current user. For example, if a salesperson is using a particular user system 2150 to interact with the MTS 2100, the user system 2150 may have the capabilities (e.g., user privileges) assigned to the salesperson. However, if an administrator is using the same user system 2150 to interact with the MTS 2100, the user system 2150 may have the capabilities (e.g., administrative privileges) assigned to the administrator. In a system with a hierarchical role model, a user with a certain permission level may access applications, data, and database information accessible to users with a lower permission level, but may not access certain applications, database information, and data accessible to users with a higher permission level. Thus, different users may have different capabilities with respect to accessing and modifying application and database information depending on the user's security or permission level. Some data structures managed by the MTS 2100 are allocated at the tenant level, while others are managed at the user level.

[0119] In some embodiments, the user system 2150 and its components are configurable using an application, such as a browser, that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 2100 (and additional instances of the MTS, if there is more than one) and their components are configurable by an operator using application(s) that include computer code executable on the processing elements. Thus, various operations described herein may be performed by executing program instructions that are stored on a non-transitory computer-readable medium and executed by the processing elements. The program instructions may be stored on a non-volatile medium, such as a hard disk, or other volatile or non-volatile memory medium or device, such as a ROM or RAM, as is well known, or provided on any medium capable of initiating program code, such as a compact disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, or the like. Additionally, the entire program code or portions thereof may be transmitted and downloaded from a software source, for example, via the Internet or from another server, as is well known, or may be transmitted over any other conventional network connection, as is well known, (e.g., extranet, VPN, LAN, etc.), using any communication medium and protocol, as is well known, (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.) as is well known. It will also be appreciated that computer code for implementing aspects of the disclosed embodiments may be implemented in any programming language that can be executed on a server or server system, for example, C, C++, HTML, Java, JavaScript, or any other scripting language, such as VBScript.

[0120] Network 2140 may be a local area network (LAN), a wide area network (WAN), a wireless network, a point-to-point network, a star network, a token ring network, a hubbed network, or any other suitable configuration. The global internetwork of networks is often referred to as the "Internet" with a capital "I" and is an example of a Transfer Control Protocol and Internet Protocol (TCP / IP) network. However, it should be understood that the disclosed embodiments may utilize any of a variety of other types of networks.

[0121] The user system 2150 may communicate with the MTS 2100 using TCP / IP, and at higher network levels, using other common Internet protocols such as HTTP, FTP, AFS, WAP, etc. For example, if HTTP is used, the user system 2150 may include an HTTP client, commonly referred to as a "browser," for sending and receiving HTTP messages from an HTTP server of the MTS 2100. Such a server may be implemented as the only network interface between the MTS 2100 and the network 2140, although other techniques may be used as well or instead. In some implementations, the interface between the MTS 2100 and the network 2140 includes a load-sharing function, such as a round-robin HTTP request distributor, to balance the load and distribute incoming HTTP requests evenly across multiple servers.

[0122] In various embodiments, the user system 2150 communicates with the application server 2122 to request and update system-level and tenant-level data from the MTS 2100, which may require one or more queries to the data storage 2112. In some embodiments, the MTS 2100 automatically generates one or more SQL statements (SQL queries) designed to access the desired information. In some cases, the user system 2150 may generate requests having a specific format that corresponds to at least a portion of the MTS 2100. As an example, the user system 2150 may request that a data object be moved to a particular environment using an object notation that describes the object-relational mapping (e.g., JavaScript object notation mapping) of a number of specified objects. Exemplary Computer System

[0123] 22, there is shown a block diagram of an exemplary computer system 2200 on which the system 100, database 110, worker node 120, director node 135, assistant node 1030, MTS 2100, and / or user system 2150 may be implemented. The computer system 2200 includes a processor subsystem 2280 coupled to a system memory 2220 and I / O interface(s) 2240 via an interconnect 2260 (e.g., a system bus). The I / O interface(s) 2240 are coupled to one or more I / O devices 2250. For convenience, a single computer system 2200 is shown in FIG. 22, however the system 2200 may be implemented as two or more computer systems operating together.

[0124] Processor subsystem 2280 may include one or more processors or processing units. In various embodiments of computer system 2200, multiple instances of processor subsystem 2280 may be coupled to interconnect 2260. In various embodiments, processor subsystem 2280 (or each processor unit within 2280) may include a cache or other form of on-board memory.

[0125] The system memory 2220 can be used to store program instructions executable by the processor subsystem 2280 to cause the system 2200 to perform various operations described herein. The system memory 2220 can be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.). The memory in the computer system 2200 is not limited to primary storage, such as the memory 2220. Rather, the computer system 2200 can also include other forms of storage, such as cache memory in the processor subsystem 2280 and secondary storage on the I / O device 2250 (e.g., hard drive, storage array, etc.). In some embodiments, these other forms of storage can also store program instructions executable by the processor subsystem 2280. In some embodiments, program instructions that, when executed, implement database application 200, in-memory cache 210, and / or orchestration application 300 may be included / stored within system memory 2220.

[0126] I / O interface 2240 may be any of a variety of types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I / O interface 2240 is a bridge chip (e.g., a southbridge) from a front side to one or more back side buses. I / O interface 2240 may be coupled to one or more I / O devices 2250 via one or more corresponding buses or other interfaces. Examples of I / O devices 2250 include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local or wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system 2200 is coupled to a network via network interface device 2250 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0127] Implementations of the subject matter of this application include, but are not limited to, the following Examples 1-20. [Example 1] 1. A method comprising: receiving a request by a worker node of a database system to execute a transaction including writing a record, the database system including a plurality of worker nodes operable to execute transactions for the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; a worker node issuing a request to a number of director nodes of a plurality of director nodes for information facilitating execution of an operation for the transaction; in response to receiving approval responses from a majority of the plurality of director nodes, the worker node proceeds to perform operations for the transaction, where none of the received responses indicates disapproval of the transaction; The method includes: [Example 2] The method of example 1, wherein the request for information specifies a record and the operation corresponds to committing a transaction, and a given one of the director nodes is operable to determine whether to approve the commit of the transaction based on whether the record conflicts with a history of approved records known to the given director node. [Example 3] The method of example 2, wherein the request to the director node specifies a snapshot time of the transaction, and the given director node is operable to check for conflicts between the record and only records in the history of approved records that were approved after the snapshot time. [Example 4] The method of example 2, wherein the history of acknowledged records includes one or more records that have not been flushed from the multiple worker nodes to persistent storage shared among the multiple worker nodes. [Example 5] 3. The method of example 2, wherein the given director node is operable to include the record in a history of approved records in response to sending an approval response to the worker node. [Example 6] 3. The method of example 2, wherein a given director node is operable to determine whether to approve a commit of a transaction without communicating with other director nodes of the plurality of director nodes. [Example 7] 3. The method of example 2, wherein a request to a director node indicates a commit time, and a given director node is operable to process requests from worker nodes at the commit time. [Example 8] after determining that permission to commit the transaction has been granted, the worker node commits the transaction and flushes the records to persistent storage shared among the multiple worker nodes. The method of example 2, further comprising: [Example 9] A non-transitory computer readable medium having stored thereon program instructions capable of causing a worker node of a database system to perform an operation, the operation comprising: receiving a request to execute a transaction including writing a record, the database system including a plurality of worker nodes operable to execute the transaction for the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; writing the records to an in-memory cache on a worker node; issuing a commit request to some of the director nodes requesting permission to commit the record; determining that permission to commit the record has been granted based on receiving an approval response from a majority of the plurality of director nodes without receiving a response indicating disapproval of the transaction; A non-transitory computer readable medium comprising: [Example 10] The operation is Aborting the transaction based on a determination that an approval response has not been received from a majority of the director nodes. The medium of example 9, further comprising: [Example 11] The operation is Aborting the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on a conflict between the record and another record known to the at least one director node. The medium of example 9, further comprising: [Example 12] The commit request specifies a proposed time period for committing the transaction, and the action is Aborting the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on arrival of the commit request after the proposed period of time. The medium of example 9, further comprising: [Example 13] The operation is receiving a request for the records from another worker node; After committing the transaction, provide the record to another worker node. The medium of example 9, further comprising: [Example 14] 1. A method comprising: a director node receiving a request from a worker node for permission to commit a transaction including a record, the director node and the worker node being part of a database system having a plurality of worker nodes operable to execute transactions and a plurality of director nodes operable to ensure transactional integrity of the transactions; a director node determining whether the record conflicts with a set of records identified by other worker nodes of a plurality of worker nodes; Based on the determination, the director node sends a response to the worker node indicating whether permission to commit the transaction has been granted; The method includes: [Example 15] The request specifies a record, the response indicates that permission to commit the transaction has been granted, and the method is the director node storing historical information identifying the records as having been committed regardless of whether the worker nodes committed the transactions. The method of example 14, further comprising: [Example 16] The request specifies a snapshot time and a proposed commit time associated with the transaction, and the determining step includes: determining whether there are any conflicting records within the set of records identified as having been committed between the snapshot time and the proposed commit time; The method of example 14, comprising: [Example 17] 17. The method of example 16, wherein the determining step is performed after the director node's local clock indicates a time equal to the commit time. [Example 18] The method of example 15, wherein the response indicates that permission to commit the transaction has not been granted, and the worker node is operable to commit the transaction in response to receiving approval from a majority of the multiple director nodes despite the response indicating that permission to commit has not been granted. [Example 19] 16. The method of example 15, wherein the set of records known to the director node includes at least one record that is not included in another set of records known to another director node of the plurality of director nodes. [Example 20] 16. The method of example 15, wherein the response provides an indication to the worker node of when the worker node is permitted to commit the transaction.

[0128] The present disclosure includes references to "embodiments" that are non-limiting implementations of the disclosed concepts. References to "embodiments," "one embodiment," "particular embodiments," "some embodiments," "various embodiments," etc. do not necessarily refer to the same embodiment. In addition to the specific embodiments described in detail, numerous possible embodiments are contemplated, including modifications or alternatives that fall within the spirit or scope of the present disclosure. Not all embodiments necessarily exhibit any or all of the potential advantages described herein.

[0129] This disclosure may discuss potential advantages that may result from the disclosed embodiments. Not all implementations of these embodiments necessarily manifest any or all of the potential advantages. Whether advantages are realized for a particular implementation depends on many factors, some of which are outside the scope of this disclosure. Indeed, there are several reasons why an embodiment within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuit configurations outside the scope of this disclosure that, in conjunction with one of the disclosed embodiments, negate or reduce one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular implementation (e.g., implementation technique or tool) may also negate or reduce the disclosed advantages. Even assuming a skilled implementation, the realization of advantages may still depend on other factors, such as the environmental conditions in which the implementation is deployed. For example, the inputs provided to a particular implementation may prevent one or more problems addressed in this disclosure from occurring in a particular case, such that the benefits of the solution may not be realized. Given the possible existence of factors external to this disclosure, it is expressly intended that any potential advantages described herein should not be construed as limitations on the scope of the claims that must be met in order to demonstrate infringement. Rather, the identification of such potential advantages is intended to illustrate the type(s) of improvement available to a designer having the benefit of this disclosure. The permissive description of such advantages (e.g., stating that a particular advantage "may result") is not intended to convey doubt as to whether such advantages are actually realizable, but rather to acknowledge the technological reality that realization of such advantages often depends on additional factors.

[0130] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims that are drafted based on this disclosure, even if only a single example of a particular feature is described. The disclosed embodiments are intended to be illustrative, not limiting, unless otherwise stated in the disclosure. Thus, the application is intended to permit claims that encompass the disclosed embodiments, as well as alternatives, modifications, and equivalents that will be apparent to those skilled in the art having the benefit of this disclosure.

[0131] For example, features in the present application may be combined in any suitable manner. Accordingly, during prosecution of this application (or an application claiming priority thereto), new claims may be prepared for such combinations of features. In particular, with reference to the appended claims, features from a dependent claim may be combined, as appropriate, with features of other dependent claims, including claims that are dependent on other independent claims. Similarly, features from each independent claim may be combined, as appropriate, with features of other dependent claims.

[0132] Thus, the accompanying dependent claims may be drafted so that each is dependent on a single other claim, but additional dependencies are also contemplated. Any combination of features in the dependent claims consistent with this disclosure is contemplated and may be claimed in this or another application. In short, combinations are not limited to those specifically recited in the accompanying claims.

[0133] It is also contemplated that, where appropriate, a claim drafted in one form or statutory type (e.g., apparatus) is intended to support a corresponding claim in another form or statutory type (e.g., method).

[0134] Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. It is hereby announced that the definitions provided in the following paragraphs, and throughout this disclosure, should be used in determining how to interpret the claims drafted based on this disclosure.

[0135] Reference to a singular item (i.e., a noun or noun phrase preceded by "a," "an," or "the") is intended to mean "one or more" unless the context clearly dictates otherwise. Thus, a reference to "an item" in a claim does not exclude additional instances of the item unless the context requires. A "plurality" of an item refers to a set of two or more of the items.

[0136] The word "may" is used herein in a permissive sense (i.e., having the potential to, being able to) and not in a mandatory sense (i.e., must).

[0137] The terms "comprising" and "including" and their forms are open-ended and mean "including, but not limited to."

[0138] When the term "or" is used in this disclosure in reference to a list of alternatives, it will generally be understood to be used in an inclusive sense unless the context dictates otherwise. Thus, a statement such as "x or y" is equivalent to "x or y, or both," and thus encompasses 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, a statement such as "either x or y, but not both" makes it clear that "or" is used in an exclusive sense.

[0139] The phrases "w, x, y, or z, or any combination thereof" or "at least one of ... w, x, y, and z" are intended to encompass all possibilities including a single element, up to the total number of elements in the set. For example, given the set [w,x,y,z], these phrases encompass any single element of the set (e.g., w but not x, y, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Thus, the phrase "at least one of ... w, x, y, and z" refers to at least one element of the set [w,x,y,z], thereby encompassing all possible combinations in this list of elements. This phrase should not be construed as requiring that there be at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0140] In this disclosure, various "labels" may precede nouns or noun phrases. Unless the context dictates otherwise, different labels used for a feature (e.g., "first circuit," "second circuit," "particular circuit," "given circuit," etc.) refer to different instances of the feature. Additionally, the labels "first," "second," and "third," when applied to features, do not imply any kind of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise stated.

[0141] The phrase "based on" is used to describe one or more factors that influence the decision. This term does not exclude the possibility that additional factors may influence the decision. That is, the decision may be based only on the specified factors or on the specified factors and other unspecified factors. Consider the phrase "determining A based on B." This phrase specifies that B is a factor that is used to determine A or that influences the decision of A. This phrase does not exclude that the decision of A may also be based on some other factor, such as C. This phrase is also intended to encompass embodiments in which A is determined only based on B. As used herein, the phrase "based on" is synonymous with the phrase "based at least in part on."

[0142] The phrases "in response to" and "responsive to" describe one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may affect or otherwise trigger the effect in conjunction with or independently of the specified factors. That is, the effect may be responsive only to these factors, or to the specified factors and other unspecified factors. Consider the phrase "performing A in response to B." This phrase specifies that B is the factor that triggers the execution of A or triggers a particular outcome of A. This phrase does not exclude that performing A may also be performed in response to some other factor, such as C. This phrase also does not exclude that performing A may be performed in response to B and C jointly. This phrase is also intended to encompass embodiments in which A is performed in response to B only. As used herein, the phrase "responsive to" is synonymous with the phrase "responsive at least in part to." Similarly, the phrase "in response to" is synonymous with the expression "at least in part in response to."

[0143] Within this disclosure, different entities (which may be variously referred to as "units," "circuits," other components, etc.) may be described or claimed as being "configured" to perform one or more tasks or operations. This formulation - an "entity configured to perform one or more tasks" - is used herein to refer to a structure (i.e., something physical). More specifically, this formulation is used to indicate that this structure is configured to perform one or more tasks during operation. A structure can be said to be "configured" to perform some task even if the structure is not currently operating. Thus, an entity described or listed as being "configured to perform" some task refers to something physical, such as a device, a circuit, a system having a memory and a processor unit that stores program instructions executable to perform the task. This expression is not used herein to refer to something intangible.

[0144] In some cases, various units / circuits / components may be described herein as performing a set of tasks or operations, and it will be understood that those entities are "configured to" perform those tasks / operations, even if not specifically stated otherwise.

[0145] The term "configured to" is not intended to mean "configurable to." For example, an unprogrammed FPGA is not considered to be "configured" to perform a particular function. However, this unprogrammed FPGA may be "configurable" to perform that function. After appropriate programming, the FPGA can be said to be "configured" to perform a particular function.

[0146] For purposes of U.S. patent applications based on this disclosure, a statement in a claim that a structure is "configured to" perform one or more tasks is an express intention not to invoke 35 U.S.C. 112(f) as to that claim element. If an applicant wishes to invoke 35 U.S.C. 112(f) during prosecution of a U.S. patent application based on this disclosure, the claim element will be recited using the construction "means for" [performing a function].

[0147] In this disclosure, different "circuits" may be described. These circuits or "circuitry" constitute hardware that includes various types of circuit elements, such as combinational logic, clocked storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memories (e.g., random access memories, embedded dynamic random access memories), programmable logic arrays, etc. Circuitry may be custom designed or obtained from standard libraries. In various implementations, circuitry may include digital components, analog components, or a combination of both, as appropriate. Certain types of circuits may be generally referred to as "units" (e.g., decoding units, arithmetic logic units (ALUs), functional units, memory management units (MMUs), etc.). Such units may also be referred to as circuits or circuitry.

[0148] Thus, the disclosed circuits / units / components and other elements shown in the drawings and described herein include hardware elements such as those described in the previous paragraph. Often, the internal arrangement of hardware elements within a particular circuit may be specified by describing the function of that circuit. For example, a particular "decode unit" may be described as performing the function of "processing the opcode of an instruction and routing the instruction to one or more of a plurality of functional units," meaning that the decode unit is "configured to" perform this function. The specification of this function is sufficient to connote to one of ordinary skill in the computer arts a set of possible configurations of the circuit.

[0149] In various embodiments, as described in the previous paragraph, circuits, units, and other elements may be defined by the functions or operations they are configured to implement. The arrangement of such circuits / units / components relative to one another and the manner in which they interact form a microarchitecture definition of the hardware that is ultimately fabricated into an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitecture definition. Thus, a microarchitecture definition will be recognized by one of ordinary skill in the art as a structure from which many physical implementations may be derived, all of which are contained within the broader structure described by the microarchitecture definition. That is, one of ordinary skill in the art presented with a microarchitecture definition provided in accordance with this disclosure may, without undue experimentation, apply routine techniques to implement the structure by coding the circuit / unit / component description in a hardware description language (HDL) such as Verilog or VHDL. HDL descriptions are often expressed in a manner that appears to be functional. However, to one of ordinary skill in the art, this HDL description is the method used to translate the structure of the circuit, unit, or component into the next level of implementation details. Such HDL descriptions may take the form of behavioral code (which is typically not synthesizable), register transfer language (RTL) code (which is typically synthesizable, as opposed to behavioral code), or structural code (e.g., a netlist that specifies logic gates and their connectivity). The HDL description may then be synthesized against a library of cells designed for a given integrated circuit manufacturing technology and modified for timing, power, and other reasons to obtain a final design database that is sent to a foundry to generate masks and ultimately manufacture the integrated circuit. Some hardware circuits, or portions thereof, may also be custom designed in a schematic editor and incorporated into the integrated circuit design along with the synthetic circuit configuration. An integrated circuit may include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.) and interconnections between the transistors and the circuit elements.Some embodiments may implement multiple integrated circuits coupled together to implement a hardware circuit, and / or in some embodiments, discrete elements may be used. Alternatively, the HDL design may be synthesized into a programmable logic array, such as a field programmable gate array (FPGA), and implemented in an FPGA. This separation between the design of a group of circuits and the subsequent low-level implementation of those circuits, as this process is performed at different stages of the circuit implementation process, typically results in a scenario where a circuit or logic designer never specifies a particular set of structures for the low-level implementation beyond describing what the circuit is configured to do.

[0150] The fact that many different low-level combinations of circuit elements may be used to achieve the same circuit specifications results in the existence of many equivalent structures for that circuit. As noted above, these low-level circuit implementations may vary with changes in manufacturing technology, the foundry selected to manufacture the integrated circuits, the libraries of cells provided for a particular project, etc. In many cases, the choices made by different design tools or methods to generate these different implementations may be arbitrary.

[0151] Furthermore, it is common for a single implementation of a particular functional specification of a circuit to include a large number of devices (e.g., millions of transistors) for a given embodiment. Thus, the volume of this information is so great that it would not be impractical to provide a complete enumeration of the low-level structures used to implement a single embodiment, let alone the vast array of equivalent possible implementations. For this reason, this disclosure describes the structure of the circuit using functional shorthand notations commonly adopted in the industry.

Claims

1. receiving a request to execute a transaction including writing a record at a worker node of a database system, the database system including a plurality of worker nodes operable to execute transactions for the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; the worker node issuing a request to a number of director nodes of the plurality of director nodes for information facilitating performance of an action for the transaction; in response to receiving approval responses from a majority of the plurality of director nodes, the worker node proceeds to perform the operation for the transaction, wherein none of the received responses indicates disapproval of the transaction; The method includes:

2. 2. The method of claim 1 , wherein the request for information specifies the record and the action corresponds to committing the transaction, and wherein a given one of the director nodes is operable to determine whether to approve the commit of the transaction based on whether the record conflicts with a history of approved records known to the given director node.

3. 3. The method of claim 2, wherein the request to the director node specifies a snapshot time of the transaction, and the given director node is operable to check for conflicts between the record and only records in the history of the approved record that were approved after the snapshot time.

4. The method of claim 2 or 3, wherein the history of approved records includes one or more records that have not been flushed from the plurality of worker nodes to persistent storage shared among the plurality of worker nodes.

5. The method of claim 2 , wherein the given director node is operable to include the record in the history of approved records in response to sending an approval response to the worker node.

6. 6. The method of claim 2, wherein the given director node is operable to determine whether to approve the commit of the transaction without communicating with other director nodes of the plurality of director nodes.

7. 7. The method of claim 2, wherein the request to the director node indicates a commit time, and the given director node is operable to process the request from the worker node at the commit time.

8. after determining that permission to commit the transaction has been granted, the worker node commits the transaction and flushes the record to persistent storage shared among the plurality of worker nodes. The method of claim 2 , further comprising:

9. 1. A computer readable medium having stored thereon program instructions for causing a worker node of a database system to perform an operation, the operation comprising: receiving a request to execute a transaction including writing a record, the database system including a plurality of worker nodes operable to execute transactions for the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; writing the record to an in-memory cache of the worker node; issuing a commit request to a number of director nodes of the plurality of director nodes requesting permission to commit the record; determining that permission to commit the record has been granted based on receiving an approval response from a majority of the plurality of director nodes without receiving a response indicating disapproval of the transaction; 13. A computer readable medium comprising:

10. The operation includes: Aborting the transaction based on a determination that an approval response has not been received from a majority of the plurality of director nodes. The computer readable medium of claim 9 further comprising:

11. The operation includes: Aborting the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on a conflict between the record and another record known to the at least one director node. The computer readable medium of claim 9 further comprising:

12. The commit request specifies a proposed time period for committing the transaction, and the action is: Aborting the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on arrival of the commit request after the proposed period of time. The computer readable medium of claim 9 further comprising:

13. The operation includes: receiving a request for the record from another worker node; providing the record to the other worker node after committing the transaction; The computer readable medium of claim 9 further comprising:

14. A computer comprising: At least one processor; A memory having stored thereon program instructions executable by said at least one processor to carry out the method of any one of claims 1 to 8; A computer comprising:

Images