History Information in Director-Based Database Systems for Transaction Integrity.

Abstract

Techniques are disclosed that relate to a database system including a worker node operable to execute a transaction and a director node operable to ensure transactional integrity of the transaction. The worker node may receive a request to execute a transaction including writing a record. The worker node may then issue a request to a director node of the database system for information facilitating execution of operations for the transaction. The director node may determine whether to approve the request based on whether the operation may cause transactional inconsistency in the database system. The worker node may proceed to execute the operations for the transaction in response to receiving approval responses from a majority of the director nodes, with none of the received responses indicating disapproval of the transaction.

Description

[Technical Field]

[0001] The present disclosure relates generally to database systems, and more particularly to addressing slowdowns in components of database systems. [Background technology]

[0002] Enterprises routinely employ modern database management systems to organize and store collections of information so that users can efficiently access and manipulate them. Historically, these systems have been deployed on high-end, expensive hardware maintained in data centers owned by these enterprises. In most cases, high-end hardware performs predictably, allowing systems to achieve fast, responsive latency. However, over time, more and more enterprises are moving their systems to the cloud and, therefore, relying on hardware supplied by cloud providers. However, cloud providers are beginning to shift toward using cheaper hardware within their cloud infrastructure, which behaves less predictably than high-end hardware. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 2 is a block diagram illustrating example elements of a system including worker nodes and director nodes, according to some embodiments. [Figure 2] FIG. 2 is a block diagram illustrating exemplary elements of a worker node, according to some embodiments. [Figure 3] FIG. 2 is a block diagram illustrating exemplary elements of a director node, according to some embodiments. [Figure 4] FIG. 2 is a block diagram illustrating example elements of a history record data structure, according to some embodiments. [Figure 5]FIG. 1 is a block diagram illustrating exemplary elements of a snapshot-related interaction involving one worker node and multiple director nodes, according to some embodiments. [Figure 6] FIG. 2 is a block diagram illustrating an example layout of history information on different director nodes, according to some embodiments. [Figure 7] FIG. 2 is a block diagram illustrating exemplary elements of a commit-related interaction involving one worker node and multiple director nodes, according to some embodiments. [Figure 8] FIG. 10 is a block diagram illustrating another example layout of history information on different director nodes, according to some embodiments. [Figure 9] FIG. 2 is a block diagram illustrating example elements related to time alignment between components of a system, according to some embodiments. [Figure 10] FIG. 1 is a block diagram illustrating example elements of an assistant node, according to some embodiments. [Figure 11] FIG. 10 is a block diagram illustrating example elements of a flush operation in which a worker node and an assistant node flush the same records to a database, according to some embodiments. [Figure 12] FIG. 10 is a flow diagram illustrating an example method for a worker node interacting with a director node to facilitate execution of operations for a transaction, according to some embodiments. [Figure 13] FIG. 1 is a flow diagram illustrating an exemplary method for a director node to ensure transaction consistency within a system, according to some embodiments. [Figure 14] FIG. 10 is a flow diagram illustrating an example method for a worker node interacting with a director node to obtain history information describing the history of an approved record, according to some embodiments. [Figure 15] FIG. 10 is a flow diagram illustrating an example method for a worker node to select a proposed time associated with performing an operation for a transaction, according to some embodiments. [Figure 16] FIG. 10 is a flow diagram illustrating an example method for a worker node to select a proposed snapshot time for a transaction, according to some embodiments. [Figure 17] FIG. 10 is a flow diagram illustrating an example method for a director node to attempt to align the time observed by the director node with the time observed by other director nodes, according to some embodiments. [Figure 18] FIG. 10 is a flow diagram illustrating an example method for a director node to attempt to align times observed by the director node, according to some embodiments. [Figure 19] FIG. 1 is a flow diagram illustrating an example method for an assistant node to observe work performed by a worker node, according to some embodiments. [Figure 20] FIG. 10 is a flow diagram illustrating another example method for an assistant node to observe work performed by a worker node, according to some embodiments. [Figure 21] FIG. 1 is a block diagram illustrating elements of a multi-tenant system, according to some embodiments. [Figure 22] FIG. 1 is a block diagram illustrating elements of a computer system for implementing various systems described in this disclosure, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] As explained, cloud providers are shifting toward using cheaper, and therefore less reliable, hardware in their cloud infrastructure. The use of cheaper hardware has resulted in an increasing number of reports of servers randomly slowing down. This random slowdown of a server (or other component) can cause both gray failures (where the server suddenly experiences a significant drop in its performance (e.g., 99% or more)) and stragglers (where the server experiences a less noticeable slowdown, such as a 25% drop in performance). When the slowdown occurs, the server does the "right" thing, but slowly enough to circumvent failure detectors that would take the server down while also causing problems for the entire system. For example, a server may operate at half its normal speed but still be able to output work and respond to requests, albeit slowly, so the server's slower performance may not be suspected by the rest of the system.

[0005] Although slowdowns can affect transactional and non-transactional systems, techniques used to address slowdowns in non-transactional systems are not applicable to transactional systems. A transaction is a set of commands logically grouped as a single unit of work. A transactional system is a type of system that processes transactions according to a set of guidelines that ensure transactional integrity. ACID (Atomicity, Consistency, Isolation, and Durability) is an example of a guideline that attempts to ensure that either all or none of the transactions succeed, ensure that data is consistent, ensure that transactions occur in isolation, and ensure that once a transaction is committed, it stays within the system. Non-transactional systems are not bound by a set of guidelines that ensure transactional integrity. One technique for addressing slowdowns in non-transactional systems is to retry some of the computational work when a time limit is reached, i.e., after a period of time has passed without a slow server being able to produce results, the work is retried on another server. In some cases, computational work is pushed to multiple servers so that if one of those servers experiences a slowdown, the other servers can still return the results of the work. This is an acceptable approach for non-transactional systems because their work is idempotent, meaning that performing the same work multiple times does not introduce unintended states into the system. As an example, fetching and returning a static web page is an idempotent operation. This contrasts with non-idempotent operations, such as incrementing a counter, where multiple retries may cause the counter to increment more than intended. Because transactional database systems adhere to principles that ensure transaction integrity, their work is generally non-idempotent, and they are usually guaranteed to ensure that only one transaction appears to be occurring at a time.Therefore, attempting to do the same work on multiple servers is not feasible for transactional database systems. This disclosure addresses, among other things, the problem of how to protect transactional database systems from slowdowns that can slow down individual components of the system.

[0006] Slowdowns also cause problems when attempting to scale a transactional database system to have more database nodes (alternatively, "worker" nodes) executing transactions. As the number of database nodes in a system increases, the likelihood of transaction inconsistency also increases. In one embodiment, a director node is implemented that ensures transaction consistency in a transactional database system by checking for conflicts between transactions executed by worker nodes. However, a system that uses a single director node is susceptible to significant performance degradation if the director node experiences slowdowns, since transactions cannot be completed without the consent of the director node. The present disclosure further addresses, among other things, the problem of how to ensure transaction consistency among worker nodes in light of the slowdowns that occur in transactional database systems.

[0007] This disclosure describes techniques for implementing mechanisms to mitigate some or all of the effects of slowdowns on a system. In various embodiments described below, the system includes a database and a set of "worker" nodes that execute transactions for the system, including writing records to the database. The system further includes a group of "director" nodes that help ensure transaction consistency among transactions executed by the worker nodes. Throughout the execution of a transaction, a worker node may interact with the director node to ensure that the worker node's actions do not interfere with the actions of other worker nodes in the system. When interacting with the group of director nodes, the worker node sends a request to the set of director nodes (e.g., some or all of them) and waits for an approval response from at least a majority of the group of director nodes before proceeding. In some cases, the director node checks for conflicts between records submitted by the worker node and the history of approved records (i.e., records approved to be committed) known to the individual director node. Identifying a conflict may result in the director node sending a disapproval response to the worker node. In other cases, the director node may check for problems when initiating a transaction and return an approval response containing information about previously approved records up to a certain time. If the worker node does not receive an approval response from a majority of the group of director nodes, or if it receives a disapproval response, the worker node may abort the transaction.

[0008] These techniques can be advantageous because they can shift the impact of slowdowns on a system while ensuring transaction integrity in that system. In particular, as described in more detail below, a system can be structured so that ensuring transaction integrity does not require all director nodes to respond, but rather only a majority. Thus, one or more director nodes can become unexpectedly slow as a result of slowdowns without slowing down the progress of transactions in the system. Consider an example in which a system includes a group of three director nodes, where acknowledgment responses from at least two director nodes (the majority) together provide guarantees of transaction integrity. A worker node may send a commit request to all three director nodes for permission to commit the transaction. In this example, because the worker node sent a request to three director nodes and only two acknowledgment responses are required to ensure transaction integrity, one of the director nodes may experience slowdowns while the other two director nodes respond. As a result, that slowdown does not slow down the entire system.

[0009] This disclosure further describes the use of assistant nodes to mitigate some or all of the effects of slowdowns on worker nodes. In various embodiments, a worker node is associated with one or more assistant nodes that observe the work performed by the worker node. The assistant nodes may follow the worker node's transaction log and store records that match records written at the worker node. During operation, a first worker node (the "read" node) may communicate with a second worker node (the "write" node, whose work is being observed by the assistant node) to attempt to retrieve a record from the second worker node. In some instances, the write worker node may experience slowdowns and be unable to provide a response in a timely manner. The read worker node may then communicate with the write worker node's assistant node to retrieve the record. Because the assistant node observes the write worker node's work, the assistant node may provide a response on behalf of the write worker node. The response may include the record. These techniques can be advantageous because they can prevent read worker nodes from being slowed down (e.g., waiting for responses) by other worker nodes (e.g., write worker nodes) that are experiencing slowdowns. This disclosure also describes concepts related to time alignment between worker nodes and director nodes of a system. An exemplary application of the techniques of this disclosure will now be described with reference to FIG. 1.

[0010] Referring now to FIG. 1 , a block diagram of system 100 is shown. System 100 includes a set of components that may be implemented via hardware or a combination of hardware and software. In the illustrated embodiment, system 100 includes database 110, worker cluster 119 having worker nodes 120, and director cluster 130 having director node 135. Also as shown, database 110 and worker node 120 include record 115. In some embodiments, system 100 is implemented differently than shown. For example, system 100 may include an assistant node that observes the work of a worker node, although in some cases, a worker node may function as an assistant node for another worker node.

[0011] In various embodiments, system 100 implements a platform service (e.g., a customer relationship management (CRM) platform service) that enables users of the service to develop, run, and manage applications. System 100 may be a multi-tenant system that provides various functions to users / tenants hosted by the multi-tenant system. Thus, system 100 may execute software routines from various different users (e.g., the provider and tenants of system 100) and provide code, web pages, and other data to users, databases (e.g., database 110), and other entities of system 100. In various embodiments, system 100 is implemented using a cloud infrastructure provided by a cloud provider. Thus, database 110, worker node 120, and director node 135 may run on and utilize available cloud resources (e.g., computing resources, storage resources, network resources, etc.) of the cloud infrastructure to facilitate their operation. As an example, worker node 120 may run within respective virtual environments hosted on server-based hardware contained within the cloud provider's data center. However, in some embodiments, the system 100 is implemented using local or private infrastructure, as opposed to a public cloud.

[0012] Database 110, in various embodiments, is a collection of information organized to allow access, storage, and manipulation of that information. Accordingly, database 110 may include support software (e.g., storage nodes) that enable worker nodes 120 to perform operations (e.g., access, store, etc.) on the information stored in database 110. In various embodiments, database 110 is implemented using single or multiple storage devices connected to each other over a network (e.g., a storage-attached network (SAN)) and configured to store information redundantly to prevent data loss. Because the storage devices may persistently store data, database 110 may serve as persistent storage for system 100. Information (e.g., records 115) written to database 110 by worker nodes 120 may be accessible to other worker nodes 120. As described in more detail with respect to FIG. 11 , records 115 may be stored in a log-structured merge tree (LSM tree) file as part of the LSM tree implemented in database 110.

[0013] As shown, database 110 stores records 115. In various embodiments, records 115 are key-value pairs that include data and a corresponding key that can be used to look up the record 115. As an example, a record 115 may correspond to a row of data in a database table and specify values ​​for one or more attributes / fields of the database table. In various embodiments, records 115 are immutable, so that to update an underlying database construct (e.g., a row in a table), a new record 115 is written. Thus, a database construct may be associated with multiple records 115, each of which is a different version of the database construct. These records 115 may be referred to as "record versions" of the database construct. As an example, a first record 115 (first record version) that stores specific values ​​for a database row may be written first, followed by a second record 115 (second subsequent record version) that updates one or more of the values ​​in the database row. The two records 115 may be accessible using the same key.

[0014] Worker node 120, in various embodiments, provides various database services such as data storage, data retrieval, and / or data manipulation. In various embodiments, worker node 120 is a set of software routines executable on hardware, although in some embodiments, worker node 120 encompasses both hardware and software routines. Database services may be provided to other components within system 100 or components external to system 100. For example, worker node 120 may receive transaction requests 125 from an application node (not shown) to perform database transactions. A database transaction, in various embodiments, is a logical unit of work (e.g., a specified set of database operations) to be performed in connection with database 110. For example, processing a database transaction may include executing an SQL SELECT command to select one or more rows from one or more database tables. The contents of the row may be specified in records 115, and thus worker node 120 may return one or more records 115 corresponding to one or more table rows. Performing a database transaction may include a worker node 120 writing one or more records 115 to the database 110. In various embodiments, the worker node 120 first writes the records 115 to a local in-memory cache and then flushes them to the database 110. Before committing and flushing those records 115, the worker node 120 may communicate with the director node 135 to ensure that the records 115 do not conflict with record writes performed by other worker nodes 120. As used herein, the expression "committing a transaction" (including committing a record) is used in accordance with its well-understood meaning to refer to the process of causing changes made during the transaction to be saved and visible outside the entity performing the transaction.Worker nodes 120 may also communicate with director node 135 when initiating transactions to obtain information regarding the state of records stored within system 100 (e.g., the history of records 115 that have potentially been committed by other worker nodes 120 but have not been flushed to be available in database 110). The various components of worker node 120 are described in more detail with respect to FIG. 2.

[0015] In various embodiments, director nodes 135 facilitate the execution of transactions and help ensure the transactional integrity of those transactions. In various embodiments, director nodes 135 are sets of software routines executable on hardware, although in some embodiments, director nodes 135 encompass both hardware and software routines. As shown, director nodes 135 form director cluster 130. The number of director nodes 135 in director cluster 130 may vary depending on the embodiment. For example, director cluster 130 may include three director nodes 135, five director nodes 135, eight director nodes 135, etc. Although director nodes 135 are part of director cluster 130, in various embodiments, director nodes 135 do not communicate with each other. Specifically, when worker node 120 sends a request to two or more director nodes 135, those director nodes 135 may not communicate information about the request among each other. Thus, a director node 135 may process a request from its own knowledge and perspective, independent of the knowledge and perspective of other director nodes 135. However, in some embodiments, director nodes 135 communicate with each other to share certain information (e.g., their respective histories of approved records) that can be used to complete a request. The various components of director node 135 are described in more detail with respect to Figures 3 and 4.

[0016] In various embodiments, to process requests from worker nodes 120, director node 135 maintains information about the state of records 115 stored within worker nodes 120. When initiating a transaction, worker node 120 may send a snapshot request to a set of director nodes 120 to obtain a snapshot time. In various embodiments, the snapshot time may be used by the owner of the snapshot time (e.g., a transaction) to identify the state of system 100 at a particular point in time and to access records available at that time. The concept of snapshot time is described in more detail with respect to FIG. 2. Accordingly, director node 135 may acknowledge the snapshot time and then return a response to worker node 120. In many cases, the response includes information about the state of records 115 so that worker node 120 can locate and access records 115 that may be relevant to processing the transaction. However, the historical information from director node 135 may be an incomplete history of acknowledged records. As a result, in some cases, worker node 120 must obtain responses from a majority of director clusters 130 to obtain the complete history. An exemplary layout of history information on director node 135 and a description of the snapshot request process are provided in more detail with respect to FIGS. 5 and 6.

[0017] When a worker node 120 desires to commit a transaction, it may send a commit request to a set of director nodes 135 for approval to commit. When deciding whether to approve the request, the director node 135 may use its historical information to determine whether the requested action(s) (e.g., permission to commit) would result in an inconsistent or undesirable state in the system 100. If there is a conflict, the director node 135 may deny the request, causing the worker node 120 to abort some or all of the database transaction. If there is no conflict from the director node's perspective, the director node 135 may approve the request and update its historical information regarding the state of the record 115 to reflect the approval. A description of an exemplary update of historical information and commit request process is provided in more detail with respect to FIGS. 7 and 8.

[0018] 2, a block diagram of exemplary elements of a worker node 120 is shown. In the illustrated embodiment, the worker node 120 includes a database application 200, an in-memory cache 210, and historical information 220. As further shown, the database application 200 maintains a local clock 205, the in-memory cache 210 stores records 115 associated with keys 215, and the historical information 220 includes a set of historical records 225. In some embodiments, the worker node 120 is implemented differently than shown. For example, the worker node 120 may store a transaction log having log records. However, in some embodiments, the transaction log is stored in storage shared among the worker nodes 120, as described with respect to FIG. 10.

[0019] Database application 200, in various embodiments, is a set of executable program instructions for managing database 110, which may include managing an LSM tree built around database 110. Thus, database application 200 may receive requests to perform database transactions involving reading and / or writing records 115 for system 100. Upon receiving a transaction request, database application 200 may initiate a database transaction based on the request, including obtaining a snapshot time for the database transaction. The snapshot time, in various embodiments, is a value that indicates which records 115 can be read by worker nodes 120. The value may be a numeric value indicating a time (e.g., a time window covering 1 millisecond) or a period of time. That is, system 100 may increment an integer commit value over time, and a snapshot time may correspond to one of the commit values. Records 115 whose commit value is less than (or, in various cases, equal to) the snapshot time may be read by worker nodes 120. For example, a transaction may be assigned a snapshot time of 445 (or a time of 10:11:20). As a result, database application 220 may read records 115 for that transaction that have a commit value less than or equal to 445 (or 10:11:20). As described further below, worker node 120 may communicate with director node 135 to obtain approval of the transaction's snapshot time.

[0020] The local clock 205, in various embodiments, identifies the local time observed by the worker node 120. In some cases, the local clock 205 is provided by an operating system running on the worker node 120, and in some cases, the database application 200 maintains its own local clock 205, independent of how other applications running on the worker node 120 observe time. In various embodiments, the database application 200 performs certain database operations based on the time on the local clock 205. For example, as described further below, the database application 200 may use the clock 205 to select and propose a snapshot time (or commit time, both of which are described in more detail below) to the director node 135 for approval. In various embodiments, the worker node 120 and the director node 135 each have their own opinion of the current time in the system 100 based on their local clocks 205. However, the local clocks 205 of the worker node 120 and the director node 135 may become out of sync with each other over time. That is, their local clocks 205 may identify different times. As a result, a node may observe a significantly different time than other nodes in system 100. This can cause the undesirable effect of a node (e.g., director node 135) processing requests significantly later than other nodes (e.g., other director nodes 135). As described in more detail with respect to FIG. 9, nodes in system 100 may attempt to gradually align their times to prevent the nodes from becoming too far out of synchronization.

[0021] In-memory cache 210, in various embodiments, is a buffer that stores data in memory (e.g., random access memory) of worker node 120. HBase™ memstore is one example of an in-memory cache 210. After obtaining sufficient acknowledgements for the transaction's snapshot time, worker node 120 may begin reading and writing records 115 to process the transaction. In various embodiments, worker node 120 first writes record 115 to its in-memory cache 210 and then flushes the record 115 to database 110 after being committed by worker node 120. In various cases, the latest / most recent record version of a row in a database table is found in record 115 stored in in-memory cache 210. However, in some embodiments, records 115 written to a worker node's in-memory cache 210 are not visible to other worker nodes 120. That is, other worker nodes 120 do not know what information is stored in in-memory cache 210 unless asked or told. Thus, in various embodiments, a worker node 120 obtains historical information 220 from the director node 135 that identifies where a particular record 115 (e.g., the latest record version of a database row) can potentially be found. Thus, the worker node 120 can use the historical information 220 to identify which records 115 are stored in the in-memory caches 130 of other worker nodes 120.

[0022] Historical information 220, in various embodiments, includes a collection of historical records 225. Historical records 225, in various embodiments, include information identifying the location of a record 225 within system 100 for a key 215 that can be used to look up the record 115 corresponding to that historical record 115. In various embodiments, historical information 220 captures different records (having different keys) and different versions of the same database construct (e.g., a data row) identified with the same key. Thus, historical information 220 may include multiple historical records 225 for multiple versions of the same database construct committed by different transactions at different times using the same key 215. When searching for a particular record version of a database construct (e.g., the latest record 115 of a database row), worker node 120 may identify a set of historical records 225 based on the key 215 corresponding to the database structure. Worker node 120 may then identify, from the timestamp, which historical record 225 corresponds to the record version and then attempt to access the record 115 using that historical record 225. In some embodiments, a single history record 225 may store the locations of multiple records 115 for the same key 215 .

[0023] When attempting to access a record 115 from a location external to the worker node 120, in various embodiments, the worker node 120 sends a record request 204 to that location. For example, as shown, the worker node 120 sends the record request 204 to the database 110. The record request 204 may specify a key 215 corresponding to the record 115, which the recipient can use to attempt to locate the record 115. The worker node 120 may then receive a record response 206. In some cases, the record response 206 includes the requested record 115. However, in some cases, the record response 206 does not include the requested record 115, even if the history record 225 indicates that the record 115 was at that location. For example, the director node 135 may create a history record 225 indicating that the record 115 may have been committed at the worker node 120. However, after being granted permission to commit the transaction associated with that record 115, that worker node 120 may abort the transaction and delete the record 115. As a result, upon receiving a record request 204 for the record 115, that worker node 120 returns a record response 206 indicating that the worker node 120 did not write the record 115 at the time specified in the received record request 204.

[0024] After processing a transaction (e.g., after writing all requested records 115 for the transaction), in various embodiments, worker node 120 sends a commit request to a majority of director nodes 135 in director cluster 130. The commit request may include the records 115 written for the transaction so that director nodes 135 can check for record conflicts. In response to receiving approval from a majority of director nodes 135 without disapproval, worker node 120 may commit the records 115. In some embodiments, disapproval is tolerated as long as a majority of director cluster 130 approves the commit request. As part of the commit, in various embodiments, worker node 120 stamps each record 115 with a value (e.g., a time, a number indicating a duration, etc.) indicating when the record 115 was committed. Committed records 115 may remain in the in-memory cache 210 until a triggering event (e.g., the in-memory cache 210 is nearly full or full) causes those records 115 to be flushed to the database 110. Records 115 stored in the in-memory cache 210 are accessible to and can be used in other transactions whose snapshot times / values ​​are greater than the commit time / value of the records 115 being accessed.

[0025] 3, a block diagram of exemplary elements of director node 135 is shown. In the illustrated embodiment, director node 135 includes orchestration application 300 and history information 220. As further shown, orchestration application 300 maintains local clock 205, and history information 220 includes history records 225. In some embodiments, director node 135 is implemented differently than shown. For example, director node 135 may interface with other director nodes 135 in director cluster 130.

[0026] In various embodiments, orchestration application 300 is a set of executable program instructions to facilitate the execution of transactions and ensure the transactional integrity of those transactions. To facilitate the execution of transactions, in various embodiments, orchestration application 300 provides some or all of its history information 220 to worker nodes 120 to enable the worker nodes 120 to learn about work performed by other worker nodes 120. As described, in various embodiments, committing a record 115 includes communicating with a director node 135 to obtain approval to commit the record 115. Because a director node 135 approving the commit is aware that the record 115 will potentially be committed, the orchestration application 300 can store an indication of that approval as part of the history information 220. In some cases, orchestration application 300 may miss a commit request (or otherwise fail to process the request) and therefore may not store a complete history of all recent records 115 that have been approved for commitment by director cluster 130. As a result, in various embodiments, the history information 220 identifies only those records 115 for which commitments have been approved by that orchestration application 300 .

[0027] Because the orchestration application 300 tracks records 115 for which commitment has been approved, the orchestration application 300 can facilitate the execution of a transaction by enabling a worker node 120 to potentially locate records 115 relevant to executing that transaction, including records 115 stored in its in-memory cache 210. Also, because the orchestration application 300 tracks records 115 that have been approved at its director node 135, in various embodiments, it can help ensure transaction consistency between transactions by ensuring that an approval-requested record 115 does not conflict with an approved record 115 identified in the director node's history information 220. Thus, upon receiving a commit request for a record 115, the orchestration application 300 can determine whether another record 115 having the same key 215 (i.e., associated with the same database reconstruction) has already been approved within a particular time frame. In various cases, the time frame corresponds to the time between the worker node's snapshot time for the transaction and the current time observed by the director node 135. If a conflicting approved record 115 exists, the orchestration application 300 can return a disapproval response to the requesting worker node 120, preventing the requested record 115 from being committed and ensuring transaction integrity. If no conflicting records 115 exist, the orchestration application 300 can return an approval response and add a history record 225 corresponding to the new approved record 115 to the history information 220. Note that in some cases, a director node 135 can actually detect a conflict with a previous transaction when the previous transaction was not approved by a majority of the director nodes 135. In this case, if enough other director nodes 135 approve the transaction, the transaction can be committed and the conflict perceived at that director node 135 was not a true conflict.

[0028] As shown, the orchestration application 300 maintains a local clock 205 that identifies the current time as observed at the director node 135 (or at least as observed by the orchestration application 300, which may be independent of the time observed by other applications executing on the director node 135). Like the worker nodes 120, the director nodes 135 may perform certain operations at specified times according to their respective local clocks 205. As an example, a commit request may be processed at the commit time proposed in the commit request. As described above, the clocks 205 of the director node 135 and the worker nodes 120 may become out of sync, and thus, a director node 135 may process a request at a logical time that is substantially slower than another director node 135. Thus, the nodes of the system 100 may attempt to gradually align their times to avoid becoming too out of sync, as described in more detail with respect to FIG. 9.

[0029] 4, a block diagram of an exemplary history record 225 is shown. In the illustrated embodiment, the history record 225 specifies a record identifier 410, a position 420, and a commit time / epoch 430. In some embodiments, the history record 225 is implemented differently than shown. As an example, the history record 225 may specify the director identifier of the director node 135 that wrote the history record 225.

[0030] The record identifier 410, in various embodiments, is information that identifies the record(s) 115 that correspond to the history record 225. In various cases, the record identifier 410 is a key 215 that can be used to look up the corresponding record(s) 115. The record identifier 410 may be specified in a commit request sent from a worker node 120 to a director node 135 to obtain approval to commit a set of corresponding records 115. In response to deciding to approve the commit request, the director node 135 may generate, for each record 115 in the set, a history record 225 whose record identifier 410 corresponds to that record and is obtained from the commit request.

[0031] Location 420, in various embodiments, identifies the location within system 100 of the record 115 that corresponds to history record 225. For example, location 420 may specify a worker node 120 or a database 110. In various cases, location 420 identifies the worker node 120 that initially obtained permission to commit the corresponding record 115. When that record 115 is committed and flushed to database 110, location 420 may be updated by director node 135 (which created the corresponding history record 225) to identify the location within database 110 where that record 115 was stored. Thus, over time, location 420 may be adjusted as the corresponding record 115 moves within system 100.

[0032] The commit time 430, in various embodiments, identifies a logical time (or time window) within which the associated record 115 is identified as being committed. In various embodiments, the worker node 120 may commit the record 115 of the transaction at the commit time 430, or may choose not to commit it. However, the worker node 120 cannot commit the transaction at any other commit time 430 without re-requesting approval at the new commit time 430. In some cases, the record 115 is actually committed at a physical time that is later than the logical time of the commit time 430. In particular, the commit request may specify a proposed commit time. The director node 135 may process the request at the proposed commit time and, if the request is approved, may generate a history record 225 whose commit time 430 is the proposed commit time. The requesting worker node 120 may receive the approval response and may commit the record 115 of that commit request at the proposed commit time 430 (if it chooses to commit).

[0033] 5, a block diagram of exemplary snapshot-related interactions is shown involving one worker node 120 and three director nodes 135A-C. In some embodiments, there may be more director nodes 135 than shown (e.g., five director nodes 135), and a worker node 120 may issue snapshot requests 510 to fewer than all director nodes 135 in a director cluster 130 (e.g., four of the five).

[0034] As described above, a worker node 120 may receive a request to perform a transaction, which may include reading and writing records 115. The request may be received from an application node, not shown. As part of processing the transaction, in various embodiments, the worker node 120 obtains a snapshot time for that transaction, which corresponds to the state of the system 100 at a particular point in time. Records 115 committed before the snapshot time may be accessed by that worker node 120 for the transaction, and when committing the transaction, the snapshot time may serve as a reference point for checking for record conflicts.

[0035] To ensure that a transaction is assigned a single snapshot time, in various embodiments, the worker node 120 selects the proposed snapshot time 505 on behalf of the director node 135. In various embodiments, the director nodes 135 may not communicate with each other, or at least may not communicate snapshot information, so the worker node 120 may select the snapshot time on behalf of the director node 135. In various embodiments, the director nodes 135 may not communicate with each other, so a situation may arise in which a director node 135 selects a different snapshot time than another director node 135. Thus, a transaction may be associated with multiple snapshot times, which may violate transaction guarantees. However, in some embodiments, the director node 135 may assign a transaction a snapshot time 505 that corresponds to the time when the snapshot request 510 from the worker node 120 was seen by the director cluster 130.

[0036] As described above, a director node 135 may process a request received at a specified time. As a result, when selecting a proposed snapshot time 505, in various embodiments, a worker node 120 selects the proposed snapshot time 505 such that a corresponding snapshot request 510 is estimated to arrive at the director node 135 before the proposed snapshot time 505 according to the director node's 135's respective local clock 205. If the snapshot request 510 arrives at the director node 135 early, the director node 135 may store the request and delay its processing until the proposed snapshot time 505 identified in the snapshot request 510 is reached according to the director node's 135's local clock 205. In various cases, a director node 135 processes requests received at the same local time but at different times from a global perspective. If a snapshot request 510 arrives at a director node 135 after the proposed snapshot time 505, the director node 135 may issue a snapshot response 520 indicating that the proposed snapshot time 505 is rejected. In some embodiments, worker nodes 120 send snapshot requests 510 to director nodes 135, allowing them to select snapshot times 505 for their corresponding transactions.

[0037] In some embodiments, in response to receiving a rejection of the proposed snapshot time 505, the worker node 120 selects another proposed snapshot time 505 that is further away from the worker node's 120's current time than the rejected snapshot time 505 was from the time selected by the worker node 120. As described in more detail with respect to FIG. 9 , the worker node 120 may track delays in communicating with the director node 135 and use those delays to adjust how far into the future the proposed snapshot time 505 is selected. In some embodiments, in response to receiving a rejection of the proposed snapshot time 505 but also receiving approval from at least a majority of the director nodes 135 of the cluster 130, the worker node 120 proceeds to execute the transaction in light of the proposed snapshot time 505. Thus, in some cases, if a particular director node 135 is moderately behind in time compared to other director nodes 135, the worker node 120 may select a proposed snapshot time 505 such that the corresponding snapshot request 510 is estimated to arrive at the other director nodes 135 before the proposed snapshot time 505, even if the corresponding snapshot request 510 is estimated to arrive at the particular director node 135 after the proposed snapshot time 505. Thus, when selecting a proposed snapshot time 505, the worker node 120 may consider a trade-off between reaching more director nodes 135 (which increases the chance of receiving responses from a majority) and selecting a proposed snapshot time 505 that is not too far in the future (which may delay the transaction).

[0038] In response to receiving snapshot request 510 with proposed snapshot time 505, in various embodiments, director node 135 determines whether to reject proposed snapshot time 505. If proposed snapshot time 505 is after the current time at director node 135 or would cause a conflict with another transaction, director node 135 may reject it. In response to approving proposed snapshot time 505, in various embodiments, director node 135 returns snapshot response 520 with director node history information 220 up to the proposed snapshot time 505. As shown, worker node 120 receives snapshot responses 520 from director nodes 135A-B, but not from director node 135C. Assuming that director cluster 130 includes only director nodes 135A-C and the received snapshot response 520 approves the proposed snapshot time 505, worker node 120 may proceed to process the transaction because approval was received from a majority of director cluster 130 with no disapproval.

[0039] 6, a block diagram of an exemplary layout of history information on director nodes 135A-C is shown. In the illustrated embodiment, there is a complete history 600 that includes four history records 225A-D. As shown, director node 135A includes history records 225A and 225C-D, director node 135B includes history records 225A-B, and director node 135C includes history records 225A-D. In some cases, complete history 600 may include more or fewer history records 225 than shown, and the layout of the history information on director nodes 135 may differ.

[0040] 5, worker node 120 receives snapshot responses 520 from director nodes 135A-B, but not from director node 135C. As explained, snapshot response 520 may include history information 220, including history records 225 up to proposed snapshot time 505. Assuming proposed snapshot time 505 is after the commit time 430 of history record 225D, in this example, worker node 120 receives history records 225A and 225C-D from director node 135A and history records 225A-B from director node 135B. In various embodiments, because a majority of director nodes 135 must approve the commit and therefore have history records 225 of approved records 115, if worker node 120 receives approval from a majority of director nodes 135, worker node 120 is guaranteed to obtain complete history 600. Continuing with this example, worker node 120 obtains complete history 600 from director nodes 135A-B (which are the majority in this example), even though none of these director nodes stores complete history 600. To obtain complete history 600, worker node 120 may union the history records 225 received from director nodes 135A-B. If worker node 120 receives snapshot responses 520 from less than a majority of director nodes 135, there is no guarantee that worker node 120 will obtain complete history 600. As an example, if worker node 120 receives snapshot responses 520 only from director node 135B, worker node 120 obtains an incomplete history 600.

[0041] 7, a block diagram of an exemplary commit-related interaction is shown involving a worker node 120 and three director nodes 135A-C. In some embodiments, there may be more director nodes 135 than shown (e.g., five director nodes 135), and a worker node 120 may issue a commit request 710 to fewer than all director nodes 135 in a director cluster 130 (e.g., four of the five).

[0042] After processing a transaction, a worker node 120 may decide to commit the transaction, including one or more records 115 written to the worker node's in-memory cache 210 for that transaction. To ensure that the records 115 do not conflict with records 115 written by other worker nodes 120, the worker node 120 issues a commit request 710 to at least a majority of the director nodes 135. As shown, the worker node 120 issues the commit request 710 to director nodes 135A-C, which in the illustrated embodiment are all director nodes 135. The commit request 710, in various embodiments, identifies the records 115 written for the transaction (e.g., via record identifiers 410), a proposed commit time 705 to apply to those records 115, and an approved snapshot time for the transaction. Similar to the proposed snapshot time 505, the proposed commit time 705 may identify a numerical value indicating a time (or time window) or period.

[0043] In various embodiments, similar to selecting a snapshot time, a worker node 120 may select a commit time on behalf of a director node 135 because director nodes 135 may not communicate with each other, or at least not communicate about commits. When selecting a proposed commit time 705, in various embodiments, a worker node 120 selects a proposed commit time 705 such that a corresponding commit request 710 is estimated to arrive at the director node 135 before the proposed commit time 705 according to the local clock 205 of each of the director nodes 135. Similar to a snapshot request 510, if a commit request 710 arrives early at a director node 135, the director node 135 may store the request and delay its processing until the proposed commit time 705 identified in the commit request 710 is reached according to the local clock 205 of that director node 135. If the commit request 710 arrives after the proposed commit time 705, the director node 135 may issue a commit response 720 indicating that the commit request 710 is rejected. As described with respect to the proposed snapshot time 505, in various embodiments, the worker node 120 may also adjust how far into the future the proposed commit time 705 is selected. For example, in response to receiving a rejection of the commit request 710 based on its proposed commit time 705, the worker node 120 may select another proposed commit time 705 that is further away from the worker node's 120's current time than the selected time is from the rejected commit time 705.

[0044] In response to receiving the commit request 710, in various embodiments, the director node 135 determines whether to reject the commit request 710. As part of determining whether to reject the commit request 710, the director node 135 may determine whether any of the records 115 identified in the request conflict with records 115 previously approved for commit by that director node 135. In various embodiments, the director node 135 records the approved records 115 in the form of history records 225, so the director node 135 can examine the history records 225 in its history information 220. In various embodiments, a first record 115 conflicts with a second record 115 if they are associated with the same record identifier, and the second record 115 is identified as having been committed after the snapshot time associated with the transaction of the first record 115. Thus, the director node 135 may search its history records 225 for ones whose record identifier 410 matches the record identifier of the record 115 being considered for approval. If a history record 225 is found, the director node 135 may determine whether its commit time 430 occurs after the snapshot time specified in the commit request 710. If a conflicting record 115 exists, the director node 135 sends a disapproved commit response 720 to the worker node 120 indicating that a record conflict exists. If no conflicting record 115 exists, the director node 135 sends an approved commit response 720. The director node 135 may also create a history record 225 for the record 115 of the approved transaction.

[0045] As shown, worker node 120 receives commit responses 720 from director nodes 135A and 135C, but not from director node 135B. Assuming that director cluster 130 includes only director nodes 135A-C and the received commit responses 720 approve the commit, worker node 120 may proceed to commit the transaction because approval has been received from a majority of director cluster 130 without any disapprovals. In response to receiving a disapproval of the proposed commit time 705, worker node 120 may select a new proposed commit time 705 and send a commit request 710 with the proposed commit time 705 to director nodes 135A-C, or worker node 120 may abort the transaction. In response to receiving a commit response 720 that does not approve the commit due to one or more record conflicts, worker node 120 may or may not abort the transaction, even if worker node 120 receives approval from a majority of director cluster 130. In some embodiments, director node 135 approving the commit records (e.g., via history record 225) that the commit occurred even if the transaction was aborted by worker node 120.

[0046] 8, a block diagram of an exemplary layout of history information on director nodes 135A-C is shown. In the illustrated embodiment, there is a complete history 600 that includes five history records 225A-E. As shown, director node 135A includes history records 225A and 225C-E, director node 135B includes history records 225A-B, and director node 135C includes history records 225A-E. In some cases, complete history 600 may include more or fewer history records 225 than shown, and the layout of the history information on director nodes 135 may differ.

[0047] To facilitate the following discussion, it is assumed that the exemplary layout of history information shown in FIG. 6 occurs before the exemplary commit-related interaction between the worker node 120 and the three director nodes 135A-C shown in FIG. 7. Also, the exemplary layout of history information shown in FIG. 8 occurs after the exemplary commit-related interaction. As described above, in various embodiments, when a director node 135 decides to approve a commit, the director node 135 stores a history record 225 that identifies the records 115 to be committed and the location where they are assumed to be committed. In FIG. 7, the illustrated worker node 120 receives a commit response 720 from director nodes 135A and 135C, but not from director node 135B. Assuming that a transaction including a particular record 115 is approved to commit, director nodes 135A and 135C, in this example, individually store a history record 225E for that particular record 115. However, director node 135B does not store history record 225E because it may be experiencing a slowdown and may be unable to process the received commit request 710 in Figure 7. As a result, the history of approved records known to one director node 135 may differ from the history of approved records known to another director node 135. Because the transaction was approved by a majority and history record 225E was stored in a majority of director nodes 135, worker nodes 120 that receive acknowledgments from a majority of director nodes for snapshot request 510 receive history record 225E, making them aware of the particular record 115.

[0048] In various embodiments, a director node 135 deletes historical records 225 from its historical information 220 over time. In various cases, a new record version of a database construct (e.g., a database row) is approved for commitment to replace a previous record version. Thus, a director node 135 may replace a historical record 225 of a previous record version with a new historical record 225 of a new record version. In some cases, a director node 135 deletes a historical record 115 in response to the record 225 becoming older than a certain threshold. For example, a record 115 associated with historical record 225A may be written to the LSM tree of database 110 and merged down a level in the LSM tree over time. After that record 115 reaches a certain level, a director node 135 may delete historical record 225A from its historical information 220 (when that director node 135 approved the transaction for that record 115 and created historical record 225A). As part of deleting the history record 225 from its history information 220, in some embodiments, the director node 135 sends the history record 225 to the database 110 so that it becomes accessible to the worker node 120 from the database 110.

[0049] Referring now to FIG. 9 , a block diagram of exemplary elements related to aligning time among components of system 100 is shown. In the illustrated embodiment, there is a worker node 120 and three director nodes 135A-C, all having their own local clocks 205. As further shown, the local clocks 205A and 205B of director nodes 135A and 135B, respectively, identify the time "10:11:20," the local clock 205C of director node 135C identifies the time "10:11:23," and the local clock 205D of worker node 120 identifies the time "10:11:19." While only three director nodes 135 and one worker node 120 are shown, in various embodiments there may be more director nodes 135 and / or worker nodes 120. Also, in some embodiments, time is represented differently than shown. For example, time may be a number indicating a duration, such as in minutes.

[0050] As described, in various embodiments, a node in system 100 includes a local clock 205 that identifies the time observed by that node for system 100. That time may differ from the time observed by another node. For example, worker node 120 observes the time "10:11:19," while director node 135 observes the time "10:11:23." As further described, a node may perform operations at a specified time that may affect other nodes. A node that observes a time that is reasonably different from the average time observed in system 100 may cause undesirable delays because one or more operations are performed later than expected by other nodes. Therefore, it may be desirable to continually attempt to align the time within system 100 so that nodes do not become significantly out of sync with one another.

[0051] In some embodiments, worker nodes 120 and director nodes 135 communicate their local times directly with each other. However, in various embodiments, worker nodes 120 and director nodes 135 do not communicate their local times with each other. Instead, time delays in communication between worker nodes 120 and director nodes 135 are tracked and may be used by the nodes to shift their local clocks 205 in an attempt to align their time with the time observed by other nodes. In various embodiments, when a worker node 120 issues a request (e.g., a snapshot request 510 or a commit request 710) to a director node 135, the worker node 120 stores a send timestamp 910 that specifies the time the request was issued according to its local clock 205. When the request is received at the director node 135, it may store an arrival timestamp 920 that specifies the time the request was received according to its local clock 205. When that director node 135 returns a response (e.g., snapshot response 520 or commit response 720) to the worker node 120, the response may include its arrival timestamp 920. The worker node 120 may calculate a time delay that is the difference between the send timestamp 910 and the arrival timestamp 920. When the worker node 120 issues another request to the director node 135, it may include the time delay in the request. As a result, the director node 135 may learn about the time delay associated with the worker node 120 communicating with itself and other director nodes 135 in the director cluster 130.

[0052] For example, worker node 120 may issue a commit request 710 to director nodes 135A-C and store a send timestamp 910 that specifies the time "10:11:17." In response to receiving that commit request 710, director node 135A may store an arrival timestamp 920A that specifies the time "10:11:20" according to local clock 205A, director node 135B may store an arrival timestamp 920B that specifies the time "10:11:20" according to local clock 205B, and director node 135C may store an arrival timestamp 920C that specifies the time "10:11:23" according to local clock 205C. The arrival timestamps 920 may be provided to worker node 120 in a commit response 720 to the commit request 710. Based on send timestamp 910 and arrival timestamps 920A-C, worker node 120 determines that there was a 3-second time delay when communicating with director nodes 135A and 135B, and a 6-second time delay when communicating with director node 135C. When worker node 120 communicates with director node 135B again, worker node 120 can provide those time delays.

[0053] Based on those time delays, in various embodiments, director node 135 determines whether its local clock 205 should speed up or slow down. For example, director node 135C may learn that commit request 710 took 6 seconds to arrive, compared to 3 seconds for director nodes 135A and 135B. From this, director node 135C may determine that its local clock 205C identifies a time that is approximately 3 seconds ahead of the time observed by director nodes 135A and 135B. Often, director node 135 attempts to consider factors that affect the time delay between director node 135 and worker node 120. Such factors may include, for example, the network distance between worker node 120 and director node 135, the number of devices a request passes through on the way from one node to another, the delay between worker node 120 sampling the time and sending the request, and the delay between director node 135 receiving the request and sampling the time. Based on one or more of these factors, the director node 135 may estimate the actual clock skew based on the time delay. In various embodiments, the worker nodes 120 maintain an average (which may be weighted for recent traffic) of the time delays in communicating with each director node 135. The director node 135 may receive the average time delays from multiple worker nodes 120 and then aggregate those time delays to generate an aggregated average time delay. Thus, in various embodiments, the director node 135 learns the average time delay across multiple worker nodes 120 in communicating with a given director node 135. The aggregated average time delay may allow the director node 135 to better estimate the actual clock skew, taking into account factors that may cause delays in communication.

[0054] Thus, director node 135 may align their time very gradually. By comparing the perceived delay in receiving requests from each worker-director pair, director node 135 may determine whether its time is out of sync with other nodes based on a weighted average of the time delays of the other director nodes 135. In response to deciding to speed up its local clock 205, in various embodiments, director node 135 shifts the time of its local clock 205 forward and increases its own execution speed (e.g., by running processor cores more frequently) to increase the number of requests it can process so that its local clock 205 can be shifted at a faster rate. In some cases, director node 135 may shift the time of its local clock 205 forward and skip work (e.g., requests) whose specified processing time falls within the time being shifted. In response to deciding to slow down its local clock 205, in various embodiments, director node 135 decreases the rate at which the time of its local clock 205 is shifted forward. In some cases, a director node 135 may freeze the time on its local clock 205 until the director node 135 considers the time aligned.

[0055] In various embodiments, a worker node 120 shifts the time of its local clock 205 based on the observed time delay. As an example, a worker node 120 may observe that its requests are taking longer and longer to reach the director node 135. Based on this observation, the worker node 120 may speed up its local clock 205. Alternatively, the worker node 120 may observe that its requests are reaching the director node 135 earlier than expected and may therefore slow down its local clock 205. In various embodiments, the worker node 120 also uses the observed time delay to determine how far into the future a proposed snapshot time 505 or commit time 705 should be selected to arrive before their proposed time at the director node 135 according to their respective local clock 205. As the time delay changes over time, the worker node 120 may adjust how far into the future the proposed snapshot time 505 or commit time 705 is selected.

[0056] 10, a block diagram of exemplary elements related to an assistant node 1030 is shown. In the illustrated embodiment, there are worker nodes 120A-B, an assistant node 1030, and shared storage 1040. As shown, worker node 120B and assistant node 1030 include records 115, and shared storage 1040 includes logs 1045. The illustrated embodiment may be implemented differently than shown. As an example, there may be multiple assistant nodes 1030 that observe the work performed by worker node 120.

[0057] As described, processing a transaction may include accessing one or more records 115, which may be stored in the database 110 or in the in-memory cache 210 of a worker node 120. If a record 115 is identified in the in-memory cache 210 (e.g., via the history record 225), the worker node 120 may send a record request 204 for that record 115 to the corresponding worker node 120. For example, as shown, worker node 120A issues a record request 204 to worker node 120B for a record 115 stored on worker node 120B. Like other components of system 100, a worker node 120 may experience slowdowns that cause the worker node 120 to become slow or unresponsive to record requests 204. As a result, the worker node 120 may delay the processing of transactions of other worker nodes 120. For example, worker node 120B may become unresponsive, so worker node 120A may not receive record responses 206 to its record requests 204. To help alleviate this problem, in various embodiments, worker node 120 has one or more assistant nodes 1030 that can answer record requests 204 on behalf of that worker node 120 when it experiences a slowdown.

[0058] In various embodiments, the assistant node 1030 observes the work performed by the worker node 120B and locally stores information corresponding to that work. In various embodiments, the assistant node 1030 is a set of software routines executable on hardware, and in some embodiments, the assistant node 1030 encompasses both hardware and software routines. In some embodiments, the assistant node 1030 is another worker node 120. As a result, the worker nodes 120 can be assigned to each other such that each worker node 120 is an assistant node 1030 for another worker node 120. In some embodiments, the assistant node 1030 runs with other assistant nodes 1030 on a single virtual machine, while in other embodiments, they each run on separate virtual machines.

[0059] To observe the work performed by worker node 120B, in various embodiments, assistant node 1030 follows worker node 120B's log 1045, reads all record changes, and places them in its own memory (e.g., memory cache 210). As shown, log 1045 is maintained in shared storage 1040 shared between worker node 120B and assistant node 1030. In some embodiments, a worker node's log 1045 is stored locally at worker node 120 rather than at shared storage 1040, such that its assistant node(s) 1030 may access log 1045 from its worker node 120 rather than from shared storage 1040. Shared storage 1040 may be part of database 110 or a separate storage mechanism. Log 1045, in various embodiments, is a collection of log records that identify changes (e.g., record changes) to information stored in a database (e.g., database 110). A log record may be a record that identifies a new data value for a database construct (e.g., a database row), a commit record that identifies the commit of a transaction, or a rollback record that allows part or all of the database to be rolled back. Thus, the log 1045 may identify pending and committed transactions, including the order in which those transactions were committed. A worker node 120 that writes a transaction's record 115 may update its log 1045 by appending a log record that identifies its work. Thus, the log 1045 may grow over time as more log records are appended by the worker node 120.

[0060] Thus, in various embodiments, assistant node 1030 reads log records written to worker node 120B's log 1045 and builds an in-memory image of recent transactions performed by worker node 120B. Thus, as worker node 120B writes records 115 to its cache 210 and writes log records to its log 1045, assistant node 1030 can learn about those records 115 from its log 1045 and recreate them in its own memory. As a result, assistant node 1030 can store one or more of the same records 115 stored at worker node 120B. Because assistant node 1030 follows worker node 120B's log 1045 and stores one or more of the same records 115, assistant node 1030 can process record requests 204 from worker node 120A as if it were worker node 120B.

[0061] In some embodiments, a worker node 120A seeking a record 115 first sends a record request 204 to a worker node 120B that has been identified as having committed that record 115 (although it may not have been committed if the transaction was aborted after the commit was approved). If that record request 204 times out and there is no record response 206, the worker node 120A may send the record request 204 to the assistant node 1030 of the worker node 120B. In some embodiments, the worker node 120A may send the record request 204 to the worker node 120B and its assistant node(s) 1030 in parallel. This may save time at the cost of sending more requests than may be needed.

[0062] In response to receiving the record request 204, in various embodiments, the Assistant node 1030 determines whether the worker node 120B committed the requested record 115. In some cases, the Assistant node 1030 determines that the requested record 115 has been committed (e.g., via a commit record written to the log 1045 of the worker node 120B). The Assistant node 1030 may access that record 115 from its own memory and return it to the worker node 120A in the record response 206. In some cases, the Assistant node 1030 determines that the requested record 115 does not exist and that the worker node 120B logged work past the time associated with the transaction that had the requested record 115. For example, the worker node 120B may abort the transaction rather than commit it and then commit another transaction associated with a later commit time; thus, the Assistant node 1030 may determine that work has been logged past the transaction that had the requested record 115. As a result, the assistant node 1030 may return a record response 206 to the worker node 120A indicating that the requested record 115 does not exist. In some cases, the assistant node 1030 has not progressed far enough in the log 1045 of the worker node 120B to know whether the requested record 115 has been committed. The assistant node 1030 may delay its response until it has fully read the log 1045 of the worker node 120B, or it may provide the worker node 120A with a record response 206 instructing the worker node 120A to retry the record request 204 later. In the former case, the assistant node 1030 may ensure that a record in the log 1045 of the worker node 120B having a timestamp indicating a time before the snapshot time specified in the record request 204 has been read.

[0063] In some cases, the assistant node 1030 determines that the log 1045 of worker node 120B has not reached the point of committing the transaction for the requested record 115, and that worker node 120B is not writing to its log 1045 or is writing at a very slow rate according to a defined threshold. Based on worker node 120B not updating its log 1045 and the requested record 115 not being committed, in various embodiments, the assistant node 1030 fences the log 1045 of worker node 120B and then returns a record response 206 to worker node 120A indicating that the requested record 115 does not exist. Because the assistant node 1030 received the record request 204 from worker node 120A, it is implied that worker node 120A has retrieved the history record 225 linked to the requested record 115. Because worker node 120A has its history record 225, it suggests that worker node 120B received permission to commit the transaction for requested record 115. In this case, to provide record response 206 indicating that requested record 115 has not been committed, assistant node 1030 fences worker node 120B's log 1045, preventing the requested record 115 from being committed after assistant node 1030 tells worker node 120A that the requested record 115 was not committed. To fence worker node 120B's log 1045, in various embodiments, assistant node 1030 sends a command to shared storage 1040 to cause shared storage 1040 to disable appending to worker node 120B's log 1045. As a result, worker node 120B cannot write log records and therefore cannot perform work for system 100. However, the contents of the log 1045 of the worker node 120B remain intact and can be read by other nodes (e.g., the assistant node 1030).After worker node 120B's log 1045 is fenced, assistant node 1030 may take a final pass at log 1045 to determine whether a commit has been added for the transaction of requested record 115. If the transaction has been committed, assistant node 1030 may return requested record 115 to worker node 120A; if not, assistant node 1030 may indicate to worker node 120A that requested record 115 has not been committed.

[0064] Referring now to FIG. 11 , a block diagram of a flush operation performed by a worker node 120 and an assistant node 1030 is shown. In the illustrated embodiment, there is a database 110, a worker node 120, and an assistant node 1030. As shown, both the worker node 120 and the assistant node 1030 contain the same record 115A. That is, they each have a record 115 that stores the same information (e.g., value) for the same database construct (e.g., database row). Also shown, the database 110 contains a log-structured merge tree (LSM tree) 1100 that receives the same identical record 115 from the worker node 120B and the assistant node 1030. The illustrated embodiment may be implemented differently than shown. As an example, there may be multiple assistant nodes 1030 that write the same record 115 as the worker node 120 to the database 110.

[0065] The log-structured merge tree 1100 is a data structure that, in various embodiments, stores files in an organized manner using a level-based scheme. The LSM tree 1100 may include two high-level components: an in-memory component implemented in the in-memory cache 210 of the worker nodes 120, and an on-disk component implemented in the database 110. In some cases, the in-memory component is considered separate from the LSM tree 1100. The worker nodes 120 may initially write records 115 to their in-memory caches 210. As their caches 210 fill, and / or at some point, the worker nodes 120 may flush the records 115 to the database 110. As part of flushing the records 115, in various embodiments, the worker nodes 120 write them to a new file at the top level of the LSM tree 1100. Over time, the records 115 of these files are merged downward through the levels of the LSM tree 1100 as newer files are written to the top level. Although the various techniques described with respect to FIG. 11 relate to LSM trees, such techniques can be applied to other types of database implementations where there are worker nodes that can write and commit records to the database.

[0066] The following discussion assumes that worker node 120 commits record 115A. In some cases, before worker node 120 flushes record 115A to LSM tree 1100, worker node 120 may experience a slowdown that causes worker node 120 to become catatonic and unable to flush record 115A in a timely manner. The efficiency of system 100 may depend on the rate at which committed records 115 are flushed to LSM tree 1100. Therefore, the catatonicity of worker node 120 may result in an undesirable slowdown of system 100. Thus, in various embodiments, both worker node 120 and assistant node 1030 flush the same record 115 to LSM tree 1100. As a result, while worker node 120 is unable to flush record 115A in a timely manner due to a slowdown, assistant node 1030 may independently flush record 115A to LSM tree 1100.

[0067] In various cases, the LSM tree 1100 stores a replica of the same record 115 as a result of both the worker node 120 and the assistant node 1030 flushing that record 115. As described above, a merge operation may be performed that moves the record 115 through levels of the LSM tree 1100. In various embodiments, the merge operation removes the replica so that only one copy of the record 115 is moved to the next level of the LSM tree 1100. Thus, the LSM tree 1100 may temporarily store a replica of the record 115 that is ultimately removed by a merge operation performed by the system 100.

[0068] 12, a flow diagram of method 1200 is shown. Method 1200 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1200 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1200 includes more or fewer steps than shown. For example, method 1200 may include a step in which a worker node sends a request (e.g., record request 204) to another worker node to retrieve a record (e.g., record 115).

[0069] Method 1200 begins at step 1210 with a worker node receiving a request to execute a transaction that includes writing a record. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions in the database system and multiple director nodes (e.g., director node 135) operable to ensure transaction integrity of the transactions. The worker nodes may write the records to the worker node's in-memory cache (e.g., in-memory cache 210).

[0070] In step 1220, the worker node issues requests (e.g., snapshot request 510, commit request 710, etc.) to some (e.g., a majority subset or all) of the multiple director nodes for information (e.g., history records 225) that facilitates execution of an action for the transaction. In some cases, the request for information may specify at least a record, and the action may correspond to committing the transaction. A given one of the director nodes may decide whether to approve committing the transaction based on whether the record conflicts with a history of approved records (e.g., history information 220) known to the given director node. The request to the director node may further specify a snapshot time for the transaction, and the given director node may check for conflicts between the record and only records in the history of approved records that were approved after the snapshot time. The history of approved records may include one or more records that have not been flushed from multiple worker nodes to persistent storage (e.g., database 110) shared among the multiple worker nodes. In response to sending an approval response to the worker node, a given director node may include the record in its history of approved records (e.g., by creating a history record 225 for the record). In various embodiments, a given director node determines whether to approve a transaction to commit without communicating with other director nodes among the multiple director nodes. A request to a director node may indicate a commit time (e.g., proposed commit time 705), and the given director node may process requests from worker nodes at that commit time.

[0071] In step 1230, the worker node proceeds to perform the operations for the transaction in response to receiving approval responses (e.g., approval snapshot response 520, approval commit response 720, etc.) from a majority of the multiple director nodes, where none of the received responses indicates disapproval of the transaction. In some cases, after determining that permission to commit the transaction has been granted, the worker node may commit the transaction and flush the record to persistent storage shared among the multiple worker nodes. In some cases, after determining that permission has been granted, the worker node may abort the transaction. The worker node may abort the transaction based on a determination that approval responses have not been received from a majority of the multiple director nodes. The worker node may abort the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on a conflict between the record and another record known to the at least one director node. The worker node may further abort the transaction in response to receiving a disapproval response from at least one of the director nodes indicating disapproval of committing the transaction based on the arrival of the commit request after the commit time.

[0072] 13, a flow diagram of method 1300 is shown. Method 1300 is one embodiment of a method performed by a director node (e.g., director node 135) of a database system (e.g., system 100) as part of ensuring transaction consistency within the database system. Method 1300 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1300 includes more or fewer steps than shown. For example, method 1200 may include a step in which a director node processes a snapshot request (e.g., snapshot request 510) received from a worker node (e.g., worker node 120).

[0073] Method 1300 begins at step 1310 with a director node receiving a request (e.g., commit request 710) from a worker node for permission to commit a transaction that includes a record (e.g., record 115). The worker node may be part of a database system that may include multiple worker nodes operable to execute transactions and multiple director nodes operable to ensure transaction integrity of those transactions. The received request may specify a record of the transaction.

[0074] In step 1320, the director node determines whether the record conflicts with a set of records (e.g., identified in history information 220) provided by another worker node of the plurality of worker nodes (e.g., as part of a commit request 710). The received request may specify a snapshot time (e.g., a previously approved snapshot time 505) and a commit time (e.g., a proposed commit time 705) associated with the transaction. As such, determining may include determining whether there are conflicting records in the set of records the director node identified as having been committed between the snapshot time and the commit time. Further, determining may be performed after a local clock (e.g., local clock 205) of the director node indicates a time equal to the commit time. In some cases, the set of records includes at least one record that is not included in another set of records stored by another director node of the plurality of director nodes.

[0075] In step 1330, based on the determination, the director node sends a response (e.g., commit response 720) to the worker node indicating whether permission to commit the transaction has been granted. In some embodiments, the multiple director nodes are not operable to communicate with each other regarding whether permission to commit the transaction has been granted. In some cases, the response indicates that permission to commit the transaction has been granted, and the director node stores historical information (e.g., history record 225) that identifies the record as committed regardless of whether the worker node commits the transaction. In some cases, the response indicates that permission to commit the transaction has not been granted, and the worker node aborts the transaction regardless of receiving approval to commit the transaction from other director nodes of the multiple director nodes. The response may provide the worker node with an indication (e.g., approval of proposed commit time 705) indicating when the worker node is authorized to commit the transaction.

[0076] 14, a flow diagram of method 1400 is shown. Method 1400 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1400 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1400 includes more or fewer steps than shown. For example, method 1400 may include a step in which a worker node sends a request (e.g., commit request 710) to a director node (e.g., director node 135) for permission to commit the transaction.

[0077] Method 1400 begins at step 1410 with a worker node receiving a request to execute a transaction that includes writing a first record (e.g., record 115). In some cases, the worker node receives a request to execute a read-only transaction that does not include writing a record. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions in the database system and multiple director nodes (e.g., director node 135) operable to ensure transaction integrity of the transactions.

[0078] In step 1420, the worker node issues a snapshot request (e.g., snapshot request 510) to some (e.g., a majority subset or all) of the director nodes, requesting history information (e.g., history information 220) describing the history of the approved record up to a specified time (e.g., proposed snapshot time 505). The specified time may be selected by the worker node, and the director node may delay processing the snapshot request until the specified time. The history information may specify, for a record in the history of the approved record, the position of the record (e.g., position 24) and the time the record was marked as committed (e.g., commit time).

[0079] In step 1430, the worker node proceeds to execute the transaction in response to receiving responses (e.g., snapshot response 520) from a majority of the plurality of director nodes. The combination of the information in the responses constitutes the requested historical information. Processing may include determining whether a second record exists that conflicts with the first record based on the requested historical information, and aborting the transaction in response to determining that a second conflicting record exists. The history of approved records known to a first one of the director nodes may differ from the history of approved records known to a second one of the director nodes. In some cases, none of the responses individually provides the requested historical information. The director nodes may not be operable to communicate with each other to share historical information.

[0080] The transaction may include a second record. The worker node may determine the location of the second record based on the requested history information and may attempt to access the second record from the location. In some cases, the location corresponds to central storage shared among multiple worker nodes. In some cases, the location corresponds to another worker node. In attempting to access the second record, the worker node may send a request for the second record (e.g., record request 204) to another worker node and may receive a response (e.g., record response 206) indicating that the second record was not committed.

[0081] 15, a flow diagram of method 1500 is shown. Method 1500 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1500 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1500 includes more or fewer steps than shown. For example, method 1500 may include a step in which a worker node sends a request (e.g., record request 204) to another worker node to retrieve a record (e.g., record 115).

[0082] Method 1500 begins with a worker node receiving a request to execute a transaction that includes writing a record at step 1510. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions in the database system and multiple director nodes (e.g., director node 135) operable to ensure transaction integrity of the transactions.

[0083] In step 1520, the worker node selects a proposed time (e.g., proposed snapshot time 505, proposed commit time 705, etc.) associated with performing an operation for the transaction (e.g., starting the transaction, committing the transaction, etc.). In various embodiments, the worker node determines a time delay in communicating with the director node. A given one of the time delays may be the delay between sending a given request from the worker node to a given one of the director nodes and receiving the given request at the given director node. Thus, the worker node may record a first time that the given request was sent to the given director node (e.g., send timestamp 910) and receive a response to the given request that identifies a second time that the given request was received at the given director node (e.g., arrival timestamp 920). A given time delay may be determined based on the first time and the second time, and a proposed time may be selected based on the time delay such that the authorization request is estimated to arrive at the director node before the proposed time according to the director node's respective clock (e.g., local clock 205). The proposed time may be selected based on a time identified by a worker node's clock. In various cases, the worker node's clock identifies a different time than the director node's clock, which identifies a different time than the clock of at least one other director node among the multiple director nodes.

[0084] In step 1530, the worker node issues an approval request to perform the operation to some of the director nodes (e.g., a majority subset or all). In various embodiments, one of the director nodes is operable to process the approval request when the proposed time arrives according to the director node's clock, which identifies the time observed by the director node. In some cases, the operation relates to initiating a transaction, and the director node is operable to return history information (e.g., history information 220) describing the history of approved records up to the proposed time. In some cases, the operation relates to committing a transaction, and the director node is operable to check for conflicts between the transaction's records and a set of approved records known to the director node that occurred before the proposed time. The worker node may include delay information in the approval request to allow a first one of the director nodes to attempt to align the time observed by the first director node with the time observed by a second one of the director nodes.

[0085] The worker node may receive a disapproval response from the director node indicating disapproval of the proposed time based on the request arriving at the director node after the proposed time according to the director node's clock. The proposed time may be selected to be a first amount of time after the current time identified by the worker node's clock. Consequently, based on receiving the disapproval response, the worker node may select another proposed time such that the another proposed time is a second amount of time after the current time identified by the worker node's clock, the second amount of time being greater than the first amount of time.

[0086] 16, a flow diagram of method 1600 is shown. Method 1600 is one embodiment of a method performed by a worker node (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1600 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1600 includes more or fewer steps than shown. For example, method 1600 may include a step in which a worker node sends a request (e.g., record request 204) to another worker node to retrieve a record (e.g., record 115).

[0087] Method 1600 begins at step 1610 with a worker node accessing delay information specifying a time delay for communicating with some of a plurality of director nodes of a database system operable to ensure transaction integrity of transactions of the database system. At step 1620, the worker node receives a request to execute a transaction.

[0088] In step 1630, the worker node selects a proposed snapshot time (e.g., proposed snapshot time 505) for the transaction based on the delay information and the worker node's clock (e.g., local clock 205). The proposed snapshot time may be selected to be a time determined according to the director node's clock to allow the authorization request to arrive at the director node before the proposed snapshot time.

[0089] In step 1640, the worker node issues a first approval request (e.g., snapshot request 510) for the proposed snapshot time to two or more of the plurality of director nodes. In various embodiments, one director node of the two or more director nodes is operable to process the first approval request when the proposed snapshot time arrives according to the director node's clock. Before issuing the first approval request, the worker node may issue a second approval request to the director node for the proposed time associated with another transaction. The worker node may record a first time (e.g., send timestamp 910) that the second approval request was issued to the director node according to the worker node's clock and receive a response to the second approval request from the director node identifying a second time (e.g., arrival timestamp 920) that the second approval request was received at the director node according to the director node's clock. The worker node's clock may identify a different time than the director node's clock, and the proposed snapshot time may be selected based on the first time and the second time.

[0090] 17, a flow diagram of method 1700 is shown. Method 1700 is one embodiment of a method performed by a director node (e.g., director node 135) of a database system (e.g., system 100) to attempt to align time observed by the director node with time observed by other director nodes. Method 1700 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1700 includes more or fewer steps than shown. For example, method 1700 may include a step in which the director node provides a response to a worker node (e.g., worker node 120) that includes historical information (e.g., historical information 220).

[0091] Method 1700 begins at step 1710 with a director node maintaining a clock (e.g., local clock 205) that identifies the time observed by the director node. The director node may perform database operations at designated times.

[0092] In step 1720, the director node receives a request (e.g., snapshot request 510) from one worker node of a plurality of worker nodes operable to process a transaction for information (e.g., historical information 220) that facilitates execution of the transaction associated with the worker node. The request may provide a first indication of a time observed by other director nodes of the plurality of director nodes. In various embodiments, the request specifies a time delay observed by the worker node when communicating with the plurality of director nodes. The director node may aggregate this time delay with time delays provided by the other worker nodes and shift its clock based on the aggregated time delay. A given one of the time delays may be the delay between sending a given request from the worker node to a given director node of the plurality of director nodes and receiving the given request by the given director node. The time delay may be derived based on the worker node's clock, which identifies the time observed by the worker node. In some cases, the request specifies a proposed time associated with the transaction (e.g., proposed snapshot time 505). The director node may process the request when the proposed time arrives according to the director node's clock. In various cases, the director node's clock identifies a different time than the clock of another director node among the multiple director nodes. Thus, the request may be processed at the director node before the request is processed at another director node.

[0093] In step 1730, the director, based on the first indication, attempts to shift its clock to align the time observed by the director node with the time observed by the other director nodes. Based on the time delay, the director node may determine that there is a greater time delay when the worker node communicates with the director node than when the worker node communicates with a majority of the other director nodes. Thus, shifting may include slowing down the clock. Based on the time delay, the director node may determine that there is a lesser time delay when the worker node communicates with the director node than when the worker node communicates with a majority of the other director nodes. Thus, shifting includes speeding up the clock. In some cases, the director node may identify the arrival time of the request according to its clock (e.g., arrival timestamp 920) and send a response to the worker node specifying the arrival time. The worker node may be operable to identify a time delay between sending the request and the director node receiving the request. In some cases, shifting the clock causes the director node to skip processing one or more requests received at the director node. Shifting the clock can be performed without directly communicating with other director nodes.

[0094] 18 , a flow diagram of method 1800 is shown. Method 1800 is one embodiment of a method performed by a director node (e.g., director node 135) of a database system (e.g., system 100) to attempt to align times observed by the director node. Method 1800 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1800 includes more or fewer steps than shown. For example, method 1800 may include a step in which the director node provides a response to a worker node (e.g., worker node 120) that includes historical information (e.g., historical information 220).

[0095] Method 1800 begins at step 1810 with director nodes maintaining respective clocks (e.g., local clock 205) that identify respective times observed by the director nodes. A given director node may perform database operations at designated times. At step 1820, the director nodes receive timing information from worker nodes of the database system. The timing information may indicate communication time delays between the worker nodes and the director node.

[0096] In step 1830, the director nodes attempt to shift their respective clocks based on the timing information to align the respective times observed by the director nodes. In various embodiments, the shift is performed without some of the director nodes communicating with other director nodes. A director node may receive a request (e.g., commit request 710) from a worker node for permission to commit a transaction, and the request may specify a proposed commit time (e.g., proposed commit time 705). A given director node may process the request if its respective clock specifies a time that is not earlier than the proposed commit time. A given director node may send a response (e.g., commit response 720) to a worker node indicating the arrival time of the request at the given director node (e.g., arrival timestamp 920) according to the given director node's respective clock. The worker node may determine the time delay for the worker node to communicate with a given director node based on the arrival time. In various cases, a request received from a worker node is processed by a given director node independently of the processing of the request by another one of the director nodes.

[0097] 19, a flow diagram of method 1900 is shown. Method 1900 is one embodiment of a method performed by an assistant node (e.g., assistant node 1030) of a database system (e.g., system 100) that observes work performed by a worker node (e.g., worker node 120). Method 1900 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1900 includes more or fewer steps than shown. For example, method 1900 may include a step in which the assistant node flushes a record (e.g., record 115) to a database (e.g., database 110) that was also flushed by the worker node.

[0098] Method 1900 begins at step 1910 with the worker node observing work performed by a first worker node of a database system including a plurality of worker nodes operable to execute transactions for the database system. The observing may include storing, at an assistant node, records associated with the first worker node. The assistant node may read a transaction log (e.g., log 1045) of the first worker node stored in a data store (e.g., shared storage 1040) shared between the first worker node and the assistant node. The records stored on the assistant node may be determined from the transaction log. In some cases, the assistant node is one of a plurality of worker nodes operable to execute transactions for the database system. In various embodiments, the assistant node and the first worker node are executed using different virtual machines.

[0099] In step 1920, the assistant node receives a request for a record associated with the first worker node (e.g., record request 204) from a second worker node of the plurality of worker nodes. The request may specify a snapshot time corresponding to a transaction of the second worker node. Before making a determination as to whether the first worker node committed the record, the assistant node may ensure that a record in the transaction log having a timestamp indicating a time prior to the snapshot time has been read by the assistant node. In various embodiments, the second worker node is operable to send a request for the record to the assistant node in response to not receiving a response from the first worker node to the request sent to the first worker node for the record.

[0100] In step 1930, the assistant node makes a determination as to whether the first worker node committed the record. Making the determination may include determining that the record is not committed based on the transaction log and that the first worker node is not responding. Based on that determination, in various embodiments, the assistant node fences the transaction log to prevent the first worker node from modifying the transaction log to commit the record, and then sends a response (e.g., record response 206) to the second worker node indicating that the record is not committed. Making the determination may include determining that the first worker node committed a transaction including the record based on the transaction log. Thus, the assistant node may access the record from records stored on the assistant node, and the response to the second worker node may include the accessed record. Making the determination may include determining that the record is not committed based on the transaction log indicating that the first worker node logged work after the commit time associated with the transaction including the record and the transaction log not indicating the record. Therefore, the response to the second worker node may indicate that the record has not been committed.

[0101] In step 1940, the assistant node sends a response to the second worker node indicating whether the record was committed by the first worker node. Based on a determination that the record was committed, the assistant node may flush the record to a log-structured merge tree (LSM tree) of the database system (e.g., LSM 1100). The first worker node may also flush the record to the LSM tree so that there are two instances of the record in the LSM tree over a period of time.

[0102] Referring now to FIG. 20 , a flow diagram of method 2000 is shown. Method 2000 is one embodiment of a method performed by an assistant node (e.g., assistant node 1030) of a database system (e.g., system 100) that observes work performed by a worker node (e.g., worker node 120). Method 2000 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 2000 includes more or fewer steps than shown. For example, method 2000 may include a step in which the assistant node flushes a record (e.g., record 115) to a database (e.g., database 110) that was also flushed by the worker node.

[0103] Method 2000 begins in step 2010 with the assistant reading the transaction log (e.g., log 1045) of the first worker node. In various embodiments, the transaction log specifies a set of records committed by the first worker node. Based on the reading of the transaction log, the assistant node may cache one or more of the set of records at the assistant node. In step 2020, the assistant node receives a request (e.g., record request 204) for records associated with the first worker node from a second worker node.

[0104] In step 2030, based on reading the transaction log, the assistant node determines whether the record has been committed. The request received from the second worker node may identify a snapshot timestamp. The assistant node may determine that the first worker node has not updated the transaction log and that the first worker node has not written a record to the transaction log having a timestamp indicating a time later than the time of the snapshot timestamp. Based on this determination, the assistant node may prevent the first worker node from updating the transaction log. A response to the second worker node may indicate that the record has not been committed. In response to determining that the record has been committed, the assistant node may access the record from one or more cached records. The response may include the record.

[0105] In step 2040, the assistant node sends a response to the second worker node indicating whether the record has been committed. In response to determining that the record has been committed, the assistant node may flush the record to a database of the database system (e.g., database 110) regardless of whether the first worker node flushes the record to the database. Exemplary Multi-Tenant Database System

[0106] Referring now to FIG. 21 , an exemplary multi-tenant database system (MTS) 2100 in which various techniques of the present disclosure can be implemented is illustrated; for example, system 100 can be the MTS 2100. In FIG. 21 , the MTS 2100 includes a database platform 2110, an application platform 2120, and a network interface 2130 connected to a network 2140. Also shown, the database platform 2110 includes a data storage 2112 and a set of database servers 2114A-N that interact with the data storage 2112, and the application platform 2120 includes a set of application servers 2122A-N having respective environments 2124. In the illustrated embodiment, the MTS 2100 is connected to various user systems 2150A-N via the network 2140. The disclosed multi-tenant system is included for illustrative purposes and is not intended to limit the scope of the present disclosure. In other embodiments, the techniques of the present disclosure are implemented in a non-multi-tenant environment, such as a client / server environment, a cloud computing environment, or a clustered computer.

[0107] The MTS 2100, in various embodiments, is a set of computer systems that together provide various services to users (alternatively referred to as "tenants") that interact with the MTS 2100. In some embodiments, the MTS 2100 implements a customer relationship management (CRM) system that provides a mechanism for tenants (e.g., businesses, government agencies, etc.) to manage relationships and interactions with customers and potential customers. For example, the MTS 2100 may enable tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), identify sales opportunities, log service issues, and manage marketing campaigns. Additionally, the MTS 2100 may enable these tenants to identify how customers communicated with them, what they purchased, when they last purchased an item, and how much they paid. To provide the services of the CRM system and / or other services, the MTS 2100 includes a database platform 2110 and an application platform 2120, as shown.

[0108] Database platform 2110, in various embodiments, is a combination of software routines and hardware elements that implement database services for storing and managing data for MTS 2100, including tenant data. As shown, database platform 2110 includes data storage 2112, which, in various embodiments, includes a set of storage devices (e.g., solid-state drives, hard disk drives, etc.) connected to each other over a network (e.g., a storage-attached network (SAN)) and configured to store data redundantly to prevent data loss. In various embodiments, data storage 2112 is used to implement a database (e.g., database 110) that includes a collection of information organized in a manner that allows the information to be accessed, stored, and manipulated. Data storage 2112 may implement a single database, a distributed database, a collection of distributed databases, a database with redundant online or offline backups or other redundancies, etc. As part of implementing a database, data storage 2112 may store files containing one or more database records with their respective data payloads (e.g., values ​​of fields in a database table) and metadata (e.g., key values, timestamps, table identifiers for tables associated with the record, tenant identifiers for tenants associated with the record, etc.).

[0109] In various embodiments, a database record may correspond to a row in a table. A table generally includes one or more data categories logically arranged as columns or fields in a viewable schema. Thus, each record in a table may include an instance of data in each category defined by a field. For example, a database may include a table describing customers with fields for basic contact information such as name, address, phone number, and fax number. Thus, a record in that table may include a value for each of the fields in the table (e.g., a name for a name field). Another table may describe a purchase order, including fields for information such as customer, product, sales price, and date. In various embodiments, standard entity tables, such as tables for account, contact, lead, and opportunity data, are provided for use by all tenants, each containing predefined fields. The MTS 2100 may store database records for one or more tenants in the same table; i.e., tenants may share tables. Thus, in various embodiments, a database record includes a tenant identifier indicating the owner of the database record. As a result, one tenant's data is kept secure and isolated from other tenants' data, so one tenant cannot access another tenant's data unless the data is explicitly shared.

[0110] In some embodiments, data stored in data storage 2112 is organized as part of a log-structured merge tree (LSM tree, e.g., LSM tree 1100). An LSM tree typically includes two high-level components: an in-memory buffer and persistent storage. During operation, database server 2114 may first write database records to a local in-memory buffer and then flush these records to persistent storage (e.g., data storage 2112). As part of flushing the database records, database server 2114 may write the database records to a new file contained in the “top” level of the LSM tree. Over time, the database records may be rewritten by database server 2114 to new files contained in lower levels as the database records move down the levels of the LSM tree. In various implementations, as database records age and are moved down the LSM tree, they are moved to increasingly slower storage devices in data storage 2112 (e.g., from solid-state drives to hard disk drives).

[0111] When database server 2114 wants to access a database record for a particular key, it may traverse different levels of the LSM tree for files that potentially contain a database record for that particular key. If database server 2114 determines that a file may contain a related database record, database server 2114 may fetch the file from data storage 2112 into its memory. Database server 2114 may then check the fetched file for a database record with the particular key. In various embodiments, database records are immutable once written to data storage 2112. Thus, if database server 2114 wants to modify a value in a row of a table (which may be identified from the accessed database record), database server 2114 writes a new database record to the top level of the LSM tree. Over time, that database record is merged down the levels of the LSM tree. Thus, an LSM tree may store various database records for a database key, with older database records for that key located at a lower level in the LSM tree than newer database records.

[0112] The database server 2114, in various embodiments, is a hardware element, software routine, or combination thereof capable of providing database services such as data storage, data retrieval, and / or data manipulation. The database server 2114 may correspond to a worker node 120. Such database services may be provided by the database server 2114 to components within the MTS 2100 (e.g., application server 2122) and to components external to the MTS 2100. As an example, the database server 2114 may receive a database transaction request from the application server 2122 requesting to write or read data to the data storage 2112. The database transaction request may specify an SQL SELECT command to select one or more rows from one or more database tables. The contents of the row may be defined in a database record, and thus the database server 2114 may locate and return one or more database records corresponding to the selected table row or rows. In various cases, a database transaction request may instruct database server 2114 to write one or more database records for an LSM tree, which database server 2114 maintains implemented on database platform 2110. In some embodiments, database server 2114 implements a relational database management system (RDMS) or an object-oriented database management system (OODBMS) that facilitates storage and retrieval of information from data storage 2112. In various cases, database servers 2114 may communicate with each other to facilitate processing of transactions. For example, database server 2114A communicates with database server 2114N to determine whether database server 2114N has written a database record for a particular key to its in-memory buffer.

[0113] The application platform 2120, in various embodiments, is a combination of software routines and hardware elements that implements and runs the CRM software application, provides associated data, code, forms, web pages, and other information to and from user systems 2150, and stores associated data, objects, web page content, and other tenant information via the database platform 2110. To facilitate these services, in various embodiments, the application platform 2120 communicates with the database platform 2110 to store, access, and manipulate data. In some cases, the application platform 2120 may communicate with the database platform 2110 via different network connections. For example, one application server 2122 may be coupled via a local area network, while another application server 2122 may be coupled via a direct network link. TCP / IP (Transfer Control Protocol and Internet Protocol) is an exemplary protocol for communicating between the application platform 2120 and the database platform 2110, although it will be apparent to those skilled in the art that other transport protocols may be used depending on the network interconnect used.

[0114] The application server 2122, in various embodiments, is a hardware element, software routine, or combination thereof capable of providing the services of the application platform 2120, including processing requests received from tenants of the MTS 2100. The application server 2122, in various embodiments, may spawn environments 2124 that can be used for various purposes, such as providing functionality for developers to develop, execute, and manage applications (e.g., business logic). Data may be transferred to an environment 2124 from another environment 2124 and / or the database platform 2110. In some cases, an environment 2124 cannot access data from other environments 2124 unless such data is explicitly shared. In some embodiments, multiple environments 2124 can be associated with a single tenant.

[0115] Application platform 2120 may provide user systems 2150 with access to multiple different hosted applications (standard and / or custom), including a CRM application and / or applications developed by tenants. In various embodiments, application platform 2120 may manage application creation, application testing, storing applications in database objects in data storage 2112, running applications in environments 2124 (e.g., virtual machines in process space), or any combination thereof. In some embodiments, application platform 2120 may add and remove application servers 2122 from the server pool at any time for any reason, and there may not be server affinity of users and / or organizations to particular application servers 2122. In some embodiments, an interface system (not shown) implementing load balancing functionality (e.g., an F5 Big-IP load balancer) sits between application servers 2122 and user systems 2150 and is configured to allocate requests to application servers 2122. In some embodiments, the load balancer routes user requests to application servers 2122 using a least-connection algorithm. Other examples of load balancing algorithms such as round robin and observed response time may also be used. For example, in certain embodiments, three consecutive requests from the same user may hit three different servers 2122, and three requests from different users may hit the same server 2122.

[0116] In some embodiments, the MTS 2100 provides security mechanisms, such as encryption, to keep each tenant's data separate unless the data is shared. When two or more servers 2114 or 2122 are used, they may be located in close proximity to each other (e.g., in a server farm located in a single building or campus) or may be distributed at distances from each other (e.g., one or more servers 2114 located in city A and one or more servers 2122 located in city B). Thus, the MTS 2100 may include one or more logically and / or physically connected servers, either locally or distributed across one or more geographic locations.

[0117] One or more users (e.g., via user systems 2150) may interact with the MTS 2100 over the network 2140. The user systems 2150 may correspond, for example, to tenants of the MTS 2100, the provider (e.g., administrator) of the MTS 2100, or a third party. Each user system 2150 may be a desktop personal computer, a workstation, a laptop, a PDA, a mobile phone, or any Wireless Access Protocol (WAP)-enabled device, or any other computing device capable of interfacing directly or indirectly to the Internet or other network connection. The user systems 2150 may include dedicated hardware configured to interface with the MTS 2100 over the network 2140. The user systems 2150 may run a graphical user interface (GUI) corresponding to the MTS 2100, an HTTP client (e.g., a browsing program such as Microsoft's Internet Explorer® browser, Netscape's Navigator® browser, Opera browser, or a WAP-enabled browser for a mobile phone, PDA, or other wireless device), or both, allowing users of the user systems 2150 (e.g., subscribers to the CRM system) to access, process, and view information and pages available from the MTS 2100 over the network 2140. Each user system 2150 may include one or more user interface devices, such as a keyboard, mouse, touch screen, pen, etc., for interacting with the graphical user interface (GUI) provided by the browser on a display monitor screen, LCD display, etc., along with pages, forms, and other information provided by the MTS 2100 or other systems or servers. As noted above, the disclosed embodiments are suitable for use with the Internet, which refers to a specific global internetwork of networks.However, it should be understood that other networks may be used in place of the Internet, such as an intranet, an extranet, a virtual private network (VPN), a non-TCP / IP-based network, any LAN or WAN, etc.

[0118] Because users of a user system 2150 may be users of different capabilities, the capabilities of a particular user system 2150 may be determined by one or more permission levels associated with the current user. For example, if a salesperson is using a particular user system 2150 to interact with the MTS 2100, the user system 2150 may have the capabilities (e.g., user privileges) assigned to the salesperson. However, if an administrator is using the same user system 2150 to interact with the MTS 2100, the user system 2150 may have the capabilities (e.g., administrative privileges) assigned to the administrator. In a system with a hierarchical role model, users with a certain permission level may access applications, data, and database information accessible to users with lower permission levels, but may not have access to certain applications, database information, and data accessible to users with higher permission levels. Thus, different users may have different capabilities with respect to accessing and modifying application and database information depending on the user's security or permission level. Some data structures managed by the MTS 2100 are allocated at the tenant level, while others are managed at the user level.

[0119] In some embodiments, the user system 2150 and its components are configurable using an application, such as a browser, that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 2100 (and additional instances of the MTS, if more than one is present) and their components are configurable by an operator using application(s) that include computer code executable on the processing elements. Accordingly, various operations described herein may be performed by executing program instructions that are stored on a non-transitory computer-readable medium and executed by the processing elements. The program instructions may be stored on a non-volatile medium such as a hard disk, or on other volatile or non-volatile memory media or devices such as ROM or RAM, as is well known, or provided on any medium capable of launching program code, such as a compact disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, or the like. Additionally, the entire program code, or portions thereof, may be transmitted and downloaded from a software source, for example, via the Internet or from another server, as is well known, or may be transmitted via any other conventional network connection, as is well known (e.g., extranet, VPN, LAN, etc.), using any communication medium and protocol, as is well known (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.). It will also be understood that computer code for implementing aspects of the disclosed embodiments may be implemented in any programming language that can be executed on a server or server system, for example, C, C++, HTML, Java, JavaScript, or any other scripting language such as VBScript.

[0120] Network 2140 may be a LAN (Local Area Network), a WAN (Wide Area Network), a wireless network, a point-to-point network, a star network, a token ring network, a hubbed network, or any other suitable configuration. The global internetwork of networks is often referred to as the "Internet" with a capital "I" and is an example of a TCP / IP (Transfer Control Protocol and Internet Protocol) network. However, it should be understood that the disclosed embodiments may utilize any of a variety of other types of networks.

[0121] The user system 2150 may communicate with the MTS 2100 using TCP / IP, and at higher network levels, may communicate using other common Internet protocols such as HTTP, FTP, AFS, WAP, etc. For example, if HTTP is used, the user system 2150 may include an HTTP client, commonly referred to as a "browser," for sending and receiving HTTP messages from an HTTP server in the MTS 2100. Such a server may be implemented as the sole network interface between the MTS 2100 and the network 2140, although other techniques may also or alternatively be used. In some implementations, the interface between the MTS 2100 and the network 2140 includes a load-sharing function, such as a round-robin HTTP request distributor, to balance the load and distribute incoming HTTP requests evenly across multiple servers.

[0122] In various embodiments, user system 2150 communicates with application server 2122 to request and update system-level and tenant-level data from MTS 2100, which may require one or more queries to data storage 2112. In some embodiments, MTS 2100 automatically generates one or more SQL statements (SQL queries) designed to access the desired information. In some cases, user system 2150 may generate requests having a specific format corresponding to at least a portion of MTS 2100. As an example, user system 2150 may request that data objects be moved to a particular environment using an object notation that describes the object-relational mapping (e.g., JavaScript object notation mapping) of specified objects. Exemplary Computer System

[0123] 22, a block diagram of an exemplary computer system 2200 is shown that may implement system 100, database 110, worker node 120, director node 135, assistant node 1030, MTS 2100, and / or user system 2150. Computer system 2200 includes a processor subsystem 2280 coupled to system memory 2220 and I / O interface(s) 2240 via an interconnect 2260 (e.g., a system bus). I / O interface(s) 2240 are coupled to one or more I / O devices 2250. For convenience, FIG. 22 shows a single computer system 2200, but system 2200 may be implemented as two or more computer systems operating together.

[0124] Processor subsystem 2280 may include one or more processors or processing units. In various embodiments of computer system 2200, multiple instances of processor subsystem 2280 may be coupled to interconnect 2260. In various embodiments, processor subsystem 2280 (or each processor unit within 2280) may include a cache or other form of on-board memory.

[0125] System memory 2220 can be used to store program instructions executable by processor subsystem 2280 to cause system 2200 to perform various operations described herein. System memory 2220 can be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.). Memory in computer system 2200 is not limited to primary storage such as memory 2220. Rather, computer system 2200 can also include other forms of storage, such as cache memory in processor subsystem 2280 and secondary storage on I / O device 2250 (e.g., hard drive, storage array, etc.). In some embodiments, these other forms of storage can also store program instructions executable by processor subsystem 2280. In some embodiments, program instructions that, when executed, implement database application 200, in-memory cache 210, and / or orchestration application 300 may be included / stored within system memory 2220.

[0126] I / O interface 2240, according to various embodiments, can be any of a variety of types of interfaces configured to couple to and communicate with other devices. In one embodiment, I / O interface 2240 is a bridge chip (e.g., a southbridge) from the front side to one or more back side buses. I / O interface 2240 can be coupled to one or more I / O devices 2250 via one or more corresponding buses or other interfaces. Examples of I / O devices 2250 include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local or wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system 2200 is coupled to a network via network interface device(s) 2250 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0127] Implementations of the subject matter of the present application include, but are not limited to, the following Examples 1-20. [Example 1] 1. A method comprising: receiving, at a first worker node of a database system, a request to execute a transaction including writing a first record, the database system including a plurality of worker nodes operable to execute transactions of the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; issuing a snapshot request from the first worker node to some of the director nodes requesting history information describing a history of approved records up to a specified time; the first worker node proceeding with execution of the transaction in response to receiving responses from a majority of the plurality of director nodes, the combination of information in the responses constituting the requested history information; A method comprising: [Example 2] The processing step includes: determining whether a second record exists that conflicts with the first record based on the requested historical information; in response to determining that a conflicting second record exists, aborting the transaction; The method of Example 1, comprising: [Example 3] The history information includes, for a record of the history of the approved record, The record, the location of said record, and The time the record was marked as committed The method of example 1, wherein [Example 4] The transaction includes a second record, and the method further comprises: the first worker node determining a location of the second record based on the requested history information; the first worker node attempting to access the second record from the location; The method of example 1, further comprising: [Example 5] 5. The method of example 4, wherein the location corresponds to a central storage shared among the plurality of worker nodes. [Example 6] 2. The method of embodiment 1, wherein a history of approved records known to a first one of the director nodes is different from a history of approved records known to a second one of the director nodes. [Example 7] 2. The method of example 1, wherein none of the responses individually provides the requested historical information. [Example 8] 2. The method of example 1, wherein the specified time is selected by the first worker node, and the director node is operable to delay processing of the snapshot request until the specified time. [Example 9] 2. The method of example 1, wherein the director nodes are not operable to communicate with each other to share historical information describing authorized records. [Example 10] 2. The method of embodiment 1, wherein the database system includes shared storage to which records are flushed from the plurality of worker nodes, and the history of acknowledged records describes records that have not been flushed from the plurality of worker nodes to the log-structured merge tree. [Example 11] A non-transitory computer-readable medium having stored thereon program instructions capable of causing a worker node of a database system to perform an operation, the operation comprising: receiving a request to execute a transaction including writing a first record, the database system including a plurality of worker nodes operable to execute the transaction in the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; issuing snapshot requests to some of the director nodes requesting history information describing a history of approved records up to a specified time; proceeding with execution of the transaction in response to receiving responses from a majority of the plurality of director nodes, the combination of information in the responses constituting the requested history information; Including, media. [Example 12] The step of advancing includes: determining whether a second record exists that conflicts with the first record based on the requested historical information; subsequent to determining that no conflicting second record exists, issuing a commit request to the director node requesting permission to commit the first record; The medium of Example 11, comprising: [Example 13] The step of advancing includes: determining, based on the historical information, the location of a second record associated with the same database key as the first record; attempting to access the second record from the location; The medium of Example 11, comprising: [Example 14] The location corresponds to another worker node, and the step of attempting to access the second record includes: sending a request for the second record to the other worker node; receiving a response indicating that the second record was not committed; The medium of Example 13, comprising [Example 15] The operation is Aborting the transaction based on a determination that a response has not been received from a majority of the plurality of director nodes. The medium of Example 11, further comprising: [Example 16] 1. A system comprising: at least one processor; a memory having stored therein program instructions executable by the at least one processor to cause the system to perform operations; and the operation comprises: receiving a request to perform a transaction that includes writing a set of records; issuing a snapshot request for a snapshot time and history information describing a history of acknowledged records up to said snapshot time to a plurality of director nodes operable to ensure transaction integrity of the transaction; proceeding to execute the transaction in response to receiving approval responses from at least a threshold number of director nodes of the plurality of director nodes, the combination of information in the responses constituting the requested history information; Including, the system. [Example 17] The above operation is including a proposed snapshot time in the snapshot request, the approval response indicating approval of the proposed snapshot time; 17. The system of Example 16, further comprising: [Example 18] 17. The system of Example 16, wherein the history information identifies, for a record of the set of records, a location of a most recent version corresponding to the record, and the operations further include accessing the most recent version from the location. [Example 19] The operation is Aborting the transaction in response to not receiving an acknowledgment response from at least a threshold number of director nodes of the plurality of director nodes. The system of example 16, further comprising: [Example 20] The operation is sending record information to some of the director nodes for updating the history information, the record information identifying the set of records; 17. The system of Example 16, further comprising:

[0128] The present disclosure includes references to "embodiments" that are non-limiting implementations of the disclosed concepts. References to "an embodiment," "one embodiment," "particular embodiment," "some embodiments," "various embodiments," etc. do not necessarily refer to the same embodiment. In addition to the specific embodiment described in detail, numerous possible embodiments are contemplated, including modifications or alternatives that fall within the spirit or scope of the present disclosure. Not all embodiments necessarily exhibit any or all of the potential advantages described herein.

[0129] This disclosure may discuss potential advantages that may result from the disclosed embodiments. Not all implementations of these embodiments necessarily manifest any or all of the potential advantages. Whether advantages are realized for a particular implementation depends on many factors, some of which are outside the scope of this disclosure. Indeed, there are several reasons why an embodiment within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuitry outside the scope of this disclosure that, in conjunction with one of the disclosed embodiments, negates or reduces one or more of the disclosed advantages. Furthermore, suboptimal design practices of a particular implementation (e.g., implementation techniques or tools) may also negate or reduce the disclosed advantages. Even assuming skilled implementation, realization of advantages may still depend on other factors, such as the environmental conditions in which the implementation is deployed. For example, inputs provided to a particular implementation may prevent one or more problems addressed in this disclosure from occurring in a particular case, resulting in the benefits of that solution not being realized. Given the possible existence of factors external to this disclosure, it is expressly intended that any potential advantages described herein should not be construed as limitations on the scope of the claims that must be met in order to demonstrate infringement. Rather, the identification of such potential advantages is intended to illustrate the type(s) of improvement available to a designer having the benefit of this disclosure. The permissive description of such advantages (e.g., stating that a particular advantage "may result") is not intended to convey doubt as to whether such advantages are actually realizable, but rather to recognize the technological reality that realization of such advantages often depends on additional factors.

[0130] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of claims drafted based on this disclosure, even if only a single example of a particular feature is described. The disclosed embodiments are intended to be illustrative, not restrictive, unless stated to the contrary in the present disclosure. Accordingly, the application is intended to permit claims that encompass the disclosed embodiments, as well as alternatives, modifications, and equivalents that will be apparent to those skilled in the art having the benefit of this disclosure.

[0131] For example, features in this application may be combined in any suitable manner. Accordingly, during prosecution of this application (or an application claiming priority thereto), new claims may be formulated for such combinations of features. In particular, with reference to the appended claims, features from dependent claims may be combined, as appropriate, with features of other dependent claims, including claims that are dependent on other independent claims. Similarly, features from each independent claim may be combined, as appropriate.

[0132] Thus, the accompanying dependent claims may each be drafted so as to depend on a single other claim, although additional dependencies are also contemplated. Any combination of features in the dependent claims consistent with this disclosure is contemplated and may be claimed in this or another application. In short, combinations are not limited to those specifically recited in the accompanying claims.

[0133] It is also contemplated that, where appropriate, claims drafted in one form or statutory type (e.g., apparatus) are intended to support corresponding claims in another form or statutory type (e.g., method).

[0134] Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. It is hereby announced that the definitions provided in the following paragraphs, and throughout this disclosure, should be used in determining how to interpret the claims drafted based on this disclosure.

[0135] Reference to a singular item (i.e., a noun or noun phrase preceded by "a," "an," or "the") is intended to mean "one or more" unless the context clearly dictates otherwise. Thus, a reference to "an item" in a claim does not exclude additional instances of the item unless the context requires otherwise. A "plurality" of an item refers to a set of two or more of the items.

[0136] The word "may" is used herein in a permissive sense (i.e., having the potential to, being able to), not in a mandatory sense (i.e., must).

[0137] The terms "comprising" and "including" and their forms are open-ended and mean "including, but not limited to."

[0138] When the term "or" is used in this disclosure in reference to a list of alternatives, it will generally be understood to be used in an inclusive sense unless the context dictates otherwise. Thus, the statement "x or y" is equivalent to "x or y, or both," and thus encompasses 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, a phrase such as "either x or y, but not both" makes clear that "or" is used in an exclusive sense.

[0139] References to "w, x, y, or z, or any combination thereof" or "at least one of ... w, x, y, and z" are intended to encompass all possibilities, including a single element, up to the total number of elements in the set. For example, given the set [w, x, y, z], these phrases encompass any single element of the set (e.g., w but not x, y, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Thus, the phrase "at least one of ... w, x, y, and z" refers to at least one element of the set [w, x, y, z], thereby encompassing all possible combinations of this list of elements. This phrase should not be construed as requiring that there be at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0140] In this disclosure, various "labels" may precede nouns or noun phrases. Unless the context dictates otherwise, different labels used for a feature (e.g., "first circuit," "second circuit," "particular circuit," "given circuit," etc.) refer to different instances of the feature. Additionally, the labels "first," "second," and "third," when applied to features, do not imply any kind of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise stated.

[0141] The phrase "based on" is used to describe one or more factors that influence a decision. This term does not exclude the possibility that additional factors may influence the decision. That is, the decision may be based only on the specified factors, or on the specified factors and other unspecified factors. Consider the phrase "determining A based on B." This phrase specifies that B is a factor used to determine A or that influences the decision on A. This phrase does not exclude that the decision on A may also be based on some other factor, such as C. This phrase is also intended to encompass embodiments in which A is determined solely on B. As used herein, the phrase "based on" is synonymous with the phrase "based at least in part on."

[0142] The phrases "in response to" and "responsive to" describe one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may influence or otherwise trigger the effect, either in conjunction with or independently of the specified factors. That is, the effect may be responsive only to these factors, or to the specified factors and other unspecified factors. Consider the phrase "performing A in response to B." This phrase specifies that B is the factor that triggers the execution of A or triggers a particular result of A. This phrase does not exclude that performing A may also be performed in response to some other factor, such as C. This phrase also does not exclude that performing A may be performed in response to B and C jointly. This phrase is also intended to encompass embodiments in which A is performed only in response to B. As used herein, the phrase "responsive to" is synonymous with the phrase "responsive at least in part to." Similarly, the phrase "in response to" is synonymous with the expression "at least in part in response to."

[0143] Within this disclosure, different entities (which may be variously referred to as "units," "circuits," other components, etc.) may be described or claimed as being "configured" to perform one or more tasks or operations. This formulation—"entity configured to perform one or more tasks"—is used herein to refer to a structure (i.e., something physical). More specifically, this formulation is used to indicate that the structure is configured to perform one or more tasks during operation. A structure can be said to be "configured" to perform some task even if the structure is not currently operating. Thus, an entity described or listed as being "configured to perform" some task refers to something physical, such as a device, a circuit, or a system having a processor unit and a memory that stores executable program instructions to perform the task. This expression is not used herein to refer to something intangible.

[0144] In some cases, various units / circuits / components may be described herein as performing a set of tasks or operations, and it will be understood that those entities are "configured to" perform those tasks / operations, even if not specifically stated otherwise.

[0145] The term "configured to" is not intended to mean "configurable to." For example, an unprogrammed FPGA is not considered to be "configured" to perform a particular function. However, this unprogrammed FPGA may be "configurable" to perform that function. After appropriate programming, the FPGA can be said to be "configured" to perform a particular function.

[0146] For purposes of U.S. patent applications based on this disclosure, a statement in a claim that a structure is "configured to" perform one or more tasks is expressly intended not to invoke 35 U.S.C. §112(f) as to that claim element. If applicant wishes to invoke 35 U.S.C. §112(f) during prosecution of a U.S. patent application based on this disclosure, the claim element will be recited using the construction "means for" [performing a function].

[0147] Different "circuits" may be described in this disclosure. These circuits or "circuitry" comprise hardware that includes various types of circuit elements, such as combinational logic, clocked storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memories (e.g., random access memory, embedded dynamic random access memory), programmable logic arrays, etc. Circuitry may be custom designed or obtained from standard libraries. In various implementations, circuitry may include digital components, analog components, or a combination of both, as appropriate. Particular types of circuits may be generally referred to as "units" (e.g., decoding units, arithmetic logic units (ALUs), functional units, memory management units (MMUs), etc.). Such units may also be referred to as circuits or circuitry.

[0148] Thus, the disclosed circuits / units / components and other elements shown in the drawings and described herein include hardware elements such as those described in the previous paragraph. Often, the internal arrangement of hardware elements within a particular circuit can be specified by describing the function of that circuit. For example, a particular "decode unit" may be described as performing the function of "processing the opcode of an instruction and routing the instruction to one or more of a plurality of functional units," meaning that the decode unit is "configured to" perform this function. This specification of this function is sufficient to connote to one skilled in the computer arts a set of possible configurations of the circuit.

[0149] In various embodiments, as described in the previous paragraph, circuits, units, and other elements may be defined by the functions or operations they are configured to implement. The arrangement of such circuits / units / components relative to one another and the manner in which they interact form a microarchitecture definition of the hardware that is ultimately fabricated into an integrated circuit or programmed into an FPGA to form the physical implementation of the microarchitecture definition. Thus, a microarchitecture definition will be recognized by those skilled in the art as a structure from which many physical implementations can be derived, all of which are contained within the broader structure described by the microarchitecture definition. That is, a person skilled in the art presented with a microarchitecture definition provided in accordance with this disclosure may, without undue experimentation, apply ordinary techniques to implement the structure by coding a description of the circuits / units / components in a hardware description language (HDL) such as Verilog or VHDL. HDL descriptions are often expressed in a manner that appears functional. However, to those skilled in the art, this HDL description is the method used to translate the structure of the circuits, units, or components into the next level of implementation detail. Such HDL descriptions may take the form of behavioral code (which is typically not synthesizable), register transfer language (RTL) code (which, in contrast to behavioral code, is typically synthesizable), or structural code (e.g., a netlist specifying logic gates and their connectivity). The HDL description may then be synthesized against a library of cells designed for a given integrated circuit manufacturing technology and modified for timing, power, and other reasons to obtain a final design database, which is sent to a foundry to generate masks and ultimately manufacture the integrated circuit. Some hardware circuits, or portions thereof, may also be custom designed in a schematic editor and incorporated into the integrated circuit design along with the synthesized circuit configuration. An integrated circuit may include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.) and interconnections between the transistors and circuit elements.Some embodiments may implement multiple integrated circuits coupled together to implement a hardware circuit, and / or in some embodiments, discrete components may be used. Alternatively, an HDL design may be synthesized into a programmable logic array, such as a field programmable gate array (FPGA), and implemented in an FPGA. This separation between the design of a group of circuits and the subsequent low-level implementation of those circuits, as this process is performed at different stages of the circuit implementation process, generally results in a scenario where a circuit or logic designer never specifies a particular set of structures for the low-level implementation beyond describing what the circuit is configured to do.

[0150] The fact that many different low-level combinations of circuit elements can be used to realize the same circuit specification results in the existence of many equivalent structures for that circuit. As noted above, these low-level circuit implementations may vary with changes in manufacturing technology, the foundry selected to manufacture the integrated circuit, the library of cells provided for a particular project, etc. In many cases, the choices made by different design tools or methods to generate these different implementations may be arbitrary.

[0151] Furthermore, it is common for a single implementation of a particular functional specification of a circuit to include a large number of devices (e.g., millions of transistors) for a given embodiment. Thus, due to the vast amount of information, it would not be impractical to provide a complete enumeration of the low-level structures used to implement a single embodiment, let alone the vast array of equivalent possible implementations. For this reason, this disclosure describes the structure of a circuit using functional shorthand notations commonly adopted in the industry.

Claims

1. 1. A method comprising: receiving, at a first worker node of a database system, a request to execute a transaction including writing a first record, the database system including a plurality of worker nodes operable to execute transactions of the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; the first worker node issuing a snapshot request to some of the director nodes requesting history information describing a history of approved records up to a specified time; the first worker node proceeding with execution of the transaction in response to receiving responses from a majority of the plurality of director nodes, the combination of information in the responses constituting the requested history information; determining, based on the requested historical information, whether there exists a second record that conflicts with the first record, that is associated with the same database key as the first record, and that has already been approved within a specified time frame; in response to determining that a conflicting second record exists, aborting the transaction, a history of approved records known to a first one of the director nodes is different from a history of approved records known to a second one of the director nodes; method.

2. The history information includes, for a record of the history of the approved record, The record, the location of said record, and The time the record was marked as committed The method of claim 1 , wherein

3. The transaction includes a second record, and the method further comprises: the first worker node determining a location of the second record based on the requested history information; the first worker node attempting to access the second record from the location; The method of claim 1 further comprising:

4. The method of claim 3 , wherein the location corresponds to a central storage shared among the plurality of worker nodes.

5. The method of claim 1 , wherein none of the responses individually provides the requested historical information.

6. The method of claim 1 , wherein the specified time is selected by the first worker node, and the director node is operable to delay processing of the snapshot request until the specified time.

7. The method of claim 1 , wherein the director nodes are not operable to communicate with each other to share historical information describing approved records.

8. 2. The method of claim 1 , wherein the database system includes shared storage to which records are flushed from the plurality of worker nodes, and the history of acknowledged records describes records that have not been flushed from the plurality of worker nodes to a log-structured merge tree.

9. 1. A computer-readable medium having stored thereon program instructions capable of causing a worker node of a database system to perform an operation, the operation comprising: receiving a request to execute a transaction including writing a first record, the database system including a plurality of worker nodes operable to execute the transaction in the database system and a plurality of director nodes operable to ensure transaction integrity of the transaction; issuing snapshot requests to some of the director nodes requesting history information describing a history of approved records up to a specified time; proceeding with execution of the transaction in response to receiving responses from a majority of the plurality of director nodes, the combination of information in the responses constituting the requested history information; determining, based on the requested historical information, whether there exists a second record that conflicts with the first record, that is associated with the same database key as the first record, and that has already been approved within a specified time frame; and issuing a commit request to the director node for permission to commit the first record following a determination that no conflicting second record exists; The history of approved records known to a first one of the director nodes is different from the history of approved records known to a second one of the director nodes. Computer-readable medium.

10. The step of advancing includes: determining, based on the historical information, the location of a second record associated with the same database key as the first record; attempting to access said second record from said location; 10. The computer-readable medium of claim 9, comprising:

11. The location corresponds to another worker node, and the step of attempting to access the second record comprises: sending a request for the second record to the other worker node; receiving a response indicating that the second record was not committed; The computer-readable medium of claim 10, comprising:

12. 1. A system comprising: at least one processor; a memory having stored thereon program instructions executable by said at least one processor to perform the method of any one of claims 1 to 8; A system comprising:

Images