Guidance-based database system for transactional consistency

Abstract

Techniques involving a database system that includes worker nodes operable to execute transactions and director nodes operable to ensure transactional consistency of the transactions are disclosed. A worker node can receive a request to execute a transaction involving a write record. The worker node can then issue a request to director nodes of the database system for information to facilitate execution of the transaction operation. The director nodes can determine whether to approve the request based on whether the operation would cause transactional inconsistency in the database system. The worker node can advance execution of the transaction operation in response to receiving approval responses from a majority of the director nodes, where none of the received responses indicate disapproval of the transaction.

Description

Background Technology Technical Field

[0002] The published text generally deals with database systems, and more specifically with the deceleration of components that handle database systems.

[0003] Related technical descriptions

[0004] Companies typically employ modern database management systems, which enable users to store organized sets of information that can be efficiently accessed and manipulated. Historically, these systems have been deployed on high-end, expensive hardware maintained by data centers owned and operated by these companies. In most cases, the characteristics of this high-end hardware were predictable, enabling systems with low latency and fast response times. However, over time, more and more companies have migrated their systems to the cloud, thus relying on hardware provided by cloud providers. However, cloud providers have begun to shift towards using cheaper hardware in their cloud infrastructure, whose characteristics are less predictable than those of high-end hardware. Attached Figure Description

[0005] Figure 1 This is a block diagram illustrating exemplary elements of a system including worker nodes and mentor nodes according to some implementation schemes.

[0006] Figure 2 This is a block diagram illustrating exemplary elements of a worker node according to some implementation schemes.

[0007] Figure 3 This is a block diagram illustrating exemplary elements of a guide node according to some implementation schemes.

[0008] Figure 4 This is a block diagram illustrating exemplary elements of a historical data structure according to some implementation schemes.

[0009] Figure 5 This is a block diagram illustrating exemplary elements involving snapshot-related interactions between worker nodes and multiple mentor nodes according to some implementation schemes.

[0010] Figure 6 This is a block diagram illustrating an exemplary layout of historical information about different guide nodes according to some implementation schemes.

[0011] Figure 7 This is a block diagram illustrating exemplary elements of submission-related interactions involving worker nodes and multiple mentor nodes according to some implementation schemes.

[0012] Figure 8 This is a block diagram illustrating another exemplary layout of historical information about different mentor nodes according to some implementation schemes.

[0013] Figure 9 This is a block diagram illustrating exemplary elements related to alignment time between components in a system, according to some embodiments.

[0014] Figure 10 This is a block diagram illustrating exemplary elements of an assistant node according to some implementation schemes.

[0015] Figure 11 This is a block diagram illustrating exemplary elements of a refresh operation according to some implementation schemes, wherein worker nodes and assistant nodes refresh the same records to the database.

[0016] Figure 12 This is a flowchart illustrating an exemplary method associated with a worker node according to some implementation schemes, the worker node interacting with a mentor node to facilitate the execution of transaction operations.

[0017] Figure 13 This is a flowchart illustrating an exemplary method related to a mentor node for ensuring transaction consistency within a system, according to some implementation schemes.

[0018] Figure 14 This is a flowchart illustrating an exemplary method associated with a worker node according to some implementation schemes, the worker node interacting with a mentor node to obtain historical information describing the history of approved records.

[0019] Figure 15 This is a flowchart illustrating an exemplary method associated with a worker node according to some implementation schemes, wherein the worker node selects a proposed time associated with the operation of performing a transaction.

[0020] Figure 16 This is a flowchart illustrating an exemplary method associated with a worker node according to some implementation schemes, wherein the worker node selects a proposed snapshot time for a transaction.

[0021] Figure 17 This is a flowchart illustrating an exemplary method associated with a mentor node according to some implementation schemes, wherein the mentor node attempts to align the time observed by the mentor node with the time observed by other mentor nodes.

[0022] Figure 18 This is a flowchart illustrating an exemplary method related to mentor nodes according to some implementation schemes, which attempts to align the time observed by these mentor nodes.

[0023] Figure 19 This is a flowchart illustrating an exemplary method associated with an assistant node according to some implementation schemes, which observes the work performed by worker nodes.

[0024] Figure 20This is a flowchart illustrating another exemplary method associated with an assistant node according to some implementation schemes, which observes the work performed by worker nodes.

[0025] Figure 21 This is a block diagram illustrating the components of a multi-tenant system according to some implementation schemes.

[0026] Figure 22 This is a block diagram illustrating the elements of a computer system for implementing the various systems described in the public text, according to some implementation schemes. Detailed Implementation

[0027] As mentioned earlier, cloud providers are shifting towards using cheaper, and therefore less reliable, hardware in their cloud infrastructure. This has led to an increase in reports of servers experiencing random slowdowns. Such random slowdowns in servers (or other components) include both gray failures (where a server suddenly experiences a significant performance drop (e.g., more than 99%) and stray failures (where a server experiences a less noticeable slowdown, e.g., a 25% performance drop). When slowed, the server does the "correct" things, but slowly enough to avoid triggering a fault detector and causing problems for the entire system. For example, the server might be running at half its normal speed but still able to output work and respond to requests, albeit at a slower pace, so that the slower performance doesn't raise suspicion in the rest of the system.

[0028] Slowdowns affect both transactional and non-transactional systems, but methods for handling slowdowns in non-transactional systems are not applicable to transactional systems. A transaction is a set of commands logically grouped together as a unit of work. A transactional system is a system that handles transactions according to a set of guidelines that ensure transaction consistency. ACID (Atomicity, Consistency, Isolation, and Durability) is an implementation of these guidelines, designed to guarantee that a transaction either succeeds entirely or fails entirely, ensuring data consistency, ensuring transactions occur in isolation, and ensuring that once a transaction is committed, it remains in the system. Non-transactional systems are not bound by this set of guidelines for ensuring transaction consistency. One way to address slowdowns in non-transactional systems is to retry partial computational work when a time limit is reached—that is, after a period of time, the work is retried on another server where the slower server failed to produce a result. In some cases, computational work is pushed across multiple servers so that even if one server slows down, the others can still return the result of that work. This is an acceptable approach for non-transactional systems because their work is idempotent, meaning the same work can be executed multiple times without causing unexpected states in the system. As an example, retrieving and returning static web pages is an idempotent task. This contrasts with non-idempotent operations, such as incrementing a counter, which, if retried multiple times, may cause the counter to increment more than expected. Because transactional database systems adhere to the principle of ensuring transaction consistency, their operations are typically non-idempotent and usually guarantee that only one transaction occurs at a time. Therefore, attempting the same work on multiple servers is not feasible for transactional database systems. The published text specifically addresses how to protect transactional database systems from slowdowns that could cause various components of the system to decelerate.

[0029] Slowdowns can also cause problems when attempting to scale a transactional database system to have more database nodes (or "worker" nodes) executing transactions. As the number of database nodes in the system increases, the likelihood of transaction inconsistencies also increases. In one implementation, a director node is implemented to ensure transaction consistency in the transactional database system by checking for conflicts between transactions executed by worker nodes. However, a system using a single director node is prone to significant performance degradation if it slows down because a transaction cannot be completed without the director node's consent. Among other things, the published text further addresses how to ensure transaction consistency among worker nodes in the event of slowdowns in the transactional database system.

[0030] The published text describes techniques for implementing mechanisms to mitigate some or all of the slowdown effects on a system. In various implementations described below, the system includes a database and a set of “worker” nodes that perform transactions for the system involving writing records to the database. The system also includes a set of “mentor” nodes that help ensure transactional consistency between transactions performed by the worker nodes. Throughout transaction execution, worker nodes may interact with mentor nodes to ensure that the actions of a worker node do not interfere with the actions of other worker nodes within the system. When interacting with a set of mentor nodes, a worker node sends a request to the set of mentor nodes (e.g., some or all of them) and waits for an approving response from at least a majority of the mentor nodes in that set before proceeding. In some cases, mentor nodes check for conflicts between records committed by worker nodes and the history of approved records (i.e., records that have been approved for commit) known to the individual mentor nodes. Identified conflicts may cause mentor nodes to send a disapproving response to worker nodes. In other cases, mentor nodes check for issues in initiating transactions and may return approving responses that include information about previously approved records up to a specific time. If a worker node does not receive an approval response from a majority of the mentor nodes in that group, or receives a disapproval response, the worker node may abort the transaction.

[0031] These techniques can be advantageous because they ensure transactional consistency within the system while mitigating slowdown effects. Specifically, as discussed in more detail below, the system can be configured such that not all mentor nodes must respond, but rather a majority of mentor nodes must, to ensure transactional consistency. Therefore, one or more mentor nodes may inadvertently slow down due to slowdown without actually slowing down the progress of system transactions. Consider an embodiment where the system comprises a set of three mentor nodes, with approval responses from at least two mentor nodes (the majority) providing a transactional consistency guarantee. A worker node can send a commit request to all three mentor nodes to obtain permission to commit the transaction. Because the worker node sends a request to all three mentor nodes, and in this embodiment only two approval responses are needed to ensure transactional consistency, one mentor node may slow down, while the other two mentor nodes will respond. Therefore, the overall system will not slow down due to this slowdown.

[0032] The public text further describes the use of assistant nodes to mitigate some or all of the slowdown effects on worker nodes. In various implementations, a worker node is associated with one or more assistant nodes that observe the work performed by the worker node. Assistant nodes may track the worker node's transaction log and store records that match those written at the worker node. During operation, a first worker node (“read” node) may communicate with a second worker node (“write” node, whose work is observed by the assistant node) to attempt to obtain a record from the second worker node. In some cases, the write worker node may slow down and fail to provide a timely response. The read worker node may then communicate with the write worker node's assistant node to attempt to obtain the record. Because the assistant node observes the write worker node's work, it can provide a response on behalf of the write worker node. This response may include the record. These techniques can be advantageous because they can prevent the read worker node from slowing down (e.g., waiting for a response) through other worker nodes that might slow down (e.g., write worker nodes). The public text also discusses concepts related to alignment time between worker nodes and mentor nodes in the system. Exemplary applications of the techniques in the public text will now be discussed from the references. Figure 1 start.

[0033] Turn now Figure 1 A block diagram of system 100 is shown. System 100 includes a set of components that can be implemented via hardware or a combination of hardware and software. In the illustrated embodiment, system 100 includes a database 110, a worker cluster 119 with worker nodes 120, and a mentor cluster 130 with mentor nodes 135. Also as shown, database 110 and worker nodes 120 include records 115. In some embodiments, system 100 is implemented differently than shown. For example, system 100 may include assistant nodes that observe the work of worker nodes, although in some cases, a worker node may also act as an assistant node to another worker node.

[0034] In various implementations, system 100 implements a platform service (e.g., a customer relationship management (CRM) platform service) that allows users of the service to develop, run, and manage applications. System 100 may be a multi-tenant system providing various functionalities to users / tenants hosted by a multi-tenant system. Therefore, system 100 can execute software routines from various different users (e.g., the provider and tenants of system 100) and provide code, web pages, and other data to users, databases (e.g., database 110), and other entities of system 100. In various implementations, system 100 is implemented using cloud infrastructure provided by a cloud provider. Database 110, worker nodes 120, and mentor nodes 135 can thus execute on and utilize the available cloud resources (e.g., computing resources, storage resources, network resources, etc.) of the cloud infrastructure for their operation. As an example, worker node 120 may execute in a corresponding virtual environment hosted on server-based hardware included within a cloud provider's data center. However, in some implementations, system 100 is implemented using on-premises or private infrastructure instead of a public cloud.

[0035] In various implementations, database 110 is a collection of information organized in a manner that allows access, storage, and manipulation of information. Therefore, database 110 may include supporting software (e.g., storage nodes) that allows worker nodes 120 to perform operations (e.g., access, storage, etc.) on the information stored in database 110. In various implementations, database 110 is implemented using one or more storage devices connected together on a network (e.g., a storage attached network (SAN)) and configured to redundantly store information to prevent data loss. The storage devices can persistently store data, thus database 110 can act as a permanent storage device for system 100. Information written to database 110 by worker node 120 (e.g., record 115) can be accessed by other worker nodes 120. See reference... Figure 11 In more detail, record 115 can be stored in a merged tree (LSM tree) file of the log structure as part of the LSM tree implemented in database 110.

[0036] As shown in the figure, database 110 stores record 115. In various embodiments, record 115 is a key-value pair that includes data and a corresponding key that can be used to look up the record 115. As one example, record 115 may correspond to a data row in a database table and specify the value of one or more attributes / fields of that database table. In various embodiments, record 115 is immutable, so a new record 115 is written to update the underlying database construct (e.g., rows in a table). Therefore, a database construct can be associated with multiple records 115, each being a different version of that database construct. Those records 115 can be referred to as "record versions" of the database construct. As one example, a first record 115 (first record version) storing some values ​​of a database row can initially be written, followed by a second record 115 (second subsequent record version) updating one or more values ​​of the database row. Both records 115 can be accessed using the same key.

[0037] In various implementations, worker node 120 provides various database services, such as data storage, data retrieval, and / or data manipulation. In various implementations, worker node 120 is a set of software routines executable on hardware, while in some implementations, worker node 120 includes both hardware and software routines. Database services may be provided to other components within system 100 or to components outside system 100. For example, worker node 120 may receive transaction requests 125 from an application node (not shown) to execute database transactions. In various implementations, a database transaction is a logical unit of work (e.g., a specified set of database operations) to be performed relative to database 110. For example, processing a database transaction may include executing an SQL SELECT command to select one or more rows from one or more database tables. The content of the row may be specified in record 115, so worker node 120 may return one or more records 115 corresponding to one or more table rows. Executing a database transaction may include worker node 120 writing one or more records 115 to database 110. In various implementations, worker node 120 first writes record 115 to a local memory cache before flushing the record to database 110. Before committing and refreshing these records 115, worker nodes 120 may communicate with mentor nodes 135 to ensure that record 115 does not conflict with record writes performed by other worker nodes 120. As used herein, the phrase "commit transaction" (including committing records) is used according to its well-known meaning and refers to the process that makes changes made during a transaction preserved and visible outside the entity performing the transaction. Worker nodes 120 may also communicate with mentor nodes 135 when initiating a transaction to obtain information about the state of records stored within system 100 (e.g., the history of record 115 that may have been committed by other worker nodes 120 but not refreshed to be available at database 110). Reference Figure 2 The various components of worker node 120 are discussed in detail.

[0038] In various implementations, mentor node 135 facilitates the execution of transactions and helps ensure the transactional consistency of those transactions. In various implementations, mentor node 135 is a set of software routines executable on hardware, while in some implementations, mentor node 135 comprises both hardware and software routines. As shown, mentor nodes 135 form mentor cluster 130. The number of mentor nodes 135 within mentor cluster 130 can vary between different implementations. For example, mentor cluster 130 may include three mentor nodes 135, five mentor nodes 135, eight mentor nodes 135, etc. Although mentor nodes 135 are part of mentor cluster 130, in various implementations, mentor nodes 135 do not communicate with each other. Specifically, when worker node 120 sends a request to two or more mentor nodes 135, these mentor nodes 135 may not exchange information related to the request with each other. Therefore, mentor nodes 135 can process requests independently of the knowledge and perspectives of other mentor nodes 135, based on their own knowledge and perspectives. However, in some implementations, the guide nodes 135 communicate with each other to share certain information that can be used to complete the request (e.g., the corresponding history of approved records). (See reference) Figure 3 and Figure 4 The various components of the mentor node 135 are discussed in detail.

[0039] To process requests from worker node 120, in various implementations, mentor node 135 maintains information about the state of record 115 stored within worker node 120. When a transaction is initiated, worker node 120 may send a snapshot request to a set of mentor nodes 120 to obtain a snapshot time. In various implementations, the snapshot time identifies the state of system 100 at a specific point in time and can be used by the owner of the snapshot time (e.g., a transaction) to access the records available at that point in time. Reference Figure 2 The concept of snapshot time is discussed in more detail. Therefore, mentor node 135 can approve a snapshot time and then return a response to worker node 120. In many cases, the response includes information about the state of record 115, allowing worker node 120 to locate and access record 115 that may be relevant to the transaction. However, the historical information from mentor node 135 may be an incomplete history of the approved records. Consequently, in some cases, worker node 120 must obtain a response from a majority of the mentor cluster 130 to obtain the complete history. References Figure 5 and Figure 6 A more detailed discussion is provided regarding the exemplary layout of the historical information for mentor node 135 and the snapshot request process.

[0040] When a worker node 120 wishes to commit a transaction, it can send a commit request to a set of mentor nodes 135 for approval. In determining whether to approve the request, mentor nodes 135 can use their historical information to determine whether the requested action (e.g., commit permission) would lead to an inconsistent or undesirable state in system 100. If a conflict exists, mentor node 135 rejects the request, prompting worker node 120 to abort part or all of the database transaction. If there is no conflict from the mentor node's perspective, mentor node 135 approves the request and can update its historical information about the state of record 115 to reflect the approval. (Reference) Figure 7 and Figure 8 A more detailed discussion of the exemplary process for updating and submitting historical information is provided.

[0041] Turn now Figure 2 A block diagram of exemplary components of worker node 120 is shown. In the illustrated embodiment, worker node 120 includes a database application 200, a memory cache 210, and historical information 220. As further shown, database application 200 maintains a local clock 205; memory cache 210 stores records 115 associated with key 215; historical information 220 includes a set of historical records 225. In some embodiments, worker node 120 is implemented differently than shown. For example, worker node 120 may store a transaction log with logging. Although in some embodiments, the transaction log is stored in a shared memory among worker nodes 120, as shown in references... Figure 10 The subject of discussion.

[0042] In various implementations, database application 200 is an executable set of program instructions for managing database 110, which may include managing the LSM tree built around database 110. Therefore, database application 200 may receive requests to execute database transactions involving reading and / or writing records 115 of system 100. When a transaction request is received, database application 200 may initiate a database transaction based on the request, including obtaining a snapshot time for the database transaction. In various implementations, the snapshot time is a value indicating which records 115 worker nodes 120 can read. This value may be a time (e.g., a time window covering one millisecond) or a numerical value indicating a time period. That is, system 100 may increment an integer commit value over time, and the snapshot time may correspond to a value of that commit value. Worker node 120 may read records 115 whose commit values ​​are less than (or in various cases equal to) the snapshot time. For example, a snapshot time “445” (or a time “10:11:20”) may be assigned to a transaction. As a result, for this transaction, database application 220 can read record 115 with a commit value less than or equal to 445 (or 10:11:20). As discussed further below, worker node 120 can communicate with mentor node 135 to obtain approval for the snapshot time of the transaction.

[0043] In various implementations, local clock 205 identifies the local time observed by worker node 120. In some cases, local clock 205 is provided by the operating system running on worker node 120; in others, database application 200 maintains its own local clock 205 independently of how other applications running on worker node 120 observe time. In various implementations, database application 200 performs certain database operations based on the time of local clock 205. For example, as discussed further below, database application 200 can use clock 205 to select and propose a snapshot time (or commit time, both of which will be discussed in more detail below) to mentor node 135 for approval. In various implementations, both worker node 120 and mentor node 135 have their own view of the current time within system 100 based on their local clock 205. However, the local clocks 205 of worker node 120 and mentor node 135 may become out of sync with each other over time—that is, their local clocks 205 may identify different times. As a result, nodes may observe a time that is significantly different from that of other nodes in system 100. This can lead to undesirable effects, where a node (e.g., mentor node 135) processes requests significantly later than other nodes (e.g., other mentor nodes 135). See reference... Figure 9 In more detail, the nodes of System 100 could try to gradually align their times so that those nodes do not become too out of sync.

[0044] In various implementations, memory cache 210 is a buffer that stores data in the memory (e.g., random access memory) of worker node 120. HBase TM The memstore is one embodiment of memory cache 210. After sufficient approval is obtained regarding the snapshot time of a transaction, worker node 120 can begin reading and writing record 115 to process the transaction. In various implementations, worker node 120 initially writes record 115 to its memory cache 210 and later flushes the record 115 to the database 110 after the record is committed by worker node 120. In various cases, the most recent / latest record version of the row in the database table is found in record 115 stored in memory cache 210. However, in some implementations, record 115 written to worker node 120's memory cache 210 is not visible to other worker nodes 120. That is, other worker nodes 120 do not know what information is stored in memory cache 210 without being asked or informed. Therefore, in various implementations, worker node 120 obtains historical information 220 from mentor node 135, which identifies where a certain record 115 (e.g., the most recent record version of a database row) might be found. In this way, worker node 120 can use historical information 220 to identify which records 115 are stored in the memory cache 130 of other worker nodes 120.

[0045] In various implementations, historical information 220 includes a collection of historical records 225. In various implementations, historical records 225 include information identifying the location of record 115 in system 100 for a key 215 that can be used to find the record 115 corresponding to that historical record 225. In various implementations, historical information 220 captures different versions of different records (with different keys) and the same database construct (e.g., data rows) identified with the same key. Therefore, historical information 220 may include multiple historical records 225 of multiple versions of the same database construct committed at different times by different transactions using the same key 215. When searching for a record version of a database construct (e.g., the latest record 115 of a database row), worker node 120 can identify a set of historical records 225 based on the key 215 corresponding to the database construct. Worker node 120 can then identify from the timestamp which historical record 225 corresponds to the record version, and can then use the historical record 225 to attempt to access record 115. In some implementations, a single historical record 225 may store the locations of multiple records 115 with the same key 215.

[0046] In various implementations, when attempting to access record 115 from a location outside worker node 120, worker node 120 sends a record request 204 to that location. For example, as shown, worker node 120 sends record request 204 to database 110. Record request 204 may specify a key 215 corresponding to record 115, which the receiver can use to attempt to locate record 115. Worker node 120 may then receive a record response 206. In some cases, record response 206 includes the requested record 115. However, in some cases, record response 206 does not include the requested record 115, even if history record 225 indicates that record 115 is at that location. For example, mentor node 135 may create history record 225, which indicates that record 115 may have already been committed at worker node 120. However, after being granted permission to commit the transaction associated with record 115, worker node 120 may abort the transaction and delete record 115. Therefore, upon receiving a record request 204 for record 115, worker node 120 returns a record response 206 indicating that worker node 120 did not write record 115 at the time specified in the received record request 204.

[0047] After processing a transaction (e.g., after writing record 115 for all requests in that transaction), in various implementations, worker node 120 sends a commit request to a majority of mentor nodes 135 in mentor cluster 130. The commit request may include record 115 written for the transaction, allowing mentor nodes 135 to check for record conflicts. In response to receiving approval from a majority of mentor nodes 135, and no disapproval, worker node 120 may commit record 115. In some implementations, disapproval is allowed as long as a majority of mentor nodes in mentor cluster 130 approve the commit request. As part of the commit, in various implementations, worker node 120 marks each record 115 with a value indicating when to commit (e.g., time, a numerical value indicating a time period, etc.). Committed records 115 may be retained in memory cache 210 until a triggering event (e.g., memory cache 210 is nearing or full) prompts those records 115 to be flushed to database 110. Record 115 stored in memory cache 210 can be accessed and used by other transactions whose snapshot time / value is greater than the commit time / value of the record 115 being accessed.

[0048] Turn now Figure 3A block diagram of exemplary elements of a mentor node 135 is shown. In the illustrated embodiment, mentor node 135 includes an orchestration application 300 and historical information 220. As further shown, the orchestration application 300 maintains a local clock 205, and the historical information 220 includes historical records 225. In some embodiments, mentor node 135 is implemented differently than shown. For example, mentor node 135 may interact with other mentor nodes 135 in mentor cluster 130.

[0049] In various implementations, orchestration application 300 is a set of executable program instructions to facilitate the execution of transactions and ensure the transactional consistency of those transactions. To facilitate transaction execution, in various implementations, orchestration application 300 provides worker node 120 with part or all of its history information 220 so that worker node 120 can be aware of work performed by other worker nodes 120. As discussed, in various implementations, committing record 115 involves communicating with mentor node 135 to obtain approval for the commit of record 115. Because the mentor node 135 that approves the commit knows the record 115 that may be committed, its orchestration application 300 is able to store that approval instruction as part of history information 220. In some cases, orchestration application 300 may miss a commit request (or fail to process it), and therefore may not store a complete history of all recent records 115 that have been approved for commit by mentor cluster 130. As a result, in various implementations, history information 220 only identifies those records 115 that have been approved for commit by its orchestration application 300.

[0050] Because orchestration application 300 tracks records 115 that have been approved for commit, it can facilitate transaction execution by enabling worker nodes 120 to potentially locate records 115 associated with their transactions, including records 115 stored in memory cache 210. Furthermore, since orchestration application 300 tracks records 115 approved at its mentor node 135, in various implementations, transactional consistency between transactions can be helped to ensure that the requesting approval record 115 does not conflict with the approved record 115 identified in the mentor node's history information 220. Therefore, upon receiving a commit request for record 115, orchestration application 300 can determine whether another record 115 with the same key 215 (i.e., associated with the same database rebuild) has already been approved within a specific time frame. In various cases, this time frame corresponds to the time between a snapshot of the worker node's transaction and the current time observed by the mentor node 135. If a conflicting approved record 115 exists, the orchestration application 300 can return a disapproval response to the requesting worker node 120, preventing the requested record 115 from being committed and ensuring transaction consistency. If a conflict-free record 115 exists, the orchestration application 300 can return an approval response and add the historical record 225 to the historical information 220 corresponding to the newly approved record 115. Note that in some cases, when an earlier transaction is not actually approved by a majority of mentor nodes 135, mentor node 135 may detect a conflict with that earlier transaction. In this case, if a sufficient number of other mentor nodes 135 do approve the transaction, the transaction can be committed, and the conflict perceived at that mentor node 135 is not a true conflict.

[0051] As shown in the figure, the orchestration application 300 maintains a local clock 205, which identifies the current time as observed at the mentor node 135 (or at least by the orchestration application 300, which can be independent of the time observed by other applications executing on the mentor node 135). Similar to worker node 120, mentor nodes 135 can perform certain operations at specified times based on their respective local clocks 205. As an example, a submission request can be processed at a submission time proposed in the submission request. As mentioned above, the clocks 205 of mentor node 135 and worker node 120 may become out of sync, so mentor node 135 may process a request at a logical time much later than the other mentor node 120. Therefore, the nodes of system 100 can attempt to gradually align their times so that they do not become too out of sync, which will refer to... Figure 9 Let's have a more detailed discussion.

[0052] Turn now Figure 4A block diagram of an exemplary history record 225 is shown. In the illustrated embodiment, history record 225 specifies a record identifier 410, a location 420, and a submission time / period 430. In some embodiments, history record 225 is implemented differently than shown. For example, history record 225 may specify a mentor identifier for the mentor node 135 that wrote the history record 225.

[0053] In various implementations, record identifier 410 is information that identifies record 115 corresponding to history record 225. In various cases, record identifier 410 is a key 215 that can be used to look up the corresponding record 115. Record identifier 410 can be specified in a commit request sent from worker node 120 to mentor node 135 to approve the commit of a corresponding set of records 115. In response to the decision to approve the commit request, mentor node 135 can generate history record 225 for each record 115 in that set, with history record identifier 410 corresponding to that record and obtained from the commit request.

[0054] In various implementations, location 420 identifies the position of record 115 corresponding to history 225 within system 100. For example, location 420 may specify worker node 120 or database 110. In various cases, location 420 initially identifies the worker node 120 that has been granted permission to submit the corresponding record 115. If the record 115 is submitted and refreshed to database 110, location 420 may be updated by mentor node 135 (which created the corresponding history 225) to identify the position in database 110 where the record 115 is stored. Therefore, location 420 can be adjusted over time as the corresponding record 115 moves within system 100.

[0055] In various implementations, commit time 430 identifies the logical time (or time window) at which the relevant record 115 is being committed. In various implementations, worker node 120 may commit record 115 of a transaction with commit time 430, or may choose not to commit; however, worker node 120 may not commit the transaction at any other commit time 430 without re-requesting approval at a new commit time 430. In some cases, record 115 is actually committed at a physical time later than the logical time of commit time 430. Specifically, a commit request may specify a proposed commit time. Mentor node 135 may process the request at the proposed commit time, and if the request is approved, a history 225 may be generated showing that commit time 430 is the proposed commit time. Requesting worker node 120 may receive an approval response and may commit the requested record 115 with the proposed commit time 430 (if it chooses to do so).

[0056] Turn now Figure 5 A block diagram illustrating exemplary snapshot-related interactions involving worker node 120 and three mentor nodes 135A-135C is shown. In some embodiments, there may be more mentor nodes 135 than shown (e.g., five mentor nodes 135), and worker node 120 may issue snapshot requests 510 to fewer than all mentor nodes 135 within the mentor cluster 130 (e.g., four out of five).

[0057] As described above, worker node 120 can receive requests to execute transactions, which may include reading and writing records 115. This request can be received from an application node (not shown). As part of processing the transaction, in various embodiments, worker node 120 obtains a snapshot time corresponding to the state of system 100 at a specific point in time for the transaction. Records 115 committed before the snapshot time can be accessed by worker node 120 for the transaction, and the snapshot time can be used as a reference point for checking for record conflicts when the transaction is committed.

[0058] To ensure that transactions are assigned a single snapshot time, in various implementations, worker node 120 selects the proposed snapshot time 505 instead of mentor node 135. In various implementations, worker node 120 may choose a snapshot time instead of mentor node 135 because mentor nodes 135 may not communicate with each other or at least not transmit snapshot information. Because mentor nodes 135 may not communicate with each other in various implementations, it is possible for mentor node 135 to select a different snapshot time than another mentor node 135. Therefore, a transaction may be associated with multiple snapshot times, which could potentially compromise transaction guarantees. However, in some implementations, mentor node 135 may assign snapshot time 505 to a transaction, corresponding to the time when mentor cluster 130 sees the snapshot request 510 from worker node 120.

[0059] As described above, the mentor node 135 can process received requests at a specified time. Therefore, in various embodiments, when a proposed snapshot time 505 is selected, the worker node 120 selects a proposed snapshot time 505 such that, based on the mentor node 135's corresponding local clock 205, it is estimated that the corresponding snapshot request 510 will arrive at the mentor node 135 before the proposed snapshot time 505. If the snapshot request 510 arrives at the mentor node 135 earlier, the mentor node 135 may store the request and delay processing until the proposed snapshot time 505 identified in the snapshot request 510 is reached according to the mentor node 135's local clock 205. In various cases, the mentor node 135 processes received requests at the same local time but at different times from a global perspective. If the snapshot request 510 arrives at the mentor node 135 after the proposed snapshot time 505, the mentor node 135 may issue a snapshot response 520 indicating that the proposed snapshot time 505 has been rejected. In some implementations, worker node 120 sends a snapshot request 510 to mentor node 135 and allows mentor node to select a snapshot time 505 for the corresponding transaction.

[0060] In some implementations, in response to receiving a rejection of the proposed snapshot time 505, worker node 120 selects another proposed snapshot time 505, which is further from worker node 120's current time than the rejected snapshot time 505 is further from the time selected by worker node 120. (See reference...) Figure 9 In more detail, worker node 120 can track the latency of communication with mentor node 135 and use these latency to adjust the timing of selecting a proposed snapshot time 505 in the future. In some implementations, in response to receiving a rejection of the proposed snapshot time 505, and also receiving approval from at least a majority of mentor nodes 135 in cluster 130, worker node 120 proceeds with the execution of the transaction based on the proposed snapshot time 505. Thus, in some cases, if a particular mentor node 135 is reasonably behind other mentor nodes 135 in time, worker node 120 can select a proposed snapshot time 505 such that the estimated corresponding snapshot request 510 arrives at other mentor nodes 135 before the proposed snapshot time 505, even if the estimated snapshot request arrives at the particular mentor node 135 after the proposed snapshot time 505. Therefore, when choosing a proposed snapshot time 505, worker node 120 can consider a trade-off between reaching more mentor nodes 135 (which increases the chance of receiving a response from most mentor nodes) and choosing a proposed snapshot time 505 that is not too far in the future (which may result in delayed transactions).

[0061] In response to receiving a snapshot request 510 with a proposed snapshot time 505, in various implementations, mentor node 135 determines whether to reject the proposed snapshot time 505. Mentor node 135 may reject the proposed snapshot time if the proposed snapshot time 505 is after the current time at mentor node 135, or if it would cause a conflict with other transactions. In response to approving the proposed snapshot time 505, in various implementations, mentor node 135 returns a snapshot response 520 with historical information 220 of mentor nodes up to the proposed snapshot time 505. As shown, worker node 120 receives snapshot responses 520 from mentor nodes 135A-135B, but not from mentor node 135C. Assuming mentor cluster 130 only includes mentor nodes 135A-135C, and the received snapshot response 520 approves the proposed snapshot time 505, worker node 120 can proceed with processing the transaction because approval has been received from a majority of mentor nodes in mentor cluster 130, with no disapprovals.

[0062] Turn now Figure 6 A block diagram illustrating an exemplary layout of historical information for mentor nodes 135A-135C is shown. In the illustrated embodiment, a complete history 600 exists, comprising four historical records 225A-225D. As shown, mentor node 135A includes historical records 225A and 225C-225D; mentor node 135B includes historical records 225A-225B; and mentor node 135C includes historical records 225A-225D. In some cases, the complete history 600 may include more or fewer historical records 225 than shown, and the layout of historical information for mentor node 135 may differ.

[0063] exist Figure 5In this embodiment, worker node 120 receives snapshot responses 520 from mentor nodes 135A-135B, but not from mentor node 135C. As explained, snapshot response 520 may include historical information 220, which includes historical records 225 up to the proposed snapshot time 505. In this embodiment, assuming the proposed snapshot time 505 is after the submission time 430 of historical record 225D, worker node 120 receives historical records 225A and 225C-225D from mentor nodes 135A and historical records 225A-225B from mentor node 135B. In various implementations, since a majority of mentor nodes 135 must approve submissions, there is a historical record 225 with approved records 115, so if worker node 120 receives approvals from a majority of mentor nodes 135, worker node 120 is guaranteed to obtain the complete history 600. Continuing with this embodiment, worker node 120 obtains the complete history 600 from mentor nodes 135A-135B (the majority in this embodiment), even if neither the worker node nor the mentor nodes store the complete history 600. To obtain the complete history 600, worker node 120 can perform a union of the historical records 225 received from mentor nodes 135A-135B. If worker node 120 receives snapshot responses 520 from fewer than a majority of mentor nodes 135, there is no guarantee that worker node 120 will obtain the complete history 600. As an example, if worker node 120 receives snapshot responses 520 only from mentor node 135B, then worker node 120 obtains an incomplete history 600.

[0064] Turn now Figure 7 The diagram illustrates an exemplary commit-related interaction involving worker node 120 and three mentor nodes 135A-135C. In some implementations, there may be more mentor nodes 135 than shown (e.g., five mentor nodes 135), and worker node 120 may issue commit requests 710 to fewer than all mentor nodes 135 within the mentor cluster 130 (e.g., four out of five).

[0065] After processing a transaction, worker node 120 may decide to commit the transaction, including writing one or more records 115 to the worker node's memory cache 210 for the transaction. To ensure that record 115 does not conflict with records 115 written by other worker nodes 120, worker node 120 issues a commit request 710 to at least a majority of mentor nodes 135. As shown, worker node 120 issues commit request 710 to mentor nodes 135A-135C, which are all mentor nodes 135 in the illustrated embodiment. In various embodiments, commit request 710 identifies the records 115 written for the transaction (e.g., via record identifier 410), a proposed commit time 705 to be applied to those records 115, and a snapshot time approved for the transaction. Similar to the proposed snapshot time 505, the proposed commit time 705 may identify a time (or time window) or a numerical value indicating a period of time.

[0066] Similar to the selection of a snapshot time, in various implementations, worker node 120 may choose a commit time instead of mentor node 135, since mentor nodes 135 may not communicate with each other or at least not regarding commits. In various implementations, when selecting a proposed commit time 705, worker node 120 selects a proposed commit time 705 such that, based on the corresponding local clock 205 of mentor node 135, it is estimated that the corresponding commit request 710 will arrive at mentor node 135 before the proposed commit time 705. Similar to snapshot request 510, if commit request 710 arrives at mentor node 135 earlier, mentor node 135 may store the request and delay processing until the proposed commit time 705 identified in commit request 710 is reached according to the local clock 205 of mentor node 135. If commit request 710 arrives after the proposed commit time 705, mentor node 135 may issue a commit response 720 indicating that commit request 710 has been rejected. As discussed regarding the snapshot time 505 of the proposal, in various implementations, worker node 120 may also adjust the time at which it selects the commit time 705 of the proposal in the future. For example, in response to receiving a rejection of a commit request 710 based on its proposed commit time 705, worker node 120 may select another proposed commit time 705 that is further from the worker node 120's current time than the rejected commit time 705 is further from the time selected by the worker node.

[0067] In response to receiving a submission request 710, in various embodiments, the mentor node 135 determines whether to reject the submission request 710. As part of determining whether to reject the submission request 710, the mentor node 135 may determine whether any record 115 identified in the request conflicts with a record 115 previously approved for submission by the mentor node 135. Because in various embodiments, the mentor node 135 records approved records 115 in the form of a history record 225, the mentor node 135 can consult the history record 225 in its history information 220. In various embodiments, if the first record and the second record are associated with the same record identifier, and the second record 115 is identified as having been submitted after the snapshot time associated with the transaction of the first record 115, then the first record 115 conflicts with the second record 115. Therefore, the mentor node 135 may search its history record 225 for records whose record identifier 410 matches the record identifier of the record 115 being considered for approval. If historical record 225 is found, the mentor node 135 can determine whether its commit time 430 occurred after the snapshot time specified in the commit request 710. If a conflicting record 115 exists, the mentor node 135 sends a disapproved commit response 720 to the worker node 120 indicating a record conflict. If no conflicting record 115 exists, the mentor node 135 sends an approved commit response 720. The mentor node 135 can also create historical record 225 for the record 115 of the approved transaction.

[0068] As shown in the diagram, worker node 120 receives commit responses 720 from mentor nodes 135A and 135C, but not from mentor node 135B. Assuming mentor cluster 130 consists only of mentor nodes 135A-135C, and the received commit response 720 approves the commit, worker node 120 can proceed with the commit transaction because it has received approval from a majority of mentor nodes 135A-135C, with no disapprovals. In response to receiving a disapproval of the proposed commit time 705, worker node 120 can choose a new proposed commit time 705 and send a commit request 710 with that proposed commit time 705 to mentor nodes 135A-135C, or worker node 120 can abort the transaction. In response to receiving a commit response 720 that disapproves the commit due to one or more record conflicts, worker node 120 can abort the transaction or not. In some implementations, the commit of the approved mentor node 135 record (e.g., via history record 225) occurs even if worker node 120 aborts the transaction.

[0069] Turn now Figure 8A block diagram illustrating an exemplary layout of historical information for mentor nodes 135A-135C is shown. In the illustrated embodiment, a complete history 600 exists, comprising five historical records 225A-225E. As shown, mentor node 135A includes historical records 225A and 225C-225E; mentor node 135B includes historical records 225A-225B; and mentor node 135C includes historical records 225A-225E. In some cases, the complete history 600 may include more or fewer historical records 225 than shown, and the layout of historical information for mentor node 135 may differ.

[0070] For the sake of the following discussion, assume that... Figure 6 The exemplary layout of the historical information shown occurs in Figure 7 Prior to the exemplary commit-related interactions between worker node 120 and the three mentor nodes 135A-135C shown. Figure 8 The exemplary layout of the historical information shown occurs after the interaction related to the exemplary submission. As described above, in various embodiments, when the mentor node 135 decides to approve the submission, the mentor node 135 stores a history record 225, which identifies the record 115 to be submitted and the assumed location of the submission record. Figure 7 In the example shown, worker node 120 receives commit responses 720 from mentor nodes 135A and 135C, but not from mentor node 135B. In this embodiment, assuming the approved commit involves a transaction with a specific record 115, mentor nodes 135A and 135C store the historical record 225E for that specific record 115, respectively. However, mentor node 135B does not store the historical record 225E because this mentor node may slow down and therefore may not be able to process transactions where the mentor node is slowing down. Figure 7 The commit request 710 is received in the process. As a result, the history of approved records known to one mentor node 135 may differ from the history of approved records known to another mentor node 135. Since the transaction is approved by a majority of mentor nodes, the history record 225E is stored on a majority of mentor nodes 135, so the worker node 120 that receives the approval for the snapshot request 510 from a majority of mentor nodes will receive the history record 225E, making these nodes aware of the specific record 115.

[0071] In various implementations, the mentor node 135 removes history record 225 from its history information 220 over time. In various cases, a new record version of a database construct (e.g., a database row) is approved to replace the commit of a previous record version. Thus, the mentor node 135 can replace the previous record version of history record 225 with a new history record 225 of the new record version. In some cases, the mentor node 135 removes history record 225 in response to record 115 aging beyond a certain threshold. For example, record 115 associated with history record 225A may be written to the LSM tree of database 110 and merged down the LSM tree levels over time. After record 115 has reached a certain level, the mentor node 135 may remove history record 225A from its history information 220 (if the mentor node 135 has approved the transaction for record 115 and history record 225A has been created). As part of removing history record 225 from its history information 220, in some implementations, mentor node 135 sends history record 225 to database 110 so that worker node 120 can access the history record from database 110.

[0072] Turn now Figure 9 A block diagram of exemplary components related to alignment time between parts of system 100 is shown. In the illustrated embodiment, there is one worker node 120 and three mentor nodes 135A-135C, each with a corresponding local clock 205. As further shown, the corresponding local clocks 205A and 205B of mentor nodes 135A and 135B identify the time "10:11:20", the local clock 205C of mentor node 135C identifies the time "10:11:23", and the local clock 205D of worker node 120 identifies the time "10:11:19". Although only three mentor nodes 135 and one worker node 120 are depicted, in various embodiments, there may be more mentor nodes 135 and / or worker nodes 120. Furthermore, in some embodiments, time is represented in a manner different from that shown—for example, time may be a numerical value indicating a time period, or it may be minutes, etc.

[0073] As explained, in various implementations, nodes in system 100 include a local clock 205 that identifies the time observed by that node for system 100. This time may differ from the time observed by another node—for example, worker node 120 observes time "10:11:19," while mentor node 135 observes time "10:11:23." As further explained, nodes may perform operations at specified times that could affect other nodes. A node observing a time that differs reasonably from the average time observed in system 100 may cause unnecessary delays because the execution of one or more operations is later than the expected time for other nodes. Therefore, it may be desirable to continuously align the time within system 100 so that nodes do not become too out of sync with each other.

[0074] In some implementations, worker node 120 and mentor node 135 directly transmit their local time to each other. However, in various implementations, worker node 120 and mentor node 135 do not transmit their local time to each other. Instead, time delays in communication between worker node 120 and mentor node 135 can be tracked by the nodes and used to modify their local clocks 205 in an attempt to align their time with the time observed by other nodes. In various implementations, when worker node 120 issues a request to mentor node 135 (e.g., a snapshot request 510 or a commit request 710), worker node 120 stores a transmission timestamp 910 that specifies the time the request was issued according to its local clock 205. When the request is received at mentor node 135, mentor node 135 may store an arrival timestamp 920 that specifies the time the request was received according to its local clock 205. When mentor node 135 provides a response to worker node 120 (e.g., a snapshot response 520 or a commit response 720), the response may include the arrival timestamp 920. Worker node 120 can calculate the time delay, which is the difference between the transmission timestamp 910 and the arrival timestamp 920. When worker node 120 issues another request to mentor node 135, the worker node can include the time delay in the request. Therefore, mentor node 135 can know the time delay involved in worker nodes 120 communicating with it and the other mentor nodes 135 in the mentor cluster 130.

[0075] For example, worker node 120 may send a commit request 710 to mentor nodes 135A-135C and store a transmission timestamp 910 specifying the time "10:11:17". In response to receiving the commit request 710, mentor node 135A may store an arrival timestamp 920A specifying the time "10:11:20" according to local clock 205A, mentor node 135B may store an arrival timestamp 920B specifying the time "10:11:20" according to local clock 205B, and mentor node 135C may store an arrival timestamp 920C specifying the time "10:11:23" according to local clock 205C. These arrival timestamps 920 may be provided to worker node 120 in a commit response 720 to the commit request 710. Based on transmission timestamp 910 and arrival timestamps 920A-920C, worker node 120 determines that there is a three-second time delay when communicating with mentor nodes 135A and 135B, and a six-second time delay when communicating with mentor node 135C. When worker node 120 communicates with mentor node 135B again, what time delays can worker node 120 provide?

[0076] Based on these time delays, in various implementations, the mentor node 135 determines whether its local clock 205 should speed up or slow down. For example, compared to three seconds for mentor nodes 135A and 135B, mentor node 135C can determine that the submission request 710 took six seconds to arrive. Thus, mentor node 135C can determine that its local clock 205C is approximately three seconds ahead of the time observed by mentor nodes 135A and 135B. In many cases, mentor node 135 attempts to account for factors affecting the time delay between itself and worker node 120. These factors may include, for example, the network distance between worker node 120 and mentor node 135, the number of devices the request passes through on its journey from one node to another, the delay between worker node 120's sampling time and sending the request, and the delay between mentor node 135's receiving the request and sampling time. Based on one or more of these factors, mentor node 135 can estimate the actual clock skew based on the time delay. In various implementations, worker node 120 maintains an average time delay for communication with each mentor node 135 (which may be weighted for recent traffic). Mentor node 135 can receive average time delays from multiple worker nodes 120 and then aggregate these time delays to produce an aggregated average time delay. Thus, in various implementations, mentor node 135 learns the average time delay for communication between multiple worker nodes 120 and a given mentor node 135. Aggregated average time delays allow mentor node 135 to better estimate actual clock skew, taking into account factors that may cause communication delays.

[0077] Therefore, mentor node 135 can adjust its time very slowly. By comparing the perceived latency of receiving requests from each worker-mentor, mentor node 135 can determine whether its time is out of sync with other nodes based on a weighted average of the time latency of other mentor nodes 135. In response to determining to accelerate its local clock 205, in various embodiments, mentor node 135 shifts the time of its local clock 205 forward and increases its own execution speed (e.g., by running processor cores at a higher frequency) to increase the number of requests it can handle, thus allowing its local clock 205 to shift at a faster rate. In some cases, mentor node 135 may shift the time of its local clock 205 forward and skip jobs (e.g., requests) whose specified processing time falls within the shifted time. In response to determining to slow down its local clock 205, in various embodiments, mentor node 135 reduces the rate at which its local clock 205 shifts forward. In some cases, mentor node 135 may freeze the time of its local clock 205 until the mentor node 135 believes it has aligned its time.

[0078] In various implementations, worker node 120 adjusts the time of its local clock 205 based on observed time delays. As one embodiment, worker node 120 may observe that its requests are taking increasingly longer to reach mentor node 135. Based on this observation, worker node 120 may speed up its local clock 205. Alternatively, worker node 120 may observe that its requests are arriving at mentor node 135 faster than expected, and therefore may slow down its local clock 205. In various implementations, worker node 120 also uses the observed time delays to determine when a future snapshot time 505 or commit time 705 should be selected for a proposal, such that, according to their respective local clocks 205, it arrives at mentor node 135 before those proposed times. As the time delay changes over time, worker node 120 may adjust the time for selecting the future snapshot time 505 or commit time 705.

[0079] Turn now Figure 10 A block diagram of exemplary components associated with assistant node 1030 is shown. In the illustrated embodiment, worker nodes 120A-120B, assistant node 1030, and shared memory 1040 are present. As shown, worker node 120B and assistant node 1030 include records 115, and shared memory 1040 includes logs 1045. The illustrated embodiment can be implemented in a manner different from that shown. As one embodiment, multiple assistant nodes 1030 may exist to observe the work performed by worker node 120.

[0080] As explained, processing a transaction may involve accessing one or more records 115, which may be stored in database 110 or in memory cache 210 of worker node 120. When record 115 is identified (e.g., via history record 225) in memory cache 210, worker node 120 may send a record request 204 for that record 115 to the corresponding worker node 120. For example, as shown, worker node 120A issues a record request 204 to worker node 120B for record 115 stored on worker node 120B. Like other components of system 100, worker node 120 may experience a slowdown that causes it to respond slowly or unresponsively to record request 204. As a result, worker node 120 may cause other worker nodes 120 to experience delays in processing transactions. For example, worker node 120B may become unresponsive, so worker node 120A may not receive a record response 206 for its record request 204. To help mitigate this problem, in various implementations, worker node 120 has one or more assistant nodes 1030 that can answer record request 204 on behalf of worker node 120 when worker node slows down.

[0081] In various implementations, assistant node 1030 observes the work performed by worker node 120B and locally stores information corresponding to that work. In various implementations, assistant node 1030 is a set of software routines executable on hardware, while in some implementations, assistant node 1030 includes both hardware and software routines. In some implementations, assistant node 1030 is another worker node 120. Therefore, worker nodes 120 can be assigned to each other such that each worker node 120 is an assistant node 1030 of another worker node 120. In some implementations, assistant node 1030 executes together with other assistant nodes 1030 on a single virtual machine, while in other implementations, these nodes each execute on a separate virtual machine.

[0082] To observe the work performed by worker node 120B, in various embodiments, assistant node 1030 follows worker node 120B's log 1045, reads all record changes, and places them into its storage (e.g., memory cache 210). As shown, log 1045 is stored in shared storage 1040 shared between worker node 120 and assistant node 1030. In some embodiments, worker node log 1045 is stored locally at worker node 120, not at shared storage 1040, so its assistant node 1030 can access log 1045 from worker node 120 instead of from shared storage 1040. Shared storage 1040 may be part of database 110 or a separate storage mechanism. In various embodiments, log 1045 is a collection of log records that identify changes (e.g., record changes) to information stored in a database (e.g., database 110). Log records may be records identifying new data values ​​in database construction (e.g., database rows), commit records identifying transaction commits, or rollback records allowing partial or full database rollback. In this way, log 1045 can identify pending and committed transactions, including the order in which those transactions were committed. The worker node 120 that writes transaction record 115 can update its log 1045 by appending log records identifying that work. Thus, as worker node 120 appends more log records, log 1045 can grow over time.

[0083] Therefore, in various implementations, assistant node 1030 reads and writes log records to worker node 120B's log 1045 and constructs an in-memory image of the most recent transactions executed by worker node 120B. Thus, when worker node 120B writes record 115 to its cache 210 and records to its log 1045, assistant node 1030 can learn which records 115 are from log 1045 and recreate the records in its own memory. As a result, assistant node 1030 can store one or more records 115 that are identical to those stored in worker node 120B. Because assistant node 1030 follows worker node 120B's log 1045 and stores one or more identical records 115, assistant node 1030 can process record requests 204 from worker node 120A as if it were worker node 120B.

[0084] In some implementations, worker node 120A, searching for record 115, initially sends record request 204 to worker node 120B identified as committing record 115 (although the record may not have been committed if its transaction was aborted after approval). If record request 204 times out and there is no record response 206, worker node 120A may send record request 204 to worker node 120B's assistant node 1030. In some implementations, worker node 120A may send record request 204 to worker node 120B and its assistant node 1030 in parallel. This can save time, but at the cost of potentially sending more requests than needed.

[0085] In response to receiving record request 204, in various implementations, assistant node 1030 determines whether worker node 120B has already committed the requested record 115. In some cases, assistant node 1030 determines (e.g., via a commit record written in log 1045 of worker node 120B) that the requested record 115 has been committed. Assistant node 1030 may access record 115 from its own memory and return the record to worker node 120A in record response 206. In some cases, assistant node 1030 determines that the requested record 115 does not exist, and worker node 120B has already recorded work beyond the time associated with the transaction containing the requested record 115. For example, worker node 120B may abort the transaction instead of committing it, and then commit another transaction associated with a later commit time, so assistant node 1030 can determine that work has already been recorded after the transaction containing the requested record 115. Therefore, assistant node 1030 can return a record response 206 to worker node 120A, indicating that the requested record 115 does not exist. In some cases, assistant node 1030 may not be far enough in worker node 120B's log 1045 to know whether the requested record 115 has been committed. Assistant node 1030 can delay its response until it has read enough of worker node 120B's log 1045, or it can provide worker node 120A with record response 206, instructing worker node 120A to retry its record request 204 later. In the former case, assistant node 1030 can ensure that it has read the record in worker node 120B's log 1045 with a timestamp indicating a time prior to the snapshot time specified in record request 204.

[0086] In some cases, assistant node 1030 determines that worker node 120B's log 1045 has not yet reached the point where a transaction to commit the requested record 115 can be made, and that worker node 120B has not written to its log 1045 or is writing at a rate that is too slow according to a defined threshold. Based on worker node 120B's failure to update its log 1045 and commit the requested record 115, in various implementations, assistant node 1030 isolates worker node 120B's log 1045 and then returns a record response 206 to worker node 120A indicating that the requested record 115 does not exist. Because assistant node 1030 received the record request 204 from worker node 120A, this indicates that worker node 120A has obtained a historical record 225 linked to the requested record 115. Because worker node 120A possesses this historical record 225, this indicates that worker node 120B has received permission to commit the transaction to the requested record 115. In this scenario, to provide a response 206 indicating that the requested record 115 has not yet been committed, the assistant node 1030 isolates the log 1045 of worker node 120B, preventing the submission of the requested record 115 after the assistant node 1030 has already informed worker node 120A that the requested record 115 is uncommitted. To isolate the log 1045 of worker node 120B, in various implementations, the assistant node 1030 sends a command to shared memory 1040 to prevent shared memory 1040 from appending to the log 1045 of worker node 120B. As a result, worker node 120B cannot write log records and therefore cannot perform work for system 100. However, the contents of the log 1045 of worker node 120B can remain intact and be read by other nodes (e.g., assistant node 1030). After the log 1045 of worker node 120B has been isolated, the assistant node 1030 can perform a final check on the log 1045 to determine whether a commit has been added for the transaction of the requested record 115. If the transaction has been committed, the assistant node 1030 can return the requested record 115 to the worker node 120A; otherwise, the assistant node 1030 can indicate to the worker node 120A that the requested record 115 has not yet been committed.

[0087] Turn now Figure 11A block diagram is shown illustrating a refresh operation performed by worker node 120 and assistant node 1030. In the illustrated embodiment, there is a database 110, worker node 120, and assistant node 1030. As shown, both worker node 120 and assistant node 1030 include the same record 115A—that is, they both have a record 115 that stores the same information (e.g., values) for the same database structure (e.g., database rows). Also as shown, database 110 includes a merged tree (LSM tree) 1100 of a log structure that receives the same record 115 from worker node 120B and assistant node 1030. The illustrated embodiment can be implemented in a manner different from that shown. As one embodiment, multiple assistant nodes 1030 may write the same record 115 as worker node 120 to database 110.

[0088] In various implementations, the log structure merge tree 1100 is a data structure that stores files in an organized manner using a level-based scheme. The LSM tree 1100 may include two high-level components: an in-memory component implemented at the memory cache 210 of worker node 120 and an on-disk component implemented at database 110. In some cases, the in-memory component is considered to be separate from the LSM tree 1100. Worker node 120 may initially write records 115 to its in-memory cache 210. As its cache 210 becomes full and / or at a specific point in time, worker node 120 may flush records 115 to database 110. As part of flushing records 115, in various implementations, worker node 120 writes them to a new file at the top level of the LSM tree 1100. Over time, as newer files are written to the top level, records 115 of these files are merged down through the levels of the LSM tree 1100. While regarding... Figure 11 The various techniques discussed are related to LSM trees, but these techniques can be applied to other types of database implementations where worker nodes exist that can write and commit records to the database.

[0089] For the following discussion, it is assumed that worker node 120 has committed record 115A. In some cases, worker node 120 may slow down before it has flushed record 115A to LSM tree 1100, causing it to become stressed and unable to flush record 115A in a timely manner. The efficiency of system 100 may depend on the speed at which record 115 is flushed to LSM tree 1100. Thus, stress on worker node 120 may lead to undesirable slowdowns in system 100. Therefore, in various implementations, both worker node 120 and assistant node 1030 flush the same record 115 to LSM tree 1100. As a result, while worker node 120 may not be able to flush record 115A in a timely manner due to slowdown, assistant node 1030 can independently flush record 115A to LSM tree 1100.

[0090] In all cases, since both worker node 120 and assistant node 1030 refresh record 115, LSM tree 1100 stores copies of the same record 115. As described above, a merge operation can be performed to move record 115 through each level of LSM tree 1100. In various implementations, the merge operation removes copies such that only one copy of record 115 moves to the next level of LSM tree 1100. Therefore, LSM tree 1100 can temporarily store copies of record 115 that are eventually removed by the merge operation performed by system 100.

[0091] Turn now Figure 12 A flowchart of method 1200 is shown. Method 1200 is an embodiment of a method executed by worker nodes (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1200 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1200 includes more or fewer steps than shown. For example, method 1200 may include the step of a worker node sending a request (e.g., record request 204) to another worker node to obtain a record (e.g., record 115).

[0092] Method 1200 begins at step 1210, where a worker node receives a request to execute a transaction involving writing records. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions against the database system and multiple mentor nodes (e.g., mentor node 135) operable to ensure transactional consistency. Worker nodes may write records to their own memory cache (e.g., memory cache 210).

[0093] In step 1220, a worker node issues a request (e.g., snapshot request 510, commit request 710, etc.) to one of the multiple mentor nodes (e.g., a majority subset or all) for information (e.g., history record 225) to facilitate the execution of a transaction operation. In some cases, the request for information may specify at least a record, and the operation may correspond to a commit transaction. A given mentor node may determine whether to approve the commit transaction based on whether the record conflicts with a history of approved records known to the given mentor node (e.g., history information 220). The request to the mentor node may further specify the snapshot time of the transaction, and the given mentor node may check for conflicts between the record and those records in the history of approved records that were approved after the snapshot time. The history of approved records may include one or more records that have not yet been flushed from the multiple worker nodes to a persistent storage (e.g., database 110) shared among the multiple worker nodes. In response to sending an approval response to a worker node, the given mentor node may include the record in the history of approved records (e.g., by creating history record 225 for the record). In various implementations, a given mentor node determines whether to approve a transaction commit without communicating with other mentor nodes among a plurality of mentor nodes. Requests to mentor nodes may indicate a commit time (e.g., a proposed commit time 705), and a given mentor node may process requests from worker nodes at the commit time.

[0094] In step 1230, a worker node proceeds with a transaction operation in response to receiving an approval response (e.g., approval snapshot response 520, approval commit response 720, etc.) from a majority of the mentor nodes, where none of the received responses indicated disapproval of the transaction. In some cases, after determining that permission to commit the transaction has been granted, the worker node may commit the transaction and flush the record to persistent storage shared among the worker nodes. In other cases, after determining that permission has been granted, the worker node may abort the transaction. A worker node may abort a transaction based on the determination that it has not yet received an approval response from a majority of the mentor nodes. A worker node may abort a transaction in response to receiving a disapproval response from at least one mentor node, indicating disapproval of the commit based on a conflict between the record and another record known to the at least one mentor node. A worker node may also abort a transaction in response to receiving a disapproval response from at least one mentor node, indicating disapproval of the commit based on the commit request arriving after the commit time.

[0095] Turn now Figure 13A flowchart of method 1300 is shown. Method 1300 is an implementation of a method performed by a mentor node (e.g., mentor node 135) of a database system (e.g., system 100) as part of ensuring transaction consistency within a database system. Method 1300 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some implementations, method 1300 includes more or fewer steps than shown. For example, method 1200 may include the step of the mentor node processing a snapshot request (e.g., snapshot request 510) received from a worker node (e.g., worker node 120).

[0096] Method 1300 begins at step 1310, where a mentor node receives from a worker node a request (e.g., commit request 710) for permission to commit a transaction involving a record (e.g., record 115). The worker node may be part of a database system that may include multiple worker nodes operable to execute transactions and multiple mentor nodes operable to ensure the transactional consistency of those transactions. The received request may specify the record for the transaction.

[0097] In step 1320, the mentor node determines whether the record conflicts with a set of records provided by other worker nodes among the multiple worker nodes (e.g., as part of commit request 710) (e.g., identified in history information 220). The received request may specify a snapshot time associated with the transaction (e.g., a previously approved snapshot time 505) and a commit time (e.g., a proposed commit time 705). Thus, the determination may include the mentor node determining whether there are conflicting records in the set of records identified as having been committed between the snapshot time and the commit time. Furthermore, the determination may be performed after the mentor node's indicated time is equal to the commit time on its local clock (e.g., local clock 205). In some cases, the set of records includes at least one record not included in another set of records stored by another mentor node among the multiple mentor nodes.

[0098] In step 1330, based on this determination, the mentor node sends a response to the worker node indicating whether permission to commit the transaction has been granted (e.g., commit response 720). In some embodiments, multiple mentor nodes are not operational to communicate with each other regarding whether permission to commit the transaction has been granted. In some cases, the response indicates that permission to commit the transaction has been granted, and the mentor node stores historical information (e.g., history record 225) that identifies the record as committed, regardless of whether the worker node has committed the transaction. In some cases, the response indicates that permission to commit the transaction has not been granted, and the worker node aborts the transaction independently of receiving approval for the transaction from other mentor nodes among the multiple mentor nodes. The response may provide the worker node with an indication of when it is permitted to commit the transaction (e.g., approval of the proposed commit time 705).

[0099] Turn now Figure 14 A flowchart of method 1400 is shown. Method 1400 is an embodiment of a method executed by worker nodes (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1400 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1400 includes more or fewer steps than shown. For example, method 1400 may include a step where the worker node sends a request (e.g., submit request 710) to a mentor node (e.g., mentor node 135) to obtain permission to submit the transaction.

[0100] Method 1400 begins at step 1410, where a worker node receives a request to execute a transaction involving writing a first record (e.g., record 115). In some cases, a worker node receives a request to execute a read-only transaction that does not involve writing any records. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions against the database system and multiple mentor nodes (e.g., mentor node 135) operable to ensure the transactional consistency of transactions.

[0101] In step 1420, the worker node issues a snapshot request (e.g., snapshot request 510) to one of the mentor nodes (e.g., a majority subset or all) to request historical information (e.g., historical information 220) describing the history of approved records up to a specified time (e.g., the proposed snapshot time 505). This specified time can be chosen by the worker node, and the mentor nodes can delay processing the snapshot request until that specified time. For each record in the history of approved records, the historical information may specify the record's location (e.g., location 24) and the time when the record was indicated to have been committed (e.g., the commit time).

[0102] In step 1430, the worker node proceeds to execute the transaction in response to receiving a response (e.g., snapshot response 520) from a majority of the mentor nodes. The union of the information in the responses constitutes the requested historical information. The process may include determining, based on the requested historical information, whether a second record conflicts with the first record, and aborting the transaction in response to the determination of a conflicting second record. The history of approved records known to the first mentor node may differ from the history of approved records known to the second mentor node. In some cases, no single response provides the requested historical information. Mentor nodes may not be able to communicate with each other to share historical information.

[0103] This transaction may involve a second record. A worker node can determine the location of the second record based on the requested historical information and can attempt to access it from that location. In some cases, this location corresponds to a central memory shared among multiple worker nodes. In other cases, this location corresponds to another worker node. When attempting to access the second record, a worker node can send a request for the second record to another worker node (e.g., record request 204) and can receive a response indicating that the second record has not been committed (e.g., record response 206).

[0104] Turn now Figure 15 A flowchart of method 1500 is shown. Method 1500 is an embodiment of a method executed by worker nodes (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1500 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1500 includes more or fewer steps than shown. For example, method 1500 may include the step of a worker node sending a request (e.g., record request 204) to another worker node to obtain a record (e.g., record 115).

[0105] Method 1500 begins at step 1510, where a worker node receives a request to execute a transaction involving writing records. A database system with worker nodes may include multiple worker nodes (e.g., worker node 120) operable to execute transactions against the database system and multiple mentor nodes (e.g., mentor node 135) operable to ensure the transactional consistency of the transactions.

[0106] In step 1520, the worker node selects a proposal time (e.g., a snapshot time 505, a commit time 705, etc.) associated with performing a transactional operation (e.g., initiating a transaction, committing a transaction, etc.). In various implementations, the worker node determines the time delay for communicating with the mentor node. A given time delay in the time delay could be the delay between the worker node sending a given request to a given mentor node and the given mentor node receiving the given request. Therefore, the worker node could record a first time when the given request is sent to the given mentor node (e.g., transmission timestamp 910) and receive a response to the given request, which identifies a second time when the given request is received at the given mentor node (e.g., arrival timestamp 920). The given time delay can be determined based on the first and second times, and the proposal time can be selected based on the time delay such that the approval request arrives at the mentor node before the proposal time, based on the mentor node's corresponding clock (e.g., local clock 205). The proposal time can also be selected based on a time identified by the worker node's clock. In various cases, the clock of the worker node is different from the clock of the mentor node, and the clock of the mentor node is different from the clock of at least one other mentor node among a plurality of mentor nodes.

[0107] In step 1530, the worker node issues a request for approval to execute an operation to the mentor nodes (e.g., a majority subset or all) among a plurality of mentor nodes. In various embodiments, the mentor nodes are operable to process the approval request upon arrival of the proposal time, based on the clock of the mentor node identifying the time observed by the mentor node. In some cases, the operation relates to initiating a transaction, and the mentor node is operable to return historical information (e.g., historical information 220) describing the history of approved records up to the proposal time. In some cases, the operation relates to committing a transaction, and the mentor node is operable to check for conflicts between transaction records and a set of approved records known to the mentor node that occurred before the proposal time. The worker node may include delay information in the approval request so that a first mentor node among the mentor nodes can attempt to align the time observed by the first mentor node with the time observed by a second mentor node among the mentor nodes.

[0108] Worker nodes can receive a disapproval response from the mentor node, indicating a disapproval proposal time based on the request arriving at the mentor node after the proposal time according to the mentor node's clock. The proposal time can be selected as a first amount of time after the current time identified by the worker node's clock. Therefore, based on receiving a disapproval response, the worker node can choose another proposal time such that this other proposal time is a second amount of time after the current time identified by the worker node's clock, where the second amount of time is greater than the first amount of time.

[0109] Turn now Figure 16 A flowchart of method 1600 is shown. Method 1600 is an embodiment of a method executed by worker nodes (e.g., worker node 120) of a database system (e.g., system 100) as part of processing a transaction. Method 1600 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1600 includes more or fewer steps than shown. For example, method 1600 may include the step of a worker node sending a request (e.g., record request 204) to another worker node to obtain a record (e.g., record 115).

[0110] Method 1600 begins with step 1610, where the worker node accesses delay information specifying the time delay for communicating with those of the multiple mentor nodes in the database system that are operational to ensure the transactional consistency of transactions in the database system. In step 1620, the worker node requests to execute the transaction.

[0111] In step 1630, the worker node selects a proposed snapshot time (e.g., proposed snapshot time 505) for the transaction based on latency information and the worker node's clock (e.g., local clock 205). The proposed snapshot time can be selected as the time at which the allowed approval request arrives at the mentor node before the proposed snapshot time, as determined by the mentor node's clock.

[0112] In step 1640, the worker node issues a first approval request (e.g., snapshot request 510) to two or more of the multiple mentor nodes. In various embodiments, one of the two or more mentor nodes is operable to process the first approval request when the proposed snapshot time arrives according to the mentor node's clock. Before issuing the first approval request, the worker node may issue a second request to the mentor nodes to approve a proposed time associated with another transaction. The worker node may record a first time (e.g., transmission timestamp 910) for issuing the second approval request to the mentor nodes according to the worker node's clock and receive a response to the second approval request from the mentor nodes, the response identifying a second time (e.g., arrival timestamp 920) according to the mentor node's clock. The worker node's clock may identify a time different from the mentor node's clock, and the proposed snapshot time may be selected based on the first and second times.

[0113] Turn now Figure 17 A flowchart of method 1700 is shown. Method 1700 is an embodiment of a method executed by a mentor node (e.g., mentor node 135) of a database system (e.g., system 100) to attempt to align the time observed by that mentor node with the time observed by other mentor nodes. Method 1700 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1700 includes more or fewer steps than shown. For example, method 1700 may include a step in which the mentor node provides a response to a worker node (e.g., worker node 120) including historical information (e.g., historical information 220).

[0114] Method 1700 begins with step 1710, where the mentor node maintains a clock (e.g., local clock 205) that identifies the time observed by the mentor node. The mentor node can perform database operations at specified times.

[0115] In step 1720, the mentor node receives a request (e.g., snapshot request 510) from one of a plurality of worker nodes operable to process a transaction, for information (e.g., historical information 220) that facilitates the execution of the transaction associated with the worker node. This request can provide a first indication of the time observed by other mentor nodes among the plurality of mentor nodes. In various embodiments, the request specifies the time delay observed by the worker node while communicating with the plurality of mentor nodes. The mentor node can aggregate the time delays with those provided by other worker nodes and modify its clock based on the aggregated time delays. A given time delay in the time delay can be the delay between the worker node sending a given request to a given mentor node among the plurality of mentor nodes and the given mentor node receiving the given request. The time delay can be derived based on the worker node's clock, which identifies the time observed by the worker node. In some cases, the request specifies a proposal time associated with the transaction (e.g., a proposed snapshot time 505). Depending on the mentor node's clock, the mentor node can process the request when the proposal time arrives. In various cases, the mentor node's clock identifies a time different from that of another mentor node among the plurality of mentor nodes. Therefore, a request can be processed on the mentor node before another mentor node processes the request.

[0116] In step 1730, the mentor changes its clock based on a first instruction to attempt to align the time observed by the mentor node with the time observed by other mentor nodes. Based on time delay, the mentor node may determine that there is a greater time delay among worker nodes communicating with the mentor node than among worker nodes communicating with most other mentor nodes. Therefore, the change may include slowing down the clock. Based on time delay, the mentor node may determine that there is a smaller time delay among worker nodes communicating with the mentor node than among worker nodes communicating with most other mentor nodes. Therefore, the change may include speeding up the clock. In some cases, the mentor node may identify the arrival time of a request based on the clock (e.g., arrival timestamp 920) and send a response to the worker node specifying the arrival time. The worker node is operable to identify the time delay between sending the request and the mentor node receiving the request. In some cases, changing the clock causes the mentor node to skip processing one or more requests received at the mentor node. Clock changes can be performed without directly communicating with other mentor nodes.

[0117] Turn now Figure 18A flowchart of method 1800 is shown. Method 1800 is an embodiment of a method executed by a mentor node (e.g., mentor node 135) of a database system (e.g., system 100) to attempt to align time as observed by the mentor node. Method 1800 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1800 includes more or fewer steps than shown. For example, method 1800 may include a step in which the mentor node provides a response to a worker node (e.g., worker node 120) including historical information (e.g., historical information 220).

[0118] Method 1800 begins with step 1810, where the mentor node maintains corresponding clocks (e.g., local clock 205) that indicate the corresponding time observed by the mentor node. A mentor node is given the option to perform database operations at a specified time. In step 1820, the mentor node receives timing information from the worker nodes of the database system. The timing information may indicate the communication time delay between the worker nodes and the mentor node.

[0119] In step 1830, the mentor node, based on timing information, changes its corresponding clock to attempt to align with the corresponding time observed by the mentor node. In various implementations, the change is performed without the mentor node communicating with other mentor nodes. The mentor node may receive a request for permission to commit a transaction from a worker node (e.g., commit request 710), and this request may specify a proposed commit time (e.g., proposed commit time 705). A given mentor node may process this request if its corresponding clock specifies a time not preceding the proposed commit time. The given mentor node may send a response to the worker node (e.g., commit response 720) indicating the time of arrival of the request at the given mentor node according to its corresponding clock (e.g., arrival timestamp 920). Worker nodes may determine the time delay among worker nodes communicating with the given mentor node based on the arrival time. In various cases, the request received from the worker node is processed by the given mentor node, regardless of the processing of the request by another mentor node.

[0120] Turn now Figure 19A flowchart of method 1900 is shown. Method 1900 is an embodiment of a method executed by an assistant node (e.g., assistant node 1030) of a database system (e.g., system 100), which observes the work performed by a worker node (e.g., worker node 120). Method 1900 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 1900 includes more or fewer steps than shown. For example, method 1900 may include the step of the assistant node refreshing a record (e.g., record 115) to a database (e.g., database 110) that is also refreshed by the worker node.

[0121] Method 1900 begins at step 1910, wherein a worker node observes work performed by a first worker node of a database system, the database system comprising multiple worker nodes operable to perform transactions against the database system. Observation may include storing records associated with the first worker node in an assistant node. The assistant node may read the transaction log of the first worker node (e.g., log 1045), which is stored in a data storage (e.g., shared memory 1040) shared between the first worker node and the assistant node. Records stored in the assistant node can be determined from the transaction log. In some cases, the assistant node is one of multiple worker nodes operable to perform transactions against the database system. In various embodiments, the assistant node and the first worker node use different virtual machines for execution.

[0122] In step 1920, the assistant node receives a request (e.g., record request 204) from a second worker node among a plurality of worker nodes for a record associated with the first worker node. This request may specify a snapshot time corresponding to the transaction of the second worker node. Before determining whether the first worker node has committed the record, the assistant node may ensure that the record in the transaction log, with a timestamp indicating a time prior to the snapshot time, has been read by the assistant node. In various embodiments, the second worker node is operable to send a request for the record to the assistant node in response to not receiving a response from the first worker node for a request for the record sent to the first worker node.

[0123] In step 1930, the assistant node determines whether the first worker node has committed the record. Making this determination may include determining, based on the transaction log, that the record has not yet been committed and that the first worker node has not responded. Based on this determination, in various embodiments, the assistant node fences the transaction log to prevent the first worker node from modifying the transaction log to commit the record, and then sends a response to the second worker node indicating that the record has not yet been committed (e.g., record response 206). Making this determination may also include determining, based on the transaction log, that the first worker node committed the transaction including the record. Therefore, the assistant node can access the record from the records stored in the assistant node, and the response to the second worker node may include the accessed record. Making this determination may also include determining, based on transaction logs indicating that the first worker node has recorded work after the commit time associated with the transaction including the record and transaction logs not indicating the record, that the record has not yet been committed. Thus, the response to the second worker node may indicate that the record has not yet been committed.

[0124] In step 1940, the assistant node sends a response to the second worker node indicating whether the record was committed by the first worker node. Based on the determination that the record has been committed, the assistant node can flush the record to the merged tree (LSM tree) of the database system's log structure (e.g., LSM 1100). The first worker node can also flush the record to the LSM tree, so that there are two instances of the record in the LSM tree for a period of time.

[0125] Turn now Figure 20 A flowchart of method 2000 is shown. Method 2000 is an embodiment of a method executed by an assistant node (e.g., assistant node 1030) of a database system (e.g., system 100), which observes the work performed by a worker node (e.g., worker node 120). Method 2000 can be executed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 2000 includes more or fewer steps than shown. For example, method 2000 may include the step of the assistant node refreshing a record (e.g., record 115) to a database (e.g., database 110) that is also refreshed by the worker node.

[0126] Method 2000 begins in step 2010, where the assistant node reads the transaction log of the first worker node (e.g., log 1045). In various implementations, the transaction log specifies a set of records committed by the first worker node. Based on the read of the transaction log, the assistant node can cache one or more records from this set at its own node. In step 2020, the assistant node receives a request from the second worker node for records associated with the first worker node (e.g., record request 204).

[0127] In step 2030, based on the reads of the transaction log, the assistant node determines whether a record has been committed. A request received from the second worker node may identify a snapshot timestamp. The assistant node may determine that the first worker node has not updated the transaction log and has not written a record to the transaction log with a timestamp indicating a time after the snapshot timestamp. Based on this determination, the assistant node may prevent the first worker node from updating the transaction log. The response to the second worker node may indicate that the record has not yet been committed. In response to determining that the record has been committed, the assistant node may access the record from one or more cached records. This response may include the record.

[0128] In step 2040, the assistant node sends a response to the second worker node indicating whether the record has been committed. In response to confirming that the record has been committed, the assistant node can refresh the record to the database of the database system (e.g., database 110), regardless of whether the first worker node refreshes the record to the database.

[0129] Exemplary multi-tenant database system

[0130] Turn now Figure 21 An exemplary multi-tenant database system (MTS) 2100 is shown, which can implement various technologies for public text. For example, system 100 may be MTS 2100. Figure 21 In this embodiment, MTS2100 includes a database platform 2110, an application platform 2120, and a network interface 2130 connected to network 2140. Also as shown, database platform 2110 includes a data storage 2112 and a set of database servers 2114A-2114N interacting with the data storage 2112, and application platform 2120 includes a set of application servers 2122A-2122N with a corresponding environment 2124. In the illustrated embodiment, MTS2100 is connected to various user systems 2150A-2150N via network 2140. The disclosed multi-tenant systems are included for illustrative purposes and are not intended to limit the scope of the disclosed text. In other embodiments, the technologies described in the disclosed text are implemented in non-multi-tenant environments, such as client / server environments, cloud computing environments, clustered computers, etc.

[0131] In various implementations, the MTS2100 is a set of computer systems that together provide various services to users (or “tenants”) interacting with the MTS2100. In some implementations, the MTS2100 implements a customer relationship management (CRM) system that provides mechanisms for tenants (e.g., companies, government agencies, etc.) to manage their relationships and interactions with customers and prospects. For example, the MTS2100 enables tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), identify sales opportunities, record service issues, and manage marketing campaigns. Furthermore, the MTS2100 enables tenants to identify how they communicate with customers, what customers have purchased, when customers last purchased items, and how much customers paid. To provide CRM system services and / or other services, as shown in the figure, the MTS2100 includes a database platform 2110 and an application platform 2120.

[0132] In various implementations, database platform 2110 is a combination of hardware components and software routines that implement database services for storing and managing data (including tenant data) of MTS 2100. As shown, database platform 2110 includes data storage 2112. In various implementations, data storage 2112 includes a set of storage devices (e.g., solid-state drives, hard disk drives, etc.) connected together on a network (e.g., a Storage Attached Network (SAN)) and configured to redundantly store data to prevent data loss. In various implementations, data storage 2112 is used to implement a database (e.g., database 110) that includes a collection of information organized in a manner that allows access, storage, and manipulation of information. Data storage 2112 can implement a single database, a distributed database, a collection of distributed databases, a database with redundant online or offline backups, or other redundancies. As part of the database implementation, the data storage 2112 can store a file comprising one or more database records having corresponding data payloads (e.g., values ​​of fields in a database table) and metadata (e.g., key values, timestamps, table identifiers of the tables associated with the records, tenant identifiers of the tenants associated with the records, etc.).

[0133] In various implementations, database records may correspond to rows in a table. A table typically contains one or more data categories, which are logically set as columns or fields in a visual schema. Therefore, each record in a table may contain a data instance for each category defined by its fields. For example, a database may include a table describing customers, with fields for basic contact information such as name, address, phone number, and fax number. Therefore, records in this table may include values ​​for each field in the table (e.g., the name of the name field). Another table might describe purchase orders, including fields for information such as customer, product, sales price, and date. In various implementations, standard entity tables are provided for use by all tenants, such as tables for account, contact, lead, and opportunity data, each containing predefined fields. The MTS2100 may store database records for one or more tenants in the same table; that is, tenants may share a single table. Therefore, in various implementations, database records include a tenant identifier indicating the owner of the database record. As a result, one tenant's data remains secure and separate from other tenants' data, preventing that tenant from accessing another tenant's data unless such data is explicitly shared.

[0134] In some implementations, the data stored in data storage 2112 is organized as part of a merged log structure tree (LSM tree, e.g., LSM tree 1100). The LSM tree typically includes two high-level components: a memory cache and persistent storage. In operation, database server 2114 may first write database records to the local memory cache and then flush these records to persistent storage (e.g., data storage 2112). As part of flushing database records, database server 2114 may write database records to a new file included at the “top level” of the LSM tree. Over time, as database records move down through the levels of the LSM tree, they can be rewritten by database server 2114 to new files included in lower levels. In various implementations, as database records age and move down the LSM tree, they are moved to increasingly slower storage devices (e.g., from solid-state drives to hard disk drives) of data storage 2112.

[0135] When database server 2114 wants to access a database record for a specific key, it can traverse different levels of the LSM tree to search for files that might contain the database record for that specific key. If database server 2114 determines that a file can include the relevant database record, it can retrieve the file from data storage 2112 into its own storage. Database server 2114 can then examine the retrieved file for the database record with the specific key. In various implementations, database records are immutable once written to data storage 2112. Therefore, if database server 2114 wants to modify the value of a row in a table (which can be identified from the accessed database record), it writes a new database record to the top level of the LSM tree. Over time, this database record is merged down the LSM tree. Thus, the LSM tree can store various database records for a database key, where older records for that key are located at lower levels of the LSM tree compared to newer records.

[0136] In various implementations, database server 2114 is a hardware component, software routine, or a combination thereof capable of providing database services (e.g., data storage, data retrieval, and / or data manipulation). Database server 2114 may correspond to worker node 120. Such database services can be provided by database server 2114 to components within MTS 2100 (e.g., application server 2122) and components outside MTS 2100. For example, database server 2114 may receive database transaction requests from application server 2122, requesting data to be written to or read from data storage 2112. Database transaction requests may specify SQL SELECT commands to select one or more rows from one or more database tables. The content of a row can be defined in a database record, so database server 2114 can locate and return one or more database records corresponding to the selected rows from one or more tables. In various cases, database transaction requests may instruct database server 2114 to write one or more database records for an LSM tree, which database server 2114 maintains, implemented on database platform 2110. In some implementations, database server 2114 implements a relational database management system (RDMS) or an object-oriented database management system (OODBMS), which facilitates the storage and retrieval of information for data storage 2112. In various cases, database servers 2114 can communicate with each other to facilitate transaction processing. For example, database server 2114A can communicate with database server 2114N to determine whether database server 2114N has written a database record to its memory cache for a specific key.

[0137] In various implementations, application platform 2120 is a combination of hardware components and software routines that implement and execute CRM software applications, and provide relevant data, code, forms, web pages, and other information to and from user system 2150, and store relevant data, objects, web page content, and other tenant information via database platform 2110. To facilitate these services, in various implementations, application platform 2120 communicates with database platform 2110 to store, access, and manipulate data. In some cases, application platform 2120 may communicate with database platform 2110 via different network connections. For example, one application server 2122 may be connected via a local area network, and another application server 2122 may be connected via a direct network link. Transmission Control Protocol and Internet Protocol (TCP / IP) are exemplary protocols for communication between application platform 2120 and database platform 2110; however, it will be apparent to those skilled in the art that other transport protocols may be used depending on the network interconnection used.

[0138] In various implementations, application server 2122 is a hardware element, software routine, or combination thereof capable of providing services to application platform 2120, including processing requests received from tenants of MTS 2100. In various implementations, application server 2122 can create environment 2124 that can be used for various purposes, such as providing developers with the ability to develop, execute, and manage applications (e.g., business logic). Data can be transferred from another environment 2124 and / or from database platform 2110 to environment 2124. In some cases, environment 2124 cannot access data from other environments 2124 unless such data is explicitly shared. In some implementations, multiple environments 2124 may be associated with a single tenant.

[0139] Application platform 2120 can provide user system 2150 with access to multiple different hosted (standard and / or custom) applications, including CRM applications and / or applications developed by tenants. In various implementations, application platform 2120 can manage application creation, application testing, application storage in database objects at data storage 2112, application execution in environment 2124 (e.g., virtual machines in process space), or any combination thereof. In some implementations, application platform 2120 can add and remove application server 2122 from the server pool at any time for any reason, even if there is no server affinity between a user and / or organization and a particular application server 2122. In some implementations, an interface system (not shown) implementing load balancing functionality (e.g., an F5 Big-IP load balancer) is located between application server 2122 and user system 2150 and configured to distribute requests to application server 2122. In some implementations, the load balancer uses a least-connections algorithm to route user requests to application server 2122. Other embodiments of the load balancing algorithm can also be used, such as round-robin and observation of response time. For example, in some implementations, three consecutive requests from the same user may hit three different servers 2122, and three requests from different users may hit the same server 2122.

[0140] In some implementations, the MTS2100 provides security mechanisms, such as encryption, to keep each tenant's data separate unless the data is shared. If more than one server 2114 or 2122 is used, the servers can be located very close to each other (e.g., a cluster of servers in a single building or campus) or distributed in locations far apart from each other (e.g., one or more servers 2114 located in city A and one or more servers 2122 located in city B). Therefore, the MTS2100 may include one or more logically and / or physically connected servers that are distributed locally or across one or more geographical locations.

[0141] One or more users (e.g., via user system 2150) can interact with MTS 2100 via network 2140. User system 2150 may correspond to, for example, a tenant of MTS 2100, a provider of MTS 2100 (e.g., an administrator), or a third party. Each user system 2150 may be a desktop PC, workstation, laptop, PDA, mobile phone, or any device that supports Wireless Access Protocol (WAP) or any other computing device capable of directly or indirectly engaging with the Internet or other network connections. User system 2150 may include dedicated hardware configured to engage with MTS 2100 via network 2140. User system 2150 may execute a graphical user interface (GUI) corresponding to MTS 2100, an HTTP client (e.g., a browser program, such as Microsoft Internet Explorer), etc. TM Browsers, Netscape's Navigator TM A browser (such as an Opera browser, or a WAP-enabled browser in the case of a cellular phone, PDA, or other wireless device, or both) allows users of user system 2150 (e.g., subscribers of a CRM system) to access, process, and view available information and pages from MTS 2100 via network 2140. Each user system 2150 may include one or more user interface devices, such as a keyboard, mouse, touchscreen, pen, etc., for interacting with a graphical user interface (GUI) provided by a browser on a display monitor screen, LCD display, etc., in conjunction with pages, forms, and other information provided by MTS 2100 or other systems or servers. As described above, the disclosed embodiments are applicable to the Internet, which refers to a specific global network interconnection. However, it should be understood that other networks can be used instead of the Internet, such as intranets, extranets, virtual private networks (VPNs), non-TCP / IP-based networks, any LAN or WAN, etc.

[0142] Because users of user system 2150 can have different capabilities, the capabilities of a specific user system 2150 may be determined by one or more permission levels associated with the current user. For example, when a salesperson interacts with MTS2100 using a specific user system 2150, that user system 2150 may have the capabilities assigned to that salesperson (e.g., user privileges). However, when an administrator interacts with MTS2100 using the same user system 2150, that user system 2150 may have the capabilities assigned to that administrator (e.g., administrative privileges). In systems with a hierarchical role model, a user at one permission level can access applications, data, and database information accessible to users at lower permission levels, but cannot access certain applications, database information, and data accessible to users at higher permission levels. Therefore, depending on the user's security or permission level, different users may have different capabilities in accessing and modifying application and database information. It is also possible that some data structures managed by MTS2100 are assigned at the tenant level, while other data structures are managed at the user level.

[0143] In some embodiments, the user system 2150 and its components can be configured using an application (e.g., a browser) that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 2100 (and additional instances of the MTS, where more than one instance exists) and its components are configurable by an operator using an application that includes computer code executable on a processing element. Therefore, the various operations described herein can be performed by executing program instructions stored on a non-transitory computer-readable medium and executed by a processing element. The program instructions can be stored on a non-volatile medium (e.g., a hard disk), or in any other well-known volatile or non-volatile storage medium or device, such as ROM or RAM, or provided on any medium capable of booting the program code, such as an optical disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, etc. Furthermore, it is well known that the entire program code or portions thereof can be transferred and downloaded from a software source (e.g., via the Internet) or from another server, or transmitted via any other well-known conventional network connection (e.g., extranet, VPN, LAN, etc.) using any well-known communication medium and protocol (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.). It is also understood that the computer code used to implement aspects of the disclosed implementation scheme can be implemented in any programming language capable of executing on a server or server system, such as C, C++, HTML, Java, JavaScript, or any other scripting language, such as VBScript.

[0144] Network 2140 can be a LAN (Local Area Network), WAN (Wide Area Network), wireless network, point-to-point network, star network, token ring network, hub network, or any other suitable configuration. The global interconnected network, often referred to as the "Internet" with a capital "I," is an embodiment of TCP / IP (Transmission Control Protocol and Internet Protocol) networking. However, it should be understood that the disclosed implementation can utilize any and many other types of networks.

[0145] User system 2150 can communicate with MTS2100 using TCP / IP, and at a higher network level, it can communicate using other common Internet protocols such as HTTP, FTP, AFS, WAP, etc. For example, in the case of using HTTP, user system 2150 may include an HTTP client, commonly referred to as a "browser," for sending and receiving HTTP messages from the HTTP server at MTS2100. This server may be implemented as the sole network interface between MTS2100 and network 2140, but other technologies can be used or alternatively. In some implementations, the interface between MTS2100 and network 2140 includes load-sharing functionality, such as a round-robin HTTP request distributor, to balance the load and distribute incoming HTTP requests evenly across multiple servers.

[0146] In various implementations, user system 2150 communicates with application server 2122 to request and update system-level and tenant-level data from MTS 2100, which may require one or more queries to data storage 2112. In some implementations, MTS 2100 automatically generates one or more SQL statements (SQL queries) designed to access the required information. In some cases, user system 2150 may generate requests with a specific format corresponding to at least a portion of MTS 2100. For example, user system 2150 may request the movement of data objects into a specific environment 2124 using object notation describing an object-relational mapping (e.g., a JavaScript object notation mapping) of specified multiple objects.

[0147] Exemplary computer system

[0148] Turn now Figure 22A block diagram of an exemplary computer system 2200 is described, which may implement system 100, database 110, worker node 120, mentor node 135, assistant node 1030, MTS 2100, and / or user system 2150. Computer system 2200 includes a processor subsystem 2280 connected to system memory 2220 and I / O interface 2240 via interconnect 2260 (e.g., system bus). I / O interface 2240 is connected to one or more I / O devices 2250. Although for convenience, in... Figure 22 A single computer system 2200 is shown, but system 2200 can also be implemented as two or more computer systems operating together.

[0149] Processor subsystem 2280 may include one or more processors or processing units. In various embodiments of computer system 2200, multiple instances of processor subsystem 2280 may be coupled to interconnect 2260. In various embodiments, processor subsystem 2280 (or each processor unit within 2280) may include cache or other forms of onboard memory.

[0150] System memory 2220 can be used to store program instructions executable by processor subsystem 2280 to enable system 2200 to perform the various operations described herein. System memory 2220 can be implemented using different physical storage media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.), etc. The memory in computer system 2200 is not limited to main memory, such as memory 2220. Instead, computer system 2200 may also include other forms of memory, such as cache memory in processor subsystem 2280 and auxiliary memory on I / O device 2250 (e.g., hard disk drive, storage array, etc.). In some embodiments, these other forms of memory may also store program instructions executable by processor subsystem 2280. In some embodiments, program instructions that implement database application 200, memory cache 210, and / or orchestration application 300 when executed may be included / stored in system memory 2220.

[0151] According to various implementations, I / O interface 2240 can be any of a variety of types of interfaces configured to connect to and communicate with other devices. In one implementation, I / O interface 2240 is a bridge chip (e.g., a southbridge) from a front end to one or more back end buses. I / O interface 2240 can be connected to one or more I / O devices 2250 via one or more corresponding buses or other interfaces. Embodiments of I / O device 2250 include storage devices (hard disk drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local area network or wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one implementation, computer system 2200 is connected to a network via network interface device 2250 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0152] The implementation of the subject matter of this application includes, but is not limited to, the following embodiments 1 to 20.

[0153] 1. A method comprising:

[0154] The database system's worker nodes receive requests to execute transactions involving writing records, wherein the database system includes multiple worker nodes operable to execute transactions against the database system and multiple supervisor nodes operable to ensure the transaction consistency of the transactions;

[0155] The worker node sends a request to one of the multiple mentor nodes for information regarding operations that facilitate the execution of a transaction; and

[0156] The worker node proceeds to execute a transaction in response to receiving an approval response from a majority of the multiple mentor nodes, where none of the received responses indicate that the transaction should be disapproved.

[0157] 2. The method according to embodiment 1, wherein a record is specified for the request for information and the operation corresponds to a commit transaction, and wherein a given mentor node in the mentor nodes is operable to determine whether to approve the commit transaction based on whether the record conflicts with the history of approved records known to the given mentor node.

[0158] 3. The method according to embodiment 2, wherein a snapshot time of the transaction is specified for the request of the mentor node, and wherein the mentor node is operable to check for conflicts between the record and those records in the history of approved records that were approved after the snapshot time.

[0159] 4. The method according to embodiment 2, wherein the history of approved records includes one or more records that have not yet been refreshed from multiple worker nodes to a persistent storage shared among multiple worker nodes.

[0160] 5. The method according to embodiment 2, wherein a given supervisor node is operable to include a record in the history of approved records in response to sending an approval response to a worker node.

[0161] 6. The method according to embodiment 2, wherein a given mentor node is operable to determine whether to approve a transaction submission without communicating with other mentor nodes among a plurality of mentor nodes.

[0162] 7. The method according to embodiment 2, wherein the request to the mentor node indicates a submission time, and wherein the mentor node is operable to process the request from the worker node at the submission time.

[0163] 8. The method according to Embodiment 2 further includes:

[0164] After confirming that permission to commit the transaction has been granted, the worker node commits the transaction and flushes the record to a persistent storage shared among multiple worker nodes.

[0165] 9. A computer-readable medium having program instructions stored thereon, the program instructions enabling worker nodes of a database system to perform operations, including:

[0166] Receive requests to execute transactions involving writing records, wherein the database system includes multiple worker nodes operable to execute transactions against the database system and multiple supervisor nodes operable to ensure transactional consistency of the transactions;

[0167] Write the records to the worker node's memory cache;

[0168] Send a commit request to one of the multiple mentor nodes to request permission to commit the record; and

[0169] Based on the fact that approval responses were received from a majority of the multiple mentor nodes and none of the received responses indicated that the transaction was not approved, it was determined that permission to commit the record had been granted.

[0170] 10. The medium according to Example 9, wherein the operation further includes:

[0171] The transaction is aborted if it is determined that an approval response has not been received from a majority of the multiple mentor nodes.

[0172] 11. The medium according to Example 9, wherein the operation further includes:

[0173] In response to receiving a disapproval response from at least one mentor node, the transaction is aborted. The disapproval response indicates that the transaction is not approved based on a conflict between the record and another record known to at least one mentor node.

[0174] 12. The medium according to Embodiment 9, wherein the submission request specifies the proposed time period for submitting the transaction, wherein the operation further includes:

[0175] In response to receiving a disapproval response from at least one mentor node, the transaction is aborted. The disapproval response indicates that the commit transaction is not approved based on the fact that the commit request arrives after the proposed time period.

[0176] 13. The medium according to Example 9, wherein the operation further includes:

[0177] Receive requests for records from another worker node; and

[0178] After the transaction is committed, the record is provided to another worker node.

[0179] 14. A method comprising:

[0180] The mentor node receives a request from the worker node for permission to commit a transaction involving records. The mentor node and the worker node are part of a database system that has multiple worker nodes operable to execute transactions and multiple mentor nodes operable to ensure the transactional consistency of transactions.

[0181] The mentor node determines whether a record conflicts with a set of records identified by other worker nodes among multiple worker nodes; and

[0182] Based on this determination, the mentor node sends a response to the worker node indicating whether permission to commit the transaction has been granted.

[0183] 15. The method according to embodiment 14, wherein the request specifies a record, and the response indicates that permission to submit the transaction has been granted, and wherein the method further comprises:

[0184] The mentor node stores historical information, which records entries as committed, regardless of whether worker nodes have committed transactions.

[0185] 16. The method according to embodiment 14, wherein the request specifies the snapshot time and proposal commit time associated with the transaction, and wherein determining includes:

[0186] Determine if there are any conflicting records in this group of records that were submitted between the snapshot time and the proposal submission time.

[0187] 17. The method according to embodiment 16, wherein the determination is performed after the local clock of the instructor node indicates that the time is equal to the submission time.

[0188] 18. The method according to embodiment 15, wherein the response indication has not yet granted permission to submit the transaction, and wherein the worker node is operable to submit the transaction in response to receiving approval from a majority of the mentor nodes among a plurality of mentor nodes, even though the response indication has not granted permission to submit.

[0189] 19. The method according to embodiment 15, wherein the set of records known to the mentor node includes at least one record that is not included in another set of records known to another mentor node among a plurality of mentor nodes.

[0190] 20. The method according to embodiment 15, wherein the response provides an indication to the worker node when it is allowed to commit a transaction.

[0191] The published text includes references to “implementation schemes,” which are non-limiting ways of realizing the disclosed concepts. References to “implementation scheme,” “an implementation scheme,” “a specific implementation scheme,” “some implementation schemes,” “various implementation schemes,” etc., do not necessarily refer to the same implementation scheme. Numerous feasible implementation schemes are envisioned, including detailed specific implementation schemes and modifications or substitutions falling within the spirit or scope of the published text. Not all implementation schemes necessarily exhibit any or all of the potential advantages described herein.

[0192] The disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all implementations of these embodiments necessarily exhibit any or all of the potential advantages. Whether an advantage is realized for a particular implementation depends on many factors, some of which are beyond the scope of the disclosure. In fact, there are many reasons why an implementation falling within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include circuitry outside the scope of the disclosure that, when combined with one of the disclosed embodiments, negates or diminishes one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular implementation (e.g., the implementation technique or tool) may also negate or diminish the disclosed advantages. Even assuming skillful implementation, the realization of an advantage may still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, the inputs provided to a particular implementation may prevent one or more problems addressed in the disclosure from occurring in a particular context, and as a result, the benefits of its solution may not be realized. Given the possibility of factors beyond the disclosure, it is expressly stated that any potential advantages described herein should not be construed as a limitation of the claims that must be satisfied to prove infringement. Rather, the identification of such potential advantages is intended to illustrate the type of improvement available to a designer who benefits from the disclosure. Describing these advantages permissively (e.g., stating that a particular advantage "may occur") is not intended to convey doubt about whether these advantages can actually be realized, but rather to acknowledge that the realization of these advantages often depends on the technological reality of additional factors.

[0193] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims drafted based on the disclosure, even if only a single embodiment is described for a particular feature. The disclosed embodiments are intended to be illustrative rather than limiting, and there is no statement to the contrary in the disclosure. Therefore, this application is intended to allow for claims covering the disclosed embodiments, as well as substitutions, modifications, and equivalents that will be apparent to those skilled in the art who benefit from the disclosure.

[0194] For example, features in this application can be combined in any suitable manner. Therefore, during the examination of this application (or an application claiming priority thereto), new claims can be made for any such combination of features. In particular, referring to the appended claims, features of dependent claims can be combined with features of other dependent claims, including claims dependent on other independent claims, where appropriate. Similarly, features from various independent claims can be combined where appropriate.

[0195] Therefore, while appended dependent claims can be drafted such that each dependent claim depends on a single other claim, additional dependent claims may also be considered. Any combination of features in dependent claims consistent with the published text is contemplated and may be claimed in this application or another application. In short, combinations are not limited to those specifically listed in the appended claims.

[0196] Where appropriate, it is also anticipated that claims drafted in one format or statutory type (e.g., apparatus) are intended to support corresponding claims in another format or statutory type (e.g., method).

[0197] Because the published text is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. Therefore, it is hereby announced that the following paragraphs, as well as the definitions provided throughout the publication, will be used to determine how to interpret the claims drafted based on the published text.

[0198] Unless the context clearly specifies otherwise, references to the singular form of an item (i.e., a noun or noun phrase preceded by "a," "an," or "the") are intended to mean "one or more." Therefore, without accompanying context, a reference to "item" in a claim does not exclude additional instances of that item. "A plurality of" items refers to a group of two or more items.

[0199] The word "may" used in this article is permissive (i.e., possible, able) rather than mandatory (i.e., required).

[0200] The terms “including” and “comprise” and their forms are open-ended, meaning “including but not limited to”.

[0201] When the term “or” is used in public texts concerning a list of options, it is generally understood to be used in an inclusive sense unless the context otherwise specifies. Therefore, a statement of “x or y” is equivalent to “x or y or both,” thus encompassing 1) x but excluding y, 2) y but excluding x, and 3) both x and y. On the other hand, phrases such as “x or y, but excluding both” indicate that “or” is used in an exclusive sense.

[0202] The statements “w, x, y, or z or any combination thereof” or “...at least one of w, x, y, and z” are intended to cover all possibilities involving a single element up to the total number of elements in the set. For example, given the set [w, x, y, z], these terms cover any single element of the set (e.g., w, but excluding x, y, or z), any two elements (e.g., w and x, but excluding y or z), any three elements (e.g., w, x, and y, but excluding z), and all four elements. The phrase “...at least one of w, x, y, and z” therefore refers to at least one element in the set [w, x, y, z], thus covering all possible combinations in the list of elements. This phrase should not be interpreted as requiring the existence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0203] In public text, various “labels” may precede nouns or noun phrases. Unless the context otherwise specifies, different labels used for a feature (e.g., “first circuit,” “second circuit,” “specific circuit,” “given circuit,” etc.) refer to different instances of that feature. Furthermore, when applied to features, the labels “first,” “second,” and “third” do not indicate any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise specified.

[0204] The phrase "based on" is used to describe one or more factors that influence the determination. This term does not exclude the possibility that additional factors might influence the determination. That is, the determination can be based solely on the specified factors or on the specified factors along with other unspecified factors. Consider the phrase "A is determined based on B." This phrase specifies that B is a factor used to determine A or a factor influencing the determination of A. This phrase does not exclude the possibility that the determination of A could also be based on another factor, such as C. This phrase is also intended to cover implementations where A is determined solely based on B. As used herein, the phrase "based on" is synonymous with the phrase "at least partially based on."

[0205] The phrases “responding to” and “responsive” describe one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may influence or otherwise trigger the effect, whether in conjunction with or independently of a specific factor. That is, an effect may respond only to those factors, or it may respond to the specified factor as well as other unspecified factors. Consider the phrase “responding to B to execute A.” This phrase specifies that B is a factor that triggers the execution of A or triggers a specific result of A. This phrase does not exclude that the execution of A may also respond to some other factor, such as C. This phrase also does not exclude that the execution of A may jointly respond to B and C. This phrase is also intended to cover implementations where A is executed only in response to B. As used herein, the phrase “responding to” is synonymous with the phrase “at least partially responding to.” Similarly, the phrase “responding to” is synonymous with the phrase “at least partially responding to.”

[0206] In public texts, different entities (which may be referred to differently as “units,” “circuits,” other components, etc.) may be described or required to be “configured” to perform one or more tasks or operations. This expression—[entity] configured to perform one or more tasks—is used herein to refer to a structure (i.e., something physical). More specifically, this expression is used to indicate that the structure is set to perform one or more tasks during operation. It can be said that a structure is “configured” to perform some tasks, even if the structure is not currently being operated. Therefore, an entity described or stated as “configured to” perform a task refers to something physical, such as a device, circuit, system having processor units and memory storing program instructions executable to perform that task, etc. This phrase is not used herein to refer to intangible things.

[0207] In some cases, various units / circuits / components may be described herein as performing a set of tasks or operations. It should be understood that these entities are "configured" to perform those tasks / operations, even if not specifically indicated otherwise.

[0208] The term "configured as" does not imply "configurable as." For example, an unprogrammed FPGA is not considered "configured as" to perform a specific function. However, such an unprogrammed FPGA can be "configurable as" to perform that function. After proper programming, an FPGA can be said to be "configured as" to perform a specific function.

[0209] For the purposes of a U.S. patent application based on a published text, a statement in the claims describing a structure “configured” to perform one or more tasks is not expressly intended to invoke 35 U.S.SC § 112(f) for that claim element. If the applicant wishes to invoke section 112(f) during the proceedings of a U.S. patent application based on a published text, a “component” structure for [performing a function] will be used to state the claim element.

[0210] Different “circuits” can be described in public texts. These circuits or “circuit systems” constitute hardware comprising various types of circuit elements, such as combinational logic, clock-controlled storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memories (e.g., random access memory, embedded dynamic random access memory), programmable logic arrays, etc. Circuits can be custom-designed or taken from standard libraries. In various implementations, circuits may appropriately include digital components, analog components, or a combination of both. Certain types of circuits are often referred to as “units” (e.g., decoding units, arithmetic logic units (ALUs), functional units, memory management units (MMUs), etc.). Such units also refer to circuits or circuit systems.

[0211] Therefore, the circuits / units / components and other elements disclosed in the accompanying drawings and described herein include hardware elements, such as those described in the preceding paragraphs. In many cases, the internal arrangement of hardware elements in a particular circuit can be specified by describing the function of the circuit. For example, a particular "decoding unit" can be described as having the function of executing "the opcode of a processing instruction and routing that instruction to one or more of a plurality of functional units," indicating that the decoding unit is "configured" to perform that function. To those skilled in the art of computers, such a functional description is sufficient to suggest a set of possible structures for the circuit.

[0212] In various implementations, as discussed in the preceding paragraphs, circuits, cells, and other elements can be configured to implement functional or operational definitions. The arrangement of these circuits / cells / components relative to each other and the manner in which they interact form a microarchitecture definition of hardware, which is ultimately fabricated in an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitecture definition. Therefore, a microarchitecture definition is considered by those skilled in the art to be a structure from which many physical implementations can be derived, all falling within the broader structure described by the microarchitecture definition. That is, a person skilled in the art with a microarchitecture definition provided in the published text can implement the structure by encoding the description of the circuits / cells / components in a Hardware Description Language (HDL) (e.g., Verilog or VHDL) without excessive experimentation and the application of ordinary skills. HDL descriptions are often expressed in a seemingly functional manner. However, to those skilled in the art, such HDL descriptions are a way of translating the structure of a circuit, cell, or component into the next level of implementation details. Such HDL descriptions can take the form of property code (typically non-synthesizable), register transfer language (RTL) code (as opposed to property code, typically synthesizable), or structural code (e.g., a netlist specifying logic gates and their connections). HDL descriptions can then be synthesized against a cell library designed for a given integrated circuit manufacturing technology and can be modified for timing, power, and other reasons to produce a final design database, which is then transferred to a foundry to generate masks and ultimately manufacture integrated circuits. Some hardware circuitry or portions thereof can also be custom-designed in a schematic editor and captured into the integrated circuit design along with the synthesized circuitry. Integrated circuits can include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.) and interconnections between transistors and circuit elements. Some implementations may realize multiple integrated circuits interconnected to implement hardware circuitry, and / or discrete elements may be used in some implementations. Alternatively, HDL designs can be synthesized into a programmable logic array such as a field-programmable gate array (FPGA) and implemented in the FPGA. This decoupling between the design of a set of circuits and their subsequent low-level implementations often results in a situation where the circuit or logic designer never specifies a particular set of structures for the low-level implementation beyond describing what the circuit is configured to do, as this process is performed at different stages of the circuit implementation process.

[0213] The fact that many different low-level combinations of circuit elements can be used to implement the same circuit specifications results in a large number of equivalent circuit structures. As mentioned above, these low-level circuit implementations can vary depending on the manufacturing technology, the foundry chosen to manufacture the integrated circuits, the cell library provided for a particular project, and so on. In many cases, the choice made by different design tools or methods to produce these different implementations can be arbitrary.

[0214] Furthermore, for a given implementation, a single implementation of the circuit's specific functional specifications typically involves a large number of devices (e.g., millions of transistors). Therefore, the sheer volume of such information makes it impractical to provide a complete description of the low-level structure used to implement a single implementation, let alone the numerous equivalent possible implementations. For this reason, the published text uses functional abbreviations commonly used in the industry to describe the circuit's structure.

Claims

1. A method comprising: A request to execute a transaction is received by a worker node of a database system, the transaction involving writing records, wherein the database system includes multiple worker nodes capable of operating to execute transactions against the database system and multiple supervisor nodes capable of operating to ensure the transaction consistency of the transaction; The worker node sends a submission request to one of the plurality of mentor nodes, granting permission to submit the transaction. and The worker node proceeds to submit the transaction in response to receiving an approval response from a majority of the mentor nodes, wherein a given mentor node is operable to determine whether to approve the submission of the transaction based on whether the record conflicts with a history of approved records known to the given mentor node.

2. The method of claim 1, wherein the submission request specifies the record.

3. The method of claim 2, wherein the commit request of the mentor node is assigned a snapshot time of the transaction, and wherein the given mentor node is operable to check for conflicts between the record and those records in the history of the approved records that were approved after the snapshot time.

4. The method of claim 2 or 3, wherein the history of the approved records includes one or more records that have not yet been flushed from the plurality of worker nodes to a persistent storage shared among the plurality of worker nodes.

5. The method of claim 2, wherein the given mentor node is operable to include the record in the history of the approved records in response to sending an approval response to the worker node.

6. The method of claim 2, wherein the given mentor node is operable to determine whether to approve the submission of the transaction without communicating with other mentor nodes among the plurality of mentor nodes.

7. The method of claim 2, wherein the submit request to the mentor node is given a submit time, and wherein the given mentor node is operable to process the submit request from the worker node at the submit time.

8. The method according to claim 2, further comprising: The worker node refreshes the record to a permanent storage shared among the multiple worker nodes.

9. A computer-readable medium having program instructions stored thereon, the program instructions enabling worker nodes of a database system to perform operations, the operations including: Receive a request to execute a transaction, the transaction involving writing records, wherein the database system includes multiple worker nodes capable of operating to execute transactions against the database system and multiple supervisor nodes capable of operating to ensure the transaction consistency of the transaction; Write the record to the worker node's memory cache; Send a commit request to one of the multiple mentor nodes to request permission to commit the transaction; Based on receiving an approval response from a majority of the mentor nodes, it is determined that permission to submit the transaction has been granted, wherein a given mentor node is operable to determine whether to approve the submission of the transaction based on whether the record conflicts with a history of approved records known to the given mentor node. and After confirming that permission has been granted to submit the transaction, submit the transaction.

10. The computer-readable medium of claim 9, wherein the operation further comprises: The transaction is aborted if it is determined that an approval response has not been received from a majority of the multiple mentor nodes.

11. The computer-readable medium of claim 9, wherein the operation further comprises: In response to receiving a disapproval response from at least one mentor node, the transaction is aborted, the disapproval response indicating that the submission of the transaction is not approved based on a conflict between the record and another record known to the at least one mentor node.

12. The computer-readable medium of claim 9, wherein the commit request specifies a proposed time period for committing the transaction, and wherein the operation further comprises: In response to receiving a disapproval response from at least one mentor node, the transaction is aborted, the disapproval response indicating that the submission of the transaction is not approved based on the submission request arriving after the proposed time period.

13. The computer-readable medium of claim 9, wherein the operation further comprises: Receive a request for the record from another worker node; and After the transaction is committed, the record is provided to the other worker node.

14. A computer, comprising: At least one processor; and A memory having program instructions stored thereon, the program instructions being executable by the at least one processor to perform the method according to any one of claims 1 to 8.

Images