Key Spatial Reference

Abstract

Techniques related to tracking record writes for a key space across a set of database nodes are disclosed. A first database node of a database system can receive a request to perform a database transaction including writing a particular record for a key included in the key space. The first database node can access a key space reference catalog that stores a plurality of indications of when the key space was written by database nodes of the database system. In response to determining that a second database node has written a record for the key space within a particular time frame, the first database node can send to the second database node a request for information indicating whether the second database node has written a record for the key. Based on the response received from the second database node, the first database node can determine whether to write the particular record.

Description

[Technical Field]

[0001] The present disclosure relates generally to database systems, and more particularly to tracking record writes to a key space across a set of database nodes. [Background technology]

[0002] Modern database systems routinely implement management systems that can store collections of information in an organized manner that users can efficiently access and manipulate. In some cases, these management systems maintain log-structured merge trees (LSM trees) with multiple levels, each storing information as key-value pairs in database records. LSM trees typically include two high-level components: an in-memory buffer and persistent storage. During operation, the database system receives transaction requests to process transactions that involve writing database records to persistent storage. The database system first writes the database records to the in-memory buffer before later flushing them to persistent storage. As part of flushing the database records, the database system writes the database records to a new file that is stored in persistent storage. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram illustrating example elements of a system that can use key space references to track record writes to a key space across a set of database nodes, according to some embodiments. [Figure 2A] FIG. 2 is a block diagram illustrating exemplary elements of a keyspace permission, according to some embodiments. [Figure 2B] FIG. 2 is a block diagram illustrating exemplary elements of a key spatial reference, according to some embodiments. [Figure 3]FIG. 1 is a block diagram illustrating exemplary elements associated with a database node that processes active database transactions using key spatial references, according to some embodiments. [Figure 4] FIG. 1 is a block diagram illustrating exemplary elements associated with a database node that updates a key spatial reference after committing a database transaction, according to some embodiments. [Figure 5] FIG. 1 is a flow diagram illustrating an example method related to processing database transactions using key spatial references, according to some embodiments. [Figure 6] FIG. 1 is a flow diagram illustrating an example method related to processing database transactions using key spatial references, according to some embodiments. [Figure 7] FIG. 1 is a block diagram illustrating elements of a multi-tenant system, according to some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating elements of a computer system that implements various systems described in this disclosure, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] As described above, modern database systems may maintain an LSM tree containing database records with key-value pairs. Often, the database system contains a single active database node responsible for writing database records to the persistent storage component of the LSM tree. However, in some cases, the database system contains multiple database nodes writing database records to the LSM tree. To prevent multiple database nodes from writing database records to the same database key at nearly the same time, database nodes may be provisioned with key permissions that identify the keys under which those database nodes are authorized to write the corresponding database records.

[0005] However, in various cases, it may be desirable to re-provision key grants from a previous owner database node to a new owner database node. As an example, an administrator may want to perform a rolling upgrade of database nodes to a new software version. To prevent delays in database transactions whose keys are associated with key grants provisioned to that database node, it may be desirable to re-provision those key grants to another database node so that the other database node can perform those database transactions while the previous database node is taken offline to be upgraded. However, in many cases, at the time re-provisioning is to occur, the previous database node still has in-progress database transactions tied to the key grants being re-provisioned. Restarting those in-progress database transactions is undesirable because some of those database transactions may take a long time to execute (e.g., an hour). The present disclosure addresses, among other things, this technical problem of how to handle in-progress transactions when re-provisioning key grants to a new owner database node.

[0006] More specifically, this disclosure describes various techniques for allowing ongoing transactions for a key space to commit or roll back at the previous owner database node, while allowing new transactions for the same key space to be initiated at the new owner database node. However, because there may be transactions running on two or more separate database nodes that commit database records for the same key space, a mechanism is desired to ensure that those writes do not cause data corruption in the database. Thus, this disclosure further addresses the technical problem of ensuring consistency within a database system when there are multiple transactions running across multiple database nodes for the same key space.

[0007] In various embodiments described below, keyspace references are used to track where database record writes for a particular keyspace should occur and where database record writes for this keyspace have previously occurred. When permission to write to a keyspace is first provisioned to a first database node in a database system, a first keyspace reference can be created that indicates that database record writes for that keyspace should occur only at the first database node. As a result, database nodes in the database system direct write requests to the first database node based on the first keyspace reference, and this keyspace reference can be included in a reference catalog that stores keyspace references for various keyspaces in the database system.

[0008] At some point during operation of the database system, a particular key space may be reprovisioned from a first database node to a second database node. When that key space is provisioned to the second database node, in some embodiments, a second key space reference is created with a state of "open," indicating that database record writes for that key space should occur at the second database node. The first key space reference associated with the first database node may be set to a state of "closed," indicating that database record writes for that key space will no longer occur at the first database node. Consequently, database nodes in the database system direct write requests to the second database node instead of the first database node. As a result, database record writes for the key space can continue to occur even if the first database node is taken offline. While new database record writes are directed to the second database node, the first database node may still have active transactions that have written but not committed records for the key space. As a result, a situation can arise where a second database node writes and commits a record before a first database node's record for the same key, resulting in an inconsistent database because the later record appears to have been written before the earlier record, even though the earlier record was written before the later record.

[0009] To avoid this situation, in various embodiments, when a second database node wants to write a database record for a key in a particular key space, the second database node revisits the key space reference in the reference catalog. Based on the key space reference, the second database node can determine whether its record write for the key space potentially conflicts with a record write for another database node within a particular time frame. For example, the second database node can determine that there is an ongoing database transaction at a first database node that has written a database record for the key space. If the second database node determines that at least one database record should be written within the key space, in various embodiments, the second database node sends a request to each database node that may have written a database record specific to the key to which the second database node intends to write the database record. For example, the second database node can send a request to the first database node to determine whether one of the database records it has written for the key space is associated with the key. If no record has been written for that key, the second database node writes the record it desires for the key. If there is a record written for the key by another database node, the second database node can abort the transaction or ensure that its record is committed after the other database node's record. By being able to learn about database record writes by other database nodes for a given key, a database node can prevent itself from writing and committing a database record for the same key in a way that would result in a corrupted database.

[0010] In some cases, a key space may be moved around several database nodes, resulting in several "read" key space references (which identify where a database record write for the key space previously occurred) and one "write" key space reference (which identifies where a database record write for the key space should occur). Thus, when a database node wants to read or write a database record for a key, it may need to check multiple key space references and their associated database nodes for the record corresponding to the key.

[0011] The techniques of the present disclosure may be advantageous over previous approaches because they allow for the reprovisioning of key spaces among database nodes while providing a mechanism for ensuring consistency in the database system by preventing those database nodes from committing records in the wrong chronological order. In particular, these techniques allow for the reprovisioning of key spaces without having to restart ongoing transactions. Because database transactions can take a long time to execute, not restarting ongoing transactions conserves database resources and saves time. Furthermore, by not having to incur the cost of restarting ongoing transactions as part of a key space transfer, updates are less expensive to perform at the database nodes. Thus, the overall operation of the database system is improved. Next, an exemplary application of the techniques of the present disclosure will be discussed, beginning with FIG. 1 .

[0012] Referring now to FIG. 1 , a block diagram of system 100 is shown. System 100 includes a set of components that may be implemented using hardware or a combination of hardware and software routines. In the illustrated embodiment, system 100 includes database 110 (having LSM file 115), database nodes 120A and 120B, and catalog manager node 140. As further illustrated, catalog manager node 140 includes storage catalog 145 having keyspace permissions 124 and keyspace references 126, one or more of which are stored in database node 120. As further illustrated, database nodes 120 include respective in-memory caches 130 that store database records 132 for keys 134. In some embodiments, system 100 is implemented differently than illustrated. As an example, there may be no catalog manager node 140; instead, the storage catalog 145 may be stored in a storage area shared by the database nodes 120, such that the database nodes 120 maintain the key space permissions 124 and key references 126. Furthermore, although the techniques of this disclosure are discussed with reference to LSM trees, these techniques may be applied to other types of database implementations in which multiple nodes write and commit database records.

[0013] In various embodiments, system 100 implements a platform service (e.g., a customer relationship management (CRM) platform service) that enables users of that service to develop, execute, and manage applications. System 100 may be a multi-tenant system that provides various functions to multiple users / tenants hosted by the multi-tenant system. Thus, system 100 may execute software routines from various different users (e.g., providers and tenants of system 100) as well as provide code, web pages, and other data to users, databases, and other entities associated with system 100. Illustratively, for example, system 100 includes database node 120 that can store, manipulate, and retrieve data from LSM file 115 of database 110 for users of system 100.

[0014] Database 110, in various embodiments, is a collection of information organized in a manner that allows that information to be accessed, stored, and manipulated. Accordingly, database 110 may include support software that enables database nodes 120 to perform operations (e.g., access, store, etc.) on the information stored in database 110. In some embodiments, database 110 is implemented by one or more storage devices connected together over a network (e.g., a storage attached network (SAN)) and configured to store information redundantly to prevent data loss. The storage devices may persistently store data, and thus database 110 may function as persistent storage. In various embodiments, database records 132 written to LSM file 115 by one database node 120 are accessible by other database nodes 120. LSM file 115 may be stored as part of a log-structured merge tree (LSM tree) implemented in database 110.

[0015] An LSM tree is a data structure that, in various embodiments, stores LSM files 115 in an organized manner using a level-based scheme. The LSM tree can include two high-level components: an in-memory component implemented in an in-memory cache 130, and an on-disk component implemented in the database 110. In some embodiments, the in-memory cache 130 is considered separate from the LSM tree. Database nodes 120 can initially write database records 132 to their in-memory cache 130. When the cache 130 becomes full and / or at certain points in time, the database nodes 120 can flush their database records 132 to the database 110. As part of flushing these database records 132, in various embodiments, the database nodes 120 write the database records 132 to a new set of LSM files 115 in the database 110.

[0016] In various embodiments, an LSM file 115 is a set of database records 132. A database record 132 may be a key-value pair containing data and a corresponding database key 134 that can be used to look up the database record. For example, a database record 132 may correspond to a row of data in a database table, where the database record 132 specifies values for one or more attributes associated with the database table. In various embodiments, a file 115 is associated with one or more database key ranges defined by the keys 134 of the database records contained in the LSM file 115. Consider an example in which the file 115 stores three database records 132 associated with keys 134 "XYA," "XYW," and "XYZ," respectively. These three keys 134 span the database key range XYA→XYZ, and therefore the LSM file 115 is associated with that database key range.

[0017] Database node 120, in various embodiments, is hardware, software, or a combination thereof that can provide database services such as data storage, data retrieval, and / or data manipulation. These database services may be provided to other components within system 100 or to components external to system 100. For example, database node A may receive a request from an application server to perform database transaction 122. Database transaction 122, in various embodiments, is a logical unit of work (e.g., a specified set of database operations) performed in connection with database 110. As an example, processing database transaction 122 may include executing an SQL SELECT command to select one or more rows from one or more database tables. The contents of the row may be specified in database records 132, and thus database node 120 may return one or more database records 132 corresponding to the selected one or more table rows. In various cases, performing database transaction 122 may include database node 120 writing one or more database records 132 to an LSM tree. In various embodiments, the database node 120 first writes those database records 132 to its in-memory cache 130 before flushing those database records 132 to the database 110 .

[0018] In-memory cache 130 is, in various embodiments, a buffer that stores data in memory (e.g., random access memory) of database node 120. TMA memstore is an example of an in-memory cache 130. As previously mentioned, a database node 120 may initially write database records 132 to its in-memory cache 130. In some cases, the latest / most recent version of a row in a database table may be found in the database record 132 stored in the in-memory cache 130. However, in some embodiments, the database records 132 written to a database node 120's in-memory cache 130 are invisible to other database nodes 120. That is, other database nodes 120 do not know what information is stored in the in-memory cache 130 of the database node 120 without asking. To prevent database record conflicts when one database node 120 may not know about database records 132 written by another database node 120, in various embodiments, database nodes 120 are provisioned with keyspace permissions 124 that control which database records 132 can be written by a given database node 120. Thus, key space permissions 124 can prevent two or more database nodes 120 from writing database records 132 for the same database key 134 within a certain time interval, preventing these database nodes 120 from flushing conflicting database records 132 to database 110.

[0019] Keyspace permission 124, in various embodiments, is information that identifies a keyspace and the corresponding owner of that keyspace. For example, as shown, database node 120B is provisioned with keyspace permission 124, and therefore database node 120B is permitted to write to its in-memory cache 130 database records 132 whose corresponding keys 134 fall within the keyspace associated with that keyspace permission 124. In various embodiments, keyspace permission 124 is provisioned to at most one database node 120 at any time. Thus, while keyspace permission 124 is provisioned to database node 120B, database node 120A is not permitted to write database records 132 whose corresponding keys 134 fall within the keyspace associated with that keyspace permission 124. To be permitted to write database records 132 for a particular key 134, in various embodiments, database node 120 can issue a permission request to catalog manager node 140 specifying the key 134. In some cases, the authorization request may specify multiple keys 134 (key spaces).

[0020] Catalog manager node 140, in various embodiments, facilitates the management and distribution of keyspace permissions 124 and keyspace references 126 among database nodes 120. As part of facilitating the management and distribution of keyspace permissions 124, in various embodiments, catalog manager node 140 updates and distributes keyspace permissions 124 in response to receiving requests from database nodes 120. For example, catalog manager node 140 may receive a request from database node 120B for permission to write record 132 to keyspace “XY.” In response, catalog manager node 140 may determine whether keyspace permissions 124 for this keyspace have already been provisioned on database node 120. If it has not been provisioned, catalog manager node 140 may update keyspace permissions 124 to provision the keyspace on database node 120B and then notify all database nodes 120, including database node 120B, of the provisioning of the keyspace. If permissions for the requested keyspace have been provisioned, in various embodiments, catalog manager node 140 identifies the owning database node 120 and sends a request to that database node 120 to relinquish the requested keyspace. That database node 120 can send a response indicating that the keyspace has been relinquished, after which catalog manager node 140 can update keyspace permissions 124 and provision the keyspace to database node 120B, and then notify all database nodes 120 about the re-provisioning of the keyspace. In various embodiments, when a keyspace is provisioned to a database node 120, a keyspace reference 126 is created.

[0021] In various embodiments, a key space reference 126 includes information identifying a time window during which a specified database node 120 wrote a database record 132 having a key 134 that falls within a specified key space. For example, a key space reference 126 may indicate that database node 120A wrote a database record 132 belonging to key space “XY” during a time window defined from a first transaction commit number (XCN) to a second, later-occurring XCN. In various embodiments, one of the key space references 126 for a key space may identify a database node 120 that is currently authorized to write database records 132 belonging to that key space. A key space reference 126 that identifies a database node 120 that is authorized to write to a key space is referred to herein as a “write” key space reference 126. This is in contrast to a “read” key space reference 126, which identifies a database node 120 that previously wrote to a key space but is no longer authorized to write to the key space (unless that database node 120 has its associated key space permissions 124 re-provisioned). Consequently, in various embodiments, a catalog manager node 140 may store multiple key space references 126 for the same key space, one of which is a write key space reference 126 and the other is a read key space reference 126. When a key space is reprovisioned to another database node 120, the current write key space reference 126 can be converted to a read key space reference 126, and the newly created key space reference 126 can become a write key space reference 126.

[0022] When a database node 120 desires that a database record 132 be written for a particular key 134, in various embodiments, the database node 120 sends a catalog request 142 for one or more key space references 126 to a catalog manager node 140. In various cases, the catalog manager node 140 may return the write key space references 126 to the requesting database node 120 as part of a catalog response 144. The requesting database node 120 may then send a record write request to database nodes 120 that are identified by the write key space references 126 as being authorized to write to the key space containing the appropriate key 134. Consider an example in which database node 120A desires to write a database record 132 for key 134 "X," but database node 120B has been provisioned with a key space permission 124 that encompasses key 134 "X." To write that database record 132, database node 120A can obtain a write keyspace reference 126 from storage catalog 145 that indicates that database node 120B is authorized to write the database record 132 for the keyspace that contains key 134 "X." As a result, database node 120A can send a record write request to database node 120B in which the particular database record 132 is to be written.

[0023] When a database node 120 wishes to write a database record 132 for key 134 to its own in-memory cache 130, in various embodiments, the database node 120 may send a catalog request 142 of one or more key space references 126 to a catalog manager node 140. The catalog manager node 140 may then return one or more read key space references 126 to the requesting database node 120 as part of a catalog response 144. In various embodiments, the database node 120 examines those read key space references 126 to determine whether another database node 120 has written to the associated key space within a particular time frame, potentially resulting in a commit conflict between the database nodes 120. As an example, database node 120B may wish to write database record 132B with key 134B. Database node 120B may determine from the set of read key space references 126 that database node 120A wrote record 132 for the key space containing key 134B, but that record 132 has not been committed. As shown, database node 120A is writing database record 132A having key 134A that belongs to this example key space.

[0024] In various embodiments, in response to determining that another database node 120 has written to a key space within a particular time frame, database node 120 sends record request 136 to database node 120 to determine whether database node 120 has specifically written record 132 for a particular key 134. Continuing the previous example, database node 120B can send record request 136 to database node 120A to determine whether it has written a database record for key 134B. Database node 120A can send record response 138 indicating whether database record 132 for key 134B has been written. If database record 132 has been written for key 134B, database node 120B can abort database transaction 122 associated with the record write or can delay the record write until other database records 132 have been committed by database node 120A. If a database record 132 has not been written for key 134B, database node 120B may write and commit a database record 132 for key 134B.

[0025] By maintaining key space permissions 124 and key space references 126 in the system 100, database nodes 120 may be able to determine where database records 132 should be written and where they can find previously written but committed database records 132. Thus, when a user reprovisions one or more key spaces to other database nodes 120 as part of updating a particular database node 120, all database nodes 120 in the system 100 can determine from the key space permissions 124 and key space references 126 which database nodes 120 are authorized to write database records 132 for the reprovisioned key spaces. As a result, database record writes to these key spaces can continue to occur while that particular database node 120 is being updated. Furthermore, ongoing database transactions 122 on the particular database node 120 being updated can be committed without conflicts, because other database nodes 120 provisioned with these key spaces can learn about what database record writes have occurred at the particular database node 120. Thus, these database nodes 120 can prevent themselves from writing and committing database records 132 that conflict with database records written at a particular database node 120 .

[0026] 2A, a block diagram of exemplary elements of a keyspace permission 124 is shown. In the illustrated embodiment, the keyspace permission 124 specifies a keyspace 210 and a node indication 220. In some embodiments, the system 100 does not include a keyspace permission 124, but uses a keyspace reference 126 to fulfill their role. In some embodiments, the keyspace permission 124 is implemented differently than shown. As an example, the keyspace permission 124 may specify an identifier that distinguishes it from other keyspace permissions 124 and allows the keyspace permission 124 to be looked up in the storage catalog 145.

[0027] In various embodiments, a key space 210 corresponds to a range of keys 134 defined by a minimum key 134 and a maximum key 134, as shown in FIG. 1 . For example, a key space 210 may correspond to a range of keys 134 from “AAAAA” to “EEEEE.” In some embodiments, a key space 210 corresponds to multiple key ranges (e.g., “AAAAAA” to “BBBBB” and “CCCCC” to “EEEEE”). In some cases, the key range of a key space 210 may be specified by a single key prefix instead of a minimum key 134 and a maximum key 134. For example, a key space 210 may specify “XY,” which encompasses all keys 134 with the prefix “XY.” In some embodiments, there is a single key space permission 124 for a given key space 210, such that at most one database node 120 is authorized to write database records 132 to that key space 210. Thus, when a non-owning database node 120 wants to write a database record 132 for a particular keyspace 210, the non-owning database node 120 can either request ownership of the keyspace 210 or issue a request to the owning database node 120 to write the database record 132. If ownership of a keyspace 210 is to be transferred, in various embodiments, the node index 220 of the corresponding keyspace grant 124 is updated to reflect the new owning database node 120.

[0028] The node index 220, in various embodiments, indicates the database node 120 associated with the keyspace grant 124. In some embodiments, the database node 120 is assigned a log window that defines a list of log files to which the database node 120 is authorized to write log information. The node index 220 specifies an identifier for the log window and may therefore be associated with the database node 120 via the log window. The node index 220 may be updated in response to the occurrence of various events. For example, the catalog manager node 140 may update the node index 220 for a particular keyspace grant 124 after receiving a request from the database node 120 for ownership of the keyspace 210 corresponding to that particular keyspace grant 124. As another example, ownership of the keyspace 210 may be transferred away from a database node 120 receiving a software update for its database application, and therefore the node index 220 may be updated to remove ownership of the keyspace 210 from that database node 120.

[0029] In various embodiments, a keyspace permission 124 can be split into multiple keyspace permissions 124. For example, a keyspace permission 124 specifying keyspace 210 "XY" may be split into two keyspace permissions 124, one specifying keyspace 210 "XYA-XYM" and the other specifying keyspace 210 "XYN-XYZ." In various embodiments, multiple keyspace permissions 124 can be merged into a single keyspace permission 124. For example, the two keyspace permissions 124 from the previous example may be merged into a single keyspace permission 124 specifying keyspace 210 "XY."

[0030] 2B, a block diagram of exemplary elements of key space reference 126 is shown. In the illustrated embodiment, key space reference 126 specifies key space 210, node index 220, epoch range 230, and state 240. In some embodiments, key space reference 126 is implemented differently than shown; for example, key space reference 126 may not specify state 240.

[0031] As previously described, a key space reference 126 may be created when a database node 120 is granted permission to write to a key space 210. When a key space reference 126 is created, in various embodiments, the key space 210 of the key space reference 126 is set to identify the granted key space, and the node index 220 is set to identify the associated database node 120. Various information included in the key space reference 126 may be added and updated over time, including after another database node 120 is granted permission to write to the identified key space. For example, the epoch range 230 (described below) of the key space reference 126 may not specify a complete range (e.g., an upper limit) until all active database transactions 122 that have written to the key space 210 have committed.

[0032] The epoch range 230, in various embodiments, identifies a time frame in which database records 132 were committed to the corresponding key space 210. When a database transaction 122 is committed, the database records 132 written for that database transaction 122 may be stamped with a transaction commit number (XCN). Database records 132 committed earlier in time may be stamped with an XCN having a lower numeric value than the XCN of database records 132 committed at a later time. Committed database records 132, in various embodiments, remain in the in-memory cache 130 of the database node 120 until they are flushed to the database 110 in response to a trigger event (e.g., the in-memory cache 130 storing a threshold amount of data). When the database node 120 flushes its in-memory cache 130, it may flush one or more database records 132 up to a particular XCN (referred to as a "flush XCN"). In various embodiments, the epoch range 230 defines a time frame by specifying a minimum and maximum XCN. The minimum XCN may identify the most recently flushed XCN when a key spatial reference 126 is created. For example, database node 120B may flush all database records 132 with an XCN less than 600. Thereafter, when a key spatial reference 126 is created in association with database node 120B, the lower bound of the epoch range 230 of that key spatial reference 126 may be set to 600.

[0033] The max XCN may identify the XCN associated with the last database transaction 122 that wrote to the key space 210 before it was awarded to another database node 120. That is, while a database node 120 owns a particular key space 210, it may execute multiple database transactions 122 that write database records 132 to that key space 210. A particular key space 210 may be awarded to another database node 120 at some point; however, these database transactions 122 may still be active. In various embodiments, these database transactions 122 are allowed to complete and are not aborted. The epoch range 230 of the key space reference 126 corresponding to these database transactions 122 may be updated to specify the XCN of the last of these transactions 122 to commit as the max XCN of the epoch range 230. Because the max XCN may not be set until the last of these transactions 122 commits, in various embodiments, the epoch range 230 initially specifies a null value for max XCN. As a result, while max XCN is set to a null value, the time period indicated by epoch range 230 may have a beginning but no end.

[0034] The state 240, in various embodiments, identifies the stage of a key space reference 126 in its lifecycle. States may include "open," "closed," and "inactive." In various embodiments, the "open" state indicates that record writes to the corresponding key space 210 are permitted at the database node 120 indicated by the corresponding node index 220. When a key space reference 126 is initially created, its state 240 may be set to "open." In various embodiments, the "closed" state indicates that 1) record writes to the key space 210 are not permitted at the corresponding database node 120, and 2) there is still an active database transaction 122 at that database node 120. An active database transaction 122, in various embodiments, refers to an ongoing database transaction 122 in which the database node 120 is writing database records 132 to its in-memory cache 130. An active database transaction 122 can become a committed database transaction 122 when the database records 132 associated with that database transaction 122 commit. When the key space 210 of a key space reference 126 is provisioned to another database node 120, the state 240 of the key space reference 126 may be set to "closed." In various embodiments, the "inactive" state indicates that active database transactions 122 associated with the key space 210 have committed on the corresponding database node 120. In various embodiments, the key space reference 126 may be deleted after committed database records 132 associated with the key space reference 126 have been flushed from the in-memory cache 130 of the corresponding database node 120 to persistent storage (e.g., the database 110).

[0035] Referring now to FIG. 3, a block diagram of an exemplary layout associated with a database node 120B using a key space reference 126 to process active database transactions 122 is shown. In the illustrated embodiment, database node 120B includes a key space reference 126, an in-memory cache 130, and a database application 300. As shown, database application 300 is processing an active database transaction 122 that has been assigned key space grant 124 "XY" and has an associated snapshot transaction commit number (snapshot XCN) 310 "445." As further shown, there are three key space references 126, each associated with a different database node 120 but corresponding to the same key space 210 "XY." In some embodiments, database node 120 is implemented differently than shown. As an example, database application 300 may process multiple active transactions 122 and multiple committed transactions 122.

[0036] Database application 300 is a set of program instructions executable to manage database 110, including, in various embodiments, managing an LSM tree built around database 110. As such, database application 300 can receive requests to perform database transactions 122, including reading and / or writing database records 132 of database 110. As an example, database node 120B may receive a transaction request from an application node to execute a set of SQL statements identified by the application node. Upon receiving the transaction request, database application 300 can initiate an active database transaction 122 based on the received transaction request. In various embodiments, active database transactions 122 are associated with a snapshot XCN. The snapshot XCN, in various embodiments, identifies the latest XCN at which the corresponding database record 132 can be read by the active database transaction 122. For example, the exemplary active database transaction 122 is associated with snapshot XCN 310 “445.” As a result, the active database transaction 122 can read committed database records 132 that are assigned XCNs less than or equal to “445.” In some cases, only database records 132 whose XCN is less than "445" can be read.

[0037] The following discussion uses an example to provide a more detailed understanding of the concepts discussed throughout this disclosure. Consider an example in which database node 120B wants to write a database record 132 having key 134 “XYZ” for an example active database transaction 122. Before writing the database record 132, in various embodiments, database node 120B considers key space reference 126 to determine whether another database node 120 has written to key space 210 “XY” within the time frame that includes snapshot XCN 310 “445.” As disclosed, key space reference 126 can be accessed from catalog manager node 140 via catalog request 142 and subsequent catalog response 144. Database node 120B can issue catalog request 142 when initiating an active database transaction 122. In some cases, catalog manager node 140 can only provide key space references 126 with epoch ranges 230 that encompass snapshot XCN 310 of the initiated database transaction 122.

[0038] As illustrated, key space reference 126A identifies database node 120B and has an open state 240, indicating that a record write should occur at database node 120B for key space 210 “XY.” Thus, key space reference 126A is considered a “write” key space reference 126, while key space references 126B and 126C are considered “read” key space references 126 because they identify locations where record writes were previously allowed to occur for key space 210 “XY.” In some cases, database node 120B may have previously written a database record 132 for key 134 “XYZ” that has not yet been committed. Therefore, in some embodiments, database node 120B first searches its in-memory cache 130 for a database record 132 with key 134 “XYZ.” If a database record 132 with key 134 “XYZ” is identified, database node 120B can write a new database record 132 for the identified database record 132. If there is no such database record 132 in the in-memory cache 130 , then database node 120 B may consider other key space references 126 .

[0039] As further illustrated, key space reference 126B identifies database node 120A, has an open epoch range (no upper XCN limit defined), and has a closed state 240 indicating that there are still active database transactions 122 that may have written database records 132 for key space 210 “XY” that have not committed. Database node 120B can first determine whether snapshot XCN 310 “445” falls within epoch range 230 of key space reference 126B. Because snapshot XCN 310 “445” falls within epoch range 230 “390-Null” and not all database transactions 122 associated with key space 210 “XY” have committed, the possibility exists that database node 120A may have written a database record 132 with key 134 “XYZ” that is not known to database node 120B. As a result, in various embodiments, database node B determines whether a database record 132 has been written at database node A for key 134 "XYZ." To do so, database node 120B can send record request 136 to database node A, requesting an indication of whether database node A has written such a database record 132. Database node 120A can return record response 138. If record response 138 indicates that database node 120A has written a database record 132 for key 134 "XYZ," database node 120B can abort the active database transaction 122 (or subtransaction portion) or wait until other database records 132 commit before writing its database record 132 for key 134 "XYZ." Record response 138, in some embodiments, includes the database records 132 written by database node 120.If the record response 138 indicates that database node 120A has not written a database record 132 for key 134 "XYZ," database node 120B may consider other key space references 126.

[0040] As illustrated, key space reference 126B identifies database node 120C, has a closed epoch range 230, and has an inactive state 240, indicating that all active database transactions 122 at database node 120C have been committed. While these database transactions 122 are committed, in some cases, the corresponding database records 132 have not been flushed to database 110 and remain in database node 120C's in-memory cache 130. Therefore, database node 120B can send database node 120C a record request 136 for an indication of whether database node 120C has written database record 132 for key 134 "XYZ." Based on a record response 138 received from database node 120C, database node 120B can abort the active database transaction 122 or wait to write that database record 132 after other database records 132 have been flushed. If the record response 138 indicates that database node 120C has not written a record 132 for key 134 "XYZ," database node 120C may consider other key space references 126 if there are additional key space references 126 associated with key 134 "XYZ" whose epoch range 230 encompasses snapshot XCN310 "445."

[0041] 4, a block diagram of an exemplary layout associated with a database node 120 that updates a key space reference 126 in response to the commitment of an active database transaction 122. In the illustrated embodiment, database node 120A includes a key space reference 126, an in-memory cache 130, and a database application 300. As shown, database application 300 has committed database transaction 122 at XCN 410 "600."

[0042] While the following discussion is made with reference to database node 120A, the discussion is applicable to other database nodes 120, such as database node 120B. During operation, database node 120A may obtain keyspace permission 124 to write database records 132 for keyspace 210 “XY.” At various times, while database node 120A holds that keyspace permission 124, database node 120A may initiate multiple active database transactions 122 that write database records 132 for keyspace 210 “XY.” While database node 120A is processing those database transactions 122, database node 120A may receive a request from catalog manager node 140 to relinquish part or all of keyspace 210 “XY.” For example, database node 120A may be requested to relinquish keyspace 210 “XYZ.” In various embodiments, database node 120A abandons the requested key space 210 but allows active database transactions 122 associated with that key space 210 to commit. While there is at least one active database transaction 122 associated with that key space 210, database node 120A may not update the corresponding key space reference 126 (e.g., key space reference 126B for key space 210 “XY”) to define the upper bound of its epoch range 230.

[0043] After processing an active database transaction 122, database node 120A can commit that database transaction 122, which results in a committed database transaction 122. As part of the commitment process, in some embodiments, database node 120A stamps an XCN 410 on each database record 132 of the database transaction 122. Illustratively, for example, committed transaction 122 has XCN 410 “600.” Thus, each record 132 associated with the committed transaction 122 may include metadata identifying XCN 410 “600.” After processing the last active transaction associated with the abandoned key space 120, in various embodiments, database node 120A updates the epoch range 230 of the corresponding key space reference 126B with the XCN 410 of that database transaction 122. Consider an example where the illustrative committed database transaction 122 was the last active transaction 122 at database node 120A for key space 210 “XY.” Thus, database node 120A can update the epoch range 230 of key spatial reference 126B to specify "XCN390-600" and the state 240 to "inactive." Database node 120A can send a reference update request 404 to catalog manager node 140 to distribute the new version of key spatial reference 140B to other database nodes 120 in system 100.

[0044] Referring now to FIG. 5, a flow diagram of method 500 is shown. Method 500 is one embodiment of a method performed by a first database node (e.g., database node 120B) of a database system (e.g., system 100) as part of processing a database transaction (e.g., database transaction 122). In some cases, method 500 may be performed by executing program instructions stored on a non-transitory computer-readable medium (e.g., memory 820). In some embodiments, method 500 includes more or fewer steps than shown. As an example, method 500 may include a step in which the first database node returns a response to the transaction requestor.

[0045] Method 500 begins at step 510 with a first database node receiving a request to perform a database transaction that includes writing a particular record (e.g., database record 132) to a key (e.g., key 134) included in a key space (e.g., key space 210). Prior to receiving the request to perform the database transaction, the first database node may issue a permission request to a database system (e.g., catalog manager node 140) for approval to write the record to the key space. Accordingly, the first database node may receive a permission (e.g., key space permission 124) to write the record to the key space. In some examples, access permissions for the key space may be re-provisioned to the first database node from a second database node (e.g., database node 120A). In some embodiments, an active transaction on the second database node that includes writing the record to the key space is allowed to commit after permission to write the record to the key space is received by the first database node. In some cases, at least one of the active transactions may have caused the record to be written to the key space before permission was received by the first database node.

[0046] In step 520, the first database node accesses a key space reference catalog (e.g., storage catalog 145), which stores multiple indicators (e.g., key space references 126) of when a key space was written by a database node of the database system. The multiple indicators may include a set of indicators that is specific to the key space. One of the set of indicators may identify database nodes that are authorized to write records for the key space, and two or more of the set of indicators may identify database nodes for reading records written for the key space. Thus, a particular indicator may indicate that all record writes to the key space should be performed by the first database node while permission is granted to the first database node. The first database node may receive a write request from a second database node to write a specific record to the key space as part of an active transaction on the second database node. Thus, the first database node may grant permission to the second database node to write the specific record. The second database node may then use this permission to complete the write.

[0047] In some cases, the particular index can identify an epoch range (e.g., epoch range 230) for the key space and is associated with a second database node. The first database node can determine that an epoch corresponding to a database transaction falls within the epoch range. This determination can indicate that the second database node potentially wrote records for the key space within a particular time frame. In some cases, the epoch range for a particular index can be modified in response to the commitment of the last active transaction linked to the particular index. The epoch range may or may not define the upper epoch of the epoch range prior to modification of the particular index. The particular index can be deleted after all records written by the second database node for the key space have been stored in a persistent database (e.g., database 110) of the database system. In various embodiments, the index is maintained only for uncommitted work or for transactions that are in main memory and have not yet been flushed to persistent storage.

[0048] In step 530, in response to determining that the second database node has potentially written a record for the key space within a particular time frame, the first database node sends to the second database node a request for information (e.g., record request 136) indicating whether the second database node has written a record for the key. In step 540, based on the response (e.g., record response 138) received from the second database node, the first database node determines whether to write the particular record. In some cases, in response to determining that the second database node has written the record for the key, the first database node may abort at least a portion of the database transaction that includes writing the particular record. In some cases, in response to determining that the second database node has written the record for the key, the first database node may wait until the record written by the second database node has been committed before writing the particular record.

[0049] Referring now to FIG. 6, a flow diagram of method 600 is shown. Method 600 is one embodiment of a method performed by a database system (e.g., system 100) as part of processing a database transaction (e.g., database transaction 122). In some cases, method 600 may be performed by executing program instructions stored on a non-transitory computer-readable medium (e.g., memory 820). In some embodiments, method 600 includes more or fewer steps than shown. For example, method 600 may include a step in which a first database node (e.g., database node 120B) of the database system returns a response to the transaction requestor.

[0050] Method 600 begins at step 610 with a database system maintaining a key space reference catalog (e.g., storage catalog 145) that stores multiple indices (e.g., key space references 126) associated with a key space (e.g., key space 210). At step 620, the database system assigns the key space to a first database node. In various cases, a first specific index of the multiple indices identifies a first time frame (e.g., epoch range 230) and a second database node (e.g., database node 120A) of the database system that was previously assigned the key space, such that the second database node was authorized to write records whose keys fall within the key space at the second database node. The database system can add a second specific index to the key space reference catalog that specifies an open state (e.g., open state 240) indicating that all record writes to the key space should occur at the first database node. The database system may update the first particular indicator to specify a closed state (e.g., closed state 240) indicating that record writes to the keyspace should not occur at the second database node. In some cases, the database system receives an upgrade request to perform a rolling upgrade at the second database node and performs the allocation in response to receiving the upgrade request.

[0051] At step 630, the first database node executes a transaction including writing a record for a key in the key space. The executing includes, at step 632, the first database node determining, based on a first particular indicator, that a first time period overlaps with a second time period associated with the transaction. At step 634, in response to the determining, the first database node sends to the second database node a request for information (e.g., record request 136) indicating whether a record has been written but not committed by the second database node for the key. In response to determining that the second database node has not written a record for the key, the first database node can write the particular record.

[0052] Exemplary Multi-Tenant Database System Referring now to FIG. 7, an exemplary multi-tenant database system (MTS) 700 is shown in which various techniques of the present disclosure can be implemented. For example, system 100 may be MTS 700. In FIG. 7, MTS 700 includes a database platform 710, an application platform 720, and a network interface 730 connected to a network 740. As further shown, database platform 710 includes a data storage 712 and a set of database servers 714A-N that interact with data storage 712, and application platform 720 includes a set of application servers 722A-N, each with its own environment 724. In the illustrated embodiment, MTS 700 is connected to various user systems 750A-N via network 740. The disclosed multi-tenant system is included for illustrative purposes and is not intended to limit the scope of the present disclosure. In other embodiments, the techniques of the present disclosure are implemented in a non-multi-tenant environment, such as a client / server environment, a cloud computing environment, or a clustered computer.

[0053] In various embodiments, MTS 700 is a set of computer systems that together provide various services to users (alternatively referred to as “tenants”) who interact with MTS 700. In some embodiments, MTS 700 implements a customer relationship management (CRM) system that provides a mechanism for tenants (e.g., businesses, government agencies, etc.) to manage their relationships and interactions with customers and potential customers. For example, MTS 700 may enable tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), identify sales opportunities, log service issues, and manage marketing campaigns. Furthermore, MTS 700 may enable these tenants to identify how they communicated with customers, what customers purchased, when they last purchased an item, and how much they paid. To provide the services of the CRM system and / or other services, MTS 700 includes a database platform 710 and an application platform 720, as shown.

[0054] Database platform 710, in various embodiments, is a combination of hardware elements and software routines that implement database services for storing and managing data for MTS 700, including tenant data. As shown, database platform 710 includes data storage 712, which, in various embodiments, includes a set of storage devices (e.g., solid-state drives, hard disk drives, etc.) connected together over a network (e.g., a storage-attached network (SAN)) and configured to store data redundantly to prevent data loss. In various embodiments, data storage 712 is used to implement a database (e.g., database 110) that includes a collection of information organized in a manner that allows the information to be accessed, stored, and manipulated. Data storage 712 can implement a single database, a distributed database, a collection of distributed databases, a database with redundant online or offline backups or other redundancies, etc. As part of implementing a database, data storage 712 can store files (e.g., files 115) containing one or more database records having respective data payloads (e.g., values of fields in a database table) and metadata (e.g., key values, timestamps, table identifiers for tables associated with the record, tenant identifiers for tenants associated with the record, etc.).

[0055] In various embodiments, a database record may correspond to a row in a table. A table typically contains one or more data categories logically arranged as columns or fields in a visible schema. Thus, each record in a table may contain an instance of data for each category defined by a field. For example, a database may contain a table that describes customers with fields for basic contact information, such as name, address, phone number, and fax number. Thus, a record in that table may contain a value for each of the fields in the table (e.g., name for a name field). Another table may describe purchase orders, including fields for information such as customer, product, sales price, and date. In various embodiments, standard entity tables, such as tables for account, contact, lead, and opportunity data, each containing predefined fields, are provided for use by all tenants. The MTS 700 may store database records for more than one tenant in the same table; i.e., tenants may share tables. Thus, in various embodiments, a database record includes a tenant identifier that indicates the owner of the database record. As a result, one tenant's data is kept securely separate from other tenants' data, so that one tenant cannot access another tenant's data unless that data is explicitly shared with them.

[0056] In some embodiments, data stored in data storage 712 is organized as part of a log-structured merge tree (LSM tree). An LSM tree typically includes two high-level components: an in-memory cache and a persistent storage. In operation, database server 714 may first write database records to a local in-memory cache before later flushing these records to persistent storage (e.g., data storage 712). As part of flushing a database record, database server 714 may write the database record to a new file contained in the “top” level of the LSM tree. Over time, as database records are moved down the levels of the LSM tree, the database record may be rewritten by database server 714 to a new file contained in a lower level. In various embodiments, as database records age and are moved down the LSM tree, they are moved to increasingly slower storage devices in data storage 712 (e.g., from solid-state drives to hard disk drives).

[0057] When database server 714 wants to access a database record for a particular key, it can traverse different levels of the LSM tree for files that potentially contain the database record for that particular key. If database server 714 determines that a file may contain a relevant database record, it can fetch the file from data storage 712 into its memory. Database server 714 may then check the fetched file for a database record with the particular key. In various embodiments, database records are immutable once written to data storage 712. Thus, when database server 714 wants to modify the values of a table row (which can be identified from the accessed database record), database server 714 writes a new database record to the top level of the LSM tree. Over time, the database records are merged down the levels of the LSM tree. Thus, an LSM tree can store various database records for a database key, with older database records for that key being located at a lower level in the LSM tree than newer database records.

[0058] Database server 714, in various embodiments, is a hardware element, software routine, or combination thereof that can provide database services such as data storage, data retrieval, and / or data manipulation. Database server 714 may correspond to database node 120. Such database services may be provided by database server 714 to components within MTS 700 (e.g., application server 722) and to components external to MTS 700. As an example, database server 714 may receive a database transaction request from application server 722 requesting that data be written to or read from data storage 712. The database transaction request may specify an SQL SELECT command that selects one or more rows from one or more database tables. The contents of the row may be defined in a database record, and thus database server 714 may identify and return one or more database records corresponding to the selected one or more table rows. In various cases, the database transaction request may instruct database server 714 to write one or more database records to an LSM tree, which database server 714 maintains implemented in database platform 710. In some embodiments, database servers 714 implement a relational database management system (RDMS) or an object-oriented database management system (OODBMS) that facilitates the storage and retrieval of information from data storage 712. In various cases, database servers 714 may communicate with each other to facilitate the processing of transactions. For example, database server 714A may communicate with database server 714N to determine whether database server 714N has written a database record for a particular key to its in-memory cache.

[0059] The application platform 720, in various embodiments, is a combination of hardware elements and software routines that implements and executes the CRM software application, as well as provides associated data, code, forms, web pages, and other information to and from user systems 750, and stores associated data, objects, web page content, and other tenant information via the database platform 710. To facilitate these services, in various embodiments, the application platform 720 communicates with the database platform 710 to store, access, and manipulate data. In some examples, the application platform 720 can communicate with the database platform 710 via different network connections. For example, one application server 722 may be coupled via a local area network, while another application server 722 may be coupled via a direct network link. Transmission Control Protocol / Internet Protocol (TCP / IP) is an exemplary protocol for communicating between the application platform 770 and the database platform 710, although those skilled in the art will recognize that other transport protocols may be used depending on the network interconnect used.

[0060] The application server 722, in various embodiments, is a hardware element, software routine, or combination thereof that can provide the services of the application platform 720, including processing requests received from tenants of the MTS 700. The application server 722, in various embodiments, can spawn environments 724 that can be used for various purposes, such as providing functionality for developers to develop, execute, and manage applications (e.g., business logic). Data may be transferred to an environment 724 from another environment 724 and / or from the database platform 710. In some cases, an environment 724 cannot access data from other environments 724 unless such data is explicitly shared. In some embodiments, multiple environments 724 can be associated with a single tenant.

[0061] Application platform 720 can provide user systems 750 with access to multiple different hosted applications (standard and / or custom), including a CRM application and / or tenant-developed applications. In various embodiments, application platform 720 can manage application creation, application testing, storing applications in database objects in data storage 712, running applications in environments 724 (e.g., virtual machines in a process space), or any combination thereof. In some embodiments, application platform 720 can add and remove application servers 722 from the server pool at any time for any reason, and there may not be user and / or organization server affinity to specific application servers 722. In some embodiments, an interface system (not shown) implementing load balancing functionality (e.g., an F5 Big-IP load balancer) is disposed between application servers 722 and user systems 750 and operates to distribute requests to application servers 722. In some embodiments, the load balancer routes user requests to application servers 722 using a least-connection algorithm. Other example load balancing algorithms, such as round robin and observed response time, can also be used. For example, in a particular embodiment, three consecutive requests from the same user may hit three different servers 722, and three requests from different users may hit the same server 722.

[0062] In some embodiments, MTS 700 provides security mechanisms, such as encryption, to keep each tenant's data separate unless the data is shared. When multiple servers 714 or 722 are used, they may be located in close proximity to each other (e.g., in a server farm located in a single building or campus), or they may be distributed in locations remote from each other (e.g., one or more servers 714 located in city A and one or more servers 722 located in city B). Thus, MTS 700 may include one or more logically and / or physically connected servers, either locally or distributed in one or more geographic locations.

[0063] One or more users (e.g., via user systems 750) can interact with the MTS 700 over the network 740. The user systems 750 can correspond, for example, to tenants of the MTS 700, providers (e.g., administrators) of the MTS 700, or third parties. Each user system 750 can be a desktop personal computer, a workstation, a laptop, a PDA, a cell phone, or any wireless access protocol (WAP)-enabled device, or any other computing device capable of directly or indirectly interfacing to the Internet or other network connection. The user systems 750 can include dedicated hardware configured to interface with the MTS 700 over the network 740. The user systems 750 can communicate with the MTS 700, a HTTP client (e.g., Microsoft® Internet Explorer), or any other application program. TM Browser: Netscape Navigator TMEach user system 750 may execute a corresponding graphical user interface (GUI) for a browser (e.g., a browsing program such as a browser, Opera® browser, or a WAP-enabled browser in the case of a cell phone, PDA, or other wireless device), or both, that allows users of user systems 750 (e.g., subscribers to a CRM system) to access, process, and view information and pages available to them from MTS 700 over network 740. Each user system 750 may include one or more user interface devices, such as a keyboard, mouse, touch screen, pen, etc., for interacting with the graphical user interface (GUI) provided by the browser on a display monitor screen, LCD display, etc., in connection with pages, forms, and other information provided by MTS 700 or other systems or servers. As discussed above, the disclosed embodiments are suitable for use with the Internet, which refers to a specific global internetwork of networks. However, it should be understood that other networks, such as an intranet, extranet, virtual private network (VPN), non-TCP / IP-based network, any LAN, or WAN, may be used in place of the Internet.

[0064] Because users of user system 750 may be users in different capacities, the capacity of a particular user system 750 may determine one or more permission levels associated with the current user. For example, when a salesperson is using a particular user system 750 to interact with MTS 700, that user system 750 may have a capacity (e.g., user privileges) assigned to that salesperson. However, when an administrator is using the same user system 750 to interact with MTS 700, that user system 750 may have a capacity (e.g., administrator privileges) assigned to that administrator. In a system with a hierarchical role model, a user at one permission level may be able to access applications, data, and database information accessible by users with lower permission levels, but may not be able to access certain applications, database information, and data accessible by users with higher permission levels. Thus, different users may have different capabilities with respect to accessing and modifying applications and database information depending on the user's security or permission level. Additionally, there may be some data structures managed by MTS 700 that are allocated at the tenant level, while other data structures are managed at the user level.

[0065] In some embodiments, the user system 750 and its components are configurable using an application, such as a browser, that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 700 (and additional instances of the MTS, if multiple instances are present) and their components are operator-configurable using applications that include computer code executable on the processing elements. Accordingly, various operations described herein may be performed by executing program instructions stored on a non-transitory computer-readable medium and executed by the processing elements. The program instructions may be stored on a non-volatile medium, such as a hard disk, or any other well-known volatile or non-volatile memory medium or device, such as a ROM or RAM, or provided on any medium capable of storing program code, such as a compact disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, or the like. Furthermore, the entire program code, or portions thereof, may be transmitted or downloaded from a software source, for example, over the Internet or from another server, as is well known, or may be transmitted over any other well-known conventional network connection (e.g., extranet, VPN, LAN, etc.) using any well-known communication medium and protocol (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.). Furthermore, it will be understood that computer code for implementing aspects of the disclosed embodiments can be implemented in any programming language executable on a server or server system, such as, for example, C, C+, HTML, Java, JavaScript, or any other scripting language such as VBScript.

[0066] Network 740 may be a LAN (Local Area Network), a WAN (Wide Area Network), a wireless network, a point-to-point network, a star network, a token ring network, a hubbed network, or any other suitable configuration. The global internetwork of networks, often referred to as the "Internet" with a capital "I," is one example of a TCP / IP (Transmission Control Protocol and Internet Protocol) network. However, it should be understood that the disclosed embodiments may utilize any of a variety of other types of networks.

[0067] The user system 750 communicates with the MTS 700 using TCP / IP, and at higher network levels, can use other common Internet protocols for communication, such as HTTP, FTP, AFS, WAP, etc. For example, if HTTP is used, the user system 750 can include an HTTP client, commonly called a "browser," for sending and receiving HTTP messages from an HTTP server in the MTS 700. Such a server may be implemented as simply a network interface between the MTS 700 and the network 740, although other approaches may also or alternatively be used. In some implementations, the interface between the MTS 700 and the network 740 includes a load-sharing function, such as a round-robin HTTP request distributor, which balances the load and distributes incoming HTTP requests evenly across multiple servers.

[0068] In various embodiments, user system 750 communicates with application server 722 to request and update system-level and tenant-level data from MTS 700, which may require one or more queries to data storage 712. In some embodiments, MTS 700 automatically generates one or more SQL statements (SQL queries) designed to access the desired information. In some cases, user system 750 may generate requests having a specific format corresponding to at least a portion of MTS 700. As an example, user system 750 may request that data objects be moved to a particular environment 724 using an object notation that describes the object-relational mapping (e.g., JavaScript object notation mapping) of specified objects.

[0069] Exemplary Computer System 8, a block diagram of an exemplary computer system 800 is shown in which system 100, database 110, database node 120, MTS 700, and / or user system 750 may be implemented. Computer system 800 includes a processor subsystem 880 coupled to a system memory 820 and an I / O interface 840 via an interconnect 860 (e.g., a system bus). I / O interface 840 is coupled to one or more I / O devices 850. For simplicity, FIG. 8 depicts a single computer system 800, but system 800 may also be implemented as multiple computer systems operating together.

[0070] Processor subsystem 880 may include one or more processors or processing units. In various embodiments of computer system 800, multiple instances of processor subsystem 880 may be coupled to interconnect 860. In various embodiments, processor subsystem 880 (or each processor unit within 880) may include a cache or other form of on-board memory.

[0071] System memory 820 can be used to store program instructions executable by processor subsystem 880 to cause system 800 to perform the various operations described herein. System memory 820 may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM, i.e., SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.). The memory of computer system 800 is not limited to primary storage devices such as memory 820. Rather, computer system 800 may further include other forms of storage, such as cache memory within processor subsystem 880 and secondary storage on I / O devices 850 (e.g., hard drives, storage arrays, etc.). In some embodiments, these other forms of storage may also store program instructions executable by processor subsystem 880. In some embodiments, program instructions that, when executed, implement database application 146 may be contained / stored within system memory 820.

[0072] I / O interface 840 may be any of a variety of types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I / O interface 840 is a bridge chip (e.g., a southbridge) from a frontside to one or more backside buses. I / O interface 840 may be coupled to one or more I / O devices 850 via one or more corresponding buses or other interfaces. Examples of I / O devices 850 include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., for local or wide area networks), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system 800 is coupled to a network via network interface device 850 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0073] Implementations of the subject matter of this application include, but are not limited to, the following Examples 1-20. 1. receiving, by a first database node of a database system, a request to perform a database transaction including writing a particular record to a key contained in a key space; accessing, by the first database node, a key space reference catalog that stores a plurality of indications of when key spaces have been written by database nodes of the database system; in response to determining that a second database node has written a record for the key space within a specified time frame, the first database node sending to the second database node a request for information indicating whether the second database node has written a record for the key space; determining, based on a response received from the second database node, whether to write the particular record; A method comprising: 2. prior to receiving the request to perform the database transaction, the first database node: issuing an authorization request to the database system for approval to write a record to the keyspace; receiving permission to write a record to the keyspace, the permission being transferred from the second database node to the first database node; The method of Example 1, further comprising: 3. The method of Example 2, wherein active transactions on the second database node that include writing records to the key space are permitted to commit after the permission to write records to the key space is received by the first database node, and at least one of the active transactions caused a record write to the key space before the permission was received by the first database node. 4. The method of Example 2, wherein a particular indicator among the plurality of indicators indicates that while the permission is granted to the first database node, all record writes to the key space identified by the particular indicator should be performed by the first database node. 5. receiving, by the first database node, a relinquish request to relinquish the permission to the second database node to permit the second database node to write records to the key space as part of an active transaction on the second database node; relinquishing, by the first database node, the permission in response to the relinquishment request; The method of Example 4, further comprising: 6. A particular index of the plurality of indexes identifies an epoch range for the key space and is associated with the second database node, the method comprising: making a determination by the first database node that an epoch corresponding to the database transaction falls within the epoch range, the determination indicating that the second database node wrote a record for the key space within the particular time frame; The method of Example 1, further comprising: 7. modifying the epoch range in response to a commitment of the last active transaction linked to the particular index, the epoch range not defining an upper epoch of the epoch range prior to the modification. The method of Example 6, further comprising: 8. storing all records written by the second database node for the key space in a persistent database of the database system, and then deleting the particular index. The method of Example 6, further comprising: 9. The method of Example 1, wherein the plurality of indices includes a set of indices for the key space, one of the set of indices identifying database nodes authorized to write records for the key space, and two or more of the set of indices identifying database nodes for reading records written for the key space. 10. In response to determining that the second database node has written a record for the key, the first database node aborts at least a portion of the database transaction that included writing the particular record. The method of Example 1, further comprising: 11. A non-transitory computer-readable medium having stored thereon program instructions executable by a first database node of a database system to cause the first database node to perform an operation, the operation comprising: receiving a request to perform a database transaction including writing a particular record for a key included in a key space; accessing a key space reference catalog that stores a plurality of indications of when a key space has been written by a database node of the database system; in response to determining that a second database node has written a record for the key space within a particular time frame, sending to the second database node a request for information indicating whether the second database node has written a record for the key space; determining whether to write the particular record based on a response received from the second database node; and Including, media. 12. The medium of Example 11, wherein a particular index among the plurality of indexes corresponding to the second database node identifies the key space by a minimum key and a maximum key, and the particular index specifies a time period that encompasses the particular time period. 13. The operation is requesting authorization to write a record to the keyspace at the first database node; receiving authorization to write a record to the keyspace; storing a specific index in the key space reference catalog, the specific index indicating that all record writes to the key space should occur at the first database node; 12. The medium of Example 11, further comprising: 14. The operation is performing another database transaction that includes writing another specific record for the key contained in the key space; using the keyspace reference catalog to determine that the permission to write records for the keyspace has been transferred to a third database node; sending a write request to the third database node to write the other particular record; 14. The medium of Example 13, further comprising: 15. The operation is responsive to determining that the second database node has written a record for the key, waiting until the record written by the second database node is committed before writing the particular record. 12. The medium of Example 11, further comprising: 16. Maintaining, by the database system, a key space reference catalog that stores a plurality of indices associated with the key space; assigning, by the database system, a key space to a first database node of the database system, wherein a first particular indicator of the plurality of indicators identifies a first time period and a second database node of the database system that was previously assigned the key space, whereby the second database node was authorized to write records whose keys fall within the key space at the second database node; executing, by the first database node, a transaction including writing a record to a key in the key space, said executing including: determining, based on the first specified indicator, that the first time frame overlaps with a second time frame associated with the transaction; in response to said determining, sending to said second database node a request for information indicating whether a record has been written but not committed by said second database node for said key; A method comprising: 17. receiving, by the database system, an upgrade request to perform a rolling upgrade on the second database node, wherein the allocating is performed in response to receiving the upgrade request; The method of Example 16, further comprising: 18. The said allocation: adding, by the database system, a second specific index to the key spatial reference catalog, the second specific index specifying an open state indicating that all record writes to the key space should occur at the first database node; updating, by the database system, the first particular indicator to specify a closed state indicating that record writes to the key space should not occur at the second database node; The method of Example 16, comprising: 19. The carrying out of the above-mentioned determining, based on the key spatial reference catalog, that at least two of the plurality of indices are associated with the key space, wherein the at least two indices include the first particular index; in response to determining that a time period identified by an index different from the first particular index among the at least two indexes overlaps with the second time period, sending to a third database node a request for information indicating whether a record has been written but not committed by the third database node for the key; The method of Example 16, comprising: 20. The method of Example 16, wherein the first database node writes the particular record in response to determining that the second database node has not written a record for the key.

[0074] This disclosure includes references to "embodiments" that are non-limiting implementations of the disclosed concepts. References to "one embodiment," "one embodiment," "particular embodiment," "some embodiments," "various embodiments," etc. do not necessarily refer to the same embodiment. Numerous possible embodiments are possible, including the specific embodiment described in detail, as well as modifications or alternatives that fall within the spirit or scope of the present disclosure. Not all embodiments necessarily exhibit some or all of the potential advantages described herein.

[0075] This disclosure includes references to "one embodiment" or groups of "embodiments" (e.g., "some embodiments" or "various embodiments"). Embodiments are different implementations or instances of the disclosed concepts. References to "one embodiment," "one embodiment," "particular embodiment," etc. do not necessarily refer to the same embodiment. Numerous possible embodiments are possible, including those specifically disclosed, as well as modifications or alternatives that fall within the spirit or scope of the disclosure.

[0076] This disclosure may discuss potential advantages that may result from the disclosed embodiments. Not all implementations of these embodiments necessarily exhibit any or all of the potential advantages. Whether advantages are realized for a particular implementation depends on many factors, some of which are beyond the scope of this disclosure. Indeed, there are several reasons why an implementation within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuitry outside the scope of this disclosure that negates or reduces one or more of the disclosed advantages relative to one of the disclosed embodiments. Furthermore, suboptimal design practices of a particular implementation (e.g., implementation techniques or tools) may also negate or reduce the disclosed advantages. Even assuming a skilled implementation, realization of advantages may still depend on other factors, such as the environmental conditions in which the implementation is deployed. For example, inputs provided to a particular implementation may prevent one or more problems addressed in this disclosure from occurring on a particular occasion, resulting in the benefits of that solution not being realized. It is expressly intended that the potential advantages described herein should not be construed as claim limitations that must be satisfied in order to prove infringement, given the existence of possible factors external to this disclosure. Rather, the identification of such potential advantages is intended to illustrate the types of improvements available to a designer having the benefit of this disclosure. The fact that such advantages are permissively described (e.g., stating that a particular advantage "may result") is not intended to convey doubt as to whether such advantages can actually be realized, but rather to recognize the technological reality that realization of such advantages often depends on additional factors.

[0077] Unless otherwise indicated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of claims drafted based on this disclosure, even if only a single example of a particular feature is described. The disclosed embodiments are intended to be illustrative, not limiting, unless stated to the contrary in the disclosure. Accordingly, the present application intends to permit claims to cover the disclosed embodiments, as well as alternatives, modifications, and equivalents that will be apparent to those skilled in the art having the benefit of this disclosure.

[0078] For example, features in this application may be combined in any suitable manner. Accordingly, during prosecution of this application (or an application claiming priority thereto), new claims may be formulated for any such combination of features. In particular, with reference to the appended claims, features from dependent claims may be combined with features of other dependent claims, including claims that are dependent on other independent claims, where appropriate. Similarly, features from each independent claim may be combined where appropriate.

[0079] Thus, the appended dependent claims may be drafted so that each depends on one other claim, but further dependencies are also contemplated. Any combination of features in the dependencies not inconsistent with this disclosure is contemplated and may be claimed in this or another application. In short, combinations are not limited to those specifically recited in the appended claims.

[0080] Furthermore, where appropriate, claims drafted in one form or statutory type (e.g., apparatus) are considered to be intended to support corresponding claims in another form or statutory type (e.g., method).

[0081] Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. Notice is hereby given that the following paragraphs, as well as the definitions provided throughout this disclosure, should be used in determining how to interpret any claims drafted based on this disclosure.

[0082] Reference to the singular form of an item (i.e., a noun or noun phrase preceded by "a," "an," or "the") is intended to mean "one or more" unless the context clearly dictates otherwise. Thus, a reference to an "item" in a claim does not exclude additional instances of the item without attendant context. A "plurality" of an item refers to a set of two or more of that item.

[0083] The word "may" is used herein in a permissive sense (i.e., having the potential, being able to), and not in a mandatory sense (i.e., must).

[0084] The terms "comprising" and "including" and their forms are open-ended and mean "including but not limited to."

[0085] In this disclosure, when the term "or" is used in connection with a list of alternatives, it is generally understood to be used in an inclusive sense unless the context dictates otherwise. Thus, a statement of "x or y" is equivalent to "x or y, or both," and thus covers 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, a phrase such as "either x or y, but not both" makes it clear that "or" is used in an exclusive sense.

[0086] The phrases "w, x, y, or z, or any combination thereof" or "... at least one of w, x, y, and z" are intended to cover all possibilities, including single elements, up to the total number of elements in the set. For example, given the set [w, x, y, z], these phrases cover any single element of the set (e.g., w but not x, y, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Thus, the phrase "... at least one of w, x, y, and z" refers to at least one element of the set [w, x, y, z], thereby covering all possible combinations within this list of elements. This phrase should not be interpreted as requiring that there be at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0087] In this disclosure, various "labels" may precede nouns or noun phrases. Unless the context dictates otherwise, different labels used for a feature (e.g., "first circuit," "second circuit," "particular circuit," "given circuit," etc.) refer to different instances of that feature. Furthermore, when applied to features, the labels "first," "second," and "third" do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise indicated.

[0088] The phrase "based on" or "based on" is used to describe one or more factors that influence the decision. This term does not exclude the possibility that additional factors may influence the decision. That is, the decision may be based only on the specified factors, or it may be based on the specified factors and other unspecified factors. Consider the phrase "determining A based on B." This phrase specifies that B is a factor used to determine A or that influences the decision on A. This phrase does not exclude that the decision on A may be further based on some other factor, such as C. This phrase is also intended to cover embodiments in which A is determined only based on B. As used herein, the phrase "based on" is synonymous with "based at least in part on."

[0089] The phrases "in response to" and "depending on" describe one or more factors that cause an effect. This phrase does not exclude the possibility that additional factors may influence or otherwise trigger the effect, either jointly with or independently of the specified factors. That is, the effect may be responsive only to those factors, or may be responsive to the specified factors and other unspecified factors. Consider the phrase "executing A in response to B." This phrase specifies that B is a factor that triggers the execution of A or triggers a particular result of A. This phrase does not exclude that executing A may be further responsive to some other factor, such as C. This phrase further does not exclude that executing A may be responsive to B and C jointly. This phrase is further intended to cover embodiments in which A is executed only in response to B. As used herein, the phrase "in response to" is synonymous with "at least partially responsive to." Similarly, the phrase "in response to" is synonymous with "at least partially responsive to."

[0090] Within this disclosure, different entities (which may be variously referred to as "units," "circuits," other components, etc.) may be described or claimed as being "configured" to perform one or more tasks or operations. This formulation—[entity] configured to [perform one or more tasks]—is used herein to refer to a structure (i.e., something physical). More specifically, this formulation is used to indicate that this structure is arranged to perform one or more tasks during operation. A structure can be said to be "configured to" perform a task even if the structure is not currently in operation. Thus, an entity described or described as being "configured to" perform a task refers to something physical, such as a device, a circuit, or a system having a processor unit and a memory that stores executable program instructions to implement the task. This phrase is not used herein to refer to something intangible.

[0091] In some cases, various units / circuits / components may be described herein as performing a set of tasks or operations, and these entities will be understood to be "configured to" perform these tasks / operations, even if not specifically described.

[0092] The term "configured to" is not intended to mean "configurable to." For example, an unprogrammed FPGA is not considered to be "configured to" perform a particular function. However, the unprogrammed FPGA may be "configurable to" perform that function. After appropriate programming, the FPGA can then be said to be "configured to" perform a particular function.

[0093] For purposes of U.S. patent applications based on this disclosure, reciting in a claim that a structure is "configured to" perform one or more tasks is expressly intended to not invoke 35 U.S.C. § 112(f) for that claim element. If applicant wishes to invoke Section 112(f) during prosecution of a U.S. patent application based on this disclosure, they should recite the claim element using the "means for" construct [to perform a function].

[0094] Different "circuits" may be described in this disclosure. These circuits or "circuitry" comprise hardware that includes various types of circuit elements, such as combinational logic, clocked storage (e.g., flip-flops, registers, latches, etc.), finite state machines, memory (e.g., random access memory, embedded dynamic random access memory), programmable logic arrays, etc. Circuitry may be custom designed or obtained from standard libraries. In various implementations, circuitry may include digital components, analog components, or a combination of both, as appropriate. Particular types of circuits may be generically referred to as "units" (e.g., decode units, arithmetic logic units (ALUs), functional units, memory management units (MMUs), etc.). Such units may also be referred to as circuits or circuitry.

[0095] Thus, the disclosed circuits / units / components and other elements shown in the drawings and described herein include hardware elements as described in the preceding paragraph. Often, the internal arrangement of hardware elements within a particular circuit can be specified by describing the circuit's function. For example, a particular "decode unit" may be described as performing the function of "processing the instruction's opcode and routing the instruction to one or more of a plurality of functional units," meaning that the decode unit is "configured to" perform this function. This specification of this function is sufficient to denote to one of ordinary skill in the computer arts a set of possible configurations of the circuit.

[0096] In various embodiments, as discussed in the previous paragraph, circuits, units, and other elements are defined by the functions or operations they are configured to implement. The arrangement of such circuits / units / components relative to one another and the manner in which they interact form a microarchitecture definition of the hardware that is ultimately fabricated in integrated circuits or programmed into FPGAs to form a physical implementation of the microarchitecture definition. Thus, a microarchitecture definition is recognized by those skilled in the art as a structure from which many physical implementations can be derived, all of which fall within the broader structure described by the microarchitecture definition. That is, presented with a microarchitecture definition provided in accordance with this disclosure, one skilled in the art can, without undue experimentation and application of ordinary skill, implement the structure by coding the circuit / unit / component description in a hardware description language (HDL) such as Verilog or VHDL. HDL descriptions are often expressed in a way that may appear functional. However, to those skilled in the art, this HDL description is a method used to transform the structure of a circuit, unit, or component to the next level of implementation detail. Such HDL descriptions may take the form of behavioral code (which is typically not synthesizable), register transfer language (RTL) code (which, in contrast to behavioral code, is typically synthesizable), or structural code (e.g., a netlist specifying logic gates and their connectivity). The HDL descriptions may then be synthesized against a library of cells designed for a given integrated circuit fabrication technology and may be modified for timing, power, and other reasons to result in a final design database that is sent to a foundry to generate masks and ultimately produce the integrated circuit. Some hardware circuits, or portions thereof, may be custom designed in a schematic editor and incorporated into the integrated circuit design along with the synthesized circuitry. Integrated circuits may include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.) and interconnections between the transistors and circuit elements.Some embodiments may implement multiple integrated circuits coupled together to implement a hardware circuit, and / or some embodiments may use discrete elements. Alternatively, an HDL design may be synthesized into and implemented in a programmable logic array, such as a field programmable gate array (FPGA). This separation between the design of a group of circuits and the subsequent low-level implementation of those circuits generally results in a scenario where a circuit or logic designer does not specify a particular set of structures for the low-level implementation beyond describing what the circuit is configured to do, because this process is performed at a different stage in the circuit implementation process.

[0097] The fact that many different low-level combinations of circuit elements may be used to implement the same specification for a circuit results in a large number of equivalent structures for that circuit. As previously mentioned, these low-level circuit implementations may vary according to changes in fabrication technology, the foundry selected to manufacture the integrated circuit, the library of cells provided for a particular project, etc. In many cases, the choices made by different design tools or methodologies to produce these different implementations may be arbitrary.

[0098] Furthermore, it is common for a single implementation of a particular functional specification of a circuit to include a large number of devices (e.g., millions of transistors) for a given embodiment. Thus, the sheer volume of this information makes it impractical to provide a complete description of the low-level structure used to implement a single embodiment, let alone the vast array of equivalent possible implementations. For this reason, this disclosure describes the structure of a circuit using functional shorthand commonly adopted in the industry.

Claims

1. receiving, at a first database node of a plurality of database nodes of a database system, a request to perform a database transaction including writing a particular record for a key contained within a key space to an in-memory cache of the first database node; accessing, by the first database node, a key spatial reference catalog stored in a catalog manager node of the database system to retrieve from the key spatial reference catalog a plurality of key spatial references, each of the plurality of key spatial references corresponding to a respective database node of the plurality of database nodes, each of the plurality of key spatial references corresponding to the respective database node including a key space to which the corresponding database node is authorized to write one or more records, and a time period during which the corresponding database node of the plurality of database nodes of the database system wrote one or more records for keys contained in the key space to an in-memory cache of the corresponding database node; determining, by the first database node based on the plurality of key space references, that a second database node of the plurality of database nodes may be writing records for the key space that potentially conflict with the particular record within a particular time frame to an in-memory cache of the second database node; in response to said determining, said first database node sending to said second database node a request for information indicating whether said second database node has written to said in-memory cache of said second database node a record for said key that conflicts with said particular record; the first database node writing the particular record to the in-memory cache of the first database node based on a response indicating that the second database node has not yet written a record that conflicts with the particular record to the in-memory cache of the second database node; Including, method.

2. prior to receiving the request to perform the database transaction, the first database node: issuing an authorization request to the catalog manager node of the database system for approval to write a record to the keyspace; receiving permission to write records to the keyspace from the catalog manager node, the permission being transferred from the second database node to the first database node; The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein active transactions on the second database node that include writing records to the key space to the in-memory cache of the second database node are permitted to commit after the permission to write records to the key space is received by the first database node, and at least one of the active transactions caused a record write to the key space before the permission was received by the first database node.

4. 3. The method of claim 2, wherein the key spatial reference catalog stores a key spatial reference that indicates that all record writes to the keyspace should occur only at the first database node while the permission to write to the keyspace is provisioned on the first database node.

5. receiving, by the first database node, from the catalog manager node, a relinquish request to relinquish the permission to write records for the keyspace to permit the second database node to write records for the keyspace to the in-memory cache of the second database node as part of an active transaction of the second database node; the first database node relinquishing the permission in response to the relinquishment request; 5. The method of claim 4, further comprising:

6. only a particular key spatial reference of the plurality of key spatial references identifies a time frame denoted as an epoch range for the key space, and no other key spatial references of the plurality of key spatial references identify the time frame, and the particular key spatial reference of the plurality of key spatial references is associated with the second database node, the method comprising: the first database node making a determination that a time window corresponding to the database transaction falls within the time window, denoted as the epoch range, the determination indicating that the second database node may be writing a record for the key space that potentially conflicts with the particular record; The method of claim 1 further comprising:

7. 7. The method of claim 6, wherein only the epoch range of the particular key spatial reference is modified in response to a commitment of a last active transaction linked to the particular key spatial reference, the epoch ranges of key spatial references other than the particular key spatial reference are not modified in response to the commitment of the last active transaction, and the epoch range of the particular key spatial reference does not define an upper epoch of the epoch range prior to the modification.

8. 7. The method of claim 6, wherein the particular key spatial reference is deleted after all records written at the second database node for the key space are stored in a persistent database of the database system, and key spatial references other than the particular key spatial reference are not deleted after all records written at the second database node for the key space are stored in a persistent database of the database system.

9. 2. The method of claim 1, wherein one of the three or more key space references identifies a database node authorized to write records to the key space, and two or more of the three or more key space references identify database nodes for reading records written to the key space.

10. based on a response indicating that the second database node has already written a record that conflicts with the particular record to the in-memory cache of the second database node, the first database node aborting at least a portion of the database transaction that included writing the particular record to the in-memory cache of the first database node; 10. The method of claim 1, further comprising:

11. A computer-readable storage medium storing a program, the program, when executed by a first database node of a plurality of database nodes in a database system, causing the first database node to: receiving, by the first database node of the database system, a request to perform a database transaction including writing a particular record for a key contained within a key space to an in-memory cache of the first database node; an operation by the first database node accessing a key spatial reference catalog stored in a catalog manager node of the database system to retrieve a plurality of key spatial references from the key spatial reference catalog, each of the plurality of key spatial references corresponding to a respective database node of the plurality of database nodes, each of the plurality of key spatial references corresponding to the respective database node including a key space to which the corresponding database node is authorized to write one or more records, and a time period during which the corresponding database node of the plurality of database nodes of the database system wrote records for keys included in the key space to an in-memory cache of the corresponding database node; determining, by the first database node based on the plurality of key space references, that a second database node of the plurality of database nodes may have written records for the key space that potentially conflict with the particular record within a particular time frame to an in-memory cache of the second database node; in response to said determining, said first database node sends to said second database node a request for information indicating whether said second database node has written to said in-memory cache of said second database node a record for said key that conflicts with said particular record; writing, by the first database node, the particular record to the in-memory cache of the first database node based on a response indicating that the second database node has not yet written a record that conflicts with the particular record to the in-memory cache of the second database node; causing an action to be performed, including A computer-readable storage medium.

12. 12. The computer-readable storage medium of claim 11, wherein a particular key spatial reference among the plurality of key spatial references corresponds to the second database node, and key spatial references other than the particular key spatial reference correspond to database nodes other than the second database node, and the particular key spatial reference and each of the key spatial references other than the particular key spatial reference identify a key space by a minimum key and a maximum key and specify a time frame.

13. The operation is the first database node requesting authorization from the catalog manager node of the database system to write a record to the keyspace at the first database node; and receiving, by the first database node, permission to write a record to the keyspace from the catalog manager node; 12. The computer-readable storage medium of claim 11, wherein a particular key spatial reference corresponding to the first database node is stored in the key spatial reference catalog, and the particular key spatial reference corresponding to the first database node indicates that all record writes to the key space included in the particular key spatial reference should occur at the first database node.

14. The operation is the first database node performing another database transaction including writing another specific record for the key contained within the key space; the first database node using the key space reference catalog to determine that the permission to write records to the key space has been transferred to a third database node; sending a write request from the first database node to the third database node to write the other particular record; 14. The computer-readable storage medium of claim 13, further comprising:

15. a first database node, the first database node comprising: at least one processor; a memory for storing a program; When the program is executed by the at least one processor, the program causes the first database node to perform the method of any one of claims 1 to 10. The first database node.

Images