Key permission distribution

The permission orchestrator node in database systems manages key permissions to prevent simultaneous writes across nodes, addressing transaction conflicts and improving processing speed by ensuring only one node writes to a key at a time, thus enhancing system efficiency.

JP7593688B2Active Publication Date: 2024-12-03SALESFORCE INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023506487
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-09-23
Filing Date
2021-09-08
Publication Date
2024-12-03
Estimated Expiration
2041-09-08

AI Technical Summary

Technical Problem

In multi-active node database systems, there is a risk of simultaneous writing of records with the same database key by multiple nodes, leading to transaction isolation violations and visible anomalies, as existing systems lack effective mechanisms for distributing and redistributing database key permissions among active nodes.

Method used

A permission orchestrator node orchestrates the distribution of key permissions among worker nodes, ensuring that only one node at a time has the authority to write to a particular key, by relinquishing and reallocating permissions while ensuring no conflicts occur, using techniques such as lock management and historical commit numbers to prevent concurrent writes.

Benefits of technology

This approach prevents simultaneous writing of records with the same key, thereby avoiding transaction isolation violations and ensuring faster transaction processing by allowing key permissions to be redistributed among active nodes, thus enhancing system efficiency and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593688000001
    Figure 0007593688000001
  • Figure 0007593688000002
    Figure 0007593688000002
  • Figure 0007593688000003
    Figure 0007593688000003
Patent Text Reader

Abstract

Techniques for distributing database key permissions are disclosed. A database system may distribute first permission information to a plurality of database nodes, the first permission information identifying a set of key range permissions for each of the plurality of database nodes. When a given key range permission is distributed to a given database node, the database node may be authorized to write records whose keys fall within the key range associated with the given key range permission. The database system may receive a request from a first database node for the first key range permission to be provisioned to a second database node. The database system may modify the first permission information to derive second permission information that provisions the first key range permission to the first database node instead of the second database node. The database system may distribute the second permission information to each of the plurality of database nodes.
Need to check novelty before this filing date? Find Prior Art

Description

Related Technology

[0001] The present disclosure relates generally to database systems, and more specifically to distributing database key permissions among database nodes. [Background technology]

[0002] Modern database systems routinely implement management systems that allow collections of information to be stored in an organized manner so that users can efficiently access and manipulate them. In some cases, these management systems maintain log-structured merge trees (LSM trees) with multiple levels, each of which stores database records as key-value pairs. LSM trees typically include two high-level components: an in-memory cache and persistent storage. During operation, the database system receives a transaction request to process a transaction that involves writing database records to persistent storage. The database system first writes the database records to the in-memory cache and then flushes them to persistent storage. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram illustrating example elements of a system including an authorization orchestrator node and a worker node, according to some embodiments.

[0004] [Diagram 2] FIG. 2 is a block diagram illustrating example elements of a worker node that relinquishes key permissions, according to some embodiments.

[0005] [Diagram 3] FIG. 2 is a block diagram illustrating example elements of an authorization orchestrator node, according to some embodiments.

[0006] [Figure 4]FIG. 2 is a block diagram illustrating exemplary elements of a worker node that obtains key permissions, according to some embodiments.

[0007] [Diagram 5] 1 is a flow diagram illustrating an example method for distributing key authorizations according to some embodiments. [Figure 6] 1 is a flow diagram illustrating an example method for distributing key authorizations according to some embodiments.

[0008] [Figure 7] FIG. 2 is a block diagram illustrating elements of a multi-tenant system, according to some embodiments.

[0009] [Figure 8] FIG. 1 is a block diagram illustrating elements of a computer system according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] As mentioned above, modern database systems may maintain an LSM tree that contains database records having key-value pairs. Often, the database system includes a single active database node (and multiple standby database nodes) responsible for writing database records to the persistent storage component of the LSM tree. However, in some cases, the database system includes multiple active database nodes that are writing database records to the LSM tree. The multiple active database nodes may share a common persistent storage, but have their own individual in-memory caches. In such an implementation, database records written by an active database node to its individual in-memory caches are not visible to other active database nodes until those records are flushed to the common persistent storage.

[0011] In implementations with multiple active database nodes, the possibility exists that two or more database nodes could write records for the same database key at nearly the same time, unless those database nodes are restricted in some way. (In contrast, in implementations with a single active database node, this scenario is of little concern, since there is a local lock manager that ensures that changes from local transactions are coordinated and do not conflict.) Thus, in some multi-active node implementations, one of the active database nodes is provisioned with a database key grant for a particular database key, permitting that database node to write records for the database key. However, the other active database node is not provisioned with the database key grant, and therefore cannot write records for the particular database key. However, in some cases, it may be desirable to reprovision the database key grant from the owner active database node to another active database node. As an example, if the owner database node is overwhelmed with other database work and there is an outstanding transaction involving writing a record for a particular database key, it may be desirable to offload the outstanding transaction to another database node. The present disclosure addresses, inter alia, this technical challenge of being able to ensure that multiple database nodes do not write records with the same key at the same time while still allowing database key permissions to be redistributed to meet the needs of the database system.

[0012] More specifically, this disclosure describes various techniques for orchestrating the distribution of database key permissions among multiple database nodes of a database system. In various embodiments described below, the database system includes an authorization orchestrator node that orchestrates the distribution of key permissions among worker nodes capable of writing database records to a database of the database system. During operation of the database system, a first worker node may receive a request to execute a transaction that involves writing a record for a particular database key (e.g., database key "XYZ"). In various embodiments, the first worker node first accesses authorization information from the authorization orchestrator node (if the first worker node has not previously obtained the authorization information) to determine whether it has authorization to write a database record for the particular database key. In some cases, the authorization information may indicate that the associated key permissions have been provisioned to the first worker node, and the first worker node may write the record. The first worker node may also ensure that no other record has been committed for the particular database key within a particular time frame before it writes the record for the transaction. Otherwise, the authorization information may indicate that the associated key authorization is not provisioned or is provisioned to another worker node. The first worker node then issues a request to the authorization orchestrator node for the associated key authorization. In some cases, the first worker node may request a key range authorization that includes the associated database key authorization (e.g., key range "XY" that includes database key "XYZ").

[0013] Upon receiving a request from a given worker node, in various embodiments, the authorization orchestrator node first determines whether the associated key grant is owned by another worker node. If the associated key grant is not owned, the authorization orchestrator node may generate updated authorization information to provision the associated key grant to the first worker node. The authorization orchestrator node may then distribute the authorization information to the worker nodes of the database system. In some embodiments, the authorization orchestrator node may provision a range of key grants (which may be referred to as a "key range grant") that includes the associated key grant. As an example, the authorization orchestrator node may provision a key grant for key range "XY" (encompassing any key beginning with "XY", including the specific database key "XYZ"). However, if the second worker node owns the associated key grant, the authorization orchestrator node, in some embodiments, sends a relinquishment request to the second worker node to relinquish the associated key grant.

[0014] Upon receiving the relinquish request, in various embodiments, the second worker node determines whether there is an active transaction on the second worker node that has locked the associated key grant for record commit. An active transaction may lock a key grant if it is attempting to commit a record for a database key corresponding to the key grant. A locked key grant cannot be used by other active transactions or relinquished back to the grant orchestrator node while the key grant is locked / held. If the associated key grant is held by an active transaction running on the second worker node, the second worker node waits until the transaction is completed and then returns the key grant. In some embodiments, the second worker node may relinquish a set of key grants that includes the associated key grant. In some embodiments, the second worker node also returns history information identifying a transaction commit number ("XCN") associated with the relinquished key grant. The XCN may indicate when the most recent database record was committed for the relinquished key grant (or the set of key grants that includes the relinquished key grant). Consider an example where the most recently committed database record for key range "XY" was committed with an XCN of "501". As a result, the second worker node may return historical information indicating a maximum XCN of "501" for the relinquished key grant "XYZ". After ensuring that the associated key grant has been revoked at the second worker node, in various embodiments, the authorization orchestrator node provisions the associated key grant to the first worker node by updating the authorization information and distributing it to the worker nodes. The updated authorization information may include the historical information returned by the second worker node so that other worker nodes may determine whether they may commit records for the particular database key.

[0015] Upon receiving the associated key grant, the first worker node may use its grant information to determine whether the record of the associated key grant was committed within a particular time frame. For example, it may determine whether the second worker node committed the record of the associated key grant while in possession of the key grant. If the record appears to be committed, the first worker node may communicate with the second worker node to determine whether the record was actually committed. If the record is committed, the first worker node aborts the transaction to ensure that the two transactions do not conflict, otherwise the first worker node writes the record of the associated key grant to its in-memory cache. Upon relinquishing the key grant (e.g., in response to receiving a relinquish request), in various embodiments, the first worker node notifies the grant orchestrator node of the write and commit of the record so that other worker nodes can know when they have acquired the key-related grant. In some embodiments, the associated key grant identifies the record commit. As a result, the key permission may be a list of locations for recent updates as well as a lock and concurrency protection.

[0016] In this manner, database key permissions may be distributed and redistributed among worker nodes by the permission orchestrator node. This may result in a "tug of war" scenario in which a worker node "tugs" a database key permission in order to write a database record for a transaction in the system. That is, a worker node may acquire a particular key permission, write a record for the particular key permission, relinquish the particular key permission, and then reacquire the particular key permission for another record write. As a result, the particular key permission may be "taken back" by a worker node attempting to write a record using the particular key permission.

[0017] These techniques may be advantageous because they allow a database system to be implemented with multiple active database nodes while ensuring that multiple database nodes are not writing records for the same database key at approximately the same time, potentially causing a violation of transaction isolation and introducing application visible anomalies. That is, key grants are allocated to at most one database node at a time, and other database nodes are prevented from writing database records with keys associated with the key grant. These techniques may be further advantageous because they allow key grants to be moved between active database nodes such that one active database node does not have to process all of the work associated with the database key. This may result in transactions being processed more quickly compared to previous database system implementations. Beginning with reference to FIG. 1, an exemplary application of these techniques will now be described.

[0018] Referring now to FIG. 1, a block diagram of a system 100 is shown. The system 100 includes a set of components that may be implemented via hardware or a combination of hardware and software routines. In the illustrated embodiment, the system 100 includes a database 110 having an LSM file 115, worker nodes 120A and 120B, and an authorization orchestrator node 150. As further illustrated, the worker nodes 120 and the authorization orchestrator node 150 include authorization information 140 that defines key range permissions 145A and 145B. As also illustrated, the worker nodes 120 include respective in-memory caches 130 that store database records 132 associated with keys 134. In some embodiments, the system 100 is implemented differently than illustrated. For example, the authorization orchestrator node 150 may implement the functionality of the worker nodes 120 in addition to orchestrating the distribution of the key range permissions 145. Additionally, although the techniques of this disclosure are described with respect to LSM trees, the techniques may be applied to other types of database implementations in which multiple nodes are writing and committing database records.

[0019] System 100, in various embodiments, implements a platform service (e.g., a customer relationship management (CRM) platform service) that enables users of that service to develop, execute, and manage applications. System 100 may be a multi-tenant system that provides various functions to multiple users / tenants hosted by the multi-tenant system. Thus, system 100 may execute software routines from various different users (e.g., providers and tenants of system 100) and provide code, web pages, and other data to users, databases, and other entities associated with system 100. As shown, for example, system 100 includes worker nodes 120 that can store, manipulate, and retrieve data from LSM files 115 of database 110 on behalf of users of system 100.

[0020] The database 110, in various embodiments, is a collection of information organized to allow access, storage, and manipulation of that information. Thus, the database 110 may include supporting software that allows the worker nodes 120 to perform operations (e.g., access, store, etc.) on the information stored in the database 110. In some embodiments, the database 110 is implemented by a single or multiple storage devices (e.g., a storage attached network (SAN)) connected to each other over a network and configured to store information redundantly to prevent data loss. The storage devices may store data persistently, and thus the database 110 may function as a persistent storage device. In various embodiments, a database record 132 written to the LSM file 115 by one worker node 120 is accessible by the other worker node 120. The LSM file 115 may be stored as part of a log-structured merge tree implemented in the database 110.

[0021] The LSM tree is a data structure that, in various embodiments, stores the LSM files 115 in an organized manner using a level-based scheme. The LSM tree may include two high-level storage elements: an in-memory storage element implemented in an in-memory cache 130, and an on-disk storage element implemented in the database 110. In some embodiments, the in-memory cache 130 is considered separate from the LSM tree. The work nodes 120, in various embodiments, first write the database records 132 to their in-memory cache 130. Once the cache 130 becomes full and / or at certain points in time, the worker nodes 120 may flush their database records 132 to the database 110. As part of flushing the database records 132, in various embodiments, the worker nodes 120 write the database records 132 to a new set of LSM files 115 in the database 110.

[0022] An LSM file 115, in various embodiments, is a set of database records 132. A database record 132 may be a key-value pair that includes data and a corresponding database key 134 that can be used to look up that database record. For example, a database record 132 corresponds to a row of data in a database table, and the database record 132 specifies values ​​for one or more attributes associated with the database table. In various embodiments, a file 115 is associated with one or more database key ranges that are defined by the keys 134 of the database records 132 contained in that LSM file 115. Consider an example in which a file 115 stores three database records 132 each associated with a key 134 "XYA", "XYW", and "XYZ", respectively. These three keys 134 span the database key range of XYA→XYZ, and therefore the LSM file 115 is associated with that database key range.

[0023] Worker node 120, in various embodiments, may be hardware, software, or a combination thereof, and may provide database services such as data storage, data retrieval, and / or data manipulation. These database services may be provided to other components within system 100 or to components external to system 100. For example, worker node 120A may receive a database transaction request to perform one or more database tasks. The request may be received from an application server seeking to access a set of database records 132. The transaction request may specify a SQL SELECT command to select one or more rows from one or more database tables. The contents of the rows may be defined in the database records 132, and thus worker node 120A may return one or more database records 132 that correspond to the selected one or more table rows. In various cases, the database transaction request may instruct worker node 120 to write one or more database records 132 to an LSM tree. In various embodiments, a worker node 120 first writes a database record 132 to its in-memory cache 130 before flushing it to the database 110 .

[0024] In-memory cache 130, in various embodiments, is a buffer that stores data in memory (e.g., random access memory) of worker node 120. HBase® Memstore is one example of an in-memory cache 130. As described above, a worker node 120 may first write database records 132 to its in-memory cache 130. In some cases, the latest / most recent version of a row in a database table may be found in the database records 132 stored in the in-memory cache 130. However, the database records 132 written to the in-memory cache 130 of a worker node 120 are not visible to other worker nodes 120, in various embodiments. That is, other worker nodes 120 do not know what information is stored in the in-memory cache 130 of the worker node 120 unless they ask. To prevent database record collisions due to one worker node 120 not knowing about database records 132 written by another worker node 120, in various embodiments, worker nodes 120 are provisioned with permission information 140 that controls which records 132 can be written by a given worker node 120. In this manner, the permission information 140 can prevent two or more worker nodes 120 from writing database records 132 for the same database key 134 within a particular time interval, preventing worker nodes 120 from flushing conflicting database records 132 to the database 110.

[0025] The permission information 140, in various embodiments, is information that identifies a corresponding set of key range permissions 145 and owners for those key range permissions. For example, worker node 120A is provisioned with key range permission 145A (shown by a solid line box on worker node 120A) and worker node 120B is provisioned with key range permission 145B (shown by a solid line box on worker node 120B). The key range permission 145, in various embodiments, indicates a range of key permissions that corresponds to a range of database keys 134 for which the owning worker node 120 is authorized to write database records 132. As an example, the key range permission 145A may indicate a key permission for key 134A. As a result, the worker node 120A can write database record 132A to its in-memory cache 130 as shown. In various embodiments, a key permission is provided to at most one worker node 120 at any given time. Consider an example where the key range permission 145B indicates a key permission for key 134B. That key permission is provisioned to worker node 120B, and worker node 120A is not authorized to write record 132 with key 134B. To be authorized to write record 132 for a particular key 134, in various embodiments, worker node 120 may issue an authorization request 112 to authorization orchestrator node 150 that specifies the particular key 134. In various cases, authorization request 112 may specify multiple keys 134 (e.g., a key range).

[0026] The authorization orchestrator node 150, in various embodiments, facilitates distribution of key grants among the worker nodes 120 and ensures that at most one worker node 120 has ownership of the key grant. As part of facilitating distribution of key grants, in various embodiments, the authorization orchestrator node 150 may update the authorization information 140 in response to receiving the authorization request 112. As shown, the authorization orchestrator node 150 receives the authorization request 112 from the worker node 120B. In response to receiving the authorization request 112, the authorization orchestrator node 150 may first determine whether a key grant for the requested database key 134 has already been provisioned. If not, the authorization orchestrator node 150 may update the authorization information 140 to provision the key grant for the worker node 120B, inform the worker node 120B of the update via the authorization response 114, and inform the other worker nodes 120 of the update via the authorization information indication 156. In various embodiments, all worker nodes 120, including worker node 120B, are notified via permission information display 156.

[0027] If a key grant (or key range grant 145 if a key range is requested) is provisioned, the authorization orchestrator node 150 may identify the owning worker node 120 and issue a release request 152 to that worker node 120. As an example, worker node 120B may issue an authorization request 112 for a key grant associated with key 134A. The authorization orchestrator node 150 may determine that the key grant is part of the key range grant 145A already provided to worker node 120A. Thus, the authorization orchestrator node 150 may issue a relinquish request 152 to worker node 120A. When relinquishing a key grant, the worker node 120, in various embodiments, ensures that the key grant to be relinquished is not in use. The worker node 120 may then relinquish the key grant and notify the authorization orchestrator node 150 via a relinquish response 154. In some cases, the worker node 120 may include historical information in the relinquish response 154 indicating whether the record 132 may have been committed using a relinquished key grant (or, in some cases, a relinquished key range grant 145).

[0028] In response to receiving the relinquish response 154, the authorization orchestrator node 150 may update the authorization information 140 to provision the key authorization to the worker node 120B, notify the worker node 120B of the update via the authorization response 114, and notify the other worker nodes 120 of the update via the authorization information indication 156. In this manner, the authorization orchestrator node 150 may distribute and redistribute ownership of the key authorization to the worker nodes 120 by updating the authorization information 140 in response to the authorization request 112 and propagating the updated authorization information 140 to the worker nodes 120.

[0029] 2, a block diagram of an exemplary layout for a worker node 120 relinquishing a key grant is shown. In the illustrated embodiment, worker node 120A includes an in-memory cache 130, grant information 140, and a database application 200. As further shown, database application 200 is processing an active transaction 210 holding lock 215 and a committed transaction 220 having an associated transaction commit number (XCN) 225. Also shown, grant information 140 includes key range grants 145A-145C having key grant 205 and corresponding history information 230 specifying XCN 225. In some embodiments, worker node 120A is implemented differently than shown. As an example, database application 200 may process multiple active transactions 210 and multiple committed transactions 220.

[0030] Database application 200 is a set of program instructions executable to manage database 110, which in various embodiments includes managing an LSM tree built around database 110. Thus, database application 200 may process database transactions to read records 132 from the LSM tree and write records 132. To aid in the processing of database transactions, in various embodiments, database application 200 maintains metadata that describes the structural layout of the LSM tree, including where files 115 are stored in database 110 and what records 132 may be contained in those files 115. In some implementations, the metadata includes tries that correspond to files 115. Database application 200 may use the metadata to perform faster and more efficient key range lookups as part of processing database transactions.

[0031] As described, the database application 200 may receive a request to execute a transaction to read and write database records 132. Upon receiving the transaction request, the database application 200 may initiate an active transaction 210 based on the received transaction request. The active transaction 210, in various embodiments, refers to an ongoing transaction in which the database application 200 is writing database records 132 in the in-memory cache 130. While a transaction is an active transaction 210, the database records 132 written for that transaction are not committed and cannot be readable / accessible by another entity other than the worker node 120 that wrote them. In various cases, the database application 200 may decide to execute a rollback of the active transaction 210, and thus the database records 132 written for the active transaction 210 are not committed and are flushed. For example, if database application 200 is not permitted to write a particular database record 132 based on a set of criteria (e.g., a time period analysis, described in more detail with respect to FIG. 4), database application 200 may roll back the active transaction 210 that includes writing that database record 132. In some cases, database application 200 may decide to perform a rollback of only a sub-portion of active transaction 210.

[0032] To write a database record 132 when processing a given active transaction 210, in various embodiments, the database application 200 determines whether the worker node 120A has the appropriate permissions 205 and acquires locks 215 for those permissions 205 before writing the database record 132. Consider an example in which an example active transaction 210 involves writing a database record 132 having a key 134 with a value "XYT" (referred to as "key 134(XYT)"). The database application 200 may first check the permission information 140 to determine whether an authorization key 205 for the key 134 has been provided to the worker node 120A. The key permission 205, in various embodiments, identifies the associated key 134 and the owner of that key 134. As shown, key range permissions 145A and 145C (illustrated by solid lined boxes) have been provisioned to the worker node 120A, and key range permission 145B (illustrated by dashed lined box) has not been provided to the worker node 120A. Thus, in the illustrated embodiment, database application 200 determines that key permissions 205A identify worker node 120A as the owner of key 134(XYT), although in other cases worker node 120A may not be the owner and therefore must request ownership of key 134(XYT). An example of requesting ownership is described in more detail with respect to FIG.

[0033] After the worker node 120 determines that the desired key grant 205 has been provisioned to the worker node 120, in various embodiments, the worker node 120 obtains a lock 215 for the associated key 134 when writing the database record 132 with the key 134. Continuing with the previous example, the worker node 120A may obtain a lock 215(XYT) for the key 134(XYT). The lock 215, in various embodiments, may prevent other entities (e.g., other worker nodes 120) from writing the database record 132 with the associated key 134, and may further prevent the associated key grant 205 from being revoked and reprovisioned to other entities. Once the lock 215 is obtained, the worker node 120 may write the database record 132 for the corresponding key 134. As described in more detail with respect to FIG. 4, before writing the database record 132, the worker node 120 may further determine whether another worker node 120 has committed a database record 132 for the same database key 134 within a certain period of time based on the XCN 225.

[0034] After processing the active transaction 210 (e.g., after writing all database records 132 required for the transaction), the worker node 120 may commit the transaction, resulting in the committed transaction 220. As part of the commit process, in various embodiments, the worker node 120 stamps the transaction commit number 225 on each database record 132 of the transaction. As an example, the committed transaction 220 has an XCN225 of T501 (referred to as "XCN225(T501)"). Thus, each record 132 associated with the committed transaction 220 may include metadata identifying XCN225(T501). XCN225 is, in various embodiments, a monotonically increasing value and can thus indicate a time period. That is, during operation, the system 100 may periodically increment the XCN225 of the database system assigned to a transaction upon commit. Since the XCN225 of the database system is periodically incremented, two committed transactions 220 may be associated with different XCN225s. For example, the first committed transaction 220 may be assigned to XCN225(T501), and the second committed transaction 220 may be assigned to XCN225(T412). Since the XCN225 of the database system is incremented, the worker node 120 (or another entity) can determine that the second committed transaction 220 occurred earlier in time than the first committed transaction 220 (T412<T501). In various embodiments, the committed database records 132 become available to other worker nodes 120 upon request and are ultimately written to the file 115 in the database 110.

[0035] As part of the commit process, in various embodiments, the worker node 120 also updates history information 230 based on the XCN 225 of the transaction being committed. In various embodiments, the history information 230 identifies, for a key permission 205 and / or key range permission 145, the XCN 225 associated with the most recent record commit with that key permission 205 / key range permission 145. That XCN 225 is referred to as the "maximum XCN" or "latest XCN" for that key permission 205 / key range permission 145. Consider the illustrated example of a database record 132 having key 134 being committed for the committed transaction 220. Worker node 120A holds key permission 205B, and since no other worker node 120 has permission to write to key 134 (XYZ) while worker node 120A holds key permission 205B, the database record 132 is the most recently committed record 132 for key 134 (XYZ). Thus, worker node 120A may update history information 230 to associate key grant 205B (or key range grant 145A that includes key grant 205B) with XCN 225(T501).

[0036] The history information 230 may be updated in different ways. In various cases, the worker node 120 may update the history information 230 to associate each key permission 205 used in a given transaction at the worker node 120 with the XCN 225 of the given transaction. In some cases, a set of key permissions 205 may be grouped and provisioned as a key range permission 145, and a portion of the history information 230 may be linked to that key range permission 145. Thus, the worker node 120 may update the history information 230 to associate the key range permission 145 with the XCN 225. This may be referred to as a "key range XCN." As shown, the key range permission 145C is associated with the XCN 225 (T412). Thus, the latest committed database record 132 for the key range permission 145C occurs in the committed transaction 220 having the XCN 225 (T412). However, other records 132 for key range grant 145C may be associated with a different, smaller XCN 225, and therefore were committed at an earlier time by another worker node 120. As will be described in more detail with respect to Figure 4, the worker node 120 may use history information 230 to determine whether to abandon writing the record.

[0037] During operation, the worker node 120A may receive a relinquish request 152 to relinquish one or more key grants 205 (or key range grants 145). The relinquish request 152 may be received from the authorization orchestrator node 150 and may identify one or more key grants 205 to be relinquished. In response to receiving the relinquish request 152, the worker node 120A, in various embodiments, first prevents any new transactions from acquiring locks 215 on the requested key grants 205. The worker node 120A may then determine whether any active transactions 210 have locks 215 on the requested key grants 205. If there are no locks 215 associated with these key grants 205, the worker node 120A may send a relinquish response 154 back to the authorization orchestrator node 150. In some cases, the worker node 120A may relinquish the key grants 205 but retain the key grants 205 for the key ranges that include the relinquished key grants 105. For example, the worker node 120A may relinquish key grant 205 “XYZ” but retain other remaining key grants 205 for key range “XY”. The relinquish response 154, in various embodiments, includes an indication that the requested key grant 205 has been relinquished and will therefore not be used for transactions unless the key grant 205 is re-provisioned to the worker node 120A. In some cases, the worker node 120 may relinquish a set of key range grants 145 that includes one or more requested key grants 205, and thus the relinquish response 154 may include an indication of the relinquished key range grants 145. For example, the relinquish request 152 may specify key grant 205B, but the worker node 120A may decide to relinquish the entire key range grant 145A. The relinquish response 154 may also include history information 230 that provides an indication of which relinquished key grants 205 were used to commit records 132 while the worker node 120 was provided with these key grants 205.

[0038] In some cases, the worker node 120A may determine that there is an active transaction 210 holding a lock 215 on the requested key grant 205. For example, the relinquish request 152 may identify the key grant 205A obtained by the active transaction 210 in the example embodiment. In some embodiments, the worker node 120A waits for the associated active transaction 210 to commit or abort. The worker node 120A may then provide a release response 154 to the orchestrator node 150 for the requested key grant 205. If the worker node 120 attempts to relinquish a key range grant 145 but a lock 215 exists on an unrequested key grant 205 within the key range grant 145, the worker node 120 may hold the locked key grant 205 but relinquish the remaining portion of the key range grant 145. As an example, the worker node 120A may receive a relinquish request 152 identifying a key grant 205B. Worker node 120A may decide to return key range grant 145A. However, because key grant 205A (not the requested key grant) is locked by active transaction 210, worker node 120A may return all key grants 205 in key range grant 145A except for key grant 205A. Worker node 120A may return key grant 205A after there is no longer a lock 215 held on key grant 205A.

[0039] 3, a block diagram of an exemplary authorization orchestrator node 150 is shown. In the illustrated embodiment, the authorization orchestrator node 150 includes an authorization engine 300 having authorization information 140. In some embodiments, the authorization orchestrator node 150 is implemented differently than shown. For example, the authorization orchestrator node 150 may be implemented as a worker node 120 and thus may further include a database application 200 and an in-memory cache 130.

[0040] As shown, the authorization orchestrator node 150 receives an authorization request 112 issued by worker node 120B. The authorization request 112 may identify one or more key permissions 205 or key range permissions 145 that worker node 120B wishes to obtain. For example, the authorization request 112 from worker node 120B may identify key range permission 145A. In response to receiving the authorization request 112, the authorization engine 300 may process the authorization request 112 and return an authorization response 114 indicating whether the requestor received the requested key permissions 205.

[0041] The authorization engine 300, in various embodiments, is a set of software routines executable to facilitate provisioning and relinquishing of key permissions 205 between worker nodes 120. In response to receiving an authorization request 112, the authorization engine 300, in various embodiments, first determines whether the requested key permissions 205 have been provided to the worker node 120. As shown, the authorization engine 300 stores the authorization information 140, and thus the authorization engine 300 may refer to the authorization information 140 to determine whether the requested key permissions 205 have been provisioned. If these key permissions 205 have not been provisioned, the authorization engine 300 may generate updated authorization information 140 that allocates the requested key permissions 205 to the requesting worker node 120. The authorization engine 300 may then distribute the updated authorization information 140 to the worker node 120. In some embodiments, the authorization engine 300 issues an authorization information display 156 to the worker node 120 that includes the updated authorization information 140. Further, in some embodiments, the permission information indication 156 indicates that the permission information 140 has been updated. Thus, the worker node 120 may retrieve the updated permission information 140 from the permission orchestrator node 150 in response to receiving the permission information indication 156.

[0042] If the requested key grant 205 has been provided to the worker node 120, the authorization engine 300 may issue a relinquish request 152 to the worker node 120. For example, the authorization engine 300 may determine from a first version of the authorization information 140 (e.g., authorization information 140A) that the key range grant 145A has been provided to the worker node 120A (as shown). Thus, the authorization engine 300 may send a relinquish request 152 to the worker node 120A to relinquish the key range grant 145A as described above. In response to receiving a relinquish response 154 indicating that the requested key grant 205 has been relinquished, the authorization engine 300, in various embodiments, updates the first version of the authorization information 140 (e.g., authorization information 140B) allocated to the worker node 120 requesting the requested key grant 205 to a second version. As shown, the authorization engine 300 updates authorization information 140A, where worker node 120A is the owner of key range permission 145A, to authorization information 140B, where worker node 120B is the owner of key range permission 145A. The authorization engine 300 may then issue an authorization information indication 156 for the updated authorization information 140. In some embodiments, the authorization engine 300 issues an authorization response 114 to the requesting worker node 120 that includes the updated authorization information 140.

[0043] When updating the authorization information 140 to modify the ownership of the key grant 205, in various embodiments, the authorization engine 300 also updates the history information 230 included in the authorization information 140. As mentioned above, the relinquish response 154 may identify the XCN 225 for the key grant 205 or key range grant 145 used in the transaction at the relinquishing worker node 120. The XCN 225 provided in the key grant 205 (or key range grant 145) may represent the most recent time period in which records 132 were committed to the corresponding keys 134 across the system 100. Thus, the authorization engine 300 may update the history information 230 to include the XCN 225 identified in the received relinquish response 154.

[0044] 4, a block diagram of an exemplary layout for a worker node 120 obtaining key grants 205 is shown. In the illustrated embodiment, worker node 120B includes an in-memory cache 130, grant information 140, and a database application 200. As shown, database application 200 includes active transactions 210 associated with locks 215 and snapshots XCN 410. Also shown, grant information 140 includes key range grants 145A-145C and associated history information 230. In some embodiments, worker node 120B is implemented differently than shown. For example, database application 200 may handle multiple active transactions 210 and multiple committed transactions 220.

[0045] As described, the database application 200 may receive requests to execute transactions to read and write database records 132. Upon receiving the transaction request, the database application 200 may initiate a new active transaction 210 based on the received transaction request. As the database application 200 processes the active transaction 210, it may write various records 132 to the in-memory cache 130. When writing a particular record 132, the database application 200 may determine, based on the permission information 140, whether the worker node 120B has been provisioned with an appropriate permission 205. If the appropriate permission 205 is not provided to the worker node 120B, the worker node 120B may request the permission 205, as described above. Once the worker node 120B possesses the appropriate permission 205, the database application 200 may obtain a lock 215 for the permission 205 before writing the database record 132. In various cases, before writing record 132, database application 200 may ensure that another record 132 has not been committed after the time corresponding to snapshot XCN 410 that identifies the state of system 100 that active transaction 210 is permitted to see.

[0046] Snapshot XCN410 is, in various embodiments, a value indicating the latest XCN225 that can be read by an active transaction 210 corresponding to a database record 132. As shown, the active transaction 210 is assigned the snapshot XCN410 (T432). Thus, the active transaction 210 may read a committed database record 132 where the XCN225 is less than or equal to "T432". (Optionally, a database record 132 less than the snapshot XCN410 may be read.) For example, the active transaction 210 may read from the database 110 a database record 132 stamped with XCN225 (T230) (T230 < T432).

[0047] To ensure the integrity of data stored in system 100, in various embodiments, database application 200 will not write a database record 132 for a particular key 134 if another database record 132 for the same particular key 134 has been committed with an XCN 225 greater than the snapshot XCN 410 assigned to the corresponding active transaction 210. Thus, in various embodiments, database application 200 determines for an active transaction 210 whether a database record 132 for a particular key 134 has been committed with an XCN 225 greater than the snapshot XCN 410 (T432) based on historical information 230. For example, active transaction 210 may involve writing a database record 132 for key 134. Before writing that database record 132, database application 200 may check historical information 230 to determine the XCN 225 associated with key grant 205A. In some embodiments, historical information 230 identifies a respective XCN 225 for each key grant 205. If XCN 225 associated with key permission 205A is greater than snapshot XCN 410 (T432), database application 200 may refrain from writing database record 132 for key 134. However, if XCN 225 is less than or equal to snapshot XCN 410 (T432), database application 200 may write that database record 132.

[0048] In some embodiments, the history information 230 identifies XCN 225 for the entire key range grant 145. As shown, XCN 225 (T501) is associated with the key range grant 145A, which includes key grant 205A. In response to determining that XCN 225 (T501) for the key range grant 145A is greater than snapshot XCN 410 (T432), the database application 200 may determine whether the key grant 205A itself is associated with a smaller XCN 225. Although XCN 225 (T501) is associated with the entire key range grant 145A, it may have been added to the history information as a result of a commit of a database record 132 associated with a different key 134 (e.g., XYA) than key 134 XYZ. To determine XCN 225 for a particular key grant 205, in various embodiments, the worker node 120B sends an XCN request 420 to the worker node 120. In some embodiments, the history information 230 identifies the last worker node 120 associated with the key grant 205, and thus the worker node 120B may send the XCN request 420 only to the identified worker node 120. In some cases, the database application 200 may receive an XCN response 425 that may identify an XCN 225 for the key grant 205 identified in the XCN request 420. If the identified XCN 225 is greater than the snapshot XCN 410 (T432), the database application 200 may abort writing the database record 132 for the key 134 (XYZ). However, if the XCN 225 is less than or equal to the snapshot XCN 410 (T432), the database application 200 may write that database record 132.

[0049] 5, a flow diagram of a method 500 is shown. Method 500 is one embodiment of a method performed by a database system (e.g., system 100) that orchestrates the distribution of key range permissions (e.g., key range permissions 145) among multiple database nodes (e.g., worker nodes 120) of the database system. Method 500 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 500 includes more or fewer steps than are illustrated. For example, method 500 may include a step of the database system generating first permission information.

[0050] Method 500 begins at step 510 with a database system distributing first authorization information (e.g., authorization information 140A) to a plurality of database nodes. The first authorization information may identify a distribution of key range authorizations to each of the plurality of database nodes. Once a given key range authorization is distributed to a given database node, the given database node may be authorized to write records (e.g., database records 132) whose keys (e.g., keys 134) fall within a key range associated with the given key range authorization. In some cases, the first authorization information may provision a second database node with a second key range authorization that includes the first key range authorization.

[0051] In step 520, the database system receives a request (e.g., grant request 112) from a first database node (e.g., worker node 120B) for a first key range grant provisioned to a second database node (e.g., worker node 120A). In various embodiments, the database system sends a relinquishment request (e.g., relinquish request 152) to the second database node to relinquish the first key range grant. The second database node may relinquish the first key range grant in response to determining that the first key range grant is not being used in a set of active transactions (e.g., active transactions 210) executing at the second database node. The database system may receive an indication (e.g., relinquish response 154) from the second database node indicating that the first key range grant has been relinquished. In some cases, the second database node may relinquish the first key range grant and retain the remaining portion of the second key range grant. Optionally, a transaction commit number (eg, XCN225) may be specified that indicates the time interval when the most recent record was committed to the first key range grant.

[0052] In step 530, the database system modifies the first permission information to derive second permission information (e.g., permission information 140B) that provisions the first key range permission to the first database node instead of the second database node. In various embodiments, the second permission information is stored in a trie data structure that includes a plurality of branches, a particular one of the plurality of branches corresponding to the first key range permission.

[0053] In step 540, the database system distributes the second authorization information to the multiple database nodes. Distributing the second authorization information may include notifying the multiple database nodes about the second authorization information (e.g., via the authorization information display 156) and returning the second authorization information in response to receiving a request for the second authorization information from the multiple database nodes.

[0054] In various cases, the second permission information may identify a key range transaction commit number that indicates a first time interval when the most recent record was committed to the first key range permission. In various embodiments, the first database node determines whether a second time interval associated with the first database node (e.g., corresponding to snapshot XCN 410) occurs after the first time interval. In response to determining that the second time interval occurs after the first time interval, the first database node may write a database record for the particular key associated with the first key range permission. In response to determining that the second time interval does not occur after the first time interval, the first database node may retrieve a record transaction commit number for the particular key from a database node that committed a most recent record for the particular key. In response to determining that the record transaction commit number indicates a time interval that does not occur before the second time interval associated with the first database node, the first database node may prevent the record write for the particular key. The first database node may write the record for the particular key in response to determining that the record transaction commit number indicates a time interval that occurs before a second time interval associated with the first database node.

[0055] 6, a flow diagram of a method 600 is shown. Method 600 is one embodiment of a method performed by a database system (e.g., system 100) that orchestrates the distribution of key range permissions (e.g., key range permissions 145) among multiple database nodes (e.g., worker nodes 120) of the database system. Method 600 may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method 600 includes more or fewer steps than shown. For example, method 600 may include a step of the database system generating first permission information.

[0056] Method 600 begins at step 610 with an authorization orchestrator database node (e.g., authorization orchestration node 150) provisioning a first key range permission (e.g., key range permission 145) to a first worker database node (e.g., worker node 120A) of the database system. The first key range permission may allow a record (e.g., database record 132) whose key (e.g., key 134) falls within a first key range associated with the first key range permission to be written. At step 620, the authorization orchestrator database node receives an authorization request (e.g., authorization request 112) from a second worker database node (e.g., worker node 120B) of the database system for a second key range permission associated with a second key range encompassed in the first key range.

[0057] In step 630, the authorization orchestrator database node, in response to receiving the authorization request, causes the first worker database node to relinquish at least a portion of the first key range permission. The relinquishing may include sending a relinquishment request (e.g., relinquish request 152) to the first worker database node to relinquish a permission associated with the second key range. The first worker database node may prevent a transaction from using a key associated with the second key range in response to receiving the relinquishment request. In various cases, the first worker database node may determine that a record associated with a key that falls within the second key range has been written for an ongoing transaction (e.g., active transaction 210). The first worker database node may commit the ongoing transaction. After committing the ongoing transaction, the first worker database node may return an indication (e.g., relinquish response 154) to the authorization orchestrator database node that a portion of the first key range permission associated with the second key range has been relinquished.

[0058] In step 640, following the first worker database node relinquishing at least a portion of the first key range permission, the authorization orchestrator database node provisions a second key range permission to the second worker database node. In various embodiments, provisioning the second key range permission to the second worker database node includes the authorization orchestrator database node providing historical information (e.g., historical information 230) to the second worker database node indicating one or more writes performed by the first worker database node. The second worker database node determines, based on the historical information, whether the first worker database node committed a record having a particular key that falls within the second key range during a particular time interval. In response to determining that the first worker database node committed a record with the particular key during a particular time interval, the second worker database node may abort a portion of the transaction that involved writing a record with the particular key. Exemplary Multitenant Database System

[0059] Referring now to FIG. 7, an exemplary multi-tenant database system (MTS) 700 is shown in which various techniques of the present disclosure may be implemented. For example, the system 100 may be the MTS 700. In FIG. 7, the MTS 700 includes a database platform 710, an application platform 720, and a network interface 730 connected to a network 740. Also as shown, the database platform 710 includes a data store 712 and a set of database servers 714A-714N that interact with the data store 712, and the application platform 720 includes a set of application servers 722A-722N with respective environments 724. In the exemplary embodiment, the MTS 700 is connected to various user systems 750A-750N via the network 740. The disclosed multi-tenant system is included for illustrative purposes and is not intended to limit the scope of the present disclosure. In other embodiments, the techniques of the present disclosure are implemented in environments that are not multi-tenant environments, such as client / server environments, cloud computing environments, clustered computers, and the like.

[0060] The MTS 700, in various embodiments, is a set of computer systems that together provide various services to users (alternatively referred to as "tenants") that interact with the MTS 700. In some embodiments, the MTS 700 implements a customer relationship management (CRM) system that provides a mechanism for tenants (e.g., businesses, government agencies, etc.) to manage relationships and interactions with customers and potential customers. For example, the MTS 700 may enable tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), specify sales opportunities, record service issues, and manage marketing campaigns. Additionally, the MTS 700 may enable those tenants to identify how to communicate with customers, what customers have purchased, when the customer last purchased an item, and what the customer has paid for. As shown, to provide the services of the CRM system and / or other services, the MTS 700 includes a database platform 710 and an application platform 720.

[0061] The database platform 710, in various embodiments, is a combination of hardware elements and software routines that implement database services for storing and managing data for the MTS 700, including tenant data. As shown, the database platform 710 includes a data store 712, which in various embodiments includes a set of storage devices (e.g., solid state drives, hard disk drives, etc.) connected to each other over a network (e.g., a storage attached network (SAN)) and configured to store data redundantly to prevent data loss. In various embodiments, the data store 712 is used to implement a database (e.g., database 110) that includes a collection of information organized to allow access, storage, and manipulation of the information. The data store 712 may implement a single database, a distributed database, a collection of distributed databases, a database with redundant online or offline backups, or other redundancies. As part of a database implementation, data store 712 may store files (e.g., files 115) that contain one or more database records having their respective data payloads (e.g., values ​​for fields in a database table) and metadata (e.g., key values, timestamps, table identifiers for tables associated with the record, tenant identifiers for tenants associated with the record, etc.).

[0062] In various embodiments, a database record may correspond to a row in a table. A table typically includes one or more data categories logically arranged as columns or fields in a viewable schema. Thus, each record in a table may include an instance of data in each category defined by a field. For example, a database may include a table that describes customers with fields for basic contact information such as name, address, phone number, fax number, etc. Thus, a record in that table may include a value for each field in the table (e.g., name for a name field). Another table might describe purchase orders that include fields for information such as customer, product, sale price, date, etc. In various embodiments, standard entity tables are provided for use by all tenants, such as account, contact, lead, opportunity data, each including predefined fields. The MTS 700 may store database records for one or more tenants in the same table. That is, tenants may share tables. Thus, a database record includes a tenant identifier that indicates the owner of the database record in various embodiments. As a result, one tenant's data is kept secure and separate from other tenants' data, so that one tenant does not have access to another tenant's data unless such data is explicitly shared.

[0063] In some embodiments, data stored in the data store 712 is organized as part of a log-structured merge tree (LSM tree). An LSM tree typically includes two high-level components: an in-memory cache and persistent storage. In operation, the database server 714 may first write database records to a local in-memory cache before later flushing those records to persistent storage (e.g., the data store 712). As part of flushing the database records, the database server 714 may write the database records to a new file contained in the "top" level of the LSM tree. Over time, the database records may be rewritten by the database server 714 to new files contained in lower levels as the database records are moved down the levels of the LSM tree. In various implementations, as database records age and move down the LSM tree, they are moved to slower and slower storage devices (e.g., from solid state drives to hard disk drives) in the data store 712.

[0064] When the database server 714 wishes to access a database record for a particular key, the database server 714 may traverse different levels of the LSM tree for files that potentially contain a database record for that particular key. If the database server 714 determines that a file may contain an associated database record, the database server 714 may fetch the file from the data store 712 into the memory of the database server 714. The database server 714 may then check the fetched file for a database record with the particular key. In various embodiments, a database record is immutable once written to the data store 712. Thus, when the database server 714 wishes to modify a value in a row of a table (which may be identified from the accessed database record), the database server 714 writes out a new database record to the top level of the LSM tree. Over time, that database record is merged down the LSM tree. Thus, the LSM tree may store various database records for a database key, with older database records for that key located at lower levels of the LSM tree, followed by newer database records.

[0065] The database server 714, in various embodiments, may be a hardware element, software, or combination thereof, and may provide database services such as data storage, data retrieval, and / or data manipulation. The database server 714 may correspond to a worker node 120. Such database services may be provided by the database server 714 to components within the MTS 700 (e.g., application server 722) and storage elements external to the MTS 700. As an example, the database server 714 may receive a database transaction request from an application server 722 requesting to write to or read from the data storage device 712. The database transaction request may specify an SQL SELECT command to select one or more rows from one or more database tables. The contents of the row may be defined in a database record, and thus the database server 714 may identify and return one or more database records corresponding to the selected one or more table rows. In various cases, the database transaction request instructs the database server 714 to write one or more database records 714 to an LSM tree, which the database server 714 maintains implemented on the database platform 710. In some embodiments, database servers 714 implement a relational database management system (RDMS) or an object-oriented database management system (OODBMS) that facilitates storage and retrieval of information from data store 712. In various cases, database servers 714 may communicate with each other to facilitate processing of transactions. For example, database server 714A may communicate with database server 714N to determine whether database server 714N has written a database record for a particular key to its in-memory cache.

[0066] The application platform 720, in various embodiments, is a combination of hardware elements and software routines that implement and execute CRM software applications, provide relevant data, code, forms, web pages, and other information to user systems 750, and store relevant data, objects, web page content, and other tenant information via the database platform 710. To facilitate these services, in various embodiments, the application platform 720 communicates with the database platform 710 to store, access, and manipulate data. In some instances, the application platform 720 may communicate with the database platform 710 through different network connections. For example, one application server 722 may be coupled through a local area network and another application server 722 may be coupled through a direct network link. TCP / IP (Transfer Control Protocol and Internet Protocol) is an exemplary protocol for communicating between the application platform 770 and the database platform 710, although it will be apparent to one skilled in the art that other transport protocols may be used depending on the network interconnect used.

[0067] The application server 722 is a hardware element, software routine, or combination thereof capable of providing the services of the application platform 720, including processing requests received from tenants of the MTS 700, in various embodiments. The application server 722 can spawn environments 724 that can be used for various purposes, such as providing functionality to developers to develop, execute, and manage applications (e.g., business logic), in various embodiments. Data may be transferred to an environment 724 from another environment 724 and / or the database platform 710. In some cases, an environment 724 cannot access data from other environments 724 unless such data is explicitly shared. In some embodiments, multiple environments 724 may be associated with a single tenant.

[0068] Application platform 720 may provide user systems 750 access to multiple different hosted (standard and / or custom) applications, including CRM applications and / or applications developed by tenants. In various embodiments, application platform 720 may manage the creation of applications, the testing of applications, the storage of applications in database objects in data storage 712, the execution of applications in environments 724 (e.g., virtual machines in process space), or any combination thereof. In some embodiments, application platform 720 may add and remove application servers 722 from the server pool at any time for any reason, and there may be no server affinity for users and / or organizations to specific application servers 722. In some embodiments, an interface system (not shown) implementing load balancing functionality (e.g., F5 Big-IP load balancer) is located between application servers 722 and user systems 750 and configured to distribute requests to application servers 722. In some embodiments, the load balancer uses a least connections algorithm to route user requests to application servers 722. Other examples of load balancing algorithms may also be used, such as round robin, observed response time, etc. For example, in a particular embodiment, three consecutive requests from the same user may hit three different application servers 722, and three requests from different users may hit the same server 722.

[0069] In some embodiments, MTS 700 provides security mechanisms, such as encryption, to keep each tenant's data separate unless the data is shared. If multiple servers 714 or 722 are used, they may be located in close proximity to each other (e.g., in a server farm located in one building or campus), or they may be distributed in locations remote from each other (e.g., one or more servers located in City A and one or more servers located in City B). Thus, MTS 700 may include one or more logically and / or physically connected servers, either locally or distributed in one or more geographic locations.

[0070] One or more users (e.g., via user systems 750) may interact with MTS 700 over network 740. User systems 750 may correspond to, for example, a tenant of MTS 700, a provider (e.g., an administrator) of MTS 700, or a third party. Each user system 750 may be a desktop personal computer, a workstation, a laptop, a PDA, a mobile phone, or any wireless access protocol (WAP) enabled device, or any other computing system capable of interfacing directly or indirectly with the Internet or other network connection. User systems 750 may include dedicated hardware configured to interface with MTS 700 over network 740. The user systems 750 execute a graphical user interface (GUI) corresponding to the MTS 700, an HTTP client (e.g., a browser program such as Microsoft's Internet Explorer browser, Netscape's Navigator browser, Opera browser, or a WAP-enabled browser for a cell phone, PDA, or other wireless device), or both, to enable a user of the user system 750 (e.g., a subscriber to the CRM system) to access, process, and view information and pages available from the MTS 700 over the network 740. Each user system 750 also typically includes one or more user interface devices, such as a keyboard, mouse, touch screen, pen, etc., for interacting with the graphical user interface (GUI) provided by the browser on a display monitor screen, LCD display, etc., along with pages, forms, and other information provided by the MTS 700 or other systems or servers. As discussed above, the disclosed embodiments are suitable for use with the Internet, which refers to a particular global internetwork of networks.However, it should be understood that other networks, such as an intranet, an extranet, a virtual private network (VPN), a non-TCP / IP based network, any LAN or WAN, may be used in place of the Internet.

[0071] Users of user systems 750 may be users with different capacities, and the capacity of a particular user system 750 may be determined by one or more permission levels associated with the current user. For example, when a salesperson uses a particular user system 750 to interact with the MTS 700, the user system 750 may have a capacity (e.g., user privileges) allocated to the salesperson. However, when an administrator uses the same user system 750 to interact with the MTS 700, the user system 750 may have a capacity (e.g., administrative privileges) allocated to the administrator. In a system with a hierarchical role model, a user at one permission level may have access to applications, data, and database information accessible to a user at a lower permission level, but may not have access to certain applications, database information, and data accessible to a user at a higher permission level. Thus, different users may have different capabilities with respect to accessing and modifying application and database information depending on the user's security or permission level. There may also be some data structures managed by the MTS 700 that are allocated at the tenant level and other data structures managed at the user level.

[0072] In some embodiments, the user system 750 and its components are configurable using an application, such as a browser, that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 700 (and additional instances of the MTS, if there are multiple) and their components are operator configurable using applications that include computer code executable on the processing elements. Thus, various operations described herein may be performed by executing program instructions stored on a non-transitory computer-readable medium and executed by the processing elements. The program instructions may be stored on a non-volatile medium, such as a hard disk, or any other known volatile or non-volatile memory medium or device, such as a ROM or RAM, or provided on any medium capable of sterling program code, such as a compact disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, or the like. Additionally, the entire program code or portions thereof may be transmitted and downloaded from a software source, for example, via the Internet or from another server known in the art, or may be transmitted over any other known conventional network connection (e.g., extranet, VPN, LAN, etc.) using any known communication medium and protocol (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.) It will also be understood that computer code for implementing aspects of the disclosed embodiments may be implemented in any programming language that can be executed on a server or server system, such as, for example, C, C+, HTML, Java, JavaScript, or any other scripting language such as VBScript.

[0073] Network 740 may be a local area network (LAN), a wide area network (WAN), a wireless network, a point-to-point network, a star network, a token ring network, a hubbed network, or any other suitable configuration. The global internetwork of networks often referred to as the "Internet" with a capital "I" is one example of a Transfer Control Protocol and Internet Protocol (TCP / IP) network. However, it should be understood that the disclosed embodiments may utilize any of a variety of other types of networks.

[0074] The user system 750 communicates with the MTS 700 using TCP / IP and, at higher network levels, may communicate using other common Internet protocols such as HTTP, FTP, AFS, WAP, etc. For example, if HTTP is used, the user system 750 may include an HTTP client, commonly referred to as a "browser," to send and receive HTTP messages from an HTTP server in the MTS 700. Such a server may be implemented as the only network interface between the MTS 700 and the network 740, although other techniques may be used as well or instead. In some implementations, the interface between the MTS 700 and the network 740 includes a load sharing function, such as a round-robin HTTP request distributor, to balance the load and distribute incoming HTTP requests evenly across multiple servers.

[0075] In various embodiments, the user system 750 communicates with the application server 722 to request and update system-level and tenant-level data from the MTS 700, which may require one or more queries to the data storage 712. In some implementations, the MTS 700 automatically generates one or more SQL statements (SQL queries) designed to access the desired information. In some cases, the user system 750 may generate a request having a particular format that corresponds to at least a portion of the MTS 700. As an example, the user system 750 may request that a data object be moved to a particular environment 724 using an object notation that describes the object-relational mapping (e.g., JavaScript object notation mapping) of a particular number of objects. An Exemplary Computer System

[0076] 8, there is shown a block diagram of an exemplary computer system 800 upon which system 100, database 110, worker node 120, MTS 700, and / or user system 750 may be implemented. Computer system 800 includes a processor subsystem 880 coupled to a system memory 820 and an I / O interface 840 via an interconnect 860 (e.g., a system bus). I / O interface 1330 may be coupled to one or more I / O devices 850. Although a single computer system 800 is shown in FIG. 8 for convenience, system 800 may also be implemented as two or more computer systems operating together.

[0077] Processor subsystem 880 may include one or more processors or processing units. In various embodiments of computer system 800, multiple instances of processor subsystem 880 may be coupled to interconnect 860. In various embodiments, processor subsystem 880 (or each processor in processor subsystem 880) may include a cache or other form of on-board memory.

[0078] The system memory 820 is operable to store program instructions executable by the processor subsystem 880, which cause the system 800 to perform various operations described herein. The system memory 820 may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RDRAM RAM, etc.), read-only memory (PROM, EEPROM, etc.). The memory in the computer system 800 is not limited to primary storage devices such as the memory 820. Rather, the computer system 800 may include other forms of storage devices, such as cache memory in the processor subsystem 880, secondary storage devices on the I / O devices 850 (e.g., hard drives, storage arrays, etc.). In some embodiments, these other forms of storage devices may also store program instructions executable by the processor subsystem 880. In some embodiments, the program instructions that, when executed, implement the database application 200 may be contained / stored within the system memory 820.

[0079] I / O interface 840 may be any of a variety of types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I / O interface 840 is a bridge chip (e.g., a southbridge) from a front side to one or more back side buses. I / O interface 840 may be coupled to one or more I / O devices 850 via one or more corresponding buses or other interfaces. Examples of I / O interfaces include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or controllers associated therewith), network interface devices (e.g., to a local or wide area network), or other devices (e.g., graphics, sound, etc.). In one embodiment, computer system 800 is coupled to a network via network interface device 850 (e.g., configured to communicate via Wi-Fi, Bluetooth, Ethernet, etc.).

[0080] Implementations of the subject matter of the present application include, but are not limited to, the following Examples 1-20. Example 1. Distributing, by a database system, first authorization information to a plurality of database nodes of the database system, the first authorization information identifying a distribution of key range permissions to each of the plurality of database nodes, wherein a given key range permission distributed to a given database node authorizes the given database node to write records whose keys fall within a key range associated with the given key range permission; receiving, by the database system, from a first database node a request for a first key range grant provisioned to a second database node; modifying, by the database system, the first authorization information to derive second authorization information that provisions the first key range authorization to the first database node on behalf of the second database node; and distributing, by the database system, the second authorization information to each of the plurality of database nodes. Example 2. Before modifying the first permission information, the database system sending a request to a second database node to relinquish the first key range grant, the second database node being operable to relinquish the first key range grant in response to determining that the first key range grant is not being used in a set of active transactions being executed at the second database node; 2. The method of example 1, further comprising: receiving an indication from the second database node that the first key range grant has been relinquished. Example 3. The method of Example 2, wherein the first authorization information provisions a second database node with a second key range authorization that is a superset of the first key range authorization, and the second database node is operable to relinquish the first key range authorization but retain a remaining portion of the second key range authorization. Example 4. The method of example 2, wherein the indication specifies a transaction commit number associated with a most recently committed record for the first key range grant, and the second grant information identifies the transaction commit number to the first database node, and the first database node is operable to determine whether to write the particular record based on the transaction commit number. Example 5. The second grant information defines the key range transaction commit number associated with the most recently committed record for the first key range grant, and the first database node: determining whether a transaction commit number associated with the first database node is greater than the key range transaction commit number; 2. The method of claim 1, operable to: in response to determining that the transaction commit number is greater than the key range transaction commit number, write a record for the particular key associated with the first key range permission. Example 6. The second grant information defines the key range transaction commit number associated with the most recently committed record for the first key range grant, and the first database node: determining whether a transaction commit number associated with the first database node is greater than the key range transaction commit number; In response to determining that the transaction commit number is less than or equal to the key range transaction commit number, the method of example 1 is operable to: retrieve a record transaction commit number for a particular key associated with the first key range permission from the database node that committed the most recently committed record. Example 7. The first database node is 7. The method of example 6, operable to prevent a record write for a particular key in response to determining that a transaction commit number associated with the first database node is less than or equal to the record transaction commit number. Example 8. The first database node is 7. The method of example 6, operable to write the record for the particular key in response to determining that a transaction commit number associated with the first database node is greater than the record transaction commit number. Example 9. The method of example 1, wherein the second permission information is stored in a trie data structure that includes a plurality of branches, a particular branch of the plurality of branches corresponding to the first key range permission. Example 10. Distributing second authorization information to each of a plurality of database nodes Communicating the second permission information to a plurality of database nodes; receiving an information request for the second authorization information from each of the plurality of database nodes; and returning a trie data structure in response to the information request. Example 11. A non-transitory computer readable medium having program instructions stored thereon executable by a database system, the program instructions providing to the database system: distributing first authorization information to a plurality of database nodes of a database system, the first authorization information identifying a distribution of key range permissions to each of the plurality of database nodes, wherein a given key range permission distributed to a given database node authorizes the given database node to write records whose keys fall within a key range associated with the given key range permission; receiving a grant request from a first database node for a first key range grant provisioned to a second database node; modifying the first authorization information to derive second authorization information that provisions the first key range authorization to the first database node instead of the second database node; Distributing the second authorization information to each of a plurality of database nodes. Example 12. The operation is sending a request to a second database node to relinquish at least the first key range grant; 12. The medium of example 11, further comprising: receiving an indication from the second database node indicating that the first key range permission and the second key range permission have been relinquished prior to modifying the first permission information. Example 13. The medium of example 12, wherein modifying the first authorization information is performed such that a superset key range authorization is provided to the first database node, the superset key range authorization including the first key range authorization and a second key range authorization that was not requested by the first database node in the authorization request. Example 14. The medium of example 12, wherein modifying the first authorization information is performed such that the second authorization information includes an indication of whether another database node committed a record having a key that falls within a key range associated with the first key range authorization during a particular time frame. Example 15. The medium of example 11, wherein modifying the first authorization information is performed such that the second authorization information includes an indication of whether another database node committed a record having a key that falls within a key range associated with the first key range authorization during a particular time frame. Example 16. A method comprising: provisioning, by an authorization orchestrator database node of the database system, a first key range authorization to a first worker database node of the database system, the first key range authorization authorizing a record to be written whose key falls in a first key range associated with the first key range authorization; receiving, by the database system authorization orchestrator database node, from a second worker database node, an authorization request for a second key range authorization associated with a second key range encompassed by the first key range; in response to receiving the authorization request, the authorization orchestrator database node causes a first worker database node to relinquish at least a portion of the first key range authorization; and following the first worker database node relinquishing at least a portion of the first key range grant, the authorization orchestrator database node provisions a second key range grant to the second worker database node. Example 17. The relinquishing includes sending a relinquishment request to a first worker database node to relinquish permissions associated with the second key range, the method comprising: determining, by a first worker database node, that a record associated with a key that falls within the second key range has been written for the ongoing transaction; committing, by a first worker database node, the ongoing transaction; and 18. The method of example 17, further comprising: after committing the ongoing transaction, the first worker database node returning an indication to the authorization orchestrator database node that a portion of the first key range authorization associated with the second key range has been relinquished. Example 18. The method of Example 17, further including preventing, by the first worker database node, the transaction from using keys associated with the second key range in response to receiving the abandon request. Example 19. Provisioning the second key range permission to the second worker database node includes providing to the second worker database node historical information indicative of one or more writes performed by the first worker database node, and the method further comprises: 17. The method of claim 16, further comprising determining, by the second worker database node, based on the historical information, whether the first worker database node committed a record having a particular key that falls within the second key range during a particular time interval. Example 20. The method of example 19, wherein in response to a first worker database node determining that it committed a record with a particular key during a particular time interval, a second worker database node aborts a portion of a transaction that involves writing a record with the particular key.

[0081] The present disclosure includes references to "embodiments" that are non-limiting implementations of the disclosed concepts. References to "embodiments," "one embodiment," "particular embodiments," "some embodiments," "various embodiments," etc. do not necessarily refer to the same embodiment. Numerous possible embodiments are contemplated, including the specific embodiments described in detail, as well as modifications or alternatives that fall within the spirit or scope of the present disclosure. Not all embodiments necessarily demonstrate any or all of the potential advantages described herein.

[0082] Unless otherwise specified, a particular embodiment is not intended to limit the scope of the claims drafted based on this disclosure to the disclosed form, even if only a single example of a particular feature is described. Thus, the disclosed embodiments are intended to be illustrative rather than limiting, and nothing to the contrary is stated. This application is intended to cover alternatives, modifications, and equivalents as will be apparent to one skilled in the art having the benefit of this disclosure.

[0083] Particular features, structures, or characteristics may be combined in any suitable manner not inconsistent with the present disclosure. Thus, the present disclosure is intended to include any feature or combination of features disclosed herein (explicitly or implicitly), or any generalization thereof. Accordingly, new claims may be formulated during prosecution of this application (or any application claiming priority thereto) for such combinations of features. In particular, with reference to the appended claims, features from the dependent claims may be combined with features of the independent claims, and features from each independent claim may be combined in any suitable manner and not just in the specific combinations recited in the appended claims.

[0084] For example, although the accompanying dependent claims are drafted to each depend on a single other claim, additional dependencies are also contemplated, including that claim 6 may depend on any of the preceding claims, claim 9 may depend on any of the preceding claims, claim 15 may depend on any of claims 11-14, and claim 19 may depend on any of claims 16-18. Where appropriate, claims drafted in one statutory type (e.g., apparatus) are also contemplated to suggest corresponding claims of other statutory types (e.g., methods).

[0085] Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. It is hereby announced that the definitions provided in the following paragraphs, and throughout this disclosure, shall be used in determining how to interpret any claims drafted based on this disclosure.

[0086] References to the singular, such as "a," "an," "the," etc., are intended to mean "one or more" unless the context clearly dictates otherwise. Thus, a reference to an "item" in a claim does not exclude additional instances of that item.

[0087] In this specification, the word "may" is used in its permissive sense (ie, having the possibility, being possible) and not its mandatory sense (ie, requiring).

[0088] "Comprising" and "including" and their variants are open ended and mean "including, but not limited to."

[0089] When the term "or" is used in this disclosure in reference to a list of options, it will generally be understood to be used in an inclusive sense unless the context provides otherwise. Thus, a statement of "x or y" is equivalent to "x or y, or both," covering x but not y, y but not x, and both x and y. On the other hand, the phrase "either x or y, but not both" makes it clear that "or" is used in an exclusive sense.

[0090] The provisions of "w, x, y, or z, or any combination thereof," or "at least one of w, x, y, and z" are intended to cover all possibilities involving a single element to the total number of elements in the set. For example, given a set [w, x, y, z], these phrases cover any single element of the set (e.g., w but not w, x, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Thus, the phrase "at least one of w, x, y, and z" refers to at least one of the elements of the set [w, x, y, z], thereby covering all possible combinations in this list of options. This phrase should not be interpreted as requiring that there be at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0091] Various "labels" may proceed nouns in this disclosure. Unless the context provides otherwise, different labels used for a feature (e.g., "first circuit," "second circuit," "particular circuit," "given circuit," etc.) refer to different instances of the feature. The labels "first," "second," and "third," when applied to particular features, do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless specifically stated otherwise.

[0092] In this disclosure, different entities (which may be variously referred to as "units," "circuits," other components, etc.) may be described or claimed as being "configured" to perform one or more tasks or operations. This formulation, i.e., an "entity" configured to perform one or more tasks, is used herein to refer to a structure (i.e., something physical). More specifically, this formulation is used to indicate that this structure is arranged to perform one or more tasks during operation. A structure can be said to be "configured" to perform some tasks even if the structure is not currently operating. Thus, an entity described or defined as being "configured" to perform some tasks refers to some physical thing, such as a device, a circuit, a memory that stores executable program instructions to perform a task. This phrase is not used herein to refer to something intangible.

[0093] The term "configured to" is not intended to mean "configurable to." For example, an unprogrammed FPGA would not be considered to be "configured" to perform any function. However, the unprogrammed FPGA may be "configurable to" perform that function.

[0094] As used herein, the phrase "based on" is used to describe one or more factors that influence the determination. This term does not exclude the possibility that additional factors may influence the determination. That is, the determination may be based only on the specified factors or on the specified factors and other unspecified factors. Consider the phrase "determine A based on B." This phrase specifies that B is a factor used to determine A or that influences the determination of A. This phrase does not exclude that the determination of A may also be based on other factors such as C. This phrase is also intended to cover embodiments in which A is determined based only on B. As used herein, the phrase "based on" is synonymous with the phrase "based at least in part on."

[0095] The phrase "in response to" lists one or more factors that trigger the effect. This phrase does not exclude the possibility that additional factors may affect or otherwise trigger the effect. That is, the effect may be responsive to only those factors or to the specified factors and other unspecified factors. Consider the phrase "performing A in response to B." This phrase specifies that B is a factor that triggers performance of A. This phrase does not exclude that performing A may also be responsive to some other factor, such as C. This phrase is also intended to cover embodiments in which A is performed in response to only B.

[0096] In this disclosure, various "modules" operable to perform specified functions are shown in the figures and described in detail above. As used herein, a "module" refers to software or hardware operable to perform a particular set of operations. A module may refer to a set of software instructions executable by a computer system to perform a set of operations. A module may also refer to hardware configured to perform a set of operations. A hardware module may comprise general-purpose hardware, as well as non-transitory computer-readable media that store program instructions, or specialized hardware such as customized ASICs. Thus, a module described as being "executable" to perform an operation refers to a software module, and a module described as being "configured" to perform an operation refers to a hardware module. A module described as being "operable" to perform an operation refers to a software module, a hardware module, or some combination thereof. Furthermore, for any description herein referring to a module being "executable" to perform a particular operation, it should be understood that these operations may, in other embodiments, be implemented by a hardware module that is "configured" to perform the operation, and vice versa.

Claims

1. A method performed by an authorization orchestrator node of a database system, comprising: distributing first permission information to a plurality of database nodes of the database system, the first permission information identifying a distribution of key range permissions to each of the plurality of database nodes, wherein a given key range permission distributed to a given database node authorizes the given database node to write records whose keys fall within a key range associated with the given key range permission to an in-memory cache of the given database node; receiving a request from a first database node for a first key range grant provisioned to a second database node; modifying the first authorization information to derive second authorization information that provisions the first key range authorization to the first database node instead of the second database node; and distributing the second authorization information to each of the plurality of database nodes.

2. Before modifying the first authorization information, sending a request to the second database node to relinquish the first key range grant, the second database node being operable to relinquish the first key range grant in response to determining that the first key range grant is not being used in a set of active transactions being executed at the second database node; 2. The method of claim 1, further comprising: receiving an indication from the second database node that the first key range grant has been relinquished.

3. 3. The method of claim 2, wherein the first authorization information provisions the second database node with second key range authorizations that are a superset of the first key range authorizations, and the second database node is operable to relinquish the first key range authorizations but retain a remaining portion of the second key range authorizations.

4. 4. The method of claim 2 or 3, wherein the indication specifies a transaction commit number associated with a most recent committed record for the first key range grant, and the second grant information identifies the transaction commit number to the first database node, and the first database node is operable to determine whether to write a particular record to an in-memory cache of the first database node based on the transaction commit number.

5. The second grant information defines a key range transaction commit number associated with a most recently committed record for the first key range grant, and the first database node: determining whether a transaction commit number associated with the first database node is greater than the key range transaction commit number; 2. The method of claim 1, operable to: in response to determining that the transaction commit number is greater than the key range transaction commit number, write a record for the particular key associated with the first key range permission to an in-memory cache of the first database node.

6. The second grant information defines a key range transaction commit number associated with a most recently committed record for the first key range grant, and the first database node: determining whether a transaction commit number associated with the first database node is greater than the key range transaction commit number; 2. The method of claim 1, operable to: in response to determining that the transaction commit number is less than or equal to the key range transaction commit number, retrieve a record transaction commit number for a particular key associated with the first key range permission from a database node that committed the most recently committed record.

7. The first database node comprises:

7. The method of claim 6, further comprising: in response to determining that the transaction commit number associated with the first database node is less than or equal to a record transaction commit number, operable to prevent writing of a record for the particular key to an in-memory cache of the first database node.

8. The first database node comprises:

7. The method of claim 6, further comprising: responsive to determining that the transaction commit number associated with the first database node is greater than a record transaction commit number, operable to write the record for the particular key to an in-memory cache of the first database node.

9. 9. The method of claim 1, wherein the second permission information is stored in a trie data structure including a plurality of branches, a particular branch of the plurality of branches corresponding to the first key range permission.

10. Distributing the second authorization information to each of the plurality of database nodes includes: notifying the plurality of database nodes of the second permission information; receiving an information request for the second authorization information from each of the plurality of database nodes; and returning a trie data structure in response to the information request.

11. A non-transitory computer readable storage medium having a program stored thereon executable by an authorization orchestrator node of a database system, the program causing the authorization orchestrator node of the database system to perform a method according to any one of claims 1 to 10.

12. An authorization orchestrator node of a database system, comprising: At least one processor; A memory storing a program executable by said at least one processor to perform the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Storage device management method

    JP2008217797A

  • System and method for bulk removal of records in a database

    US20200097558A1

  • System and method for bulk removal of records in a database

    WO2020068856A1