Cloud storage for databases using object storage

Abstract

Techniques are disclosed that relate to managing distributed storage of data with low latency access and durable storage. The distributed storage system may include a storage cache distributed across availability zones in front of a shared object storage. Database nodes collocated with the storage cache may handle storing, retrieving, and manipulating data in the storage cache and the shared object storage. The shared object storage cache provides persistent storage of data for the distributed storage system, and storing of data in the storage cache is implemented to provide low latency retrieval of the storage. The storage cache across availability zones may be populated via the shared object storage rather than moving data across availability zones.

Description

[Technical field]

[0001] The present disclosure relates generally to storage systems, and more specifically to replicating and retrieving data using cache clusters and object storage. [Background technology]

[0002] Modern database systems routinely implement management systems that allow users to store collections of information in an organized manner that can be efficiently accessed and manipulated. These database systems often include database nodes and storage nodes that cooperate to perform database services. Database nodes often process database transactions to read and manipulate data, and storage nodes act to ensure that the results of those transactions are stored in an efficiently accessible manner. Management systems also often attempt to ensure that the data in a database system is sufficiently replicated across various zones to prevent data loss in the event that a portion of the database system fails or becomes unavailable. [Brief description of the drawings]

[0003] [Figure 1A] FIG. 1 is a block diagram illustrating example elements of a cloud environment, according to some embodiments. [Figure 1B] FIG. 1 illustrates a block diagram showing an example of a storage cache, according to some embodiments. [Diagram 2] FIG. 1 is a block diagram illustrating example elements of a cloud environment including a cloud-based service configured to process write requests to the cloud-based service, according to some embodiments. [Diagram 3] FIG. 1 illustrates a block diagram illustrating example elements of a cloud environment including a cloud-based service configured to process write and read requests to the cloud-based service, according to some embodiments. [Figure 4] FIG. 1 illustrates a block diagram illustrating example elements of a cloud environment including a cloud-based service having three availability zones configured to process write and read requests to the cloud-based service, according to some embodiments. [Diagram 5] FIG. 1 illustrates a block diagram illustrating example elements of a cloud environment including a cloud-based service configured to process write and read requests to the cloud-based service along with writing to and reading from a log of write and read requests, according to some embodiments. [Figure 6] FIG. 1 illustrates a block diagram showing example elements of a cloud environment, including cloud-based services and a cluster manager, in accordance with some embodiments. [Figure 7] FIG. 1 illustrates a block diagram illustrating example elements of a cloud environment including a cloud-based service configured to store write requests directly in shared object storage for later injecting into local storage in the cloud-based service, in accordance with some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating example elements of a cloud environment including a cloud-based service configured to process write and read requests from various entities based on metadata associated with the various entities, according to some embodiments. [Figure 9] FIG. 2 is a block diagram illustrating an example database node configured to identify a cache for storing data related to a first entity, according to some embodiments. [Figure 10A] 4 is a block diagram illustrating an example allocation of cache space based on entity classification according to some embodiments. [Figure 10B] FIG. 1 is a block diagram illustrating an example system crash in a given availability zone, according to some embodiments. [Figure 11]FIG. 1 is a flow diagram illustrating an example method for processing a write request using a class-based technique, according to some embodiments. [Figure 12] 1 is a block diagram illustrating an example method for processing read requests for data stored in one or more caches using class-based techniques, according to some embodiments. [Figure 13] FIG. 1 is a flow diagram illustrating an example method for processing a write request in a cloud-based service, according to some embodiments. [Figure 14] FIG. 1 is a flow diagram illustrating another exemplary method for processing a write request in a cloud-based service, according to some embodiments. [Figure 15] 1 illustrates an example multi-tenant database system (MTS) in which various techniques of this disclosure may be implemented. [Figure 16] FIG. 1 illustrates a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Cloud computing services (or simply "cloud") are becoming increasingly popular as enterprises seek to migrate their infrastructure to the cloud. As used herein, the term "cloud" is used according to its well-understood meaning to refer to the on-demand availability of computer resources, such as data storage and computing power, available to one or more organizations via the Internet. One example of a public cloud is Amazon Web Services® (AWS), which is offered by Amazon® (the provider) to multiple enterprises (organizations) to host and run software. Public cloud resources (e.g., computing hardware, storage hardware, etc.) are often distributed across multiple geographic areas (called "cloud regions"), with each cloud region having multiple isolated networks of those resources (called "availability zones"). An enterprise may instantiate its software on computer systems located within a particular zone of a geographically appropriate cloud region.

[0005] As described herein, a distributed database storage system may store data in a data storage cluster that includes storage servers distributed across multiple availability zones. Data stored in the storage servers in each availability zone may be replicated across servers in additional availability zones for persistent data storage redundancy within the data storage cluster. Often, these storage servers are not natively designed for public clouds; instead, they are designed for a single centralized network, so there is no concept of cloud regions or availability zones. Thus, a distributed database storage system that is distributed across multiple availability zones may not properly account for availability zone failures, along with the latency and cost associated with communicating across zones in the cloud. For example, if an availability zone fails, data may be replicated across availability zones in response to the failure. However, moving data across availability zones may degrade performance, as latency increases when moving data across availability zones. This degradation in performance may be problematic for transactional databases that require high performance (low latency) for fast access to data. Additionally, the cost of moving data across availability zones may be expensive. The present disclosure addresses at least this technical problem of replicating data without the latency issues and costs associated with moving data across availability zones.

[0006] This disclosure describes various techniques for enabling cloud-based services to move data across availability zones by integrating the use of ephemeral storage caches (such as internal non-volatile memory (NVMe)-based storage) and shared object storage (such as Amazon S3®). Shared object storage provides cost-effective, durable, and scalable storage for persistent data. However, shared object storage does not have the performance (such as low latency for fast access to data) required for transactional databases. Ephemeral storage caches provide low-latency data transfer, but are temporary storage where data is lost during power outages or other failure events. Thus, this disclosure contemplates techniques for providing low-cost, low-latency, and durable persistent storage for distributed database storage systems, such as systems that provide transactional database storage.

[0007] In various embodiments described below, the cloud-based storage service has multiple storage caches instantiated across multiple availability zones in front of a shared object storage. The storage caches may be instantiated in a data storage cluster with storage caches distributed across multiple availability zones for data replication and availability across availability zones. Each availability zone can write data to and read data from the shared object storage separately. The shared object storage behind the data storage cluster provides persistent and durable replicated storage for the data in the data storage cluster. In various cases, an orchestration service or cluster manager such as Kubernetes® or Apache Zookeeper® is used to instantiate the storage caches in different availability zones. In various embodiments, the availability zones include database nodes, which are described further herein. The database nodes handle the retrieval, manipulation, and storage of data for the data storage cluster. For example, the database nodes may write data to both the storage cache and the shared object storage in response to a request to store data in the cloud-based storage service. The database nodes may retrieve data from the storage cache in response to a data retrieval request, providing fast data retrieval by taking advantage of the low latency of the storage cache. The shared object storage provides a backup for the data in case the storage cache fails.

[0008] These techniques may be advantageous because they allow low latency data retrieval while providing persistent data storage to prevent data loss. Additionally, these techniques provide low latency and low cost movement of data within a cloud environment. For example, data lost in a storage cache may be retrieved directly from the shared object storage rather than from another storage cache, since the cost of retrieving data from a shared object storage is typically lower than the cost of moving data across availability zones. Additionally, the shared object storage may be used to populate data across availability zones rather than populating data from one availability zone to another. For example, replacement data for older or invalid data in one availability zone may be retrieved from the shared object storage after identification of newer or valid data in another availability zone, the newer or valid data also being stored in the shared object storage. An exemplary application of these techniques will now be described with reference to FIG. 1A.

[0009] 1A, a block diagram of exemplary elements of a cloud environment 100 is shown. Cloud environment 100 includes a set of components that may be implemented via hardware or a combination of hardware and software. In the illustrated embodiment, cloud environment 100 includes a cloud-based service 105 that includes availability zones 110A and 110B. Cloud environment 100 is a cloud infrastructure that includes various components (e.g., hardware, virtualized resources, storage, and network resources) for providing cloud computing services to users in various embodiments. In some embodiments, cloud environment 100 is distributed across multiple geographic locations, and each location may define a "region" of cloud environment 100. A region may include a set of data centers that may be clustered together. Within a region, there may be one or more availability zones 110.

[0010] As used herein, the term "availability zones" refers to two or more cloud areas that are geographically separated and across which data is replicated. Availability zones 110 may be located within the same region or may be in separate regions. Availability zones 110 are, in various embodiments, a logical or physical grouping of components (e.g., computing resources, storage resources, etc.) within a region. Often, components of a given availability zone 110 are isolated from failures of components of other availability zones 110. For example, availability zone 110A may be a first datacenter in a particular region and availability zone 110B may be a second datacenter in the same region. Availability zone 110A may be separated such that a failure at a datacenter of availability zone 110B does not impact the datacenter of availability zone 110A. In some cases, availability zones 110A and 110B may be the same datacenter but may accommodate components on separate networks such that one availability zone 110 is not affected by the other availability zone 110.

[0011] In various embodiments, a data storage cluster (e.g., a cache cluster or storage cluster described herein) is distributed across two or more availability zones 110, and data is replicated across availability zones. Data may be replicated across availability zones to provide data redundancy between availability zones. In certain embodiments, availability zones 110 are geographically separated by a distance such that an external failure (e.g., a power outage or natural disaster) in one availability zone does not impact another availability zone, and stored data is still available for retrieval. In the context of cloud-based services 105, availability zones 110 may be referred to as "computer zones." While the illustrated embodiment of FIG. 1A and various embodiments described herein show two availability zones 110A and 110B, it should be understood that cloud-based services 105 may include any number of availability zones. For example, in many contemplated embodiments, cloud-based services 105 include three availability zones 110, as shown in FIG. 4.

[0012] In a particular embodiment, database node 120, storage cache 130, and log storage 140 are distributed across availability zones 110. In the illustrated embodiment, availability zone 110A includes database node 120A, and availability zone 110B includes database node 120B. As used herein, the term "database node" refers to a component that handles the storage, retrieval, and manipulation of data in cloud environment 100. A database node may be, for example, a component implemented by an application running on a server in cloud environment 100. For example, database node 120 may be a software routine executable by an application running on a server colocated in availability zone 110 with storage cache 130 and log storage 140. Thus, database node 120 may interface with storage cache 130, log storage 140, and shared object storage 150 to handle the storage, retrieval, and manipulation of data in cloud environment 100. For example, a database node may service requests to storage cache 130, log storage 140, or shared object storage 150 for reading and writing data on behalf of cloud-based service 105.

[0013] In various embodiments, database nodes 120 are instantiated in cloud environment 100 by cluster manager 160. In particular embodiments, at least one database node 120 is instantiated in each availability zone 110 such that an instance of the database node exists in each availability zone. One or more database nodes 120 in availability zone 110 may implement a database (e.g., a distributed database storage system) that may be used by other applications to store and query data. Distributing database nodes 120 across each availability zone 110 provides redundancy of database nodes between availability zones. Thus, if a database node in one availability zone fails, a database node in another availability zone may be available to replace the failed database node. For example, database node 120A in availability zone 110A may be able to interface with storage cache 130B or log storage 140B in availability zone 110B if database node 120B fails.

[0014] In some embodiments, database node 120 is instantiated on multiple servers within an availability zone 110 to provide redundancy within the availability zone. In such embodiments, a primary instance of the database node is running on one server within the availability zone, and a replica of the database node is instantiated on an additional server as a backup. Thus, if the primary instance of the database node fails, a backup (secondary) instance of the database node may be activated to replace the failed primary instance.

[0015] In some embodiments, only one database node 120 is instantiated in cloud environment 100 such that only one instance of the database node exists across availability zones 110. In various embodiments, only one instance of database node 120 across availability zones 110 is active as a primary instance, with inactive (replica) database nodes residing in other availability zones (or the same availability zone). Thus, if the primary instance of the database node fails, an inactive (replica) database node in any availability zone (e.g., another availability zone or the same availability zone) may be activated to provide redundancy for the operation of the database node.

[0016] In the illustrated embodiment, availability zone 110A includes storage cache 130A, and availability zone 110B includes storage cache 130B. FIG. 1B illustrates a block diagram illustrating an example of storage cache 130, according to some embodiments. As used herein, the term "storage cache" refers to an ephemeral cache for storing data. For example, storage cache 130 may include internal non-volatile memory (e.g., NVMe) based storage. In various embodiments, storage cache 130 includes cache nodes 180 (e.g., storage nodes) and storage devices 190. In certain embodiments, cache nodes 180 are sets of software routines executable to implement storage cache 130 and may provide storage functionality in cloud-based service 105. In some embodiments, hardware used to execute the sets of software routines is considered part of cache node 180. Cache node 180 may receive requests to read and write data on behalf of cloud-based service 105, and in various instances, specifically on behalf of its storage cache 130. As an example, cache node 180 may receive a request to write or read data stored in storage device 190 .

[0017] In certain embodiments, storage device 190 is a data storage element where data is maintained in storage cache 130. In various embodiments, storage device 190 is an ephemeral cache storage element, such as an internal non-volatile memory (e.g., NVMe) based storage. Other embodiments of cache storage elements, such as, but not limited to, volatile memory based storage, may also be contemplated. In certain embodiments, storage device 190 includes multiple servers (e.g., server blades with internal non-volatile memory) collocated with database nodes 120 in availability zone 110, which collectively and in combination with cache nodes 180 provide storage cache 130. In some embodiments, storage cache 130 is instantiated by database nodes 120. In various embodiments, storage cache 130 provides low latency data transfers, but is typically susceptible to losing data upon power loss or failure.

[0018] In various embodiments, storage cache 130 is implemented as a cache for data stored in shared object storage 150. Storage cache 130 may receive requests to read and write data on behalf of cloud-based services 105, and in various cases, specifically on behalf of their availability zones 110. As an example, storage cache 130A may receive requests from database node 120A to return data stored on the servers that make up storage cache 130A. In various embodiments, storage cache 130, along with database node 120, practices data replication such that data stored by one storage cache 130 is stored by at least one other storage cache 130. For example, data stored by storage cache 130A may be stored by storage cache 130B. In some cases, portions of the data may be stored in different storage caches 130 such that a complete copy is not kept by a single storage cache 130. In various embodiments, the stored data is replicated across different availability zones 110 such that if one availability zone 110 becomes unavailable, the data is still accessible through another availability zone 110, or such that if a particular portion of the cloud-based service 105 on an availability zone 110 becomes unavailable, the data is still accessible through another portion of the cloud-based service 105.

[0019] In the illustrated embodiment, availability zone 110A includes log storage 140A, and availability zone 110B includes log storage 140B. As used herein, the term “log storage” refers to a storage element used to store a log of transactions (e.g., storage and retrieval) over time, including data managed by cloud-based service 105 (e.g., data stored in and retrieved from storage cache 130). In some embodiments, log storage 140 is instantiated by database node 120. As shown in the illustrated embodiment, log storage 140A corresponds to availability zone 110A and storage cache 130A, and log storage 140B corresponds to availability zone 110B and storage cache 130B. Thus, in a particular embodiment, log storage 140A stores logs of transactions associated with availability zone 110A and storage cache 130A, and log storage 140B stores logs of transactions associated with availability zone 110B and storage cache 130B.

[0020] In some embodiments, log storage 140 may store a copy of data managed in cloud-based service 105. For example, log storage 140 may store a copy of data along with a log record. In some embodiments, the copy of data stored in log storage 140 may be used to recover data in the event of a problem with one of the other storage elements (e.g., storage cache 130 or shared object storage 150). In certain embodiments, log storage 140 includes a persistent data storage element where logs and data are retained in the event of a power loss or failure. For example, log storage 140 may include a block storage element, such as an Elastic Block Storage (EBS) element. In various embodiments, log storage 140 implements a set of software routines executable to provide storage functionality in cloud-based service 105 that corresponds to the functionality implemented in storage cache 130. For example, log storage 140 may practice data replication such that data stored by one log storage 140 is stored by at least one other log storage 140.

[0021] As shown in FIG. 1A, the cloud environment 100 includes a shared object storage 150. As used herein, the term "shared object storage" refers to persistent data storage for objects that may be shared across resources. One example of a shared object storage is Amazon S3® from Amazon Web Services® (AWS). In certain embodiments, the shared object storage 150 implements a web service interface that allows for the storage and retrieval of any amount of data with high bandwidth. Data may be stored and retrieved at any time from anywhere on the web. In some instances, the shared object storage 150 incurs costs only for the transaction of data and not for the amount of data transferred. Thus, any amount of data may be stored or retrieved in a single transaction based on the same cost. In various embodiments, the cloud-based service 105 interfaces with the shared object storage 150 for the storage and retrieval of data managed by the cloud-based service. For example, as described herein, database nodes 120 in availability zone 110 may interface with shared object storage 150 to provide data to or retrieve data from the shared object storage for storage. In particular embodiments, shared object storage 150 is implemented by database nodes 120 as the main data storage, and storage cache 130 is implemented as local, low-latency data storage for a subset of the data stored in shared object storage 150.

[0022] In the illustrated embodiment, the cloud environment 100 includes a cluster manager 160 having a metadata store 170. As used herein, the term “cluster manager” refers to a centralized management or orchestration service that facilitates deployment of cloud-based services 105 using resources of availability zones 110 and shared object storage 150. In various embodiments, the cluster manager 160 includes a set of executable software routines to provide management or orchestration services. Kubernetes® and Apache Zookeeper® are examples of cluster managers. The cluster manager 160 may deploy containerized applications that implement the cloud-based services 105. In some embodiments, the cluster manager 160 is associated with a region and is therefore responsible for facilitating deployment within the availability zones 110 of the region. As shown in FIG. 1A , for example, cluster manager 160 interfaces with both availability zone 110A and availability zone 110B, and thus can facilitate deployment of database nodes 120, storage cache 130, and log storage 140 within those availability zones 110. Although cluster manager 160 is shown separate from availability zones 110, in various embodiments, cluster manager 160 is instantiated within availability zone 110 (e.g., availability zone 110A). Although cluster manager 160 may be instantiated in one of availability zones 110, cluster manager 160 may still facilitate deployment within the other availability zone 110 (e.g., availability zone 110B). However, in some embodiments, cluster manager 160 may facilitate deployment only within its own availability zone 110. Thus, multiple cluster managers 160 may be instantiated to manage the deployment of each availability zone 110 and enable cloud-based services 105 to be distributed across those availability zones 110.

[0023] In various embodiments, cluster manager 160 maintains resource information describing the resources (e.g., processors, storage devices, network ports, servers, virtual machines, etc.) of availability zone 110 that cluster manager 160 can access to deploy cloud-based service 105. Cluster manager 160 may receive a deployment request (e.g., from an administrator of an organization) to deploy cloud-based service 105. In various embodiments, the deployment request includes a specification describing what type of cloud-based service 105 should be deployed and how it should be deployed (e.g., whether a storage service or cluster should be deployed across at least two availability zones 110). Based on receiving the deployment request, cluster manager 160 may consider the requirements of the specification and the availability zone resources it has available to meet those requirements. Cluster manager 160 may then attempt to deploy the requested cloud-based service 105 using the resources of availability zone 110. In various embodiments, cluster manager 160 stores location information describing the locations where cluster manager 160 instantiated components of cloud-based service 105. As an example, the information may indicate that storage cache 130A of the illustrated cloud-based service 105 is instantiated on resources in availability zone 110A.

[0024] In particular embodiments, cluster manager 160 maintains information about various elements of cloud-based service 105, such as the health of database nodes 120, storage cache 130, and log storage 140, as well as metadata used by various nodes to determine how to handle requests from various entities for data. In particular embodiments, the metadata is stored in metadata store 170, which is instantiated in cluster manager 160. In various embodiments, metadata store 170 is a repository that stores metadata, which may relate to the operation of cloud-based service 105. The metadata may specify, for example, where particular data is stored in cloud-based service 105. As an example, the metadata may specify that records for a particular key range are stored in storage caches 130A and 130B. Metadata store 170 may be accessible to various components of cloud environment 100, including database nodes 120. In some embodiments, cloud environment 100 is implemented differently than that shown in FIG. 1A. For example, cluster manager 160 and metadata store 170 may be implemented using nodes in availability zones 110A and 110B. As another example, metadata store 170 may be distributed across availability zones 110A and 110B such that metadata is available locally to a particular availability zone 110.

[0025] In various embodiments, cloud-based service 105 is a service or system that provides a set of functionality and is deployable on the cloud infrastructure of cloud environment 100. Cloud-based service 105 may include, for example, a distributed storage service, a database management system, an email service, a web application server, and the like. In various embodiments, cloud-based service 105 is implemented by executing a set of program instructions. As such, cluster manager 160 may deploy cloud-based service 105 across one or more availability zones 110 by causing its corresponding program instructions to execute on the resources of those availability zones 110. In the illustrated embodiment, cloud-based service 105 is a storage service having multiple storage caches 130 instantiated across availability zones 110A and 110B. In some cases, storage caches 130 of cloud-based service 105 may form one or more clusters (e.g., cache clusters) as described herein. In a cache cluster, storage caches 130 of a particular cluster act on behalf of that cluster. For example, storage caches 130A and 130B may form a cache cluster, such as cache cluster 200 shown in FIG. 2. In some cases, log storage 140 of cloud-based services 105 may form one or more clusters (e.g., log storage clusters) as described herein, where log storage 140 of a particular cluster acts on behalf of that cluster. For example, log storages 140A and 140B may form a log storage cluster, such as log storage cluster 500 shown in FIG. 5.

[0026] Referring now to FIG. 2, a block diagram is shown illustrating exemplary elements of a cloud environment including a cloud-based service configured to process write requests to the cloud-based service. In a particular embodiment, storage cache 130A and storage cache 130B are part of a cache cluster 200. As used herein, the term "cache cluster" refers to an organization of storage caches 130 distributed across multiple availability zones 110 for data replication and availability across availability zones. In various embodiments, cloud-based service 105 receives a write request 202 for a data set 204. The request may be from an entity, such as, for example, a user, tenant, organization, enterprise, etc., at a local database node within cloud-based service 105.

[0027] In the illustrated embodiment, database node 120A is a local database node that receives write request 202 for dataset 204. Database node 120A determines where to store dataset 204 in cloud environment 100. For example, database node 120A may determine where to store dataset 204 based on metadata associated with write request 202 (e.g., metadata of the entity submitting the request). In certain embodiments, database node 120A determines to store dataset 204 in shared object storage 150. In some embodiments, database node 120A also determines to store dataset 204 in storage cache 130A for storage locally in availability zone 110A. As described above, database node 120A may store dataset 204 in shared object storage 150 as the main storage location for the dataset, with storage cache 130A used to store the dataset locally. It should be noted that database node 120A may decide to store data set 204 in one storage element and then the other, or in both storage elements in parallel. For example, database node 120A may store to storage cache 130A before shared object storage 150 due to the low latency of transmission to storage cache 130A.

[0028] 2, when a data set 204 is stored in both storage cache 130A and shared object storage 150, the data set is stored in a low-latency storage element (storage cache 130A) for fast retrieval of the data, and the data set is stored in a persistent storage element (shared object storage 150) for cost-effective, durable, and scalable storage of the data. Database node 120A may send metadata to cluster manager 160 for storing data set 204 in both storage cache 130A and shared object storage 150, as described herein.

[0029] In particular embodiments, database node 120B, which is in a different availability zone 110B than database node 120A in availability zone 110A, determines to retrieve data set 204 from shared object storage 150. Database node 120B may decide to retrieve data set 204 for a variety of reasons. In some embodiments, database node 120B determines to retrieve data set 204 from shared object storage 150 in response to database node 120B receiving a request to retrieve the data set, as described in the embodiment of FIG. 3 below. In various embodiments, database node 120B determines to retrieve data set 204 from shared object storage 150 based on an indication that new or updated data (e.g., there are updates to the data) has been received at cloud-based service 105. For example, metadata received by database node 120B from cluster manager 160 may include an indication that new or updated data has been received by database node 120A (and subsequently stored in shared object storage 150). In other embodiments, database node 120B may determine to retrieve dataset 204 from shared object storage 150 when database node 120B determines that data in storage cache 130B has suffered a data failure (e.g., data is lost, contains errors, or is invalid), which may be determined based on metadata received from cluster manager 160.

[0030] In certain embodiments, as shown in FIG. 2, storage cache 130B (and any other storage caches described herein) retrieves data directly from shared object storage 150 for local storage in availability zone 110B. For example, storage cache 130B may receive instructions received from its co-located database node 120B or a database node 120 in another availability zone to retrieve data set 204. In various embodiments, the retrieval of data set 204 is performed by cache node 180 (shown in FIG. 1B), which includes a set of software routines executable to implement the retrieval of data directly from shared object storage 150. By retrieving data set 204 directly from shared object storage 150, storage cache 130B can retrieve the data set without the data set passing through database node 120B or another database node. In some embodiments, cache node 180 itself may be responsible for determining to retrieve data from shared object storage 150 and performing the retrieval. In some embodiments, cache node 180 may also be able to send (write) data to shared object storage 150 without the data passing through database node 120B (or another database node). In some contemplated embodiments, database node 120B first retrieves data set 204 from shared object storage 150 and then sends the retrieved data set to storage cache 130B for storage locally in availability zone 110B.

[0031] Data retrieval from shared object storage 150 is typically less expensive than data transmission across availability zones. For example, data retrieval from shared object storage 150 may be determined on a per-transaction basis, and data transmission across availability zones is determined based on the amount of data in the transmission. Thus, retrieving a data set 204 from shared object storage 150, rather than retrieving the data set from storage cache 130A or database node 120A for storage in availability zone 110B, may reduce the cost of populating a data set in storage cache 130B.

[0032] When storing dataset 204 locally in storage cache 130B, the dataset is replicated across availability zones 110A and 110B and is available for retrieval locally in either availability zone. Storing locally in availability zones 110A and 110B allows database nodes in either availability zone to retrieve data directly from local storage (e.g., local storage cache) in response to a data retrieval request. For example, database node 120A may retrieve requested data from storage cache 130A if the source of the request is local to availability zone 110A, or database node 120B may retrieve requested data from storage cache 130B if the source of the request is local to availability zone 110B. Thus, storing locally in availability zones 110A and 110B provides low latency data retrieval to cloud-based service 105, while storing in shared object storage 150 provides durable main storage for the data.

[0033] FIG. 3 illustrates a block diagram showing example elements of a cloud environment including a cloud-based service configured to process write and read requests to the cloud-based service. In the illustrated embodiment, database node 120B receives 302 a read request to retrieve 302 a data set 204. The read request 302 may be received from an entity as described herein. In various embodiments, the read request 302 is received from an entity local to availability zone 110B (e.g., availability zone 110B is a local availability zone to the entity making the read request). In response to receiving the read request 302, database node 120B determines whether the requested data is available in storage cache 130B. For example, upon receiving the read request 302, database node 120B may retrieve metadata for data set 204 from cluster manager 160. Database node 120B may then determine from where to retrieve data to satisfy read request 302. If the data is determined to be available in storage cache 130B, database node 120B may retrieve the data from storage cache 130B (not shown) and transmit the data set to the entity making the read request 302.

[0034] In various embodiments, database node 120B determines that the requested data is not available in storage cache 130B, and database node 120B retrieves the data from shared object storage 150. In some embodiments, database node 120B determines that the data in storage cache 130B is older, and database node 120B retrieves the data from shared object storage 150. For example, the versions of the data in storage cache 130A and shared object storage 150 have been updated compared to the version of the data in storage cache 130B. In some embodiments, database node 120B determines that the data in storage cache 130B is invalid or has errors, and database node 120B retrieves the data from shared object storage 150. The dataset 204 retrieved by database node 120B is then sent to storage cache 130B to store the dataset locally in availability zone 110B (e.g., populate the dataset into availability zone 110B) and to the entity making the read request 302. In these embodiments, database node 120B retrieves data set 204 from shared object storage 150 instead of storage cache 130A to reduce the cost of retrieving data, as discussed above.

[0035] As described herein, embodiments of the cloud-based service 105 may include more than two availability zones 110. In such embodiments, the additional availability zones 110 may be populated similar to the embodiment described above for populating availability zone 110B. FIG. 4 illustrates a block diagram showing exemplary elements of a cloud environment including a cloud-based service having three availability zones 110 configured to handle write and read requests to the cloud-based service. In the illustrated embodiment, storage cache 130A and storage cache 130B are populated with data set 204 as described in the embodiments above. In various embodiments, database node 120C receives a read request 402 to retrieve data set 204. The read request 402 may be received from an entity as described herein.

[0036] In response to receiving the read request 402, database node 120C determines a storage element from which to retrieve the requested data. For example, database node 120C determines whether to retrieve the requested data from storage cache 130C or from shared object storage 150. Database node 120C may make the decision of where to retrieve the requested data from in a manner similar to the decision made by database node 120B for the embodiment of FIG. 4 described above. The data set 204 retrieved by database node 120C is then sent to the entity making the read request 402. When database node 120C retrieves data set 204 from shared object storage 150, database node 120C also sends data set 204 to storage cache 130C for storing the data set locally in availability zone 110C (e.g., populating the data set in availability zone 110C).

[0037] FIG. 5 illustrates a block diagram showing exemplary elements of a cloud environment including a cloud-based service configured to process write and read requests to the cloud-based service along with writing and reading to a log of the write and read requests. In the illustrated embodiment, the cloud-based service 105 includes log storage 140 in availability zone 110 (also shown in FIG. 1A ) in addition to database node 120 and storage cache 130. Log storage 140A is colocated with database node 120A and storage cache 130A in availability zone 110A. Log storage 140B is colocated with database node 120B and storage cache 130B in availability zone 110B. In a particular embodiment, log storage 140A and log storage 140B are part of a log storage cluster 500. As used herein, the term “log storage cluster” refers to an organization of log storage 140 distributed across multiple availability zones 110 for replication and availability of log data across availability zones.

[0038] In various embodiments, database node 120A and database node 120B process transactions in cloud environment 100 as described herein. Examples of transactions include "read / write requests" and "data reads / writes" to storage cache 130A, storage cache 130B, or shared object storage 150, as shown in FIG. 5. In some embodiments, database node 120A and database node 120B may also process data transfer(s) transactions between database nodes as needed (e.g., if the data is not available in shared object storage 150). Database node 120A or database node 120B may generate a log of the transaction as it occurs and store (e.g., write) the log of the transaction in a corresponding log storage (e.g., either log storage 140A or log storage 140B, respectively). Thus, log storages 140A and 140B are used by respective database nodes 120A and 120B to store logs of transactions occurring over time.

[0039] In certain embodiments, the transaction log includes a timestamp of the transaction and a copy of the data of the transaction. As an example, database node 120A may generate a log of transactions that write data to storage cache 130A in response to a write request. Thus, the transaction log stored by database node 120A in log storage 140A includes a copy of the data written to storage cache 130A along with a timestamp of the transaction. In various embodiments, the transaction logs stored in log storage 140A and log storage 140B are used to facilitate recovery of data in the event of a problem in cloud environment 100. For example, if a transaction involving writing data to storage cache 130A fails, it may be necessary to roll back one or more writes of data to the storage cache. The transaction log stored in log storage 140A may be read by database node 120A and used to determine which writes were performed and which writes need to be rolled back or undone.

[0040] In some embodiments, transaction logs stored in log storage in one availability zone (e.g., log storage 140A in availability zone 110A) are replicated to log storage in the other availability zone (e.g., log storage 140B in availability zone 110B). In one contemplated embodiment, transaction logs may be replicated similarly to replicating a data set (e.g., data set 204, discussed above). For example, transaction logs for transactions processed by database node 120A may be stored in shared object storage 150 along with the data set (in addition to the transaction logs stored in log storage 140A). The transaction logs may then be retrieved from shared object storage 150 by database node 120B along with the data set, e.g., in response to a read request. Database node 120B may then store the transaction logs in log storage 140B in addition to storing the data set in storage cache 130B. If the cost of retrieving data from shared object storage is per transaction, it may be more cost effective to retrieve the transaction logs along with the data set from shared object storage 150. In another contemplated embodiment, a database node that generates a transaction log (e.g., database node 120A) may send the transaction log directly to each log storage in each availability zone (because the cost of sending and storing the transaction log in the log storage may be reasonable).

[0041] In various embodiments, as described herein, the metadata may be used by various nodes (e.g., database nodes 120 or cache nodes 180 associated with storage cache 130) to determine how to process requests and transactions for data in cloud environment 100. FIG. 6 illustrates a block diagram showing example elements of a cloud environment including a cloud-based service and a cluster manager. As shown in FIG. 6, in certain embodiments, the metadata is stored in metadata store 170 and instantiated by cluster manager 160. Examples of information available in the metadata include the location where particular data was stored in cloud-based service 105 or where particular data is stored in cloud-based service 105. Other examples of information available in the metadata include timestamps, tags, or classifications of data that was or is stored in cloud-based service 105.

[0042] In certain embodiments, database nodes 120 in cloud-based services 105 may access metadata store 170. For example, as shown in FIG. 6, database nodes 120 may implement reading / writing metadata from / to metadata store 170. In various embodiments, database nodes 120 access the metadata to determine where to store data or determine where data is stored based on the metadata. For example, for a write request received by database node 120A, the database node may access the metadata to determine where to send the data in the write request for storage. In some embodiments, metadata in metadata store 170 may be accessed directly by storage cache 130 in cloud-based services 105. Additional embodiments for use of metadata by database nodes 120 and storage cache 130 are described herein.

[0043] Various embodiments may be contemplated in which database node 120 first stores a data set in shared object storage 150 before populating local storage for the database node (e.g., storage cache 130). FIG. 7 illustrates a block diagram showing exemplary elements of a cloud environment including a cloud-based service configured to directly store write requests in shared object storage for later populating local storage in the cloud-based service. The illustrated embodiment may be implemented, for example, in a situation where the data set in the write request is a large data set. In such a situation, it may be more economical for the database node to directly store the data in shared object storage 150 rather than in a local storage cache to take advantage of the high bandwidth capacity and low per-transaction cost of the shared object storage. Additionally, depending on the size of the local storage cache 130, storing a large data set may be difficult. After storing data in shared object storage 150, local storage in the availability zone 110 (e.g., storage cache 130) may be populated from shared object storage 150 as needed. In some cases, the data populated into the local storage cache 130 may be a subset of the data stored in the shared object storage 150 (e.g., if the data set stored in the shared object storage 150 is too large for the local storage cache).

[0044] In the illustrated embodiment of Figure 7, database node 120A receives a write request 702 to store a data set 704. In particular embodiments, database node 120A determines that the data set 704 should be stored in shared object storage 150 before being stored locally in storage cache 130A. For example, the data set 704 may be too large for storage cache 130A, or the data set may be large enough that for economic or other reasons high-bandwidth storage in shared object storage 150 first is preferable. Database node 120A then transmits the data set 704 for storage in shared object storage 150.

[0045] In some embodiments, after the data set 704 is stored in the shared object storage 150, database node 120A or database node 120B may retrieve the data set and store it in storage cache 130A or storage cache 130B, respectively. In other contemplated embodiments, storage cache 130A or storage cache 130B may retrieve data set 704 directly from the shared object storage 150. For example, cache node 180 (shown in FIG. 1B ) implemented in storage cache 130 may retrieve data directly from the shared object storage 150 for storage in the storage cache. In some embodiments, cache node 180 may retrieve data directly in response to an instruction from its co-located database node 120 or a database node 120 in another availability zone. Note that various embodiments described herein may also include retrieval of data directly from the shared object storage 150 by storage cache 130 (via cache node 180).

[0046] In some embodiments, database node 120 (or storage cache 130) retrieves dataset 704, or a subset of the dataset, in response to storing the dataset in shared object storage 150. For example, database node 120 may receive an indication (based on metadata) that new or updated data has been stored in shared object storage 150 and retrieve dataset 704, or a subset of the dataset, in response to the indication. In some embodiments, as described above, database node 120 (or storage cache 130) retrieves dataset 704, or a subset of the dataset, in response to a read request received by the database node. In some embodiments, database node 120 (or storage cache 130) retrieves dataset 704, or a subset of the dataset, in response to a determination that data in the storage cache is invalid, missing, or has an error (e.g., data failure). Class-Based Cache Management

[0047] This disclosure describes various techniques for managing multiple storage caches in a distributed storage system according to a classification of various entities that utilize the distributed storage system. Traditionally, cache management is performed by the application layer of the distributed storage system, for example, by allowing cache data to fall and become rehydrated using explicit or implicit cache population upon cache misses. However, the importance of different data sets may not be the same. That is, data cached for a first entity may be more valuable than data cached for a second, different entity. For example, data that is accessed frequently and quickly after initial storage may be referred to as "hot" data. As such, data stored for some entities (e.g., hot data) may require higher availability than data stored for other entities. In addition, retrieval of high value-added data may require lower latency than low value-added data.

[0048] Specifically, the present disclosure provides variable availability of a cache implemented in different availability zones of a data cluster used by a distributed storage system to store data. As described above with reference to FIG. 1A, the cache is located in different availability zones coupled to database nodes included in the data cluster of the distributed storage system. The database nodes included in the data cluster handle retrieval, manipulation, and storage of data in both the cache cluster and the shared object storage for their corresponding availability zone. During normal operation of the data storage, the database nodes write data to both the cache cluster (for low latency retrieval) and the shared object storage (which provides high bandwidth) such that data is replicated between the two storage elements, where the cache cluster provides temporary (e.g., ephemeral) storage and the shared object storage provides persistent storage. During normal data retrieval operations, data is read from the cache cluster (e.g., from a given cache located in an availability zone) for low latency data retrieval, assuming that the data was previously stored in the cache.

[0049] Because caches store data temporarily, data stored in a given cache may be lost in the event of a failure or power loss to the given cache. Following data loss in a cache, a database node coupled to the cache associated with the lost data may retrieve the lost data directly from the shared object storage to repopulate the given cache. Retrieving data directly from the shared object storage is more cost-effective than retrieving data across availability zones in the event of data loss. Thus, the disclosed cache management techniques may advantageously reduce costs associated with data management in the event of, for example, a system failure. Additionally, the disclosed cache management techniques may advantageously increase the availability of space for high priority data, thereby improving response times for databases provided via cloud-based services.

[0050] In some situations, the use of caches for low latency data retrieval may be expensive, have limited availability, or both. As such, the present disclosure describes techniques for varying the availability of caches to store various data based on the class associated with the data to be stored. In this manner, data is prioritized based on class to more efficiently utilize limited cache storage. This ensures, for example, that sufficient space is available for hot data. The class assigned to a data set to be stored in a cache cluster is determined by either the entity (e.g., user, tenant, organization, enterprise, etc.) associated with that data set, or the type of data contained in the data set, or both. In some embodiments, a system administrator assigns classifications to different entities.

[0051] Different classes of entities dictate how data about these entities is tagged and ultimately stored in the cache. The distributed storage system may determine a number of availability zones for storing a data set based, for example, on the user ID of a user associated with the data set. As one particular example, a "gold" customer may have copies of the data stored across three different availability zones, while a "bronze" customer may have only one copy of the data stored in a single local availability zone. A customer may be classified as "gold" if they pay at least a threshold amount for the database service provided by the distributed storage system. That is, for example, gold customers pay more for the service than bronze customers. In this particular example, if a cache failure occurs in their local availability zone (e.g., due to a power outage in the availability zone where the bronze customers store their data), the bronze customers must wait for the system to fetch the requested data from the shared object storage (which is associated with a higher latency than retrieving data from the cache). In contrast, if a system failure occurs in their local availability zone, the gold customers' data can be retrieved from another externally located availability zone to respond to queries from the gold customers. While responding to the gold customer using data from another availability zone, the system also repopulates the failed cache for the gold customer by pulling the data from the shared object storage.

[0052] Retrieving data from a "local" availability zone involves a database node retrieving data from a cache located in the same availability zone as the database node. That is, a cache is local to a given database node if it is co-located with the given database node in the same geographic cloud region. In some embodiments, the computer system utilized by an entity to request data from the distributed storage system is co-located with the database node utilized to retrieve the requested data. In contrast, retrieving data from an "external" availability zone involves a database node retrieving data from a cache located in a different availability zone than the database node. For example, a database node may retrieve requested data from a cache that is not located in the same geographic region as the database node.

[0053] 8, a block diagram illustrating example elements of a cloud environment including a cloud-based service 105 configured to process write requests from various entities based on metadata 822 associated with the various entities is shown. In the illustrated embodiment, the cloud-based service 105 receives a write request 802 for a dataset 804 from a first entity 810A at a local database node 120A.

[0054] The database node 120A may be co-located with the first entity 810A (e.g., a computing device utilized by this entity) in the illustrated embodiment. For example, the database node 120A and the first entity 810A are located in the same geographic region. For example, the database node 120A is located in an availability zone 110A that is located in a similar geographic location as the first entity 810A. To fulfill the request 802, the database node 120A communicates with the cluster manager 160 to obtain metadata 822 of the first entity 810A. The first entity 810A may be a user, a customer, a tenant, an organization, a business, etc. In various embodiments, the first entity 810A may be associated with an identifier. For example, the database node 120A may provide an identifier (ID) of the first entity 810A to the cluster manager 160.

[0055] The cluster manager 160 locates the metadata 822 of the first entity in the metadata store 170 and provides this metadata of the first entity 810A to the database node 120A. The metadata 822 includes a classification of the first entity 810A. This classification can be used by various database nodes to determine, for example, how to store data about the first entity 810A. If the first entity 810A is, for example, a "gold" customer, this customer data may have a higher priority than, for example, a "silver" customer. In some embodiments, the metadata includes tags maintained by the cluster manager for the different data sets based on the classification of the entities associated with the different data sets. The tags of the data set 804 may, for example, indicate the availability zone of one or more caches that store copies of the data set 804.

[0056] After obtaining the metadata 822 of the first entity 810A, the database node 120A determines where to store the data set 804. The database node 120A first stores the data set 804 locally in the storage cache 130A, and then stores the data set 804 in the shared object storage 150. Note that the database node 120A may store the data in the storage cache 130A before storing the data in the shared object storage 150 due to the lower latency associated with the storage 150. In other situations, the database node 120A may perform storage in two different storage elements in parallel. In various embodiments, the manipulation of data by the databased node 120 is affected by the classification of the different entities. For example, storing a data set may involve determining a cache allocation size based on the class of the entity and determining whether to evict the data to comply with the determined size limit. The manipulation of data based on the entity classification is described in more detail below with reference to Figures 10A and 10B.

[0057] In some embodiments, database node 120A sends data set 804 and metadata 822 to database node 120B. Note that database node 120B is located in a different availability zone 110B than database node 120A. Database node 120B stores a copy of data set 804 in storage cache 130B. Database node 120B may then determine, based on metadata 822 of first entity 810A, whether to send data set 804 to third database node 120C for storage in third storage cache 130C. In other embodiments, database node 120B receives metadata 822 from database node 120A and determines, based on metadata 822, whether to pull data from storage cache 130A for storage in storage cache 130B. For example, database node 120B makes its own storage decision based on metadata 822 rather than receiving instructions from database node 120A to store a copy of data set 804. In some embodiments, database node 120B determines how to store copies of data set 804 in storage cache 130B based on the metadata. For example, database node 120B may instruct storage cache 130B to keep data set 804 in storage for only 10 minutes, while data relating to another entity may be kept for only 1 hour. Storing data based on information contained in the metadata of various entities is described in more detail below with reference to FIG. 9.

[0058] As discussed above with reference to FIG. 1A, the cluster manager 160 maintains information of the various elements of the cloud-based service 105, such as the health of the database nodes 120 (e.g., which nodes have crashed), as well as metadata used by the various nodes to determine how to handle requests for data from various entities. That is, the cluster manager 160 is operable to maintain information of the cache cluster 200. As such, the cloud-based service 105 is zone aware. For example, the service 105 can identify the location (availability zone 110) of the cached data. In various embodiments, the database nodes 120 are configured to provide the location of the data to the cluster manager 160 for storage in the metadata store 170 after caching the data in their respective storage caches 130. This allows the cloud-based service 105 to identify, for example, from which availability zone to start retrieving data when a read request is received. In this way, the service 105 is aware of the most recent availability zone it accessed and the zone in which the requested data is located.

[0059] In some embodiments, database node 120A receives a read request from one or more entities. For example, a first entity 810A may request a different data set than the data set 804 specified in the write request 802. Database node 120A is configured to identify a storage location (e.g., shared object storage 150, or one of storage caches 130A) from which to retrieve the requested data based on metadata 822 of the first entity 810A. In various embodiments, the read request is subject to the classification of the different entities. For example, if database node 120A suffers a cache miss, the node must decide whether to retrieve the data from a slower shared object storage or incur additional costs by pulling the requested data from a faster storage, e.g., a storage cache located in another availability zone. Such class-based decisions are described in more detail below with reference to Figures 10A and 10B.

[0060] 9 is a block diagram illustrating an example database node configured to identify a cache for storing data related to a first entity. In the illustrated embodiment, database node 120A includes a class identification module 910, a storage determination module 920, a tagging module 930, a location determination module 940, and a retrieval determination module 950.

[0061] The database node 120A, in the illustrated embodiment, receives metadata 822 of the first entity 810A from the cluster manager 160 and provides the metadata to the class identification module 910. In the illustrated embodiment, the class identification module 910 identifies a classification 912 of the first entity 810A and transmits the classification to the storage determination module 930. For example, the class identification module 910 may use the ID of the first entity 810A to look up the class of the entity in a table maintained by the cluster manager 160. As another example, the classification of the entity may be included in the ID of the first entity 810A, such as a tenant ID. In this particular example, the module 910 extracts the classification from the tenant ID. In various embodiments, the class identification module 910 may determine that the first entity 810A is a new customer (e.g., has been using the cloud-based service 105 for only two weeks). Based on this determination, the module 910 may classify the first entity 810A as a "low priority" customer.

[0062] The storage determination module 920 determines where to store the data received from the first entity 810A in the write request (e.g., the data set 804 in the write request 820) based on the classification 912. For example, the storage determination module 930 may determine to store data for the high priority entity in both a local cache (e.g., located in the same availability zone as the database node 120A) and several caches located across different availability zones. Such multi-zone storage of data may advantageously reduce or prevent the high priority entity from experiencing cache misses (e.g., for critical data). For example, a large tenant (e.g., a company spread across the United States or the world) is likely to access a database from multiple availability zones simultaneously. In this example, if data is required at one database node, the same data is likely also required by other database nodes in other availability zones for the large entity. As such, the disclosed system is configured to preload data into various caches across availability zones for the large entity in anticipation of the data being requested across various zones, thereby avoiding cache misses.

[0063] In some embodiments, the storage determination module 920 may determine not to cache any data related to a given entity. For example, the storage determination module 920 may determine that data related to a second entity (e.g., an entity with a lower priority) should be stored only in a shared object storage (e.g., storage 150). The storage determination module 920 then outputs one or more identified caches 922 (e.g., cache 130) for storing the data set 804 received from the first entity 810A, for example.

[0064] The tagging module 930, in the illustrated embodiment, assigns a tag to the data set 804 based on the identified cache 922. The tag may be used by database node 120A or other database nodes 120, or both, to determine where additional copies of the data set 804 should be stored. For example, if database node 120A sends the tagged data set 804 to database node 120B, node 120B may store the data set in storage cache 130B and then determine based on the tag whether to send the data set 804 to a third database node 120C for storage in a third storage cache 130C. In this example, database node 120B decides to store and send the data set 804 based on the tag assigned by tagging module 930.

[0065] In some embodiments, database node 120A also provides metadata 822 to location module 940. For example, database node 120A may provide metadata to module 940 when attempting to read data. In the illustrated embodiment, as indicated by the dashed lines, location module 940 may determine one or more availability zones 942 that currently store a copy of data set 804 based on tags assigned to data set 804. For example, metadata 822 may include various tags indicating different availability zones with various different caches. As described herein, database node 120A may assign tags to different data sets before storing the data in a cache according to tags maintained by cluster manager 160. In other words, the tag assigned to data set 804 and stored by the cluster manager in metadata 822 of first entity 810A may indicate one or more availability zones in which the data is currently stored. After locating the availability zones indicated by the tags, location module 940 transmits information indicating the identified availability zones 942 to retrieval determination module 950.

[0066] The retrieval determination module 950 selects an availability zone 942 for retrieving the dataset 804 or determines to retrieve the dataset from the shared object storage 150 (e.g., if the requesting entity is classified as low priority). For example, the module 950 may determine which availability zone will provide the most efficient retrieval of the data 804. That is, if a copy of the dataset 804 is stored in the same availability zone as the database node 120A, retrieval of the data from this same availability zone will be more efficient than retrieval of the data from an outside availability zone. The retrieval determination module 950 outputs the identified cache (e.g., located in one of the availability zones 942) for retrieval of the dataset 804.

[0067] Database node 120A, in the illustrated embodiment, outputs database instructions 924. For example, database node 120A may send instructions 924 to storage cache 130A specifying storing data set 804 based on a tag assigned by tagging module 930. As another example, database node 120A may send instructions 924 to storage cache 130A requesting retrieval of data set 804 based on retrieval decision module 950 identifying cache 130A for retrieval of this data.

[0068] In some embodiments, the database instructions 924 generated by the database node 120A specify the amount of time that the data should be stored by one or more caches. For example, the instructions 924 may instruct that the data set 804 be stored for one hour, but data relating to another entity be stored for only ten minutes, based on, for example, the classification 912 of the first entity 810A and the classifications of the other entities. As one particular example, data relating to gold customers may be stored for a longer time than data relating to other customers.

[0069] 10A is a block diagram illustrating an example allocation of cache space based on entity classification. In the illustrated embodiment, example 1000A shows an example allocation of cache space to a first entity 810A and a second entity 1010B based on metadata 822A and 1022B associated with these two entities.

[0070] In the illustrated embodiment, database node 120A retrieves metadata 822A of a first entity 810A and metadata 1022B of a second entity 1010B from cluster manager 160. Based on this metadata, database node 120A sends instructions to storage cache 130A specifying to allocate a larger amount of space 1032A to the first entity than to the second entity. In the illustrated embodiment, in storage cache 130A, the allocated space 1032A for the first entity 810A is larger than the allocated space 1032B for the second entity 1010B. Database node 120A may determine these space allocations based on the classifications specified in the metadata 822A and 1022B of the two entities. For example, an entity classified as "high priority" may be allocated a larger amount of space in storage cache 130 across various availability zones 1010 than another entity classified as "low priority."

[0071] 10A for clarity, it should be noted that in other embodiments, any of various other storage caches 130 may receive instructions from their respective database nodes 120 specifying the amount of space to be allocated to different entities. In addition, different storage caches 130 may allocate the same or different amounts of space to a given entity. For example, storage cache 130A may allocate a different amount of space to first entity 810A than storage cache 130B.

[0072] In some embodiments, the storage space instructions sent by database node 120A include timing instructions. For example, in addition to specifying the amount of space to be allocated to first entity 810A, database node 120A may specify the amount of time to hold information of first entity 810A in this allocated space 1032A. As one particular example, storage cache 130A is configured to store 100 objects, with 10 of these object slots allocated to gold customers and 1 of these object slots allocated to bronze customers. In this particular example, database node 120A instructs storage cache 130A to hold objects in the 10 slots allocated to gold customers for 1 hour and to hold objects in the 1 slot allocated to bronze customers for 10 minutes.

[0073] In some embodiments, copies of a data set for a given entity are stored in multiple caches across different availability zones. In other embodiments, a single copy of a data set for a given entity is stored in a single cache. For example, based on a first entity 810A having a high priority classification, database node 120A stores data for this entity in both storage cache 130A and one or more caches located outside availability zone 110A. As one particular example, a gold customer may have data stored in multiple different availability zones (e.g., to ensure high availability of the data), while a silver customer may have a single copy of the data stored in a local availability zone (e.g., providing lower availability of the data). Copies of data for high priority entities may be retrieved from a cache in a first availability zone for storage in a cache in another availability zone. However, data for low priority entities may be retrieved, for example, from shared storage cache 150. That is, replication of data for high priority entities is direct (e.g., faster), whereas replication of data for low priority entities is indirect. Although retrieving data across availability zones is faster than retrieving data from slower storage (e.g., shared object storage 150), this type of data retrieval is more expensive than slower data retrieval. Thus, data for lower priority entities may be served from shared object storage rather than served across availability zones.

[0074] In some embodiments, data relating to the second entity is evicted from the cache before data relating to the first entity. For example, when a storage cache reaches capacity, it evicts data stored for silver customers before evicting data stored for gold customers. A database node in the cache's availability zone may instruct the cache to perform such an eviction, for example, based on metadata received from cluster manager 160.

[0075] In some embodiments, database node 120A is configured to prioritize storing data for a first entity 810A over storing data for a second entity 1010B based on the classification of these entities. For example, if database node 120A receives requests from first entities 810A and 1010B in parallel, database node 120A may store data for the first entity 810A in storage cache 130A before storing data for the second entity in cache 130A. Additionally, if storage cache 130A is near capacity when database node 120A receives two write requests, database node 120A is configured to store data for the first entity 810A in storage cache 130A but store data for the second entity 1010B in shared object storage 150 (e.g., data for the second entity is not cached). Handling cache misses

[0076] In some embodiments, the database node is configured to handle the cache miss based on a classification of the entity associated with the cache miss. For example, when database node 120A gets a read request from first entity 810A (not shown in FIG. 10A) and a cache miss occurs, the node is configured to retrieve data from a slower source of memory (e.g., shared object storage 150) or from a storage cache in another availability zone based on the classification of the first entity. If first entity 810A is a gold customer, for example, database node 120A retrieves the requested data from a cache in another availability zone. That is, if the classification of first entity 810A indicates that copies of data for this entity are stored across multiple availability zones, database node 120A pulls data from a storage cache in another availability zone to respond to the read request. However, to repopulate the cache with missing data, database node 120A is configured to pull data from shared storage cache 150 (e.g., from slower memory to avoid the high costs associated with faster cross-zone data retrieval). As one particular example, retrieving data across availability zones may take 2 milliseconds, while retrieving data from shared object storage 150 may take 100 milliseconds.

[0077] In situations where an entity has a low priority classification, database node 120A is configured to simply retrieve data from shared object storage 150. For example, lower priority entities must wait longer to receive requested data because copies of the data may not be stored across availability zones. In some situations, the lowest priority entity (e.g., a bronze customer) may not have data repopulated into the cache associated with the miss at all. In such situations, future requests for data received from this entity are simply satisfied by retrieving the data from shared object storage 150. In other situations, even if the entity stores copies of the data across multiple availability zones, in the event of a cache miss, the data for this entity is retrieved from the shared object storage rather than across availability zones due to the entity's classification (e.g., the entity is low priority). As such, this entity may experience higher latency when retrieving data than other entities (e.g., whose data is retrieved across availability zones).

[0078] 10B is a block diagram illustrating an example system crash in a given availability zone. In the illustrated embodiment, example 1000B shows a system crash occurring within availability zone 110A.

[0079] Availability node 110A, in the illustrated embodiment, experiences a system crash. The crash may be due to a power outage or some other system failure. The crash causes storage cache 130A to be temporarily unavailable until, for example, cloud-based service 105 is able to bring this particular availability zone 110A back online. In the illustrated embodiment, once availability zone 110A recovers, storage cache 130A retrieves metadata 822 from cluster manager 160 to determine what data to retrieve and store for various entities. Based on this determination, cache 130A sends a request 1020A for data belonging to different entities to database node 120A. Database node 120A retrieves the requested data 1022 from shared object storage 150. For example, based on the first entity's metadata 822 specifying that the first entity is a high priority entity, storage cache 130A sends a request to database node 120A indicating to retrieve data regarding the first entity from a cache in another availability zone. As another example, based on the second entity's metadata 822 specifying that the second entity is a low priority entity, the storage cache sends a request to database node 120A indicating to retrieve data for the second entity from shared object storage 150. As yet another example, cache 130A may determine, upon coming back online after a system crash, based on the third entity's metadata 822, not to repopulate data for this entity at all. That is, storage cache 130A may not need to immediately repopulate data for this third entity. Rather, storage cache 130A may wait until the third entity submits a read request for this data (at which point the data is also cached for the third entity) before asking database node 120A to retrieve the data from shared object storage 150.

[0080] Database node 120A, in the illustrated embodiment, may send data request 1020B to database node 120B based on data request 1020A from storage cache 130A that specifies retrieval of data across availability zones (rather than from shared object storage 150), for example. Database node 120B may retrieve data 1022 from storage cache 130B. Database node 120B may then provide data 1022 to database node 120A for storage in cache 130A. Exemplary Methods

[0081] 11 is a flow diagram illustrating an exemplary method for processing a write request using a class-based technique, according to some embodiments. The method 1100 illustrated in FIG. 11 may be used in conjunction with any of the computer circuits, systems, devices, elements, or components disclosed herein, among other devices. In various embodiments, some of the method elements shown may be performed simultaneously, in a different order than shown, or omitted. Additional method elements may be performed as desired. The method 1100 may be performed, for example, by database node 120A.

[0082] At element 1110, in the illustrated embodiment, a database node of the distributed storage system receives a request to store a data set from a first entity of the plurality of entities. In some embodiments, the database node receives a request for a second data set from a second entity of the plurality of entities. In some embodiments, the database node identifies a first cache of the plurality of caches that stores the second data set based on metadata associated with the second entity, the first cache being located in a first availability zone. In some embodiments, the database node responds to the request for the second data set, where responding is performed based on retrieving the second data set from the first cache.

[0083] In some embodiments, the database node determines that a cache miss has occurred in the first cache. In some embodiments, the database node determines whether to service a query for data missing from the first cache using data from the shared object storage based on a classification of an entity associated with the cache miss. In some embodiments, the database node receives a request for a second data set from a second entity of the plurality of entities. In some embodiments, the database node retrieves the second data set from the shared object storage, the retrieval being performed based on a classification indicated in metadata associated with the second entity.

[0084] At 1120, a database node obtains metadata associated with a first entity, the metadata specifying one of a plurality of classifications for the plurality of entities. In some embodiments, the database node obtains the metadata by receiving metadata associated with different ones of the plurality of entities from a cluster manager of the distributed storage system, the metadata including tags maintained by the cluster manager for the different data sets based on classifications of the entities associated with the different data sets. In some embodiments, the tags for the data sets indicate availability zones for each of a number of caches that store copies of the data set.

[0085] In some embodiments, the multiple caches are further configured to store multiple copies of data for a first entity across different availability zones based on a classification of the first entity. In some embodiments, the multiple caches are further configured to store a single copy of data for a second entity based on a classification of a second entity of the multiple entities. In some embodiments, the multiple caches are further configured to perform cache eviction of data stored for the second entity prior to performing cache eviction of data stored for the first entity based on a classification of the second entity. In various embodiments, the multiple caches are configured to store data according to various different criteria, such as latency (e.g., the speed at which an entity can retrieve data), storage size allocation, fast refresh of cache, etc., based on the classification of the different entities.

[0086] At 1130, the database node provides the dataset to one or more of a plurality of caches for storage, the plurality of caches configured to store the dataset based on a classification of the first entity identified in metadata associated with the first entity, the plurality of caches located in two or more availability zones. In some embodiments, the plurality of caches are further configured to allocate a greater amount of cache space to the first entity than a second entity based on the classification of the first entity as indicated in the metadata associated with the first entity and the classification of the second entity as indicated in metadata associated with a second entity of the plurality of entities. For example, gold customers may be allocated more space in the plurality of storage caches than silver customers.

[0087] At 1140, the database node stores the dataset in a shared object storage coupled to the database node. In some embodiments, in response to a system failure, the multiple caches are configured to determine to repopulate data related to the first entity from the shared object storage into one or more of the multiple caches based on a classification of the first entity. In some embodiments, in response to a system failure, the multiple caches are configured to determine not to repopulate data related to a second entity of the multiple entities into one or more of the multiple caches based on a classification of the second entity, wherein repopulating the data related to the first entity is performed without being triggered by a cache miss.

[0088] In some embodiments, the plurality of caches prioritizes storing data for a first entity over storing data for a second entity of the plurality of entities based on a classification of the first entity. In some embodiments, the first availability zone includes a first cache that replicates data for at least one other cache in a second availability zone.

[0089] FIG. 12 is a block diagram illustrating an example method for processing read requests for data stored in one or more caches using class-based techniques, according to some embodiments. The method 1200 illustrated in FIG. 12 may be used in conjunction with any of the computer circuits, systems, devices, elements, or components disclosed herein, among other devices. In various embodiments, some of the method elements shown may be performed simultaneously, in a different order than shown, or omitted. Additional method elements may be performed as desired. The method 1200 may be performed, for example, by database node 120B.

[0090] At element 1210, in the illustrated embodiment, a first database node in a first availability zone receives a request for a first data set from a first entity of the plurality of entities. In some embodiments, the first database node is further configured to receive a request from the first entity to store a second data set. In some embodiments, the first database node is further configured to provide the second data set to one or more of the plurality of caches for storage and store the second data set in a shared object storage. In some embodiments, providing the second data set to one or more of the plurality of caches for storage includes providing instructions specifying storing multiple copies of the second data set in multiple caches located across different availability zones, the instructions being provided based on a classification of the first entity specified in metadata associated with the first entity.

[0091] At 1220, a first database node retrieves metadata associated with the first entity, the metadata specifying one of a plurality of classifications for the plurality of entities. In some embodiments, the metadata associated with the first entity is retrieved from a cluster manager of the distributed storage system, and the metadata maintained by the cluster manager for different ones of the plurality of entities indicates respective availability zones in which data for the different ones of the plurality of entities is stored.

[0092] At 1230, a database node communicates with a first cache of the multiple storage caches to retrieve the first data set, the first cache being located in a first availability zone, and the multiple storage caches being configured to store data for the multiple entities based on the multiple classifications.

[0093] At 1240, the database node identifies, based on the communication, that a cache miss has occurred (e.g., the cache from which the database node attempted to retrieve the first data set does not currently store the first data set), and therefore the cache returns a message to the database node indicating the lack of data.

[0094] At 1250, the database node determines, based on the cache miss, whether to respond to the request for the first data set using the shared object storage or using a second cache located in the second availability zone. In some embodiments, in response to the system failure, the multiple caches are further configured to determine, based on a classification of the first entity, to repopulate one or more of the multiple caches with data regarding the first entity using the data retrieved from the shared object storage. In some embodiments, in response to the system failure, the multiple caches are further configured to determine, based on a classification of the second entity, not to repopulate one or more of the multiple storage caches with data regarding the second entity, wherein repopulating the data regarding the first entity is performed without being triggered by a cache miss.

[0095] FIG. 13 is a flow diagram illustrating an exemplary method for processing a write request in a cloud-based service, according to some embodiments. The method 1300 illustrated in FIG. 13 may be used in conjunction with any of the computer circuits, systems, devices, elements, or components disclosed herein, among other devices. In various embodiments, some of the method elements shown may be performed simultaneously, in a different order than shown, or omitted. Additional method elements may be performed as desired. The method 1300 may be performed by executing a set of program instructions stored on a non-transitory computer-readable medium.

[0096] At 1310, in the illustrated embodiment, a request to store a dataset is received at a first database node in a first availability zone.

[0097] At 1320, in the illustrated embodiment, the first database node sends the data set to a first storage cache in the first availability zone for storing the data set in the first availability zone. In some embodiments, the first database node is operable to retrieve requested data from the first storage cache in the first availability zone in response to a database query.

[0098] At 1330, in the illustrated embodiment, the first database node transmits the dataset to shared object storage for storing the dataset in the first availability zone.

[0099] At 1340, in the depicted embodiment, the second database node in the second availability zone retrieves the data set from the shared object storage. In some embodiments, the second database node in the second availability zone is operable to retrieve the data set from the shared object storage in response to a data failure in the second storage cache. In some embodiments, the second database node in the second availability zone is operable to retrieve the data set from the shared object storage in response to a determination that the data set in the shared object storage includes an update to the data set stored by the second storage cache in the second availability zone.

[0100] At 1350, in the illustrated embodiment, the second database node sends the data set to a second storage cache in the second availability zone for storing the data set in the second availability zone. In some embodiments, the second database node is operable to retrieve the requested data from the second storage cache in the second availability zone if the requested data is not found in the first storage cache in the first availability zone. In some embodiments, the first database node or the second database node is operable to retrieve the data from the shared object storage if the requested data is not found in either the first storage cache in the first availability zone or the second storage cache in the second availability zone.

[0101] In some embodiments, the storage cache is hosted by servers that store cached data in internal non-volatile memory, and the shared object storage is a persistent object storage. In some embodiments, the database nodes are operable to send logs for operations by the multiple database nodes to a log storage cluster for storage in log storage elements separated into two or more availability zones, the availability zone of the log storage elements corresponding to the availability zone of the storage cache. In some embodiments, logs for a specified availability zone of the storage cache are stored in the availability zone of the log storage element corresponding to the specified availability zone.

[0102] FIG. 14 is a flow diagram illustrating another exemplary method for processing a write request in a cloud-based service, according to some embodiments. The method 1400 illustrated in FIG. 14 may be used in conjunction with any of the computer circuits, systems, devices, elements, or components disclosed herein, among other devices. In various embodiments, some of the method elements illustrated may be performed simultaneously, in a different order than illustrated, or omitted. Additional method elements may be performed as desired. The method 1400 may be performed by executing a set of program instructions stored on a non-transitory computer-readable medium.

[0103] At 1410, in the illustrated embodiment, a first database node in a distributed storage system receives a request to store a data set, the first database node located in a first availability zone of the distributed storage system. In some embodiments, the first database node sends a log about storing the data set in a first storage cache in the first availability zone to a log storage cluster having multiple log storage elements, the log being stored in the first availability zone in a log storage element corresponding to the first availability zone in the cache cluster.

[0104] At 1420, in the illustrated embodiment, the first database node sends the data set to a first storage cache in the first availability zone for storing the data set in the first availability zone.

[0105] At 1430, in the illustrated embodiment, the first database node transmits the dataset to shared object storage for storing the dataset in the first availability zone.

[0106] At 1440, in the illustrated embodiment, a second database node in a second availability zone receives a request to retrieve the data set.

[0107] In the illustrated embodiment, the second database node retrieves the dataset from the shared object storage at 1450. In some embodiments, the second database node determines to retrieve the version of the dataset from the shared object storage in response to determining that the version of the dataset in the second storage cache is missing, invalid, or older than the version of the dataset in the shared object storage.

[0108] At 1460, in the illustrated embodiment, the second database node sends the dataset to a second storage cache in the second availability zone for storing the dataset in the second availability zone. In some embodiments, the second database node sends the retrieved dataset to the entity initiating the request.

[0109] In some embodiments, the first database node transmits metadata to the cluster manager for transmission and storage of the dataset in the shared object storage and the first storage cache. In some embodiments, the second database node retrieves the metadata from the cluster manager in response to the request and determines, based on the metadata, to retrieve the dataset from the shared object storage to satisfy the request. In some embodiments, determining to retrieve the dataset from the shared object storage includes determining that a version of the dataset in the first storage cache and the shared object storage is updated compared to a version of the dataset in the second storage cache.

[0110] In some embodiments, in response to a failure of the dataset in the first storage cache in the first availability zone, the first database node retrieves a version of the dataset from the shared object storage and replaces the failed dataset in the first storage cache in the first availability zone with the retrieved version of the dataset.

[0111] In some embodiments, a third database node in a third availability zone receives the request to retrieve the dataset, retrieves the dataset from the shared object storage, and sends the retrieved dataset to a third storage cache in the third availability zone for storage in the third availability zone. Exemplary Multitenant Database System

[0112] Referring now to FIG. 15, an exemplary multi-tenant database system (MTS) 1500 is shown in which various techniques of the present disclosure can be implemented. In FIG. 15, the MTS 1500 includes a database platform 1510, an application platform 1520, and a network interface 1530 connected to a network 1540. Also as shown, the database platform 1510 includes a data storage 1512 and a set of database servers 1514A-N that interact with the data storage 1512, and the application platform 1520 includes a set of application servers 1522A-N, each having an environment 1524. In the illustrated embodiment, the MTS 1500 is connected to various user systems 1550A-N via the network 1540. The disclosed multi-tenant system is included for illustrative purposes and is not intended to limit the scope of the present disclosure. In other embodiments, the techniques of the present disclosure are implemented in a non-multi-tenant environment, such as a client / server environment, a cloud computing environment, a clustered computer, or the like.

[0113] The MTS 1500, in various embodiments, is a set of computer systems that together provide various services to users (alternatively referred to as "tenants") that interact with the MTS 1500. In some embodiments, the MTS 1500 implements a customer relationship management (CRM) system that provides a mechanism for tenants (e.g., businesses, government agencies, etc.) to manage relationships and interactions with customers and potential customers. For example, the MTS 1500 may enable tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), identify sales opportunities, record service issues, and manage marketing campaigns. Additionally, the MTS 1500 may enable these tenants to identify how they communicated with customers, what customers purchased, when they last purchased an item, and how much they paid. To provide the services of the CRM system and / or other services, as shown, the MTS 1500 includes a database platform 1510 and an application platform 1520.

[0114] The database platform 1510, in various embodiments, is a combination of hardware elements and software routines that implement a database service for storing and managing data for the MTS 1500, including tenant data. As shown, the database platform 1510 includes data storage 1512. The data storage 1512, in various embodiments, includes a set of storage devices (e.g., solid state drives, hard disk drives, etc.) connected to each other over a network (e.g., a storage attached network (SAN)) and configured to store data redundantly to prevent data loss. In various embodiments, the data storage 1512 is used to implement a database (e.g., cloud-based service 105) that includes a collection of information organized to allow access, storage, and manipulation of the information. The data storage 1512 may implement a single database, a distributed database, a collection of distributed databases, a database with redundant online or offline backups or other redundancies, etc. As part of implementing a database, the data storage 1512 may store files (e.g., datasets 204) that include one or more database records having their respective data payloads (e.g., values ​​of fields in a database table) and metadata (e.g., key values, timestamps, table identifiers for tables associated with the record, tenant identifiers for tenants associated with the record, etc.).

[0115] In various embodiments, a database record may correspond to a row in a table. A table generally includes one or more data categories logically arranged as columns or fields in a viewable schema. Thus, each record in a table may include an instance of data in each category defined by a field. For example, a database may include a table describing a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Thus, a record in that table may include a value for each of the fields in the table (e.g., name for the name field). Another table may describe a purchase order that includes fields for information such as customer, product, sale price, date, etc. In various embodiments, standard entity tables, such as tables for account, contact, lead, and opportunity data, are provided for use by all tenants, each including predefined fields. The MTS 1500 may store database records for one or more tenants in the same table. That is, tenants may share tables. Thus, a database record includes a tenant identifier that indicates the owner of the database record in various embodiments. As a result, one tenant's data is kept secure and isolated from other tenants' data, so that the tenant cannot access another tenant's data unless such data is explicitly shared.

[0116] In some embodiments, data stored in data storage 1512 is organized as part of a log-structured merge tree (LSM tree). An LSM tree typically includes two high-level components: an in-memory buffer and persistent storage. During operation, database server 1514 may first write database records to a local in-memory buffer and then flush these records to persistent storage (e.g., data storage 1512). As part of flushing the database records, database server 1514 may write the database records to a new file contained in the "top" level of the LSM tree. Over time, the database records may be rewritten by database server 1514 to new files contained in lower levels as the database records move down the levels of the LSM tree. In various implementations, as database records age and move down the LSM tree, they are moved to slower and slower storage devices in data storage 1512 (e.g., from solid state drives to hard disk drives).

[0117] When the database server 1514 wants to access a database record for a particular key, the database server 1514 may traverse different levels of the LSM tree for files that potentially contain a database record for that particular key. If the database server 1514 determines that a file may contain a related database record, the database server 1514 may fetch the file from the data storage 1512 into the memory of the database server 1514. The database server 1514 may then check the fetched file for a database record having the particular key. In various embodiments, a database record is immutable once written to the data storage 1512. Thus, when the database server 1514 wants to modify a value in a row of a table (which may be identified from an accessed database record), the database server 1514 writes out a new database record to the top level of the LSM tree. Over time, that database record is merged down a level of the LSM tree. Thus, an LSM tree may store various database records for a database key, with older database records for that key being located at a lower level in the LSM tree than newer database records.

[0118] The database server 1514, in various embodiments, is a hardware element, software routine, or combination thereof capable of providing database services such as data storage, data retrieval, and / or data manipulation. The database server 1514 may correspond to a database node 120. Such database services may be provided by the database server 1514 to components within the MTS 1500 (e.g., application server 1522) and components external to the MTS 1500. As an example, the database server 1514 may receive a database transaction request from the application server 1522 requesting to write or read data to the data storage 1512. The database transaction request may specify an SQL SELECT command to select one or more rows from one or more database tables. The contents of the row may be defined in a database record, and thus the database server 1514 may locate and return one or more database records corresponding to the selected table row or rows. In various cases, a database transaction request may instruct database server 1514 to write one or more database records for an LSM tree, which database server 1514 maintains implemented on database platform 1510. In some embodiments, database server 1514 implements a relational database management system (RDMS) or an object-oriented database management system (OODBMS) that facilitates storage and retrieval of information from data storage 1512. In various cases, database servers 1514 may communicate with each other to facilitate processing of transactions. For example, database server 1514A may communicate with database server 1514N to determine whether database server 1514N has written a database record to its in-memory buffer for a particular key.

[0119] The application platform 1520, in various embodiments, is a combination of hardware elements and software routines that implements and executes CRM software applications, as well as providing associated data, code, forms, web pages, and other information to and from user systems 1550, and storing associated data, objects, web page content, and other tenant information via the database platform 1510. To facilitate these services, in various embodiments, the application platform 1520 communicates with the database platform 1510 to store, access, and manipulate data. In some instances, the application platform 1520 may communicate with the database platform 1510 via different network connections. For example, one application server 1522 may be coupled via a local area network and another application server 1522 may be coupled via a direct network link. TCP / IP (Transfer Control Protocol and Internet Protocol) is an exemplary protocol for communicating between the application platform 1520 and the database platform 1510, although it will be apparent to one skilled in the art that other transport protocols may be used depending on the network interconnect used.

[0120] The application server 1522 is a hardware element, software routine, or combination thereof capable of providing the services of the application platform 1520, including, in various embodiments, processing requests received from tenants of the MTS 1500. The application server 1522 can, in various embodiments, spawn environments 1524 that can be used for various purposes, such as providing functionality for developers to develop, execute, and manage applications (e.g., business logic). Data can be transferred to an environment 1524 from another environment 1524 and / or the database platform 1510. In some cases, an environment 1524 cannot access such data from other environments 1524 unless such data is explicitly shared. In some embodiments, multiple environments 1524 can be associated with a single tenant.

[0121] The application platform 1520 may provide access to multiple different hosted (standard and / or custom) applications to the user systems 1550, including CRM applications and / or applications developed by tenants. In various embodiments, the application platform 1520 may manage the creation of applications, the testing of applications, the storage of applications in database objects in the data storage 1512, the execution of applications in the environments 1524 (e.g., virtual machines in the process space), or any combination thereof. In some embodiments, the application platform 1520 may add and remove application servers 1522 from the server pool at any time for any reason, and there may be no server affinity of a user and / or organization to a particular application server 1522. In some embodiments, an interface system (not shown) implementing load balancing functionality (e.g., F5Big-IP load balancer) is located between the application servers 1522 and the user systems 1550 and configured to distribute requests to the application servers 1522. In some embodiments, the load balancer routes user requests to the application servers 1522 using a least connections algorithm. Other examples of load balancing algorithms, such as round robin and observed response time, may also be used. For example, in a particular embodiment, three consecutive requests from the same user may hit three different servers 1522, or three requests from different users may hit the same server 1522.

[0122] In some embodiments, the MTS 1500 provides security mechanisms, such as encryption, to keep each tenant's data separate unless the data is shared. If more than one server 1514 or 1522 is used, they may be located in close proximity to each other (e.g., in a server farm located in a single building or campus) or may be distributed at distances from each other (e.g., one or more servers 1514 are located in city A and one or more servers 1522 are located in city B). Thus, the MTS 1500 may include one or more logically and / or physically connected servers, either locally or distributed across one or more geographic locations.

[0123] One or more users may interact with the MTS 1500 over the network 1540 (e.g., via user systems 1550). The user systems 1550 may correspond to, for example, a tenant of the MTS 1500, a provider of the MTS 1500 (e.g., an administrator), or a third party. Each user system 1550 may be a desktop personal computer, a workstation, a laptop, a PDA, a mobile phone, or any wireless access protocol (WAP)-enabled device, or any other computing device capable of interfacing directly or indirectly to the Internet or other network connection. The user systems 1550 may include dedicated hardware configured to interface with the MTS 1500 over the network 1540. The user systems 1550 may execute a graphical user interface (GUI) corresponding to the MTS 1500, an HTTP client (e.g., a browsing program such as Microsoft's Internet Explorer® browser, Netscape's Navigator® browser, Opera's browser, or a WAP-enabled browser in the case of a cell phone, PDA, or other wireless device), or both, to enable a user of the user system 1550 (e.g., a subscriber to the CRM system) to access, process, and view information and pages available from the MTS 1500 over the network 1540. Each user system 1550 may include one or more user interface devices, such as a keyboard, mouse, touch screen, pen, etc., for interacting with a graphical user interface (GUI) provided by a browser on a display monitor screen, LCD display, etc., along with pages, forms, and other information provided by the MTS 1500 or other systems or servers. As discussed above, the disclosed embodiments are suitable for use with the Internet, which refers to a particular global internetwork of networks.However, it should be understood that other networks may be used in place of the Internet, such as an intranet, an extranet, a virtual private network (VPN), a non-TCP / IP based network, any LAN or WAN, etc.

[0124] Because users of the user system 1550 may be users of different capabilities, the capabilities of a particular user system 1550 may be determined by one or more permission levels associated with the current user. For example, when a salesperson is interacting with the MTS 1500 using a particular user system 1550, the user system 1550 may have the capabilities (e.g., user privileges) assigned to the salesperson. However, when an administrator is interacting with the MTS 1500 using the same user system 1550, the user system 1550 may have the capabilities (e.g., administrator privileges) assigned to the administrator. In a system with a hierarchical role model, a user at one permission level may access applications, data, and database information accessible by a user at a lower permission level, but may not access certain applications, database information, and data accessible by a user at a higher permission level. Thus, different users may have different capabilities with respect to accessing and modifying application and database information depending on the user's security or permission level. Some data structures managed by the MTS 1500 are assigned at the tenant level, while other data structures are managed at the user level.

[0125] In some embodiments, the user system 1550 and its components are configurable using an application, such as a browser, that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 1500 (and additional instances of the MTS, if there is more than one) and their components are operator configurable using application(s) that include computer code executable on the processing elements. Thus, various operations described herein may be performed by executing program instructions that are stored in a non-transitory computer-readable medium and executed by the processing elements. The program instructions may be stored in a non-volatile medium, such as a hard disk, or may be stored in any other volatile or non-volatile memory medium or device, as is well known, such as a ROM or RAM, or may be provided on any medium capable of storing program code, such as a compact disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, etc. Additionally, the entire program code or portions thereof may be transmitted and downloaded from a software source, e.g., via the Internet or from another server, as is well known, or transmitted via any other conventional network connection, as is well known, (e.g., extranet, VPN, LAN, etc.), using any communication medium and protocol, as is well known, (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.) as is well known. It will also be understood that computer code for implementing aspects of the disclosed embodiments may be implemented in any programming language that can be executed on a server or server system, such as, for example, C, C+, HTML, Java, JavaScript, or any other scripting language, such as VBScript.

[0126] Network 1540 may be a local area network (LAN), a wide area network (WAN), a wireless network, a point-to-point network, a star network, a token ring network, a hubbed network, or any other suitable configuration. A global internetwork of networks is sometimes referred to as the "Internet" with a capital "I" and is an example of a Transfer Control Protocol and Internet Protocol (TCP / IP) network. However, it should be understood that the disclosed embodiments may utilize any of a variety of other types of networks.

[0127] The user system 1550 communicates with the MTS 1500 using TCP / IP and, at higher network levels, may communicate using other common Internet protocols such as HTTP, FTP, AFS, WAP, etc. For example, if HTTP is used, the user system 1550 may include an HTTP client, commonly referred to as a "browser," for sending and receiving HTTP messages from an HTTP server in the MTS 1500. Such a server may be implemented as the only network interface between the MTS 1500 and the network 1540, although other techniques may be used as well or instead. In some implementations, the interface between the MTS 1500 and the network 1540 includes a load sharing function, such as a round-robin HTTP request distributor, to balance the load and distribute incoming HTTP requests evenly across multiple servers.

[0128] In various embodiments, the user system 1550 communicates with the application server 1522 to request and update system-level and tenant-level data from the MTS 1500, which may require one or more queries to the data storage 1512. In some embodiments, the MTS 1500 automatically generates one or more SQL statements (SQL queries) designed to access the desired information. In some cases, the user system 1550 may generate requests having a specific format that corresponds to at least a portion of the MTS 1500. As an example, the user system 1550 may request that a data object be moved to a particular environment using an object notation that describes the object-relational mapping (e.g., JavaScript object notation mapping) of a number of specified objects. Exemplary Computer System

[0129] 16, a block diagram of an exemplary computer system 1600 is shown that may implement cloud environment 100, cloud-based services 105, database nodes 120, storage cache 130, log storage 140, shared object storage 150, cluster manager 160, MTS 1500, and / or user systems 1550. Computer system 1600 includes a processor subsystem 1680 coupled to system memory 1620 and I / O interface(s) 1640 via an interconnect 1660 (e.g., a system bus). I / O interface(s) 1640 are coupled to one or more I / O devices 1650. For convenience, a single computer system 1600 is shown in FIG. 16, but system 1600 may be implemented as two or more computer systems operating together.

[0130] Processor subsystem 1680 may include one or more processors or processing units. In various embodiments of computer system 1600, multiple instances of processor subsystem 1680 may be coupled to interconnect 1660. In various embodiments, processor subsystem 1680 (or each processor unit within 1680) may include a cache or other form of on-board memory.

[0131] The system memory 1620 can be used to store program instructions executable by the processor subsystem 1680 to cause the system 1600 to perform various operations described herein. The system memory 1620 can be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.). The memory in the computer system 1600 is not limited to primary storage, such as the memory 1620. Rather, the computer system 1600 can also include other forms of storage, such as cache memory in the processor subsystem 1680 and secondary storage on the I / O devices 1650 (e.g., hard drives, storage arrays, etc.). In some embodiments, these other forms of storage can also store program instructions executable by the processor subsystem 1680.

[0132] I / O interface 1640 may be any of a variety of types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I / O interface 1640 is a bridge chip (e.g., a southbridge) from a frontside to one or more backside buses. I / O interface 1640 may be coupled to one or more I / O devices 1650 via one or more corresponding buses or other interfaces. Examples of I / O devices 1650 include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local or wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system 1600 is coupled to a network via network interface device 1650 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0133] The present disclosure includes references to "embodiments" that are non-limiting implementations of the disclosed concepts. References to "an embodiment," "one embodiment," "a particular embodiment," "some embodiments," "various embodiments," and the like do not necessarily refer to the same embodiment. Numerous possible embodiments are contemplated, including the specific embodiment described in detail, as well as modifications or alternatives that fall within the spirit or scope of the present disclosure. Not all embodiments necessarily exhibit any or all of the potential advantages described herein.

[0134] The disclosure includes references to "an embodiment" or groups of "embodiments" (e.g., "several embodiments" or "various embodiments"). The embodiments are different implementations or instances of the disclosed concepts. References to "an embodiment," "one embodiment," "a particular embodiment," etc. do not necessarily refer to the same embodiment. Numerous possible embodiments are contemplated, including those specifically disclosed, as well as modifications or alternatives that fall within the spirit or scope of the disclosure.

[0135] This disclosure may describe potential advantages that may result from the disclosed embodiments. All implementations of these embodiments do not necessarily manifest any or all of the potential advantages. Whether advantages are realized for a particular implementation depends on many factors, some of which are outside the scope of this disclosure. Indeed, there are several reasons why an implementation within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other components outside the scope of this disclosure that, in conjunction with one of the disclosed embodiments, nullify or reduce one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular implementation (e.g., implementation technique or tool) may also nullify or reduce the disclosed advantages. Even assuming a skilled implementation, the realization of advantages may still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, inputs provided to a particular implementation may prevent one or more problems addressed in this disclosure from occurring in a particular case, such that the benefits of the solution may not be realized. Given the existence of possible factors external to this disclosure, it is expressly intended that any potential advantages described herein should not be construed as limitations on the scope of the claims that must be met in order to demonstrate infringement. Rather, the identification of such potential advantages is intended to illustrate the type(s) of improvement available to a designer having the benefit of this disclosure. Where such advantages are permissively described (e.g., describing a particular advantage as "may result"), this is not intended to convey doubt as to whether such advantages may actually be realized, but rather to acknowledge the technological reality that the realization of such advantages often depends on additional factors.

[0136] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims that are drafted based on this disclosure, even if only a single example is described with respect to a particular feature. The disclosed embodiments are intended to be illustrative rather than limiting, unless otherwise stated in the disclosure. Thus, the present application is intended to enable claims that encompass the disclosed embodiments, as well as such alternatives, modifications, and equivalents that would be apparent to those skilled in the art having the benefit of this disclosure.

[0137] For example, features in the present application may be combined in any suitable manner. Accordingly, during prosecution of this application (or an application claiming priority thereto), new claims may be prepared for any such combination of features. In particular, with reference to the appended claims, features from a dependent claim may be combined, as appropriate, with features of other dependent claims, including claims that are dependent on other independent claims. Similarly, features from each independent claim may be combined, as appropriate, with features of other dependent claims.

[0138] Thus, the accompanying dependent claims may be drafted so that each is dependent on a single other claim, although additional dependencies are also contemplated. Any combination of features in the dependent claims consistent with this disclosure is contemplated and may be claimed in this or another application. In short, combinations are not limited to those specifically recited in the accompanying claims.

[0139] It is also contemplated that, where appropriate, a claim drafted in one form or statutory type (e.g., apparatus) is intended to support a corresponding claim in another form or statutory type (e.g., method).

[0140] Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. Notice is hereby given that the definitions provided in the following paragraphs, as well as throughout this disclosure, should be used in determining how to interpret the claims drafted based on this disclosure.

[0141] Reference to a singular item (i.e., a noun or noun phrase preceded by "a," "an," or "the") is intended to mean "one or more" unless the context clearly dictates otherwise. Thus, a reference to "an item" in a claim does not exclude additional instances of that item absent attendant context. A "plurality" of an item refers to a set consisting of two or more of that item.

[0142] The word "may" is used herein in a permissive sense (i.e., having the possibility, being able to) and not in a mandatory sense (i.e., must).

[0143] The terms "comprising" and "including" and their forms are open ended and mean "including, but not limited to."

[0144] When the term "or" is used in this disclosure in reference to a list of alternatives, it will generally be understood to be used in an inclusive sense unless the context provides otherwise. Thus, a statement of "x or y" is equivalent to "x or y, or both," and thus encompasses 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, a phrase such as "either x or y, but not both" makes it clear that "or" is used in an exclusive sense.

[0145] The phrases "w, x, y, or z, or any combination thereof" or "at least one of ... w, x, y, and z" are intended to encompass all possibilities involving a single element to the total number of elements in the set. For example, given the set [w,x,y,z], these phrases encompass any single element of the set (e.g., w but not x, y, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Thus, the phrase "at least one of ... w, x, y, and z" refers to at least one element of the set [w,x,y,z], thereby encompassing all possible combinations of elements in the set. This phrase should not be construed as requiring that there be at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0146] In this disclosure, various "labels" may precede a noun or noun phrase. Unless otherwise dictated by context, different labels used for a feature (e.g., "first database node," "second database node," "particular database node," "given database node," etc.) refer to different instances of the feature. Additionally, the labels "first," "second," and "third," when applied to features, do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise noted.

[0147] The phrase "based on" is used to describe one or more factors that influence the decision. This term does not exclude the possibility that additional factors may influence the decision. That is, the decision may be based only on the specified factors or on the specified factors and other unspecified factors. Consider the phrase "determining A based on B." This phrase specifies that B is a factor that is used to determine A or that influences the decision of A. This phrase does not exclude that the decision of A may also be based on some other factor, such as C. This phrase is also intended to encompass embodiments in which A is determined solely based on B. As used herein, the phrase "based on" is synonymous with the phrase "based at least in part on."

[0148] The phrases "in response to" and "responsive to" describe one or more factors that cause an effect. The phrases do not exclude the possibility that additional factors may affect or otherwise trigger the effect, either in conjunction with or independently of the specified factors. That is, the effect may be responsive only to these factors, or to the specified factors and other unspecified factors. Consider the phrase "performing A in response to B." The phrase specifies that B is the factor that triggers the execution of A or triggers a particular outcome of A. The phrase does not exclude that performing A may also be responsive to some other factor, such as C. The phrase also does not exclude that performing A may be responsive to B and C jointly. The phrase is also intended to encompass embodiments in which A is performed in response to B only. As used herein, the phrase "in response to" is synonymous with the phrase "responsive at least in part to." Similarly, the phrase "in response to" is synonymous with the phrase "at least partially in response to."

[0149] Within this disclosure, different entities (which may be variously referred to as "units," "nodes," other components, etc.) may be described or claimed as being "configured" to perform one or more tasks or operations. This formulation - an "entity" configured to perform one or more tasks - is used herein to refer to a structure (i.e., something physical). More specifically, this formulation is used to indicate that this structure is arranged to perform one or more tasks during operation. A structure may be said to be "configured to" perform some task even if the structure is not currently operating. Thus, an entity described or described as being "configured to" perform some task refers to something physical, such as a device, a node, a system having a memory and a processor unit that store executable program instructions to perform the task. This phrase is not used herein to refer to something intangible.

[0150] In some cases, various units / nodes / components may be described herein as performing a set of tasks or operations, and it will be understood that those entities are "configured to" perform those tasks / operations, even if not specifically stated otherwise.

[0151] The term "configured to" is not intended to mean "configurable to." For example, an unprogrammed FPGA is not considered to be "configured to" perform a particular function. However, this unprogrammed FPGA may be "configurable" to perform that function. After appropriate programming, the FPGA can be said to be "configured" to perform a particular function.

[0152] For purposes of filing a U.S. patent application based on this disclosure, a recitation in a claim that a structure is "configured to" perform one or more tasks is expressly intended not to invoke 35 U.S.C. 112(f) as to that claim element. If an applicant wishes to invoke 35 U.S.C. 112(f) during prosecution of a U.S. patent application based on this disclosure, the applicant will recite the claim element using the construction "means for [performing a function]."

Claims

1. 1. A distributed storage system, comprising: A cache cluster including multiple storage caches separated into two or more availability zones; a shared object storage coupled to the cache cluster; at least one database node located in at least one of the availability zones; Equipped with The at least one database node Receiving a request to store a dataset; sending the data set to a first storage cache in the first availability zone for storing the data set in the first availability zone; transmitting the data set to the shared object storage for storing the data set in the shared object storage; and operable to The cache nodes in the second storage cache in the second availability zone Retrieving the data set from the shared object storage; storing the data set in the second availability zone in a second storage cache in the second availability zone; The system is operable to:

2. The system of claim 1 , wherein the storage cache is hosted by a server that stores cached data in internal non-volatile or volatile memory, and the shared object storage is a persistent object storage.

3. The system of claim 1 , wherein the cache nodes in the second storage cache are operable to retrieve the data set from the shared object storage in response to a data failure in the second storage cache.

4. 2. The system of claim 1, wherein the cache nodes in the second storage cache are operable to retrieve the data set from the shared object storage in response to a determination that the data set in the shared object storage includes an update to a data set stored by the second storage cache in the second availability zone.

5. 2. The system of claim 1, further comprising a log storage cluster including a plurality of log storage elements separated into two or more availability zones, the availability zones of the log storage elements corresponding to the availability zones of the storage cache, and the at least one database node operable to send logs of operations by the at least one database node to the log storage cluster for storage in the log storage elements.

6. 6. The system of claim 5, wherein the logs for a specified availability zone of the storage cache are stored in an availability zone of the log storage element that corresponds to the specified availability zone.

7. 2. The system of claim 1, wherein the at least one database node is operable to retrieve requested data from the first storage cache in the first availability zone in response to a database query.

8. 8. The system of claim 7, wherein the at least one database node is operable to retrieve the requested data from the second storage cache in the second availability zone if the requested data is not found in the first storage cache in the first availability zone.

9. 9. The system of claim 8, wherein the at least one database node is operable to retrieve data from the shared object storage if the requested data is not found in either the first storage cache in the first availability zone or the second storage cache in the second availability zone.

10. 1. A method comprising: receiving a request to store a dataset at at least one database node in a distributed storage system, the at least one database node being located in at least one availability zone of the distributed storage system; the at least one database node sending the data set to a first storage cache in the first availability zone for storing the data set in the first availability zone; the at least one database node transmitting the data set to a shared object storage for storage in the shared object storage; receiving a request by the at least one database node to retrieve the data set from a second availability zone; a cache node in the second availability zone retrieving the data set from the shared object storage; storing the data set retrieved from the shared object storage in a second storage cache in the second availability zone by the cache node for storage in the second availability zone; The method includes:

11. 11. The method of claim 10, further comprising the at least one database node sending the data set retrieved by the cache node to an entity initiating the request to retrieve the data set from the second availability zone.

12. The method of claim 10 , further comprising the at least one database node further sending metadata to a cluster manager for storing the data set in the shared object storage and the first storage cache.

13. the cache node retrieving the metadata for storing the data set from the cluster manager in response to the request to retrieve the data set from the second availability zone; determining, based on the metadata, to retrieve the data set from the shared object storage to satisfy the request; The method of claim 12 further comprising:

14. 14. The method of claim 13, wherein determining to retrieve the data set from the shared object storage comprises determining that versions of the data set in the first storage cache and the shared object storage are updated compared to a version of the data set in the second storage cache.

15. the at least one database node retrieving a version of the data set from the shared object storage in response to a failure of the data set in the first storage cache in the first availability zone; the at least one database node replacing the failed data set in the first storage cache in the first availability zone with the retrieved version of the data set; The method of claim 10 further comprising:

16. The at least one database node sending a log about storing the data set in the first storage cache in the first availability zone to a log storage cluster having a plurality of log storage elements; storing the log in a first availability zone of the log storage element corresponding to the first availability zone in the distributed storage system; The method of claim 10 further comprising:

17. receiving a request from a cache node in a third availability zone to retrieve the data set from the third availability zone; the cache node in the third availability zone retrieving the data set from the shared object storage; storing the retrieved data set in a third storage cache in the third availability zone for storage by the cache node in the third availability zone in the third availability zone; The method of claim 10 further comprising:

18. 1. A non-transitory computer-readable medium having stored thereon program instructions executable by a distributed storage system in a cloud-based cache cluster, the program instructions including: receiving, by at least one database node in the distributed storage system, a request to store a dataset, the at least one database node being located in at least one availability zone of the distributed storage system; transmitting, by the at least one database node, the data set to a first storage cache in the first availability zone for storing the data set in the first availability zone; transmitting, by the at least one database node, the data set to a shared object storage for storage in the shared object storage; receiving, by the at least one database node, a request to retrieve the data set from a second availability zone; retrieving the data set from the shared object storage by a cache node in the second availability zone; storing, by the cache nodes, the data set retrieved from the shared object storage in a second storage cache in the second availability zone for storage in the second availability zone; A non-transitory computer-readable medium for causing a computer to perform operations including:

19. 20. The non-transitory computer-readable medium of claim 18, wherein the cache node determines to retrieve the version of the dataset from the shared object storage in response to determining that a version of the dataset in the second storage cache is missing, invalid, or older than a version of the dataset in the shared object storage.

20. The request to retrieve the data set from the second availability zone is a database query from an entity, and the program instructions are configured to: transmitting, by the cache node, the retrieved data set to the entity; 20. The non-transitory computer-readable medium of claim 18, executable by the distributed storage system to perform operations including:

Images