Cloud database storage system

Abstract

To provide a system for reducing a blast radius of a storage node failure and time and a resource necessary for full redundancy after the storage node failure.SOLUTION: A system and a method include a memory of a database table shadow in a plurality of storage nodes. Each shadow may be stored as a plurality of duplication and each replication of a given shadow is stored in a different storage node. In a case of a failure of the given storage node, all of the shadow duplication stored in the given storage node are copied from another storage node to one of a new storage node and an existing storage node. The storage node can store a shadow of a database table related to a plurality of tenants.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a cloud database storage system and related methods.

Background Art

[0002] Modern organizations often utilize a system landscape consisting of distributed computing systems that provide various computing services. For example, an organization may deploy services within an on-premises data center (which may itself be located within different geographical locations) and within a computing system located within a data center provided by one or more platform (PaaS: Platform-as-a-Service) providers as a service. The computing system may include a cloud-based computing system (e.g., providing services using scalable on-demand virtual machines).

[0003] In the software as a service (SaaS) model, a service provider deploys a service (e.g., a database) on a cloud-based computing system (which may in some cases be provided by a separate PaaS provider). The service can be used by customers in exchange for payment of fees to the service provider. The customer is not involved in the operation, management, and scaling of the service, which are all performed by the service provider.

[0004] In the case of a database provided as a service, the service provider attempts to ensure scalability for both the storage device and the computing layer of the database. Some deployments disaggregate the storage device and the computing layer by using separate node clusters for each layer. This separation enables (theoretically) independent scaling of the layers.

[0005] The storage layer of such a database service may consist of many storage nodes, each containing local disk space and computing resources (e.g., RAM and CPU) to run the storage node process. Typically, the database service is a single-tenant system, where all the storage nodes in the storage layer collectively store a single database. Because each tenant requires a separate storage layer, these systems are costly for small tenants. The elasticity of such systems is not suitable for larger tenants because adding or removing individual storage nodes does not provide a reasonably fine-grained scaling increase.

[0006] A database instance failure results in a correlated failure of all tenants hosted on that database instance, a large "blast radius." To provide high availability, legacy database systems sequentially replicate the entire primary database instance to separately running failover instances. Users switch to the failover instance if the primary instance fails. While such systems reduce perceived downtime to seconds, they require expensive additional infrastructure (i.e., failover instances) and considerable time to restore full redundancy (i.e., through replicating the entire database from the failover instance to a new instance).

[0007] To provide sufficient performance, networking resources are provisioned to offer bandwidth suitable for data-intensive operations such as recovery and scaling. These resources are therefore over-provisioned for typical loads, resulting in wasted costs. [Overview of the project] [Means for solving the problem]

[0008] What is needed is a system that reduces the blast radius of a memory node failure and the time and resources required for full redundancy after a memory node failure. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows a distributed database system according to several embodiments. [Figure 2] This figure shows the storage of database table shards on multiple storage nodes according to several embodiments. [Figure 3] This diagram shows how to access database table shards in several different embodiments. [Figure 4] This is a flowchart illustrating the process for accessing database table shards on a storage node, according to several embodiments. [Figure 5a] This figure shows a scale-out process according to several embodiments. [Figure 5b] This figure shows a scale-out process according to several embodiments. [Figure 5c] This figure shows a scale-out process according to several embodiments. [Figure 6a] This figure shows a scale-in process according to several embodiments. [Figure 6b] This figure shows a scale-in process according to several embodiments. [Figure 6c] This figure shows a scale-in process according to several embodiments. [Figure 7] This figure shows the storage of database table shard replicas on multiple storage nodes according to several embodiments. [Figure 8] This diagram shows how to access database table shards in several different embodiments. [Figure 9a]This figure shows a scale-out process using database table shard replication in several embodiments. [Figure 9b] This figure shows a scale-out process using database table shard replication in several embodiments. [Figure 9c] This figure shows a scale-out process using database table shard replication in several embodiments. [Figure 10a] This figure shows a scale-in process using database table shard replication in several embodiments. [Figure 10b] This figure shows a scale-in process using database table shard replication in several embodiments. [Figure 10c] This figure shows a scale-in process using database table shard replication in several embodiments. [Figure 11a] This figure shows a recovery process using database table shard replication in several embodiments. [Figure 11b] This figure shows a recovery process using database table shard replication in several embodiments. [Figure 11c] This figure shows a recovery process using database table shard replication in several embodiments. [Figure 12] This figure shows the storage of database table shard replicas for multiple tenants on multiple storage nodes, according to several embodiments. [Figure 13] This figure shows how multiple tenants can access database table shards in several different embodiments. [Figure 14] This is a block diagram of a cloud-based virtual machine providing a database service, based on several embodiments. [Modes for carrying out the invention]

[0010] The following description is provided to enable a person of ordinary skill in the art to make and use the described embodiments. However, various modifications will still be readily apparent to a person of ordinary skill.

[0011] According to some embodiments, the data of each tenant of a database system is logically split into shards, and the shards of a given tenant are distributed across multiple storage nodes. The shards are large enough to amortize the operational cost for individual data items by enabling batch request processing, and small enough to enable preferably fast overall shard operations (e.g., for backup / restore, shard reallocation). In some embodiments, the shards are sized substantially equally and can be in the range of 1 GB to 10 GB per shard.

[0012] Over time, the data added to a shard may grow the shard larger than the specified maximum shard size. In this case, the shard can be split into two shards, with one of the two shards remaining on the original storage node and the other shard being moved to a different storage node. Similarly, when the size of a shard is reduced below the specified minimum shard size, the shard can be merged with a shard related to an adjacent key range that may be located on a different storage node. Embodiments can thus provide fine-grained elastic space allocation for each tenant by maintaining as many shards as necessary for a particular tenant.

[0013] Embodiments can also decouple storage node failures. Specifically, a failure of a storage node restricts access to the shards stored on that storage node, but does not restrict access to other shards stored on other storage nodes. Assuming a number of storage nodes across which the shards of each tenant are relatively evenly distributed, a storage node failure will render unavailable only a small portion of the data of a given tenant. Such unavailability may be temporary in view of the recovery options described herein.

[0014] Furthermore, because tenant data is distributed across storage nodes, compute nodes can access the data in parallel in some embodiments. For example, instead of traversing a large table located on a single storage node, embodiments may allow compute nodes to simultaneously query hundreds of storage nodes containing shards of the table. This parallelization can provide performance improvements of at least an order of magnitude in some scenarios.

[0015] To achieve high availability, each shard can be replicated multiple times between different storage nodes. Relatively small shard sizes and large inter-node bandwidth allow such replication to be processed quickly. In the event of a storage node failure or decommissioning, all data in all shards stored by that storage node remains accessible through the replication of those shards stored on other storage nodes.

[0016] Furthermore, since shard replication is typically distributed across the entire remaining storage system, surviving nodes can generate new replications to restore full redundancy. For example, each shard replication can be sent from its storage node to a new storage node. When database shards are distributed across many storage nodes, the re-replication of data to restore full database redundancy can proceed in parallel across all nodes (i.e., using n:n communication rather than 1:1 communication). Thus, in some embodiments, the restoration of full redundancy can be completed within seconds. In contrast, other systems require the redistribution of the entire database across the network and / or the copying of data across all nodes to compensate for a failed node.

[0017] Allocating disk space to tenants on a small scale and sharing storage infrastructure across tenants significantly reduces the fixed cost per tenant (i.e., to the cost of the smallest possible shard size) compared to systems that require at least one separate virtual machine per tenant. Some embodiments employ a stateless compute layer where the compute layer can be shared across tenants, further reducing the incremental cost for smaller tenants to near zero.

[0018] Figure 1 shows a system in several embodiments. The components shown in Figure 1 can be implemented using any suitable combination of known or to be known computing hardware and / or software. Such combinations may include cloud-based implementations in which computing resources are elastically virtualized and allocated. In some embodiments, two or more components are implemented by a single computing device.

[0019] Clients 110-116 may include computing systems running applications or services that issue database queries. Clients 110-116 may issue database queries in response to requests entered into them by users (not shown). The compute layer 120 receives queries (for example, via APIs provided by the compute layer 120) and, working with data stored in the storage layer 130, provides the corresponding query results to clients 110-116.

[0020] The computing layer 120 includes a plurality of computing nodes 122-128. Similarly, the storage layer 130 includes a plurality of storage nodes 132-138. Each of the nodes 122-128 and 132-138 includes at least one processing unit (e.g., a CPU), random access memory, and at least one data storage device (e.g., an NVMe SSD), and may include a single computer server, a group of computer servers, a virtual machine, and any other combination known or to be known. In some embodiments, the computing layer 120 is entirely stateless and booted over a network, and therefore the computing nodes 122-128 are not required to include storage devices.

[0021] In some embodiments, the compute layer 120 and the storage layer 130 are implemented by a Kubernetes cluster. Kubernetes is an open-source system for automated deployment, scaling, and managing containerized applications. A Kubernetes cluster consists of many nodes, which may have physical servers and / or virtual machines. The nodes run applications within software containers. One or more containers may be grouped into pods, where each container in a given pod shares the same computing resources and local network.

[0022] According to some embodiments, Figure 1 represents a single-tenant database system. Storage nodes 132-138 store data related to a single tenant, and compute node 122 processes queries received from clients 110-116 operated by users of the single tenant. As described above, each of the storage nodes 132-138 stores its own shard of data related to the single tenant to provide parallelism and resilience.

[0023] Figure 1 may represent a multi-tenant database system in several embodiments. In such an implementation, storage nodes 132-138 store data related to multiple tenants. The data is stored in shards distributed across storage nodes 132-138. Each of the compute nodes 122-128 may run a tenant-specific single-tenant database instance. Therefore, if client 110 is accessed by a user of the first tenant, queries issued by client 110 are directed to one of the compute nodes 122-128 associated with the first tenant.

[0024] In yet another example, memory nodes 132-138 store data related to multiple tenants, and each of the compute nodes 122-128 runs a multi-tenant database instance. Therefore, regardless of which tenant is querying the data, any of the compute nodes 122-128 can process the queries received from clients 110-116.

[0025] Figure 2 illustrates the storage of a single tenant's database table shards on multiple storage nodes according to several embodiments. Database tables 210, 220, and 230 are logical representations of data associated with a given database tenant. Database tables 210, 220, and 230 may conform to specific schemas known in the art. According to some embodiments, each row in database tables 210, 220, and 230 contains a value describing an instance of an object. Embodiments are not limited to any type of object.

[0026] Each of the database tables 210, 220, and 230 is split into multiple shards. Specifically, database table 210 is split into shards 212, 214, and 216; database table 220 is split into shards 222 and 224; and database table 230 is split into shards 232, 234, 236, and 238. Each shard is associated with a key range. In some embodiments, the shards are sized substantially equally, and the size of each shard can range from 1GB to 10GB. Embodiments include any algorithm for splitting tables into shards and defining the key ranges of the shards. In one implementation, the storage layer 130 is a key / value store, and the key range of a shard is the key range of keys stored within that shard. According to some embodiments, a shard may contain data from two or more database tables.

[0027] In this example, the shards are stored across storage nodes 132, 134, and 136 of storage layer 130. Storage node 132 stores shards 212, 214, 216, and 236; storage node 134 stores shards 222, 232, and 238; and storage node 136 stores shards 224 and 234. It is assumed that storage node 138 and any other storage nodes in storage layer 130 store shards of other unshown database tables for the tenant.

[0028] The distribution of shards across the storage nodes of the storage layer 130 can be based on any preferred algorithm. Since storage node 132 stores all shards 212, 214, and 216 of table 210, a failure of node 132 results in the complete inaccessibility of all data in table 210. In some embodiments, each shard of the table is preferably stored on a different node (e.g., shards 222 and 224 of table 220) such that a node failure results in the inaccessibility of at most one shard of the table. In some embodiments, each storage node may store at most one shard of a given tenant.

[0029] Figure 3 illustrates access to database table shards in several embodiments. The database compute frontend 310 may have a database instance running on a compute node as described above. The database compute frontend 310 may need to access database table data in response to queries received from clients. However, with the shard storage described above, the database compute frontend 310 must first determine the location of the shard containing the desired data in order to direct its request to the appropriate storage node.

[0030] The topology manager 320 consists of one or more nodes that store shard metadata 325. The shard metadata 325 may be associated with a well-known ID in the system and stored in one or more storage nodes 132 to 136 in the same format as other shards stored therein. According to some embodiments, the shard metadata 325 specifies the storage location of the database shards. For example, the shard metadata 325 may identify each shard by a table ID and key range. Furthermore, each shard is also associated with an identifier of the storage node in which it is stored. Thus, the database compute frontend 310 queries the topology manager 320 for the location of shards relevant to a query (for example, all shards in a given table) and receives, in response, instructions for the storage node in which the desired shards are stored. The database compute frontend 310 then issues database operations to the node in which the shards are stored. As described above, if the shards reside on two or more storage nodes, the operations may be advantageously performed in parallel.

[0031] In some embodiments, tenant creation causes the creation of a single shard on a storage node. This shard may be considered the root shard, and the root shard contains shard metadata 325 associated with that tenant, rather than the topology manager 320. Therefore, in response to an incoming query, the database compute frontend 310 queries the topology manager for the location of the root shard of the tenant related to the query, retrieves the shard metadata from that root shard, and based on that, determines the storage node where the desired shard is stored.

[0032] Figure 4 is a flowchart of process 400 for accessing database table shards on a storage node, according to several embodiments. Process 400 and other processes described herein may be implemented using any preferred combination of hardware and software. The software program code embodying these processes may be stored on any non-temporary tangible medium, including fixed disks, volatile or non-volatile random access memory, DVDs, flash drives, or magnetic tape, and may be executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Such processors, processor cores, and processor threads may be implemented by virtual machines provisioned within a cloud-based architecture. Embodiments are not limited to the examples described below.

[0033] Process 400 assumes the prior storage of database table shards on multiple storage nodes, as described above. In S410, a request is received for the location of the database table shards. The request may identify the shard using the database table identifier and one or more primary keys (or ranges of primary keys) of the identified table. The request is received from a database frontend, which may be running on a compute node in the compute layer, as described above. The shards may contain data required by the database frontend to serve queries received from client applications.

[0034] Upon request, the storage node storing the shard is determined based on the stored metadata. According to some embodiments, the stored metadata is shard metadata stored by a topology manager as described above. Such a topology manager can therefore receive a request in S410 and determine a storage node in S420 based on the request and the shard metadata. In one example, the request includes a database table identifier and a primary key range. In S420, the topology manager can determine the shard containing the primary key range of the identified table based on the shard metadata and further determine the storage node storing the shard based on the shard metadata. In some embodiments, S420 includes determining the root shard of the tenant related to the query and determining the storage node where the desired shard is stored based on the metadata stored within the root shard. In S430, the identifier of the determined storage node is provided to the requester.

[0035] In S420, it may be determined that the primary key range extends to two or more shards, in which case S420 determines the storage node that stores the two or more shards. In another example, the request includes only a table identifier and does not include a primary key or key range. Therefore, S420 includes determining all shards of the identified table and their corresponding storage nodes. The received request may include any number of table identifiers and associated key ranges.

[0036] In S440, a database operation for a shard is received at the database node where the shard is stored. The database operation may be issued by a database frontend from which a request is received in S410, and if two or more storage nodes are identified, the database operation may be issued to each of the other nodes and received at each of the other nodes in S440, thereby enabling parallel execution of operations by two or more storage nodes.

[0037] Figures 5a to 5c illustrate the scale-out process according to several embodiments. Storage nodes 132, 134, and 136 in Figure 5a store their respective shards, some of which are larger in size than those in Figure 3. Therefore, it can be determined manually or automatically (based on resource consumption parameters) to reduce the amount of storage consumed by one of the storage nodes by adding another storage node to the corresponding storage layer. Such determination can be performed by a topology manager according to several embodiments. Alternatively, it can be determined that shards 234 and 238 should be split because their size exceeds a specified maximum shard size.

[0038] Figure 5b shows the addition of storage node 510 to the storage layer. Figure 5b also shows the movement of shards from storage nodes 132, 134, and 136 to storage node 510. This movement may be intended to achieve load balancing, which reduces memory consumption per node.

[0039] Figure 5c shows the resulting storage layer. As shown, shard 236 has been moved from storage node 132 to storage node 510. Shard 238 has been split into shards 238a and 238b, with shard 238b being moved from storage node 134 to storage node 510. Similarly, shard 234 has been split into shards 234a and 234b, with shard 234b being moved from storage node 136 to storage node 510. Embodiments are not limited to splitting shards into equally sized shards.

[0040] Advantageously, each shard moved to node 510 is moved over a different (and substantially independent) inter-node network connection. The shard movement can therefore occur substantially faster than if all three shards were moved from a single node to node 510. Moreover, this move requires the node from which the shards were moved to use only 1 / n of its respective bandwidth. As a result, the impact on node performance and system response time is minimal.

[0041] Shard splitting and the movement of resulting shards can occur regardless of whether or not storage nodes are added. The addition of storage nodes and the movement of shards to those storage nodes can also occur regardless of whether or not any shards have been split.

[0042] Figures 6a to 6c illustrate the scale-in process according to several embodiments. Figure 6a shows storage nodes 132, 134, 136, and 138, some of which are smaller than those shown in Figure 5c. Therefore, it can be determined manually or automatically (again, based on resource consumption parameters) to reduce the number of storage nodes in the corresponding storage layer. This determination can be performed by a topology manager according to some embodiments.

[0043] In this example, it is determined that memory node 136 should be decommissioned. Therefore, Figure 6b shows the movement of shards of memory node 136 to memory nodes 132 and 510. The memory nodes to which the shards are moved are determined in some embodiments to achieve load balancing.

[0044] Figure 6c shows the resulting storage layer, where shard 224 has been moved from storage node 136 to storage node 132, and shard 234a has been moved from storage node 136 to storage node 510. Storage node 136 no longer stores any shards and can therefore be decommissioned. According to some embodiments, scaling in may also or alternatively involve merging two or more smaller shards into one shard and, based on considerations of load balancing and resource consumption, placing one shard on a storage node. According to some embodiments, by the key range associated with each shard, shards should only be merged with shards associated with adjacent key ranges.

[0045] Figure 7 illustrates the storage of a single tenant's database table shard replicas across multiple storage nodes in several embodiments. As described with respect to Figure 2, database tables 210, 220, and 230 are logical representations of data associated with a given database tenant and are split into multiple shards associated with different key ranges.

[0046] As shown in Figure 7, the storage layer 130 stores multiple copies of each of the shards 212, 214, 216, 222, and 224. Replication can be performed in any preferred manner, such as employing erasure coding across the replication using a consensus protocol, each of which may have different advantages and disadvantages depending on the use case. The distribution of replication across the storage nodes of the storage layer 130 can be based on any preferred algorithm that can provide load balancing and high availability.

[0047] Figure 7 shows three replicas per shard, but embodiments can utilize any number of replicas. In the case of a protocol that replicates each shard across three different storage nodes, the data in a particular shard remains fully accessible even if one of the three storage nodes serving that shard fails or is decommissioned. If the replication factor is set to 4 or greater, the system will continue to operate normally even if multiple nodes fail (for example, two node failures using a replication factor of 5, or k node failures with a replication factor of 2k+1).

[0048] Figure 7 reflects the consensus protocol in which each shard is represented by a leader replica and two subordinate replicas. The leader replica is determined by a leader election algorithm and is clearly outlined in Figure 7. All actions on a given shard are directed to the storage node that stores the shard's leader replica, and changes made to it are replicated to the shard's subordinate replicas. If the node containing the shard's leader replica fails, one of the shard's remaining subordinate replicas is designated as the leader replica, and the new leader replica begins replicating its changes to the remaining subordinate replicas. Furthermore, as described below, the new subordinate replica is quickly instantiated on another (or new) storage node to re-establish the desired consensus protocol.

[0049] Figure 8 illustrates access to shard leader replicas in several embodiments. The database compute frontend 310 determines the location of the shard leader replica containing the desired data in order to direct its request to the appropriate storage node. The shard metadata 325 of the topology manager 320 specifies the storage locations of all replicas for each shard and appends the current leader replica for each shard. If a storage node containing a shard leader replica fails or is decommissioned, a new leader replica for the shard is elected, and the shard metadata 325 is updated accordingly. Any subordinate replicas stored on the failed / decommissioned storage node are removed from the shard metadata 325. The shard metadata 325 also specifies the table IDs and key ranges for each shard.

[0050] Therefore, the database compute frontend 310 queries the topology manager 320 for the location of the shards relevant to the query (for example, all shards of a given table) and receives from it the identifier of the storage node where the leader replicas of the desired shards are stored. The database compute frontend 310 then issues database operations to the node where the leader replicas of the shards are stored. Again, these operations can be advantageously performed in parallel if the shards reside on two or more storage nodes.

[0051] The database compute frontend 310 may also receive identifiers of the storage nodes where the dependent replicas of the desired shard are stored. Therefore, if there is a failure in the operation against the identified leader replica, the database compute frontend 310 can seamlessly retry the operation against one of the corresponding dependent replicas stored on a different storage node.

[0052] In some embodiments, the root shard stores shard metadata 325 instead of the topology manager 320, and the topology manager 320 simply stores the location of the root shard. The root shard may also be stored in multiple copies among storage nodes 132-138, in which case the topology manager 320 stores the location of each copy.

[0053] Figures 9a to 9c illustrate a scale-out process using database table shard replication in several embodiments. Storage nodes 132, 134, 136, and 138 in Figure 9a store their respective shards as shown in Figures 7 and 8. Suppose it is determined, manually or automatically, to add another storage node to the corresponding storage layer, for example, to reduce the amount of storage consumed by one of the storage nodes or to provide improved availability.

[0054] Figure 9b shows the addition of storage node 910 to the storage layer. As also shown in Figure 9b, the shards are moved from storage nodes 132, 134, and 138 to storage node 910, respectively. Figure 9c shows the resulting storage layer. Shard 222 has been moved from storage node 132 to storage node 910, shard 212 has been moved from storage node 134 to storage node 910, and shard 214 has been moved from storage node 138 to storage node 910. In all examples described herein, the shard metadata 325 is updated to reflect the new storage location of the replicas due to the movement of shards between storage nodes.

[0055] The shards are moved to node 910 (i.e., from 132 to 910, from 134 to 910, and from 138 to 910) via different (and substantially independent) inter-node network connections. By duplicating one shard instead of three, the burden on nodes 132, 134, and 138 is significantly less than in other implementations.

[0056] Figures 10a to 10c illustrate a scale-in process using database table shard replication in several embodiments. Figure 10a reflects a storage layer as described in relation to Figure 9c. Alternatively, it is assumed that decommissioning of storage node 136 is determined manually or automatically based on underutilization of one or more storage nodes in the shown storage layer.

[0057] Figure 10b shows the movement of shards from storage node 136 to storage nodes 132, 134, and 138. The shards to be moved and the storage nodes to which they are moved may be determined in some embodiments based on load balancing and / or resource consumption algorithms. Figure 10c shows the resulting storage layer, where shard 224 has been moved from storage node 136 to storage node 132, shard 214 has been moved from storage node 136 to storage node 134, and shard 212 has been moved from storage node 136 to storage node 138. Since storage node 136 no longer stores any shards, it may be decommissioned. The shard metadata 325 is then updated to replicate the new replicated storage locations due to the aforementioned movement of shards between storage nodes.

[0058] Figures 11a to 11c illustrate a recovery process using database table shard replication in several embodiments. Figure 11a reflects the storage layer as described in relation to Figure 10a, but in this case, it is assumed that storage node 136 has unexpectedly failed. In some implementations, the decommissioning of node 136 may be carried out as described below with respect to Figures 11a to 11c.

[0059] When a failure is detected, the topology manager 320 accesses the shard metadata 325 to determine which shards were stored on the storage node 136. Next, based on the shard metadata 325, the topology manager 320 identifies which of the remaining storage nodes is storing a copy of the shard that was stored on storage node 136. The topology manager 320 then instructs the storage layer to copy each copy of the shard that was stored on storage node 136 from one of the remaining storage nodes to a different remaining storage node.

[0060] Figure 11b illustrates the copying of each copy of a shard stored on the failed storage node 136 to other storage nodes. For example, shard metadata 325 indicates that the failed storage node 136 stored copies of shards 224, 214, and 212, and that the copy of shard 214 was the leader copy. Shard metadata 325 also indicates that copies of shard 224 are located on nodes 134 and 138, copies of shard 214 are located on nodes 132 and 910, and copies of shard 212 are located on nodes 132 and 910. Thus, Figure 11b shows the copying of shard 224 from node 138 to node 132, shard 214 from node 910 to node 134, and shard 212 from node 910 to node 138. As described above, copying between different sets of nodes allows each copying operation to proceed in parallel and utilize the full available inter-node bandwidth. Figure 11c shows the resulting storage layer after the copying operation is complete.

[0061] The specific storage node to which the shard is replicated in between may be determined, according to some embodiments, based on load balancing and / or resource consumption algorithms. The shard metadata 325 is updated to reflect the new replication storage location. The update may also include the selection of a replica of shard 214 stored as a leader replica within node 134, and instructions for that selection within the shard metadata 325.

[0062] Figure 12 shows the storage of database table shard replicas for multiple tenants on multiple storage nodes according to several embodiments. Database table 210 contains data for tenant 1, and database table 710 contains data for tenant 2. Database table 210 conforms to the schema associated with tenant 1, and database table 710 conforms to the schema associated with tenant 2. Each tenant may be associated with many other database tables, each containing their respective data. As is known in the art, it is intended that the data for tenant 1 is accessible only to users of tenant 1, and the data for tenant 2 is intended to be accessible only to users of tenant 2.

[0063] In the example shown, table 210 is divided into shards 212, 214, and 216, and table 710 is divided into shards 712 and 714. The storage layer 130 stores multiple copies of each of shards 212, 214, 216, 712, and 714, as described above with respect to Figure 7. Figure 12 represents a consensus protocol for each shard consisting of a leader copy (i.e., shown by a clear outline) and two subordinate copies. These copies can be distributed across the storage nodes of the storage layer 130 based on any preferred algorithm for providing load balancing and high availability. The node scaling out, scaling in, and disaster recovery shown in Figure 12 may proceed in the manner described above with respect to Figures 9a-9c, 10a-10c, and 11a-11c.

[0064] Figure 13 illustrates access to database table shards for multiple tenants in several embodiments.

[0065] The database compute frontend 1310 serves users of tenant 1, and the database compute frontend 1320 serves users of tenant 2. The database compute frontend 1310 may have a single-tenant database instance running on a single compute node of the compute layer, and the database compute frontend 1320 may have a single-tenant database instance running on another single compute node of the compute layer. In some embodiments, the database compute frontend 1310 and the database compute frontend 1320 run on the same compute node.

[0066] The topology manager 320 may operate as described above. Also as described above, the shard metadata 325 can associate each shard with a table identifier, key range, location of each replica, and identifier of the leader replica. However, since tables 210 and 710 conform to different schemas, the table identifier and key range of the tables in tenant 1 may overlap / conflict with the table identifier and key range of the tables in tenant 2. Therefore, the shard metadata 325 also associates each shard with a tenant identifier. If the database compute frontends 1310 and 1320 are single-tenant database frontends, the tenant identifier may be the identifier of the container in which the frontend is running.

[0067] In response to an incoming query, the database compute frontend 1310 (or frontend 1320) queries the topology manager 320 for the location of the shard associated with that query. The topology manager 320's query includes the tenant identifier, table identifier, and key range (for example, as a concatenated string). Based on the query received from the database compute frontend 1310 (or frontend 1320) and the shard metadata 325, the topology manager 320 determines the identifier of the storage node where the leader replica of the desired shard is stored. The database compute frontend 1310 (or frontend 1320) then issues database operations to the node where the leader replica of the shard is stored. If the shard resides on two or more storage nodes, the storage nodes may perform these operations in parallel.

[0068] In some embodiments, each tenant is associated with a root shard containing shard metadata for that tenant, and the topology manager 320 stores the location of the root shard for each tenant. The database compute frontend 1310 or 1320 then queries the topology manager 320 for the storage node that stores the root shard for a given tenant, reads the root shard metadata from the storage node, and determines the location of the desired shard based on the metadata. Each root shard may also be stored in multiple copies among storage nodes 132-138, in which case the topology manager 320 stores the location of each copy of each root shard.

[0069] Some embodiments include a multi-tenant database compute frontend that serves users of tenant 1 and users of tenant 2. The multi-tenant database compute frontend accesses shard replication as described above for both database compute frontend 1310 and database compute frontend 1320, using the appropriate tenant identifier to query the topology manager 320.

[0070] Figure 14 is a block diagram of a cloud-based environment in several embodiments. Each component can be implemented using any preferred combination of known or to be known hardware and software. For example, each component may include a physical computer server or a virtual machine. Such virtual machines may reside in one or more public clouds that provide self-service and immediate provisioning, auto-scaling, security, compliance, and identity management features.

[0071] The application server 1410 runs one or more applications that issue database queries. The applications are accessible by users (not shown) who request information from the applications, thereby triggering the issuance of database queries. The application server 1410 sends queries to the database compute node 1420 that runs a database instance (e.g., an index server). Based on those queries, the database instance commands CRUD operations on the data stored in the storage nodes 1432, 1434, and 1436 of the storage layer 1430.

[0072] The data is stored in the storage nodes 1432, 1434, and 1436 of the storage layer 1430 in one of the formats described above. For example, the data of a given table may be stored in shards distributed through storage nodes 1432, 1434, and 1436, and each shard may be replicated across multiple storage nodes. The data may contain data from two or more tenants, in which case each shard of a given tenant is identified in the metadata by a tenant identifier, a table identifier, and a key range. The database compute node 1420 uses the metadata to determine the location of the shard for which operation is desired.

[0073] The aforementioned diagrams represent logical architectures for distributed processes in several embodiments, and actual implementations may include more or different components arranged in other ways. Other topologies may be used with other embodiments. Furthermore, each component or device described herein may be implemented by any number of devices communicating over any number of other public and / or private networks. Two or more such computing devices may be located remotely from each other and may communicate with each other over any known type of network and / or private connection. Each component or device may comprise any number of hardware and / or software elements suitable for providing the functions described herein and any other functions. For example, any computing device used in an implementation of the system in some embodiments may include a processor for executing program code so that the computing device operates as described herein.

[0074] All systems and processes discussed herein may be embodied in program code stored on one or more non-temporary computer-readable media. Such media may include, for example, hard disks, DVD-ROMs, flash devices, magnetic tapes, and solid-state random-access memory (RAM) or read-only memory (ROM) storage units. Embodiments are therefore not limited to any particular combination of hardware and software.

[0075] The embodiments described herein are for illustrative purposes only. Those skilled in the art will recognize that other embodiments can be put into practice by modifications and alterations to the embodiments described above. [Explanation of symbols]

[0076] 110,112,114,116 clients 120 Computation Layers 122,124,126,128 Computing Nodes 130 Memory Layers 132,134,136,138 Memory nodes 210 Database Tables 212,214,216 shards 220 database tables 222,224 shards 230 Database Tables 232,234,234a,234b,236,238,238a,238b shards 310 Database Computation Frontend 320 Topology Manager 325 Shard Metadata 400 processes 510 storage nodes 710 Database Tables 712,714 shards 910 Storage Nodes 1310,1320 Database Computation Frontend 1410 Application Server 1420 database compute nodes 1430 Memory Layer 1432, 1434, 1436 memory nodes

Claims

1. A first storage node comprising a first processing unit and a first data storage device, wherein the first data storage device stores a first copy of a first shard of a first database table and a first copy of a third shard of the first database table, the first shard includes a first key range of the first database table, and the third shard includes a third key range of the first database table, and the first storage node comprises a first storage node comprising a first processing unit and a first data storage device, wherein the first data storage device stores a first copy of a first shard of a first database table and a first copy of a third shard of the first database table, and the third shard includes a third key range of the first database table, A second storage node comprising a second processing unit and a second data storage device, wherein the second data storage device stores a second copy of the first shard of the first database table, A third storage node comprising a third processing unit and a third data storage device, wherein the third data storage device stores a third copy of the first shard of the first database table and a second copy of the third shard of the first database table. A fourth storage node comprising a fourth processing unit and a fourth data storage device, wherein the fourth data storage device stores a first copy of a second shard of the first database table, and the second shard includes a second key range of the first database table, and the fourth storage node comprises a fourth storage node, A fifth storage node comprising a fifth processing unit and a fifth data storage device, wherein the fifth data storage device stores a second copy of the second shard of the first database table. A sixth storage node comprising a sixth processing unit and a sixth data storage device, wherein the sixth data storage device stores a third copy of the second shard of the first database table, A manager node configured to, in response to the detection of a failure in the first storage node, instruct the second storage node to copy the second copy of the first shard to one of the fourth, fifth, or sixth storage nodes, and to instruct the third storage node to copy the second copy of the third shard to one of the fourth, fifth, or sixth storage nodes, A system that includes this.

2. The first database table is a table of the first tenant conforming to the first tenant schema, the first data storage device stores a first replica of a third shard of the second database table, the third shard includes a third key range of the second database table, the second database table is a table of the second tenant conforming to the second tenant schema, and the system is A seventh storage node comprising a seventh processing unit and a seventh data storage device, wherein the seventh data storage device stores a second copy of the third shard of the second database table, An eighth storage node comprising an eighth processing unit and an eighth data storage device, wherein the eighth data storage device stores a third copy of the third shard of the second database table. The system according to claim 1, further comprising:

3. A first tenant root shard including first metadata that associates the first shard with the first tenant, the first database table, the first key range, the first storage node, the second storage node, and the third storage node, A second tenant root shard which includes second metadata that associates the second shard with the second tenant, the second database table, the third key range, the fourth storage node, the fifth storage node, and the sixth storage node, and The system according to claim 2, further comprising:

4. The system according to claim 3, wherein the system stores a plurality of copies of the first tenant root shard and a plurality of copies of the second tenant root shard.

5. A root shard containing metadata that associates the first shard with the first database table, the first key range, the first storage node, the second storage node, and the third storage node. The system according to claim 1, further comprising:

6. The system according to claim 5, wherein the system stores a plurality of copies of the root shard.

7. A computer implementation method, A step of determining a first shard of a first database table, a second shard of the first database table, and a third shard of the first database table, wherein the first shard includes a first key range of the first database table, the second shard includes a second key range of the first database table, and the third shard includes a third key range of the first database table. The steps include storing a first copy of the first shard of the first database table and a first copy of the third shard of the first database table in a first storage node, The steps include storing a second copy of the first shard of the first database table in a second storage node, The steps include storing a third copy of the first shard of the first database table and a second copy of the third shard of the first database table in a third storage node, The steps include storing a first copy of the second shard of the first database table in a fourth storage node, The steps include storing a second copy of the second shard of the first database table in a fifth storage node, The steps include storing a third copy of the second shard of the first database table in a sixth storage node, and Steps include: in response to the detection of a failure in the first storage node, instructing the second storage node to copy the second copy of the first shard to one of the fourth storage node, the fifth storage node, or the sixth storage node, and instructing the third storage node to copy the second copy of the third shard to one of the fourth storage node, the fifth storage node, or the sixth storage node; Methods that include...

8. The first database table is a table of the first tenant conforming to the first tenant schema, and the method is A step of determining a third shard of a second database table, wherein the second database table is a second tenant table conforming to a second tenant schema, and the third shard includes a third key range of the second database table. The steps include storing a first copy of the third shard of the second database table in the first storage node, The steps include storing a second copy of the third shard of the second database table in a seventh storage node, The steps include storing a third copy of the third shard of the second database table in an eighth storage node, and The method according to claim 7, further comprising:

9. The steps include storing a first tenant root shard which includes first metadata that associates the first shard with the first tenant, the first database table, the first key range, the first storage node, the second storage node, and the third storage node, The steps include storing a second tenant root shard which includes second metadata that associates the second shard with the second tenant, the second database table, the third key range, the fourth storage node, the fifth storage node, and the sixth storage node, and The method according to claim 8, further comprising:

10. Steps to store multiple copies of the first tenant root shard and multiple copies of the second tenant root shard. The method according to claim 9, further comprising:

11. The step of storing a root shard which includes metadata that associates the first shard with the first database table, the first key range, the first storage node, the second storage node, and the third storage node. The method according to claim 7, further comprising:

12. Steps to store multiple copies of the root shard. The method according to claim 11, further comprising:

Images