Cloud database storage system

Abstract

Systems and methods include storing database table fragments on a plurality of storage nodes. Each fragment may be stored as a plurality of copies, wherein each copy of a given fragment is stored on a different storage node. In the event of a failure of a given storage node, all fragment copies stored on the given storage node are copied from other storage nodes to new or existing storage nodes. A storage node may store fragments of database tables associated with a plurality of tenants.

Description

Background Art

[0001] Modern organizations often utilize a system landscape consisting of distributed computing systems that provide a variety of computing services. For example, an organization may deploy services within computing systems located in on-premises data centers (which themselves may be located in different geographic locations) and within data centers provided by one or more platform as a service (PaaS) providers. The computing systems may include cloud-based computing systems (e.g., using on-demand scalable virtual machines to provide services).

[0002] In the Software as a Service (SaaS) model, a service provider deploys a service (e.g., a database) onto a cloud-based computing system (which in some cases may be provided by a separate PaaS provider). The service can be used by the customer in exchange for paying a fee to the service provider. The customer is not involved in the operation, management, and scaling of the service, all of which is performed by the service provider.

[0003] In the case of databases provided as services, the service provider attempts to ensure scalability on both the storage and compute layers of the database. Some deployments decompose the storage and compute layers by utilizing separate node clusters for each layer. This decomposition (in theory) allows independent scaling of the layers.

[0004] The storage layer of such a database service may consist of many storage nodes, each of which contains local disk space and computing resources (e.g., RAM and CPU) to execute storage node processes. Typically, database services are single-tenant systems, and all storage nodes of the storage layer combine to store a single database. Since each tenant requires a separate storage layer, these systems are cost-prohibitive for small tenants. For larger tenants, the elasticity of such systems is not suitable because the addition or removal of individual storage nodes cannot provide appropriately fine-grained expansion increments.

[0005] The failure of a database instance results in a related failure of all tenants hosted on that database instance, known as a large "blast radius". To provide high availability, traditional database systems continuously replicate the entire primary database instance to a separately running failover instance. If the primary instance fails, users are switched to the failover instance. While such systems reduce perceived downtime to a few seconds, they require additional expensive infrastructure (i.e., failover instances) and considerable time to restore full redundancy (i.e., via copying the entire database from the failover instance to the new instance).

[0006] To provide adequate performance, network resources are provisioned to provide appropriate bandwidth for data-intensive operations such as recovery and expansion. As a result, resources are over-provisioned for typical loads and result in wasted costs.

[0007] What is needed is a system that reduces the blast radius of a storage node failure and the time and resources required for full redundancy following a storage node failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A distributed database system according to some embodiments is shown.

[0009] Figure 2 The storage of database table fragments on multiple storage nodes is shown according to some embodiments.

[0010] Figure 3 Access to a database table fragment is shown in accordance with some embodiments.

[0011] Figure 4 is a flow diagram of a process for accessing database table fragments on a storage node according to some embodiments.

[0012] Figure 5a to Figure 5c A scale-out process according to some embodiments is shown.

[0013] Figures 6a to 6c A scale-in process according to some embodiments is shown.

[0014] Figure 7 The storage of database table shard replicas on multiple storage nodes is shown according to some embodiments.

[0015] Figure 8 Access to a database table fragment is shown in accordance with some embodiments.

[0016] Figures 9a to 9c An expansion process using database table shard replicas according to some embodiments is shown.

[0017] Figures 10a to 10c A shrink process using database table shard copies is shown in accordance with some embodiments.

[0018] Figures 11a to 11c A recovery process using database table fragment copies according to some embodiments is shown.

[0019] Fig.12 The storage of database table shard replicas for multiple tenants on multiple storage nodes is shown according to some embodiments.

[0020] Fig.13 Access to database table shards for multiple tenants is illustrated in accordance with some embodiments.

[0021] Fig.14 is a block diagram of a cloud-based virtual machine providing database services according to some embodiments. DETAILED DESCRIPTION

[0022] The following description is provided to enable any person skilled in the art to make and use the described embodiments. However, various modifications will still be apparent to those skilled in the art.

[0023] According to some embodiments, the data of each tenant of the database system is logically split into shards, and the shards of a given tenant are distributed across multiple storage nodes. The shards are large enough to amortize the cost of operations on individual data items by allowing batch request processing, and small enough to allow reasonably fast operations across the shards (e.g., for backup / restore, shard relocation). In some embodiments, the shards can be substantially equal in size, and in the range of 1GB-10GB per shard.

[0024] Over time, data added to a shard may cause the shard to grow larger than the specified maximum shard size. In this case, the shard may be split into two shards, with one of the two shards remaining on the original storage node and the other shard being migrated to another storage node. Similarly, if the size of a shard shrinks below a specified minimum shard size, the shard may be merged with shards associated with an adjacent key range, which may be located on a different storage node. Thus, embodiments may provide fine-grained elastic space allocation for each tenant by maintaining as many shards as required for a particular tenant.

[0025] Embodiments may also decorrelate storage node failures. Specifically, a failure of a storage node limits access to shards stored on the storage node, but does not limit access to other shards stored on other storage nodes. Assuming that each tenant's shards are relatively evenly distributed across a large number of storage nodes, a storage node failure will only render a small portion of a given tenant's data unavailable. Given the recovery options described herein, this unavailability may be short-lived.

[0026] In addition, because the tenant's data is distributed across storage nodes, in some embodiments, the computing nodes can access the data in parallel. For example, instead of scanning a large table located on a single storage node, an embodiment can allow a computing node to query hundreds of storage nodes including fragments of the table at the same time. In some scenarios, this parallelization can provide at least an order of magnitude improvement in performance.

[0027] To achieve high availability, each shard can be replicated multiple times between different storage nodes. The relatively small size of the shards and the large node-to-node bandwidth allow this replication to occur quickly. In the event of a storage node failure or outage, the data of all shards stored by that storage node remains accessible via copies of those shards stored on other storage nodes.

[0028] In addition, since copies of shards are typically distributed throughout the remaining storage system, the surviving nodes are able to generate new copies to restore full redundancy. For example, a copy of each shard can be transferred from its storage node to a new storage node. When the shards of a database are distributed across many storage nodes, the re-replication of data used to restore full redundancy of the database can be performed in parallel across all nodes (i.e., using n:n communication instead of 1:1 communication). Therefore, in some embodiments, the restoration of full redundancy can be completed in seconds. In contrast, other systems require redistributing the entire database and / or replicating the data of an entire node over the network to adjust the failed node.

[0029] Allocating disk space to tenants in small units and sharing storage infrastructure across tenants significantly reduces the fixed cost per tenant (i.e., down to the cost of the smallest possible shard size) compared to systems where each tenant requires at least one separate virtual machine. Some embodiments employ a stateless compute layer, where the compute layer can be shared across tenants, further driving the incremental cost for small tenants to near zero.

[0030] Figure 1 A system according to some embodiments is shown. Figure 1 The components shown in the can be implemented using any suitable combination of known or becoming known computing hardware and / or software. Such a combination may include a cloud-based implementation in which computing resources are virtualized and elastically allocated. In some embodiments, two or more components are implemented by a single computing device.

[0031] Clients 110-116 may include computing systems that execute applications or services that issue database queries. Clients 110-116 may issue database queries in response to requests input thereto by users (not shown). Computing layer 120 receives the queries (e.g., via an API provided by computing layer 120) and operates in conjunction with data stored in storage layer 130 to provide corresponding query results to clients 110-116.

[0032] The computing layer 120 includes a plurality of computing nodes 122-128. Similarly, the storage layer 130 includes a plurality of storage nodes 132-138. Each of the nodes 122-128 and 132-138 includes at least one processing unit (e.g., a CPU), a random access memory, and at least one data storage device (e.g., an NVMe SSD), and may include a single computer server, a group of computer servers, a virtual machine, and any other combination known or becoming known. In some embodiments, the computing layer 120 is completely stateless and boots over a network, so the computing nodes 122-128 do not need to include storage devices.

[0033] In some embodiments, the computing layer 120 and the storage layer 130 are implemented by a Kubernetes cluster. Kubernetes is an open source system for automating the deployment, scaling, and management of containerized applications. A Kubernetes cluster consists of many nodes, which in turn may include physical servers and / or virtual machines. Nodes execute applications within software containers. One or more containers may be grouped into pods, where each container in a given pod shares the same computing resources and local network.

[0034] According to some embodiments, Figure 1 Represents a single-tenant database system. Storage nodes 132-138 store data associated with a single tenant, and compute node 122 processes queries received from clients 110-116 operated by users of the single tenant. As described above, each of storage nodes 132-138 stores a respective shard of data associated with a single tenant in order to provide parallelization and elasticity.

[0035] Figure 1 The multi-tenant database system in some embodiments may be represented. In such an embodiment, the storage nodes 132-138 store data associated with multiple tenants. The data is stored in fragments distributed across the storage nodes 132-138. Each of the computing nodes 122-128 can execute a single tenant database instance specific to a tenant. Therefore, in the case where the client 110 is being accessed by a user of the first tenant, a query issued by the client 110 is directed to one of the computing nodes 122-128 associated with the first tenant.

[0036] In yet another example, storage nodes 132-138 store data associated with multiple tenants, and each of compute nodes 122-128 executes a multi-tenant database instance. Thus, any one of compute nodes 122-128 is capable of processing queries received from clients 110-116 regardless of the tenant whose data is being queried.

[0037] Figure 2 The storage of database table fragments of a single tenant on multiple storage nodes according to some embodiments is shown. Database tables 210, 220, and 230 are logical representations of data associated with a given database tenant. Database tables 210, 220, and 230 may conform to a specific schema known in the art. According to some embodiments, each row of each of database tables 210, 220, and 230 includes a value describing an instance of an object. Embodiments are not limited to any type of object.

[0038] Each of database tables 210, 220, and 230 is split into multiple fragments. Specifically, database table 210 is split into fragments 212, 214, and 216, database table 220 is split into fragments 222 and 224, and database table 230 is split into fragments 232, 234, 236, and 238. Each fragment is associated with a key range. In some embodiments, the fragments can be substantially equal in size, and the size of each fragment can be between 1GB and 10GB. Embodiments include any algorithm for splitting a table into fragments and for defining the key range of a fragment. In one embodiment, storage layer 130 is a key / value store, and the key range of a fragment is the key range of the key stored in the fragment. According to some embodiments, a fragment may include data of more than one database table.

[0039] According to the present example, the shards are stored across storage nodes 132, 134, and 136 of storage tier 130. Storage node 132 stores shards 212, 214, 216, and 236, storage node 134 stores shards 222, 232, and 238, and storage node 136 stores shards 224 and 234. It is assumed that storage node 138 and any other storage nodes of storage tier 130 store shards of other, not shown, database tables of the tenant.

[0040] The distribution of shards across the storage nodes of the storage layer 130 may be based on any suitable algorithm. Since the storage node 132 stores all shards 212, 214, 216 of the table 210, the failure of the node 132 causes all data of the table 210 to be inaccessible. In some embodiments, each shard of the table is preferably stored on a different node (e.g., shards 222 and 224 of the table 220), so that the failure of a node causes the inaccessibility of at most one shard of the table. Some embodiments may store at most one shard of a given tenant per storage node.

[0041] Figure 3 3. Access to database table fragments according to some embodiments is shown. The database computing front end 310 may include a database instance executed on a computing node as described above. The database computing front end 310 may need to access database table data in response to a query received from a client. However, due to the fragmented storage described above, the database computing front end 310 must initially determine the location of the fragment containing the desired data in order to direct its request to the appropriate storage node.

[0042] The topology manager 320 consists of one or more nodes that store shard metadata 325. The shard metadata 325 can be associated with a well-known ID in the system and stored in one or more storage nodes 132 to 136 in the same manner as other shards stored herein. According to some embodiments, the shard metadata 325 specifies the storage location of the database shards. For example, the shard metadata 325 can identify each shard by a table id and a key range. In addition, each shard is also associated with an identifier of the storage node where it is stored. Therefore, the database computing front end 310 queries the topology manager 320 for the location of the shards associated with the query (e.g., all shards of a given table), and in response receives an indication of the storage node storing the desired shard. The database computing front end 310 then issues database operations to the nodes storing the shards. As described above, if the shards are located on more than one storage node, the operations can be advantageously performed in parallel.

[0043] In some embodiments, the creation of a tenant results in the creation of a single shard on a storage node. This shard can be considered a root shard, which (rather than the topology manager 320) includes shard metadata 325 associated with the tenant. Thus, in response to a received query, the database compute front end 310 queries the topology manager for the location of the root shard of the tenant associated with the query, retrieves the shard metadata from the root shard, and based thereon determines a storage node to store the desired shard.

[0044] Figure 4 4 is a flow chart of a process 400 for accessing database table fragments on a storage node according to some embodiments. Any suitable combination of hardware and software may be used to perform process 400 and other processes described herein. The software program code embodying these processes may be stored by any non-transitory tangible medium, including a fixed disk, a volatile or non-volatile random access memory, a DVD, a flash drive, or a tape, and executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Such processors, processor cores, and processor threads may be implemented by virtual machines provided in a cloud-based architecture. Embodiments are not limited to the examples described below.

[0045] Process 400 assumes prior storage of database table fragments on multiple storage nodes, as described above. At S410, a request for the location of a database table fragment is received. The request may identify the fragment using a database table identifier and one or more primary keys (or primary key ranges) of the identified table. The request is received from a database front end, which may be executed on a compute node of a compute layer as described above. The fragment may include data required to service a query received by the database front end from a client application.

[0046] In response to the request, a storage node storing the shard is determined based on the stored metadata. According to some embodiments, the stored metadata is shard metadata stored by the topology manager as described above. Therefore, such a topology manager can receive the request at S410 and determine the storage node based on the request and the shard metadata at S420. In one example, the request includes a database table identifier and a primary key range. The topology manager can determine the shard including the primary key range of the identified table based on the shard metadata at S420, and also determine the storage node storing the shard based on the shard metadata. In some embodiments, S420 includes determining the root shard of the tenant associated with the query and determining the storage node storing the desired shard based on the metadata stored in the root shard. At S430, an identifier of the determined storage node is provided to the requester.

[0047] It may be determined at S420 that the primary key range spans two or more fragments, in which case the storage nodes storing the two or more fragments are determined at S420. In another example, the request includes only a table identifier and does not include a primary key or a key range. Thus, S420 includes determining all fragments of the identified table and their corresponding storage nodes. The received request may include any number of table identifiers and associated key ranges.

[0048] At S440, a database operation for the shard is received at the database node storing the shard. The database operation may be issued by the database front end from which the request was received at S410, and may utilize the identifier provided at S430. If more than one storage node has been identified, the database operation may be issued to each other node and received at each other node at S440, thereby enabling operations to be performed in parallel by more than one storage node.

[0049] Figure 5a to Figure 5c An expansion process according to some embodiments is shown. Figure 5a The storage nodes 132, 134 and 136 store corresponding fragments, some of which have sizes relatively large. Figure 3 Thus, a determination may be made manually or automatically (based on resource consumption parameters) to reduce the amount of storage consumed by one of the storage nodes by adding another storage node to the corresponding storage tier. According to some embodiments, such a determination may be performed by a topology manager. A determination may also or alternatively be made to split shards 234 and 238 because their size exceeds a specified maximum shard size.

[0050] Figure 5b The addition of storage node 510 to the storage tier is shown. Figure 5bAlso shown is the movement of shards from each of storage nodes 132, 134, and 136 to storage node 510. This movement may be intended to achieve load balancing, which reduces the memory consumption of each node.

[0051] Figure 5c The resulting storage layer is shown. As shown, fragment 236 has been moved from storage node 132 to storage node 510. Fragment 238 has been divided into fragments 238a and 238b, and fragment 238b has been moved from storage node 134 to storage node 510. Similarly, fragment 234 has been divided into fragments 234a and 234b, and fragment 234b has been moved from storage node 136 to storage node 510. Embodiments are not limited to dividing fragments into fragments of equal size.

[0052] Advantageously, the shards moved to node 510 each move over a different (and substantially independent) node-to-node network connection. Thus, the movement of the shards can occur significantly faster than if all three shards traveled from a single node to node 510. Furthermore, the movement requires only 1 / n of the bandwidth of the node from which the shard is moved. Thus, the impact on node performance and system response time is minimal.

[0053] The partitioning of shards and the movement of the resulting shards can occur with or without adding storage nodes. The addition of storage nodes and the movement of shards thereto can also occur with or without partitioning any shards.

[0054] Figures 6a to 6c The zoom-out process is shown according to some embodiments. Figure 6a Storage nodes 132, 134, 136, and 138 are shown, some of which are smaller than Figure 5c Thus, it may be determined manually or automatically (again, based on resource consumption parameters) to reduce the number of storage nodes of the corresponding storage layer. According to some embodiments, this determination may be performed by a topology manager.

[0055] In this example, it is determined that storage node 136 is deactivated. Therefore, Figure 6b The movement of shards of storage node 136 to storage nodes 132 and 510 is shown. In some embodiments, the storage nodes to which the shards are moved are determined in order to achieve load balancing.

[0056] Figure 6cThe resulting storage layer is shown, where shard 224 has been moved from storage node 136 to storage node 132, and shard 234a has been moved from storage node 136 to storage node 510. Storage node 136 no longer stores any shards, and can therefore be decommissioned. According to some embodiments, shrinking may also or alternatively include merging two or more small shards into one shard, and placing the one shard on a storage node based on load balancing and resource consumption considerations. According to some embodiments, due to the key range associated with each shard, shards should only be merged with shards associated with adjacent key ranges.

[0057] Figure 7 FIG. 4 shows the storage of database table fragment copies of a single tenant on multiple storage nodes according to some embodiments. Figure 2 As described, database tables 210 , 220 , and 230 are logical representations of data associated with a given database tenant and are split into multiple shards associated with different key ranges.

[0058] like Figure 7 As shown, storage layer 130 stores multiple copies of each of shards 212, 214, 216, 222, and 224. Replication can be performed in any suitable manner, such as by using a consensus protocol, employing erasure coding across replicas, etc., each of which can present different advantages and disadvantages based on the use case. The distribution of replicas across storage nodes of storage layer 130 can be based on any suitable algorithm that can provide load balancing and high availability.

[0059] Despite Figure 7 , but embodiments may utilize any number of replicas. In the case of a protocol that replicates each shard across three different storage nodes, if one of the three storage nodes serving that shard fails or is disabled, the data for the particular shard remains fully accessible. If the replication factor is set to greater than 3, the system continues to operate normally even in the event of multiple node failures (e.g., two node failures using a replication factor of 5, or k node failures with a replication factor of 2k+1 in general).

[0060] Figure 7 Reflects the consensus protocol where each shard is represented by a leader replica and two follower replicas. The leader replica is determined by the leader election algorithm and is Figure 7. All operations on a given shard are directed to the storage node storing the leader copy of the shard, and changes to it are replicated to the follower copies of the shard. If the node containing the leader copy of the shard fails, one of the remaining follower copies of the shard is designated as the leader copy, and the new leader copy begins replicating its changes to the remaining follower copies. In addition, as will be described below, a new follower copy is quickly instantiated on another (or new) storage node to re-establish the desired consensus protocol.

[0061] Figure 8 Access to leading replicas of shards is shown according to some embodiments. The database compute front end 310 determines the location of the leading replica of the shard containing the desired data in order to direct its request to the appropriate storage node. The shard metadata 325 of the topology manager 320 specifies the storage location of all replicas of each shard and annotates the current leading replica of each shard. If the storage node including the leading replica of the shard fails or is disabled, a new leading replica of the shard is elected and the shard metadata 325 is updated accordingly. Any follower replicas stored on the failed / disabled storage node are removed from the shard metadata 325. The shard metadata 325 also specifies the table ID and key range of each shard.

[0062] Thus, the database compute front end 310 queries the topology manager 320 for the location of the shards associated with the query (e.g., all shards of a given table), and receives therefrom the identifier of the storage node storing the leading replica of the desired shard. The database compute front end 310 then issues the database operation to the node on which the leading replica shard is stored. Again, if the shards are located on more than one storage node, the operations can be advantageously performed in parallel.

[0063] The database computing front end 310 may also receive an identifier of a storage node where a follower replica of the desired shard is stored. Thus, if an operation on the identified leading replica fails, the database computing front end 310 may seamlessly retry the operation on one of the corresponding follower replicas stored on a different storage node.

[0064] In some embodiments, the root shard replaces the topology manager 320 to store the shard metadata 325, and the topology manager 320 only stores the location of the root shard. The root shard can also be stored in multiple copies between the storage nodes 132-138, in which case the topology manager 320 stores the location of each copy.

[0065] Figures 9a to 9c An expansion process using database table shard replicas according to some embodiments is shown. Figure 9a The storage nodes 132, 134, 136 and 138 store Figure 7 and 8It will be assumed that a manual or automatic determination is made to add another storage node to the corresponding storage tier, for example to reduce the amount of storage consumed by one of the storage nodes or to provide improved availability.

[0066] Figure 9b 9 shows the addition of storage node 910 to the storage layer. Figure 9b As shown, fragments are moved from each of storage nodes 132 , 134 , and 138 to storage node 910 . Fig.9c The resulting storage layer is shown. Shard 222 has been moved from storage node 132 to storage node 910, shard 212 has been moved from storage node 134 to storage node 910, and shard 214 has been moved from storage node 138 to storage node 910. In all of the examples described herein, shard metadata 325 is updated to reflect the new replica storage locations resulting from the movement of shards between storage nodes.

[0067] The shards are moved to node 910 via different (and substantially independent) node-to-node network connections (i.e., 132 to 910, 134 to 910, and 138 to 910). By replicating one shard instead of three, nodes 132, 134, and 138 are each much less burdened than other implementations.

[0068] Figures 10a to 10c A shrink process using database table shard copies is shown in accordance with some embodiments. Fig.10a Reflects on Fig.9c Described Storage Tiers. It will be assumed that a manual or automatic determination to decommission a storage node 136 may be made based on underutilization of one or more storage nodes of the depicted storage tier.

[0069] Fig.10b The movement of shards of storage node 136 to storage nodes 132, 134, and 138 is shown. In some embodiments, the shards to be moved and the storage nodes to which the shards are moved may be determined based on a load balancing and / or resource consumption algorithm. Fig.10c The resulting storage layer is shown, where shard 224 has been moved from storage node 136 to storage node 132, shard 214 has been moved from storage node 136 to storage node 134, and shard 212 has been moved from storage node 136 to storage node 138. Since storage node 136 no longer stores any shards, it can be decommissioned. The shard metadata 325 is then updated to reflect the new replica storage locations resulting from the above-described movement of shards between storage nodes.

[0070] Figures 11a to 11c A recovery process using database table fragment copies according to some embodiments is shown. Fig.11a Reflects on Fig.10a The storage layer described above, however, assumes in this case that storage node 136 unexpectedly fails. In some implementations, the decommissioning of node 136 can also be as described below with respect to Figures 11a-11c Proceed as described.

[0071] Upon detecting a failure, the topology manager 320 accesses the shard metadata 325 to determine the shards stored on the storage node 136. Next, the topology manager 320 identifies which of the remaining storage nodes stores a copy of the shards stored on the storage node 136 based on the shard metadata 325. The topology manager 320 then instructs the storage layer to replicate a copy of each shard stored on the storage node 136 from the remaining storage nodes to a different remaining storage node.

[0072] Fig.11b It is shown that a copy of each shard stored on the failed storage node 136 is replicated to other storage nodes. For example, the shard metadata 325 indicates that the failed storage node 136 stores copies of shards 224, 214, and 212, and that the copy of shard 214 is the leading copy. The shard metadata 325 also indicates that a copy of shard 224 is located on nodes 134 and 138, a copy of shard 214 is located on nodes 132 and 910, and a copy of shard 212 is located on nodes 132 and 910. Therefore, Fig.11b Shard 224 is shown being replicated from node 138 to node 132, shard 214 is replicated from node 910 to node 134, and shard 212 is replicated from node 910 to node 138. As described above, replication between different sets of nodes allows each replication operation to proceed in parallel and use the full available node-to-node bandwidth. Fig.11c The resulting storage tier after the copy operation is completed is shown.

[0073] According to some embodiments, the specific storage nodes from which the shards are replicated and to which they are copied can be determined based on load balancing and / or resource consumption algorithms. The shard metadata 325 is updated to reflect the new replica storage location. The update can also include electing a replica of the shard 214 stored in the node 134 as the leading replica, and indicating the election in the shard metadata 325.

[0074] Fig.12The storage of database table shard copies of multiple tenants on multiple storage nodes is shown according to some embodiments. Database table 210 includes data for Tenant 1, and database table 710 includes data for Tenant 2. Database table 210 conforms to a schema associated with Tenant 1, and database table 710 conforms to a schema associated with Tenant 2. Each tenant may be associated with many other database tables including their respective data. As is known in the art, Tenant 1's data is intended to be accessible only to users of Tenant 1, and Tenant 2's data is intended to be accessible only to users of Tenant 2.

[0075] In the example shown, table 210 is divided into shards 212, 214, and 216, and table 710 is divided into shards 712 and 714. Storage layer 130 stores multiple copies of each of shards 212, 214, 216, 712, and 714, as described above with respect to Figure 7 As described. Fig.12 Represents a consensus protocol consisting of a leading replica (ie, represented by a distinct outline) and two follower replicas of each shard. The replicas may be distributed across the storage nodes of the storage layer 130 based on any suitable algorithm for providing load balancing and high availability. Fig.12 The expansion, reduction, and failure recovery of the nodes shown in Figure 9a-9c , 10a-10c and 11a-11c.

[0076] Fig.13 Access to database table shards for multiple tenants is illustrated in accordance with some embodiments.

[0077] Database computing front end 1310 serves users of tenant 1, and database computing front end 1320 serves users of tenant 2. Database computing front end 1310 may include a single-tenant database instance executed on a single computing node of the computing layer, and database computing front end 1320 may include a single-tenant database instance executed on another single computing node of the computing layer. In some embodiments, database computing front end 1310 and database computing front end 1320 are executed on the same computing node.

[0078] The topology manager 320 may operate as described above. Also as described above, the shard metadata 325 may associate each shard with a table identifier, a key range, the location of each replica, and an identifier of a leading replica. However, because the tables 210 and 710 conform to different schemas, the table identifier and key range of the table of Tenant 1 may overlap / conflict with the table identifier and key range of the table of Tenant 2. Therefore, the shard metadata 325 also associates each shard with a tenant identifier. If the database compute front ends 1310 and 1320 are single-tenant database front ends, the tenant identifier may be an identifier of the container in which the front end is executing.

[0079] In response to the received query, the database computing front end 1310 (or front end 1320) queries the topology manager 320 for the location of the shard associated with the query. The query of the topology manager 320 includes a tenant identifier, a table identifier, and a key range (e.g., as a concatenated string). The topology manager 320 determines the identifier of the storage node storing the leading replica of the desired shard based on the query and the shard metadata 325 received from the database computing front end 1310 (or front end 1320). The database computing front end 1310 (or front end 1320) then issues database operations to the node(s) on which the leading replica shard(s) are stored. If the shard is located on more than one storage node, the storage nodes can perform these operations in parallel.

[0080] In some embodiments, each tenant is associated with a root shard including the tenant's shard metadata, and the topology manager 320 stores the location of each tenant's root shard. Thus, the database computing front end 1310 or 1320 can query the topology manager 320 for a storage node storing a given tenant's root shard, read the metadata of the root shard from the storage node, and determine the location of the desired shard based on the metadata. Each root shard can also be stored in multiple copies between the storage nodes 132-138, in which case the topology manager 320 stores the location of each copy of each root shard.

[0081] Some embodiments include a multi-tenant database compute front end that serves users of Tenant 1 and users of Tenant 2. The multi-tenant database compute front end accesses the shard replicas, as described above with respect to both database compute front end 1310 and database compute front end 1320, querying the topology manager 320 using the appropriate tenant identifier.

[0082] Fig.14 is a block diagram of a cloud-based environment according to some embodiments. Each component may be implemented using any suitable combination of hardware and software that is or becomes known. For example, each component may include a physical computer server or a virtual machine. Such virtual machines may reside in one or more public clouds, providing self-service and immediate provisioning, automatic scaling, security, compliance, and identity management features.

[0083] Application server 1410 executes one or more applications that issue database queries. The application can be accessed by a user (not shown) who requests information from the application, thereby triggering the issuance of a database query. Application server 1410 sends the query to database computing node 1420, which executes a database instance (e.g., indexserver). Based on the query, the database instance instructs CRUD operations on the data stored in storage nodes 1432, 1434, and 1436 of storage layer 1430.

[0084] The data is stored in the storage nodes 1432, 1434, and 1436 of the storage layer 1430 in any of the above-described ways. For example, the data for a given table may be stored in fragments distributed throughout the storage nodes 1432, 1434, and 1436, and each fragment may be replicated across multiple storage nodes. The data may include data for more than one tenant, where each fragment of a given tenant is identified in the metadata by a tenant identifier, a table identifier, and a key range. The database compute node 1420 uses the metadata to determine the location of the fragment for which the operation is desired.

[0085] The foregoing diagrams represent the logical architecture for describing the process according to some embodiments, and the actual implementation may include more or different components arranged in other ways. Other topologies may be used in combination with other embodiments. In addition, each component or device described herein may be implemented by any number of devices communicating via any number of other public and / or private networks. Two or more such computing devices may be located remotely from each other and may communicate with each other via any known network mode and / or dedicated connection. Each component or device may include any number of hardware and / or software elements suitable for providing the functions described herein and any other functions. For example, any computing device used in the implementation of the system according to some embodiments may include a processor to execute program code so that the computing device operates as described herein.

[0086] All systems and processes discussed herein may be embodied in program code stored on one or more non-transitory computer-readable media. Such media may include, for example, hard disks, DVD-ROMs, flash drives, magnetic tapes, and solid-state random access memory (RAM) or read-only memory (ROM) storage units. Therefore, embodiments are not limited to any particular combination of hardware and software.

[0087] The embodiments described herein are for illustration purposes only. Those skilled in the art will appreciate that other embodiments may be practiced through modifications and alterations to the embodiments described above.

Claims

1. A system comprising: a first storage node, the first storage node comprising a first processing unit and a first data storage device, the first data storage device storing a first copy of a first shard of a first database table, and the first shard comprising a first key range of the first database table; a second storage node, the second storage node comprising a second processing unit and a second data storage device, the second data storage device storing a second copy of the first fragment of the first database table; a third storage node, the third storage node comprising a third processing unit and a third data storage device, the third data storage device storing a third copy of the first fragment of the first database table; a fourth storage node, the fourth storage node comprising a fourth processing unit and a fourth data storage device, the fourth data storage device storing a first copy of a second shard of the first database table, and the second shard comprising a second key range of the first database table; a fifth storage node, the fifth storage node comprising a fifth processing unit and a fifth data storage device, the fifth data storage device storing a second copy of the second fragment of the first database table; as well as A sixth storage node, the sixth storage node comprising a sixth processing unit and a sixth data storage device, the sixth data storage device storing a third copy of the second fragment of the first database table.

2. The system of claim 1, wherein the first database table is a table of a first tenant that conforms to a first tenant schema, the first data storage device stores a first copy of a third shard of a second database table, the third shard includes a third key range of the second database table, and the second database table is a table of a second tenant that conforms to a second tenant schema, further comprising: a seventh storage node, comprising a seventh processing unit and a seventh data storage device, the seventh data storage device storing a second copy of the third shard of the second database table; as well as An eighth storage node includes an eighth processing unit and an eighth data storage device, wherein the eighth data storage device stores a third copy of the third fragment of the second database table.

3. The system according to claim 2, further comprising: a first tenant root shard, the first tenant root shard comprising first metadata associating the first shard with the first tenant, the first database table, the first key range, the first storage node, the second storage node, and the third storage node; as well as A second tenant root shard, the second tenant root shard comprising second metadata associating the second shard with the second tenant, the second database table, the third key range, the fourth storage node, the fifth storage node, and the sixth storage node.

4. The system according to claim 3, wherein: The system stores multiple copies of the first tenant root shard and multiple copies of the second tenant root shard.

5. The system according to claim 1, further comprising: A root shard comprising metadata associating the first shard with the first database table, the first key range, the first storage node, the second storage node, and the third storage node.

6. The system according to claim 5, wherein: The system stores multiple copies of the root shard.

7. The system of claim 1, further comprising: A manager node comprising a seventh processing unit and a seventh data storage device, the manager node being configured to: detecting a failure of the first storage node, and In response to detecting the failure, instructing the second storage node to copy the second copy of the first shard to the sixth storage node.

8. A system comprising: a first storage node comprising a first processing unit and a first data storage device, the first data storage device storing a first fragment of a first database table, and the first fragment comprising a first key range of the first database table; a second storage node comprising a second processing unit and a second data storage device, the second data storage device storing a second fragment of the first database table, and the second fragment comprising a second key range of the first database table; a third storage node, comprising a third processing unit and a third data storage device; as well as A manager node, the manager node comprising a fourth processing unit and a fourth data storage device, the manager node being used to instruct the first storage node to divide the first fragment into a third fragment associated with a third key range and a fourth fragment associated with a fourth key range, to store the third fragment in the first data storage device of the first storage node, and to store the fourth fragment in the third data storage device of the third storage node.

9. The system according to claim 8, further comprising: a fourth storage node comprising a fourth processing unit and a fourth data storage device, the fourth data storage device storing a fifth shard of the first database table, and the fifth shard comprising a fifth key range of the first database table, Wherein, the manager node is used for: instructing the fourth storage node to merge the second shard with the fifth shard to create a sixth shard associated with a sixth key range, the sixth key range including the second key range and the fifth key range; instructing the second storage node to delete the second fragment; and Deactivate the second storage node.

10. The system of claim 8, wherein the third data storage device stores a fifth shard of a second database table, and the fifth shard includes a fifth key range of the second database table.

11. The system according to claim 10, wherein: The first database table is a table of a first tenant conforming to a first tenant schema, and the second database table is a table of a second tenant conforming to a second tenant schema.

12. The system according to claim 11, further comprising: a fourth storage node comprising a fourth processing unit and a fourth data storage device, the fourth data storage device storing a sixth fragment of the first database table, and the sixth fragment comprising a sixth key range of the first database table, Wherein, the manager node is used for: instructing the fourth storage node to merge the second fragment with the sixth fragment to create a seventh fragment associated with a seventh key range, the seventh key range including the second key range and the sixth key range; instructing the second storage node to delete the second fragment; and Deactivate the second storage node.

13. A computer-implemented method comprising: Determining a first fragment of a first database table and a second fragment of the first database table, the first fragment including a first key range of the first database table, and the second fragment including a second key range of the first database table; storing a first copy of the first shard of the first database table on a first storage node; storing a second copy of the first shard of the first database table on a second storage node; storing a third copy of the first shard of the first database table on a third storage node; storing a first copy of a second fragment of the first database table on a fourth storage node; storing a second copy of the second shard of the first database table on a fifth storage node; as well as A third copy of the second shard of the first database table is stored on a sixth storage node.

14. The method according to claim 13, wherein: The first database table is a table of the first tenant that conforms to the first tenant schema, and the method further comprises: determining a third shard of the second database table, the second database table being a table of a second tenant conforming to a second tenant schema, and the third shard including a third key range of the second database table; storing a first copy of the third shard of the second database table in the first storage node; storing a second copy of the third shard of the second database table in a seventh storage node; and A third copy of the third shard of the second database table is stored in an eighth storage node.

15. The method according to claim 14, further comprising: storing a first tenant root shard, the first tenant root shard comprising first metadata associating the first shard with the first tenant, the first database table, the first key range, the first storage node, the second storage node, and the third storage node; as well as A second tenant root shard is stored, the second tenant root shard comprising second metadata associating the second shard with the second tenant, the second database table, the third key range, the fourth storage node, the fifth storage node, and the sixth storage node.

16. The method of claim 15, further comprising storing multiple copies of the first tenant root shard and multiple copies of the second tenant root shard.

17. The method according to claim 13, further comprising: A root shard is stored, the root shard comprising metadata associating the first shard with the first database table, the first key range, the first storage node, the second storage node, and the third storage node.

18. The method of claim 17, further comprising storing multiple copies of the root shard.

19. The method according to claim 13, further comprising: detecting a failure of the first storage node, and In response to detecting the failure, instructing the second storage node to copy the second copy of the first shard to the sixth storage node.

Images