Incremental recovery of cloud databases

Abstract

Systems and methods include storing a backup of a first fragment of a first database table of a database in a first backup location, the first fragment including a first key range of the first database table, storing a backup of a second fragment of the first database table in a second backup location, the second fragment including a second key range of the first database table, an instruction to restore the database is received, and in response to the instruction, the first fragment is restored from the first backup location to the first storage node, and in parallel, the second fragment is restored from the second backup location to the second storage node.

Description

Technical Field

[0001] The present disclosure relates generally to databases, and in particular to incremental recovery of cloud databases. Background Art

[0002] Modern organizations often utilize a system landscape consisting of distributed computing systems that provide a variety of computing services. For example, an organization may deploy services within computing systems located in on-premises data centers (which themselves may be located in different geographic locations) and within data centers provided by one or more platform as a service (PaaS) providers. The computing systems may include cloud-based computing systems (e.g., using on-demand scalable virtual machines to provide services).

[0003] In the Software as a Service (SaaS) model, a service provider deploys a service (e.g., a database) onto a cloud-based computing system (which in some cases may be provided by a separate PaaS provider). The service can be used by the customer in exchange for paying a fee to the service provider. The customer is not involved in the operation, management, or scaling of the service, all of which is performed by the service provider.

[0004] In the case of a database provided "as a service", the service provider wants to ensure scalability on both the storage and compute layers of the database. Some deployments decompose the storage and compute layers by utilizing separate node clusters for each layer. This decomposition (in theory) allows independent scaling of each layer.

[0005] The storage layer of such a database service may consist of many storage nodes, each of which contains local disk space and computing resources (e.g., RAM and CPU) to execute storage node processes. Typically, the database service is a single-tenant system, and all storage nodes of the storage layer combine to store a single database. Since each tenant requires a separate storage layer in such an implementation, these systems are cost-prohibitive for smaller tenants. For larger tenants, the elasticity of such a system is not suitable because the addition or removal of individual storage nodes cannot provide appropriately fine scaling increments.

[0006] The failure of a database instance results in a related failure of all tenants hosted on that database instance, known as a large "blast radius". In addition, recovery of data typically requires loading the data of a large backup file from cloud object storage into memory and writing the data to the data and log volumes of one or more storage nodes. Networking resources are provided to provide appropriate bandwidth during such data-intensive recovery operations. As a result, resources are inefficiently allocated (i.e., over-provisioned) for typical operational loads and result in wasted expenditures.

[0007] What is needed is a system that reduces the time and resources required for recovery and full redundancy after a storage node failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A distributed database system according to some embodiments is shown.

[0009] Figure 2 The storage of database table shards on multiple storage nodes is shown according to some embodiments.

[0010] Figure 3 Access to a database table fragment is shown in accordance with some embodiments.

[0011] Figure 4 is a flow diagram of a process for accessing database table fragments on a storage node according to some embodiments.

[0012] Figure 5a to Figure 5c A scale-out process is shown in accordance with some embodiments.

[0013] Figures 6a to 6c A scale-in process is shown according to some embodiments.

[0014] Figure 7 The storage of database table shard replicas on multiple storage nodes is shown according to some embodiments.

[0015] Figure 8 Access to a database table fragment is shown in accordance with some embodiments.

[0016] Figures 9a to 9c A scale-out process using database table shard replicas is shown in accordance with some embodiments.

[0017] Figures 10a to 10c A scale-in process using database table shard replicas is shown in accordance with some embodiments.

[0018] Figures 11a to 11c A recovery process using database table fragment copies according to some embodiments is shown.

[0019] Fig.12 The storage of database table shard replicas for multiple tenants on multiple storage nodes is shown according to some embodiments.

[0020] Fig.13 Access to database table shards for multiple tenants is illustrated in accordance with some embodiments.

[0021] Fig.14 Storage of sharded backups from multiple storage nodes to multiple backup locations is illustrated in accordance with some embodiments.

[0022] Fig.15 is a flow diagram of a process for restoring a shard backup to a storage node according to some embodiments.

[0023] Fig.16 Restoration of a shard backup to a storage node is shown in accordance with some embodiments.

[0024] Fig.17 is a flow diagram of a process for incrementally restoring a shard backup to a storage node according to some embodiments.

[0025] Fig.18a and Fig.18b Incremental restore of a shard backup to a storage node is shown in accordance with some embodiments.

[0026] Fig.19 is a block diagram of a cloud-based virtual machine providing database services according to some embodiments. DETAILED DESCRIPTION

[0027] The following description is provided to enable any person skilled in the art to make and use the described embodiments. However, various modifications will still be apparent to those skilled in the art.

[0028] According to some embodiments, the data of each tenant of the database system is logically split into fragments, and the fragments of a given tenant are distributed across multiple storage nodes. The fragments are large enough to amortize the cost of operations on individual data items by allowing batch request processing, and small enough to allow reasonably fast operations on the entire fragment (e.g., for backup / restore, fragment relocation). In some embodiments, the fragments can be substantially equal in size and in the range of 1GB-10GB per fragment.

[0029] Each shard stored on a storage node may be backed up to a backup layer implemented by an object storage service. The backup layer node may store backup data of one or more shards for each of one or more tenants. In a specific example, a system according to some embodiments may include multiple storage nodes and a backup layer, each storage node storing one or more shards for a given tenant, and the backup layer storing backup data for each shard of a given tenant in a separately identifiable location.

[0030] Some embodiments may be operable to receive a request to restore a database table of a database tenant to a database system. In response, a backup location of all database table fragments of the database tenant is identified. Next, all database table fragments of the database tenant are restored from the identified backup location to a plurality of storage nodes of the database system.

[0031] Using a 1GB / s connection between the storage layer and the backup layer, the system described in the background technology above can theoretically restore 1TB of backup data to the storage layer in 17 minutes. In contrast, using a similar 1GB / s backup layer to storage node bandwidth, some embodiments can complete the entire restore process in a few seconds, especially if the backup layer to storage node bandwidth is large compared to the size of the fragmented backup.

[0032] Some embodiments may also provide a restore operation that allows for near instantaneous use of the data to be restored. According to this so-called incremental restore, a request to restore a database table to a database system comprising a plurality of storage nodes is received. In response, metadata defining shards of the database table is restored. The metadata may specify the tenant, database table, key range, and backup location associated with each shard. Restoration of the metadata may occur in less than a second.

[0033] Next, a request to perform a database operation on one or more shards is received from the database front end. Based on the shard metadata, it is determined whether the one or more shards have been restored to a storage node of the database system. Assuming that none of the one or more shards have been restored, the shards are restored from the backup location associated with the shard in the metadata, and the metadata is updated to specify the storage node in which the shard is now stored. Identifiers of the one or more storage nodes are provided to the database front end, and the database front end uses the identifiers to send a request to each of the one or more storage nodes to perform an operation on the one or more shards. Thus, using incremental recovery, the shards are restored "on demand" and the shards are available for use almost immediately.

[0034] To achieve high availability, each recovered shard can be replicated multiple times between different storage nodes. In some embodiments, the relatively small size of the shards and the large node-to-node bandwidth allow such replication to be performed within seconds. In the event of a storage node failure or outage, the data of all shards stored by the storage node remains accessible via copies of those shards stored on other storage nodes. In addition, since copies of shards are typically distributed across the remaining storage nodes, the remaining nodes are able to generate new copies to restore full redundancy. For example, a copy of each shard can be sent from its storage node to a new storage node. When the shards of a database are distributed across many storage nodes, re-replication of data for restoring full redundancy of the database can be performed in parallel across all nodes (i.e., using n:n communication instead of 1:1 communication). In contrast, other systems require redistributing the entire database and / or replicating the data of an entire node over a network to adjust a failed node.

[0035] Figure 1 A system according to some embodiments is shown. Figure 1The components shown may be implemented using any suitable combination of known or becoming known computing hardware and / or software. Such combinations may include cloud-based implementations in which computing resources are virtualized and elastically allocated. In some embodiments, two or more components are implemented by a single computing device.

[0036] Clients 110-116 may include computing systems that execute applications or services that issue database queries. Clients 110-116 may issue database queries in response to requests (not shown) input thereto by users. Computing layer 120 receives the queries (e.g., via an API provided by computing layer 120) and operates in conjunction with database data stored in storage layer 130 to provide corresponding query results to clients 110-116.

[0037] The computing layer 120 includes a plurality of computing nodes 122-128. Similarly, the storage layer 130 includes a plurality of storage nodes 132-138. Each of the nodes 122-128 and 132-138 includes at least one processing unit (e.g., a CPU), a random access memory, and at least one data storage device (e.g., an NVMe SSD), and may include a single computer server, a group of computer servers, a virtual machine, and any other combination known or becoming known. In some embodiments, the computing layer 120 is completely stateless and booted over a network, so the computing nodes 122-128 do not need to include storage devices.

[0038] In some embodiments, the computing layer 120 and the storage layer 130 are implemented by a Kubernetes cluster. Kubernetes is an open source system for automating the deployment, scaling, and management of containerized applications. A Kubernetes cluster consists of many nodes, which in turn can include physical servers and / or virtual machines. Nodes execute applications within software containers. One or more containers can be grouped into pods, where each container of a given pod shares the same computing resources and local network.

[0039] The backup layer 140 provides object storage to the storage layer 130. Each storage node 132-138 of the storage layer can store its block-based data as an object within the backup layer 140. In some embodiments, the backup layer 140 can provide object storage through a web service interface. The backup layer 140 may include an object storage service provided by a hyperscale cloud service provider (e.g., AWS S3). The storage layer 130 identifies the shard backups stored in the backup layer 140 via an abstract backup location identifier. In some embodiments, the backup location identifier may include a combination of a corresponding tenant ID, a shard ID, and a version counter.

[0040] According to some embodiments, Figure 1Represents a single-tenant database system. Storage nodes 132-138 store data associated with a single tenant, and compute node 122 processes queries received from clients 110-116 operated by users of the single tenant. As described above, each of storage nodes 132-138 stores a respective shard of data associated with a single tenant in order to provide parallelization and elasticity.

[0041] Each of the storage nodes 132-138 may store backup data for each of its corresponding shards to the backup tier 140. In the event that the storage nodes 132-138 store multiple copies of each shard of the database, the backup data for only one copy of each shard may be stored in the backup tier 140.

[0042] Figure 1 The multi-tenant database system in some embodiments may be represented. In such an embodiment, the storage nodes 132-138 store data associated with multiple tenants. The data is stored in fragments distributed across the storage nodes 132-138. Each of the computing nodes 122-128 can execute a single tenant database instance specific to a tenant. Therefore, in the case where the client 110 is being accessed by a user of the first tenant, a query issued by the client 110 is directed to one of the computing nodes 122-128 associated with the first tenant.

[0043] In yet another example, the storage nodes 132-138 store data associated with multiple tenants, and each of the compute nodes 122-128 executes a multi-tenant database instance. Thus, any one of the compute nodes 122-128 is capable of processing queries received from the clients 110-116 regardless of the tenant whose data is being queried. In the case of a multi-tenant database system, the storage nodes 132-138 store backup data for each shard of each tenant database in the backup layer 140. Thus, the backup layer 140 can store backup data for one or more shards of one or more tenant databases.

[0044] Figure 2 The storage of database table fragments of a single tenant on multiple storage nodes according to some embodiments is shown. Database tables 210, 220, and 230 are logical representations of data associated with a given database tenant. Database tables 210, 220, and 230 may conform to a specific schema known in the art. According to some embodiments, each row of each of database tables 210, 220, and 230 includes a value describing an instance of an object. Embodiments are not limited to any type of object.

[0045] Each of the database tables 210, 220, and 230 is split into a plurality of fragments. Specifically, the database table 210 is split into fragments 212, 214, and 216, the database table 220 is split into fragments 222 and 224, and the database table 230 is split into fragments 232, 234, 236, and 238. Each fragment is associated with a key range. In some embodiments, the fragments may be substantially equal in size, and the size of each fragment may be between 1 GB and 10 GB. Embodiments include any algorithm for splitting a table into fragments and for defining a key range for a fragment. In one embodiment, the storage layer 130 is a key / value store, and the key range of a fragment is the key range of the key stored in the fragment. According to some embodiments, a fragment may include data for more than one database table.

[0046] According to the present example, the shards are stored across storage nodes 132, 134, and 136 of storage tier 130. Storage node 132 stores shards 212, 214, 216, and 236, storage node 134 stores shards 222, 232, and 238, and storage node 136 stores shards 224 and 234. It is assumed that storage node 138 and any other storage nodes of storage tier 130 store shards of other, not shown, database tables of the tenant.

[0047] The distribution of shards across the storage nodes of the storage layer 130 may be based on any suitable algorithm. Since the storage node 132 stores all shards 212, 214, 216 of the table 210, the failure of the node 132 causes all data of the table 210 to be inaccessible. In some embodiments, each shard of the table is preferably stored on a different node (e.g., shards 222 and 224 of the table 220), so that the failure of a node causes the inaccessibility of at most one shard of the table. Some embodiments may store at most one shard of a given tenant per storage node.

[0048] Figure 3 3. Access to database table fragments according to some embodiments is shown. The database computing front end 310 may include a database instance executed on a computing node as described above. The database computing front end 310 may need to access database table data in response to a query received from a client. However, due to the fragmented storage described above, the database computing front end 310 must initially determine the location of the fragment containing the desired data in order to direct its request to the appropriate storage node.

[0049] The topology manager 320 consists of one or more nodes that store shard metadata 325. The shard metadata 325 can be associated with a well-known ID in the system and stored in one or more storage nodes 132 to 136 in the same manner as other shards stored herein. According to some embodiments, the shard metadata 325 specifies the storage location of the database shards. For example, the shard metadata 325 can identify each shard by a table ID and a key range. In addition, each shard is also associated with an identifier of the storage node where it is stored. Therefore, the database computing front end 310 queries the topology manager 320 for the location of the shards associated with the query (e.g., all shards of a given table) and receives an indication of the storage node storing the desired shard. The database computing front end 310 then issues database operations to the nodes storing the shards. As described above, if the shards are located on more than one storage node, the operations can be advantageously performed in parallel.

[0050] In some embodiments, the creation of a tenant causes a single shard to be created on a storage node. This shard can be considered a root shard, which includes shard metadata 325 associated with the tenant, rather than the topology manager 320. Therefore, in response to a received query, the database compute front end 310 queries the topology manager for the location of the root shard of the tenant associated with the query, retrieves the shard metadata from the root shard, and based thereon determines a storage node to store the desired shard.

[0051] Figure 4 4 is a flow chart of a process 400 for accessing database table fragments on a storage node according to some embodiments. Any suitable combination of hardware and software may be used to perform process 400 and other processes described herein. The software program code embodying these processes may be stored by any non-transitory tangible medium, including a fixed disk, a volatile or non-volatile random access memory, a DVD, a flash drive, or a tape, and executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Such processors, processor cores, and processor threads may be implemented by virtual machines provided in a cloud-based architecture. Embodiments are not limited to the examples described below.

[0052] Process 400 assumes previous storage of database table fragments on multiple storage nodes, as described above. At S410, a request for the location of a database table fragment is received. The request may identify the fragment using a database table identifier and one or more primary keys (or primary key ranges) of the identified table. The request is received from a database front end, which may be executed on a compute node of a compute layer as described above. The fragment may include data required to service a query received by the database front end from a client application.

[0053] In response to the request, a storage node storing the shard is determined based on the stored metadata. According to some embodiments, the stored metadata is shard metadata stored by the topology manager as described above. Therefore, such a topology manager can receive the request at S410 and determine the storage node based on the request and the shard metadata at S420. In one example, the request includes a database table identifier and a primary key range. The topology manager can determine the shard including the primary key range of the identified table based on the shard metadata at S420, and also determine the storage node storing the shard based on the shard metadata. In some embodiments, S420 includes determining the root shard of the tenant associated with the query and determining the storage node storing the desired shard based on the metadata stored in the root shard. At S430, the identifier of the determined storage node is provided to the requester.

[0054] It may be determined at S420 that the primary key range spans two or more fragments, in which case the storage nodes storing the two or more fragments are determined at S420. In another example, the request includes only a table identifier and does not include a primary key or key range. Thus, S420 includes determining all fragments of the identified table and their corresponding storage nodes. The received request may include any number of table identifiers and associated key ranges.

[0055] At S440, a database operation for the shard is received at the database node storing the shard. The database operation may be issued by the database front end from which the request was received at S410, and may utilize the identifier provided at S430. If more than one storage node has been identified, the database operation may be issued to each other node at S440 and received at each other node, thereby enabling operations to be performed in parallel by more than one storage node.

[0056] Figure 5a to Figure 5c A scale-out process is shown in accordance with some embodiments. Figure 5a The storage nodes 132, 134 and 136 store corresponding fragments, some of which have sizes relatively large. Figure 3 Thus, it may be determined manually or automatically (based on resource consumption parameters) to reduce the amount of storage consumed by one of the storage nodes by adding another storage node to the corresponding storage tier. According to some embodiments, such a determination may be performed by a topology manager. It may also or alternatively be determined to split shards 234 and 238 because their size exceeds a specified maximum shard size.

[0057] Figure 5b The addition of a storage node 510 to the storage tier is shown. Figure 5bAlso shown is the movement of shards from each of storage nodes 132, 134, and 136 to storage node 510. This movement may be intended to achieve load balancing, which reduces the memory consumption of each node.

[0058] Figure 5c The resulting storage layer is shown. As shown, fragment 236 has been moved from storage node 132 to storage node 510. Fragment 238 has been divided into fragments 238a and 238b, and fragment 238b has been moved from storage node 134 to storage node 510. Similarly, fragment 234 has been divided into fragments 234a and 234b, and fragment 234b has been moved from storage node 136 to storage node 510. Embodiments are not limited to dividing fragments into fragments of equal size.

[0059] Advantageously, the shards moved to node 510 each move over a different (and substantially independent) node-to-node network connection. Thus, the movement of the shards can occur significantly faster than if all three shards traveled from a single node to node 510. Furthermore, the movement requires only 1 / n of the bandwidth of the node from which the shard is moved. Thus, the impact on node performance and system response time is minimal.

[0060] The partitioning of shards and the movement of the resulting shards can occur with or without adding storage nodes. The addition of storage nodes and the movement of shards thereto can also occur with or without partitioning any shards.

[0061] Figures 6a to 6c An expand-in process is shown according to some embodiments. Figure 6a Storage nodes 132, 134, 136, and 138 are shown, some of which are smaller than Figure 5c Thus, it may be determined manually or automatically (again, based on resource consumption parameters) to reduce the number of storage nodes of the corresponding storage layer. According to some embodiments, this determination may be performed by a topology manager.

[0062] In this example, it is determined that storage node 136 is deactivated. Therefore, Figure 6b The movement of shards of storage node 136 to storage nodes 132 and 510 is shown. In some embodiments, the storage nodes to which the shards are moved are determined in order to achieve load balancing.

[0063] Figure 6cThe resulting storage layer is shown, where shard 224 has been moved from storage node 136 to storage node 132, and shard 234a has been moved from storage node 136 to storage node 510. Storage node 136 no longer stores any shards, and can therefore be deactivated. According to some embodiments, scaling in may also or alternatively include merging two or more small shards into one shard, and placing the one shard on a storage node based on load balancing and resource consumption considerations. According to some embodiments, due to the key ranges associated with each shard, shards should only be merged with shards associated with adjacent key ranges.

[0064] Figure 7 FIG. 4 shows the storage of database table fragment copies of a single tenant on multiple storage nodes according to some embodiments. Figure 2 As described, database tables 210 , 220 , and 230 are logical representations of data associated with a given database tenant and are split into multiple shards associated with different key ranges.

[0065] like Figure 7 As shown, storage layer 130 stores multiple copies of each of shards 212, 214, 216, 222, and 224. Replication can be performed in any suitable manner, such as by using a consensus protocol, employing erasure coding across replicas, etc., each of which can present different advantages and disadvantages based on the use case. The distribution of replicas across storage nodes of storage layer 130 can be based on any suitable algorithm that can provide load balancing and high availability.

[0066] Despite Figure 7 , but embodiments may utilize any number of replicas. In the case of a protocol that replicates each shard across three different storage nodes, if one of the three storage nodes serving that shard fails or is disabled, the data for the particular shard remains fully accessible. If the replication factor is set to greater than 3, the system continues to operate normally even in the event of multiple node failures (e.g., two node failures using a replication factor of 5, or k node failures with a replication factor of 2k+1 in general).

[0067] Figure 7 Mirrors the consensus protocol, where each shard is represented by a leader replica and two follower replicas. The leader replica is determined by the leader election algorithm and is Figure 7. All operations on a given shard are directed to the storage node storing the leader replica of the shard, and changes to it are replicated to the follower replicas of the shard. If the node that includes the leader replica of the shard fails, one of the remaining follower replicas of the shard is designated as the leader replica, and the new leader replica begins replicating its changes to the remaining follower replicas. In addition, as will be described below, a new follower replica is quickly instantiated on another (or new) storage node to re-establish the desired consensus protocol.

[0068] Figure 8 Access to the leader replica of a shard is shown in accordance with some embodiments. The database compute front end 310 determines the location of the leader replica of the shard containing the desired data in order to direct its requests to the appropriate storage nodes. The shard metadata 325 of the topology manager 320 specifies the storage location of all replicas of each shard and records the current leader replica of each shard. If the storage node including the leader replica of the shard fails or is deactivated, a new leader replica of the shard is elected and the shard metadata 325 is updated accordingly. Any follower replicas stored on the failed / deactivated storage node are removed from the shard metadata 325. The shard metadata 325 also specifies the table ID and key range for each shard.

[0069] Thus, the database compute front end 310 queries the topology manager 320 for the location of the shards associated with the query (e.g., all shards of a given table), and receives therefrom the identifier of the storage node storing the leader replica of the desired shard. The database compute front end 310 then issues the database operation to the node on which the leader replica shard is stored. Again, if the shards are located on more than one storage node, the operations can be advantageously performed in parallel.

[0070] The database computing front end 310 may also receive an identifier of a storage node where a follower copy of the desired shard is stored. Thus, if an operation on the identified leader copy fails, the database computing front end 310 may seamlessly retry the operation on one of the corresponding follower copies stored on a different storage node.

[0071] In some embodiments, the root shard stores shard metadata 325 instead of the topology manager 320, and the topology manager 320 stores only the location of the root shard. The root shard may also be stored between storage nodes 132-138 in multiple copies, in which case the topology manager 320 stores the location of each copy.

[0072] Figures 9a to 9c A scale-out process using database table shard replicas is shown in accordance with some embodiments. Figure 9a The storage nodes 132, 134, 136 and 138 store Figure 7 and Figure 8 It will be assumed that a manual or automatic determination is made to add another storage node to the corresponding storage tier, for example to reduce the amount of storage consumed by one of the storage nodes or to provide improved availability.

[0073] Figure 9b 9 shows the addition of storage node 910 to the storage layer. Figure 9b As shown, fragments are moved from each of storage nodes 132 , 134 , and 138 to storage node 910 . Fig.9c The resulting storage layer is shown. Shard 222 has been moved from storage node 132 to storage node 910, shard 212 has been moved from storage node 134 to storage node 910, and shard 214 has been moved from storage node 138 to storage node 910. In all of the examples described herein, shard metadata 325 is updated to reflect the new replica storage locations resulting from the movement of shards between storage nodes.

[0074] The shards are moved to node 910 via different (and substantially independent) node-to-node network connections (i.e., 132 to 910, 134 to 910, and 138 to 910). By replicating one shard instead of three, nodes 132, 134, and 138 are each much less burdened than other implementations.

[0075] Figures 10a to 10c A scale-in process using database table shard replicas is shown in accordance with some embodiments. Fig.10a Reflects on Fig.9c Described Storage Tiers. It will be assumed that a determination to decommission a storage node 136 may be made manually or automatically based on underutilization of one or more storage nodes of the depicted storage tier.

[0076] Fig.10b The movement of shards of storage node 136 to storage nodes 132, 134, and 138 is shown. In some embodiments, the shards to be moved and the storage nodes to which the shards are moved may be determined based on a load balancing and / or resource consumption algorithm. Fig.10c The resulting storage layer is shown, where shard 224 has been moved from storage node 136 to storage node 132, shard 214 has been moved from storage node 136 to storage node 134, and shard 212 has been moved from storage node 136 to storage node 138. Since storage node 136 no longer stores any shards, it can be decommissioned. The shard metadata 325 is then updated to reflect the new replica storage locations resulting from the above-described movement of shards between storage nodes.

[0077] Figures 11a to 11cA recovery process using database table fragment copies according to some embodiments is shown. Fig.11a Reflects on Fig.10a The storage layer described above, however, assumes in this case that storage node 136 unexpectedly fails. In some embodiments, the decommissioning of node 136 can also be as described below with respect to Figure 11a-Figure 11c Proceed as described.

[0078] Upon detecting a failure, the topology manager 320 accesses the shard metadata 325 to determine the shards stored on the storage node 136. Next, the topology manager 320 identifies which of the remaining storage nodes stores a copy of the shards stored on the storage node 136 based on the shard metadata 325. The topology manager 320 then instructs the storage layer to replicate a copy of each shard stored on the storage node 136 from the remaining storage nodes to a different remaining storage node.

[0079] Fig.11b It is shown that a copy of each shard stored on failed storage node 136 is replicated to other storage nodes. For example, shard metadata 325 indicates that failed storage node 136 stores copies of shards 224, 214, and 212, and that the copy of shard 214 is the leader copy. Shard metadata 325 also indicates that a copy of shard 224 is located on nodes 134 and 138, a copy of shard 214 is located on nodes 132 and 910, and a copy of shard 212 is located on nodes 132 and 910. Therefore, Fig.11b Shard 224 is shown being replicated from node 138 to node 132, shard 214 is replicated from node 910 to node 134, and shard 212 is replicated from node 910 to node 138. As described above, replication between different sets of nodes allows each replication operation to proceed in parallel and use the full available node-to-node bandwidth. Fig.11c The resulting storage tier after the copy operation is completed is shown.

[0080] According to some embodiments, the specific storage nodes from which the shards are replicated and to which the shards are replicated can be determined based on a load balancing and / or resource consumption algorithm. The shard metadata 325 is updated to reflect the new replica storage location. The update can also include electing a replica of the shard 214 stored in the node 134 as the leader replica, and indicating the election in the shard metadata 325.

[0081] Fig.12The storage of database table shard copies of multiple tenants on multiple storage nodes is shown according to some embodiments. Database table 210 includes data for Tenant 1, and database table 710 includes data for Tenant 2. Database table 210 conforms to a schema associated with Tenant 1, and database table 710 conforms to a schema associated with Tenant 2. Each tenant may be associated with many other database tables that include their respective data. As is known in the art, Tenant 1's data is intended to be accessible only to users of Tenant 1, and Tenant 2's data is intended to be accessible only to users of Tenant 2.

[0082] In the example shown, table 210 is divided into shards 212, 214, and 216, and table 710 is divided into shards 712 and 714. Storage layer 130 stores multiple copies of each of shards 212, 214, 216, 712, and 714, as described above with respect to Figure 7 As described. Fig.12 Represents a consensus protocol consisting of a leader replica (i.e., represented by a distinct outline) and two follower replicas for each shard. The replicas may be distributed across the storage nodes of the storage layer 130 based on any suitable algorithm for providing load balancing and high availability. Fig.12 The node scale-out, scale-in, and failover shown in the figure can be described in the same way as above. Figure 9a-9c , Figure 10a-Figure 10c and Figure 11a-Figure 11c Described manner.

[0083] Fig.13 Access to database table shards for multiple tenants is illustrated in accordance with some embodiments.

[0084] Database computing front end 1310 serves users of tenant 1, and database computing front end 1320 serves users of tenant 2. Database computing front end 1310 may include a single-tenant database instance executed on a single computing node of the computing layer, and database computing front end 1320 may include a single-tenant database instance executed on another single computing node of the computing layer. In some embodiments, database computing front end 1310 and database computing front end 1320 are executed on the same computing node.

[0085] The topology manager 320 may operate as described above. Also as described above, the shard metadata 325 may associate each shard with a table identifier, a key range, the location of each replica, and an identifier of the leader replica. However, because the tables 210 and 710 conform to different schemas, the table identifier and key range of the table of Tenant 1 may overlap / conflict with the table identifier and key range of the table of Tenant 2. Therefore, the shard metadata 325 also associates each shard with a tenant identifier. If the database compute front ends 1310 and 1320 are single-tenant database front ends, the tenant identifier may be an identifier of the container in which the front end is executing.

[0086] In response to the received query, the database computing front end 1310 (or front end 1320) queries the topology manager 320 for the location of the shard associated with the query. The query of the topology manager 320 includes a tenant identifier, a table identifier, and a key range (e.g., as a concatenated string). The topology manager 320 determines the identifier of the storage node storing the leader replica of the desired shard based on the query and the shard metadata 325 received from the database computing front end 1310 (or front end 1320). The database computing front end 1310 (or front end 1320) then issues database operations to the node on which the leader replica shard is stored. If the shard is located on more than one storage node, the storage nodes can perform these operations in parallel.

[0087] In some embodiments, each tenant is associated with a root shard including the tenant's shard metadata, and the topology manager 320 stores the location of each tenant's root shard. Thus, the database compute front end 1310 or 1320 can query the topology manager 320 for a storage node storing a given tenant's root shard, read the metadata of the root shard from the storage node, and determine the location of the desired shard based on the metadata. Each root shard can also be stored between storage nodes 132-138 in multiple replicas, in which case the topology manager 320 stores the location of each replica of each root shard.

[0088] Some embodiments include a multi-tenant database compute front end that serves users of Tenant 1 and users of Tenant 2. The multi-tenant database compute front end accesses the shard replicas, as described above with respect to both database compute front end 1310 and database compute front end 1320, querying the topology manager 320 using the appropriate tenant identifier.

[0089] Fig.14Storing shard backups from multiple storage nodes to a backup tier according to some embodiments is shown. As described above, storage nodes 1420, 1430, 1440 store shards in their respective storage devices 1422, 1432, 1442, and 1452. Database compute front end 1410 accesses various storage nodes in storage nodes 1420, 1430, 1440, and 1450 to perform operations on the shards stored therein.

[0090] For simplicity, it is assumed that each of the storage nodes 1420, 1430, 1440, and 1450 stores a corresponding single shard of a single tenant database. In addition, from time to time during operation, and as is known in the art, each of the storage nodes 1420, 1430, 1440, and 1450 stores backup data of its corresponding shard in a corresponding one of the backup locations 1425, 1435, 1445, and 1455. The stored backup data may include snapshots and / or snapshots and corresponding logs, as is known in the art. The backup data may include object data suitable for storage in the backup locations 1425, 1435, 1445, and 1455, rather than block-based data. Embodiments are not limited to a one-to-one correspondence between storage nodes and shards.

[0091] Fig.15 14 is a flow chart of a process 1500 for restoring a fragmented backup stored in a backup tier including backup locations 1425, 1435, 1445, and 1455 according to some embodiments. Initially, at S1510, a request to restore a database table to a database system may be received. The request may be issued by an administrator in response to a database crash or for any other suitable reason.

[0092] Fig.16 A restore of a shard backup to a storage node is shown according to some embodiments of process 1500. An administrator may interact with a user interface of administrator component 1610 to submit instructions to restore database tables of a database tenant from corresponding backup data.

[0093] In response, at S1520, the topology manager 320 determines a backup location (i.e., location ID) storing backup data for each shard of each database table. This determination may be based on the shard metadata 325. Next, the topology manager 320 instructs the storage layer 1620 to restore each shard from its backup location. Thus, at S1530, the database table shards are restored from each of the determined backup locations to a storage node of the storage layer 1620. The shard metadata 325 may specify a storage node corresponding to each shard. Restoration may include storing the snapshot in the storage node and replaying any corresponding logs.

[0094] In some embodiments, each database table fragment restored on a given storage node is replicated to two or more other storage nodes. Such replication can occur before or after the restored fragments are available for front-end consumption. Management of fragment replication and replicas during operation can be performed as described above.

[0095] Fig.17 17 is a flow chart of a process 1700 for incrementally restoring a shard backup from a backup location to a storage node according to some embodiments. At S1710, a request is received to restore a database table of a database to a database system. In response, metadata defining shards of the database table is restored at S1720. S1720 assumes an implementation in which the database is associated with a root shard stored in the storage layer 1620, and the root shard includes metadata defining each shard of each database table and their corresponding locations. This root shard implementation is not limited to incremental recovery scenarios and can be used in the above-mentioned embodiments. Fig.15 and Fig.16 The system described is used.

[0096] Fig.18a The administrator component 1610 is shown from which the topology manager 320 receives an instruction to restore the database at S1710. According to this example, the shard metadata 325 specifies the storage node (or multiple nodes, if shard copies are employed) of the root shard of the database within the storage layer 1620. Then, at S1720, the root shard is restored from the backup location corresponding to the specified storage node.

[0097] After S1720, the storage layer 1620 may be made accessible to the database computing front end. It is worth noting that this access exists before any database table fragments are restored to the storage layer 1620. Therefore, after receiving a request to restore a database table at S1710, access may be reestablished in a near instantaneous manner.

[0098] Next, at S1730, a request to access a shard of a database table is received. As described above, the request may be received by the topology manager 320. In response, the topology manager 320 determines at S1740 and based on the shard metadata 325 whether the shard has been restored to the storage layer 1620. If not, at S1750, the shard is restored to the storage node from the backup location specified by the metadata of the root shard, such as Fig.18a 14 is shown by the arrow from backup location 1435 to storage node 1430. Shard metadata 325 is also updated to specify the storage node where the shard is now stored.

[0099] At S1760, the topology manager 320 provides the requestor with an identifier of the storage node. Fig.18aIn this example, the requester (e.g., front end 1410) communicates with storage node 1430 using an identifier to request an operation on a fragment. At S1770, the operation is performed on the fragment in the storage node. The process returns to S1730 to wait for another request for access to the fragment.

[0100] It will be assumed that a request to access the same shard is received at S1730. Therefore, it is determined at S1740 based on the shard metadata 325 that the shard has been restored to the storage node 1430. Therefore, the process proceeds to S1760 to provide the requestor with an identifier of the storage node 1430, as described above. Therefore, according to some embodiments, each shard of the database is restored from the backup location only in response to the first request to access the shard. If no request to access the shard is received, the shard may never be restored.

[0101] The request received at S1730 may request access to multiple fragments of a database table and / or other fragments of several database tables. In this case, S1740 to S1770 are performed in parallel for each fragment requested. Continuing with the above example, the next request received at S1730 may request access to the first requested fragment and another fragment that has not yet been restored. For the first requested fragment, the process proceeds to S1740, S1760, and S1770 to simply request operations on the fragments stored in storage node 1430, such as Fig.18b shown.

[0102] Assuming that the backup data of the requested other fragment is stored in the backup location 1445, the process proceeds from S1740 to S1750 to restore the fragment to the node 1440, such as Fig.18b The process continues to S1760 and S1770 to receive the identifier of the storage node 1440 and request an operation on the fragment stored therein, as shown in Fig.18b shown.

[0103] Shards restored to a respective storage node according to process 1700 may be replicated to two or more other storage nodes to provide the above benefits. The replication may occur in response to restoration of a shard, asynchronously, in batches of restored shards, and the like.

[0104] Figure 14 to Figure 18b The foregoing description applies to a multi-tenant architecture where the storage layer 1620 stores shards of database tables for two or more tenants. In such an architecture, the shard metadata 325 can specify the tenant, database table, key range, storage node (if any), and backup location for each shard. Thus, a received restore request specifies a specific tenant database, and only those shards associated with the database of the specific tenant are restored, as described with respect to process 1500 or 1700.

[0105] Fig.19 is a block diagram of a cloud-based environment according to some embodiments. Each component may be implemented using any suitable combination of hardware and software that is or becomes known. For example, each component may include a physical computer server or a virtual machine. Such virtual machines may reside in one or more public clouds, providing self-service and immediate provisioning, automatic scaling, security, compliance, and identity management features.

[0106] Application server 1910 executes one or more applications that issue database queries. The application can be accessed by a user (not shown) who requests information from the application, thereby triggering the issuance of a database query. Application server 1910 sends the query to database computing node 1920, which executes a database instance (e.g., indexserver). Based on the query, the database instance instructs CRUD operations on the data stored in storage nodes 1932, 1934, and 1936 of storage layer 1930.

[0107] The data is stored in the storage nodes 1932, 1934, and 1936 of the storage layer 1930 in any of the above-described ways. For example, the data for a given table may be stored in shards distributed throughout the storage nodes 1932, 1934, and 1936, and each shard may be replicated across multiple storage nodes. The data may include data for more than one tenant, where each shard for a given tenant is identified in the metadata by a tenant identifier, a table identifier, and a key range. The database compute node 1920 uses the metadata to determine the location of the shard for the desired operation.

[0108] The fragmented data stored in the storage nodes 1932, 1934, and 1936 of the storage layer 1930 are periodically backed up to the backup locations 1942, 1944, and 1946 of the backup layer 1940. When a request to restore the database is received, the fragments of the database are restored from those backup locations of the backup layer 1940 storing the corresponding backup data. All fragments can be as described in detail. Fig.15 Parallel recovery as described, or as described in Fig.17 Described is recovery incrementally based on access requests.

[0109] The foregoing diagrams represent the logical architecture for describing the process according to some embodiments, and the actual implementation may include more or different components arranged in other ways. Other topologies may be used in combination with other embodiments. In addition, each component or device described herein may be implemented by any number of devices communicating via any number of other public and / or private networks. Two or more such computing devices may be located remotely from each other and may communicate with each other via any known network mode and / or dedicated connection. Each component or device may include any number of hardware and / or software elements suitable for providing the functions described herein and any other functions. For example, any computing device used in the implementation of the system according to some embodiments may include a processor to execute program code so that the computing device operates as described herein.

[0110] All systems and processes discussed herein may be embodied in program code stored on one or more non-transitory computer-readable media. Such media may include, for example, hard disks, DVD-ROMs, flash drives, magnetic tapes, and solid-state random access memory (RAM) or read-only memory (ROM) storage units. Therefore, embodiments are not limited to any particular combination of hardware and software.

[0111] The embodiments described herein are for illustration purposes only. Those skilled in the art will appreciate that other embodiments may be practiced through modifications and alterations to the embodiments described above.

Claims

1. A system for data storage, comprising: A backup layer, including one or more data storage devices, the backup layer including: a first backup location storing a backup of a first fragment of a first database table of the database, the first fragment including a first key range of the first database table; and The second backup location stores a backup of a second fragment of the first database table, where the second fragment includes a second key range of the first database table.

2. The system according to claim 1, further comprising: A first storage node, comprising a first processing unit and a first data storage device; as well as The second storage node includes a second processing unit and a second data storage device, In response to an instruction to restore the database, the first fragment is restored from the first backup location to the first storage node, and in parallel, the second fragment is restored from the second backup location to the second storage node.

3. The system according to claim 2, wherein: In response to the instruction to restore the database, metadata is created that associates the new tenant with the first shard and the second shard, the first shard with the first storage node, and the second shard with the second storage node.

4. The system according to claim 2, further comprising: a third storage node, comprising a third processing unit and a third data storage device, the third data storage device storing a copy of the first fragment; as well as The fourth storage node includes a fourth processing unit and a fourth data storage device, and the fourth data storage device stores a copy of the second fragment.

5. The system according to claim 1, wherein: In response to an instruction to restore the database, metadata is created that associates the new tenant with the first shard and the second shard, the first shard with the first storage node and the third storage node, and the second shard with the second storage node and the fourth storage node.

6. The system according to claim 1, further comprising: A storage layer, including a plurality of storage nodes, wherein the plurality of storage nodes include corresponding processing units and data storage devices; as well as A manager node, which stores metadata associating the new tenant with the first shard and the second shard, Manager nodes are used to: receiving a first request from a requestor to access a first shard; In response to the first request: determining, based on the metadata, that the first fragment is not stored in any of the plurality of storage nodes; as well as In response to determining: issuing an instruction to restore the first shard from the first backup location to a first storage node among the plurality of storage nodes; as well as An identifier of the first storage node is returned to the requestor.

7. The system according to claim 6, wherein: The manager node is used to: receiving a second request from the requestor to access the first shard; as well as In response to the second request: determining, based on the metadata, that the first shard is stored in a first storage node among the plurality of storage nodes; as well as In response to determining that the first shard is stored in the first storage node, an identifier of the first storage node is returned to the requester.

8. The system of claim 1, further comprising: A storage layer, including a plurality of storage nodes, wherein the plurality of storage nodes include corresponding processing units and data storage devices; as well as A manager node, which stores metadata associating the new tenant with the first shard and the second shard, Manager nodes are used to: receiving a first request from a requester to access a first shard and a second shard; In response to the first request: determining, based on the metadata, that the first shard and the second shard are not stored in any of the plurality of storage nodes; as well as In response to the determination: issuing instructions to restore the first shard from the first backup location to a first storage node among the plurality of storage nodes, and in parallel, restore the second shard from the second backup location to a second storage node among the plurality of storage nodes; as well as An identifier of the first storage node and an identifier of the second storage node are returned to the requester.

9. A computer-implemented method comprising: storing a backup of a first fragment of a first database table of the database in a first backup location, the first fragment including a first key range of the first database table; as well as A backup of a second shard of the first database table is stored in a second backup location, the second shard including a second key range of the first database table.

10. The method according to claim 9, further comprising: receiving an instruction to restore a database; as well as In response to the instruction, The first shard is restored from the first backup location to the first storage node, and in parallel, the second shard is restored from the second backup location to the second storage node.

11. The method according to claim 10, further comprising: In response to the instruction to restore the database, metadata is created that associates the new tenant with the first shard and the second shard, the first shard with the first storage node, and the second shard with the second storage node.

12. The method according to claim 10, further comprising: storing a copy of the first shard on a third storage node; as well as A copy of the second shard is stored on a fourth storage node.

13. The method according to claim 12, further comprising: In response to an instruction to restore the database, metadata is created that associates the new tenant with the first shard and the second shard, the first shard with the first storage node and the third storage node, and the second shard with the second storage node and the fourth storage node.

14. The method according to claim 9, further comprising: receiving a first request from a requestor to access a first shard; In response to the first request: determining that the first fragment is not stored in any storage node of the plurality of storage nodes of the storage layer; as well as In response to the determination: issuing an instruction to restore the first shard from the first backup location to a first storage node among the plurality of storage nodes; as well as An identifier of the first storage node is returned to the requestor.

15. The method according to claim 14, further comprising: receiving a second request from the requestor to access the first shard; as well as In response to the second request: determining that a first fragment is stored in a first storage node among the plurality of storage nodes; as well as In response to determining that the first shard is stored in the first storage node, an identifier of the first storage node is returned to the requester.

16. The method according to claim 9, further comprising: receiving a first request from a requester to access a first shard and a second shard; In response to the first request: Determining that the first fragment and the second fragment are not stored in any storage node of the plurality of storage nodes of the storage layer; as well as In response to the determination: issuing instructions to restore the first shard from the first backup location to a first storage node among the plurality of storage nodes, and in parallel, restore the second shard from the second backup location to a second storage node among the plurality of storage nodes; as well as An identifier of the first storage node and an identifier of the second storage node are returned to the requester.

17. A non-transitory computer readable medium storing program code executable by a computing system to: storing a backup of a first fragment of a first database table of the database in a first backup location, the first fragment including a first key range of the first database table; storing a backup of a second shard of the first database table in a second backup location, the second shard including a second key range of the first database table; receiving an instruction to restore a database; as well as In response to the instruction, The first shard is restored from the first backup location to the first storage node, and in parallel, the second shard is restored from the second backup location to the second storage node.

18. The non-transitory computer readable medium of claim 17, the program code being executable by a computing system to: storing a copy of the first shard on a third storage node; storing a copy of the second shard on a fourth storage node; and In response to an instruction to restore the database, metadata is created that associates the new tenant with the first shard and the second shard, the first shard with the first storage node and the third storage node, and the second shard with the second storage node and the fourth storage node.

19. The non-transitory computer readable medium of claim 17, the program code being executable by a computing system to: receiving a first request from a requestor to access a first shard; In response to the first request: determining that the first fragment is not stored in any storage node of the plurality of storage nodes of the storage layer; as well as In response to the determination: issuing an instruction to restore the first shard from the first backup location to a first storage node among the plurality of storage nodes; Returning the identifier of the first storage node to the requester; receiving a second request from the requestor to access the first shard; as well as In response to the second request: determining that a first fragment is stored in a first storage node among the plurality of storage nodes; as well as In response to determining that the first shard is stored in the first storage node, an identifier of the first storage node is returned to the requester.

20. The non-transitory computer readable medium of claim 17, the program code being executable by a computing system to: receiving a first request from a requester to access a first shard and a second shard; In response to the first request: Determining that the first fragment and the second fragment are not stored in any storage node of the plurality of storage nodes of the storage layer; as well as In response to determining: issuing instructions to restore the first shard from the first backup location to a first storage node among the plurality of storage nodes, and in parallel, restore the second shard from the second backup location to a second storage node among the plurality of storage nodes; as well as An identifier of the first storage node and an identifier of the second storage node are returned to the requester.

Images