Techniques for Replication Checkpoints During Disaster Recovery

Abstract

During file storage replication between file systems in different cloud infrastructure regions, techniques are described for checkpointing multiple key ranges in parallel and simultaneously. In one embodiment, multiple range threads that process multiple key ranges, each a single thread per key range, create checkpoints for their respective key ranges in parallel and simultaneously after processing a predetermined number of B-tree keys. In one embodiment, each thread requests a lock on a central checkpoint record and updates a status byte in alternation while continuing to process B-tree keys within the key range for which it is responsible. In one embodiment, upon the occurrence of a failure event, either a system crash or a thread failure, each thread resumes B-tree key processing from the B-tree keys after the most recent checkpoint. In one embodiment, two generation numbers are assigned to two groups of key-value pairs of the processed B-tree, one before and one after a failure event within the key range.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Non - Provisional Patent Application No. 18 / 304,161, entitled "TECHNIQUES FOR REPLICATION CHECKPOINTING DURING DISASTER RECOVERY", filed on April 20, 2023, and U.S. Non - Provisional Patent Application No. 18 / 304,226, entitled "TECHNIQUES FOR MAINTAINING DATA CONSISTENCY DURING DISASTER RECOVERY", filed on April 20, 2023. All of these U.S. non - provisional patent applications claim the benefit and priority under 35 U.S.C. 119(e) of U.S. Provisional Patent Application No. 63 / 352,992, filed on June 16, 2022, U.S. Provisional Patent Application No. 63 / 357,526, filed on June 30, 2022, U.S. Provisional Patent Application No. 63 / 412,243, filed on September 30, 2022, and U.S. Provisional Patent Application No. 63 / 378,486, filed on October 5, 2022. The disclosures of these are hereby incorporated by reference in their entirety for all purposes.

[0002] Field This disclosure generally relates to file systems. More particularly, but not by way of limitation, techniques are described for checkpointing multiple key ranges in parallel and simultaneously while maintaining data consistency when a failure event occurs during file storage replication between file systems in different cloud infrastructure regions.

Background Art

[0003] Background Replication processes for disaster recovery can fail for a variety of reasons, including crashes, lease expirations, object storage failures, deployment failures, patch application, etc. Checkpoints play an important role during the file system replication process by periodically saving the necessary information, minimizing the risk of data loss or corruption, saving time, and allowing restart from a consistent point without having to start from the beginning. Additionally, parallel processing is used to improve performance. However, there is a need to improve checkpoints and data consistency during replication. Summary of the Invention

[0004] Brief Summary The present disclosure generally relates to file systems. More particularly, but not by way of limitation, techniques are described for checkpointing multiple key ranges in parallel and simultaneously during file storage replication between file systems within different cloud infrastructure regions. Various embodiments are described herein, including methods, systems, programs, code, or instructions executable by one or more processors, and non-transitory computer-readable storage media storing the same. Means for Solving the Problems

[0005] In one embodiment, a technique including a method is provided, the method including receiving, by a computing system, a request for file system replication between a source file system and a target file system, the source file system and the target file system being in different regions, the method further including creating, by a first processing thread of the computing system, a checkpoint in a first key range after meeting a predefined requirement for processing key-value pairs of a binary tree (B-tree) within the first key range, requesting, by the first processing thread of the computing system, a lock for updating a central record after creating the checkpoint in the first key range, creating, by a second processing thread of the computing system, a checkpoint in a second key range after meeting a predefined requirement for processing key-value pairs of a B-tree within the second key range, requesting, by the second processing thread of the computing system, a lock for updating the central record after creating the checkpoint in the second key range, and granting, by the computing system, the lock for updating the central record to the first processing thread, the request by the first processing thread being earlier than the request by the second processing thread.

[0006] In yet another embodiment, the lock prevents the second processing thread from updating the central record.

[0007] In yet another embodiment, the method further includes further processing, by the first processing thread, key-value pairs of the B-tree within the first key range after updating the central record, and granting, by the computing system, the lock for updating the central record to the second processing thread before the first processing thread completes the update of the central record.

[0008] In yet another embodiment, the method further includes, by a second processing thread, further processing key-value pairs of the B-tree within a second key range while waiting for the lock.

[0009] In yet another embodiment, the first processing thread operates independently of the second processing thread.

[0010] In yet another embodiment, processing the key-value pairs of the B-tree within the first key range by the first processing thread is executed in parallel with processing the key-value pairs of the B-tree within the second key range by the second processing thread.

[0011] In yet another embodiment, the method further includes, when detecting a system failure, resuming processing of the key-value pairs of the B-tree within the first key range from the B-tree keys after the checkpoint within the first key range by the first processing thread.

[0012] In yet another embodiment, the method further includes, when detecting a failure of the first processing thread, resuming processing of the key-value pairs of the B-tree within the first key range from the B-tree keys after the checkpoint within the first key range by a third processing thread.

[0013] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium having instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0014] In various embodiments, the non-transitory computer-readable medium stores computer-executable instructions that, when executed by one or more processors, cause one or more processors of a computer system to perform one or more of the methods disclosed herein.

[0015] In various embodiments, a computer program product includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0016] The features, embodiments, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

DETAILED DESCRIPTION OF THE INVENTION

[0018] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of an embodiment. However, it is evident that various embodiments may be practiced without these specific details. Each figure and description are not intended to be limiting.

[0019] This disclosure generally relates to file systems. More particularly, but not by way of limitation, techniques are described for checkpointing multiple key ranges in parallel and simultaneously during file storage replication between file systems in different cloud infrastructure regions. When the system crashes during the replication process, there are several issues. One is how to resume from the last point and complete it without repeating the whole process. The second issue is that the failed thread may suddenly start unexpectedly and collide with another thread that has taken over the failed thread, which may cause data corruption. The third issue is how to hold the checkpoint information in an accessible place even after a crash without performance issues for recovery.

[0020] The techniques disclosed in this disclosure create an efficient recovery scheme by splitting a B-tree into multiple key ranges, each having its own checkpoint. As a result, all these key ranges can resume and recover in parallel and simultaneously from their respective latest checkpoints. Thus, the recovery performance is much better than that of only one checkpoint for a large key range. Further, one range thread is assigned to process a key range, and all range threads operate independently. Each key range may have a checkpoint record for tracking the processed information. The checkpoint record may have a range-level lock for updating this record by one or more range threads processing that key range. Further, all range threads may update a central master checkpoint record for their respective statuses to achieve synchronization between threads. The central master checkpoint record may have a replication job-level lock for updating this master record by all range threads.

[0021] Furthermore, in certain embodiments, the technology uses a generation number (GenNum) to distinguish different generations of deltas that are processed during the delta generation process (within the source file system) or the delta application process (within the target file system) when a failure event occurs. The generation number can prevent data corruption or collision issues that could lead to data inconsistencies during replication that occur when both an unreliable failed thread and an alternate thread (i.e., a thread that takes over from the failed thread) update the same record to process the same key range.

[0022] Finally, in some embodiments, the proposed solution compresses the information for recovery into a concise format such as a manifest file for each checkpoint passed from the source file system (FS) to the target file system. Thus, the concise information is stored in memory with a small footprint, and the recovery performance can be improved.

[0023] Explanation of Terms in Certain Embodiments The "Recovery Time Objective" (RTO) refers, in certain embodiments, to the period of time that a user needs to make a replication available within a secondary (or target) region after a failure occurs within the availability domain (AD) of the primary (or source) region, regardless of whether the failure is planned or unplanned.

[0024] The "Recovery Point Objective" (RPO) refers, in certain embodiments, to the maximum allowable range with respect to the time of data loss between a failure of the primary region (usually due to an unplanned failure) and the availability of the secondary region.

[0025] In certain embodiments, a "replicator" may refer to a component (e.g., a virtual machine (VM)) within the data plane of a file system that uploads differences to a remote object store (i.e., an object storage service) when the component is located in the source region or downloads differences from the object storage for applying the differences when the component is located in the target region. The replicator is formed as a fleet (i.e., a plurality of VMs or replicator threads) called a replicator fleet and can execute the inter-region (or cross-region) replication process (e.g., uploading differences to the target region) in parallel.

[0026] In certain embodiments, a "delta generator" (DG) may refer to a component within the data plane of a file system that extracts the differences (i.e., changes) between the keys and values of two snapshots when the component is located in the source region or applies differences to the latest snapshot in the B-tree of the file system when the component is located in the target region. The delta generator in the source region can use multiple threads (referred to as delta generator threads or range threads for multiple split B-tree key ranges) to execute the extraction of differences (or B-tree scanning) in parallel. The delta generator in the target region can use multiple threads to apply the downloaded differences to the latest snapshot in parallel.

[0027] For the purposes of the present disclosure, in certain embodiments, a "shared database" (SDB) may refer to a key-value store that components (e.g., a replicator fleet) within both the control plane and the data plane of a file system can read from and write to in order to communicate with each other. In certain embodiments, the SDB may be part of a B-tree.

[0028] The "file system communicator" (FSC) may refer to, in certain embodiments, the file manager layer that runs on a storage node within the data plane of a file system. This service is useful for file creation requests, deletion requests, read requests, and write requests, and cooperates with the FNS server (e.g., Orca) to service I / O to clients. The replicator fleet can communicate with a number of storage nodes, thereby distributing file system data read / write operations among the storage nodes.

[0029] "Blob" may refer to, in certain embodiments, a data type for storing information (e.g., a formatted binary file) in a database. Blobs are generated during replication by the source region and uploaded to an object store (i.e., object storage) within the target region. A blob may include the keys and values of a binary tree (B-tree) as well as file data. Blobs within an object store are referred to as objects. The key-value pairs of the B-tree and the data associated with them are packed together into the blob that is uploaded to the object store within the target region.

[0030] In certain embodiments, a "manifest" may refer to information transmitted by a file system in a source region (referred to herein as the source file system) to a file system in a target region (referred to herein as the target file system) to facilitate an inter-region replication process. There are two types of manifest files: a master manifest and a checkpoint manifest. A range manifest file (or master manifest file) is created by the source file system at the start of the replication process and describes information (e.g., B-tree key ranges) required by the target file system. A checkpoint manifest file is created after a checkpoint in the source file system and notifies the target file system of the number of blobs included in the checkpoint and uploaded to the object store, whereupon the target file system can download that number of blobs.

[0031] In certain embodiments, a "difference" may refer to the differences identified between two specific snapshots after a replicator has recursively visited all nodes of a B-tree (also referred to herein as scanning the B-tree). A difference generator identifies key-value pairs of the B-tree with respect to the differences, traverses the B-tree nodes, and retrieves the file data associated with the B-tree keys. The difference between two snapshots may include multiple blobs. The term "difference" may include blobs and manifests when used in the context of uploading information by a source file system to an object store and downloading by a target file system from the object store.

[0032] In certain embodiments, an "object" may refer to a partial set of information representing the entire difference during an inter-region replication cycle, and is stored in an object store. An object may be sized in the order of several megabytes and stored at a specific location within a bucket of the object store. An object may contain many differences (i.e., blobs and manifests). A blob uploaded and stored in an object store is called an object.

[0033] In certain embodiments, a "bucket" may refer to a container that stores objects in a compartment within an object storage namespace (tenancy). In the present disclosure, a bucket is used by a source replicator to store differences protected using server-side encryption (SSE), and is also used by a target replicator to download changes and apply them to a snapshot.

[0034] In certain embodiments, "difference application" may refer to the process of applying differences downloaded by a target file system to the latest snapshot to create a new snapshot. Difference application may include analyzing a manifest file, applying snapshot metadata, inserting B-tree keys and values into a B-tree, and storing data associated with B-tree keys (i.e., file data or the data portion of a blob) in local storage. Snapshot metadata is created and applied at the start of a replication cycle.

[0035] In certain embodiments, a "region" may refer to a logical abstraction corresponding to a geographical area. Each region can include one or more connected data centers. A region is independent of other regions and can be separated by a vast distance.

[0036] End-to-end inter-region replication architecture An end-to-end inter-region replication architecture provides new techniques for end-to-end file storage replication and security between file systems within different cloud infrastructure regions. In certain embodiments, a file storage service generates differences between snapshots within a source file system and, upon disaster recovery, transfers the differences and associated data via high-throughput object storage to recreate new snapshots on a target file system located in a different region. The file storage service utilizes new techniques to achieve scalable, reliable, and restartable end-to-end replication. New techniques for ensuring secure transfer and consistency of information during end-to-end replication are also described.

[0037] In the context of the cloud, a realm refers to a logical collection of one or more regions. Realms are typically separated from each other and do not share data. Within a region, the data centers within the region can be organized into one or more availability domains (ADs). Availability domains are separated from each other, are fault tolerant, and have a very low probability of failing simultaneously. An AD is configured such that a failure in one AD within a region is unlikely to affect the availability of other ADs within the same region.

[0038] Current disaster recovery practices can include taking periodic snapshots and resynchronizing those snapshots to another file system within a different availability domain (AD) or region. Resynchronization is manageable and maintained by the customer but lacks a user interface for displaying progress, is a slow serialized process, and is not easy to manage as data grows over time.

[0039] Accordingly, different approaches are needed to address these and other issues. The file storage replication of a cloud service provider (e.g., Oracle Cloud Infrastructure (OCI)) disclosed in this disclosure is based on incremental snapshots and provides a consistent point-in-time view of the entire file system by propagating the differences in the changed data from the primary AD within a region to the secondary AD within the same or a different region. As used herein, the primary site (or source side) may refer to the location where the file system is located and where the replication process for disaster recovery is initiated (e.g., an AD or a region). The secondary site (or target side) may refer to the location where the file system receives information from the file system within the primary site during the replication process and becomes the new operational file system after disaster recovery (e.g., an AD or a region). The file system located at the primary site is called the source file system, and the file system located at the secondary site is called the target file system. Accordingly, the primary site, source side, source region, primary file system, or source file system (referring to one of the file systems on the source side) may be used interchangeably. Similarly, the secondary site, target side, target region, secondary file system, or target file system (referring to one of the file systems on the target side) may be used interchangeably.

[0040] The file storage service (FSS) of the present disclosure supports complete disaster recovery for failover or failback with minimal management effort. Failover is a series of actions to make the secondary site / target site the primary / source (i.e., start providing services for the workload), which may include planned failover and / or unplanned failover. A planned failover (sometimes called a planned migration) is initiated by the user to perform a planned failover from the source side (e.g., source region) to the target side (e.g., target region) without data loss. An unplanned failover is the case where, for example, due to a disaster, the source side stops unexpectedly and the source side is lost, so the user needs to start using the target side. Failback is to restore the primary side / source side to become the primary / source again before the failover. Failback may occur when the user wants to reuse the source side as the primary AD after the planned failover or unplanned failover and the trigger event (e.g., power outage) ends, by reversing the failover process. The user can resume either from the last point in time on the source side before the trigger event or from the latest changes on the target side. The replication process described in the present disclosure can maintain the identity of the file system after round-trip replication. In other words, the source file system can resume providing services for the workload again after performing a failover and then a failback.

[0041] The techniques disclosed in this disclosure (e.g., methods, computer-readable media, and systems) use consistent snapshot information to replicate the differences between snapshots from a source region to multiple remote (or target) regions, and then scan (or recursively visit) all keys and values within one or more file trees (e.g., B-trees) of the source file system (referred to herein as "scanning the B-tree" or "scanning keys") to construct consistent information (e.g., the differences or discrepancies between the keys and values of two snapshots created at different times), including region-to-region replication of file system data and / or metadata. The constructed consistent information is in blob form and is transferred to the remote side (e.g., the target region) using an object interface, such as an object store (described later), so that the target file system on the remote side can immediately detect the information transferred through the object interface and start downloading and applying it. This process is realized using a control plane and can be extended to thousands of file systems and hundreds of replication machines. Both the source file system and the target file system can operate simultaneously and asynchronously. Operating simultaneously means that the data upload process by the source file system and the data download process by the target file system can occur simultaneously. Operating asynchronously means that the source file system and the target file system can each operate at their own pace without waiting for each other at all stages, e.g., with different start times, end times, processing speeds, etc.

[0042] In one embodiment, multiple file systems may exist in the same region and are represented by the same B-tree. Each of these file systems within the same region can be replicated independently across regions. For example, file system A may have a set of parallel execution replicator threads that scan the B-tree to perform replication of file system A. File system B, represented by the same B-tree, may have another set of such parallel execution replicator threads that scan the same B-tree to perform replication of file system B.

[0043] Regarding security, cross-region replication is completely secure. Information is transferred securely and applied securely. The disclosed technology provides separation between the source region and the target region such that keys are not shared between the two without being encrypted. Thus, if the source key is involved, the target is not affected. Further, the disclosed technology includes ways to read keys, convert those keys into a certain format, and upload and download those keys securely. Since different keys are created and used in different regions, separate keys are created at the target and applied to the information with a target-centric security mechanism. For example, FSS generates a session key that is only valid during one replication cycle or session to encrypt data uploaded from the source region to the object store and decrypt data downloaded from the object store to the target region. Separate keys are used locally within the source region and the target region.

[0044] In the disclosed technology, each upload process and download process via the object store during replication has different pipeline stages. For example, the upload process has multiple pipeline stages including scanning a B-tree to generate a difference, accessing storage I / O, and uploading data (or blobs) to the object store. The download process has multiple pipeline stages including downloading data, applying the difference to a snapshot, and storing the data in storage. Each of these pipelines also includes parallel processing threads to improve the throughput and performance of the replication process. Further, the parallel processing threads can take over a failed processing thread and resume the replication process from the point of failure without restarting from the beginning. Thus, the replication process is highly scalable and reliable.

[0045] Figure 1 shows an exemplary concept of the target recovery point in time (RPO) and the target recovery time (RTO) for an unplanned failover according to an embodiment. The RPO is the maximum allowable range of data loss between a primary site failure and the availability of the secondary site (usually specified in minutes). As shown in Figure 1, the primary site A102 encounters an unplanned incident at time 110 and triggers the failover replication process by copying the latest snapshot and its delta to the secondary site B104. The information first copied reaches the secondary site B104 at time 112. The primary site A102 completes the copy of information to the secondary site B104 at time 114, and the secondary site B104 completes the replication process at time 116. Thus, the secondary site B104 becomes fully operational at time 116. As a result, the user's data is not accessible within the primary site A110 from point 110 until the point 116 where the data becomes available again. Thus, the RPO is the time between point 110 and point 116. For example, if there is data equivalent to 10 minutes that the user is not interested in, the RPO is 10 minutes. If the data loss exceeds 10 minutes, the RPO is not met. An RPO of 0 means synchronous replication.

[0046] RTO is the time it takes for the secondary to become fully operational after a failure, so that the user can access the data again (usually specified in minutes). RTO is considered from the perspective of the secondary site. Referring again to Figure 1, the primary site A102 starts the failover replication process at time 120. However, the secondary site B104 remains operational until time 122 when it recognizes the incident (or power outage) at the primary site A102. Therefore, the secondary site B104 stops its service at time 122. The secondary site B104 becomes fully operational at time 126 using the same failover replication process as described for RPO. Therefore, RTO is the time between 122 and 126. Here, the secondary site B104 can take over the role of the primary site. However, for customers using the primary site A102, the service loss is between times 120 and 126.

[0047] The primary (or source) site is where the action is taking place, and the secondary (or target) site is inactive and cannot be used until a disaster occurs. However, the customer may be provided with a point in time to continue using for test-related activities at the secondary site. This relates to how the customer sets up the replication, how the customer can start using the target if any problems occur, and how the customer can return to the source after the source has failed over.

[0048] Figure 2 is a simplified block diagram showing an architecture for inter-region remote replication according to an embodiment. In Figure 2, the end-to-end replication architecture shown includes two regions: a source region 290 and a target region 292. Each region may include one or more file systems. In one embodiment, the end-to-end replication architecture includes data planes 202 and 212, a control plane (only control APIs 208a-n and 218a-n are shown), local storage 204 and 214, an object store 260, and a key management service (KMS) 250 for both the source region 290 and the target region 292. Figure 2 shows only one file system 280 within the source region 290 and one file system 282 within the target region 292 for simplicity. If there are two or more file systems in one region, the same replication architecture is applied to each pair of source and target file systems. In one embodiment, multiple inter-region replications can occur simultaneously between each pair of source and target file systems by utilizing parallel processing threads. In some embodiments, one source file system can be replicated to different target file systems located in the same target region. Further, file systems within a region may share resources. For example, certain resources within the KMS 250, the object store 260, and the data plane may be shared by many file systems within the same region, depending on the implementation.

[0049] The data plane within the architecture includes local storage nodes 204a - n and 214a - n, as well as replicators (or replicator fleets) 206a - n and 216a - n. The control API hosts within each region perform all orchestration between different regions. The FSS receives from a customer a request to set up replication between a source file system 280 and a target file system 282 where the customer's data will be moved. The control plane 208 obtains the request, performs resource allocation, and notifies the replicator fleet 206a - n within the source data plane 202 to start uploading data 230a from different snapshots to the object storage 260 (or sometimes called differential upload). An API is available to assist the customer in setting the target time and the recovery time objective (RTO) of the replication. The replication model disclosed in this disclosure is a "push - based" model based on snapshot differentials, that is, the source region starts the replication.

[0050] As used herein, the data 230a and 230b transferred between the source file system 280 and the target file system 282 are general terms and may include an initial snapshot, keys and values of different B - trees between two snapshots, file data (e.g., fmap), snapshot metadata (i.e., a set of B - tree keys of snapshots reflecting different snapshots taken within the source file system), and other information (e.g., manifest files) that helps facilitate the replication process.

[0051] Regarding the data plane of the inter-region replication architecture, the replicator is a component within the data plane of the file system. The replicator performs differential generation or differential application for the file system according to the region where the file system is located. For example, the replicator fleet 206 within the file system 280 of the source region performs the generation and replication of the difference 230a. The replicator fleet 216 within the file system 282 of the target region downloads the differences 230b and applies them to the latest snapshot within the file system 282 of the target region. The file system 282 of the target region can also use the control plane and workflows to ensure end-to-end transfer.

[0052] All incremental operations are based on snapshots, which are existing resources within file storage as a service. A snapshot is a point in time, data point, or image of what is happening within the file system and is executed periodically within the file system 280 of the source region. In the very first replication (e.g., where replication has not been obtained before), FSS obtains a base snapshot, which is a snapshot of all the contents of the source file system, and transfers all of that content to the target system. In other words, the replicator reads from the storage layer of that specific file system and stores all the data in the object storage bucket.

[0053] After the data plane 202 of the source file system 280 uploads all the data 230a to the object storage (or object store) 260, the source-side control plane 208 notifies the target-side control plane 218 that there is new work to be done on the target side, and then this notification is relayed to the target-side replicator. Thereafter, the target-side replicators 216a~n start downloading objects (e.g., initial snapshots and deltas) from the object storage bucket 260 and applying the deltas captured on the source side.

[0054] For a base copy (e.g., the entire contents of the file system up to a point in time ranging from the past 5 days to 5 years), the upload process may take time. To assist in meeting service-level goals regarding time and performance, the source system 280 can take replication snapshots at specific intervals, such as once an hour. The source side 280 can then transfer all the data within that hour to the target side 282 and take new snapshots every hour. If there is any cache with many changes, the replication can be set to a shorter replication interval.

[0055] To illustrate the above, consider a situation where a first snapshot is created on a file system within a source region (referred to as the source file system). Replication is performed periodically, and thus the first snapshot is replicated to a file system within a target region (referred to as the target file system). Thereafter, when some update is performed within the source file system, a second snapshot is created. If an unplanned power outage occurs after the second snapshot is created, the source file system attempts to replicate the second snapshot to the target file system. During failover, the source file system may well identify the difference (i.e., the delta) between the first snapshot and the second snapshot, which includes the keys and values of the B-tree, and the file data associated therewith, within the B-tree representing both the first snapshot and the second snapshot. Next, deltas 230a and 230b are transferred from the source file system to the target file system via the object store 260 within the target region, and the target file system recreates the second snapshot by applying the deltas to the first snapshot previously established within the target region. When the second snapshot is created on the target file system, the failover replication process is complete and the target file system is ready to operate.

[0056] Regarding the control plane and its application programming interfaces (APIs), the control plane provides instructions for the data plane that includes replicators as executors to execute instructions. Storage (204 and 214) and replicator fleets (206 and 216) are both within the data plane. The control plane is not shown in Figure 2. As used herein, a "cycle" can refer to a period that starts when the source file system 280 begins to transfer data 230a to the target file system 282 and ends when the target file system 282 has received all the data 230b and completed the application of the received data. The data 230a - b is captured on the source side and then applied on the target side. When all the changes on the target side are applied to the cycle, the source file system 280 takes another snapshot and starts another cycle.

[0057] The control APIs (208a - n and 218a - n) are a set of hosts within the overall architecture of the control plane and execute the configuration of the file system. The control APIs are responsible for communicating state information between different regions. State machines that track various state activities within a region, such as the progress of a job, the location of keys, and future tasks to be executed, are distributed across multiple regions. All this information is stored in the control plane of each region and communicated between regions via the control APIs. In other words, the state information relates to the details of the life cycle, the details of the differences, and the life cycle of the resources. The state machine can also help track the progress of replication and cooperate with the data plane to estimate the time taken for replication. Therefore, the state machine can provide the user with status regarding whether the replication is proceeding as expected and the normality of the job.

[0058] Furthermore, the communication between the control APIs (208a - n) of the source file system 280 and the control APIs (218a - n) of the target file system 218 in a different region includes the transfer of snapshots and the metadata for creating an exact copy from the source to the target. For example, when a customer periodically takes snapshots within the source file system, the control plane can ensure that snapshots of the same user, including the tracking, transfer, and recreation of metadata, are created in the target file system.

[0059] The object store 260 in FIG. 2 (also referred to as an "object" herein) is an object storage service (e.g., Oracle's object storage service) that enables reading blobs and writing files for archival purposes. The advantages of using an object store are, first, ease of configuration, second, ease of streaming data to the object store, and third, having the advantage of secure streaming as a reliable repository for maintaining information, all of which are because there is no network loss, the data can be immediately downloaded, and it exists permanently. Direct communication between replicators within the source region and the target region is possible, but direct communication requires the configuration of an inter-region network, which is not scalable and difficult to manage.

[0060] For example, if there is a large amount of data being moved from a source to a target, the source can upload the data to the object store 260, and the target 282 does not need to wait for all the information uploaded to the object store 260 to start downloading. Thus, both the source 280 and the target 282 can operate continuously and simultaneously. The use of the object store enables the system to scale and achieve a higher throughput. Further, a key management service (KMS) 250 can control access to the object store 260 to ensure security. In other words, the source tries to move the data out of the source region as fast as possible and hold the data somewhere so that the data is not lost before it can be applied to the target.

[0061] Compared to using a network pipe with packet loss and recovery issues, the use of the object store 260 between the source region and the target region enables continuous data streaming where hundreds of file systems can be written from the source region to the object store, and at the same time, the target region can apply hundreds of files simultaneously. Thus, data streaming via the object store can achieve high throughput. Further, both the source region and the target region can operate at their own speeds for uploading and downloading.

[0062] Every time a user changes some data in the source file system 280, a snapshot is taken and the difference before and after the change is updated. These changes are accumulated in the source file system 280 and can be streamed to the object store 260. The target file system 282 can detect that the data is available in the object store 260 and immediately download the changes and apply them to that file system. In some embodiments, only the differences are uploaded to the object storage after the base snapshot.

[0063] In some embodiments, the replicator can communicate with many different regions (e.g., from Phoenix to Ashburn and further to other remote regions), and the file system can manage many different endpoints on the replicator. Each replicator 206 within the source file system 280 can maintain a cache of these object storage endpoints, and further, in cooperation with the KMS 250, generate a transfer key (e.g., a session key) for encrypting the data addresses of the data in the object storage 260 (e.g., server-side encryption or SSE) to protect the data stored in the bucket. There is one master bucket for each AD within the target region. A bucket is a container that stores objects in a compartment within the object storage namespace (tenancy). Since all remote clients can communicate with the bucket and write information in a specific format, the information of each file system can be uniquely identified, preventing the mixing of data from different customers or file systems.

[0064] The object store 260 is a high-throughput system, and the techniques disclosed in this disclosure can utilize the object store. In one embodiment, the replication process includes multiple pipeline stages, a B-tree scan within the source file system 280, storage IO access, data upload to the object store 260, data download from the object store 260, and differential application within the target file system 282. Each stage includes parallel processing threads that participate in improving the performance of data streaming from the source region 290 to the target region 292 via the object store 260.

[0065] In one embodiment, each file system within the source region may include a set of replicator threads 206a - n that are executed in parallel to upload the differences to the object store 260. Each file system within the target region may also include a set of replicator threads 216a - n that are executed in parallel to download the differences from the object store 260. Since both the source side and the target side operate asynchronously simultaneously, the source can upload as fast as possible, while the target can start downloading after detecting that the differences are available in the object store. Thereafter, the target file system applies the differences to the latest snapshot and deletes the differences in the object store after application. Thus, the FSS consumes little space in the object store, and the object store has a very high throughput (e.g., gigabytes of transfer).

[0066] In one embodiment, multiple threads are also executed in parallel for storage IO access (e.g., DASD) 204a - n and 214a - n. Thus, all processes related to the replication process, including accessing storage, uploading snapshots and data 230a from the source file system 280 to the object store 260, and downloading snapshots and data 230b to the target file system 282, include multiple threads that are executed in parallel to perform data streaming.

[0067] File storage is a local service of AD. When a file system is created, that file system is within a specific AD. To transfer or replicate data from one file system to another file system within the same region or a different region, artifact (also called manifest) transfer may need to be used.

[0068] As an alternative to using an object store to transfer data, a network connection between remote machines (e.g., between source and target replicator nodes) can be set up, and VCN peering can be used to use Classless Inter-Domain Routing (CIDR) for each region.

[0069] Referring again to FIG. 2, the Key Management System (KMS) 250 provides security for replication and provides storage services to a cloud service provider (e.g., OCI). In some embodiments, the file systems 280 on the source (or primary) side and the target (or secondary) side use separate KMS keys, and key management is hierarchical. The reason for using separate keys is that if the source is compromised, an attacker cannot decrypt the target using the same key. The FSS has a three-tier key architecture. Since the source and target use different keys during data transfer, the source must first decrypt the data, re-encrypt it using an intermediate key, and then re-encrypt the data on the target side. The FSS defines a session, and each session is one data cycle. A key is created to transfer data in that session. In other words, a new key is used for each new session. In other embodiments, a key can be used for two or more sessions (e.g., two or more data transfers) before creating another key. The key is not transferred via the object store 260, and the key is only available on the source side and is not visible from outside the source for security reasons.

[0070] The replication cycle (also called a session) is periodic and adjustable. For example, the replicators (206a~n and 216a~n) execute replication once per hour. The cycle starts when a new snapshot is created on the source side 280 and ends when all the differences 230b have been applied to the target side 282 (i.e., the target has reached the DONE state). Each session is completed before another session starts. Therefore, there is always only one session and no overlap between sessions.

[0071] Secret management (i.e., replication using the KMS) processes the transfer of confidential materials between the source (primary) file system 290 and the target (or secondary) file system 292 using the KMS 250. The source file system 280 calculates the differences, reads the file data, and then decrypts the file data in cooperation with the key management service using the encryption key of the local file system. Next, the source file system 280 generates a session key (referred to as a delta encryption key (DEK)), encrypts it to become an encrypted session key (referred to as a delta transfer key (DTK)), and transfers the DTK to the target file system 282 via the respective control planes 208 and 218. The source file system 280 further encrypts the data 230a using the DEK and uploads the encrypted data 230a to the object store 260 via the Transport Layer Security (TLS) protocol. Next, the object store 260 uses server-side encryption (SSE) to ensure the security for the storage of the data (e.g., differences, manifests, and metadata) 230a.

[0072] The target file system 282 securely obtains the encrypted session key DTK via the control plane 218 (using HTTPS via inter-region API communication), decrypts the session key DTK via the KMS 250 to obtain the DEK, and places the DEK at a location within the target region 292. When a replication job is scheduled within the target file system 282, the DEK is provided to a replicator (one of the replication fleets 216a - n), and the replicator uses this key to decrypt the data (e.g., the delta including file data) 230b downloaded from the object store 260 for application and re-encrypts the file data using the local file system key.

[0073] Replication between the source file system 280 and the target file system 282 is a parallel process, and both the source file system 280 and the target file system 282 operate at their own paces. When the source side completes the upload (which can occur before the target download process), the source side cleans up the memory and removes all keys. When the target completes the application of the delta to the latest snapshot, it similarly cleans up the memory and removes all keys. The FSS service also releases the KMS key. In other words, there are two copies of the session key, one within the source file system 280 and another within the target file system 282. Both copies are deleted at the end of each session, and a new session key is generated for the next replication cycle. This process ensures that the same key is not used for different purposes. Further, the session key is encrypted by the file system key, creating a two-fold protection. This is to ensure that only a specific file system can use this session key.

[0074] Figure 3 is a simplified schematic diagram of components involved in inter-region remote replication according to an embodiment. In one embodiment, components called a differential generator (DG) 310 within source region A 302 and 330 within target region B 304 are part of a replicator fleet 318 and operate on thousands of storage nodes within the fleet. The replicator 318 within source region A makes remote procedural calls (RPCs) to the differential generator 310 (e.g., getting a set of keys and values, locking a block, etc.), and collects keys, values, and data pages of the B-tree from a direct-access storage device (DASD) 314, which is a replication storage service for accessing storage and is regarded as a data server. The DG 310 within source region A is a helper to the replicator 318, divides the key range of the difference, and packs all keys / values within a specific range into a blob that is returned to the replicator 318. There are multiple storage nodes 322 and 342 connected to DASDs 314 and 334 in both regions, and each node contains a large number of disks (e.g., over 10TB).

[0075] In one embodiment, file system communicators (FSCs) 312 and 332 in both regions are metadata servers that help update the source file system for user updates to the system. FSCs 312 and 332 are used for file system communication, and the differential generator 310 is used for replication. Both DG 310 and 330 and FSCs 312 and 332 are metadata servers. User traffic passes through FSCs 312 and 332 and DASDs 314 and 334, while replication traffic passes through the DG. In an alternative embodiment, the function of the FSC can be merged with the function of the DG.

[0076] In one embodiment, the shared databases (SDBs) 316 and 336 of both regions are key-value stores, and through these components, both the control plane and the data plane (e.g., the replicator fleet) can read and write for themselves to communicate with each other. The control planes 320 and 340 of both regions can put new jobs into queues in their respective shared databases 316 and 336, and the replicator fleets 318 and 338 constantly read the queues in the shared databases 316 and 336, and when the replicator fleets 318 and 338 detect job requests, they can initiate file system replication. In other words, the shared databases 316 and 336 are conduits between the replicator fleet and the control plane. Further, the shared databases 316 and 336 are resources distributed across different regions, and the IO traffic between the shared databases 316 and 336 should be minimized. Similarly, the IO traffic with the DASD needs to be minimized so as not to affect the user's performance. However, the replication process may be adjusted because it is a secondary service compared to the primary service.

[0077] The replicator fleet 318 within the source region A can cooperate with the DG310 to start scanning the B-tree in the file system within the source region A, collect keys and values, and convert those keys and values into flat files or blobs to be uploaded to the object store. Once the data blobs (including keys and values and the actual data) are uploaded, the target can immediately apply those data blobs without waiting for a large number of blobs to be present in the object store 360. The object store 360 is located in the target region B for disaster recovery reasons. The goal is to push from the source to the target region B as quickly as possible and keep the data safe.

[0078] Optimize space by using lower-cost machines with smaller footprint, and schedule as many replications as possible while ensuring fair bandwidth allocation among those machines. To replicate thousands of file systems, there are multiple replicators. Replicator fleets 318 and 338 in both regions are run on virtual machines that can be automatically scaled up and down to build the entire fleet for running replications. Replicators and replication services can dynamically adapt based on capacity to support each job. If the load on one replicator is high, another replicator can be selected to share the load. Different replicators in the fleet can balance the load among themselves to ensure that jobs can continue and do not stop due to overloading individual replicators.

[0079] FIG. 4 is a simplified flowchart showing steps executed during inter-region remote replication according to an embodiment.

[0080] Step S1: When the customer sets up replication, the customer provides a source (or primary) file system (A) 402, a target (or secondary) file system (B) 404, and an RPO. File systems are uniquely identified by file system identification information (e.g., Oracle Cloud ID or OCID), which is a globally unique identifier for the file system. Data is stored in the file storage service (「FSS」) control plane database.

[0081] Step S2: The source (A) control plane (CP-A) 410 coordinates to periodically create system snapshots at regular intervals (less than the RPO), and notifies the data plane (including replicator / uploader 412) of the latest snapshot and the last snapshot that was successfully copied to the target (B) file system 404.

[0082] Step S3: CP-A410 notifies the replicator 412 (or uploader), which is a component within the data plane, to copy the latest snapshot.

[0083] S3a: The replicator 412 of the source (A) scans the B-tree to calculate the difference between two specific snapshots. The existing key infrastructure is used to decrypt the file system data.

[0084] S3b: These differences 414 are uploaded to the object store 430 within the target (B) region (the data can be compressed and / or deduplicated during copying). This upload can be executed in parallel by multiple replicator threads 412.

[0085] Step S4: CP-A410 notifies the target (B) control plane (CP-B) 450 of the completion of the upload.

[0086] Step S5: CP-B450 calls the target replicator B452 (or downloader) to apply the differences.

[0087] S5a: The replicator B452 downloads the data 454 from the object store 430.

[0088] S5b: The replicator B452 applies these differences to the target file system (B).

[0089] Step S6: After the difference application is completed, CP-A410 is notified of the new snapshot now available at the target (B).

[0090] Step 7: The inter-region remote replication process repeats from Step S2 to Step S6.

[0091] Figure 5 is a simplified diagram showing a high-level concept of B-tree traversal according to an embodiment. The B-tree structure can be used within a file system. The difference generator traverses the B-tree and ensures the consistency of the traversal. In other words, the traversal checks that at the end of the traversal, the keys and values are as expected so that data corruption cannot occur, and captures all information between any two snapshots. The file system is a transactional file system that may be modified, and since another user may update the same transaction or data, the user needs to know about the changes and re-do the transaction.

[0092] The keys, values, and snapshots are immutable (i.e., they cannot be changed except that a garbage collector may remove them). As shown in Figure 5, there are many snapshots (Snapshot 1 to Snapshot N) in the file system. When the difference generator is traversing the B-tree keys (510 to 560) in the source file system, the garbage collector 580 may come in and clean up the keys of the snapshots that it considers garbage, so the snapshots may be removed. When the difference generator traverses the B-tree keys, the difference generator needs to ensure that the keys associated with the remaining snapshots (i.e., the keys not removed by the garbage collector) are copied. When keys, such as 540 and 550, are removed by the garbage collector 580, the B-tree page can be shrunk, for example, from 2 pages before garbage collection to 1 page after garbage collection. A way for the difference generator to ensure consistency when traversing the B-tree keys is for the garbage collector 580 to confirm that it has not changed or deleted any of the keys in the page (or section between two snapshots) that the difference generator has just traversed (e.g., between two keys). Once consistency is confirmed, the difference generator collects the keys and sends them to the replicator for processing and uploading.

[0093] The B-tree key can indicate what has changed. The techniques disclosed in this disclosure can determine which B-tree keys are new and what has been updated between two snapshots. The diff generator can collect the metadata portion, keys and values, and related data, and then send it to the target. The target can understand that the received information is within the range of the two snapshots and applies to the target file system. The diff generator (or a thread of the diff generator) scans the section between two keys, confirms its consistency, and then uses the last end key as the next start key for the next scan. This process is repeated until all keys are checked, and the diff generator collects related data each time the consistency is confirmed.

[0094] For example, when a file is changed within the file system (e.g., created, deleted, and then recreated), this process creates multiple versions of the corresponding file directory entry. During the replication process, the garbage collector may clean up (or remove) the version of the file directory entry corresponding to the deleted file, which may cause a consistency issue called a whiteout. A whiteout occurs when there is a mismatch between the source file system and the target file system, because the target file system may fail to reconstruct the original snapshot chain that includes the changed file. The disclosed techniques can detect whiteout files (i.e., changed files affected by the garbage collector) during B-tree scanning, extract the version of the changed file that is not affected, and provide related information to the target file system within the same replication cycle to ensure the consistency between the source file system and the target file system by properly reconstructing the correct snapshot chain.

[0095] Figures 6A and 6B are diagrams showing the pipeline stages of inter-region replication according to an embodiment. The inter-region replication of the source file system disclosed in the present disclosure includes four pipeline stages, namely, the start of inter-region replication, the B-tree scan within the source file system (i.e., the differential generation pipeline stage), the storage IO access for retrieving data (i.e., the data read pipeline stage), and the data upload to the object store (i.e., the data upload pipeline stage) within the source file system. The target file system includes four pipeline stages in a similar but reverse order, namely, the preparation for inter-region replication, the download of data from the object store, the application of the difference within the target file system, and the storage IO access for storing the data. Figure 6A shows the four pipeline stages within the source file system, but the same concept applies to the target file system as well. Figure 6B shows the processes and interactions between the components involved in the pipeline stages. These pipeline stages can all operate in parallel. Each pipeline stage operates independently and can pass information to the next pipeline stage when the processing at the current stage is completed. Each pipeline stage receives a portion of the overall bandwidth and is guaranteed not to use more than necessary. In other words, resources are fairly allocated among all jobs. When no other jobs are operating within the system, the operating job can acquire as many resources as possible.

[0096] Threads within each pipeline stage also execute tasks independently of each other in parallel (or simultaneously) within the same pipeline stage (i.e., if a thread fails, it does not affect other threads). Further, the tasks (or replication jobs) executed by threads at each pipeline stage are restartable, i.e., if a thread fails, a new thread (also called an alternative thread) can take over from the failed thread and continue the original task from the last successful point.

[0097] In some embodiments, the B-tree scan can be performed using parallel processing threads within the source file system 280. The B-tree can be divided into a plurality of key ranges between the first key and the last key in the file system. The number of key ranges can be determined by the customer. A plurality of (e.g., about 8 to 16) range threads can be used for the B-tree scan for each file system. One range thread can perform a B-tree scan of one key range, and all range threads operate in parallel simultaneously. The number of threads used varies depending on factors such as the size of the file system, the availability of resources, and the bandwidth for balancing resource and traffic congestion. Usually, the number of key ranges is more than the number of available range threads for fully utilizing the range threads. Therefore, the B-tree scan is scalable and can be processed by simultaneous parallel scans (e.g., using multiple threads).

[0098] After the difference generator scans the pages, if some keys are missing and thus some keys are inconsistent, the system can remove the ongoing uncommitted transactions and return to the starting point to scan again. During the repetition of the B-tree scan due to the inconsistency, the difference generator can ignore the missing keys and the data associated with them by not collecting them because these associated data are regarded as garbage in order to minimize the amount of information to be processed or uploaded to the target side. Therefore, the B-tree scan and data transfer can be made more efficient. Further, the difference generator does not need to wait for the garbage collector to remove the information to be deleted before scanning the B-tree keys. For example, keys have dependencies on each other. If a key or iNode points to a block that has been deleted by the garbage collector or should be deleted, the system (or the difference generator) can itself understand that a particular block is garbage and the difference generator does not need to carry it.

[0099] The differential generator usually makes no changes on the source side (e.g., does not delete the keys or blocks of data considered as garbage), and simply does not copy them to the target side. The B-tree scanning process and garbage collection are asynchronous processes. For example, when the block of data pointed to by a key no longer exists, the file system can flag the key as garbage, notify that the key should not be modified (e.g., is immutable), and only the garbage collector can remove the key. The differential generator can continue scanning the next key without waiting for the garbage collector. In other words, the differential generator and the garbage collector can proceed at their own pace.

[0100] In FIG. 6A, when the source region starts an inter-region replication process that can include multiple file systems, the main threads 610a - n select a replication job (one job per file system). The main thread of the file system (e.g., 610a or 610 for later use) within the source region (i.e., the source file system) communicates with the differential generator 620 (shown in FIG. 6B) to obtain the number of key ranges requested by the customer and update the corresponding records in the SDB 622. Once the main thread 610 of the source file system knows the number of key ranges required, it further creates a set of range threads 612a - n based on the number of key ranges required. These range threads 612a - n are executed by the differential generator 620. These range threads 612a - n initialize the GETKEYVAL buffer 640 (shown in FIG. 6B), update the checkpoint record 642 in the SDB 622 (shown in FIG. 6B), and perform storage I / O access 644 by exchanging information with the DASD I / O threads 614a - n.

[0101] In one embodiment, each main thread 610 is responsible for monitoring all range threads 612a - n that it creates. During replication, the main thread 610 may generate a master manifest file that summarizes the entire replication. The range threads 612a - n generate a range manifest file that includes the number of key ranges (i.e., the subdivision of the entire replication), and then generate a checkpoint manifest (CM) file for each range to provide updates to the target file system regarding the number of blobs per checkpoint, where the checkpoints are created during B - tree traversal. One checkpoint is created by the range thread 612. When the main thread 610 determines that all range threads 612a - n are complete, it creates a final checkpoint manifest (CM) file that includes an end - of - file marker, and then uploads the CM file to the object store so that the target file system can understand the progress within the source file system. The CM file includes an overview of all individual ranges, such as the number of ranges, the final state of the checkpoint records, and other information.

[0102] Range threads 612a~n are used for parallel processing to significantly reduce the time of B-tree traversal of a large source file system. In one embodiment, the B-tree keys are divided into ranges of approximately equal size. One range thread can perform a B-tree traversal of one key range. The number of range threads 612a~n used varies depending on factors such as the size of the file system, resource availability, and bandwidth to balance resources, the amount of data generated, and traffic congestion. Usually, the number of key ranges is about two to four times more than the number of available range threads 612a~n to fully utilize the range threads. Each of the range threads 612a~n has a dedicated buffer (GETKEYVAL) 640 that contains jobs available for work. Each range thread 612 operates independently of other range threads and periodically updates the checkpoint record 642 in the SDB622.

[0103] Range threads 612a~n may need to collect file data (e.g., FMAP) associated with the B-tree keys and request IO access 644 to storage when traversing the B-tree (i.e., when recursively visiting all nodes of the B-tree). These IO requests are queued by each range thread 612 so that the DASD IO threads 614a~n (i.e., the data read pipeline stage) can handle those IO requests. These DASD IO threads 614a~n are common threads shared by all range threads 612a~n. After the DASD IO threads 614a~n obtain the requested data, the data is placed in the output buffer 646 to serialize the data into a blob so that the replica object threads 616a~n (i.e., the data upload pipeline stage) can upload it to the object store located in the target region. Each object thread selects an upload job that can include a portion of all the data to be uploaded, and all object threads execute the uploads in parallel.

[0104] FIG. 7 is a diagram showing a hierarchical structure in the FSS data plane according to an embodiment. In FIG. 7, the replicator fleet 710 includes four layers: a job layer 712, a delta generator client 714, encryption / DASD IO 716, and an object 718. The replicator fleet 710 is a single process that serves to exchange information with a storage fleet 720, a KMS 730, and an object storage 740. In an embodiment, the job layer 712 polls the SDB 704 for a job 706 that is queued as either an upload job or a download job. The replicator fleet 710 includes VMs (or threads) that select enqueue replication jobs up to their maximum capacity. A replicator thread may own a part of a replication job, but coordinates with another replicator thread that owns the remaining part of the same replication job to complete the entire replication job simultaneously. The replication job executed by the replicator fleet 710 is restartable in that if a replicator thread fails during replication, another replicator thread can take over and continue from the last successful point to complete the job that the failed replicator thread originally owned. If a strayed replicator thread (e.g., a replicator thread that fails and restarts) competes with another replicator thread, the FSS can avoid the conflict by using a mechanism called a generation number to cause both replicator threads to update different records.

[0105] The differential generator client layer 714 performs a B-tree scan by accessing the differential generator server 724 in which the B-tree in the storage fleet 720 exists. The encryption / DASD IO layer 716 assumes the roles of security and storage access. After the B-tree scan, the replicator fleet 710 may request IO access via the encryption / DASD IO layer 716 to access the DASD range 722 of file data associated with the differences identified during the B-tree scan. The replicator fleet 710 and the storage fleet 720 both periodically update the status of the control API 702 (e.g., checkpoint and lease of the replicator fleet 710) via the SDB 704, enabling the control API 702 to trigger an alarm or execute an action if necessary.

[0106] During the inter-region replication process, the encryption / DASD IO layer 716 exchanges information with the KMS and the FSK fleet 730 on the target side to create a session key (or snapshot encryption key), and uses the FSK for encryption and decryption of the session key. Finally, the object layer 718 is responsible for uploading differential and file data from the source file system to the object store 740 and downloading them from the object store 740 to the target file system.

[0107] The data plane of the FSS is responsible for differential generation. The data plane stores FSS data using a B-tree, which includes various types of key-value pairs including, but not limited to, a leader block, a superblock, an iNode, a file name key, a cookie map (cookies associated with directory entries), and a block map (also called FMAP in the case of file content data).

[0108] These B-tree keys are processed together by the replicator and the differential generator within the data plane. An algorithm for calculating the pairs of keys and values (i.e., part of the difference) that have changed between two specific snapshots within the file system continuously reads the keys, returns the keys to the replicator using the transaction budget, and finally ensures that the transaction is confirmed to obtain a consistent pair of keys and values for processing.

[0109] In other embodiments, the difference generation and calculation may be scalable. A scalable approach can calculate the difference (i.e., the change in the pairs of keys and values) between two snapshots by utilizing multiple threads to divide the B-tree into many key ranges. A pool of threads (i.e., differential generators) can perform a scan of the B-tree (i.e., traverse the B-tree) and calculate the differences in parallel.

[0110] FIG. 8 shows a simplified exemplary binary large object (BLOB) format according to an embodiment. A blob is a data type for storing information (e.g., binary data) in a database. A blob is generated by a source region during replication and uploaded to an object store. The target region needs to download and apply the blob. Blob and object may be used interchangeably in the context.

[0111] During the B-tree scan, when the difference generator encounters the iNode of a specific file (i.e., data content) and its block map (also called FMAP, data associated with the B-tree key), the difference generator cooperates with the replicator to traverse all the pages within the blocks (FMAP blocks) within the DASD range pointed to by the FMAP, read them into the data buffer, decrypt the data using the local encryption file key, put it into the output buffer, and serialize it into a blob for the replicator to upload to the object store. In other words, the difference generator needs to collect all the FMAPs of the identified differences in order to obtain all the data related to the differences between the two snapshots.

[0112] The snapshot differences stored in the object store may span multiple blobs (or objects if stored in the object store). The blob format of these blobs includes a key, a value, and, if present, data associated with the key. For example, in Figure 8, the snapshot difference 800 includes at least three blobs 802, 804, and 806. The first blob 802 includes a prefix 810 indicating the key and value types, the key length, and the value length, followed by a key 812 (key 1) and a value 814 (value 1). The second blob 804 includes a prefix 820 (key and value types, key length, and value length), a key 822 (key 2), a value 824 (value 2), a data length 826, and data 828 (data 2). In the prefix 820 of this second blob 804, since this blob includes additional data 828 associated with the key 822, the key and value types are fmap. The third blob 830 includes a format similar to that of the first blob 810, for example, a prefix 830, a key 832 (key 3), and a value 834 (value 3).

[0113] The data is decrypted, collected, and then written to a blob. All processes are executed in parallel. Multiple blobs can be processed and updated simultaneously. When all processes are complete, the data is written in blob format (shown in Figure 8) and can then be uploaded to the object store in the format (shown in Figure 9) or path name.

[0114] Figure 9 shows an exemplary replication bucket format according to an embodiment. A "bucket" can refer to a container that stores objects in a compartment within an object storage namespace. In one embodiment, a bucket is used by a source replicator to store data protected using server-side encryption (SSE) technology and is also used by a target replicator to download changes and apply them to a snapshot. Replication data for all file systems in a target region can share a bucket within that region.

[0115] The data layout of a bucket in an object store has a directory structure that includes, but is not limited to, a file system ID (e.g., Oracle Cloud ID), a difference including a start snapshot number and an end snapshot number, a manifest that describes the content of the information in the object's layout, and blobs. For example, the bucket in FIG. 9 includes two objects 910 and 930. The first object 910 includes two differences 912 and 920. This object starts with a path name 911 (e.g., ocid1.filesystem.oc1.iad...) that uses the source file system ID as a prefix, followed by a first difference 912 generated from snapshot 1 and snapshot 2, and a second snapshot 920 generated from snapshot 2 and snapshot 3. Each difference includes one or more blobs that represent the content of that difference. The first difference 912 has two blobs 914 and 916 stored in the order of generation. The second difference 920 includes only one blob 922. Each difference also includes a manifest that describes the content of the information in the layout of this difference, for example, manifest 918 of the first difference 912 and manifest 924 of the second difference 920. The manifest in the bucket is content that describes the differences, such as the file system number and snapshot range. The manifest can be a master manifest, a range manifest, or a checkpoint manifest depending on the stage of the replication process.

[0116] The second object 930 also includes two differences 932 and 940 in a similar format starting with path name 931. The two objects 910 and 930 in the bucket come from different source regions, namely IAD for object 910 and PHX for object 930. After the blobs are applied, the corresponding information in the layout can be removed to reduce space utilization.

[0117] The final manifest object (i.e., the checkpoint manifest, CM file) is uploaded from the source region to the object store, and the source file system indicates to the target region that it has completed uploading the snapshot difference of a specific object. The source CP transmits this event to the target CP, and the target CP can notify the target DP via the SDB to trigger the download process of that object by the target replicator.

[0118] The control plane within the source region or the target region coordinates all of the replication workflow and drives the replication of data. The control plane performs the following functions: (1) creates the underlying system snapshot for creating the differences, (2) determines when such snapshots need to be created, (3) initiates replication based on the snapshots, (4) monitors the replication, (5) triggers the download of the differences by the secondary (or target side), and (6) indicates to the primary (or source) side that the snapshots have reached the secondary.

[0119] The file system has several operations for processing its resources, including but not limited to creating, reading, updating, and deleting (CRUD). These operations are usually synchronized within the same region, starting a workflow when the file system receives an HTTPS request from the API server, making changes in the backend to create a resource, and returning a response to the customer. Resources are divided into a source region and a target region. The state is maintained for the same resource between the source region and the target region. Therefore, there is asynchronous communication between the source region and the target region. A customer can create or update a resource by contacting the source region, and these creations or updates can be automatically reflected in secondary or auxiliary resources within the target region. The state machine in the control plane also targets recovery in many aspects, including but not limited to fleet failures, key management failures, disk failures, and object failures.

[0120] Regarding the application programming interfaces (APIs) within the control plane, there are various APIs for users to configure replication. The control API for any new resource only functions within the region where the object was created. In the target file system, a field named "IsTargetable" can be set in the API to ensure that the target file system receiving replication cannot be accidentally used by consumers. In other words, setting this field to false means that consumers can view the target file system, but no one can export the target file system or access any data within the live system. Since export is not a read-only permission but a read / write permission for export, any export can potentially modify the data. Therefore, during the replication process, exports are not permitted to prevent any changes to the target file system. Consumers can only access data within old snapshots that have already been replicated. Any newly created or replicated file system can have this field set to true. The reason is that the target can only obtain data from a single source. Otherwise, collisions may occur when data is written or deleted. The system needs to know whether the target file system in use is already part of some replication. Setting the "IsTargetable" field to "true" means that replication is not in progress, and setting it to "false" means that the target file system cannot be used.

[0121] Regarding inter-region communication between components of the control plane, the primary resource on the source file system is called an application, and the auxiliary (or secondary) source on the target file system is called an application target. Source objects and target objects, when created, have a single replication relationship. Both objects can be updated only from the source side, including changes to compartments, editing of details, or deletion. If the user wants to delete the target side, the replication itself can be deleted. In the case of a planned failover, it is possible to delete the source side, and both the source side and target replication are deleted. In the case of an unplanned failover, the source side is not available, so only the target replication can be deleted. In other words, there are two resources for a single replication, and those resources should be kept in a synchronized state. There are various workflows for updating metadata on both the source side and the target side. Additionally, inter-region APIs for retries, fault handling, and failover are also part of the inter-region communication process.

[0122] When creating the necessary security and other related artifacts, the source uploads the security and artifacts to the object store, starts a job at the target (i.e., notifies the target that the job is available), and the target can then start downloading the artifacts (e.g., snapshots or deltas). The target then continues to look for an end-of-file marker (also referred to herein as a checkpoint manifest (CM) file) within the object store. The CM file is used as a mechanism for the source side and the target side to communicate the completion of the upload of the object during the replication process. At every checkpoint, the source side uploads this CM file containing information such as the number of blobs uploaded up to this checkpoint, enabling the target side to download this number of blobs and apply them to the current snapshot. This CM file is a mechanism for the source side to communicate to the target side that the upload of the object to the object store is complete and for the target to start working on that object. In other words, the target continues to download until there are no more objects in the object storage. Thus, this approach enables concurrent processing on both the source side and the target side.

[0123] Figure 10 is a flowchart showing a state machine for simultaneous source upload and target download according to an embodiment. As previously explained, both the source file system and the target file system can perform replication simultaneously and thus can each have its own state machine. In one embodiment, each file system can have its own state machine while sharing some common job-level states. In Figure 10, the source file system has states 1030 - 1034 for session key generation and transfer in addition to states 1002 - 1018 for performing data upload. The target file system has states 1050 - 1068 related to data download. The session key can be generated at any time within the source file system while the differences are being uploaded to the object storage. Thus, the session key transfer has its own state sequence 1030 - 1034. In Figure 10, the target file system cannot start the replication download process (i.e., Ready_to_Reconcile state 1050) until it receives an indication that at least an object has been uploaded to the object storage by the source file system (i.e., Mainfest_Copied state 1014) and that it is ready to download the session key (i.e., Copied_DTK state 1034).

[0124] In the source file system, multiple functional blocks such as a snapshot generator, a control API, and a delta monitor are part of the CP. The replicator fleet is part of the DP. The snapshot generator is responsible for periodically generating snapshots. The delta monitor periodically monitors the progress of the replicator in replication-related tasks, including the creation of snapshots and the replication schedule. When the delta monitor detects that the replicator has completed a replication job, it transitions the state to a copied state on the source side (e.g., Manifest_Copied state 1014) or a replicated state on the target side (e.g., Replicated state 1058). In certain embodiments, multiple file systems can concurrently perform replication from a source region to a target region.

[0125] Referring to FIG. 10, in one embodiment, in the source file system, in the concurrent mode state machine, after creating a snapshot signal to the delta monitor indicating that a snapshot has been generated, the snapshot generator. The delta monitor that executes the CP replication state (CpRpSt) workflow is responsible for initiating the upload of snapshot metadata to the object store on the target side. The snapshot metadata can include the type of snapshot, snapshot identification information, the time of the snapshot, etc. The CpRpSt workflow sets the Ready_to_Copy_Metadata state 1002 for the replicator fleet to start copying the metadata. When the replicator acquires a replication job, it creates a copy of the snapshot metadata (i.e., Snapshot_Metadata_Copying state 1004) and uploads those copies to the object store. When all replicators have completed uploading the snapshot metadata, the state is set to the Snapshot_Metadata_Copied state 1006. Thereafter, the CpRpSt workflow continues to poll the source SDB for the session key.

[0126] Here, the CpRtSt workflow returns control to the differential monitor to monitor the differential upload process that transitions to the Ready_to_Copy state 1008 indicating that the differential calculation is scheduled. Next, the source CP API sends a request to the replicator to start the next stage of replication by uploading the differential and creating a copy of the manifest. The replicator that selects the replication job can start creating a copy of the manifest (i.e., the Mainfest_Copying state 1010). When the source file system completes the copy of the manifest, it transitions to the Manifest_Copied state 1014 and at the same time notifies the target file system that it can start the internal state (the Ready_to_Reconcile state 1050).

[0127] As described above, the session key can be generated by the source file system during the upload of data. The replicator of the source file system communicates with the target KMS vault to obtain the master key that can be provided by the customer and creates a session key (referred to herein as the differential encryption key or DEK). Next, the replicator encrypts the session key using the local file system key (FSK: file system key) (here, it becomes the encrypted DEK, also referred to herein as the differential transfer key (DTK)). Then, the DTK is stored in the SDB within the source region and reused by the replicator thread during the replication cycle. The state machine transitions to the Ready_to_Copy_DTK state 1030.

[0128] The source file system transfers the resource identification information of DTK and KMS to the target API, and then the target API puts those resource identification information into the SDB within the target region. During this transfer process, the state machine is set to the Copying_DTK state 1032. When the CpRpSt workflow in the source file system finishes polling the source SDB for the session key, the target file system downloads the session key (DTK) and sends a notification to the target side indicating that it is ready to use that session key to decrypt the downloaded differences for application. Then, the state machine transitions to the Copied_DTK state 1034. The replicator on the target side retrieves the DTK from the SDB and requests the KMS API to decrypt the DTK into the plaintext DEK (i.e., the decrypted session key).

[0129] When the source file system completes the upload of data for a specific replication cycle including session key transfer, the difference monitor notifies the target control API of the status such as validity confirmation information and transitions to the X-region_Copied_Done state 1016. This can occur before the target file system completes the download and application of the data. The source file system further cleans up the memory and removes all keys. Then, the source file system transitions to the Awaiting_Target_Response state 1018 and waits for a response from the target file system to start a new replication cycle.

[0130] As described above, the target file system cannot start the replication download process until it receives an indication that at least the object has been uploaded to the object storage by the source file system (i.e., the Mainfest_Copied state 1014), and that it is ready to download the session key (i.e., the Copied_DTK state 1034). When these two conditions are met, the state machine transitions to the Ready_To_Reconcile state 1050. Next, in the Reconciling state 1052, the target file system starts an adjustment process with the source side, such as synchronizing snapshots of the source file system and the target file system, takes a snapshot, and performs some internal CP management operations including generating statistical values. This internal state includes communication within the target file system between the delta monitor and the CP API.

[0131] After the adjustment process is completed, the replication job is passed to the target replicator (i.e., the Ready_to_Replicate state 1054). The target replicator monitors the checkpoint manifest (CM) file uploaded by the source file system. The CM file is marked by the target. Then, the target replicator thread starts to download the manifest and apply the downloaded and decrypted deltas (i.e., the Replicating state 1056). The target replicator thread also reads the FMAP data blocks from the blobs downloaded from the object store, communicates with the local FSK service to obtain the file system key FSK, and the FSK is used to re-encrypt each FMAP data block and store it in local storage.

[0132] When the source file system finishes uploading data, it updates the final CM file by setting the end-of-file (eof) field to true and uploads it to the object store. As soon as the target file system detects this final CM file, it finishes downloading the blobs and applies them, and the state machine transitions to the Replicated state 1058.

[0133] After the target file system applies all the deltas (or blobs), it continues to download the snapshot metadata from the object store and inputs the information of the source file system's snapshot into the target file system's snapshot (i.e., the Snapshot_metadata_Populating state 1060). When the target file system's snapshot is input, the state machine transitions to the Snapshot_Metadata_Populated state 1062.

[0134] In the Snapshot_Deleting state 1064, the target file system deletes all the blobs in the object store for the blobs that have been downloaded and applied to the latest snapshot. Then, the target control API notifies the target delta monitor when the blobs in the object store are deleted and proceeds to the Snapshot_Deleted state 1066. The target file system further cleans up the memory and removes all the keys. The FSS service also releases the KMS keys.

[0135] Once the target DP finishes applying the differences and cleaning up, it uses the target control API to verify the validity of the source file system's status and whether it has received an X-region_Copied_Done notification from the source file system. If the notification has been received, the target difference monitor transitions to the X-region DONE state 1068 and sends an X-region DONE notification to the source file system. In some embodiments, the target file system checks whether the end of the file exists for all key ranges and all upload processing threads because all objects uploaded to the object store have special markers such as file end markers in the CM file, thereby detecting whether the source file system has completed the upload.

[0136] Referring again to the state machine of the source file system, while in the Awaiting_Target_Response state 1018, the source file system checks whether the status of the target CP has changed to completed, indicating that all differences downloaded by the target have been applied and the file data has been stored locally. If the status of the target CP changes to completed, this marks the end of the replication cycle.

[0137] The source and target sides operate asynchronously. Once the source file system completes the replication upload, it notifies the target control API of the X-region_Copied_Done notification. Then, once the target file system completes the replication process, the difference monitor target communicates in the reverse direction with the source control API using the X-region DONE notification. The source file system returns to the Ready_to_Copy_Metadata state 1002 and starts another replication cycle.

[0138] FIG. 11 is an exemplary flow diagram showing the exchange of information between the data plane and the control plane within a source region according to an embodiment. The data plane components and the control plane components communicate with each other using a shared database (SDB), such as 1106. The SDB is a key-value store that both the control plane components and the data plane components can read from and write to. The data plane components include a replicator and a difference generator. The exchange of information between the components within source region A1101 and target region B1102 is also shown.

[0139] In FIG. 11, at step S1, the source control plane (CPa) 1103 requests the object store within the target region B (OSb) 1112 to create a bucket. At step S2, the source replicator (REPLICATORa) 1108 periodically updates the heartbeat status to the source SDB (SDBa) 1106. The heartbeat is a concept used to track the progress of replication executed by the replicator. The heartbeat can use a mechanism called lease, where the heartbeat is continuously updated each time the replicator works on a job, enabling the control plane to recognize the overall release information. For example, the byte count is continuously moving on the job. If the replicator cannot function properly, the heartbeat may become stale, and then another replicator can detect and take over, continuing the work on the remaining jobs. Therefore, if the system crashes midway, the system can accurately start from the last point based on the checkpoint mechanism. The checkpoint helps the system know where the last point of progress was and be able to continue from that point without re-executing the entire work.

[0140] In step S3, CPa1103 further requests the File System Service Workflow (FSW_CPa) 1104 to create snapshots periodically. In step S4, FSW_CPa1104 notifies CPa1103 about the new snapshot. In step S5, next, CPa1103 stores the snapshot information in SDBa1106. In step S6, REPLICATORa1108 polls SDB1106 for any changes to the existing snapshots. If a change is detected, in step S7, it retrieves the job specification. When REPLICATORa1108 detects a change to the snapshot in step S8, this initiates the replication process. In step S8, REPLICATORa1108 provides information about two snapshots (SNa and SNb) including the changes between the snapshots to the Difference Generator (DGa) 1110. In step S9, REPLICATORa1108 enters work item information such as the number of key ranges into SDBa1106. In step 10, REPLICATORa1108 checks the replication job queue in SDBa1106 to obtain work items. In step S11, it assigns those work items to the Difference Generator (DGa) 1110 to scan the B-tree keys of the snapshot (i.e., traverse the B-tree) and calculate the differences and the corresponding key-value pairs. In step 12, REPLICATORa1108 decrypts the file data associated with the identified B-tree keys and packs them together with the key-value pairs into blobs. In step 13, REPLICATORa1108 encrypts the blobs using the session key and uploads them as objects to OSb1112. In step S14, REPLICATORa performs a checkpoint and stores the checkpoint record in SDBa1106. This replication process (S8 - S14) repeats (as a loop) until all differences are identified and the data is uploaded to OSb1112.In step S15, REPLICATORa1108 then notifies SDBa1106 of the details of the replication job, and this detail is then passed to CPa1103 in step S16 and further relayed to CPb1114 as the final CM file in step S17. In step S18, CPb1114 stores the job details in SDBb1116.

[0141] The exchange of information between the data plane and the control plane within target region B is similar. At the end of applying the delta to the target file system, the control plane within target region B notifies the control plane within source region A that the snapshot has been successfully applied. Thereby, the control plane within source region A can start over using the new snapshot.

[0142] Authentication is performed on all components. There is an authentication mechanism that uses the replication ID and the file system number, from the replicator to the file system key (FSK). The key can be given to the replicator only if the replicator provides appropriate content. Thus, the authentication mechanism can prevent fraudsters from obtaining the decryption key. Other security mechanisms include blocking network ports. A component called the file system key server (FSKS) is a gatekeeper for properly checking the requester by checking metadata such as the job executed by the requester and other information. For example, assume that the replicator is trying to request the key of the file system. In that case, FSKS can check whether the replicator is associated with a specific job (e.g., whether the replication is actually associated with that file system) to confirm the validity of the requester.

[0143] Availability addresses situations where a machine can automatically restart after going down, or where services remain available while software deployment is in progress. For example, since all replicators are stateless, losing a replicator is transparent to the customer because another replicator can pick up and continue the job's work. The job's state is maintained not locally, but in a shared database and other reliable locations. The shared database is a service such as the database used by the control plane to maintain information about the file system and is based on a B-tree.

[0144] The system has thousands of storage nodes that enable any storage node to perform differential replication, so storage availability in the FSS of this disclosure is high. By utilizing many machines that can take over from each other in case of some failure, the availability of the control plane is increased. For example, the progress of replication is not simply blocked by the failure of a single control plane. Thus, there is no single point of failure. The availability of network access is made such that the source node does not become overloaded by using congestion management that includes various types of throttling.

[0145] Replication is durable by writing the replication state to the shared database and by utilizing checkpointing where the replicator is stateless. The replication process is idempotent. Idempotency can refer to deterministic reapplication where, in case an operation fails, retrying the same operation, for example, using the same key, upload process, or scan process, should result in the same outcome.

[0146] Operations within multiple regions should be equivalent. In the control plane, the actions taken should be stored. For example, in the case of repeated HTTP requests, an equivalence cache can be useful for remembering that a particular operation has been performed and that it is the same operation. For example, in the data plane, when a block is allocated, the block and the file system's file map key are written together. Thus, if the block is allocated again, the block can be identified. If the block is sealed, the write operation fails. The equivalence mechanism can know that the block has been sealed in the past and the write operation does not need to be retried. In yet another example, the equivalence mechanism stores a chain of steps that need to be performed for the processing of a particular key and value. In other words, the equivalence mechanism allows all operations to be checked to ensure they are in the correct state. Thus, the system can simply move on to the next step without repeating.

[0147] Atomic replay enables the application of deltas to start as soon as the first delta object reaches the object store when a snapshot is rolled back, for example, when going back from snapshot 10 to snapshot 5. To make the replay atomic, the entire delta needs to be maintained in the object store before it can be applied.

[0148] Regarding the expansion of replicators, the FSS of the present disclosure enables adding the number of replication machines (e.g., replicator virtual machines (“VMs”)) required to support many file systems. The number of replicators can be dynamically increased or decreased by considering the bandwidth requirements and availability of resources. Regarding the expansion of storage, thousands of storages can be used to parallelize the process and improve the working speed. Regarding the inter-region bandwidth, the bandwidth allocation is automatically adjusted, such as by adjusting all inter-region bandwidths by grasping the increase in latency and reducing the required speed, to ensure that each workload is not overused or does not exceed a predefined throughput limit. All replicator processors (or threads) have this function.

[0149] In the expansion of checkpoint storage, the uploader and downloader checkpoint the progress to persistent storage, and the shared storage is used as a work queue for dividing key ranges. If the checkpoint workload overly burdens the shared database, for the purpose of expansion, the checkpoint storage function can be added to the differential generator. The current workload of the shared database may consume less than 10 IOPs.

[0150] FIG. 12 is a schematic diagram showing a failback mode according to an embodiment. The failback mode enables restoring the primary side / source side to become the primary / source again before failover. As shown in FIG. 12, the primary AD 1202 includes the source file system 1206, and the secondary AD 1204 includes the target file system 1208. The secondary AD 1204 may exist in the same region or a different region from the region of the primary AD 1202.

[0151] In FIG. 12, snapshot 1 1220 and snapshot 2 1222 in the source file system 1206 exist before the failover due to a power outage event. Similarly, snapshot 1 1240 and snapshot 2 1242 of the target file system 1208 exist before the failover. When a power outage occurs in snapshot 3 1224 in the primary AD 1202, the FSS performs an unplanned failover 1250, and snapshot 3 1224 in the source file system 1206 is replicated to the target file system 1208 and becomes the new snapshot 3 1224. After the target file system 1208 starts operating, the customer can make changes to create snapshot 4 1246 for the target file system 1208.

[0152] If the customer decides to reuse the source file system again, the FSS service may execute a failback. When executing the failback, the user has two options: (1) the last point in time in the source file system before the trigger event 1251, or (2) the latest changes in the target file system 1252.

[0153] In the case of the first option, the user can resume from the last point in the source file system 1206 prior to the trigger event (i.e., snapshot 3 1224). In other words, since snapshot 3 1224 has previously failed over successfully to the target file system 1208, it becomes the snapshot for use after failback. To perform the failback 1251, the state of the source file system 1206 is changed to inaccessible. Next, the FSS service identifies the last point in the source file system 1206 before the failover was successful, snapshot 3 1224. The FSS may execute a clone of snapshot 3 1224 within the primary AD 1202 (i.e., a replication within the same region). Now, the primary AD 1202 returns to its initial settings before the power outage, and the user can reuse the source file system 1206 again. Since snapshot 3 1224 already exists in the file system being used, no data transfer from the secondary AD 1204 to the primary AD 1202 is required.

[0154] In the case of the second option, the user wants to reuse the source file system with the latest changes in the target file system 1208. In other words, since snapshot 4 1246 in the target file system 1208 was the latest change in the target file system 1208, it becomes the snapshot for use after failback. The failback process 1252 for this option includes reverse replication (i.e., reversing the roles of the source file system and the target file system for the replication process), and the FSS performs the following steps.

[0155] Step 1. The state of the source file system 1206 is changed to inaccessible. Step 2. Next, the FSS service identifies the latest snapshot in the successfully replicated target file system 1208, for example, snapshot 3 1244.

[0156] Step 3. The FSS service also detects the corresponding snapshot 3 1224 in the source file system 1206 and performs a clone (i.e., a copy within the same region).

[0157] Step 4. The FSS service starts reverse replication 1252 in the same process as described in relation to FIG. 4, but in the reverse direction. In other words, both the source file system 1206 and the target file system 1208 need to be synchronized, and then the target file system 1208 can upload the differences to the object store in the primary AD 1202. The source file system 1206 can download the differences from the object store, complete the application to snapshot 3 1224, and create a new snapshot 4 1226.

[0158] Here, the primary AD 1202 returns to the initial settings before the power outage, and the user can reuse the source file system 1206 again without transferring the data that already exists in both the source file system 1206 and the target file system 1208, for example, snapshots 1 to 3 (1220 to 1224) in the source file system 1206. This saves time and prevents unnecessary bandwidth.

[0159] Replication Checkpoints and Data Consistency for Disaster Recovery As described above, differential generation is a time-consuming task, especially for large file systems. If the B-tree keys of the source file system are divided into a number of key ranges of approximately the same size, each key range can be processed by a range thread of the differential generator in the data plane. Each range thread processing a key range can maintain an active checkpoint record and one or more generation records for tracking the B-tree scan and blob upload process. Both the checkpoint record and the generation record of differential generation can be stored in the source shared database (SDB). A checkpoint refers to, and may well represent, the state of the differences processed at a specific point in time for both the source file system and the target file system, and the differential generation in the source FS or the differential application in the target FS should continue from that point. Both the source FS and the target FS are checkpointable, i.e., both file systems can create a checkpoint after processing a predetermined amount of differences (e.g., key-value pairs of the B-tree or transformed blobs) or after either a predetermined period (e.g., several minutes), whichever is earlier, since the progress and pace of each range thread processing a key range may vary. For simplicity, the present disclosure may use a predetermined amount of differences. The generation record may include a generation number, and each generation number (GenNum) is associated with a group of blobs processed and uploaded by a range thread. In other words, the generation number may refer to or indicate the number of restarts or crashes during the differential generation and application process of replication, or the number of generations the differential generation or application job has passed through with respect to a change in thread ownership. The generation number (GenNum) may be stored in a central location accessible to all range threads (e.g., the source SDB or the central master checkpoint record), and the same GenNum may be used for each generation record. When the differential generation of a key range is completed, the memory space storing the checkpoint record and one or more generation records can be freed.

[0160] FIG. 13 is a simplified illustrative diagram showing key range processing for differential generation during inter-region replication according to an embodiment. In FIG. 13, the B-tree keys (i.e., the key portions of key-value pairs) of the source file system (FS) 1302 within the source region may be divided into three key ranges: key range 1 (1310), key range 2 (1312), and key range 3 (1314). In some embodiments, each key range may include 1000 keys, for example, key numbers K1 to K1000 of key range 1. The keys may be converted into blobs for uploading as objects to an object store. As previously explained in connection with FIG. 8, a blob may include key-value pairs of multiple B-trees (sometimes simply referred to as keys within a key range). Some B-tree keys may include associated file data and may occupy more space within the corresponding blob. Since the size of each blob may be fixed, each converted blob may include different numbers of key-value pairs (or keys). As a result, even if the number of keys in each key range is approximately the same, the same number of keys may be converted into different numbers of blobs for different key ranges. For example, a key range containing 1000 keys may be converted into 100 blobs for uploading. Another key range containing 1000 keys may be converted into only 50 blobs for uploading because some keys within this key range may have a larger size of associated data. Further, another key range containing 1000 keys may be converted into 200 blobs for uploading because most keys in this key range include small associated data. In some embodiments, after being divided into many key ranges, the key numbers are consecutive. However, the blob sequence number for each key range may start from B1 and continue until all keys within that key range are processed, which is because the range threads operating on different key ranges operate independently and convert keys into blobs at their own pace for uploading.Each blob may include a field called key_value count (kv_count) to indicate the number of B-tree keys the blob holds, which helps the target FS understand the number of keys downloaded by the target FS and the relationship between the downloaded blobs and keys.

[0161] For simplicity, Figure 13 assumes that one blob contains 10 keys. Thus, there are 100 blobs for each key range. For example, there are B1 - B100 for key range 1, B1 - B100 for key range 2, and B1 - B100 for key range 3. The key numbers are consecutive, such as K1 - K1000 for key range 1, K1001 - 2000 for key range 2, and K2001 - 3000 for key range 3. Each key range is processed by a range thread of the difference generator, and all range threads (T1 - T3) process the key ranges simultaneously, in parallel, and independently. If a thread (e.g., thread T2 1322) fails, a new range thread (e.g., T4) may take over. Range threads can function independently because they do not need to communicate with each other when processing their respective key ranges. Since each range thread can process the keys within its corresponding key range simultaneously without waiting for other range threads because the keys in different key ranges are different, range threads function simultaneously and in parallel. Thus, each range thread can proceed at its own pace for its corresponding key range. Additionally, each blob may contain a fixed number of keys. As a result, the source FS can know the same number as the processed keys. Furthermore, all key ranges start from generation number (GenNum) G1. GenNum increases by 1 each time a failure event occurs in the source FS. In Figure 13, all key ranges have three GenNums (G1, G2, and G3) due to two failure events: system crash 1320 and thread failure 1322.

[0162] In some embodiments, each range thread also maintains a checkpoint record (or called a range checkpoint record to distinguish it from the central master checkpoint record), and one or more generation records created when the differential generation process starts. If a failure event (either a system crash or a thread failure) occurs, a new generation record with a new GenNum incremented by 1 from the previous GenNum can be created. Thus, one generation record corresponds to one GenNum. For example, since two failure events occur, namely a system crash and a thread failure within another key range, range thread T1 has checkpoint record 1 (1350) for key range 1 and three generation records (1353a, 1352b, and 1352c).

[0163] Referring to FIG. 13, range thread T2 has checkpoint record 2 (1360) of key range 2 and three generation records (1362a - c). Two of the generation records (1362a - b) are for T2 due to system failure, and one (1362c) is for T4 due to the failure of thread T2. Range thread T3 has checkpoint record 3 (1370) of key range 3 and three generation records (1372a - c) because two failure events occur. Each key range has four checkpoints (C1, C2, C3, and C4), and each checkpoint targets 25 blobs. For example, checkpoint record 1 is created at point C1 when range thread T1 of key range 1 processes 25 blobs (i.e., up to blob number B25). Checkpoint record 1 is updated at point C2 when range thread T1 processes another 25 blobs up to B50, and so on. Since each range thread processes at its own pace and is independent of each other, the checkpoint times for each key range may be different, but the number of blobs processed per checkpoint is the same, e.g., 25 blobs at C1 for key range 1, 25 blobs at C1 for key range 2, and 25 blobs at C1 for key range 3. In some embodiments, the checkpoint may be created after a predetermined period rather than after a predetermined number of processed blobs.

[0164] In one embodiment, the generation record of each range thread is updated (i.e., increment the generation number GenNum) whenever either a system crash 1320 (or an event affecting all key ranges), e.g., between checkpoints C1 and C2, or a thread failure 1322, e.g., between checkpoints C2 and C3 for key range 2 only, occurs. In either situation, the differential generation process can resume in parallel and simultaneously by all range threads from the latest checkpoint within all key ranges. A system failure affects the entire system and thus all key ranges. Accordingly, all key ranges are marked with a system failure (i.e., "X"). However, a thread failure may occur only in the range thread that is processing a particular key range. In FIG. 13, range thread T2 fails (i.e., 1322) between checkpoints C2 and C3 within range 2 (i.e., marked as a filled black dot), but no thread failure occurs within the other key ranges (i.e., marked as white dots). As a result, a new range thread T4 (also referred to herein as an alternative range thread) takes over and creates a new generation record (1362c) with GenNum G3 accordingly. As shown in FIG. 13, since all key ranges execute the same differential generation process, the generation number (GenNum) of all generation records changes from G1 to G2 when a system failure 1320 occurs and from G2 to G3 when a thread failure 1322 occurs. In other words, all range threads maintain the same generation number.

[0165] In some embodiments, failed range threads may sometimes start up unexpectedly after a while and resume processing their key ranges. Since the new alternative range threads are also processing the same key range, both the failed range threads and the alternative range threads may update the same records and generate blobs for uploading to the object store. This situation may lead to data corruption. Therefore, a new generation record (e.g., 1362c) is created for the alternative range thread T4. The generation number (GenNum) within the generation record 1362c of thread T4 should start from G3 (i.e., increment from G2 to G3). However, since the GenNum within the generation record 1362b of the old thread T2 is unaware of its own failure, it remains at G2. As a result, the duplicate blob numbers processed and uploaded by both the failed range thread T2 and the new alternative range thread T4 may be associated with different GenNums, enabling the target FS to distinguish between the blobs processed by threads T2 and T4.

[0166] After a checkpoint is created, the information within the checkpoint record and the generation record is integrated for each key range and can be copied into a checkpoint manifest file that is uploaded to the object store. The checkpoint manifest file includes the generation number of the checkpoint for that key range and the associated blob group. This checkpoint manifest file may be downloaded by the target file system, which uses this information to download blobs from the object store and apply the keys, values, and file data to the B-tree. The checkpoint manifest file will be described in more detail later. In some embodiments, the blobs for each key range are of the form / REPL-<job ID> / .... / RANGE-<range ID> / BLOB-<blob sequence number>- <gennum>can be uploaded to the object storage path of that specific key range, such as. replication-job_id indicates a specific replication between the source file system and the target file system. The range ID indicates a specific key range for the blob upload. The uploaded blob is placed at a specific path under the starting blob sequence number that includes the associated GenNum.

[0167] Figure 13 shows the generation of differences within the source file system. However, since similar key range processing is performed within the target FS, it may be applicable to the application of differences within the target file system. In some embodiments, the target FS may include many key ranges, and one range thread processes one key range to perform the download of blobs and manifest files from the object store. Each range thread also maintains checkpoint records and generation records stored in the target shared database (target SDB) during the difference application process in case any system crash or thread failure occurs.

[0168] End-to-end process flow FIG. 14 is a flowchart showing an end-to-end process flow for differential generation and differential application during inter-region replication according to an embodiment. In step 1410 of FIG. 14, upon receiving a replication request, the source FS divides the B-tree key into a plurality of key ranges of approximately equal size, assigns a range thread to each key range, and executes differential generation simultaneously and in parallel. In step 1412, the source FS creates a range manifest file containing information including at least the total number of key ranges, and transfers it to the target FS via the object store to notify the target FS to prepare accordingly for the total number of key ranges being processed by the source FS. In step 1420, each range thread working on a key range within the source FS creates and updates checkpoint records and one or more generation records during the differential generation process. These checkpoint records and generation records can be stored in the source SDB. During the differential generation process, each range thread can resume the differential generation process for the key range it is responsible for based on the information in the checkpoint record if a failure event occurs. Further, the generation record for each key range is also updated, and data corruption that may occur due to a failure event can be resolved. Details of the update of these records will be described in more detail later.

[0169] In step 1422, after each checkpoint within the key range, the range thread working on that key range can copy and integrate the information within that checkpoint record and generation record into the checkpoint manifest file. In step 1424, each range thread uploads the checkpoint manifest file to the object store for the target FS to download. In step 1426, the target FS downloads the checkpoint manifest file for each key range, creates a key range that mirrors what is in the source FS, downloads the blobs based on the information in the checkpoint manifest file, and then may clean up the manifest file. The target FS recognizes that the group of blobs to be downloaded is from the last checkpoint stage of that key range when it detects the final checkpoint manifest file with the end-of-file (eof) field for the key range set to true. In step 1428, each range thread within the target FS may create and update checkpoint records and one or more generation records during the differential application process. These checkpoint records and generation records can be stored in the target SDB. In step 1430, the process repeats steps 1420 - 1428 until all range threads have completed generating the differences for the key ranges they are responsible for in the source FS and applying those differences to those key ranges in the target FS.

[0170] Data Consistency for Disaster Recovery As previously explained in connection with FIG. 13, the generation number (GenNum) can be used to resolve potential data corruption between two range threads that are processing the same key range. In some embodiments, each time a replicator obtains a replication job, a GenNum is assigned to all range threads that are processing different key ranges, and the GenNum is also recorded in the source shared database (SDB). When a new range thread starts, a generation record is created for either a new job or a handover job. The new range thread can check the current generation number in the source SDB and then increment the number by one. Thus, two range threads cannot write to the same generation record. For example, in FIG. 13, when the differential generation process starts, the generation number G1 is assigned to all generation records of the range threads (e.g., T1, T2, and T3). If the range thread T2 that is processing key range 2 fails, for example, due to a timeout (i.e., lease expiration) by not updating the heartbeat status, a new alternative range thread T4 can take over and resume the differential generation process (or can resume from the next key after the latest checkpoint created by thread T2). The new alternative thread T4 can check the source SDB and detect that the generation number of thread T2 is G2. Thus, the new alternative thread T4 can set its generation number to G3. As a result, threads T2 and T4 are processing the same key range 2 but can update different records (e.g., generation record 1362b and generation record 1362c), and the processed blobs thereof are associated with different generation numbers (G2 and G3). In one embodiment, the blobs processed and uploaded by threads T2 and T4 can be placed in different buffers assigned different generation numbers. If the target file system downloads the blobs uploaded by both threads T2 and T4, the target file system can discard the duplicate blob numbers associated with the lower generation number G2 and use the duplicate blob numbers associated with the higher generation number G3.

[0171] FIG. 15 shows a generation record for a difference generation process and a difference application process according to an embodiment. In FIG. 15, the generation record 1500 may include key-value pairs, one key-value pair for each generation record within the key range. The key-value pairs may be stored in tabular form (e.g., in a memory table stored within source SDB). In some embodiments, the fields of the key 1510 may include, but are not limited to, the type of replication, replication ID, range ID, and generation number (GenNum). Since both file systems contain their respective generation records in the same format, the type of replication may indicate whether this generation record is for the source file system or for the target file system. In the case of the source file system, the generation record may be used to track a group of blobs uploaded to the object store. In the case of the target file system, the generation record may be used to track a group of blobs downloaded from the object store. The replication ID can be either a source file system ID or a target file system ID for identifying the file system performing inter-region replication. The range ID indicates the key range with which this generation record is associated. The generation number is the generation in which a group of blobs is processed and uploaded by the range thread that owns this generation record (i.e., the number of restarts or system failures or changes in thread ownership). For example, after a difference generation job starts, the blobs that are processed and uploaded belong to the first generation (i.e., GenNum = 1). After the system crashes and restarts, the blobs that are processed and uploaded belong to the second generation (i.e., GenNum = 2), and so on.

[0172] In FIG. 15, in some embodiments, the field of value 1520 may include, but is not limited to, start_blob_seqno and end_blob_seqno. These fields represent the start blob sequence number and the end blob sequence number that define a group of blobs associated with the generation number of the key. For example, in table 1530 of thread T1, processing the blobs within key range 1 (1310) may include three key-value pairs to represent three generations of blobs processed by thread T1, as shown in FIG. 13. For simplicity and for the purpose of illustration, not all fields are listed in the table. The first key-value pair within the first generation record (i.e., the first entry in 1352a of FIG. 13 or table 1530) includes a key with range_id R1 and GenNum G1 associated with a group of blobs from sequence number (seqno or blob number) 1 (or B1) to 35 (or B35). When thread T1 completes blob number 35, the system crashes. Therefore, thread T1 creates a second generation record with GenNum incremented by 1 to G2. Thread T1 may resume the differential generation process for key range 1 from the next key after the previous checkpoint at B25 (shown in FIG. 13), i.e., from B26. Since blobs B25 to B35 have already been processed and uploaded to the object store, GenNum G2 records only blobs B36 and later.

[0173] The second key-value pair (i.e., the second generation record 1352b in FIG. 13 or the second entry in table 1530) includes a key with GenNum G2 associated with a group of blobs from blob number 36 to blob number 70, at which point a thread failure occurs in key range 2 (1312 in FIG. 13). No thread failure occurs in key range 1, but the GenNum within all generation records is updated. Thus, thread T1 creates a third generation record where the GenNum is incremented by one again to become G3. Thread T1 can resume the differential generation process for key range 1 from the next key after the previous checkpoint at B50 (shown in FIG. 13) until all 100 blobs are completed. Thus, the third key-value pair (i.e., the second generation record 1352c in FIG. 13) includes a value representing GenNum G3 and a group of blobs numbered 71 to 100.

[0174] For further explanation, key range 2 (1312 in FIG. 13) includes both a system crash at B35 and a thread failure at B70. FIG. 15 has two tables. Table 1532 includes two entries of key-value pairs representing two generations of the blobs (i.e., two generation records) processed by thread T2 in FIG. 13. Table 1534 includes one entry of a key-value pair representing one generation of the blob processed by thread T4 in FIG. 13. Returning to FIG. 13, key range 2 includes a system crash at B35 and a failure of thread T2 at B70. Thus, the first generation record (1362a) of thread T2 includes the first key-value pair with range ID R2 and GenNum G1 associated with the group of blobs from B1 to B35. Next, thread T2 creates a second generation record (1362b in FIG. 13 or the second entry in table 1532), increments the GenNum by one to G2, and resumes the differential generation process for key range 2 from the next key after the previous checkpoint B25 until it fails at B70. When thread T2 fails, a new alternative thread T4 takes over and creates a new generation record stored in table 1534. Thread T4 checks the source SDB and detects that the current GenNum for key range 2 is G2. Thus, thread T4 creates a third generation record (1362c in FIG. 13 or the first entry in table 1534), increments the GenNum by one to G3, and selects from the failure point B70 of thread T2 to finish all the blobs (up to B100) in key range 2. Thus, table 1534 of thread T4 includes GenNum G3 associated with blobs B71 to B100.

[0175] However, a failed thread may unexpectedly start and continue processing the blob. For example, thread T2 that fails at B70 may start and continue processing the blob up to B85 which also fails again from B70. As a result, the second key-value pair (or the second generation record) in table 1532 of thread T2 includes GenNum G2 associated with blobs B36 - B85 because thread T2 processes additional blobs (B71 - B85) after unexpectedly starting after failure. Threads T2 and T4 both work on the same key range 2 (range ID R2) and include different generation records (i.e., tables 1532 and 1534) with different GenNums, thereby preventing data corruption in the overlapping blobs B71 - B85 when thread T2 unexpectedly starts. When the information in the generation records of the key range is transferred to the target FS via the object store, the target FS may identify a group of overlapping blobs (B71 - B85) between the key-value pair (GenNum = G2, blobs B36 - B85) owned by thread T2 and another key-value pair (GenNum = G3, blobs B71 - B100) owned by thread T4. Since the overlapping blobs (B71 - B85) associated with the higher GenNum are the recovered blobs, the target FS may download and apply the blobs B1 - B35 associated with GenNum G1, the blobs B36 - B70 associated with GenNum G2, and the blobs B71 - B100 associated with GenNum G3.

[0176] FIG. 16 is a flowchart showing key range processing including generation records for difference generation and difference application during inter-region replication according to an embodiment. FIG. 16 can expand and explain further details regarding step 1420 of FIG. 14. In step 1610, each range thread working on a key range creates generation records by assigning a generation number (GenNum) to a group of blobs having a start sequence number and an end sequence number. The range thread processes this group of blobs by identifying differences (e.g., key-value pairs) from the B-tree of the source FS, converting the differences into blob format, and uploading the blobs to the object store. If a failure event occurs, the range thread may need to identify whether the failure event is a system crash or a thread failure. In step 1612, if the failure event is a system crash (e.g., 1320 of FIG. 13), in step 1614, the current range thread can create a new generation record and increment the GenNum (e.g., 1510 of FIG. 15) (i.e., create a new generation number). In step 1616, the current thread can check the corresponding checkpoint record and resume the difference generation process from the next key processed after the previous checkpoint. The difference generation resume process can be described in more detail later.

[0177] In step 1620, if the failure event is a thread failure (e.g., 1322 in FIG. 13), in step 1622, a new alternative range thread (e.g., thread T4 in FIG. 13) takes over the failed thread (e.g., thread T2 in FIG. 13), a new generation record (e.g., table 1534 in FIG. 15) can be created, and GenNum can be incremented by 1 (e.g., increment from G2 to G3 as shown in FIGS. 13 and 15). In step 1624, the alternative range thread checks the checkpoint record for key range 2 (range ID R2) and can resume the difference generation process from the next key processed after the previous checkpoint. After resuming the difference generation process in response to either a system crash or a thread failure, the process may proceed to step 1630. In step 1630, after processing a predetermined number of blobs that meet the checkpoint requirements (or after a predetermined period), an active thread (e.g., either T1 or T4) can copy the information in the generation record to a checkpoint manifest file for uploading to the object store.

[0178] Replication Checkpoint Figure 17 shows checkpoint records for a differential generation process and a differential application process according to an embodiment. In Figure 17, a generation record 1700 may include key-value pairs, where each key-value pair is a key-value pair for each checkpoint within a key range. The key-value pairs may be stored in a table format. In some embodiments, the fields of the key 1710 may include, but are not limited to, the type of replication, the replication ID, and the range_ID. Since both file systems contain their respective generation records in the same format, the type of replication may indicate whether this checkpoint record is for the source file system or for the target file system. The replication ID can be either a source file system ID or a target file system ID for identifying the file system performing inter-region replication. The range ID may be the key range identification information for this checkpoint record.

[0179] In some embodiments, the field with value 1720 may include, but is not limited to, information in three categories: (1) processed data, (2) the position of the processed key, and (3) a generation number for cross-referencing between the generation record and this checkpoint record. Information regarding the processed data, which is the first category, may include, but is not limited to, the number of objects or blobs uploaded as part of this checkpoint (i.e., num_objects), the number of keys uploaded by the source file system (i.e., num_keys_uploaded) or the number of keys applied by the target file system (i.e., num_keys_applied), the cumulative difference size (in bytes) uploaded so far for this replication by the source file system (i.e., total_delta_size_uploaded) or the cumulative difference size (in bytes) applied by the target file system (i.e., total_delta_size_applied). For example, each blob may contain 100 bytes. Thus, uploading 10 blobs may include a difference size of 1000 bytes. total_delta_size_uploaded is measured from the start of the entire key range. Since the delta generator divides B-tree keys into multiple key ranges, the delta generator has information regarding the number of keys and blobs for each key range. This information can be used to calculate the accumulated difference size when a thread creates a checkpoint record. The position of the processed key, which is the second category, may be the next key to start after this checkpoint (next_delta_key) and the last key replicated by this source file system (last_delta_key). Finally, since the checkpoint number and generation number may vary depending on the position of the failure event, the generation number can serve to cross-reference between the generation record and this checkpoint record.

[0180] In some embodiments, each range thread may checkpoint a predetermined number of processed blobs. Since the progress of each range thread may be different, the range threads may update their respective checkpoint records at different times after completing a predetermined number of processed blobs. In some embodiments, a blob may contain a fixed number of keys. In other embodiments, the number of keys within a blob may vary depending on the implementation. The number of keys uploaded (num_keys_uploaded) and the number of blobs (total_delta_size_uploaded) to the object store by the source file system can inform the target file system of the number of blobs to download for delta application and the progress of the entire replication process. The next key (next_delta_key) to start after this checkpoint can be used if the system crashes or a thread fails and the working range thread cannot recognize the position to resume the delta generation process in the source file system or the delta application process in the target file system. As an example of the use of checkpoints, after creating a checkpoint, a range thread may continue to process the B-tree keys of the key range and upload the converted blobs to the object store. At some point, a system crash occurs. The range thread may need to resume the delta generation process after the system recovers. Instead of going back to the beginning of the file system, each range thread can simply go back to the checkpoint just before the crash point and resume from there.

[0181] Figure 18 is an exemplary checkpoint record in table form according to an embodiment. For simplicity and for the purpose of explanation, not all fields are listed in the table. Referring to table 1810 of Figure 18 for thread T1, table 1810 contains, for each of the four checkpoints (C1, C2, C3, and C4), such as C1 at B25, C2 at B50, C3 at B75, and C4 at B100, one of four key-value pairs per checkpoint. In the case of the first key-value pair, the key range ID (range_ID) is R1. The number of objects or blobs for this checkpoint is 25. Assuming each blob contains 100 bytes, the total number of delta bytes uploaded by the source file system (total_delta_size_uploaded) is also 2500. Assuming 10 keys within each blob, the total number of keys processed up to this checkpoint is 250. Therefore, the next key (next_delta_key) to start after this checkpoint is 251. As shown in Figure 13, the generation number (GenNum) cross-referenced to the generation record of thread T1 is G1. Thereafter, as shown in Figure 13, thread T1 can continue processing B-tree keys up to blob B35 where the system crashes. After the system recovers, thread T1 recognizes, by checking the checkpoint record, that blobs B1 - B25 have been processed successfully and uploaded to the object store. Therefore, thread T1 can simply resume key processing from key number K251 after checkpoint C1 (i.e., next_delta_key or the first key within blob B26). Thread T1 can create another checkpoint C2 when processing another 25 blobs.Therefore, the value part of the key-value pair of checkpoint C2 may include that the number of objects (num_objects) is 25, the total number of uploaded blobs in key range 2 (total_delta_size_uploaded) is 5000, the next key to start (next_delta_key) is 501, and GenNum is G2 because the system crash increments the generation number by one. When thread T1 finishes processing all 100 blobs in key range 2, a similar process is executed for checkpoint C3 and checkpoint C4. The next_delta_key field does not need to be used for checkpoint C4 because thread T1 has finished processing the last key within key range 1. Checkpoint C4 exists so that in case a failure event occurs after all blobs in the key range have been uploaded by the range thread but before the checkpoint manifest file is uploaded to the object store, the range thread can use the checkpoint C4 information to resume generating and uploading the checkpoint manifest file.

[0182] Continuing to refer to FIG. 18, in the case of key range 2, as shown in FIG. 13, since a thread failure occurs between checkpoints C2 and C3, thread T2 may already have created and updated the checkpoint records of checkpoint numbers C1 and C2 as table 1820. After taking over thread T2, thread T4 can continue to update the checkpoint record table 1820 by obtaining a range-level lock on this checkpoint record (from the replicator or delta generator) for checkpoint numbers C3 and C4 to prevent the failed thread T2 from performing unexpected updates. When taking over thread T2, thread T4 can resume the delta generation process from the next_delta_key K1501 after checkpoint C2 by searching the summarized information in table 1820 stored in source SDB or in the central master checkpoint record described later.

[0183] Each checkpoint record of a key range may have a range-level lock for the range thread of that key range to update the checkpoint record. For example, if a thread failure occurs, multiple range threads may process the key range. A new alternative thread that updates the checkpoint record (i.e., the thread that takes over the failed thread) may acquire a range-level lock so that the failed thread cannot update the same record.

[0184] Resume of Difference Generation and Update of Checkpoint Status All range threads execute the differential generation of their key ranges simultaneously, in parallel, and independently, but need to update the progress byte in the central master checkpoint record to enable the source file system to coordinate or synchronize all range threads. The central master checkpoint can be managed by the data plane's replicator. The master checkpoint record with range_ID set to 0 can include a summary of the source FS processing information, such as the generation number, the number of checkpoint records the target FS expects, the total number of B-tree keys, and the B-tree keys processed. This information can be useful for reporting the progress of replication to the customer. When the checkpoint is complete, to synchronize all range threads, each range thread can acquire an alternating lock (e.g., a lock at the mutex replication job level) and update the progress byte of the central checkpoint record. After completing the update, the range thread can release the lock to allow another range thread to update the progress byte. Additionally, the master checkpoint record may need to periodically write the progress byte to the source shared database (SDB) for the control plane of the source file system to track the status of the differential generation progress. Thus, all range threads can be blocked from accessing the progress byte periodically (e.g., for a few minutes) while the master checkpoint record performs such an update to the SDB. This process can be illustrated in Figure 19 below.

[0185] Figure 19A is a flowchart showing a differential generation restart process using checkpoints according to an embodiment. Figure 19B is a flowchart showing a synchronization mechanism for checkpoint status updates between all key ranges according to an embodiment. Both the flowcharts of Figure 19A and Figure 19B further elaborate on step 1420 of Figure 14 regarding checkpoint updates.

[0186] Referring to FIG. 19A for the differential generation restart process, in some embodiments, at step 1910, after meeting checkpoint requirements such as the default requirements for processing key-value pairs of a binary tree (B-tree), each range thread of the source file system creates a checkpoint and updates the checkpoint record for the key range it is responsible for. The default requirements can be either the processing of a default number of keys (i.e., the keys and values of the B-tree representing the difference) or the converted blobs within the source FS, or the earlier of a default period (e.g., several minutes). At step 1912, each range thread continues to process the B-tree keys and create blobs for uploading to the object store.

[0187] In step 1920, when a failure event occurs, the process checks whether the failure event is a system crash or a thread failure. If the failure event is a thread failure (e.g., thread T2 in FIG. 13), the process may proceed to step 1922 through additional steps, where a new thread may take over the failed thread by obtaining ownership of the checkpoint record before resuming the differential generation process. Further, as described in connection with the generation record, the new thread also creates a new generation record with GenNum incremented by one. Otherwise (i.e., not a thread failure event), the process proceeds to step 1924, where all active range threads, including the new thread, check and use the next_delta_key field in the checkpoint record to resume in parallel and simultaneously from the next differential key (i.e., B-tree key) after each respective latest checkpoint. For example, if a failure event occurs between checkpoints C1 and C2, all range threads can resume from the next_delta_key of checkpoint C1 in their respective checkpoint tables. Assuming the failure event is a system crash (1320 in FIG. 13), thread T1 can resume at key number 251 by checking table 1810 (see FIG. 18). If the failure event is a thread failure such as thread T2 (1322 in FIG. 13), thread T4 can take over thread T2 at B170 (shown in FIG. 13), obtain ownership of this checkpoint record, and then resume at key number K1501 by checking table 1820 (see FIG. 18). In step 1926, the process may repeat steps 1910 - 1924 until all keys within each key range are processed by the responsible range thread.

[0188] Referring to FIG. 19B for the checkpoint status update process, in some embodiments, at step 1930, as previously described, when the source FS receives an inter-region replication request between the source FS and the target FS in different regions, the B-tree key is split into multiple key ranges, and a range thread, which is one range thread per key range, can be assigned to the key range to perform differential generation. At step 1940, after meeting the checkpoint requirements for processing the key-value pairs of the binary tree (B-tree), such as the default requirements for the number of processed blobs or the period, each range thread of the source file system creates a checkpoint and updates the checkpoint record for the key range for which it is responsible. At step 1942, in the case of the range thread that creates the checkpoint first among all the range threads (referred to as the cp first thread), after creating the checkpoint, the cp first thread may request a lock on the central master checkpoint record to update a central record such as the progress byte. At step 1943, the cp first thread may obtain a lock from the lock manager of the central master checkpoint record (or be granted a lock by the lock manager) to update the progress byte, and all other range threads may wait until the cp first thread completes the update and releases the lock. This lock mechanism is to ensure that only one range thread can update the progress byte at a time (or prevent other threads from updating simultaneously). At step 1944, all other range threads that have completed creating their respective checkpoints can obtain the lock alternately and update the central record (e.g., the status byte). For example, the first range thread (e.g., the cp first thread) may create the checkpoint for the first key range before the second range thread creates the checkpoint for the second key range. Thus, the first range thread may request a lock on the central master checkpoint record before the second range thread requests the same lock.As a result, the first range thread may be granted a lock to update the central record, and the second range thread may need to wait for the first range thread to release the lock.

[0189] In some embodiments, when two or more threads request a lock simultaneously, a contention resolution policy may be used to resolve such contention. The contention resolution policy may include, but is not limited to, a round-robin mechanism, thread priority, first-come-first-served, etc. For example, a round-robin mechanism may be used to select the requesting thread based on the thread IDs of these requesting threads, such as selecting a lower thread ID associated with the thread that starts the differential generation process first. Thread priority may, where applicable, select the range thread designated as having a higher priority. Requests may be queued and serviced to acquire the lock. In one embodiment, while a thread is waiting to update the progress byte, the thread can still continue to process keys within the key range for which it is responsible.

[0190] In step 1946, after updating the central master checkpoint record, the cp first thread continues the differential generation process for the next checkpoint stage of the key range for which it is responsible while other threads are updating the central master checkpoint record. Since each thread progresses at its own pace, the cp first thread that acquires the lock at the next checkpoint stage can be a different thread. For example, in FIG. 13, assume that thread T1 first reaches checkpoint C1 within key range 1 (1310) (i.e., the cp first thread) and acquires the lock. Then threads T2 and T3 may wait for the lock to be released later. After finishing updating the progress byte, T1 moves to the next checkpoint stage, creates checkpoint C2 for key range 1, and threads T2 and T3 may acquire the lock alternately and update the progress byte. After updating the progress byte, assume that thread T3 processes faster at the next checkpoint stage and first reaches checkpoint C2 within key range 3 (1304) and becomes the cp first thread for the checkpoint C2 stage. Then thread T3 can acquire the lock, and threads T1 and T2 may need to wait for the lock to be released. In step 1948, the process repeats steps 1940 - 1946 for the remaining checkpoint stages until all range threads have completed differential generation for all key ranges.

[0191] When performing differential application, the target file system may utilize the same resume process, such as a new alternative thread inheriting ownership of the checkpoint record from the failed thread and a synchronization mechanism for checkpoint status updates among its range threads.

[0192] Checkpoint manifest file As previously explained in connection with FIG. 14, after each checkpoint, the checkpoint records and generation records for each key range are integrated to create a checkpoint manifest file and can be uploaded as a manifest blob to the object store by the responsible range thread. The manifest blob can be uploaded by the range thread after all the delta blobs for the checkpoint have been uploaded. When the target FS detects the existence of the checkpoint manifest blob in the object store, the target FS first downloads the checkpoint manifest blob, reads and uses the information in this checkpoint manifest, downloads the number of delta blobs uploaded by the source FS, and can apply them to the corresponding key range in the target FS. The above process continues until the target FS receives a manifest blob with the end-of-file (eof) field set to true, which notifies the target FS of the checkpoint phase for the key range. This checkpoint manifest blob functions as communication between the source file system and the target file system using concise information.

[0193] Figure 20 is an exemplary checkpoint manifest file according to an embodiment. In some embodiments, the fields in the checkpoint manifest file may include, but are not limited to, the number of objects or blobs uploaded as part of this checkpoint (i.e., num_objects), the number of keys uploaded by the source file system (i.e., num_keys_uploaded) or the number of keys applied by the target file system (i.e., num_keys_applied), the cumulative delta size (in blob units) uploaded so far for this replication by the source file system (i.e., total_delta_size_uploaded) or the cumulative delta size (in blob units) applied by the target file system (i.e., total_delta_size_applied). The source replication ID (i.e., src_repl_id) and the target replication ID (i.e., tgt_repl_id) are used to identify a specific replication job between the source FS and the target FS. The cumulative delta size (in bytes) uploaded so far for this replication by the source file system (i.e., total_delta_size_uploaded) or the cumulative delta size (in bytes) applied by the target FS (i.e., total_delta_size_applied) can help verify the deltas received and applied by the target. The key range used to generate this manifest file (i.e., range_ID), and the total number of range threads working on this key range (i.e., total_sub_tasks) can help the target FS, for example, in the case of a failed thread, to determine whether more than one thread was working on this key range. The end-of-file setting (i.e., eof) can help the target FS track the end of the key range.Finally, as described in connection with FIG. 15 with respect to generation records, to track failure events during the replication process and avoid potential data corruption, there is an array of lists of generation numbers (i.e., GenNum) that includes the latest generation number (i.e., latest_gennum), and a list of corresponding blob sequence numbers (i.e., start_blob_seqno and end_blob_seqno). Since multiple failure events may occur during the checkpointing stage, the checkpoint manifest blob includes an array of GenNum and a group of blob sequence numbers.

[0194] As previously explained in connection with FIG. 13, the checkpoint manifest file can help the target FS to be aware of the blobs available for download, the progress of the replication process, and solve potential data corruption problems. As an example, consider key range 2 that includes a thread failure (e.g., thread T2 at B70 shown in FIG. 13). Referring to FIG. 20, until the delta generation for key range 2 is complete, this key range may have uploaded the following four checkpoint manifest files / blobs to the object store (see also FIG. 13 for key range 2(1312) and FIGS. 18 for threads T2 and T4). First checkpoint manifest blob of C1 (created by thread T2): num_objects = 25, delta_size_uploaded = 2500, (assuming 100 bytes per blob) total_sub_tasks = 1, range_id = R2, eof = 0, latest_gennum = G1, [G1: start_blob_seqno, end_blob_seqno) = (1, 25)] Second checkpoint manifest blob of C2 (created by thread T2): num_objects = 25, delta_size_uploaded = 5000, total_sub_tasks = 1, range_id = R2, eof = 0, latest_gennum = G2, [G1: (start_blob_seqno, end_blob_seqno) = (26, 35)] [G2: (start_blob_seqno, end_blob_seqno) = (36, 50)] The third checkpoint manifest blob of C3 (created by thread T4): num_objects = 25, delta_size_uploaded = 7500, total_sub_tasks = 2, range_id = R2, eof = 0, latest_gennum = G3, [G2: (start_blob_seqno, end_blob_seqno) = (51, 70)] [G3: (start_blob_seqno, end_blob_seqno) = (71, 75)] The fourth checkpoint manifest blob of C4 (created by thread T4): num_objects = 25, delta_size_uploaded = 10000, total_sub_tasks = 2, range_id = R2, eof = 1, latest_gennum = G3, [G3: (start_blob_seqno, end_blob_seqno) = (76, 100)] For example, when the target FS receives the second checkpoint manifest blob of checkpoint C2 in key range 2 (range ID R2), the target FS recognizes that it needs to download 25 blobs from the object store based on the num_objects field, and based on the delta_size_uploaded field, it should have received a total of 5000 bytes for this key range so far. Based on the total_sub_tasks field, only one range thread, T2, is working on this key range. These 25 blobs that the target FS is scheduled to download are not the last group of blobs for this key range based on the eof field. Finally, this group of blobs belongs to generation number G1 associated with sequence numbers 26 - 35 and generation number G2 associated with sequence numbers 36 - 50 because a system crash occurred at B35.

[0195] When the target FS receives the third checkpoint manifest blob of C3, this information can help the target FS detect that a failure event has occurred in the source FS during the delta generation process. Based on the total_sub_tasks, range_id, and GenNum fields, the target FS can recognize that since the latest generation number (latest_gennum) has increased from G2 to G3, thread T4 is the alternative thread and the second range thread working on this key range. The 25 blobs that the target FS is scheduled to download belong to GenNum G2 associated with blob sequence numbers 51 - 70 and GenNum G3 associated with sequence numbers 71 - 75. If the failed thread T2 unexpectedly starts after B70 (see Figure 13) and uploads another checkpoint manifest blob of C3 but fails again, the target FS may receive the fifth checkpoint manifest blob uploaded by thread T2, although the timing is uncertain. 5th checkpoint manifest blob of C3 (by thread T2): num_objects = 25, delta_size_uploaded = 7500, total_sub_tasks = 1, range_id = R2, latest_gennum = G2, eof = 0, [G2: (start_blob_seqno, end_blob_seqno) = (51, 75)] The 3rd and 5th checkpoint manifest blobs contain overlapping blob_seqnos (51 - 70 and 71 - 75). Since this 5th checkpoint manifest blob has a smaller GenNum G2 for blobs B71 - B75 (by the failed thread T2), as previously explained in relation to Figure 15, the target FS should discard this group of blobs and use the recovered blobs B71 - B75 associated with GenNum G3 (by thread T4). When the new replacement thread T4 takes over the failed thread T2, it can detect from the checkpoint record that the checkpoint manifest file has been generated but not uploaded. Thread T4 can then generate and upload the converted checkpoint manifest blob and update the checkpoint record to indicate that the manifest file has been uploaded. The failed thread T2 may upload the checkpoint manifest blob after startup, but it does not need to update the checkpoint record for key range 2 and thus cannot generate any checkpoint manifest files in the future. For example, if thread T2 starts before the expiration of the job lease, it cannot update the checkpoint record because the ownership of the checkpoint record has changed after thread T4 took over. If thread T2 starts after the expiration of the job lease, it cannot update and extend the lease because thread T4 has taken over the differential generation job. The ownership of the checkpoint record and the job lease mechanism can be coordinated among the data plane (e.g., replicator, differential generator, and range threads), the source SDB, and the control plane.

[0196] When the target FS receives the fourth checkpoint manifest blob, since the eof field is set to 1, it recognizes that this blob is the last group of 25 blobs in this key range. When the target FS downloads the manifest blob and applies the differential blobs for each checkpoint, it can delete the manifest blob in the object store to leave more space for future blob uploads by the source FS. Further, in some embodiments, in the source FS, after the checkpoint manifest file is created, the checkpoints and one or more generation records in the source SDB can be deleted to improve resource utilization.

[0197] Exemplary Cloud Architecture As described above, infrastructure as a service (IaaS) as a service is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services as they arise in connection with those infrastructure components (examples of services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Thus, since these services can be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.

[0198] In some cases, IaaS customers may access resources and services via a wide area network (WAN) such as the Internet and use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and install enterprise software on the VM. Next, the customer can use the provider's services to perform various functions, including load balancing network traffic, troubleshooting application problems, monitoring performance, and managing disaster recovery.

[0199] In most cases, the cloud computing model requires the participation of a cloud provider. The cloud provider can be a third-party service that specializes in providing (e.g., offering, lending, selling) IaaS, but it doesn't have to be. An entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0200] In some examples, the deployment of IaaS is the process of placing a new application or a new version of an application on a prepared application server, etc. This process may include the process of preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by the cloud provider under the hypervisor layer (e.g., server, storage, network hardware, and virtualization). Therefore, the customer can play a role in handling the deployment of (e.g., an operating system (OS), middleware, and / or an application) on top of (e.g., a self-service virtual machine that can be spun up on demand).

[0201] In some examples, IaaS provisioning can also refer to obtaining computers or virtual hosts for use and installing the required libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, which may need to be done first.

[0202] In some cases, there are two different issues with IaaS provisioning. First, there is the initial issue of provisioning the initial set of infrastructure before anything is run. Second, after everything is provisioned, there is the issue of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two issues can be addressed by enabling the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., which components are needed and how those components communicate with each other) can be defined by one or more configuration files. In this way, the entire infrastructure topology (e.g., which resources depend on which resources and how each of those resources interact) can be described declaratively. In some cases, after the topology is defined, a workflow can be generated to create and / or manage the various components described in the configuration files.

[0203] In some examples, the infrastructure can include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., a configurable and / or shared pool of computing resources, sometimes on-demand), which can be configured and / or shared. In some examples, there may be one or more inbound traffic / outbound traffic group rules provisioned to define how inbound and / or outbound traffic of the network is set, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. may be provisioned. As more infrastructure elements are desired and / or added, the infrastructure can gradually develop.

[0204] In some cases, continuous deployment techniques can be employed to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques can enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes globally). However, in some examples, the infrastructure to which the code is deployed must be set up first. In some cases, provisioning can be done manually, provisioning tools can be used to provision resources, and / or after the infrastructure is provisioned, deployment tools can be used to deploy the code.

[0205] FIG. 21 is a block diagram 2100 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2102 can be communicatively coupled to a secure host tenancy 2104 that can include a virtual cloud network (VCN) 2106 and a secure host subnet 2108. In some examples, the service operator 2102 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google® Glass head-mounted display) that run software such as Microsoft Windows Mobile®, and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and have Internet, email, short message service (SMS), BlackBerry®, or other communication protocols enabled. Alternatively, the client computing device can be a general-purpose personal computer, including, for example, personal computers and / or laptop computers that run various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. The client computing device can be a workstation computer that runs any of various commercially available UNIX® or UNIX-like operating systems, including but not limited to various GNU / Linux operating systems such as Google Chrome OS.Alternatively or in addition, the client computing device can be any other electronic device such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device), and / or a personal messaging device that can communicate via a network and / or the Internet accessible to the VCN 2106.

[0206] The VCN 2106 can include an LPG 2110 that can be communicatively coupled to an SSH VCN 2112 via a local peering gateway (LPG) 2110 included in a secure shell (SSH) VCN 2112. The SSH VCN 2112 can include an SSH subnet 2114 and can be communicatively coupled to a control plane VCN 2116 via an LPG 2110 included in the control plane VCN 2116. Also, the SSH VCN 2112 can be communicatively coupled to a data plane VCN 2118 via the LPG 2110. The control plane VCN 2116 and the data plane VCN 2118 can be included in a service tenancy 2119 that can be owned and / or operated by an IaaS provider.

[0207] The control plane VCN 2116 can include a control plane demilitarized zone (DMZ) layer 2120 that functions as a border network (e.g., a part of the enterprise network between the enterprise intranet and the external network). The servers based on the DMZ have limited responsibilities and can help keep the intrusion contained. Further, the DMZ layer 2120 can include one or more load balancer (LB) subnets 2122, a control plane application layer 2124 that can include an application subnet 2126, and a control plane data layer 2128 that can include a database (DB) subnet 2130 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 2122 included in the control plane DMZ layer 2120 can be communicatively coupled to the application subnet 2126 included in the control plane application layer 2124 that can be included in the control plane VCN 2116 and the Internet gateway 2134, and the application subnet 2126 can be communicatively coupled to the DB subnet 2130 included in the control plane data layer 2128 as well as the service gateway 2136 and the network address translation (NAT) gateway 2138. The control plane VCN 2116 can include the service gateway 2136 and the NAT gateway 2138.

[0208] The control plane VCN 2116 can include a data plane mirror app layer 2140 that can include an app subnet 2126. The app subnet 2126 included in the data plane mirror app layer 2140 can include a virtual network interface controller (VNIC) 2142 that can execute a compute instance 2144. The compute instance 2144 can communicatively couple the app subnet 2126 of the data plane mirror app layer 2140 to the app subnet 2126 that can be included in the data plane app layer 2146.

[0209] The data plane VCN 2118 can include a data plane app layer 2146, a data plane DMZ layer 2148, and a data plane data layer 2150. The data plane DMZ layer 2148 can include an LB subnet 2122 that can be communicatively coupled to the app subnet 2126 of the data plane app layer 2146 and the Internet gateway 2134 of the data plane VCN 2118. The app subnet 2126 can be communicatively coupled to the service gateway 2136 and the NAT gateway 2138 of the data plane VCN 2118. The data plane data layer 2150 can also include a DB subnet 2130 that can be communicatively coupled to the app subnet 2126 of the data plane app layer 2146.

[0210] The Internet gateways 2134 of the control plane VCN 2116 and the data plane VCN 2118 can be communicatively coupled to a metadata management service 2152 that can be communicatively coupled to the public Internet 2154. The public Internet 2154 can be communicatively coupled to the NAT gateways 2138 of the control plane VCN 2116 and the data plane VCN 2118. The service gateways 2136 of the control plane VCN 2116 and the data plane VCN 2118 can be communicatively coupled to cloud services 2156.

[0211] In some examples, the service gateway 2136 of the control plane VCN 2116 or the data plane VCN 2118 can make application programming interface (API) calls to the cloud service 2156 without going through the public Internet 2154. The API call from the service gateway 2136 to the cloud service 2156 can be unidirectional, and the service gateway 2136 can make an API call to the cloud service 2156, and the cloud service 2156 can send the requested data to the service gateway 2136. However, the cloud service 2156 does not need to initiate an API call to the service gateway 2136.

[0212] In some examples, the secure host tenancy 2104 can be directly connected to the service tenancy 2119, or otherwise can be separated. The secure host subnet 2108 can communicate with the SSH subnet 2114 via the LPG 2110, and the LPG 2110 can enable two-way communication on a separated system if not. Connecting the secure host subnet 2108 to the SSH subnet 2114 can give the secure host subnet 2108 access to other entities within the service tenancy 2119.

[0213] The control plane VCN 2116 may enable a user of the service tenancy 2119 to set or otherwise provision desired resources. Desired resources provisioned within the control plane VCN 2116 may be deployed or otherwise used in the data plane VCN 2118. In some examples, the control plane VCN 2116 may be separable from the data plane VCN 2118, and the data plane mirror app layer 2140 of the control plane VCN 2116 may communicate with the data plane app layer 2146 of the data plane VCN 2118 via VNICs 2142 that may be included in the data plane mirror app layer 2140 and the data plane app layer 2146.

[0214] In some examples, a user or customer of the system may perform requests, such as create, read, update, or delete (CRUD) operations, via the public internet 2154 that can communicate requests to the metadata management service 2152. The metadata management service 2152 may communicate the requests to the control plane VCN 2116 via the internet gateway 2134. The requests may be received by the LB subnet 2122 included in the control plane DMZ layer 2120. The LB subnet 2122 may determine that the requests are valid, and in response, the LB subnet 2122 may send the requests to the app subnet 2126 included in the control plane app layer 2124. If the validity of the requests is confirmed and the requests require calls to the public internet 2154, the calls to the public internet 2154 may be sent to the NAT gateway 2138 that can make calls to the public internet 2154. Metadata that may desirably be stored by the requests may be stored within the DB subnet 2130.

[0215] In some examples, the data plane mirror application layer 2140 can facilitate direct communication between the control plane VCN 2116 and the data plane VCN 2118. For example, it may be desirable for changes, updates, or other appropriate modifications to the configuration to be applied to the resources included in the data plane VCN 2118. Through the VNIC 2142, the control plane VCN 2116 can communicate directly with the resources included in the data plane VCN 2118, thereby enabling changes, updates, or other appropriate modifications to the configuration of the resources.

[0216] In some embodiments, the control plane VCN 2116 and the data plane VCN 2118 may be included in the service tenant 2119. In this case, the user or customer of the system does not have to own or operate either the control plane VCN 2116 or the data plane VCN 2118. Instead, the IaaS provider may own or operate both the control plane VCN 2116 and the data plane VCN 2118, which may both be included in the service tenancy 2119. This embodiment can enable network isolation that can prevent a user or customer from exchanging information with the resources of other users or other customers. Also, this embodiment can enable the private storage of databases by the user or customer of the system without having to rely on the public Internet 2154, which may not have the desired level of threat prevention for storage.

[0217] In other embodiments, the LB subnet 2122 included in the control plane VCN 2116 can be configured to receive signals from the service gateway 2136. In this embodiment, the control plane VCN 2116 and the data plane VCN 2118 can be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 2154. A customer of the IaaS provider may desire this embodiment because the databases used by the customer may be controlled by the IaaS provider and may be stored in a service tenancy 2119 that can be isolated from the public Internet 2154.

[0218] FIG. 22 is a block diagram 2200 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2202 (e.g., the service operator 2102 of FIG. 21) can be communicatively coupled to a secure host tenancy 2204 (e.g., the secure host tenancy 2104 of FIG. 21) that can include a virtual cloud network (VCN) 2206 (e.g., the VCN 2106 of FIG. 21) and a secure host subnet 2208 (e.g., the secure host subnet 2108 of FIG. 21). The VCN 2206 can be communicatively coupled to a Secure Shell (SSH) VCN 2212 (e.g., the SSH VCN 2112 of FIG. 21) via a local peering gateway (LPG) 2210 (e.g., the LPG 2110 of FIG. 21) included in the SSH VCN 2212. The SSH VCN 2212 can include an SSH subnet 2214 (e.g., the SSH subnet 2114 of FIG. 21), and the SSH VCN 2212 can be communicatively coupled to a control plane VCN 2216 (e.g., the control plane VCN 2116 of FIG. 21) via the LPG 2210 included in the control plane VCN 2216. The control plane VCN 2216 can be included in a service tenancy 2219 (e.g., the service tenancy 2119 of FIG. 21), and a data plane VCN 2218 (e.g., the data plane VCN 2118 of FIG. 21) can be included in a customer tenancy 2221 that can be owned or operated by a user or customer of the system.

[0219] The control plane VCN 2216 can include a control plane DMZ layer 2220 (e.g., the control plane DMZ layer 2120 in FIG. 21) that can include an LB subnet 2222 (e.g., the LB subnet 2122 in FIG. 21), a control plane application layer 2224 (e.g., the control plane application layer 2124 in FIG. 21) that can include an application subnet 2226 (e.g., the application subnet 2126 in FIG. 21), and a control plane data layer 2228 (e.g., the control plane data layer 2128 in FIG. 21) that can include a database (DB) subnet 2230 (similar to the DB subnet 2130 in FIG. 21). The LB subnet 2222 included in the control plane DMZ layer 2220 can be communicatively coupled to the application subnet 2226 included in the control plane application layer 2224 that can be included in the control plane VCN 2216, and to an Internet gateway 2234 (e.g., the Internet gateway 2134 in FIG. 21). The application subnet 2226 can be communicatively coupled to the DB subnet 2230 included in the control plane data layer 2228, as well as to a service gateway 2236 (e.g., the service gateway 2136 in FIG. 21) and a network address translation (NAT) gateway 2238 (e.g., the NAT gateway 2138 in FIG. 21). The control plane VCN 2216 can include the service gateway 2236 and the NAT gateway 2238.

[0220] The control plane VCN 2216 can include a data plane mirror application layer 2240 (e.g., the data plane mirror application layer 2140 of FIG. 21) that can include an application subnet 2226. The application subnet 2226 included in the data plane mirror application layer 2240 can include a virtual network interface controller (VNIC) 2242 (e.g., the VNIC 2142) that can execute a compute instance 2244 (e.g., similar to the compute instance 2144 of FIG. 21). The compute instance 2244 can facilitate communication between the application subnet 2226 of the data plane mirror application layer 2240 and an application subnet 2226 that can be included in the data plane application layer 2246 (e.g., the data plane application layer 2146 of FIG. 21) via the VNIC 2242 included in the data plane mirror application layer 2240 and the VNIC 2242 included in the data plane application layer 2246.

[0221] The internet gateway 2234 included in the control plane VCN 2216 can be communicatively coupled to a metadata management service 2252 (e.g., the metadata management service 2152 of FIG. 21) that can be communicatively coupled to the public internet 2254 (e.g., the public internet 2154 of FIG. 21). The public internet 2254 can be communicatively coupled to the NAT gateway 2238 included in the control plane VCN 2216. The service gateway 2236 included in the control plane VCN 2216 can be communicatively coupled to a cloud service 2256 (e.g., the cloud service 2156 of FIG. 21).

[0222] In some examples, the data plane VCN 2218 may be included in the customer's tenancy 2221. In this case, the IaaS provider may provide a control plane VCN 2216 for each customer, and the IaaS provider may configure the specific compute instances 2244 included in the service tenancy 2219 for each customer. Each compute instance 2244 may enable communication between the control plane VCN 2216 included in the service tenancy 2219 and the data plane VCN 2218 included in the customer's tenancy 2221. The compute instance 2244 may enable the resources provisioned within the control plane VCN 2216 included in the service tenancy 2219 to be deployed or otherwise used in the data plane VCN 2218 included in the customer's tenancy 2221.

[0223] In other examples, a customer of an IaaS provider may have a database that persists in the customer's tenancy 2221. In this example, the control plane VCN 2216 can include a data plane mirror app layer 2240 that can include an app subnet 2226. The data plane mirror app layer 2240 can exist in the data plane VCN 2218, but the data plane mirror app layer 2240 does not have to persist in the data plane VCN 2218. That is, the data plane mirror app layer 2240 can have access rights to the customer's tenancy 2221, but the data plane mirror app layer 2240 does not have to exist in the data plane VCN 2218 and does not have to be owned or operated by the customer of the IaaS provider. The data plane mirror app layer 2240 can be configured to make calls to the data plane VCN 2218, but does not have to be configured to make calls to any entity included in the control plane VCN 2216. The customer may wish to deploy or otherwise use resources within the data plane VCN 2218 that are provisioned within the control plane VCN 2216, and the data plane mirror app layer 2240 can facilitate the desired deployment or other use of the customer's resources.

[0224] In some embodiments, a customer of an IaaS provider can apply a filter to the data plane VCN 2218. In this embodiment, the customer can determine which data plane VCN 2218s are accessible, and the customer can restrict access from the data plane VCN 2218 to the public Internet 2254. The IaaS provider does not have to be able to apply the filter or otherwise control access of the data plane VCN 2218 to any external network or database. Applying the filter and control by the customer to the data plane VCN 2218 included in the customer's tenancy 2221 can help to isolate the data plane VCN 2218 from other customers and from the public Internet 2254.

[0225] In some embodiments, cloud service 2256 can be invoked by service gateway 2236 to access services that may not exist on any of public Internet 2254, control plane VCN 2216, or data plane VCN 2218. The connection between cloud service 2256 and control plane VCN 2216 or data plane VCN 2218 need not be operational or continuous. Cloud service 2256 can exist on a different network owned or operated by an IaaS provider. Cloud service 2256 may be configured to receive calls from service gateway 2236 and may be configured not to receive calls from public Internet 2254. Some cloud services 2256 may be isolated from other cloud services 2256, and control plane VCN 2216 may be isolated from cloud services 2256 that may not exist in the same region as control plane VCN 2216. For example, control plane VCN 2216 may be located in "Region 1," and a "deployment 21" of cloud services may be located in Region 1 and "Region 2." When a call to deployment 21 is made by service gateway 2236 included in control plane VCN 2216 located in Region 1, this call can be sent to deployment 21 within Region 1. In this example, control plane VCN 2216, or deployment 21 within Region 1, need not be communicatively coupled to, or otherwise communicate with, deployment 21 within Region 2.

[0226] FIG. 23 is a block diagram 2300 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2302 (e.g., the service operator 2102 of FIG. 21) can be communicatively coupled to a secure host tenancy 2304 (e.g., the secure host tenancy 2104 of FIG. 21) that can include a virtual cloud network (VCN) 2306 (e.g., the VCN 2106 of FIG. 21) and a secure host subnet 2308 (e.g., the secure host subnet 2108 of FIG. 21). The VCN 2306 can be communicatively coupled to an SSH VCN 2312 (e.g., the SSH VCN 2112 of FIG. 21) via an LPG 2310 (e.g., the LPG 2110 of FIG. 21) included in the SSH VCN 2312. The SSH VCN 2312 can include an SSH subnet 2314 (e.g., the SSH subnet 2114 of FIG. 21), and the SSH VCN 2312 can be communicatively coupled to a control plane VCN 2316 (e.g., the control plane VCN 2116 of FIG. 21) via an LPG 2310 included in the control plane VCN 2316 and to a data plane VCN 2318 (e.g., the data plane 2118 of FIG. 21) via an LPG 2310 included in the data plane VCN 2318. The control plane VCN 2316 and the data plane VCN 2318 can be included in a service tenancy 2319 (e.g., the service tenancy 2119 of FIG. 21).

[0227] The control plane VCN 2316 can include a control plane DMZ layer 2320 (e.g., the control plane DMZ layer 2120 of FIG. 21) that can include a load balancer (LB) subnet 2322 (e.g., the LB subnet 2122 of FIG. 21), a control plane application layer 2324 (e.g., the control plane application layer 2124 of FIG. 21) that can include an application subnet 2326 (similar to the application subnet 2126 of FIG. 21), and a control plane data layer 2328 (e.g., the control plane data layer 2128 of FIG. 21) that can include a DB subnet 2330. The LB subnet 2322 included in the control plane DMZ layer 2320 can be communicatively coupled to the application subnet 2326 included in the control plane application layer 2324 that can be included in the control plane VCN 2316, and to an Internet gateway 2334 (e.g., the Internet gateway 2134 of FIG. 21). The application subnet 2326 can be communicatively coupled to the DB subnet 2330 included in the control plane data layer 2328, as well as to a service gateway 2336 (e.g., the service gateway of FIG. 21) and a network address translation (NAT) gateway 2338 (e.g., the NAT gateway 2138 of FIG. 21). The control plane VCN 2316 can include the service gateway 2336 and the NAT gateway 2338.

[0228] The data plane VCN 2318 can include a data plane application layer 2346 (e.g., the data plane application layer 2146 of FIG. 21), a data plane DMZ layer 2348 (e.g., the data plane DMZ layer 2148 of FIG. 21), and a data plane data layer 2350 (e.g., the data plane data layer 2150 of FIG. 21). The data plane DMZ layer 2348 can include a reliable application subnet 2360 and an unreliable application subnet 2362 of the data plane application layer 2346, and an LB subnet 2322 communicatively coupled to an Internet gateway 2334 included in the data plane VCN 2318. The reliable application subnet 2360 can be communicatively coupled to a service gateway 2336 included in the data plane VCN 2318, a NAT gateway 2338 included in the data plane VCN 2318, and a DB subnet 2330 included in the data plane data layer 2350. The unreliable application subnet 2362 can be communicatively coupled to a service gateway 2336 included in the data plane VCN 2318 and a DB subnet 2330 included in the data plane data layer 2350. The data plane data layer 2350 can include a DB subnet 2330 communicatively coupled to a service gateway 2336 included in the data plane VCN 2318.

[0229] The untrusted application subnet 2362 can include one or more primary VNICs 2364(1)-(N) communicatively coupled to tenant virtual machines (VMs) 2366(1)-(N). Each tenant VM 2366(1)-(N) can be communicatively coupled to respective application subnets 2367(1)-(N) that can be included in respective container egress VCNs 2368(1)-(N) that can be included in respective customer tenancies 2370(1)-(N). Each secondary VNIC 2372(1)-(N) can facilitate communication between the untrusted application subnet 2362 included in the data plane VCN 2318 and the application subnets included in the container egress VCNs 2368(1)-(N). Each container egress VCN 2368(1)-(N) can include a NAT gateway 2338 communicatively coupled to the public internet 2354 (e.g., the public internet 2154 of FIG. 21).

[0230] The internet gateway 2334 included in the control plane VCN 2316 and in the data plane VCN 2318 can be communicatively coupled to a metadata management service 2352 (e.g., the metadata management system 2152 of FIG. 21) communicatively coupled to the public internet 2354. The public internet 2354 can be communicatively coupled to the NAT gateway 2338 included in the control plane VCN 2316 and in the data plane VCN 2318. The service gateway 2336 included in the control plane VCN 2316 and in the data plane VCN 2318 can be communicatively coupled to cloud services 2356.

[0231] In some embodiments, the data plane VCN 2318 can be integrated with the customer's tenancy 2370. This integration can be useful or desirable for customers of the IaaS provider in some cases, such as when they may desire support when running code. A customer may provide code to execute that can be disruptive, communicate with other customers' resources, or otherwise cause undesirable effects. In response, the IaaS provider can determine whether to execute code provided to the IaaS provider by the customer.

[0232] In some examples, a customer of an IaaS provider may grant the IaaS provider temporary network access rights and request a function connected to the data plane application layer 2346. The code for executing this function may be executed in VMs 2366(1) to (N), and this code need not be configured to execute elsewhere on the data plane VCN 2318. Each of VMs 2366(1) to (N) may be connected to the tenancy 2370 of one customer. Each container 2371(1) to (N) included in VMs 2366(1) to (N) may be configured to execute the code. In this case, there can be a two-fold separation (for example, the containers 2371(1) to (N) executing the code, the containers 2371(1) to (N) may be included in at least VMs 2366(1) to (N) included in the untrusted application subnet 2362), which can help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging the networks of different customers. The containers 2371(1) to (N) may be communicatively coupled to the customer's tenancy 2370 and may be configured to send or receive data with the customer's tenancy 2370. The containers 2371(1) to (N) need not be configured to send or receive data with any other entity within the data plane VCN 2318. Upon completion of the execution of the code, the IaaS provider may force the containers 2371(1) to (N) to terminate or otherwise discard them.

[0233] In some embodiments, the trusted application subnet 2360 can execute code that can be owned or operated by an IaaS provider. In this embodiment, the trusted application subnet 2360 may be communicatively coupled to the DB subnet 2330 and may be configured to execute CRUD operations within the DB subnet 2330. The untrusted application subnet 2362 may be communicatively coupled to the DB subnet 2330, but in this embodiment, the untrusted application subnet may be configured to execute read operations within the DB subnet 2330. The containers 2371(1) to (N) that can execute customer code, which may be included in each customer's VMs 2366(1) to (N), do not have to be communicatively coupled to the DB subnet 2330.

[0234] In other embodiments, the control plane VCN 2316 and the data plane VCN 2318 do not have to be directly communicatively coupled. In this embodiment, there does not have to be a direct communication between the control plane VCN 2316 and the data plane VCN 2318. However, the communication can occur indirectly by at least one method. The LPG 2310 may be established by the IaaS provider, thereby facilitating the communication between the control plane VCN 2316 and the data plane VCN 2318. In another example, the control plane VCN 2316 or the data plane VCN 2318 can make calls to the cloud service 2356 via the service gateway 2336. For example, a call from the control plane VCN 2316 to the cloud service 2356 can include a request for a service that can communicate with the data plane VCN 2318.

[0235] FIG. 24 is a block diagram 2400 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2402 (e.g., the service operator 2102 of FIG. 21) can be communicatively coupled to a secure host tenancy 2404 (e.g., the secure host tenancy 2104 of FIG. 21) that can include a virtual cloud network (VCN) 2406 (e.g., the VCN 2106 of FIG. 21) and a secure host subnet 2408 (e.g., the secure host subnet 2108 of FIG. 21). The VCN 2406 can be communicatively coupled to an SSH VCN 2412 (e.g., the SSH VCN 2112 of FIG. 21) via an LPG 2410 (e.g., the LPG 2110 of FIG. 21) included in the SSH VCN 2412, and the VCN 2406 can include the LPG 2410. The SSH VCN 2412 can include an SSH subnet 2414 (e.g., the SSH subnet 2114 of FIG. 21), and the SSH VCN 2412 can be communicatively coupled to a control plane VCN 2416 (e.g., the control plane VCN 2116 of FIG. 21) via the LPG 2410 included in the control plane VCN 2416 and to a data plane VCN 2418 (e.g., the data plane 2118 of FIG. 21) via the LPG 2410 included in the data plane VCN 2418. The control plane VCN 2416 and the data plane VCN 2418 can be included in a service tenancy 2419 (e.g., the service tenancy 2119 of FIG. 21).

[0236] The control plane VCN 2416 can include a control plane DMZ layer 2420 (e.g., the control plane DMZ layer 2120 in FIG. 21) that can include an LB subnet 2422 (e.g., the LB subnet 2122 in FIG. 21), a control plane application layer 2424 (e.g., the control plane application layer 2124 in FIG. 21) that can include an application subnet 2426 (e.g., the application subnet 2126 in FIG. 21), and a control plane data layer 2428 (e.g., the control plane data layer 2128 in FIG. 21) that can include a DB subnet 2430 (e.g., the DB subnet 2330 in FIG. 23). The LB subnet 2422 included in the control plane DMZ layer 2420 can be communicatively coupled to the application subnet 2426 included in the control plane application layer 2424 that can be included in the control plane VCN 2416, and to an Internet gateway 2434 (e.g., the Internet gateway 2134 in FIG. 21). The application subnet 2426 can be communicatively coupled to the DB subnet 2430 included in the control plane data layer 2428, as well as to a service gateway 2436 (e.g., the service gateway in FIG. 21) and a network address translation (NAT) gateway 2438 (e.g., the NAT gateway 2138 in FIG. 21). The control plane VCN 2416 can include the service gateway 2436 and the NAT gateway 2438.

[0237] The data plane VCN 2418 can include a data plane application layer 2446 (e.g., the data plane application layer 2146 in FIG. 21), a data plane DMZ layer 2448 (e.g., the data plane DMZ layer 2148 in FIG. 21), and a data plane data layer 2450 (e.g., the data plane data layer 2150 in FIG. 21). The data plane DMZ layer 2448 can include a trusted application subnet 2460 (e.g., the trusted application subnet 2360 in FIG. 23) and an untrusted application subnet 2462 (e.g., the untrusted application subnet 2362 in FIG. 23) of the data plane application layer 2446, and an LB subnet 2422 communicatively coupled to an Internet gateway 2434 included in the data plane VCN 2418. The trusted application subnet 2460 can be communicatively coupled to a service gateway 2436 included in the data plane VCN 2418, a NAT gateway 2438 included in the data plane VCN 2418, and a DB subnet 2430 included in the data plane data layer 2450. The untrusted application subnet 2462 can be communicatively coupled to the service gateway 2436 included in the data plane VCN 2418 and the DB subnet 2430 included in the data plane data layer 2450. The data plane data layer 2450 can include a DB subnet 2430 communicatively coupled to the service gateway 2436 included in the data plane VCN 2418.

[0238] The untrusted application subnet 2462 can include primary VNICs 2464(1) to (N) communicatively coupled to tenant virtual machines (VMs) 2466(1) to (N) present within the untrusted application subnet 2462. Each tenant VM 2466(1) to (N) can execute code within its respective containers 2467(1) to (N) and can be communicatively coupled to an application subnet 2426 that can be included in a data plane application layer 2446 that can be included in a container egress VCN 2468. Each secondary VNIC 2472(1) to (N) can facilitate communication between the untrusted application subnet 2462 included in the data plane VCN 2418 and the application subnet included in the container egress VCN 2468. The container egress VCN can include a NAT gateway 2438 communicatively coupled to a public internet 2454 (e.g., the public internet 2154 of FIG. 21).

[0239] The internet gateway 2434 included in the control plane VCN 2416 and in the data plane VCN 2418 can be communicatively coupled to a metadata management service 2452 (e.g., the metadata management system 2152 of FIG. 21) communicatively coupled to a public internet 2454. The public internet 2454 can be communicatively coupled to a NAT gateway 2438 included in the control plane VCN 2416 and in the data plane VCN 2418. The service gateway 2436 included in the control plane VCN 2416 and in the data plane VCN 2418 can be communicatively coupled to a cloud service 2456.

[0240] In some examples, the pattern shown by the architecture of block diagram 2400 in FIG. 24 may be considered an exception to the pattern shown by the architecture of block diagram 2300 in FIG. 23, which may be desirable for the customers of the IaaS provider when the IaaS provider cannot communicate directly with a customer (e.g., a disconnected region). Each of the containers 2467(1)-(N) included in VM2466(1)-(N) for each customer may be accessed in real time by the customer. The containers 2467(1)-(N) may be configured to make calls to respective secondary VNICs 2472(1)-(N) included in the application subnet 2426 of the data plane application layer 2446 that may be included in the container egress VCN 2468. The secondary VNICs 2472(1)-(N) may be able to send the calls to the NAT gateway 2438, and the NAT gateway 2438 may be able to send the calls to the public Internet 2454. In this example, the containers 2467(1)-(N) that may be accessed in real time by the customer can be separated from the control plane VCN 2416 and may be separated from other entities included in the data plane VCN 2418. The containers 2467(1)-(N) may also be separated from the resources of other customers.

[0241] In another example, a customer can call cloud service 2456 using containers 2467(1) to (N). In this example, the customer can execute the code within containers 2467(1) to (N) that requests a service from cloud service 2456. Containers 2467(1) to (N) can send this request to secondary VNICs 2472(1) to (N), and secondary VNICs 2472(1) to (N) can send this request to a NAT gateway, and the NAT gateway can send this request to public internet 2454. Public internet 2454 can send this request to LB subnet 2422 included in control plane VCN 2416 via internet gateway 2434. In response to determining that this request is valid, the LB subnet can send this request to app subnet 2426, and app subnet 2426 can send this request to cloud service 2456 via service gateway 2436.

[0242] It should be understood that the IaaS architectures 2100, 2200, 2300, 2400 shown in the figures may include components other than the components shown. Further, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may include more or fewer components than the components shown in the figures, combine two or more components, or may have a different configuration or arrangement of components.

[0243] In one embodiment, the IaaS system described herein may include the provision of a series of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0244] FIG. 25 shows an exemplary computer system 2500 in which various embodiments may be implemented. System 2500 may be used to implement any of the computer systems described above. As shown in the figure, computer system 2500 includes a processing unit 2504 that communicates with a plurality of peripheral subsystems via a bus subsystem 2502. These peripheral subsystems may include a processing acceleration unit 2506, an I / O subsystem 2508, a storage subsystem 2518, and a communication subsystem 2524. Storage subsystem 2518 includes tangible computer-readable storage media 2522 and system memory 2510.

[0245] The bus subsystem 2502 provides a mechanism for the various components and subsystems of the computer system 2500 to communicate with each other as intended. Although the bus subsystem 2502 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2502 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include an ISA (Industry Standard Architecture) bus, an MCA (Micro Channel Architecture) bus, an EISA (Enhanced ISA) bus, a VESA (Video Electronics Standards Association) local bus, and a PCI (Peripheral Component Interconnect) bus implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard.

[0246] The processing unit 2504, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2500. One or more processors may be included in the processing unit 2504. These processors can include single-core processors or multi-core processors. In some embodiments, the processing unit 2504 may be implemented as one or more independent processing units 2532 and / or 2534, with a single-core processor or multi-core processor included in each processing unit. In other embodiments, the processing unit 2504 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0247] In various embodiments, the processing unit 2504 can execute various programs according to program code and can maintain a plurality of programs or processes to be executed simultaneously. At any given time, some or all of the program code to be executed can be present in the processor 2504 and / or in the storage subsystem 2518. With appropriate programming, the processor 2504 can provide the various functions described above. The computer system 2500 may further include a processing acceleration unit 2506 that can include a digital signal processor (DSP), an application specific processor, and / or the like.

[0248] The I / O subsystem 2508 may include user interface input devices and user interface output devices. User interface input devices can include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can enable a user to interact with information by controlling input devices such as a Microsoft Xbox (registered trademark) 360 game controller via a natural user interface using gestures and spoken commands, and can include motion detection devices and / or gesture recognition devices such as a Microsoft Kinect (registered trademark) motion sensor. User interface input devices may include gesture recognition devices such as a Google Glass (registered trademark) blink detector that detects a user's eye activity (e.g., a blink when taking a photo and / or selecting a menu) and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Additionally, user interface input devices can include a voice recognition detection device that enables a user to interact with a voice recognition system (e.g., a Siri (registered trademark) navigator) via voice commands.

[0249] The user interface input device may include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, gamepad, and graphic tablet, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser distance meters, and eye tracking devices. Further, the user interface input device may include medical image input devices such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasonic examination devices. The user interface input device may include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0250] The user interface output device may include, among others, visual displays other than a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), or a flat panel device using a plasma display, a projection device, a touch screen, or the like. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2500 to the user or another computer. For example, the user interface output device may include, but is not limited to, various display devices for visually transmitting text information, graphics information, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0251] Computer system 2500 may include a storage subsystem 2518 that provides a tangible non-transitory computer-readable storage medium for storing software and data structures that provide the functionality of the embodiments described in this disclosure. The software can include programs, code modules, instructions, scripts, etc., and when executed by one or more cores or processors of processing unit 2504, provides the aforementioned functionality. Storage subsystem 2518 may provide a repository for storing data used in accordance with this disclosure.

[0252] As shown in the example of FIG. 25, storage subsystem 2518 can include various components including system memory 2510, computer-readable storage medium 2522, and computer-readable storage medium reader 2520. System memory 2510 may store program instructions that are readable and executable by processing unit 2504. System memory 2510 may also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs may be loaded into system memory 2510 including, but not limited to, client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0253] System memory 2510 may store an operating system 2516. Examples of operating systems 2516 include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations where computer system 2500 runs one or more virtual machines, the virtual machines may be loaded into system memory 2510 along with guest operating systems (GOS) and executed by one or more processors or cores of processing unit 2504.

[0254] System memory 2510 may be provided in different configurations depending on the type of computer system 2500. For example, system memory 2510 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Various types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, system memory 2510 may include a basic input / output system (BIOS) that contains basic routines that help transfer information between elements within computer system 2500, such as during startup.

[0255] The computer-readable storage medium 2522 contains computer-readable information for use by the computer system 2500, including instructions executable by the processing unit 2504 of the computer system 2500, and temporarily and / or more persistently contains and stores the information. In addition to the storage medium for storing, it can represent a remote storage device, a local storage device, a fixed storage device, and / or a removable storage device.

[0256] The computer-readable storage medium 2522 can include any suitable medium known in or used in the art, including storage media and communication media such as volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information, but not limited to these. The computer-readable storage medium 2522 can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer-readable media.

[0257] As an example, computer-readable storage medium 2522 can include a hard disk drive that reads from or writes to a removable non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk such as a CD ROM, DVD, and Blu-ray (registered trademark) disk, or other optical medium. Computer-readable storage medium 2522 can include, but is not limited to, a Zip (registered trademark) drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. Computer-readable storage medium 2522 can include a solid-state drive (SSD) based on non-volatile memory such as a flash memory-based semiconductor drive, an enterprise flash drive, a semiconductor ROM, an SSD based on volatile memory such as a semiconductor RAM, a dynamic RAM, a static RAM, a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD that uses a combination of DRAM and a flash memory-based SSD. Disk drives and associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data of computer system 2500.

[0258] Machine-readable instructions executable by one or more processors or cores of processing unit 2504 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium can include a physically tangible memory or storage device that includes a volatile memory storage device and / or a non-volatile storage device. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy (registered trademark) drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0259] Communication subsystem 2524 provides an interface to other computer systems and networks. Communication subsystem 2524 functions as an interface for receiving data from other systems of computer system 2500 and for transmitting data to other systems. For example, communication subsystem 2524 may enable computer system 2500 to connect to one or more devices via the Internet. In some embodiments, communication subsystem 2524 may include components of a radio frequency (RF) transceiver for accessing wireless voice and / or data networks (such as cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (registered trademark) (IEEE 802.11 family of standards, or other mobile communication technologies, or any combination thereof), components of a global positioning system (GPS) receiver, and / or other components. In some embodiments, communication subsystem 2524 may provide a wired network connection (such as Ethernet (registered trademark)) in addition to, or instead of, a wireless interface.

[0260] In some embodiments, communication subsystem 2524 may receive input communications in the form of structured and / or unstructured data feeds 2526, event streams 2528, event updates 2530, etc., on behalf of one or more users who may use computer system 2500.

[0261] As an example, the communication subsystem 2524 can be configured to receive in real time a data feed 2526 from social networks such as Twitter (registered trademark) feeds, Facebook (registered trademark) updates, and / or other communication services, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0262] Furthermore, the communication subsystem 2524 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2528 and / or event updates 2530 of real-time events that have no explicit end, are essentially continuous, or need not have boundaries. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0263] The communication subsystem 2524 may be configured to output structured and / or unstructured data feeds 2526, event streams 2528, event updates 2530, etc. to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2500.

[0264] The computer system 2500 can be one of various types, including a handheld portable device (e.g., an iPhone (registered trademark) mobile phone, an iPad (registered trademark) computing tablet, a PDA), a wearable device (e.g., a Google Glass (registered trademark) head-mounted display), a PC, a workstation, a mainframe, a ticket vending machine, a server rack, or any other data processing system.

[0265] Due to the constantly changing nature of computers and networks, the description of the computer system 2500 shown in the figures is merely intended to be a specific example. Many other configurations are possible that include more or fewer components than the system shown in the figures. For example, customized hardware may be used and / or certain elements may be implemented in hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other methods and / or ways to implement various embodiments.

[0266] Although specific embodiments have been described, various modifications, changes, alternative structures, and equivalents are also included within the scope of the present disclosure. Embodiments are not limited to operating within a particular data processing environment and can operate freely within multiple data processing environments. Further, although embodiments have been described using a particular series of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the series of transactions and steps described. The various features and aspects of the foregoing embodiments may be used individually or together.

[0267] Furthermore, although embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented using only hardware, only software, or a combination of these. The various processes described herein may be implemented on the same processor or on different processors in any combination. Thus, when a component or service is described as being configured to perform an operation, such a configuration may be realized, for example, by designing an electronic circuit to perform this operation, by programming a programmable electronic circuit (such as a microprocessor) to perform this operation, or by any combination of these. Processes can communicate using a variety of techniques including, but not limited to, prior art techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0268] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is obvious that additions, deletions, omissions, as well as other modifications and changes can be made without departing from the broader ideas and scope as set forth in the claims. Thus, while specific embodiments of the disclosure have been described, these are not intended to be limiting. Various changes and equivalents are within the scope of the appended claims.

[0269] The use of the terms "a", "an", and "the" and similar referents in the context of describing the disclosed embodiments (in particular, in the context of the appended claims) should be construed to cover both the singular and the plural unless specifically indicated otherwise herein or clearly contradicted by the context. The terms "comprising", "having", "including", and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" should be construed to mean either internally or partially or completely contained within, connected to, or joined together with, even if there is something intervening. The recitation of a range of values herein is merely intended to provide a convenient way of referring individually to each separate value falling within the range, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order unless specifically indicated otherwise herein or clearly contradicted by the context. The use of any examples, or exemplary language (e.g., "such as") provided herein is merely intended to better clarify the embodiments and does not impose a limitation on the scope of the disclosure unless otherwise claimed. No language in this specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0270] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally intended, unless otherwise explicitly stated, to convey in context that items, conditions, etc. can be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended, and should not be taken, to mean that a particular embodiment requires the presence of at least one of each of at least one of X, at least one of Y, or at least one of Z.

[0271] In this specification, preferred embodiments of the disclosure are described, including the best mode known to the applicant for carrying out the disclosure. Variations of such preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to adopt such variations as appropriate, and the disclosure may be practiced otherwise than as specifically described herein. Accordingly, the disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Further, any combination of the foregoing elements in all possible variations of the embodiments is included in the disclosure unless otherwise specifically indicated herein.

[0272] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference in their entirety, to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0273] In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments of the present specification. Those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or together. Further, embodiments may be utilized in any number of environments and applications beyond those described herein and within the broader spirit and scope of the present specification without departing therefrom. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0274] Embodiments may be implemented by using a computer program product that includes computer programs / instructions, which, when executed by a processor, cause the processor to execute any of the methods described in the present disclosure.

[0275] Item 1: A method comprising: a computing system receiving a request for file system replication between a source file system and a target file system, the source file system and the target file system being in different regions, and the method further comprising: the computing system processing key-value pairs of a plurality of binary trees (B-trees) that are different between two snapshots of the source file system; the computing system assigning a first identifier to a first group of key-value pairs among the key-value pairs of the plurality of B-trees; the computing system, when detecting a failure event during file system replication, assigning a second identifier to a second group of key-value pairs among the key-value pairs of the plurality of B-trees.

[0276] Item 2: The method according to Item 1, wherein processing comprises identifying key-value pairs of a plurality of B-trees and uploading them to an object storage.

[0277] Claim 3: The method according to claim 1 or 2, wherein the failure event is caused by the failure of the first processing thread.

[0278] Claim 4: The method according to claim 3, wherein a first group of key-value pairs among the key-value pairs of the plurality of B-trees is processed by the first processing thread, and a second group of key-value pairs among the key-value pairs of the plurality of B-trees is processed by the second processing thread.

[0279] Claim 5: The method according to claim 4, further comprising creating a record associated with the second processing thread to track a second group of key-value pairs among the key-value pairs of the plurality of B-trees. The record associated with the second processing thread is different from the record associated with the first processing thread for tracking a first group of key-value pairs among the key-value pairs of the plurality of B-trees.

[0280] Claim 6: The method according to claim 1 or 2, wherein the failure event is caused by a system crash.

[0281] Claim 7: The method according to claim 6, wherein both the first and second groups of key-value pairs among the key-value pairs of the plurality of B-trees are processed by a single processing thread.

[0282] Claim 8: The method according to any one of claims 1 to 7, wherein the first identifier and the second identifier are numerical values, and the second identifier has a numerical value greater than that of the first identifier.

[0283] Claim 9: The method according to claim 8, further comprising discarding an overlapping portion of a first group of key-value pairs among the key-value pairs of the plurality of B-trees by the target file system, at least partially based on the second identifier having a numerical value greater than that of the first identifier, when the first group and the second group include overlapping key-value pairs.

[0284] Claim 10: The method according to any one of Claims 1 to 9, wherein the key-value pairs of the plurality of B-trees belong to one of the plurality of key ranges.

[0285] Claim 11: A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors of a computing system, cause the one or more processors to perform operations, the operations including the computing system receiving a request for file system replication between a source file system and a target file system, the source file system and the target file system being in different regions, and the operations further including the computing system processing key-value pairs of a plurality of different binary trees (B-trees) between two snapshots of the source file system, the computing system assigning a first identifier to a first group of key-value pairs among the key-value pairs of the plurality of B-trees, the computing system detecting a failure event during file system replication and assigning a second identifier to a second group of key-value pairs among the key-value pairs of the plurality of B-trees.

[0286] Claim 12: The non-transitory computer-readable medium according to Claim 11, wherein the failure event is caused by the failure of a first processing thread.

[0287] Claim 13: The non-transitory computer-readable medium according to Claim 12, wherein the first group of key-value pairs among the key-value pairs of the plurality of B-trees is processed by a first processing thread, and the second group of key-value pairs among the key-value pairs of the plurality of B-trees is processed by a second processing thread.

[0288] Item 14: The non-transitory computer-readable medium according to Item 11, wherein the failure event is caused by a system crash.

[0289] Item 15: The non-transitory computer-readable medium according to Item 14, wherein both the first and second groups of key-value pairs among the key-value pairs of the plurality of B-trees are processed by a single processing thread.

[0290] Item 16: The non-transitory computer-readable medium according to any one of Items 11 to 15, wherein the operation further includes, when the first and second groups include overlapping key-value pairs, the target file system discarding the overlapping portion of the first group of key-value pairs among the key-value pairs of the plurality of B-trees, at least partially based on the second identifier having a larger numerical value than the first identifier, wherein the first identifier and the second identifier are numerical values, and the second identifier has a larger numerical value than the first identifier.

[0291] Item 17: A system, one or more processors, and one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to receive a request for file system replication between a source file system and a target file system, wherein the source file system and the target file system are in different regions, and the instructions further cause the system to process key-value pairs of a plurality of binary trees (B-trees) that are different between two snapshots of the source file system, assign a first identifier to a first group of key-value pairs among the key-value pairs of the plurality of B-trees, When a failure event is detected during file system replication, cause to assign a second identifier to a second group of key-value pairs among the key-value pairs of a plurality of B-trees.

[0292] Claim 18: The system according to claim 17, wherein the failure event is caused by the failure of a first processing thread, wherein a first group of key-value pairs among the key-value pairs of a plurality of B-trees is processed by a first processing thread, and a second group of key-value pairs among the key-value pairs of a plurality of B-trees is processed by a second processing thread.

[0293] Claim 19: The system according to claim 17, wherein the failure event is caused by a system crash, wherein both a first and a second group of key-value pairs among the key-value pairs of a plurality of B-trees are processed by a single processing thread.

[0294] Claim 20: The system according to any one of claims 17 to 19, wherein when the system further includes key-value pairs in which the first group and the second group overlap, the target file system causes the overlapping portion of the first group of key-value pairs among the key-value pairs of a plurality of B-trees to be discarded at least partially based on the second identifier having a numerical value greater than that of the first identifier, and the first identifier and the second identifier are numerical values, and the second identifier has a numerical value greater than that of the first identifier.< / gennum>

Claims

1. It is a method, The computing system includes receiving a request for filesystem replication between a source filesystem and a target filesystem, wherein the source filesystem and the target filesystem are located in different regions, and the method further includes: The first processing thread of the computing system, in response to the file system replication request, creates a checkpoint in the first key range after fulfilling the default requirements for processing key-value pairs of binary trees (B-trees) within the first key range. The first processing thread of the computing system requests a lock to update the central record after creating the checkpoint in the first key range, The second processing thread of the computing system, in response to the file system replication request, creates a checkpoint in the second key range after satisfying the default requirements for processing key-value pairs of a B-tree within the second key range. The second processing thread of the computing system requests the lock to update the central record after creating the checkpoint in the second key range, A method comprising the computing system granting the lock for updating the central record to the first processing thread, wherein the request for the lock for updating the central record by the first processing thread is earlier than the request by the second processing thread.

2. The method according to claim 1, wherein the lock prevents the second processing thread from updating the central record.

3. The first processing thread, after updating the central record, further processes the key-value pairs of the B-tree within the first key range, The method according to claim 2, further comprising the computing system granting the second processing thread the lock for updating the central record after the first processing thread has completed updating the central record.

4. The method according to claim 2, further comprising the second processing thread further processing key-value pairs of the B-tree in the second key range while waiting for the lock.

5. The method according to claim 1, wherein the first processing thread operates independently of the second processing thread.

6. The method according to claim 1, wherein the processing of key-value pairs of the B-tree within the first key range by the first processing thread is performed in parallel with the processing of key-value pairs of the B-tree within the second key range by the second processing thread.

7. The method according to claim 1, further comprising the first processing thread resuming processing of key-value pairs of the B-tree in the first key range from the B-tree key after the checkpoint in the first key range when a system failure is detected.

8. The method according to claim 1, further comprising detecting a failure of the first processing thread, and having a third processing thread resume processing key-value pairs of the B-tree in the first key range from the B-tree key after the checkpoint in the first key range.

9. A program that causes one or more processors of a computing system to execute the method according to any one of claims 1 to 8.

10. It is a system, One or more processors, A computer comprises one or more computer-readable media storing executable instructions, and when an instruction is executed by the one or more processors, the system... The system is instructed to receive a request for filesystem replication between a source filesystem and a target filesystem, wherein the source filesystem and the target filesystem are in different regions, and the instruction further instructs the system to: The first processing thread of the system, in response to the file system replication request, creates a checkpoint in the first key range after fulfilling the default requirements for processing key-value pairs of binary trees (B-trees) within the first key range. The first processing thread of the system requests a lock to update the central record after creating the checkpoint in the first key range, The second processing thread of the system, in response to the file system replication request, creates a checkpoint in the second key range after satisfying the default requirements for processing key-value pairs of the B-tree within the second key range. The second processing thread requests the lock to update the central record after creating the checkpoint in the second key range, A system that causes the first processing thread to grant the lock for updating the central record, wherein the request for the lock by the first processing thread for updating the central record is made earlier than the request by the second processing thread, and the lock is configured to prevent the second processing thread from updating the central record.

11. The aforementioned system further, After updating the central record, the first processing thread of the system is further processed by the key-value pairs of the B-tree within the first key range. The system according to claim 10, wherein the lock for updating the central record is granted to the second processing thread before the first processing thread has completed updating the central record.

12. The system according to claim 10 or 11, wherein the first processing thread operates independently of the second processing thread.

13. The system according to claim 10 or 11, wherein the processing of key-value pairs of the B-tree within the first key range by the first processing thread is performed in parallel with the processing of key-value pairs of the B-tree within the second key range by the second processing thread.

14. The system according to claim 10 or 11, wherein, upon detecting a system failure, the system can have the first processing thread resume processing of key-value pairs of the B-tree in the first key range, starting from the B-tree key after the checkpoint in the first key range.

15. The system according to claim 10 or 11, wherein, upon detecting a failure of the first processing thread, the system allows a third processing thread to resume processing of key-value pairs of the B-tree in the first key range, starting from the B-tree key after the checkpoint in the first key range.

Images