Techniques for Efficient Replication and Recovery

Abstract

Techniques for maintaining efficient replication and snapshot data consistency during file storage replication between file systems in different cloud infrastructure regions are described. In one embodiment, the lineage ID is used to efficiently identify the starting point of the inter-region replication process (e.g., the base snapshot), saving cloud resources while reducing network traffic and IO traffic. In one embodiment, create and delete requests for snapshots that occur during inter-region replication can be temporarily held until an appropriate time to safely execute such requests, depending on the timing relationship between such requests and the inter-region replication cycle.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Non - Provisional Patent Application No. 18 / 169,121, filed on February 14, 2023, entitled "TECHNIQUES FOR EFFICIENT REPLICATION AND RECOVERY", Attorney Docket No. 088325 - 1342650 (347410US), and U.S. Non - Provisional Patent Application No. 18 / 169,124, filed on February 14, 2023, entitled "TECHNIQUES FOR MAINTAINING SNAPSHOT DATA CONSISTENCY DURING FILE SYSTEM CROSS - REGION REPLICATION", Attorney Docket No. 088325 - 1364772 (347480US). All of these U.S. non - provisional patent applications claim the benefit and priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 63 / 352,992, filed on June 16, 2022, U.S. Provisional Patent Application No. 63 / 357,526, filed on June 30, 2022, U.S. Provisional Patent Application No. 63 / 412,243, filed on September 30, 2022, and U.S. Provisional Patent Application No. 63 / 378,486, filed on October 5, 2022. The disclosures of these are hereby incorporated by reference in their entireties for all purposes.

[0002] Field This disclosure generally relates to file systems. More particularly, but not by way of limitation, techniques for efficient replication and maintaining the consistency of snapshot data during file storage replication between file systems in different cloud infrastructure regions (e.g., data centers within a particular geographic area) are described.

Background Art

[0003] Background Enterprise business contains extremely important data. File system replication enhances the availability of extremely important data and provides fault tolerance. However, it is necessary to improve the efficiency of file system replication during replication and the consistency of snapshot data.

Summary of the Invention

[0004] Brief Summary This disclosure generally relates to file systems. More specifically, but not by way of limitation, techniques for maintaining efficient replication and snapshot data consistency during file storage replication between file systems in different cloud infrastructure regions (e.g., data centers within a specific geographic area) are described.

Problems to be Solved by the Invention

[0005] In one embodiment, a computing system generates a first snapshot and a second snapshot in a source file system within a source region, and the computing system assigns first provenance identification information to the first snapshot and second provenance identification information to the second snapshot within the source file system, where the first provenance identification information is unique among all snapshots within all regions and the second provenance identification information is unique among all snapshots within all regions. The computing system receives a request to perform replication between the source file system within the source region and a target file system within a target region, where the source region and the target region are in different regions. In response to this request, the computing system compares the first provenance identification information within the source file system with the provenance identification information of an existing snapshot within the target region, and the computing system identifies a matching snapshot having the first provenance identification information within the target region and uses it as the base snapshot for replication based at least in part on this comparison. The computing system includes a technique that includes performing replication using the difference between the second snapshot within the source file system and the base snapshot.

[0006] In yet another embodiment, the method further includes selecting the matching snapshot as the base snapshot, at least in response to the fact that the matching snapshot having the first provenance identification information within the target region is within the target file system.

[0007] In yet another embodiment, the target region includes a non-target file system that includes a snapshot associated with the first provenance identification information.

[0008] In yet another embodiment, the method further includes, at least in response to there being no matching snapshot having the first provenance identification information within the target file system in the target region, performing an in-region copy of a matching snapshot having the first provenance identification information from a non-target file system to the target file system, and selecting an in-region copy of the matching snapshot within the target file system as a base snapshot.

[0009] In yet another embodiment, an in-region copy of a matching snapshot within the target file system has the same first provenance identification information as the matching snapshot within the non-target file system, but has different resource identification information.

[0010] In yet another embodiment, the method further includes, at least in response to no matching snapshot having the first provenance identification information being detected within the target region, selecting a first snapshot having the first provenance identification information within a source file system as a base snapshot.

[0011] In yet another embodiment, the method further includes performing an inter-region copy of a first snapshot having the first provenance identification information from the source file system to the target file system before generating a difference between the second snapshot and the base snapshot within the source file system.

[0012] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0013] In various embodiments, the non - transitory computer - readable medium stores computer - executable instructions that, when executed by one or more processors, cause one or more processors of a computer system to perform one or more of the methods disclosed herein.

[0014] In various embodiments, the computer program product includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0015] The technologies described above and below can be implemented in a plurality of methods and a plurality of situations. Referring to the following figures, which are described in more detail below, a plurality of exemplary implementations and situations are provided. However, the following implementations and situations are only a part of many implementations and situations.

[0016] The features, embodiments, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Best Mode for Carrying Out the Invention

[0018] Detailed Description In the following description, for the purpose of explanation, specific details are shown so that one embodiment can be fully understood. However, it is clear that various embodiments can be practiced without those specific details. Each figure and description is not intended to be limiting.

[0019] In this specification, technologies for a file system service (FSS) are disclosed that utilize snapshots and data models to create, process, and replicate snapshots and the data associated therewith, and to ensure efficient replication, recovery, and consistency between a source file system (FS) and a target FS. For the efficiency of replication and recovery, lineage IDs are utilized to efficiently identify the starting point (e.g., the base snapshot) of a cross-region (or x-region) replication process.

[0020] A lineage ID is special identification information that uniquely identifies a snapshot across regions, regardless of whether the snapshot is a system snapshot or a user snapshot. Before cross-region replication begins, the source region and the target region can compare the lineage IDs of existing snapshots within their respective regions. When two snapshots with the same lineage ID are detected in both the source file system within the source region and the target file system within the target region, these two snapshots can be used as the base snapshot without copying the complete base snapshot from the source file system to the target file system. During cross-region replication, only the differences between subsequent snapshots in the source file system and the base snapshot need to be transferred to the target file system. Therefore, the lineage ID technology saves valuable cloud resources while reducing network traffic and IO traffic for performing cross-region replication.

[0021] In some embodiments, if a snapshot with a matching lineage ID is in a non-target file system within the target region, this non-target file system may perform in-region cloning of the snapshot with the matching lineage ID to the target FS to create a base snapshot. Thereafter, inter-region replication may be performed between the source file system and the target file system. Since in-region cloning does not involve additional encryption / decryption, data transfer via object storage, etc., in-region cloning also saves cloud resources.

[0022] The lineage ID can also help with efficient recovery in case of replication failure by quickly identifying a common starting point between the source FS and the target FS in different regions without requiring a full base copy to resume a failed replication. Finally, control plane communication between the source FS and the target FS of the snapshot and data model helps achieve the goal of improving the efficiency of file system replication by exchanging snapshot metadata information during the replication process.

[0023] The snapshot data consistency techniques disclosed herein can help protect data integrity by temporarily holding a request until an appropriate time to safely execute such a request when snapshot creation and deletion requests occur during inter-region replication. Control plane communication between the source FS and the target FS of the snapshot and data model helps achieve the goal of maintaining the consistency of snapshot data during replication by exchanging snapshot metadata information during the replication process.

[0024] In some embodiments, system snapshots are created and deleted periodically by the FSS, while user snapshots can be created and deleted at any time by the user according to a scheduled snapshot policy. Depending on the request timing for creating and deleting user snapshots and how these timings coincide with the inter-region replication cycle, several possibilities may occur, and in some cases, it may lead to snapshot consistency issues between the source FS and the target FS. To ensure snapshot consistency, snapshot deletion and replication can be delayed.

[0025] In one embodiment, if a request to delete a user snapshot occurs during a replication cycle (e.g., replication cycle N), the FSS may hold off on the deletion until the end of the replication cycle (e.g., replication cycle N) in the source FS and the end of the next replication cycle (e.g., replication cycle N + 1) in the target FS. In another embodiment, if a user snapshot is created and then required to be deleted between two replication cycles (e.g., between replication cycles N and N + 1), the snapshot may not need to be replicated at all. In yet another embodiment, if a user snapshot is created during a replication cycle (e.g., replication cycle N) but not required to be deleted until after more than one replication cycle (e.g., after replication cycle N + 1), the replication of the user snapshot can be delayed by one cycle (i.e., it can occur in replication cycle N + 1).

[0026] Explanation of terms in a certain embodiment "Recovery Time Objective" (RTO) refers to, in one embodiment, the period of time that a user needs to make replication available within a secondary (or target) region, regardless of whether the disruption is planned or unplanned, after a disruption occurs within the availability domain (AD) of the primary (or source) region.

[0027] "Recovery Point Objective" (RPO) refers to, in one embodiment, the maximum allowable scope with respect to the time of data loss between a disruption in the primary region (usually due to an unplanned disruption) and the availability of the secondary region.

[0028] "Replicator" can refer to, in one embodiment, a component within the data plane of a file system (e.g., a virtual machine (VM)) that uploads differences to a remote object store (i.e., an object storage service) when the component is located in the source region, or downloads differences from the object storage for applying differences when the component is located in the target region. The replicator is formed as a fleet (i.e., a plurality of VMs or replicator threads) called a replicator fleet and can execute the inter-region (or cross-region) replication process (e.g., uploading differences to the target region) in parallel.

[0029] A "delta generator" (DG) can refer to a component within the data plane of a file system that, in one embodiment, extracts the differences (i.e., changes) between the keys and values of two snapshots when the component is located in the source region or applies the differences to the latest snapshot within the B-tree of the file system when the component is located in the target region. The delta generator within the source region can use multiple threads (referred to as delta generator threads or range threads for multiple split B-tree key ranges) to perform the extraction of differences (or B-tree traversal) in parallel. The delta generator within the target region can use multiple threads to apply the downloaded differences to the latest snapshot in parallel.

[0030] A "shared database" (SDB) can refer, for the purposes of this disclosure, in one embodiment, to a key-value store that components (e.g., replicator fleet) within both the control plane and the data plane of a file system can read from and write to in order to communicate with each other. In one embodiment, the SDB can be part of a B-tree.

[0031] A "file system communicator" (FSC) can refer, in one embodiment, to the file manager layer that runs on storage nodes within the data plane of a file system. This service is useful for file creation requests, deletion requests, read requests, and write requests and works with an FNS server (e.g., Orca) to service IO to clients. The replicator fleet can communicate with a number of storage nodes, thereby distributing the read / write operations of file system data across the storage nodes.

[0032] In certain embodiments, a "blob" can refer to a data type for storing information (e.g., a formatted binary file) in a database. A blob is generated during replication by a source region and uploaded to an object store (i.e., object storage) within a target region. A blob can include keys and values of a binary tree (B-tree) as well as file data. A blob within an object store is called an object. Pairs of B-tree keys and values and the data associated therewith are packed together into the blob that is uploaded to the object store within the target region.

[0033] In certain embodiments, a "manifest" can refer to information transmitted by a file system within a source region (referred to herein as the source file system) to a file system within a target region (referred to herein as the target file system) to facilitate the inter-region replication process. There are two types of manifest files: a master manifest and a checkpoint manifest. A range manifest file (or master manifest file) is created by the source file system at the start of the replication process and describes information (e.g., B-tree key ranges) required by the target file system. A checkpoint manifest file is created after a checkpoint within the source file system and notifies the target file system of the number of blobs included in the checkpoint and uploaded to the object store, in response to which the target file system can then download that number of blobs.

[0034] "Difference", in certain embodiments, may refer to the differences identified between two specific snapshots after a replicator has recursively visited all nodes of the B-tree (also referred to herein as traversing the B-tree). A difference generator identifies key-value pairs of the B-tree with respect to the differences, traverses the B-tree nodes, and obtains file data associated with the B-tree keys. The difference between two snapshots may include multiple blobs. The term "difference", when used in the context of uploading information by a source file system to an object store and downloading from the object store by a target file system, may include blobs and manifests.

[0035] "Object", in certain embodiments, may refer to a partial set of information representing the entire difference during an inter-region replication cycle, and is stored in an object store. An object may be of a size of several megabytes stored at a specific location within a bucket of the object store. An object may contain many differences (i.e., blobs and manifests). A blob uploaded and stored in an object store is called an object.

[0036] "Bucket", in certain embodiments, may refer to a container that stores objects in a compartment within an object storage namespace (tenant). In the present disclosure, a bucket is used by a source replicator to store differences protected using server-side encryption (SSE), and is also used by a target replicator to download changes and apply them to a snapshot.

[0037] "Difference application" may, in certain embodiments, refer to the process of applying the differences downloaded by the target file system to the latest snapshot to create a new snapshot. Difference application may include analyzing the manifest file, applying snapshot metadata, inserting the keys and values of the B-tree into the B-tree, and storing the data associated with the B-tree keys (i.e., the file data or the data portion of the blob) in local storage. Snapshot metadata is created and applied at the start of the replication cycle.

[0038] "Region" may, in certain embodiments, refer to a logical abstraction corresponding to a geographic area. Each region may include one or more connected data centers. Regions are independent of other regions and may be separated by vast distances.

[0039] End-to-End Inter-Region Replication Architecture The end-to-end inter-region replication architecture provides new technologies for end-to-end file storage replication and security between file systems within different cloud infrastructure regions. In certain embodiments, the file storage service generates differences between snapshots in the source file system and, during disaster recovery, transfers the differences and related data via high-throughput object storage to recreate a new snapshot in the target file system located in a different region. The file storage service utilizes new technologies to achieve scalable, reliable, and restartable end-to-end replication. New technologies for ensuring the secure transfer and consistency of information during end-to-end replication are also described.

[0040] In the context of the cloud, a realm refers to a logical collection of one or more regions. Realms are typically separated from each other and do not share data. Within a region, the data centers within the region can be organized into one or more availability domains (ADs). Availability domains are separated from each other, are fault tolerant, and have a very low probability of failing simultaneously. An AD is configured such that a failure in one AD within a region is unlikely to affect the availability of other ADs within the same region.

[0041] Current disaster recovery practices can include taking periodic snapshots and resynchronizing those snapshots to a different availability domain (AD) or another file system within the region. Resynchronization is manageable and maintained by the customer, but lacks a user interface to display progress, is a slow serialized process, and is not easy to manage as data grows over time.

[0042] Accordingly, different approaches are needed to address these and other challenges. The file storage replication of a cloud service provider (e.g., Oracle Cloud Infrastructure (OCI)) disclosed in this disclosure is based on incremental snapshots and provides a consistent point-in-time view of the entire file system by propagating the differences in changed data from a primary AD within a region to a secondary AD within the same or a different region. As used herein, the primary site (or source side) may refer to the location where the file system is located and where the replication process for disaster recovery is initiated (e.g., an AD or a region). The secondary site (or target side) may refer to the location where the file system receives information from the file system within the primary site during the replication process and becomes the new operational file system after disaster recovery (e.g., an AD or a region). The file system located at the primary site is called the source file system, and the file system located at the secondary site is called the target file system. Accordingly, the primary site, source side, source region, primary file system, or source file system (referring to one of the file systems on the source side) may be used interchangeably. Similarly, the secondary site, target side, target region, secondary file system, or target file system (referring to one of the file systems on the target side) may be used interchangeably.

[0043] The file storage service (FSS) of the present disclosure supports complete disaster recovery for failover or failback with minimal management effort. Failover is a series of actions to make the secondary site / target site the primary / source (i.e., start providing services for the workload), which may include planned failover and / or unplanned failover. A planned failover (sometimes called a planned migration) is initiated by the user to perform a planned failover from the source side (e.g., source region) to the target side (e.g., target region) without data loss. An unplanned failover is the case where, for example, due to a disaster, the source side stops unexpectedly and the source side is lost, so the user needs to start using the target side. Failback is to restore the primary side / source side to become the primary / source again before the failover. Failback may occur when, after a planned failover or an unplanned failover and a trigger event (e.g., power outage) ends, the user wants to reuse the source side as the primary AD by reversing the failover process. The user can resume either from the last point in time on the source side before the trigger event or from the latest changes on the target side. The replication process described in the present disclosure can maintain the identity of the file system after round-trip replication. In other words, the source file system can provide services for the workload again after performing a failover and then a failback.

[0044] The technologies disclosed in this disclosure (e.g., methods, computer-readable media, and systems) use consistent snapshot information to replicate differences between snapshots from a source region to multiple remote (or target) regions, and then scan (or recursively visit) all keys and values within one or more file trees (e.g., B-trees) of a source file system (referred to herein as "scanning the B-tree" or "scanning keys") to construct consistent information (e.g., differences or discrepancies between keys and values of two snapshots created at different times), including region-to-region replication of file system data and / or metadata. The constructed consistent information is in blob format and transferred to the remote side (e.g., the target region) using an object interface, such as an object store (described later), so that the target file system on the remote side can immediately detect the information transferred via the object interface and start downloading and applying it. This process is realized using a control plane and can be extended to thousands of file systems and hundreds of replication machines. Both the source file system and the target file system can operate simultaneously and asynchronously. Operating simultaneously means that the data upload process by the source file system and the data download process by the target file system can occur simultaneously. Operating asynchronously means that the source file system and the target file system can each operate at their own pace without waiting for each other at all stages, e.g., with different start times, end times, processing speeds, etc.

[0045] In one embodiment, multiple file systems may exist in the same region and be represented by the same B-tree. Each of these file systems within the same region can be replicated independently across regions. For example, file system A may have a set of parallel execution replicator threads that scan the B-tree to perform replication of file system A. File system B, which is represented by the same B-tree, may have another set of such parallel execution replicator threads that scan the same B-tree to perform replication of file system B.

[0046] Regarding security, cross-region replication is completely secure. Information is transferred securely and applied securely. The disclosed technology provides separation between the source region and the target region such that keys are not shared between the two without being encrypted. Thus, if the source key is involved, the target is not affected. Further, the disclosed technology includes ways to read keys, convert those keys into a certain format, and upload and download those keys securely. Since different keys are created and used in different regions, separate keys are created at the target and applied to the information with a target-centric security mechanism. For example, FSS generates a session key that is only valid during one replication cycle or session to encrypt data uploaded from the source region to the object store and decrypt data downloaded from the object store to the target region. Separate keys are used locally within the source region and the target region.

[0047] In the disclosed technology, each upload process and download process through the object store during replication has different pipeline stages. For example, the upload process has multiple pipeline stages including scanning a B-tree to generate differences, accessing storage I / O, and uploading data (or blobs) to the object store. The download process has multiple pipeline stages including downloading data, applying differences to a snapshot, and storing the data in storage. Each of these pipelines also includes parallel processing threads to improve the throughput and performance of the replication process. Further, the parallel processing threads can take over a failed processing thread and resume the replication process from the point of failure without restarting from the beginning. Thus, the replication process is highly scalable and reliable.

[0048] Figure 1 illustrates an exemplary concept of the target recovery point in time (RPO) and the target recovery time (RTO) for an unplanned failover according to an embodiment. The RPO is the maximum allowable range of data loss between a primary site failure and the availability of the secondary site (usually specified in minutes). As shown in Figure 1, the primary site A102 encounters an unplanned incident at time 110 and triggers the failover replication process by copying the latest snapshot and its delta to the secondary site B104. The information first copied reaches the secondary site B104 at time 112. The primary site A102 completes the copy of the information to the secondary site B104 at time 114, and the secondary site B104 completes the replication process at time 116. Thus, the secondary site B104 becomes fully operational at time 116. As a result, the user's data is not accessible within the primary site A110 starting from point 110 until the point 116 where the data becomes available again. Therefore, the RPO is the time between point 110 and point 116. For example, if there is data equivalent to 10 minutes that the user is not interested in, the RPO is 10 minutes. If the data loss exceeds 10 minutes, the RPO is not met. An RPO of 0 means synchronous replication.

[0049] RTO is the time it takes for the secondary to become fully operational after a failure, so that the user can access the data again (usually specified in minutes). RTO is considered from the perspective of the secondary site. Referring again to Figure 1, the primary site A102 starts the failover replication process at time 120. However, the secondary site B104 remains operational until time 122 when it recognizes the incident (or power outage) at the primary site A102. Therefore, the secondary site B104 stops its service at time 122. The secondary site B104 becomes fully operational at time 126 using the same failover replication process as described for RPO. Therefore, RTO is the time between 122 and 126. Here, the secondary site B104 becomes able to assume the role of the primary site. However, for customers using the primary site A102, the service loss is between time 120 and 126.

[0050] The primary (or source) site is where the action is taking place, and the secondary (or target) site is inactive and cannot be used until a disaster occurs. However, the customer may be provided with a point in time to continue using for test-related activities at the secondary site. This relates to how the customer sets up replication, how the customer can start using the target if something goes wrong, and how the customer can return to the source after the source has failed over.

[0051] FIG. 2 is a simplified block diagram showing an architecture for inter-region remote replication according to an embodiment. In FIG. 2, the end-to-end replication architecture shown includes two regions: a source region 290 and a target region 292. Each region may include one or more file systems. In one embodiment, the end-to-end replication architecture includes data planes 202 and 212, control planes (only control APIs 208a - n and 218a - n are shown), local storages 204 and 214, an object store 260, and a key management service (KMS) 250 for both the source region 290 and the target region 292. FIG. 2 shows only one file system 280 within the source region 290 and one file system 282 within the target region 292 for simplicity. If there are two or more file systems in one region, the same replication architecture is applied to each pair of source and target file systems. In one embodiment, by utilizing parallel processing threads, multiple inter-region replications can occur simultaneously between each pair of source and target file systems. In some embodiments, one source file system can be replicated to different target file systems located in the same target region. Further, file systems within a region may share resources. For example, certain resources within the KMS 250, the object store 260, and the data plane may be shared by many file systems within the same region, depending on the implementation.

[0052] The data plane within the architecture includes local storage nodes 204a - n and 214a - n, as well as replicators (or replicator fleets) 206a - n and 216a - n. The control API hosts within each region perform all orchestration between different regions. The FSS receives a request from a customer to set up replication between a source file system 280 and a target file system 282 where the customer's data will be moved to. The control plane 208 obtains the request, performs resource allocation, and notifies the replicator fleet 206a - n within the source data plane 202 to start uploading data 230a from different snapshots to the object storage 260 (or sometimes called that only the differences are uploaded). APIs are available to assist the customer in setting the target time and the recovery time objective (RTO) of the replication. The replication model disclosed in this disclosure is a "push - based" model based on snapshot differences, that is, the source region initiates the replication.

[0053] As used herein, the data 230a and 230b transferred between the source file system 280 and the target file system 282 are general terms and may include an initial snapshot, keys and values of different B - trees between two snapshots, file data (e.g., fmap), snapshot metadata (i.e., a set of B - tree keys of snapshots reflecting different snapshots obtained within the source file system), and other information (e.g., manifest files) that helps facilitate the replication process.

[0054] Regarding the data plane of the inter-region replication architecture, the replicator is a component within the data plane of the file system. The replicator performs differential generation or differential application on the file system according to the region where the file system is located. For example, the replicator fleet 206 within the file system 280 of the source region performs the generation and replication of the differential 230a. The replicator fleet 216 within the file system 282 of the target region downloads the differentials 230b and applies them to the latest snapshot within the file system 282 of the target region. The file system 282 of the target region can also use the control plane and workflows to guarantee end-to-end transfer.

[0055] All incremental operations are based on snapshots, which are existing resources within file storage as a service. A snapshot is a point in time, data point, or image of what is happening within the file system and is executed periodically within the file system 280 of the source region. For the very first replication (e.g., where replication has not been obtained before), FSS obtains a base snapshot, which is a snapshot of all the contents of the source file system, and transfers all of that content to the target system. In other words, the replicator reads from the storage layer of that specific file system and stores all the data in the object storage bucket.

[0056] After the data plane 202 of the source file system 280 uploads all the data 230a to the object storage (or object store) 260, the source-side control plane 208 notifies the target-side control plane 218 that there is new work to be done on the target side, and then this notification is relayed to the target-side replicator. Thereafter, the target-side replicators 216a~n start to download objects (e.g., initial snapshots and deltas) from the object storage bucket 260 and apply the deltas captured on the source side.

[0057] In the case of a base copy (e.g., the entire contents of the file system up to a point in time ranging from the past 5 days to 5 years), the upload process may take time. To assist in meeting service-level goals regarding time and performance, the source system 280 can take replication snapshots at specific intervals such as one hour. The source side 280 can then transfer all the data within that one hour to the target side 282 and take new snapshots every hour. If there is any cache with many changes, the replication can be set to a shorter replication interval.

[0058] To illustrate the above, consider a situation where a first snapshot is created on a file system within a source region (referred to as the source file system). Replication is performed periodically, and thus the first snapshot is replicated to a file system within a target region (referred to as the target file system). Thereafter, when some update is performed within the source file system, a second snapshot is created. If an unplanned power outage occurs after the second snapshot is created, the source file system attempts to replicate the second snapshot to the target file system. During failover, the source file system may well identify the difference (i.e., the delta) between the first snapshot and the second snapshot, which includes the keys and values of the B-tree and the file data associated therewith within the B-tree representing both the first snapshot and the second snapshot. Next, deltas 230a and 230b are transferred from the source file system to the target file system via the object store 260 within the target region, and the target file system recreates the second snapshot by applying the deltas to the first snapshot previously established within the target region. When the second snapshot is created in the target file system, the failover replication process is complete and the target file system is ready to operate.

[0059] Regarding the control plane and its application programming interfaces (APIs), the control plane provides instructions for the data plane that includes replicators as executors to execute instructions. Storage (204 and 214) and replicator fleets (206 and 216) are both within the data plane. The control plane is not shown in Figure 2. As used herein, a "cycle" may refer to a period that starts when the source file system 280 begins to transfer data 230a to the target file system 282 and ends when the target file system 282 has received all the data 230b and completed the application of the received data. The data 230a - b is captured on the source side and then applied on the target side. When all the changes on the target side are applied to the cycle, the source file system 280 acquires another snapshot and starts another cycle.

[0060] The control APIs (208a - n and 218a - n) are a set of hosts within the overall architecture of the control plane and execute the configuration of the file system. The control APIs are responsible for transmitting state information between different regions. State machines that track various state activities within a region, such as the progress of a job, the location of keys, and future tasks to be executed, are distributed across multiple regions. All of this information is stored in the control plane of each region and transmitted between regions via the control APIs. In other words, the state information relates to the details of the life cycle, the details of the differences, and the life cycle of resources. The state machine can also be useful for tracking the progress of replication and cooperating with the data plane to estimate the time taken for replication. Therefore, the state machine can provide the user with status regarding whether the replication is proceeding as planned and the normality of the job.

[0061] Furthermore, communication between the control APIs (208a - n) of the source file system 280 and the control APIs (218a - n) of the target file system 218 in a different region includes the transfer of snapshots and metadata for creating an exact copy from the source to the target. For example, if a customer periodically takes snapshots within the source file system, the control plane can ensure that snapshots of the same user, including metadata tracking, transfer, and recreation, are created in the target file system.

[0062] The object store 260 in FIG. 2 (also referred to as an "object" herein) is an object storage service (e.g., Oracle's object storage service) that enables reading blobs and writing files for archival purposes. The advantages of using an object store are, first, ease of configuration, second, ease of streaming data to the object store, and third, having the advantage of security streaming as a reliable repository for maintaining information, all because there is no network loss, the data can be downloaded immediately, and it exists permanently. Direct communication between replicators within the source region and the target region is possible, but direct communication requires the configuration of an inter-region network, which is not scalable and difficult to manage.

[0063] For example, if there is a large amount of data being moved from a source to a target, the source can upload the data to the object store 260, and the target 282 does not need to wait for all the information uploaded to the object store 260 to start downloading. Thus, both the source 280 and the target 282 can operate continuously and simultaneously. The use of the object store enables the system to scale and achieve a higher throughput. Further, the Key Management Service (KMS) 250 can control access to the object store 260 to ensure security. In other words, the source tries to move the data from the source region as fast as possible and hold the data somewhere so that the data is not lost before it can be applied to the target.

[0064] Compared to using a network pipe with packet loss and recovery issues, the use of the object store 260 between the source region and the target region enables continuous data streaming where hundreds of file systems can be written from the source region to the object store, and at the same time, the target region can apply hundreds of files simultaneously. Thus, data streaming via the object store can achieve high throughput. Further, both the source region and the target region can operate at their own speeds for uploading and downloading.

[0065] Each time a user changes some data within the source file system 280, a snapshot is taken and the difference before and after the change is updated. These changes are accumulated in the source file system 280 and can be streamed to the object store 260. The target file system 282 can detect that the data is available in the object store 260 and immediately download the changes and apply them to that file system. In some embodiments, only the differences are uploaded to the object storage after the base snapshot.

[0066] In some embodiments, the replicator can communicate with many different regions (e.g., from Phoenix to Ashburn and further to other remote regions), and the file system can manage many different endpoints on the replicator. Each replicator 206 within the source file system 280 can maintain a cache of these object storage endpoints, and further, in cooperation with the KMS 250, generate a transfer key (e.g., a session key) for encrypting the data addresses of the data in the object storage 260 (e.g., server-side encryption or SSE) to protect the data stored in the bucket. There is one master bucket for each AD within the target region. A bucket is a container that stores objects in a compartment within the object storage namespace (tenancy). Since all remote clients can communicate with the bucket and write information in a specific format, the information of each file system can be uniquely identified, preventing the mixing of data from different customers or file systems.

[0067] The object store 260 is a high-throughput system, and the techniques disclosed in the present disclosure can utilize the object store. In one embodiment, the replication process includes multiple pipeline stages, a B-tree scan within the source file system 280, storage IO access, data upload to the object store 260, data download from the object store 260, and delta application within the target file system 282. Each stage includes parallel processing threads that participate in improving the performance of data streaming from the source region 290 to the target region 292 via the object store 260.

[0068] In one embodiment, each file system within the source region may include a set of replicator threads 206a - n that are executed in parallel to upload the differences to the object store 260. Each file system within the target region may also include a set of replicator threads 216a - n that are executed in parallel to download the differences from the object store 260. Since both the source side and the target side operate asynchronously simultaneously, the source can upload as fast as possible, while the target can start downloading after detecting that the differences are available in the object store. Thereafter, the target file system applies the differences to the latest snapshot and deletes the differences in the object store after the application. Thus, the FSS consumes very little space in the object store, and the object store has a very high throughput (e.g., gigabyte - scale transfers).

[0069] In one embodiment, multiple threads are also executed in parallel for storage I / O access (e.g., DASD) 204a - n and 214a - n. Thus, all processes related to the replication process, including accessing storage, uploading snapshots and data 230a from the source file system 280 to the object store 260, and downloading snapshots and data 230b to the target file system 282, include multiple threads that are executed in parallel to perform data streaming.

[0070] File storage is a local service of the AD. When a file system is created, that file system is within a specific AD. When a customer wants to transfer or replicate data from one file system to another file system within the same region or a different region, artifact (also called manifest) transfer may need to be used.

[0071] As an alternative to using an object store to transfer data, a network connection between remote machines (e.g., between source and target replicator nodes) can be set up, and VCN peering can be used to use Classless Inter-Domain Routing (CIDR) per region.

[0072] Referring again to FIG. 2, the Key Management System (KMS) 250 provides security for replication and provides storage services to a cloud service provider (e.g., OCI). In certain embodiments, the file systems 280 on the source (or primary) side and the target (or secondary) side use separate KMS keys, and key management is hierarchical. The reason for using separate keys is that if the source is compromised, an unauthorized actor cannot decrypt the target using the same key. FSS has a three-tier key architecture. Since the source and target use different keys during data transfer, the source first decrypts the data, re-encrypts it using an intermediate key, and then re-encrypts the data on the target side. FSS defines a session, and each session is one data cycle. A key for transferring data in that session is created. In other words, a new key is used for each new session. In other embodiments, a key can be used for two or more sessions (e.g., two or more data transfers) before creating another key. The key is not transferred via the object store 260, and the key is only available on the source side and is not visible from outside the source for security reasons.

[0073] The replication cycle (also called a session) is periodic and adjustable. For example, the replicators (206a~n and 216a~n) perform replication once every hour. The cycle starts when a new snapshot is created on the source side 280 and ends when all the differences 230b have been applied to the target side 282 (i.e., the target has reached the DONE state). Each session is completed before another session starts. Thus, there is always only one session and no overlap between sessions.

[0074] Secret management (i.e., replication using the KMS) processes the transfer of confidential materials between the source (primary) file system 290 and the target (or secondary) file system 292 using the KMS250. The source file system 280 calculates the differences, reads the file data, and then uses the encryption key of the local file system to decrypt the file data in cooperation with the key management service. Next, the source file system 280 generates a session key (referred to as a delta encryption key (DEK)), encrypts it to become an encrypted session key (referred to as a delta transfer key (DTK)), and transfers the DTK to the target file system 282 via the respective control planes 208 and 218. The source file system 280 further encrypts the data 230a using the DEK and uploads the encrypted data 230a to the object store 260 via the Transport Layer Security (TLS) protocol. Next, the object store 260 uses server-side encryption (SSE) to ensure the security for the storage of the data (e.g., differences, manifest, and metadata) 230a.

[0075] The target file system 282 securely obtains the encrypted session key DTK via the control plane 218 (using HTTPS via inter-region API communication), decrypts the session key DTK via the KMS 250 to obtain the DEK, and places the DEK at a location within the target region 292. When a replication job is scheduled within the target file system 282, the DEK is provided to a replicator (one of the replication fleets 216a - n), and the replicator uses this key to decrypt the data (e.g., the delta containing file data) 230b downloaded from the object store 260 for application and re-encrypts the file data using the local file system key.

[0076] Replication between the source file system 280 and the target file system 282 is a parallel process, and both the source file system 280 and the target file system 282 operate at their own paces. When the source side completes the upload (which can occur before the target download process), the source side cleans up the memory and removes all keys. When the target completes the application of the delta to the latest snapshot, it similarly cleans up the memory and removes all keys. The FSS service also releases the KMS key. In other words, there are two copies of the session key, one within the source file system 280 and another within the target file system 282. Both copies are deleted at the end of each session, and a new session key is generated for the next replication cycle. This process ensures that the same key is not used for different purposes. Further, the session key is encrypted by the file system key, creating a two-fold protection. This is to ensure that only a specific file system can use this session key.

[0077] Figure 3 is a simplified schematic diagram of the components involved in inter-region remote replication according to an embodiment. In one embodiment, components called the difference generator (DG) 310 in the source region A302 and 330 in the target region B304 are part of the replicator fleet 318 and operate on thousands of storage nodes within the fleet. The replicator 318 in the source region A makes remote procedural calls (RPCs: Remote Procedural Call) to the difference generator 310 (e.g., obtaining a set of keys and values, locking a block, etc.), and collects the keys, values, and data pages of the B-tree from a direct-access storage device (DASD: Direct-Access Storage Device) 314, which is a replication storage service for accessing storage and is regarded as a data server. The DG 310 in the source region A is a helper for the replicator 318, divides the key range of the differences, and packs all the keys / values in a specific range into a blob to be returned to the replicator 318. Both regions have a plurality of storage nodes 322 and 342 connected to DASDs 314 and 334, and each node contains a large number of disks (e.g., 10TB or more).

[0078] In one embodiment, the file system communicators (FSCs) 312 and 332 in both regions are metadata servers that help update the source file system for user updates to the system. The FSCs 312 and 332 are used for file system communication, and the difference generator 310 is used for replication. Both the DG 310 and 330 and the FSCs 312 and 332 are metadata servers. User traffic passes through the FSCs 312 and 332 and the DASDs 314 and 334, while replication traffic passes through the DG. In an alternative embodiment, the function of the FSC can be merged with the function of the DG.

[0079] In one embodiment, the shared databases (SDBs) 316 and 336 of both regions are key value stores, and through these components, both the control plane and the data plane (e.g., the replicator fleet) can read and write for each other to communicate with each other. The control planes 320 and 340 of both regions can place new jobs into queues within their respective shared databases 316 and 336, and the replicator fleets 318 and 338 continuously read the queues within the shared databases 316 and 336, and when the replicator fleets 318 and 338 detect a job request, they can initiate file system replication. In other words, the shared databases 316 and 336 are conduits between the replicator fleet and the control plane. Further, the shared databases 316 and 336 are resources distributed across different regions, and the IO traffic to or from the shared databases 316 and 336 should be minimized. Similarly, the IO traffic to or from the DASD needs to be minimized so as not to affect the user's performance. However, the replication process may be adjusted because it is a secondary service compared to the primary service.

[0080] The replicator fleet 318 within the source region A can cooperate with the DG310 to start scanning the B-tree in the file system within the source region A, collect keys and values, and convert those keys and values into flat files or blobs to be uploaded to the object store. Once the data blobs (including keys, values, and actual data) are uploaded, the target can immediately apply those data blobs without waiting for a large number of blobs to be present in the object store 360. The object store 360 is located in the target region B for disaster recovery reasons. The goal is to push from the source to the target region B as soon as possible and keep the data safe.

[0081] To replicate thousands of file systems, multiple replicators exist that optimize space using lower-cost machines with smaller footprint and schedule as many replications as possible while ensuring fair bandwidth allocation among those machines. The replicator fleets 318 and 338 in both regions are run on virtual machines that can be automatically scaled up and down to build the entire fleet for running replications. The replicators and replication services can adapt dynamically based on capacity to support each job. If the load on one replicator is high, another replicator can be selected to share the load. Different replicators in the fleet can balance the load among each other to ensure that the job can continue and is not stopped due to overloading an individual replicator.

[0082] FIG. 4 is a simplified flowchart showing steps executed during inter-region remote replication according to one embodiment.

[0083] Step S1: When the customer sets up replication, the customer provides a source (or primary) file system (A) 402, a target (or secondary) file system (B) 404, and an RPO. The file systems are uniquely identified by file system identification information (e.g., Oracle Cloud ID or OCID), which is a globally unique identifier for the file system. The data is stored in a file storage service ("FSS") control plane database.

[0084] Step S2: The source (A) control plane (CP-A) 410 adjusts to periodically create system snapshots at regular intervals (smaller than the RPO), and notifies the data plane (including the replicator / uploader 412) of the latest snapshot and the last snapshot that was successfully copied to the target (B) file system 404.

[0085] Step S3: CP-A 410 notifies the replicator 412 (or uploader), which is a component within the data plane, to copy the latest snapshot.

[0086] S3a: The replicator 412 of the source (A) scans the B-tree to calculate the difference between two specific snapshots. The existing key infrastructure is used to decrypt the file system data.

[0087] S3b: These differences 414 are uploaded to the object store 430 within the target (B) region (the data can be compressed and / or deduplicated during the copy). This upload can be executed in parallel by multiple replicator threads 412.

[0088] Step S4: CP-A 410 notifies the target (B) control plane (CP-B) 450 of the completion of the upload.

[0089] Step S5: CP-B 450 calls the target replicator B452 (or downloader) to apply the differences.

[0090] S5a: Replicator B452 downloads the data 454 from the object store 430.

[0091] S5b: Replicator B452 applies these differences to the target file system (B).

[0092] Step S6: After the differential application is completed, CP-A410 is notified of the new snapshots currently available at target (B).

[0093] Step 7: The inter-region remote replication process repeats from Step S2 to Step S6.

[0094] Figure 5 is a simplified diagram showing a high-level concept of B-tree scanning according to an embodiment. The B-tree structure can be used within a file system. The differential generator scans the B-tree and ensures the consistency of the scan. In other words, the scan confirms that the keys and values are as expected at the end of the scan so that data corruption cannot occur, and captures all the information between any two snapshots. The file system is a transactional file system that may be modified, and since another user may update the same transaction or data, the user needs to be aware of the changes and repeat the transaction.

[0095] Keys, values, and snapshots are immutable (i.e., they cannot be changed except that the garbage collector can remove them). As shown in FIG. 5, there are many snapshots (Snapshot 1 to Snapshot N) in the file system. When the diff generator scans the B-tree keys (510 to 560) in the source file system, the garbage collector 580 may come in and clean up the keys of the snapshots that it considers garbage, so the snapshots may be removed. When the diff generator scans the B-tree keys, the diff generator needs to ensure that the keys associated with the remaining snapshots (i.e., the keys not removed by the garbage collector) are copied. When keys, such as 540 and 550, are removed by the garbage collector 580, the B-tree page can be reduced, for example, from 2 pages before garbage collection to 1 page after garbage collection. A way for the diff generator to ensure consistency when scanning B-tree keys is for the garbage collector 580 to confirm that it has not changed or deleted any keys in the page (or section between two snapshots) that the diff generator has just scanned (e.g., between two keys). Once consistency is confirmed, the diff generator collects the keys and sends them to the replicator for processing and uploading.

[0096] The B-tree key can indicate what has changed. The techniques disclosed in this disclosure can determine which B-tree keys are new and what has been updated between two snapshots. The diff generator can collect the metadata portion, keys and values, and related data, and then send it to the target. The target can understand that the received information is within the range of the two snapshots and applies to the target file system. The diff generator (or a thread of the diff generator) scans the section between two keys, verifies its consistency, and then uses the last end key as the next start key for the next scan. This process is repeated until all keys are checked, and the diff generator collects related data each time the consistency is verified.

[0097] For example, when a file is changed within a file system (e.g., created, deleted, and then recreated), this process creates multiple versions of the corresponding file directory entry. During the replication process, the garbage collector may clean up (or remove) the version of the file directory entry corresponding to the deleted file, which may cause a consistency issue called a whiteout. A whiteout occurs when there is a mismatch between the source file system and the target file system, because the target file system may fail to reconstruct the original snapshot chain that includes the changed file. The disclosed techniques can detect whiteout files (i.e., changed files affected by the garbage collector) during B-tree scanning, extract the version of the changed file that is not affected, and provide relevant information to the target file system within the same replication cycle to ensure the consistency between the source file system and the target file system by properly reconstructing the correct snapshot chain.

[0098] Figures 6A and 6B are diagrams showing the pipeline stages of inter-region replication according to an embodiment. The inter-region replication of the source file system disclosed in the present disclosure includes four pipeline stages, namely, the start of inter-region replication, the B-tree scan in the source file system (i.e., the differential generation pipeline stage), the storage IO access for retrieving data (i.e., the data read pipeline stage), and the data upload to the object store (i.e., the data upload pipeline stage), which are included within the source file system. The target file system includes four pipeline stages in a similar but reverse order, namely, the preparation for inter-region replication, the download of data from the object store, the application of the difference within the target file system, and the storage IO access for storing the data. Figure 6A shows the four pipeline stages within the source file system, but the same concept applies to the target file system as well. Figure 6B shows the processes involved in the pipeline stages and the interactions between components. These pipeline stages can all operate in parallel. Each pipeline stage operates independently and can pass information to the next pipeline stage when the processing at the current stage is completed. Each pipeline stage receives a portion of the overall bandwidth and is guaranteed not to use more than necessary. In other words, resources are fairly allocated among all jobs. When no other jobs are operating within the system, the operating job can acquire as many resources as possible.

[0099] Threads within each pipeline stage also execute tasks (i.e., when a thread fails, it does not affect other threads) independently of each other in parallel (or simultaneously) within the same pipeline stage. Further, the tasks (or replication jobs) executed by threads at each pipeline stage are restartable, i.e., when a thread fails, a new thread (also called a replacement thread) can take over from the failed thread and continue the original task from the last successful point.

[0100] In some embodiments, the B-tree scan may be performed using parallel processing threads within the source file system 280. The B-tree may be divided into a plurality of key ranges between the first key and the last key in the file system. The number of key ranges may be determined by the customer. For each file system, a plurality of (e.g., about 8 to 16) range threads may be used for the B-tree scan. One range thread can perform a B-tree scan of one key range, and all range threads operate simultaneously in parallel. The number of threads used varies depending on factors such as the size of the file system, the availability of resources, and the bandwidth for balancing resource and traffic congestion. Usually, the number of key ranges is more than the number of available range threads to fully utilize the range threads. Thus, the B-tree scan is scalable and can be processed by simultaneous parallel scans (e.g., using multiple threads).

[0101] After the differential generator scans the pages, if some keys are missing and thus some keys are inconsistent, the system may remove the ongoing uncommitted transactions and return to the starting point for re-scanning. During the repetition of the B-tree scan due to the inconsistency, the differential generator may ignore the missing keys and the data associated with them by not collecting them because these associated data are regarded as garbage in order to minimize the amount of information to be processed or uploaded to the target side. Thus, the B-tree scan and data transfer can be made more efficient. Further, the differential generator does not need to wait for the garbage collector to remove the information to be deleted before scanning the B-tree keys. For example, keys have dependencies on each other. If a key or iNode points to a block that has been deleted or should be deleted by the garbage collector, the system (or the differential generator) can itself grasp that a particular block is garbage and that the differential generator does not need to carry it.

[0102] The differential generator usually makes no changes on the source side (e.g., does not delete the keys or blocks of data considered as garbage), and simply does not copy them to the target side. The B-tree scanning process and garbage collection are asynchronous processes. For example, when the block of data pointed to by a key no longer exists, the file system can flag that key as garbage, and notify that the key should not be changed (e.g., is immutable) and only the garbage collector can remove that key. The differential generator can continue scanning the next key without waiting for the garbage collector. In other words, the differential generator and the garbage collector can proceed at their own paces.

[0103] In FIG. 6A, when the source region starts the inter-region replication process that can include many file systems, the main threads 610a~n select replication jobs (one job per file system). The main thread of the file system (e.g., 610a or 610 for later use) within the source region (i.e., the source file system) communicates with the differential generator 620 (shown in FIG. 6B) to obtain the number of key ranges requested by the customer and update the corresponding records in the SDB 622. Once the main thread 610 of the source file system knows the number of key ranges required, it further creates a set of range threads 612a~n based on the number of key ranges required. These range threads 612a~n are executed by the differential generator 620. These range threads 612a~n initialize the GETKEYVAL buffer 640 (shown in FIG. 6B), update the checkpoint record 642 in the SDB 622 (shown in FIG. 6B), and perform storage I / O access 644 by interacting with the DASD I / O threads 614a~n.

[0104] In one embodiment, each main thread 610 is responsible for monitoring all range threads 612a - n that it creates. During replication, the main thread 610 may generate a master manifest file that summarizes the entire replication. The range threads 612a - n generate a range manifest file that includes the number of key ranges (i.e., the further subdivision of the entire replication), and then generate a checkpoint manifest (CM) file for each range to provide updates to the target file system regarding the number of blobs per checkpoint, where the checkpoints are created during a B - tree scan. One checkpoint is created by a range thread 612. When the main thread 610 determines that all range threads 612a - n have completed, it creates a final checkpoint manifest (CM) file that includes an end - of - file marker, and then uploads the CM file to the object store so that the target file system can understand the progress within the source file system. The CM file includes an overview of all individual ranges, such as the number of ranges, the final state of the checkpoint records, and other information.

[0105] Range threads 612a~n are used for parallel processing to significantly reduce the time for B-tree traversal of a large source file system. In one embodiment, the B-tree keys are divided into ranges of approximately equal size. One range thread can perform a B-tree traversal of one key range. The number of range threads 612a~n used varies depending on factors such as the size of the file system, resource availability, and bandwidth for balancing resources, the amount of data generated, and traffic congestion. Typically, the number of key ranges is about two to four times greater than the number of available range threads 612a~n for fully utilizing the range threads. Each of the range threads 612a~n has a dedicated buffer (GETKEYVAL) 640 that contains jobs available for work. Each range thread 612 operates independently of other range threads and periodically updates checkpoint records 642 in the SDB622.

[0106] Range threads 612a~n may need to collect file data (e.g., FMAP) associated with the B-tree keys and request IO access 644 to storage when traversing the B-tree (i.e., when recursively visiting all nodes of the B-tree). These IO requests are queued by each range thread 612 so that DASD IO threads 614a~n (i.e., the data read pipeline stage) can handle those IO requests. These DASD IO threads 614a~n are common threads shared by all range threads 612a~n. After the DASD IO threads 614a~n obtain the requested data, the data is placed in the output buffer 646 to serialize the data into blobs so that the object threads 616a~n of the replicator (i.e., the data upload pipeline stage) upload it to the object store located in the target region. Each object thread selects an upload job that may include a portion of all the data to be uploaded, and all object threads execute the uploads in parallel.

[0107] FIG. 7 is a diagram showing a hierarchical structure in the FSS data plane according to an embodiment. In FIG. 7, the replicator fleet 710 includes four layers: a job layer 712, a delta generator client 714, an encryption / DASD IO 716, and an object 718. The replicator fleet 710 is a single process responsible for communicating information with a storage fleet 720, a KMS 730, and an object storage 740. In one embodiment, the job layer 712 polls the SDB 704 for a job 706 that has been queued, which can be either an upload job or a download job. The replicator fleet 710 includes VMs (or threads) that select enqueue replication jobs up to the maximum capacity. A replicator thread may own a part of a replication job, but coordinates with another replicator thread that owns the remaining part of the same replication job to complete the entire replication job simultaneously. The replication jobs executed by the replicator fleet 710 are restartable in that if a replicator thread fails during replication, another replicator thread can take over and continue from the last successful point to complete the job that the failed replicator thread originally owned. If a strayed replicator thread (e.g., a replicator thread that fails and restarts) competes with another replicator thread, the FSS can avoid the conflict by using a mechanism called a generation number to cause both replicator threads to update different records.

[0108] The differential generator client layer 714 performs a B-tree scan by accessing the differential generator server 724 in which the B-tree in the storage fleet 720 exists. The encryption / DASD IO layer 716 is responsible for security and storage access roles. After the B-tree scan, the replicator fleet 710 may request IO access via the encryption / DASD IO layer 716 to access the DASD range 722 of the file data associated with the differences identified during the B-tree scan. The replicator fleet 710 and the storage fleet 720 both periodically update the status of the control API 702 (e.g., checkpoint and lease of the replicator fleet 710) via the SDB 704, enabling the control API 702 to trigger an alarm or execute an action if necessary.

[0109] During the inter-region replication process, the encryption / DASD IO layer 716 exchanges information with the KMS and FSK fleet 730 on the target side to create a session key (or snapshot encryption key), and uses the FSK for encryption and decryption of the session key. Finally, the object layer 718 is responsible for uploading differential and file data from the source file system to the object store 740 and downloading them from the object store 740 to the target file system.

[0110] The data plane of the FSS is responsible for differential generation. The data plane stores FSS data using a B-tree, which includes various types of key-value pairs including, but not limited to, a leader block, a superblock, an iNode, a file name key, a cookie map (cookies associated with directory entries), and a block map (also called FMAP in the case of file content data).

[0111] These B-tree keys are processed together by replicators and diff generators within the data plane. An algorithm for calculating the pairs of keys and values (i.e., part of the diff) that have changed between two specific snapshots within the file system continuously reads the keys, returns the keys to the replicator using a transaction budget, and finally ensures that the transaction is committed to obtain a consistent pair of keys and values for processing.

[0112] In other embodiments, diff generation and calculation can be scalable. A scalable approach can calculate the diff (i.e., the change in the pairs of keys and values) between two snapshots by utilizing multiple threads to divide the B-tree into many key ranges. A pool of threads (i.e., diff generators) can perform a scan of the B-tree (i.e., traverse the B-tree) and calculate the diffs in parallel.

[0113] FIG. 8 shows a simplified exemplary binary large object (BLOB) format according to an embodiment. A blob is a data type for storing information (e.g., binary data) in a database. The blob is generated by the source region during replication and uploaded to the object store. The target region needs to download and apply the blob. Blob and object can be used interchangeably depending on the context.

[0114] During the B-tree scan, when the difference generator encounters the iNode of a specific file (i.e., data content) and its block map (also called FMAP, data associated with the B-tree key), the difference generator, in cooperation with the replicator, traverses all the pages within the blocks (FMAP blocks) within the DASD range pointed to by the FMAP, reads them into the data buffer, decrypts the data using the local encryption file key, puts it into the output buffer, and serializes it into a blob for the replicator to upload to the object store. In other words, the difference generator needs to collect all the FMAPs of the identified differences in order to obtain all the data related to the differences between the two snapshots.

[0115] Snapshot differences stored in the object store may span multiple blobs (or objects if stored in the object store). The blob format of these blobs includes a key, a value, and, if present, data associated with the key. For example, in FIG. 8, the snapshot difference 800 includes at least three blobs 802, 804, and 806. The first blob 802 includes a prefix 810 indicating the key and value types, the key length, and the value length, followed by a key 812 (key 1) and a value 814 (value 1). The second blob 804 includes a prefix 820 (key and value types, key length, and value length), a key 822 (key 2), a value 824 (value 2), a data length 826, and data 828 (data 2). In the prefix 820 of this second blob 804, since this blob includes additional data 828 associated with the key 822, the key and value types are fmap. The third blob 830 includes a format similar to that of the first blob 810, for example, a prefix 830, a key 832 (key 3), and a value 834 (value 3).

[0116] The data is decrypted, collected, and then written to a blob. All processes are executed in parallel. Multiple blobs can be processed and updated simultaneously. When all processes are complete, the data is written in blob format (shown in FIG. 8) and can then be uploaded to an object store in the format (shown in FIG. 9) or path name.

[0117] FIG. 9 shows an exemplary replication bucket format according to an embodiment. A "bucket" can refer to a container that stores objects in a compartment within an object storage namespace. In one embodiment, buckets are used by source replicators to store data protected using server-side encryption (SSE) techniques and by target replicators to download changes and apply them to snapshots. Replication data for all file systems in a target region can share a bucket within that region.

[0118] The data layout of a bucket in the object store has a directory structure that includes, but is not limited to, a file system ID (e.g., Oracle Cloud ID), a difference including a start snapshot number and an end snapshot number, a manifest that describes the content of the information within the object's layout, and blobs. For example, the bucket of FIG. 9 includes two objects 910 and 930. The first object 910 includes two differences 912 and 920. This object starts with a path name 911 (e.g., ocid1.filesystem.oc1.iad...) that uses the source file system ID as a prefix, followed by a first difference 912 generated from snapshot 1 and snapshot 2, and a second snapshot 920 generated from snapshot 2 and snapshot 3. Each difference includes one or more blobs that represent the content of that difference. The first difference 912 stores two blobs 914 and 916 in the order of generation. The second difference 920 includes only one blob 922. Each difference also includes a manifest that describes the content of the information within the layout of this difference, for example, manifest 918 of the first difference 912 and manifest 924 of the second difference 920. The manifest within the bucket is content that describes the difference, such as the file system number and snapshot range. The manifest can be a master manifest, a range manifest, or a checkpoint manifest depending on the stage of the replication process.

[0119] The second object 930 also includes two differences 932 and 940 in a similar format starting with path name 931. The two objects 910 and 930 within the bucket come from different source regions, namely IAD for object 910 and PHX for object 930. After the blobs are applied, the corresponding information within the layout can be removed to reduce space utilization.

[0120] The final manifest object (i.e., the checkpoint manifest, CM file) is uploaded from the source region to the object store, and the source file system indicates to the target region that it has completed uploading the snapshot delta of a specific object. The source CP communicates this event to the target CP, and the target CP can notify the target DP via the SDB to trigger the download process of that object by the target replicator.

[0121] The control plane within the source region or the target region coordinates all of the replication workflow and drives the replication of data. The control plane performs the following functions: (1) creates the underlying system snapshot for creating deltas, (2) determines when such snapshots need to be created, (3) initiates replication based on the snapshots, (4) monitors the replication, (5) triggers the download of deltas by the secondary (or target side), and (6) indicates to the primary (or source) side that the snapshot has reached the secondary.

[0122] The file system has several operations for processing its resources, including but not limited to creating, reading, updating, and deleting (CRUD). These operations are typically synchronized within the same region, starting a workflow when the file system receives an HTTPS request from the API server, making changes in the backend to create a resource, and returning a response to the customer. The resources are split into a source region and a target region. The state is maintained for the same resource between the source region and the target region. Thus, there is asynchronous communication between the source region and the target region. A customer can contact the source region to create or update a resource, and these creations or updates can be automatically reflected in secondary or auxiliary resources within the target region. The state machine in the control plane also targets recovery in many aspects, including but not limited to failures in the fleet, key management, disk, and object.

[0123] Regarding the application programming interface (API) within the control plane, there are various APIs for users to configure replication. The control API for any new resource only functions within the region where the object was created. In the target file system, a field named "IsTargetable" can be set in the API to ensure that the target file system receiving replication cannot be accidentally used by consumers. In other words, setting this field to false means that consumers can view the target file system, but no one can export the target file system or access any data within the live system. Since export is not a read-only permission but a read / write permission for export, any export may change the data. Therefore, during the replication process, exports are not permitted to prevent any changes to the target file system. Consumers can only access data within the old snapshots that have already been replicated. All newly created or replicated file systems can have this field set to true. The reason is that the target can only obtain data from a single source. Otherwise, conflicts may occur when data is written or deleted. The system needs to know whether the target file system in use is already part of some replication. Setting the "IsTargetable" field to "true" means that replication is not in progress, and setting it to "false" means that the target file system cannot be used.

[0124] Regarding inter-region communication between components of the control plane, the primary resource on the source file system is called an application, and the auxiliary (or secondary) source on the target file system is called an application target. Source objects and target objects have a single replication relationship when created. Both objects can be updated only from the source side, including changes to compartments, editing of details, or deletion. If the user wants to delete the target side, the replication itself can be deleted. In the case of a planned failover, it is possible to delete the source side, and both the source side and the target replication are deleted. In the case of an unplanned failover, the source side is unavailable, so only the target replication can be deleted. In other words, there are two resources for a single replication, and those resources should be kept in a synchronized state. There are various workflows for updating metadata on both the source side and the target side. Additionally, inter-region APIs for retries, fault handling, and failover are also part of the inter-region communication process.

[0125] When creating the necessary security and other related artifacts, the source uploads the security and artifacts to the object store, starts a job at the target (i.e., notifies the target that the job is available), and the target can then start downloading the artifacts (e.g., snapshots or deltas). The target then continues to look for an end-of-file marker (also referred to herein as a checkpoint manifest (CM) file) within the object store. The CM file is used as a mechanism for the source side and the target side to communicate the completion of the upload of an object during the replication process. At every checkpoint, the source side uploads this CM file that contains information such as the number of blobs uploaded up to this checkpoint, enabling the target side to download this number of blobs and apply them to the current snapshot. This CM file is a mechanism for the source side to communicate to the target side that the upload of an object to the object store is complete, and for the target to start working on that object. In other words, the target continues to download until there are no more objects in the object storage. Thus, this approach enables concurrent processing on both the source side and the target side.

[0126] Figure 10 is a flowchart showing a state machine for simultaneous source upload and target download according to an embodiment. As previously explained, both the source file system and the target file system can perform replication simultaneously and thus can each have its own state machine. In one embodiment, each file system can have its own state machine while sharing some common job-level states. In Figure 10, the source file system has states 1030 - 1034 for session key generation and transfer in addition to states 1002 - 1018 for performing data upload. The target file system has states 1050 - 1068 related to data download. The session key can be generated at any time within the source file system while the differences are being uploaded to the object storage. Thus, the session key transfer has its own state sequence 1030 - 1034. In Figure 10, the target file system cannot start the replication download process (i.e., Ready_to_Reconcile state 1050) until it receives an indication that at least an object has been uploaded to the object storage by the source file system (i.e., Mainfest_Copied state 1014) and that it is ready to download the session key (i.e., Copied_DTK state 1034).

[0127] In the source file system, multiple functional blocks such as a snapshot generator, a control API, and a differential monitor are part of the CP. The replicator fleet is part of the DP. The snapshot generator is responsible for periodically generating snapshots. The differential monitor periodically monitors the progress of the replicator in replication-related tasks, including the creation of snapshots and the replication schedule. When the differential monitor detects that the replicator has completed a replication job, it transitions the state to a copied state on the source side (e.g., the Manifest_Copied state 1014) or a replicated state on the target side (e.g., the Replicated state 1058). In certain embodiments, multiple file systems can simultaneously execute replication from a source region to a target region.

[0128] Referring to FIG. 10, in certain embodiments, in the source file system, in the simultaneous mode state machine, after creating a snapshot signal to the differential monitor indicating that a snapshot has been generated, the snapshot generator. The differential monitor that executes the CP replication state (CpRpSt) workflow is responsible for starting the upload of snapshot metadata to the object store on the target side. The snapshot metadata can include the type of snapshot, snapshot identification information, the time of the snapshot, and the like. The CpRpSt workflow sets the Ready_to_Copy_Metadata state 1002 for the replicator fleet to start copying the metadata. When the replicator obtains a replication job, it creates a copy of the snapshot metadata (i.e., the Snapshot_Metadata_Copying state 1004) and uploads those copies to the object store. When all replicators have completed the upload of the snapshot metadata, the state is set to the Snapshot_Metadata_Copied state 1006. Thereafter, the CpRpSt workflow continues to poll the source SDB for the session key.

[0129] Here, the CpRtSt workflow returns control to the differential monitor to monitor the differential upload process that transitions to the Ready_to_Copy state 1008 indicating that the differential calculation is scheduled. Next, the source CP API sends a request to the replicator to start the next stage of replication by uploading the differential and creating a copy of the manifest. The replicator that selects the replication job can start creating a copy of the manifest (i.e., the Mainfest_Copying state 1010). When the source file system completes the copy of the manifest, it transitions to the Manifest_Copied state 1014 and simultaneously notifies the target file system that it can start the internal state (the Ready_to_Reconcile state 1050).

[0130] As described above, the session key can be generated by the source file system during the upload of data. The replicator of the source file system communicates with the target KMS vault to obtain a master key that can be provided by the customer and creates a session key (referred to herein as the differential encryption key or DEK). Next, the replicator encrypts the session key using the local file system key (FSK: file system key) (which becomes the encrypted DEK, also referred to herein as the differential transfer key (DTK)). Thereafter, the DTK is stored in the SDB within the source region and reused by the replicator thread during the replication cycle. The state machine transitions to the Ready_to_Copy_DTK state 1030.

[0131] The source file system transfers the resource identification information of the DTK and KMS to the target API. Then, the target API puts those resource identification information into the SDB within the target region. During this transfer process, the state machine is set to the Copying_DTK state 1032. When the CpRpSt workflow in the source file system finishes polling the source SDB for the session key, the target file system downloads the session key (DTK) and sends a notification to the target side indicating that it is ready to use that session key to decrypt the downloaded differences for application. Then, the state machine transitions to the Copied_DTK state 1034. The replicator on the target side retrieves the DTK from the SDB and requests the KMS API to decrypt the DTK into the plaintext DEK (i.e., the decrypted session key).

[0132] When the source file system completes the upload of data for a specific replication cycle including session key transfer, the difference monitor notifies the target control API of the status such as the validation information and transitions to the X-region_Copied_Done state 1016. This can occur before the target file system completes the download and application of the data. The source file system further cleans up the memory and removes all keys. Then, the source file system transitions to the Awaiting_Target_Response state 1018 and waits for a response from the target file system to start a new replication cycle.

[0133] As described above, the target file system cannot start the replication download process until it receives instructions that at least the object has been uploaded to the object storage by the source file system (i.e., Mainfest_Copied state 1014) and that it is ready to download the session key (i.e., Copied_DTK state 1034). When these two conditions are met, the state machine transitions to the Ready_To_Reconcile state 1050. Next, in the Reconciling state 1052, the target file system starts an adjustment process with the source side, such as synchronizing snapshots of the source file system and the target file system, obtaining snapshots, and generating statistics, and also performs some internal CP management operations. This internal state includes communication within the target file system between the delta monitor and the CP API.

[0134] After the adjustment process is completed, the replication job is passed to the target replicator (i.e., Ready_to_Replicate state 1054). The target replicator monitors the checkpoint manifest (CM) file uploaded by the source file system. The CM file is marked by the target. Then, the target replicator thread starts to download the manifest and apply the downloaded and decrypted deltas (i.e., Replicating state 1056). The target replicator thread also reads the FMAP data blocks from the blobs downloaded from the object store, communicates with the local FSK service to obtain the file system key FSK, and the FSK is used to re-encrypt each FMAP data block and store it in local storage.

[0135] When the source file system finishes uploading data, it updates the final CM file by setting the end-of-file (eof) field to true and uploads it to the object store. As soon as the target file system detects this final CM file, it ends the download of the blobs and applies them, and the state machine transitions to the Replicated state 1058.

[0136] After the target file system applies all the differences (or blobs), it continues to download the snapshot metadata from the object store and inputs the information of the source file system's snapshot into the target file system's snapshot (i.e., the Snapshot_metadata_Populating state 1060). When the target file system's snapshot is input, the state machine transitions to the Snapshot_Metadata_Populated state 1062.

[0137] In the Snapshot_Deleting state 1064, the target file system deletes all the blobs in the object store for the blobs that have been downloaded and applied to the latest snapshot. Then, the target control API notifies the target difference monitor when the blobs in the object store are deleted and proceeds to the Snapshot_Deleted state 1066. The target file system further cleans up the memory and removes all the keys. The FSS service also releases the KMS key.

[0138] Once the target DP finishes applying the differences and cleaning up, it uses the target control API to verify the validity regarding the status of the source file system and whether it has received an X-region_Copied_Done notification from the source file system. If the notification has been received, the target difference monitor transitions to the X-region DONE state 1068 and sends an X-region DONE notification to the source file system. In some embodiments, the target file system checks whether there is an end-of-file for all key ranges and all upload processing threads because all objects uploaded to the object store have special markers such as end-of-file markers in the CM file, so as to detect whether the source file system has completed the upload.

[0139] Referring back to the state machine of the source file system, while the source file system is in the Awaiting_Target_Response state 1018, it checks whether the status of the target CP has changed to completed, indicating that all differences downloaded by the target have been applied and the file data has been stored locally. If the status of the target CP changes to completed, this marks the end of the replication cycle.

[0140] The source side and the target side operate asynchronously. When the source file system completes the replication upload, it notifies the target control API of the X-region_Copied_Done notification. Then, when the target file system completes the replication process, the difference monitor target communicates in the reverse direction with the source control API using the X-region DONE notification. The source file system returns to the Ready_to_Copy_Metadata state 1002 and starts another replication cycle.

[0141] FIG. 11 is an exemplary flow diagram showing the exchange of information between the data plane and the control plane within a source region according to an embodiment. The data plane components and the control plane components communicate with each other using a shared database (SDB), such as 1106. The SDB is a key-value store that both the control plane components and the data plane components can read from and write to. The data plane components include a replicator and a difference generator. The exchange of information between the components within the source region A 1101 and the target region B 1102 is also shown.

[0142] In FIG. 11, at step S1, the source control plane (CPa) 1103 requests the object store within the target region B (OSb) 1112 to create a bucket. At step S2, the source replicator (REPLICATORa) 1108 periodically updates the heartbeat status to the source SDB (SDBa) 1106. The heartbeat is a concept used to track the progress of replication executed by the replicator. The heartbeat can use a mechanism called lease, where the heartbeat is continuously updated each time the replicator works on a job, enabling the control plane to recognize the entire release information. For example, the byte count is continuously moving on the job. If the replicator cannot function properly, the heartbeat may become old, and then another replicator can detect and take over to continue working on the remaining jobs. Therefore, if the system crashes midway, the system can accurately start from the last point based on the checkpoint mechanism. The checkpoint helps the system know where the last point of progress was and enables it to continue from that point without re-executing the entire job.

[0143] In step S3, CPa1103 further requests the file system service workflow (FSW_CPa) 1104 to create snapshots periodically. In step S4, FSW_CPa1104 notifies CPa1103 about the new snapshot. In step S5, next, CPa1103 stores the snapshot information in SDBa1106. In step S6, REPLICATORa1108 polls SDB1106 for any changes to the existing snapshots. If a change is detected, in step S7, it retrieves the job specifications. In step S8, when REPLICATORa1108 detects a change to the snapshot, this initiates the replication process. In step S8, REPLICATORa1108 provides information about two snapshots (SNa and SNb) including the changes between the snapshots to the difference generator (DGa) 1110. In step S9, REPLICATORa1108 enters work item information such as the number of key ranges into SDBa1106. In step 10, REPLICATORa1108 checks the replication job queue in SDBa1106 to obtain work items. In step S11, it assigns those work items to the difference generator (DGa) 1110 to scan the B-tree keys of the snapshot (i.e., traverse the B-tree) and calculate the differences and the corresponding key-value pairs. In step 12, REPLICATORa1108 decrypts the file data associated with the identified B-tree keys and packs them together with the key-value pairs into blobs. In step 13, REPLICATORa1108 encrypts the blobs using the session key and uploads them as objects to OSb1112. In step S14, REPLICATORa executes a checkpoint and stores the checkpoint record in SDBa1106. This replication process (S8 - S14) repeats (as a loop) until all differences are identified and the data is uploaded to OSb1112.In step S15, REPLICATORa1108 next notifies SDBa1106 of the details of the replication job, and then this detail is passed to CPa1103 in step S16 and further relayed to CPb1114 as the final CM file in step S17. In step S18, CPb1114 stores the job details in SDBb1116.

[0144] The exchange of information between the data plane and the control plane within target region B is similar. At the end of applying the difference to the target file system, the control plane within target region B notifies the control plane within source region A that the snapshot has been successfully applied. Thereby, the control plane within source region A can start over using the new snapshot.

[0145] Authentication is performed on all components. There is an authentication mechanism that uses the replication ID and file system number, from the replicator to the file system key (FSK). The key can be given to the replicator only if the replicator provides appropriate content. Thus, the authentication mechanism can prevent fraudsters from obtaining the decryption key. Other security mechanisms include blocking network ports. A component called the file system key server (FSKS) is a gatekeeper for properly checking requesters by checking metadata such as the jobs the requester is running and other information. For example, assume the replicator is attempting to request the key to the file system. In that case, FSKS can check whether the replicator is associated with a specific job (e.g., whether the replication is actually associated with that file system) to confirm the validity of the requester.

[0146] Availability addresses situations where a machine can automatically restart after going down, or where services remain available while software deployment is in progress. For example, since all replicators are stateless, losing a replicator is transparent to customers because another replicator can pick up and continue the job's work. The job's state is maintained not locally, but in a shared database and other reliable locations. The shared database is a service such as a database used by the control plane to maintain information about the file system and is based on a B-tree.

[0147] The system has thousands of storage nodes that enable any storage node to perform differential replication, so storage availability in the FSS of the present disclosure is high. By utilizing many machines that can take over from each other in case of some failure, the availability of the control plane is increased. For example, the progress of replication is not simply hindered by the failure of a single control plane. Thus, there is no single point of failure. The availability of network access is made such that the source node does not become overloaded by using congestion management that includes various types of regulation.

[0148] Replication is durable by writing the replication state to the shared database and by utilizing checkpointing where the replicator is stateless. The replication process should be idempotent. Idempotency can refer to deterministic reapplication where, in case an operation fails, retrying the same operation, for example, using the same key, upload process, or scan process, should result in the same outcome.

[0149] Operations within multiple regions should be idempotent. In the control plane, the actions that are executed need to be stored. For example, in the case where an HTTP request itself is repeated, an idempotency cache can be useful for remembering that a particular operation has been executed and that it is the same operation. For example, in the data plane, when a block is allocated, the block and the file map key of the file system are written together. Thus, if the block is allocated again, the block can be identified. If the block is sealed, the write operation fails. The idempotency mechanism can know that the block has been sealed in the past and the write operation does not need to be retried. In yet another example, the idempotency mechanism remembers the chain of steps that need to be executed for the processing of a particular key and value. In other words, the idempotency mechanism allows all operations to be checked to ensure that they are in the correct state. Thus, the system can simply proceed to the next step without repeating.

[0150] Atomic replay enables the application of deltas to start as soon as the first delta object reaches the object store when a snapshot is rolled back, e.g., when going back from snapshot 10 to snapshot 5. To make the replay atomic, the entire delta needs to be maintained in the object store before it can be applied.

[0151] Regarding the expansion of replicators, the FSS of the present disclosure enables adding the number of replication machines (e.g., replicator virtual machines ("VMs")) required to support many file systems. The number of replicators can be dynamically increased or decreased by considering the bandwidth requirements and availability of resources. Regarding the expansion of storage, thousands of storages can be used to parallelize the process and improve the working speed. Regarding the inter-region bandwidth, the bandwidth allocation is automatically adjusted, such as adjusting all inter-region bandwidths by grasping the increase in latency and reducing the required speed, to ensure that each workload is not overused or does not exceed a predefined throughput limit. All replicator processors (or threads) have this function.

[0152] In the expansion of checkpoint storage, the uploader and downloader checkpoint the progress to persistent storage, and the shared storage is used as a work queue for splitting key ranges. If the checkpoint workload overly burdens the shared database, for the purpose of expansion, the checkpoint storage function can be added to the differential generator. The current workload of the shared database may consume less than 10 IOPs.

[0153] FIG. 12 is a schematic diagram showing a failback mode according to an embodiment. The failback mode enables restoring the primary / source side to become the primary again before failover. As shown in FIG. 12, the primary AD 1202 includes the source file system 1206, and the secondary AD 1204 includes the target file system 1208. The secondary AD 1204 may exist in the same region or a different region from the region of the primary AD 1202.

[0154] In FIG. 12, snapshot 1 1220 and snapshot 2 1222 in the source file system 1206 exist before a failover due to a power outage event. Similarly, snapshot 1 1240 and snapshot 2 1242 of the target file system 1208 exist before the failover. When a power outage occurs in snapshot 3 1224 in the primary AD 1202, the FSS performs an unplanned failover 1250, and snapshot 3 1224 in the source file system 1206 is replicated to the target file system 1208 and becomes the new snapshot 3 1224. After the target file system 1208 becomes operational, the customer can make changes to create snapshot 4 1246 for the target file system 1208.

[0155] If the customer decides to reuse the source file system, the FSS service may execute a failback. When executing the failback, the user has two options: (1) the last point in time in the source file system before the trigger event 1251, or (2) the latest changes in the target file system 1252.

[0156] In the case of the first option, the user can resume from the last point in the source file system 1206 prior to the trigger event (i.e., snapshot 3 1224). In other words, since snapshot 3 1224 has previously successfully failed over to the target file system 1208, it becomes the snapshot for use after failback. To perform the failback 1251, the state of the source file system 1206 is changed to inaccessible. Next, the FSS service identifies the last point in the source file system 1206 before the failover was successful, snapshot 3 1224. The FSS may execute a clone of snapshot 3 1224 in the primary AD 1202 (i.e., a replication within the same region). Thus, the primary AD 1202 returns to its initial settings prior to the power outage, and the user can reuse the source file system 1206 again. Since snapshot 3 1224 already exists in the file system being used, no data transfer from the secondary AD 1204 to the primary AD 1202 is required.

[0157] In the case of the second option, the user wants to reuse the source file system with the latest changes in the target file system 1208. In other words, since snapshot 4 1246 in the target file system 1208 was the latest change in the target file system 1208, it becomes the snapshot for use after failback. The failback process 1252 for this option includes reverse replication (i.e., reversing the roles of the source file system and the target file system for the replication process), and the FSS performs the following steps.

[0158] Step 1. The state of the source file system 1206 is changed to inaccessible. Step 2. Next, the FSS service identifies the latest snapshot in the successfully replicated target file system 1208, e.g., snapshot 3 1244.

[0159] Step 3. The FSS service also detects the corresponding snapshot 3 1224 in the source file system 1206 and performs a clone (i.e., a copy within the same region).

[0160] Step 4. The FSS service starts reverse replication 1252 in the same process as described in relation to FIG. 4, but in the reverse direction. In other words, both the source file system 1206 and the target file system 1208 need to be synchronized, after which the target file system 1208 can upload the differences to the object store in the primary AD 1202. The source file system 1206 can download the differences from the object store, complete the application to snapshot 3 1224, and create a new snapshot 4 1226.

[0161] Here, the primary AD 1202 returns to the initial settings before the power outage, and the user can reuse the source file system 1206 again without transferring the data that already exists in both the source file system 1206 and the target file system 1208, for example, snapshots 1 to 3 (1220 to 1224) in the source file system 1206. This saves time and prevents unnecessary bandwidth.

[0162] Snapshot and Data Model Snapshot In one embodiment, there are two types of snapshots: a system snapshot and a user (or customer) snapshot. The system snapshot is controlled by the FSS, while the user snapshot is controlled by the customer. The system snapshot is periodically created by a snapshot generator in the source FS and is cleaned up in both the source file system and the target file system at the end of the replication cycle. The customer can also create a user snapshot in the source region based on a scheduled snapshot policy. The customer can distinguish between the system snapshot and the user snapshot based on details such as different names, for example.

[0163] Since the system snapshot can be used to specify the start of a replication cycle, there is one system snapshot per replication cycle. On the other hand, the user snapshot can be generated and deleted at any time by the user and cannot be used to specify the start of a replication cycle.

[0164] The system snapshot cannot be changed or deleted by the customer. However, the FSS can delete the system snapshot after the target FS has successfully completed the application of the delta. In one embodiment, at least one system snapshot is maintained in both the source FS and the target FS. For example, when both the source FS and the target FS have completed replication cycle N, both file systems can delete the system snapshot of the just-completed replication cycle N and instead delete the system snapshot of replication cycle N - 1.

[0165] The replication process identifies changes (i.e., differences) between two system snapshots. The replication process starts from the base snapshots (i.e., the snapshots established as the starting points) of both the source FS and the target FS. For example, if the base snapshot exists only in the source FS, the FSS needs to create a copy of the base snapshot in the target FS by transferring the entire base snapshot from the source FS to the target FS. If the source FS and the target FS already contain the same base snapshot respectively, the replication process can start calculating the differences within the source FS (i.e., the differences between the new snapshot and the base snapshot) and transfer these differences to the target FS immediately.

[0166] History ID The technology disclosed in the File System Service (FSS) realizes efficient replication, including using the History ID to save cloud resources and reduce network traffic and IO traffic. The History ID is special identification information that uniquely identifies snapshots across regions, regardless of whether the snapshot is a system snapshot or a user snapshot. Assume that two file systems have the same History ID for a specific snapshot. In that case, this means that the snapshots contained in each of these two file systems are very similar up to that point, both have a common ancestor or the same known point in time, and can be used as the base snapshot for cross-region (or x-region) replication. The History ID applies to both system snapshots and user snapshots.

[0167] A snapshot is a point-in-time picture of a file system and is immutable (i.e., write-protected). There are two types of replicas for snapshots: clones and replicas. Clones are sometimes referred to as writable snapshots and are typically created in the same region. When a clone is created, each clone can be written to independently using I / O. These clones all have the same lineage. If one clone is created between two file systems, both file systems share the same copy of the snapshot for reading. A different copy is created only when one of the file systems needs to write to the clone. A replica is a replicated snapshot created in a different region (i.e., between regions or different data centers) by a replication process.

[0168] Replication and cloning can differ in that replication realizes by first copying all data from the source region to the target region and then copying the differences between the snapshots. On the other hand, cloning copies only the data necessary to create the clone client. Intra-region cloning is much faster than inter-region replication because cloning does not involve many stages of the pipeline that replication requires, such as additional encryption / decryption, object storage transfer. Since clones do not receive further changes after being created, they only capture point-in-time snapshots.

[0169] In one embodiment, all snapshots may have three pieces of information associated with the snapshot, namely, a snapshot number (snapNum), a provenance ID (ProvID or PID), and a resource ID (e.g., an OCID). Since the snapshot consumes resources, the resource ID is a globally unique ID for identifying the resource. The snapshot number is for internal housekeeping and tracking within the file system. The provenance ID is for external use and is unique among all snapshots either within or across regions. The provenance ID is set at the time the snapshot is created and does not change when the snapshot is cloned or replicated. These three pieces of information together uniquely identify the history of the snapshot (e.g., the parent-child relationship between all snapshots) and can distinguish the snapshot from other resources within the cloud infrastructure. Additionally, a file system number (FS number) helps track clones within a region and replicas across regions. Across different regions, the provenance ID helps track the history of the snapshot by holding the provenance ID of the original parent snapshot.

[0170] In some embodiments, before replication begins, the source FS and the target FS can compare the history IDs of their respective snapshots to detect matching pairs of snapshots. If a particular pair of snapshots has the same history ID, the source FS and the target FS can start replication from the identified pair without having to transfer the entire copy of the base snapshot from the source FS to the target FS at the start of replication. As a result, this saves resources and avoids traffic associated with data transfer. For example, assume that a previous replication between the source FS and the target FS replicated snapshots S1 - S100 and then stopped. After a while, these two file systems plan to perform another replication and need to find a starting point for this new replication. Assume that the source FS is already at snapshot S200. In that case, the source FS can trace back from S200 to S1 and compare the history ID of that snapshot with the history ID of the last snapshot of the target FS (this comparison process is also referred to as a trace in this specification) and may detect that S100 in both the source FS and the target FS is a matching pair. At that point, S100 can be used as the starting point (i.e., the base snapshot) in both the source file system and the target file system for the new replication process. The source FS can calculate the difference between snapshot S100 (i.e., the base snapshot) and snapshot 200 (i.e., the new snapshot), and then transfer those differences to the target FS, which can apply those differences to S100 to create S200 in the target FS. The source FS does not need to transfer snapshot 100 to the target FS again as a copy of the base snapshot for the replication process to start. This saves a lot of data and IO transfers.

[0171] In some embodiments, the provenance ID can be useful for all file systems within the same region by cloning a snapshot from another file system within the same target region to the target FS when the snapshot replicated from the source region already exists in the target region but does not exist in the target FS. This can be shown in FIG. 13.

[0172] FIG. 13 is a diagram showing an exemplary use of the provenance ID according to an embodiment. In FIG. 13, the FSS creates clones of three snapshots, snapNum 1 / ProvID S1 / OCID S1, snapNum 2 / ProvID S2 / OCID S2, and snapNum 3 / ProvID S3 / OCID S3, of the file system FS1 within the same region 1, such that they become snapNum 1 / ProvID S1 / OCID K1, snapNum 2 / ProvID S2 / OCID K2, and snapNum 3 / ProvID S3 / OCID K3 of the file system FS2 (step 1310). Further, a new snapshot snapNum 5 / ProvID K5 / OCID K5 is also created on FS2. The clones within FS2 have different resource IDs (S* becomes K*) because they use different resources within the same region. Note that snapshot 4 of FS1 is not cloned.

[0173] Next, the FSS creates replicas of snapshots 1, 2, 3, and 5 of file system FS2 such that they become snapNum 1 / ProvID S1 / OCID M1, snapNum 2 / ProvID S2 / OCID M2, snapNum 3 / ProvID S3 / OCID M3, and snapNum 5 / ProvID K5 / OCID M5 of the file system FS3 in region 2 (i.e., step 1320). After snapNum 5 has been replicated, the replication is then deleted (i.e., step 1322), that is, regions 1 and 2 no longer communicate with each other. Further, snapshots snapNum 6 / ProvID G6 / OCID M6 and snapNum 7 / ProvID G7 / OCID M7 are then created in FS3 within region 2.

[0174] After a while, FSS attempts to perform replication of snapshots 1, 2, 3, and 7 of FS3 in region 2 to FS4 in region 1 (that is, attempts to create replicas at step 1330). Since FS4 (i.e., the target FS) does not exist in region 1 but FS1 (i.e., the non-target FS) already exists in the same region, before replication, FS3 in region 2 and FS1 in region 1 compare the history IDs of their snapshots (that is, step 1340). This comparison may detect that snapshots 1, 2, and 3 of FS3 have the same history IDs (S1, S2, S3) as snapshots 1, 2, and 3 of FS1 in region 1. Therefore, to save resources and network bandwidth, FS1 located in the same region 1 as FS4 can first create clones of snapshots 1, 2, and 3 of FS1 (snapNum 1 / ProvID S1 / OCID S1, snapNum 2 / ProvID S2 / OCID S2, and snapNum 3 / ProvID S3 / OCID S3) to be the same as (snapNum 1 / ProvID S1 / OCID P1, snapNum 2 / ProvID S2 / OCID P2, and snapNum 3 / ProvID S3 / OCID P3) of FS4 in the same region 1 as the base copy of the snapshot (that is, step 1342). Then, FS3 only needs to replicate snapshot 7 (snapNum 7 / ProvID G7 / OCID M7) of FS3 in region 2 to become snapshot 7 (snapNum 7 / ProvID G7 / OCID P4) of FS4 in region 1 by transferring the difference between snapshot 3 (ProvID S3) and snapshot 7 (ProvID G7) (that is, step 1344).In other words, the periodic inter-region replication of four snapshots 1, 2, 3, and 7 from FS3 in region 2 to FS4 in region 1 can be simplified to be, in addition to the three in-region clones of snapshots 1, 2, and 3 between FS1 and FS4 in the same region, the inter-region replication of snapshot 7 between FS3 in region 2 and FS4 in region 1. As a result, the use of the history ID saves resources, data transfer traffic (i.e., network traffic or IO traffic), and time.

[0175] FIG. 14 is a flowchart showing a process for identifying a base snapshot for inter-region replication using a provenance ID according to an embodiment. As shown in FIG. 14, in step 1401, a source FS in a source region may periodically generate a system snapshot and may also generate a user snapshot upon user request. In step 1402, each snapshot may be assigned a unique provenance ID and other identification information (e.g., snapshot ID and resource ID). In step 1404, the source FS may receive a request to perform inter-region replication between the source FS and a target FS due to either a power outage or a planned failover. In step 1408, as described above, in some embodiments, both the source FS in the source region and the file system in the target region compare the provenance IDs of their respective snapshots to identify a base snapshot (i.e., matching snapshots having the same or matching provenance IDs) for the purpose of inter-region replication or in response to a request to perform inter-region replication. For example, in FIG. 13, FS3 (i.e., the source FS) in source region 2 compares the provenance ID of its snapshot with the provenance IDs of the snapshots of both the target FS (i.e., FS4) and the non-target FS (i.e., FS1) (i.e., step 1340). In other embodiments, the comparison of the provenance IDs may first be performed between the source FS and the target FS in the target region. If no match is detected, the source FS may perform a comparison of the provenance ID with the non-target FS in the target region.

[0176] In step 1410, if no matching history ID is detected between the source FS and the file system within the target region, then in step 1412, the inter-region replication process may use the latest snapshot of the source FS as the selected base snapshot. In other words, the source FS may need to transfer a copy of the entire base snapshot (i.e., the selected base snapshot) to the target FS as shown in step 1420, and then perform any necessary differential transfer to the target FS. In step 1410, if a matching history ID is detected between the source FS and the file system within the target region, then in step 1414, the process further determines to which snapshot of either the target FS or the non-target FS within the target region the matching history ID belongs.

[0177] In step 1414, if the matching history ID (i.e., the matching snapshots with the same history ID) does not belong to the snapshots of the target FS (i.e., belongs to the snapshots of a non-target FS), in step 1416, the non-target FS may perform in-region cloning of the snapshots with the matching history ID to the target FS to create a base snapshot. Thereafter, at 1420, inter-region replication can use the cloned base snapshot of the target FS as the selected base snapshot. In other words, the source FS can generate the difference between the latest snapshot and the selected base snapshot with the matching history ID, and transfer only this difference to the target FS via the object store. This eliminates the need to transfer a complete copy of the base snapshot. For example, in FIG. 13, the non-target FS1 may clone snapshots S1, S2, and S3 to the target FS4 within the same region 1 (i.e., step 1342). Since the three snapshots (S1, S2, and S3) have matching history IDs, all three snapshots can be used as base snapshots. In an embodiment, the source FS can use the latest snapshot among the three snapshots (i.e., S3) as the selected base snapshot to generate the difference between snapshot S3 and G7 for inter-region replication (i.e., step 1344).

[0178] In step 1414, if the matching history ID belongs to the snapshots of the target FS, in step 1418, both the source FS and the target FS use the snapshot with the matching history ID as the selected base snapshot. In step 1420, the source FS can generate the difference between its latest snapshot and the selected base snapshot, and transfer the difference to the target FS for difference application during inter-region replication.

[0179] In addition to selecting a base snapshot for inter-region replication, in some embodiments, the history ID may also be useful for resumability in the event that replication fails or is accidentally deleted. For example, as described above, multiple inter-region replications may occur between regions. If one inter-region replication fails during the replication process, the corresponding source file system and target file system can use the history ID to search for and detect a snapshot of the target file system or non-target file system within the target region, and use it as the base snapshot to resume the inter-region replication. Since FSS performs replication using incremental deltas, the better the FSS can more easily and quickly identify a unique starting point common to both the source file system and the target file system, the better the FSS can resume the replication process and recover from failures. The history ID can avoid the need for a complete base copy every time a failure occurs.

[0180] Consistency of Snapshot Data This disclosure also discloses techniques for maintaining the consistency of snapshots between a source FS and a target FS, including the creation and deletion of snapshots. A first aspect for maintaining the consistency of snapshots between the source FS and the target FS is the order in which snapshot keys and file data are processed. In certain embodiments, the snapshot and data model of the FSS processes the snapshot key first and then the file data, thereby processing the snapshot key and file data in a certain order. The snapshot key (sometimes called the snap key) is the B-tree key of the snapshot. Each time a new snapshot is created in the source region, the source data plane performs a delta generation that includes identifying the new snapshot key of the new snapshot, transferring this new snapshot key to the target region, and the target FS applies and inserts this new snapshot key into its B-tree. Otherwise, the new snapshot key may be collected by the garbage collector within the source region. Since the snapshot key represents the snapshot and helps distinguish the differences between snapshots, the snapshot key needs to be processed first (i.e., identified and transferred to the target region) before reading the data blocks within the source region. Further, the file data is associated with the B-tree key. Therefore, accessing the file data before the B-tree key is created in the target FS may lead to file system inconsistencies. Finally, in some embodiments, the snapshot key is involved in billing measurement and needs to be established first.

[0181] The snapshot key is the marker key for the snapshot. When an epoch is created, the marker key is also created. Based on the epoch that tracks the file system time, a snapshot number is created. For example, when the epoch advances from N to N+1, the source file system number becomes N+1, and the source FS creates a snapshot number N (either a system snapshot or a user snapshot).

[0182] A second aspect of maintaining snapshot consistency between the source FS and the target FS is to handle snapshot deletion. In one embodiment, the FSS processes snapshot creation using the data plane (DP), and processes snapshot deletion using the CP. As previously explained, the snapshot generator in the source DP periodically generates system snapshots in addition to user snapshots generated by the customer. The difference is calculated between two specific system snapshots and replicated from the source FS to the target FS. However, snapshots may be deleted during the replication process. System snapshots are maintained in both the source FS and the target FS until the target FS completes applying the difference, but user snapshots may be updated or deleted at any time within the source region during the replication process, provided that they cannot be updated or deleted within the target region. Both the source CP and the target CP need to track and execute snapshot deletion according to the replication policy. Failure to do so may lead to inconsistencies between the source FS and the target FS due to improper handling of snapshot deletion.

[0183] Snapshots created in the source region need not be visible to the user until these snapshots are applied by the target file system. For example, if the source FS contains three user snapshots S1, S2, and S3, the source CP and the target CP do not notify the user that snapshots S1, S2, and S3 are available in the target region until these snapshots are recreated in the target FS. The purpose is to prevent the user from cloning these snapshots when any of them in the target region is not ready. In some embodiments, multiple replications to one or more target file systems in different regions from the source FS may be performed for multiple existing user snapshots (e.g., S1, S2, and S3). Those existing user snapshots in the source FS may need to be copied to one or more target file systems. However, the source FS may create a new system snapshot (e.g., snapshot S4) as a base copy for initial synchronization between the source FS and one or more target file systems before performing the replication.

[0184] In one embodiment, the deletion of a snapshot key is tracked by the source CP, held temporarily in its persistent memory, and then applied to both the source FS and the target FS at the end of the replication cycle. The temporary hold or deferral means that the deletion is not immediate and is postponed for a short period depending on other factors. The reason is that if the deletion is applied immediately during the replication window / process, the garbage collector may remove part of the snapshot key before the target FS can apply the snapshot key, interfering with the replication process and causing inconsistencies. In other words, if a file deletion occurs during the replication window, the deletion is temporarily blocked until replication is complete and then applied to both the source FS and the target FS. Thus, the application of the deletion is the last step in the snapshot model. The FSS utilizes a method called deferred snapshot deletion, which is applicable only to user snapshots.

[0185] FIG. 15 is a diagram showing deferred snapshot deletion and replication for maintaining consistency between a source FS and a target FS according to an embodiment. In FIG. 15, the FSS has three replication cycles starting from the source FS 1510 and ending at the target FS 1530, and the source FS and the target FS are in different regions. Replication cycle 1 includes a source cycle 1 (1512) and a target cycle 1 (1532). Replication cycle 2 includes a source cycle 2 (1514) and a target cycle 2 (1534). Replication cycle 3 includes a source cycle 3 (1516) and a target cycle 3 (1536). Each replication cycle starts with a system snapshot, e.g., system snapshot S10 for replication cycle 1, system snapshot S20 for replication cycle 2, and system snapshot S30 for replication cycle 3.

[0186] As shown in FIG. 15, in one embodiment, when a user snapshot is deleted, typically at the end of the current replication cycle, the source FS holds off on deleting the snapshot until it receives from the target FS a notification that the deleted snapshot has been applied by the target FS. However, the deletion of the snapshot does not become effective at the target FS until the end of the next replication cycle. This deferred deletion prevents uncertainty and ensures consistency between the source FS and the target FS.

[0187] For illustration purposes, in FIG. 15, source FS 1510 creates two user snapshots S5 and S7 (shown as “+” for creating a snapshot), and then creates a system snapshot S10, thereby starting replication cycle 1 (1512) in the source region at 18:00 UTC. After a while, snapshots S5, S7, and S10 are applied by target FS 1530 during target cycle 1 (1532) (shown as +S5, +S7, and +S10), starting at 18:05 UTC (when the delta becomes available for the target FS to download) and ending at 18:15 UTC (i.e., when the delta application of snapshots S5, S7, and S10 is complete). This (1512 and 1532) completes replication cycle 1 for both file systems.

[0188] In one embodiment, while these snapshots are being replicated and transferred from source FS 1510 to target FS 1530 during replication cycle 1 (1512 and 1532), user snapshot S5 is deleted within source FS 1510 during source cycle 1 (1512) (shown as a "-" to delete the snapshot) at time UTC 18:05, before target FS 1530 begins applying these snapshots. Source CP 1510 allows S5 to continue to be transferred to target FS 1530, but temporarily holds off on the deletion (i.e., maintains snapshot S5 in a "deleting" state), and then deletes S5 at the end of entire replication cycle 1 (or target cycle 1 (1532)) after receiving a notification from target FS 1530 indicating that S5 has been applied by target FS 1530 at time 18:15 UTC (i.e., the CP changes to the "deleted" state for S5). At this point, since the internal state is set to the "deleting" state (i.e., deletion on hold), any other requests related to S5 may receive an HTTP 409 response (i.e., indicating a conflict between the other request and the current state of the resource). However, as shown in FIG. 15, snapshot S5 is not actually deleted by target FS 1530 until the end of replication cycle 2 (or target cycle 2 (1534)) at 19:15 UTC (shown as "-S5"). This delayed deletion of S5 is sometimes referred to as block deletion because the snapshot is blocked from immediate deletion.

[0189] In source FS1510, between replication cycle 1 (1512 and 1532) and before replication cycle 2 (1514 and 1534), additional snapshots (e.g., S16 and S18) are created while some snapshots (e.g., S7 and S16) are being deleted. Replication cycle 2 starts at 19:00 UTC from source cycle 2 (1514) and ends at 19:15 UTC within target cycle 2 (1534). User snapshot S7 was deleted between replication cycles 1 and 2 (shown as “-S7”), so S7 was deleted by the source FS at the time of the deletion request (also sometimes called non-blocking deletion) and by the target FS at the end of replication cycle 2 at 19:15 UTC.

[0190] In one embodiment, when a snapshot is deleted, the state of the corresponding snapshot key (a type of marker key) changes from visible to invisible, so that the user cannot read it. When the snapshot key is removed by garbage collection, the state changes from invisible to irrecoverable, and the snapshot key is deleted from the B-tree (i.e., no longer exists in memory). This can be described with respect to snapshot S16 in FIG. 15 below.

[0191] In FIG. 15, in source FS1510, within the same difference range or replication cycle window (i.e., after replication cycle 1 (1532) is completed and before replication cycle 2 (1514) starts the difference calculation), the snapshot S16 is created (shown as “+S16”) and deleted (shown as “-S16”), and since it becomes invisible even within source FS1510 for replication purposes, it cannot be replicated to target FS1530. Therefore, S16 becomes an unreachable entry and is not replicated to the target FS at all. Here, S16 becomes visible after being created (i.e., “+S16”), and then becomes invisible when it is deleted within the source FS (i.e., “-S16”). After the garbage collector removes S16, S16 becomes irrecoverable within the same replication cycle. This method may help save some replication resources. Therefore, when the snapshot is replicated from source FS1510 to target FS1530, it may include a gap (i.e., missing S16) in S15 - S17.

[0192] As described above, the user snapshot is controlled by the user. The user snapshot can be deleted only when the user requests deletion of the created snapshot. For example, in FIG. 15, the user snapshot S18 is created between replication cycle 1 (1512 and 1532) and replication cycle 2 (1514 and 1534), but is never deleted by the user. Therefore, the snapshot S18 can continue to exist and cannot be cleaned up by the FSS. In contrast, the snapshot S7 is created before replication cycle 1 (1512 and 1532) and is later deleted by the user between replication cycle 1 and replication cycle 2.

[0193] In some embodiments, if a snapshot is created after a replication cycle has started in the source FS (i.e., the difference has been calculated between two existing snapshots), that snapshot may not need to be transferred from the source FS to the target FS until the next replication cycle. For example, in FIG. 15, snapshot 22 is created in source FS 1510 during source replication cycle 2 (1514) (shown as “+S22”). Since the difference between system snapshot S20 and the previous snapshot has been calculated and the process of transferring from source FS 1510 to target FS 1530 is in progress, S22 does not need to be replicated to target FS 1530 during the current replication cycle that is already in progress (i.e., 1514 and 1534 of cycle 2) until the next replication cycle (i.e., 1516 and 1536 of cycle 3). However, if S22 is deleted before replication cycle 3 (1516) starts, S22 cannot be replicated to target FS 1530 because it has become irrecoverable, similar to the situation of snapshot S16 described above. Further, if S22 receives a delete request during source replication cycle 3 (1516), a delayed snapshot deletion scheme may be applied, similar to snapshot S5 described above.

[0194] FIG. 16 is a flowchart showing a process of delayed snapshot deletion and replication after detecting a snapshot deletion request according to an embodiment. At step 1601, the source FS may generate one or more snapshots in the source region. At step 1602, the source FS and the target FS may periodically perform inter-region replication. At step 1604, when the source FS detects a snapshot deletion request, e.g., a request to delete a user snapshot, at step 1606, the source FS needs to determine whether the snapshot deletion request occurs during the inter-region replication cycle of the source FS. If the snapshot deletion request does not occur during the inter-region replication cycle of the source FS, it means that the snapshot deletion request occurs between two replication cycles, e.g., after replication cycle N but before replication cycle N+1. At step 1608, the source FS can simply delete the requested snapshot without performing inter-region replication for this deleted snapshot.

[0195] The snapshot deleted in step 1608 may or may not have been replicated within the previous replication cycle (i.e., replication cycle N), depending on when the snapshot was created. Neither situation affects the operations within the current replication cycle (i.e., replication cycle N+1). For example, in FIG. 15, snapshot S7 is created before source cycle 1 (1512) and is then required to be deleted between source cycle 1 (1512) and source cycle 2 (1514). Since S7 has passed through replication cycle 1 (1512 and 1532), there is no need to replicate S7 again. In contrast, snapshot S16 is required to be deleted between source cycle 1 (1512) and source cycle 2 (1514) after it is created. Source FS1510 is not replicated after deleting S16.

[0196] In step 1606, if a snapshot deletion request occurs during the inter-region replication cycle of the source FS, in step 1620, the source FS may hold off (or temporarily refrain from) deleting the snapshot, but may still be able to continue to perform inter-region replication on the requested snapshot. In other words, the source FS may transfer the requested snapshot to the target FS, and the target FS can notify the source FS after performing differential application on the requested snapshot. In step 1622, the source FS may delete the requested snapshot at the end of the replication cycle when the target FS has completed the inter-region replication. In step 1624, the target FS does not have to delete the requested snapshot applied by the target FS within the current replication cycle until the end of the next replication cycle (i.e., wait for another replication cycle). For example, in FIG. 15, the deletion request for snapshot S5 occurs during source cycle 1 (1512). Source FS 1510 holds off the deletion and passes S5 to target FS 1530. After the target FS applies S5 and completes the replication and notifies the source FS in target cycle 1 (1532), the source FS deletes S5 at 18:15 UTC.

[0197] FIG. 17 is a flowchart showing the process of delayed snapshot deletion and replication after detecting a snapshot creation event according to an embodiment. In one embodiment, a snapshot is created during an inter-region replication cycle (e.g., replication cycle N), but if a snapshot deletion request is not received by the source FS before or during the next inter-region replication cycle (e.g., replication cycle N+1), the replication of the newly created snapshot is delayed until the next replication cycle.

[0198] In step 1702 of FIG. 17, source FS and target FS may periodically perform inter-region replication. In step 1704, if the source FS detects a new snapshot creation event, for example, a newly created user snapshot, then in step 1706, the source FS needs to determine whether the new snapshot is created during the inter-region replication cycle of the source FS. If the new snapshot is not created during the inter-region replication cycle of the source FS, it means that the new snapshot is created between two replication cycles, for example, after replication cycle N but before replication cycle N+1. In step 1708, source FS and target FS may replicate the new snapshot during the next replication cycle. For example, in FIG. 15, snapshot S18 is created between source cycle 1 (1512) and source cycle 2 (1514). Next, source FS 1510 and target FS 1530 may replicate S18 between replication 2, source cycle 2 (1514), and target cycle 2 (1534).

[0199] In step 1706, if the new snapshot is created during the inter-region replication cycle of the source FS, then in step 1720, if the source FS does not receive a snapshot deletion request before or during the next replication cycle, the source FS may delay the replication of the new snapshot until the next replication cycle. For example, in FIG. 15, snapshot S22 is created during source cycle 2 (1514), and no snapshot deletion request is received before or during source cycle 3 (1516). Therefore, source FS 1510 does not start replicating S22 until source cycle 3 (1516), and then transfers S22 to target FS 1530 for application during target cycle 3 (1536).

[0200] The delayed snapshot deletion and replication technique uses a schema table in the source FS's SDB and another schema table in the target FS's SDB to temporarily store the metadata of the snapshot to be deleted (e.g., snapshot S5 during source cycle 1 (1512) in FIG. 15) and can track the deleted snapshots.

[0201] In one embodiment, the schema table can be a key-value store containing metadata information, including but not limited to, a replication number, an applied snapshot number, a snapshot number of the deleted snapshot, and a workflow ID of the replicator that processes those snapshots. In other words, such a schema table can help track snapshot deletions of blocks and non-blocks within the source FS, and the snapshots may be applied to the target FS.

[0202] Snapshot Metadata Transfer between Source CP and Target CP FIG. 18 is a flow diagram showing the workflow of the control plane for a source region and a target region according to an embodiment. This workflow may include collection of snapshot metadata by the source region, transfer between the source region and the target region, and application of the metadata by the target region. As previously described in connection with FIG. 10, in one embodiment, the state machine for the inter-region replication process can be roughly divided into five parts: (1) collection and transfer of metadata in the source region, (2) generation and transfer of differences in the source region, (3) generation and transfer of a session key between the source region and the target region, (4) download and application of differences in the target region, and (5) download and application of metadata in the target region. FIG. 18 focuses on parts 1 (i.e., collection and transfer of metadata in the source region) and 5 (i.e., download and application of metadata in the target region) of the inter-region replication process.

[0203] At a high level regarding metadata processing, the source FS may extract snapshot metadata that includes, for example, the state of a state machine from Ready_to_Copy_Metadata to Snapshot_Metadata_Copied (which may also be referred to herein as a differential state), and upload it to the object store at the start of a replication cycle. Uploading metadata at the start of a replication cycle can help detect and resolve replication issues early before a large amount of data transfer begins. The target FS inputs the snapshot metadata and, after the differential application is complete and the metadata is added to the existing data, executes deletion of the snapshot. The snapshot metadata transfer may include, but is not limited to, a history ID, the type of snapshot (e.g., system snapshot and user snapshot), and the snapshot time. Further, snapshot records such as creation and deletion of snapshots are also part of this snapshot metadata transfer. Compared with FIG. 2 that illustrates differential transfer between a source region and a target region, FIG. 18 relates to control plane communication between a source region and a target region, particularly related to snapshot metadata information.

[0204] Generally, the source CP tracks the status of copy and deletion activities of snapshots within the source region and receives validation from the target CP. The source CP and the target CP communicate via the SDB within both regions. In step S1 of FIG. 18, the CP API 1810 may start recording the snapshot status, including all snapshots that have been deleted after the replication process has started. In step S2, the source snapshot generator 1812 (a separate thread within the CP API service) scans the replication policy and creates a system snapshot. In step S3, if the source CP API 1810 detects a snapshot deletion request during the replication cycle (which may also be referred to as a differential range from a data perspective), it records the pending deletion in the source SDB 1814 (e.g., the aforementioned schema table). In step S4, the source data plane (DP) / replicator 1816 may check the status of snapshot creation (e.g., whether a system snapshot has been created). If a new system snapshot has been created, in step S5, the differential monitor of the source CP API 1810 may update its differential state to Snapshot_Metadata_Copying (see step 1004 of FIG. 10). The differential monitor may be a thread in the CP API that manages and migrates the differential state. In step S6, the source CP 1810 may prepare information regarding the user snapshots within the current replication cycle by extracting the snapshot records, in addition to metadata such as the history ID, type of snapshot, and snapshot time. Thereafter, the source CP may store the extracted information in the source SDB 1814.

[0205] Next, at step S7, the replicator 1816 of the source DP can obtain metadata information from the source SDB 1814 and upload it to the object store 1850. In the case of periodic differences between system snapshots, they can be uploaded to the object store at different stages of the same replication cycle. At step S8, the source CP 1810 can change its differential state to Snapshot_Metadata_Copied (see step 1006 in FIG. 10) and update the source SDB 1814 accordingly. Next, the source CP API 1810 notifies the target CP API 1830 (i.e., the target CP host of the FSS) that the snapshot metadata is in the copied state. The source CP 1810 can also clean up the deleted snapshot records stored in the source SDB 1814 for the current replication cycle. Note that parts 2 (i.e., generation and transfer of differences in the source region) and 3 (i.e., generation and transfer of session keys) of the above-mentioned inter-region replication process are not described in FIG. 18.

[0206] After the source FS completes its difference generation and transfer process, at step S9, when the target CP 1830 receives a notification from the source CP API 1810, it can update the target SDB 1834 and accordingly change the differential state to Ready_To_Replicate (see step 1054 in FIG. 10). At step S10, when the target data plane (DP) (e.g., replicator) 1836 detects that the system snapshot is in the Copied state (within the source region) and Ready_To_Replicate (within the target region) and is ready to replicate, the target DP proceeds to the next step for difference application and then to metadata application.

[0207] In step S11, for differential replication, the target DP 1836 retrieves the difference from the object store. In step S12, the target replicator 1836 can apply the difference to the base snapshot of the target FS within the DP. In step S13, after completing the difference application, the target DP 1836 then notifies the target CP 1830 (e.g., the difference monitor) to update the difference state to the Replicated state (see step 1058 in FIG. 10).

[0208] Thereafter, the target FS can proceed to prepare for downloading and applying the metadata. In step S14, the target CP 1830 can change the difference state to Snapshot_Metadata_Populating (see step 1060 in FIG. 10). In step S15, the target DP 1836 downloads the snapshot metadata of the current replication cycle (or between the last snapshot number in the schema and the current snapshot number) from the object store and can input the metadata of all snapshots within this range. The target DP 1836 also downloads the deleted snapshot records of the current replication cycle from the object store. In step S16, the target CP 1830 then updates the difference state to Snapshot_Metadata_Populated (see step 1062 in FIG. 10) and notifies the source CP 1810.

[0209] In step S17, the target DP 1836 can then locally delete the system snapshot and clean up the corresponding snapshot metadata and deleted snapshot records. Thereby, the current replication cycle (or differential cycle) can be completed. In step S18, the target CP 1830 can then transition to the state X-region_Done (see step 1068 in FIG. 10) and notify the source CP 1810 about the completion of the current inter-region replication.

[0210] Infrastructure architecture as an exemplary service As described above, infrastructure as a service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also provide various services that arise in connection with those infrastructure components (examples of services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, since these services can be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.

[0211] In some cases, IaaS customers may access resources and services via a wide area network (WAN), such as the Internet, and use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and install enterprise software on the VM. Next, the customer can use the provider's services to perform various functions, including load balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0212] In most cases, the cloud computing model requires the participation of a cloud provider. The cloud provider can be a third-party service that specializes in providing (e.g., offering, lending, selling) IaaS, but it doesn't have to be. An entity can also choose to deploy a private cloud and become its own provider of infrastructure services.

[0213] In some examples, an IaaS deployment is a process of placing a new application or a new version of an application on a prepared application server or the like. This process may include a process of preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by a cloud provider under a hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, a customer may play a role in handling the deployment of an (OS), middleware, and / or application on top of, for example, a self-service virtual machine (e.g., that can be spun up on demand).

[0214] In some examples, IaaS provisioning may also refer to obtaining computers or virtual hosts for use and installing the required libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning may need to be done first.

[0215] In some cases, there are two different challenges in IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is executed. Second, after everything is provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two challenges can be addressed by enabling the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., which components are required and how those components exchange information) can be defined by one or more configuration files. In this way, the entire infrastructure topology (e.g., which resources depend on which resources and how each of those resources cooperate) can be described declaratively. In some cases, after the topology is defined, a workflow for creating and / or managing the various components described in the configuration file can be generated.

[0216] In some examples, the infrastructure can include many interconnected elements. For example, there can be one or more virtual private clouds (VPCs), also known as core networks (e.g., a pool of configurable and / or shared computing resources, optionally on demand). In some examples, there can also be one or more inbound traffic / outbound traffic group rules provisioned to define how the inbound and / or outbound traffic of the network is set, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. may be provisioned. As more infrastructure elements are desired and / or added, the infrastructure can evolve gradually.

[0217] In some cases, continuous deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Further, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is desirably deployed to one or more, but in many cases a number of, different production environments (e.g., across various geographical locations, sometimes worldwide). However, in some examples, the infrastructure to which the code is deployed must first be provisioned. In some cases, provisioning can be done manually, provisioning tools may be utilized to provision resources, and / or deployment tools may be utilized to deploy the code after the infrastructure has been provisioned.

[0218] FIG. 19 is a block diagram 1900 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1902 may be communicatively coupled to a secure host tenancy 1904 that may include a virtual cloud network (VCN) 1906 and a secure host subnet 1908. In some examples, the service operator 1902 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google® Glass head-mounted display) that run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and have Internet, email, short message service (SMS), BlackBerry®, or other communication protocols enabled. Alternatively, the client computing device can be a general-purpose personal computer, including, for example, personal computers and / or laptop computers that run various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. The client computing device can be a workstation computer that runs any of various commercially available UNIX® or UNIX-like operating systems, including but not limited to various GNU / Linux® operating systems such as Google® Chrome OS.Alternatively or in addition, the client computing device can be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect (registered trademark) gesture input device), and / or a personal messaging device, that can communicate via a network and / or the Internet accessible to the VCN 1906.

[0219] The VCN 1906 can include an LPG 1910 that can be communicatively coupled to an SSH VCN 1912 via a local peering gateway (LPG) 1910 included in a secure shell (SSH) VCN 1912. The SSH VCN 1912 can include an SSH subnet 1914 and can be communicatively coupled to a control plane VCN 1916 via an LPG 1910 included in the control plane VCN 1916. Also, the SSH VCN 1912 can be communicatively coupled to a data plane VCN 1918 via the LPG 1910. The control plane VCN 1916 and the data plane VCN 1918 can be included in a service tenancy 1919 that can be owned and / or operated by an IaaS provider.

[0220] The control plane VCN1916 may include a control plane demilitarized zone (DMZ) layer 1920 that functions as a border network (e.g., a part of an enterprise network between an enterprise intranet and an external network). Servers based on the DMZ have limited responsibilities and can help contain intrusions. Further, the DMZ layer 1920 may include one or more load balancer (LB) subnets 1922, a control plane application layer 1924 that may include an application subnet 1926, and a control plane data layer 1928 that may include a database (DB) subnet 1930 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1922 included in the control plane DMZ layer 1920 can be communicatively coupled to the application subnet 1926 included in the control plane application layer 1924 that may be included in the control plane VCN1916 and to the Internet gateway 1934, and the application subnet 1926 can be communicatively coupled to the DB subnet 1930 included in the control plane data layer 1928 as well as to the service gateway 1936 and the network address translation (NAT) gateway 1938. The control plane VCN1916 may include the service gateway 1936 and the NAT gateway 1938.

[0221] The control plane VCN1916 may include a data plane mirror application layer 1940 that may include an application subnet 1926. The application subnet 1926 included in the data plane mirror application layer 1940 may include a virtual network interface controller (VNIC) 1942 that can execute a compute instance 1944. The compute instance 1944 can communicatively couple the application subnet 1926 of the data plane mirror application layer 1940 to the application subnet 1926 that may be included in the data plane application layer 1946.

[0222] The data plane VCN 1918 may include a data plane application layer 1946, a data plane DMZ layer 1948, and a data plane data layer 1950. The data plane DMZ layer 1948 may include an LB subnet 1922 communicatively coupled to the application subnet 1926 of the data plane application layer 1946 and the Internet gateway 1934 of the data plane VCN 1918. The application subnet 1926 may be communicatively coupled to the service gateway 1936 of the data plane VCN 1918 and the NAT gateway 1938 of the data plane VCN 1918. The data plane data layer 1950 may also include a DB subnet 1930 communicatively coupled to the application subnet 1926 of the data plane application layer 1946.

[0223] The Internet gateways 1934 of the control plane VCN 1916 and the data plane VCN 1918 may be communicatively coupled to a metadata management service 1952 communicatively coupled to the public Internet 1954. The public Internet 1954 may be communicatively coupled to the NAT gateways 1938 of the control plane VCN 1916 and the data plane VCN 1918. The service gateways 1936 of the control plane VCN 1916 and the data plane VCN 1918 may be communicatively coupled to a cloud service 1956.

[0224] In some examples, the service gateway 1936 of the control plane VCN 1916 or the data plane VCN 1918 can make application programming interface (API) calls to the cloud service 1956 without going through the public Internet 1954. The API calls from the service gateway 1936 to the cloud service 1956 can be one-way, and the service gateway 1936 can make API calls to the cloud service 1956, and the cloud service 1956 can send the requested data to the service gateway 1936. However, the cloud service 1956 does not need to initiate API calls to the service gateway 1936.

[0225] In some examples, the secure host tenancy 1904 can be directly connected to the service tenancy 1919, or otherwise can be separated. The secure host subnet 1908 can communicate with the SSH subnet 1914 via the LPG 1910, and the LPG 1910 can enable two-way communication on a separated system if not. Connecting the secure host subnet 1908 to the SSH subnet 1914 can give the secure host subnet 1908 access to other entities within the service tenancy 1919.

[0226] The control plane VCN 1916 may enable users of service tenancy 1919 to set or otherwise provision desired resources. Desired resources provisioned within the control plane VCN 1916 may be deployed or otherwise used in the data plane VCN 1918. In some examples, the control plane VCN 1916 may be separable from the data plane VCN 1918, and the data plane mirror app layer 1940 of the control plane VCN 1916 may communicate with the data plane app layer 1946 of the data plane VCN 1918 via VNICs 1942 that may be included in the data plane mirror app layer 1940 and the data plane app layer 1946.

[0227] In some examples, a user or customer of the system may perform requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1954 that can communicate requests to the metadata management service 1952. The metadata management service 1952 can communicate the requests to the control plane VCN 1916 via the internet gateway 1934. The requests may be received by the LB subnet 1922 included in the control plane DMZ layer 1920. The LB subnet 1922 may determine that the requests are valid and, in response, the LB subnet 1922 may send the requests to the app subnet 1926 included in the control plane app layer 1924. If the validity of the requests is confirmed and the requests require calls to the public internet 1954, the calls to the public internet 1954 may be sent to the NAT gateway 1938 that can make calls to the public internet 1954. Metadata that may desirably be stored by the requests may be stored within the DB subnet 1930.

[0228] In some examples, the data plane mirror application layer 1940 can facilitate direct communication between the control plane VCN 1916 and the data plane VCN 1918. For example, it may be desirable for changes, updates, or other appropriate modifications to the configuration to be applied to the resources included in the data plane VCN 1918. Through the VNIC 1942, the control plane VCN 1916 can communicate directly with the resources included in the data plane VCN 1918, thereby enabling changes, updates, or other appropriate modifications to the configuration of the resources.

[0229] In some embodiments, the control plane VCN 1916 and the data plane VCN 1918 may be included in the service tenant 1919. In this case, the user or customer of the system does not have to own or operate either the control plane VCN 1916 or the data plane VCN 1918. Instead, the IaaS provider may own or operate both the control plane VCN 1916 and the data plane VCN 1918, which may both be included in the service tenancy 1919. This embodiment can enable network separation that can prevent a user or customer from exchanging information with the resources of other users or other customers. Also, this embodiment can enable a user or customer of the system to privately store a database without relying on the public Internet 1954, which may not have the desired level of threat prevention for storage.

[0230] In other embodiments, the LB subnet 1922 included in the control plane VCN 1916 can be configured to receive signals from the service gateway 1936. In this embodiment, the control plane VCN 1916 and the data plane VCN 1918 can be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1954. A customer of the IaaS provider may desire this embodiment because the databases used by the customer may be controlled by the IaaS provider and may be stored in a service tenancy 1919 that can be isolated from the public Internet 1954.

[0231] FIG. 20 is a block diagram 2000 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2002 (e.g., service operator 1902 of FIG. 19) may be communicatively coupled to a secure host tenancy 2004 (e.g., secure host tenancy 1904 of FIG. 19) that may include a virtual cloud network (VCN) 2006 (e.g., VCN 1906 of FIG. 19) and a secure host subnet 2008 (e.g., secure host subnet 1908 of FIG. 19). The VCN 2006 may include a local peering gateway (LPG) 2010 (e.g., LPG 1910 of FIG. 19) and may be communicatively coupled to a secure shell (SSH) VCN 2012 (e.g., SSH VCN 1912 of FIG. 19) via the LPG 1910 included in the SSH VCN 2012. The SSH VCN 2012 may include an SSH subnet 2014 (e.g., SSH subnet 1914 of FIG. 19), and the SSH VCN 2012 may be communicatively coupled to a control plane VCN 2016 (e.g., control plane VCN 1916 of FIG. 19) via the LPG 2010 included in the control plane VCN 2016. The control plane VCN 2016 may be included in a service tenancy 2019 (e.g., service tenancy 1919 of FIG. 19), and the data plane VCN 2018 (e.g., data plane VCN 1918 of FIG. 19) may be included in a customer tenancy 2021 that may be owned or operated by a user or customer of the system.

[0232] The control plane VCN 2016 may include a control plane DMZ layer 2020 (e.g., the control plane DMZ layer 1920 in FIG. 19) that may include an LB subnet 2022 (e.g., the LB subnet 1922 in FIG. 19), a control plane application layer 2024 (e.g., the control plane application layer 1924 in FIG. 19) that may include an application subnet 2026 (e.g., the application subnet 1926 in FIG. 19), and a control plane data layer 2028 (e.g., the control plane data layer 1928 in FIG. 19) that may include a database (DB) subnet 2030 (e.g., similar to the DB subnet 1930 in FIG. 19). The LB subnet 2022 included in the control plane DMZ layer 2020 is communicably coupled to the application subnet 2026 included in the control plane application layer 2024 that may be included in the control plane VCN 2016, and an Internet gateway 2034 (e.g., the Internet gateway 1934 in FIG. 19), and the application subnet 2026 is communicably coupled to the DB subnet 2030 included in the control plane data layer 2028, as well as a service gateway 2036 (e.g., the service gateway 1936 in FIG. 19) and a network address translation (NAT) gateway 2038 (e.g., the NAT gateway 1938 in FIG. 19). The control plane VCN 2016 may include the service gateway 2036 and the NAT gateway 2038.

[0233] The control plane VCN 2016 may include a data plane mirror app layer 2040 (e.g., the data plane mirror app layer 1940 of FIG. 19) that may include an app subnet 2026. The app subnet 2026 included in the data plane mirror app layer 2040 may include a virtual network interface controller (VNIC) 2042 (e.g., the VNIC 1942) that can execute a compute instance 2044 (e.g., similar to the compute instance 1944 of FIG. 19). The compute instance 2044 can facilitate communication between the app subnet 2026 of the data plane mirror app layer 2040 and an app subnet 2026 that may be included in the data plane app layer 2046 (e.g., the data plane app layer 1946 of FIG. 19) via the VNIC 2042 included in the data plane mirror app layer 2040 and the VNIC 2042 included in the data plane app layer 2046.

[0234] The internet gateway 2034 included in the control plane VCN 2016 may be communicatively coupled to a metadata management service 2052 (e.g., the metadata management service 1952 of FIG. 19) that may be communicatively coupled to the public internet 2054 (e.g., the public internet 1954 of FIG. 19). The public internet 2054 may be communicatively coupled to the NAT gateway 2038 included in the control plane VCN 2016. The service gateway 2036 included in the control plane VCN 2016 may be communicatively coupled to a cloud service 2056 (e.g., the cloud service 1956 of FIG. 19).

[0235] In some examples, the data plane VCN 2018 may be included in the customer's tenancy 2021. In this case, the IaaS provider may provide a control plane VCN 2016 for each customer, and the IaaS provider may configure, for each customer, the specific compute instances 2044 included in the service tenancy 2019. Each compute instance 2044 may enable communication between the control plane VCN 2016 included in the service tenancy 2019 and the data plane VCN 2018 included in the customer's tenancy 2021. The compute instance 2044 may enable resources provisioned within the control plane VCN 2016 included in the service tenancy 2019 to be deployed or otherwise used in the data plane VCN 2018 included in the customer's tenancy 2021.

[0236] In other examples, a customer of the IaaS provider may have a database that persists in the customer's tenancy 2021. In this example, the control plane VCN 2016 may include a data plane mirror app layer 2040 that may include an app subnet 2026. The data plane mirror app layer 2040 may be present in the data plane VCN 2018, but the data plane mirror app layer 2040 need not persist in the data plane VCN 2018. That is, the data plane mirror app layer 2040 may have access rights to the customer's tenancy 2021, but the data plane mirror app layer 2040 need not be present in the data plane VCN 2018 and need not be owned or operated by the customer of the IaaS provider. The data plane mirror app layer 2040 may be configured to make calls to the data plane VCN 2018, but need not be configured to make calls to any entity included in the control plane VCN 2016. The customer may desire to deploy or otherwise use resources within the data plane VCN 2018 that are provisioned within the control plane VCN 2016, and the data plane mirror app layer 2040 may facilitate the desired deployment or other use of the customer's resources.

[0237] In some embodiments, a customer of an IaaS provider can apply filters to the data plane VCN 2018. In this embodiment, the customer can determine which data plane VCNs 2018 are accessible, and the customer can restrict access from the data plane VCN 2018 to the public Internet 2054. The IaaS provider may not be able to apply filters or otherwise control access of the data plane VCN 2018 to any external network or database. Applying filters and controls by the customer to the data plane VCNs 2018 included in the customer's tenancy 2021 can help isolate the data plane VCNs 2018 from other customers and from the public Internet 2054.

[0238] In some embodiments, cloud service 2056 can be invoked by service gateway 2036 to access services that may not exist on either public Internet 2054, control plane VCN 2016, or data plane VCN 2018. The connection between cloud service 2056 and control plane VCN 2016 or data plane VCN 2018 may not be operational or continuous. Cloud service 2056 may exist on a different network owned or operated by an IaaS provider. Cloud service 2056 may be configured to receive calls from service gateway 2036 and may be configured not to receive calls from public Internet 2054. Some cloud services 2056 may be isolated from other cloud services 2056, and control plane VCN 2016 may be isolated from cloud services 2056 that may not exist in the same region as control plane VCN 2016. For example, control plane VCN 2016 may be located in "Region 1", and "Deployment 19" of the cloud service may be located in Region 1 and "Region 2". When a call to Deployment 19 is made by service gateway 2036 included in control plane VCN 2016 located in Region 1, this call can be sent to Deployment 19 within Region 1. In this example, control plane VCN 2016, or Deployment 19 within Region 1, may or may not be communicatively coupled to Deployment 19 within Region 2.

[0239] FIG. 21 is a block diagram 2100 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2102 (e.g., service operator 1902 of FIG. 19) can be communicatively coupled to a secure host tenancy 2104 (e.g., secure host tenancy 1904 of FIG. 19) that can include a virtual cloud network (VCN) 2106 (e.g., VCN 1906 of FIG. 19) and a secure host subnet 2108 (e.g., secure host subnet 1908 of FIG. 19). The VCN 2106 can include an LPG 2110 (e.g., LPG 1910 of FIG. 19) and can be communicatively coupled to an SSH VCN 2112 (e.g., SSH VCN 1912 of FIG. 19) via the LPG 2110 included in the SSH VCN 2112. The SSH VCN 2112 can include an SSH subnet 2114 (e.g., SSH subnet 1914 of FIG. 19), and the SSH VCN 2112 can be communicatively coupled to a control plane VCN 2116 (e.g., control plane VCN 1916 of FIG. 19) via the LPG 2110 included in the control plane VCN 2116 and to a data plane VCN 2118 (e.g., data plane 1918 of FIG. 19) via the LPG 2110 included in the data plane VCN 2118. The control plane VCN 2116 and the data plane VCN 2118 can be included in a service tenancy 2119 (e.g., service tenancy 1919 of FIG. 19).

[0240] The control plane VCN 2116 may include a control plane DMZ layer 2120 (e.g., the control plane DMZ layer 1920 in FIG. 19) that may include a load balancer (LB) subnet 2122 (e.g., the LB subnet 1922 in FIG. 19), a control plane application layer 2124 (e.g., similar to the control plane application layer 1924 in FIG. 19) that may include an application subnet 2126 (e.g., similar to the application subnet 1926 in FIG. 19), and a control plane data layer 2128 (e.g., the control plane data layer 1928 in FIG. 19) that may include a DB subnet 2130. The LB subnet 2122 included in the control plane DMZ layer 2120 is communicatively coupled to the application subnet 2126 included in the control plane application layer 2124 that may be included in the control plane VCN 2116, and to an Internet gateway 2134 (e.g., the Internet gateway 1934 in FIG. 19). The application subnet 2126 is communicatively coupled to the DB subnet 2130 included in the control plane data layer 2128, as well as to a service gateway 2136 (e.g., the service gateway in FIG. 19) and a network address translation (NAT) gateway 2138 (e.g., the NAT gateway 1938 in FIG. 19). The control plane VCN 2116 may include the service gateway 2136 and the NAT gateway 2138.

[0241] The data plane VCN 2118 may include a data plane application layer 2146 (e.g., the data plane application layer 1946 of FIG. 19), a data plane DMZ layer 2148 (e.g., the data plane DMZ layer 1948 of FIG. 19), and a data plane data layer 2150 (e.g., the data plane data layer 1950 of FIG. 19). The data plane DMZ layer 2148 may include a LB subnet 2122 communicatively coupled to a trusted application subnet 2160 and an untrusted application subnet 2162 of the data plane application layer 2146 and an Internet gateway 2134 included in the data plane VCN 2118. The trusted application subnet 2160 may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118, a NAT gateway 2138 included in the data plane VCN 2118, and a DB subnet 2130 included in the data plane data layer 2150. The untrusted application subnet 2162 may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118 and a DB subnet 2130 included in the data plane data layer 2150. The data plane data layer 2150 may include a DB subnet 2130 communicatively coupled to a service gateway 2136 included in the data plane VCN 2118.

[0242] The untrusted application subnet 2162 can include one or more primary VNICs 2164(1) to (N) communicatively coupled to tenant virtual machines (VMs) 2166(1) to (N). Each tenant VM 2166(1) to (N) can be communicatively coupled to respective application subnets 2167(1) to (N) that can be included in respective container egress VCNs 2168(1) to (N) that can be included in respective customer tenancies 2170(1) to (N). Each secondary VNIC 2172(1) to (N) can facilitate communication between the untrusted application subnet 2162 included in the data plane VCN 2118 and the application subnets included in the container egress VCNs 2168(1) to (N). Each container egress VCN 2168(1) to (N) can include a NAT gateway 2138 communicatively coupled to the public Internet 2154 (e.g., the public Internet 1954 of FIG. 19).

[0243] The Internet gateway 2134 included in the control plane VCN 2116 and in the data plane VCN 2118 can be communicatively coupled to a metadata management service 2152 (e.g., the metadata management system 1952 of FIG. 19) communicatively coupled to the public Internet 2154. The public Internet 2154 can be communicatively coupled to the NAT gateway 2138 included in the control plane VCN 2116 and in the data plane VCN 2118. The service gateway 2136 included in the control plane VCN 2116 and in the data plane VCN 2118 can be communicatively coupled to a cloud service 2156.

[0244] In some embodiments, the data plane VCN 2118 can be integrated with the customer's tenancy 2170. This integration can be useful or desirable for customers of the IaaS provider in some cases, such as when they may want support when running code. The customer may provide code to run that can be disruptive, communicate with other customers' resources, or otherwise cause unwanted effects. In response, the IaaS provider can determine whether to execute the code provided to the IaaS provider by the customer.

[0245] In some examples, a customer of an IaaS provider may grant the IaaS provider temporary network access rights and request a function connected to the data plane application layer 2146. The code for executing this function may be executed in VMs 2166(1) to (N), and this code need not be configured to execute elsewhere on the data plane VCN 2118. Each of VMs 2166(1) to (N) may be connected to the tenancy 2170 of one customer. Each container 2171(1) to (N) included in VMs 2166(1) to (N) may be configured to execute the code. In this case, there can be a two-fold separation (for example, the containers 2171(1) to (N) executing the code, the containers 2171(1) to (N) may be included in at least the VMs 2166(1) to (N) included in the untrusted application subnet 2162), which can help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging the networks of different customers. The containers 2171(1) to (N) may be communicatively coupled to the customer's tenancy 2170 and may be configured to send or receive data with the customer's tenancy 2170. The containers 2171(1) to (N) need not be configured to send or receive data with any other entity within the data plane VCN 2118. Upon completion of the execution of the code, the IaaS provider may force the containers 2171(1) to (N) to terminate or otherwise discard them.

[0246] In some embodiments, the trusted application subnet 2160 may execute code that can be owned or operated by an IaaS provider. In this embodiment, the trusted application subnet 2160 may be communicatively coupled to the DB subnet 2130 and may be configured to perform CRUD operations within the DB subnet 2130. The untrusted application subnet 2162 may be communicatively coupled to the DB subnet 2130, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 2130. The containers 2171(1)~(N) that can execute customer code, which may be included in each customer's VM2166(1)~(N), may not be communicatively coupled to the DB subnet 2130.

[0247] In other embodiments, the control plane VCN 2116 and the data plane VCN 2118 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 2116 and the data plane VCN 2118. However, communication can occur indirectly by at least one method. The LPG 2110 may be established by the IaaS provider, thereby facilitating communication between the control plane VCN 2116 and the data plane VCN 2118. In another example, the control plane VCN 2116 or the data plane VCN 2118 may make calls to the cloud service 2156 via the service gateway 2136. For example, a call from the control plane VCN 2116 to the cloud service 2156 may include a request for a service that can communicate with the data plane VCN 2118.

[0248] FIG. 22 is a block diagram 2200 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2202 (e.g., service operator 1902 of FIG. 19) may be communicatively coupled to a secure host tenancy 2204 (e.g., secure host tenancy 1904 of FIG. 19) that may include a virtual cloud network (VCN) 2206 (e.g., VCN 1906 of FIG. 19) and a secure host subnet 2208 (e.g., secure host subnet 1908 of FIG. 19). The VCN 2206 may include an LPG 2210 (e.g., LPG 1910 of FIG. 19) and may be communicatively coupled to an SSH VCN 2212 (e.g., SSH VCN 1912 of FIG. 19) via the LPG 2210 included in the SSH VCN 2212. The SSH VCN 2212 can include an SSH subnet 2214 (e.g., SSH subnet 1914 of FIG. 19), and the SSH VCN 2212 may be communicatively coupled to a control plane VCN 2216 (e.g., control plane VCN 1916 of FIG. 19) via the LPG 2210 included in the control plane VCN 2216 and to a data plane VCN 2218 (e.g., data plane 1918 of FIG. 19) via the LPG 2210 included in the data plane VCN 2218. The control plane VCN 2216 and the data plane VCN 2218 may be included in a service tenancy 2219 (e.g., service tenancy 1919 of FIG. 19).

[0249] The control plane VCN 2216 may include a control plane DMZ layer 2220 (e.g., the control plane DMZ layer 1920 in FIG. 19) that may include an LB subnet 2222 (e.g., the LB subnet 1922 in FIG. 19), a control plane application layer 2224 (e.g., the control plane application layer 1924 in FIG. 19) that may include an application subnet 2226 (e.g., the application subnet 1926 in FIG. 19), and a control plane data layer 2228 (e.g., the control plane data layer 1928 in FIG. 19) that may include a DB subnet 2230 (e.g., the DB subnet 2130 in FIG. 21). The LB subnet 2222 included in the control plane DMZ layer 2220 can be communicatively coupled to the application subnet 2226 included in the control plane application layer 2224 that may be included in the control plane VCN 2216, and to an Internet gateway 2234 (e.g., the Internet gateway 1934 in FIG. 19). The application subnet 2226 can be communicatively coupled to the DB subnet 2230 included in the control plane data layer 2228, as well as to a service gateway 2236 (e.g., the service gateway in FIG. 19) and a network address translation (NAT) gateway 2238 (e.g., the NAT gateway 1938 in FIG. 19). The control plane VCN 2216 may include a service gateway 2236 and a NAT gateway 2238.

[0250] The data plane VCN 2218 may include a data plane application layer 2246 (e.g., the data plane application layer 1946 of FIG. 19), a data plane DMZ layer 2248 (e.g., the data plane DMZ layer 1948 of FIG. 19), and a data plane data layer 2250 (e.g., the data plane data layer 1950 of FIG. 19). The data plane DMZ layer 2248 may include a trusted application subnet 2260 (e.g., the trusted application subnet 2160 of FIG. 21) and an untrusted application subnet 2262 (e.g., the untrusted application subnet 2162 of FIG. 21) of the data plane application layer 2246, as well as an LB subnet 2222 communicatively coupled to an Internet gateway 2234 included in the data plane VCN 2218. The trusted application subnet 2260 may be communicatively coupled to a service gateway 2236 included in the data plane VCN 2218, a NAT gateway 2238 included in the data plane VCN 2218, and a DB subnet 2230 included in the data plane data layer 2250. The untrusted application subnet 2262 may be communicatively coupled to a service gateway 2236 included in the data plane VCN 2218 and a DB subnet 2230 included in the data plane data layer 2250. The data plane data layer 2250 may include a DB subnet 2230 communicatively coupled to a service gateway 2236 included in the data plane VCN 2218.

[0251] The untrusted application subnet 2262 may include primary VNICs 2264(1) to (N) communicatively coupled to tenant virtual machines (VMs) 2266(1) to (N) present within the untrusted application subnet 2262. Each tenant VM 2266(1) to (N) can execute code within its respective containers 2267(1) to (N) and can be communicatively coupled to an application subnet 2226 that may be included in a data plane application layer 2246 that may be included in a container egress VCN 2268. Each secondary VNIC 2272(1) to (N) can facilitate communication between the untrusted application subnet 2262 included in the data plane VCN 2218 and the application subnet included in the container egress VCN 2268. The container egress VCN may include a NAT gateway 2238 communicatively coupled to a public internet 2254 (e.g., the public internet 1954 of FIG. 19).

[0252] The internet gateway 2234 included in the control plane VCN 2216 and the data plane VCN 2218 can be communicatively coupled to a metadata management service 2252 (e.g., the metadata management system 1952 of FIG. 19) communicatively coupled to the public internet 2254. The public internet 2254 can be communicatively coupled to a NAT gateway 2238 included in the control plane VCN 2216 and the data plane VCN 2218. The service gateway 2236 included in the control plane VCN 2216 and the data plane VCN 2218 can be communicatively coupled to a cloud service 2256.

[0253] In some examples, the pattern shown by the architecture of block diagram 2200 in FIG. 22 may be considered an exception to the pattern shown by the architecture of block diagram 2100 in FIG. 21, which may be desirable for the customers of the IaaS provider when the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). Each of the containers 2267(1)-(N) included in VM2266(1)-(N) for each customer may be accessed in real time by the customer. The containers 2267(1)-(N) may be configured to make calls to the respective secondary VNICs 2272(1)-(N) included in the app subnet 2226 of the data plane app layer 2246 that may be included in the container egress VCN 2268. The secondary VNICs 2272(1)-(N) may be able to send the calls to the NAT gateway 2238, and the NAT gateway 2238 may be able to send the calls to the public Internet 2254. In this example, the containers 2267(1)-(N) that may be accessed in real time by the customer may be separable from the control plane VCN 2216 and separable from other entities included in the data plane VCN 2218. The containers 2267(1)-(N) may also be separable from the resources of other customers.

[0254] In other examples, a customer can call the cloud service 2256 using the containers 2267(1) to (N). In this example, the customer can execute the code within the containers 2267(1) to (N) that requests a service from the cloud service 2256. The containers 2267(1) to (N) can send this request to the secondary VNICs 2272(1) to (N), which can send this request to a NAT gateway, which can send this request to the public Internet 2254. The public Internet 2254 can send this request to the LB subnet 2222 included in the control plane VCN 2216 via the Internet gateway 2234. In response to determining that this request is valid, the LB subnet can send this request to the app subnet 2226, which can send this request to the cloud service 2256 via the service gateway 2236.

[0255] It should be understood that the IaaS architectures 1900, 2000, 2100, 2200 shown in the figures may include components other than those shown. Further, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may include more or fewer components than those shown in the figures, combine two or more components, or have a different configuration or arrangement of components.

[0256] In one embodiment, the IaaS system described herein may include the provision of a series of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0257] FIG. 23 shows an exemplary computer system 2300 in which various embodiments may be implemented. System 2300 may be used to implement any of the computer systems described above. As shown in the figure, computer system 2300 includes a processing unit 2304 that communicates with a plurality of peripheral subsystems via a bus subsystem 2302. These peripheral subsystems may include a processing acceleration unit 2306, an I / O subsystem 2308, a storage subsystem 2318, and a communication subsystem 2324. Storage subsystem 2318 includes tangible computer-readable storage media 2322 and system memory 2310.

[0258] The bus subsystem 2302 provides a mechanism for the various components and subsystems of the computer system 2300 to communicate with each other as intended. Although the bus subsystem 2302 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2302 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus that uses any of various bus architectures. For example, such an architecture can include an ISA (Industry Standard Architecture) bus, an MCA (Micro Channel Architecture) bus, an EISA (Enhanced ISA) bus, a VESA (Video Electronics Standards Association) local bus, and a PCI (Peripheral Component Interconnect) bus implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard.

[0259] A processing unit 2304, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2300. One or more processors may be included in the processing unit 2304. These processors can include single-core processors or multi-core processors. In certain embodiments, the processing unit 2304 may be implemented as one or more independent processing units 2332 and / or 2334, with a single-core processor or multi-core processor included in each processing unit. In other embodiments, the processing unit 2304 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0260] In various embodiments, processing unit 2304 can execute various programs according to program code and can maintain multiple programs or processes to be executed simultaneously. At any given time, some or all of the program code to be executed can reside in processor 2304 and / or storage subsystem 2318. With appropriate programming, processor 2304 can provide the various functions described above. Computer system 2300 can further include a processing acceleration unit 2306 that can include a digital signal processor (DSP), an application specific processor, and / or the like.

[0261] The I / O subsystem 2308 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated in a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device having a voice command recognition system, a microphone, and other types of input devices. The user interface input devices may enable a user to interact with information by controlling an input device such as a Microsoft Xbox (registered trademark) 360 game controller via a natural user interface using gestures and spoken commands, and may include motion detection devices and / or gesture recognition devices such as a Microsoft Kinect (registered trademark) motion sensor. The user interface input devices may include gesture recognition devices such as a Google Glass (registered trademark) blink detector that detects a user's eye activity (e.g., a "blink" when taking a picture and / or selecting a menu) and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Further, the user interface input devices may include a voice recognition detection device that enables a user to interact with a voice recognition system (e.g., a Siri (registered trademark) navigator) via a voice command.

[0262] The user interface input device may include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, game pad, and graphics tablet, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser distance meters, and eye tracking devices. Further, the user interface input device may include medical image input devices such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasonic examination devices. The user interface input device may include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0263] The user interface output device may include, among others, visual displays other than display subsystems, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as a flat panel device using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, a touch screen, or the like. Generally, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2300 to the user or other computers. For example, the user interface output device may include, but is not limited to, various display devices for visually communicating text information, graphics information, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0264] Computer system 2300 may include a storage subsystem 2318 that provides a tangible, non-transitory computer-readable storage medium for storing software and data structures that provide the functionality of the embodiments described in this disclosure. The software can include programs, code modules, instructions, scripts, etc., and when executed by one or more cores or processors of processing unit 2304, provides the aforementioned functionality. The storage subsystem 2318 may also provide a repository for storing data used in accordance with this disclosure.

[0265] As shown in the example of FIG. 23, the storage subsystem 2318 may include various components including system memory 2310, a computer-readable storage medium 2322, and a computer-readable storage medium reader 2320. The system memory 2310 may store program instructions that are readable and executable by the processing unit 2304. The system memory 2310 may also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs may be loaded into the system memory 2310 including, but not limited to, client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0266] System memory 2310 may also store an operating system 2316. Examples of operating systems 2316 include Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux® operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations where computer system 2300 runs one or more virtual machines, the virtual machines may be loaded into system memory 2310 along with guest operating systems (GOS) and executed by one or more processors or cores of processing unit 2304.

[0267] System memory 2310 can be provided in different configurations depending on the type of computer system 2300. For example, system memory 2310 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Various types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), and the like. In some implementations, system memory 2310 can include a basic input / output system (BIOS) that contains basic routines useful for transferring information between elements within computer system 2300, such as during startup.

[0268] Computer-readable storage medium 2322 represents a storage medium for temporarily and / or more persistently containing and storing computer-readable information for use by computer system 2300, including instructions executable by processing unit 2304 of computer system 2300, in addition to remote storage devices, local storage devices, fixed storage devices, and / or removable storage devices.

[0269] The computer-readable storage medium 2322 can include any suitable medium known in or used in the art, including but not limited to storage media and communication media such as volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information. The computer-readable storage medium 2322 can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer-readable media.

[0270] As an example, computer-readable storage medium 2322 can include a hard disk drive that reads from and writes to a removable non-volatile magnetic medium, a magnetic disk drive that reads from and writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from and writes to a removable non-volatile optical disk such as a CD ROM, DVD, and Blu-ray (registered trademark) disk, or other optical media. Computer-readable storage medium 2322 can include, but is not limited to, a Zip (registered trademark) drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. Computer-readable storage medium 2322 can include SSDs based on non-volatile memory such as flash memory-based semiconductor drives (SSDs), enterprise flash drives, semiconductor ROMs, SSDs based on volatile memory such as semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data of computer system 2300.

[0271] Machine-readable instructions executable by one or more processors or cores of processing unit 2304 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include a physically tangible memory or storage device, including a volatile memory storage device and / or a non-volatile storage device. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy (registered trademark) drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0272] Communication subsystem 2324 provides an interface to other computer systems and networks. Communication subsystem 2324 functions as an interface for receiving data from other systems of computer system 2300 and for transmitting data to other systems. For example, communication subsystem 2324 may enable computer system 2300 to connect to one or more devices via the Internet. In some embodiments, communication subsystem 2324 may include components of a radio frequency (RF) transceiver for accessing wireless voice and / or data networks (such as cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.11 family of standards, or other mobile communication technologies, or any combination thereof), components of a global positioning system (GPS) receiver, and / or other components. In some embodiments, communication subsystem 2324 may provide a wired network connection (such as Ethernet (registered trademark)) in addition to, or instead of, a wireless interface.

[0273] In some embodiments, communication subsystem 2324 may also receive input communications on behalf of one or more users who may use computer system 2300, in the form of structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, and the like.

[0274] As an example, the communication subsystem 2324 may be configured to receive the data feed 2326 in real time from social networks such as Twitter (registered trademark) feeds, Facebook (registered trademark) updates, and / or other communication services, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0275] Furthermore, the communication subsystem 2324 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2328 of real-time events and / or event updates 2330 that are essentially continuous without an explicit end or may have no boundaries. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0276] The communication subsystem 2324 may be configured to output structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc. to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2300.

[0277] The computer system 2300 can be one of various types, including a handheld portable device (e.g., an iPhone (registered trademark) mobile phone, an iPad (registered trademark) computing tablet, a PDA), a wearable device (e.g., a Google Glass (registered trademark) head-mounted display), a PC, a workstation, a mainframe, a ticket vending machine, a server rack, or any other data processing system.

[0278] Due to the constantly changing nature of computers and networks, the description of the computer system 2300 shown in the figures is merely intended to be a specific example. Many other configurations are possible that include more or fewer components than the system shown in the figures. For example, customized hardware may be used and / or certain elements may be implemented in hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other methods and / or ways to implement various embodiments.

[0279] Embodiments may be implemented by using a computer program product that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure.

[0280] Although specific embodiments have been described, various modifications, changes, alternative structures, and equivalents are also included within the scope of this disclosure. Embodiments are not limited to operating within a particular data processing environment and can operate freely within multiple data processing environments. Further, although embodiments have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that the scope of this disclosure is not limited to the series of transactions and steps described. The various features and aspects of the foregoing embodiments may be used individually or together.

[0281] Furthermore, although embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented using only hardware, only software, or combinations thereof. The various processes described herein may be implemented on the same processor or different processors in any combination. Thus, when a component or service is described as being configured to perform an operation, such a configuration may be realized, for example, by designing an electronic circuit to perform this operation, by programming a programmable electronic circuit (such as a microprocessor) to perform this operation, or by any combination thereof. Processes can communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0282] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is obvious that additions, deletions, omissions, as well as other modifications and changes may be made without departing from the broader ideas and scope as set forth in the claims. Thus, although specific embodiments of the disclosure have been described, these are not intended to be limiting. Various changes and equivalents are within the scope of the appended claims.

[0283] In the context of describing the disclosed embodiments (in particular, in the context of the appended claims), the use of the terms "a," "an," and "the" and similar referents should be construed to cover both the singular and the plural, unless specifically indicated otherwise herein or clearly contradicted by the context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" should be construed to mean that, even in the presence of intervening elements, it is either partially or wholly contained within, connected to, or joined together with. The recitation of a range of values herein is merely intended to serve as a convenient way of referring individually to each separate value falling within the range, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order unless specifically indicated otherwise herein or clearly contradicted by the context. The use of any examples, or exemplary language (e.g., "such as") provided herein is merely intended to better clarify the embodiments and does not impose a limitation on the scope of the disclosure unless otherwise claimed. No language in this specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0284] Disjunctive language, such as the phrase "at least one of X, Y, or Z", is generally intended, unless otherwise explicitly stated, to convey that items, conditions, etc. can be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z) within the context in which it is used. Thus, such disjunctive language is generally not intended, and should not be taken, to mean that a particular embodiment requires the presence of at least one of each of at least one of X, at least one of Y, or at least one of Z.

[0285] In this specification, preferred embodiments of the present disclosure are described, including the best mode known to the applicant for carrying out the present disclosure. Variations of such preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to adopt such variations as needed, and the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Further, any combination of the foregoing elements in all possible variations of the embodiments is included in the present disclosure unless otherwise specifically indicated herein.

[0286] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference in their entirety, to the same extent as if each reference were individually and specifically set forth and incorporated by reference herein.

[0287] In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments of the present specification. However, those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or together. Furthermore, embodiments may be utilized in any number of environments and applications beyond those described herein and within the broader spirit and scope of the present specification without departing from it. Accordingly, the present specification and drawings are to be regarded as illustrative rather than restrictive.

[0288] Claim 1: A method executed by a computer, comprising: a computing system generating a snapshot of a source file system within a source region; a computing system performing a first inter-region replication and a second inter-region replication between the source file system within the source region and a target file system within a target region, wherein the source region and the target region are in different regions, and the method further comprises: a computing system receiving a snapshot deletion request to delete the snapshot within the source file system; a computing system determining a timing of the snapshot deletion request within the source file system; a computing system performing a first operation according to a determination that the timing of the snapshot deletion request is during the first inter-region replication; a computing system performing a second operation according to a determination that the timing of the snapshot deletion request is between the first inter-region replication and the second inter-region replication.

[0289] Item 2: The method according to Item 1, wherein the first operation includes the source file system holding the snapshot deletion request until the completion of the first inter-region replication.

[0290] Item 3: The method according to Item 1 or Item 2, wherein holding the snapshot deletion request includes the source file system storing the metadata information of the snapshot in a database to communicate with the target file system.

[0291] Item 4: The method according to any one of Items 1 to 3, wherein holding the snapshot deletion request includes the source file system transferring the snapshot to the target file system and completing the first inter-region replication, and the target file system deleting the snapshot at the end of the second inter-region replication.

[0292] Item 5: The method according to any one of Items 1 to 4, wherein the second operation includes the source file system deleting the snapshot without transferring the snapshot to the target file system.

[0293] Item 6: The method according to any one of Items 1 to 5, further including determining a second timing for generating a snapshot of the source file system.

[0294] Item 7: The method according to Item 6, further including transferring the generated snapshot to the target file system during the second inter-region replication according to the determination that the second timing of snapshot generation is during the first inter-region replication and according to the determination that the timing of the snapshot deletion request is after the second inter-region replication.

[0295] Item 8: A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: the computing system generating a snapshot of the source file system within the source region; the computing system performing a first inter-region replication and a second inter-region replication between the source file system within the source region and the target file system within the target region, the source region and the target region being in different regions, and the operations further including: the computing system receiving a snapshot deletion request to delete the snapshot within the source file system; the computing system determining the timing of the snapshot deletion request within the source file system; the computing system performing a first operation in accordance with determining that the timing of the snapshot deletion request is during the first inter-region replication; the computing system performing a second operation in accordance with determining that the timing of the snapshot deletion request is between the first inter-region replication and the second inter-region replication.

[0296] Item 9: The non-transitory computer-readable medium according to Item 8, wherein the first operation includes the source file system holding the snapshot deletion request until the end of the first inter-region replication.

[0297] Item 10: The non-transitory computer-readable medium according to Item 8 or 9, wherein holding the snapshot deletion request includes storing, by the source file system, the metadata information of the snapshot in a database to communicate with the target file system.

[0298] Item 11: The non-transitory computer-readable medium according to any one of Items 8 to 10, wherein holding the snapshot deletion request includes transferring, by the source file system, the snapshot to the target file system and completing the first inter-region replication; and deleting, by the target file system, the snapshot at the end of the second inter-region replication.

[0299] Item 12: The non-transitory computer-readable medium according to any one of Items 8 to 11, wherein the second operation includes deleting, by the source file system, the snapshot without transferring the snapshot to the target file system.

[0300] Item 13: The non-transitory computer-readable medium according to any one of Items 8 to 12, wherein the operation further includes determining a second timing for generating a snapshot of the source file system.

[0301] Item 14: The non-transitory computer-readable medium according to Item 13, wherein the operation further includes transferring, during the second inter-region replication, the generated snapshot to the target file system according to the determination that the second timing of snapshot generation is during the first inter-region replication and according to the determination that the timing of the snapshot deletion request is after the second inter-region replication.

[0302] Item 15: A system, One or more processors and, one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to, create a snapshot of a source file system within a source region and, perform a first inter-region replication and a second inter-region replication between the source file system within the source region and a target file system within a target region, the source region and the target region being in different regions, and the computer-executable instructions further cause the system to, receive a snapshot deletion request to delete the snapshot within the source file system, determine a timing of the snapshot deletion request within the source file system, perform a first operation in accordance with a determination that the timing of the snapshot deletion request is during the first inter-region replication, and, perform a second operation in accordance with a determination that the timing of the snapshot deletion request is between the first inter-region replication and the second inter-region replication.

[0303] Clause 16: The system according to Clause 15, wherein the first operation includes retaining the snapshot deletion request by the source file system until the end of the first inter-region replication.

[0304] Clause 17: The system according to Clause 15 or Clause 16, wherein retaining the snapshot deletion request includes, transferring the snapshot to the target file system by the source file system and completing the first inter-region replication, Including deleting a snapshot at the end of the second inter-region replication by the target file system.

[0305] Clause 18: The system according to any one of Clauses 1 to 17, wherein the second operation includes deleting the snapshot by the source file system without transferring the snapshot to the target file system.

[0306] Clause 19: The system according to any one of Clauses 1 to 18, wherein the system is further caused to determine a second timing for generating a snapshot of the source file system.

[0307] Clause 20: The system according to Clause 19, wherein the system causes the generated snapshot to be transferred to the target file system during the second inter-region replication according to the determination that the second timing of snapshot generation is during the first inter-region replication and according to the determination that the timing of the snapshot deletion request is after the second inter-region replication.

Claims

1. A method performed by a computer, The computing system generates a first snapshot and a second snapshot in the source file system within the source region, The computing system includes assigning a first origin identification information to the first snapshot and assigning a second origin identification information to the second snapshot within the source file system, wherein the first origin identification information is unique among all snapshots in all regions, and the second origin identification information is unique among all snapshots in all regions, and the method further includes The computing system includes receiving a request to perform replication between the source file system in the source region and the target file system in the target region, wherein the source region and the target region are in different regions, and the method further includes The computing system, in response to the request, compares the first origin identification information in the source file system with the origin identification information of an existing snapshot in the target region, The computing system identifies a matching snapshot having the first origin identification information within the target region, at least in part, based on the comparison, and uses it as the base snapshot for the replication. A method performed by a computer, comprising the computing system performing the replication using the difference between the second snapshot and the base snapshot in the source file system.

2. A method performed by a computer according to claim 1, further comprising selecting the matching snapshot as the base snapshot in at least in response to the fact that the matching snapshot having the first origin identification information in the target region is located in the target file system.

3. The method performed by the computer according to claim 1, wherein the target region includes a non-target file system that includes a snapshot associated with the first provenance identification information.

4. In response to the fact that the matching snapshot having the first origin identification information within the target region does not exist within the target file system, perform an intra-region copy of the matching snapshot having the first origin identification information from the non-target file system to the target file system, The method performed by a computer according to claim 3, further comprising selecting an in-region copy of the matching snapshot in the target file system as the base snapshot.

5. The method performed by a computer according to claim 4, wherein the intraregion copy of the matching snapshot in the target file system has the same first origin identification information as the matching snapshot in the non-target file system, but has different resource identification information.

6. A method performed by a computer according to claim 1, further comprising selecting the first snapshot having the first origin identification information in the source file system as the base snapshot, in at least in response to the fact that no matching snapshot having the first origin identification information is found within the target region.

7. The method performed by a computer according to claim 1, further comprising performing an interregional copy of the first snapshot having the first origin identification information from the source file system to the target file system before generating the difference between the second snapshot and the base snapshot in the source file system.

8. A program for causing one or more processors to perform the method according to any one of claims 1 to 7.

9. It is a system, One or more processors, A computer comprises one or more computer-readable media storing executable instructions, and when an instruction is executed by the one or more processors, the system... The first and second snapshots are generated on the source file system within the source region, The system is instructed to assign a first origin identifier to the first snapshot and a second origin identifier to the second snapshot within the source file system, wherein the first origin identifier is unique among all snapshots in all regions, and the second origin identifier is unique among all snapshots in all regions, and the instruction further instructs the system to The system receives a request to perform replication between the source file system in the source region and the target file system in the target region, wherein the source region and the target region are in different regions, and the instruction further to the system In response to the above request, the first origin identification information in the source file system is compared with the origin identification information of an existing snapshot in the target region, Based at least in part on the comparison, identify a matching snapshot having the first origin identification information within the target region and use it as the base snapshot for the replication. A system that further performs replication between the source file system and the target file system by using the difference between the second snapshot and the base snapshot in the source file system.

10. The system according to claim 9, wherein the system is further configured to select the matching snapshot as the base snapshot in response to at least the presence of the matching snapshot having the first origin identification information within the target region in the target file system.

11. The system according to claim 9 or 10, wherein the target region includes a non-target file system that includes a snapshot associated with the first provenance identification information.

12. The aforementioned system further, In response to the fact that the matching snapshot having the first origin identification information within the target region does not exist in the target file system, an intra-region copy of the matching snapshot having the first origin identification information from the non-target file system to the target file system is performed. The system according to claim 11, wherein the system is configured to select an in-region copy of the matching snapshot in the target file system as the base snapshot.

13. The system according to claim 9 or 10, wherein the system is further configured to select the first snapshot having the first origin identification information in the source file system as the base snapshot, at least in response to the fact that no matching snapshot having the first origin identification information is found within the target region.

14. The system according to claim 13, wherein the system is further made to perform an interregional copy of the first snapshot having the first origin identification information from the source file system to the target file system before generating the difference between the second snapshot and the base snapshot in the source file system.

Images