Delta transfer of a scalable, secure cross-region and optimized file system for cloud scale

Abstract

This specification discloses new techniques for end-to-end file storage replication and security between file systems in different cloud infrastructure regions. In one embodiment, a file storage service generates differences between snapshots in a source file system and, upon disaster recovery, transfers the differences and associated data via high-throughput object storage to recreate new snapshots in a target file system located in a different region. The file storage service utilizes new techniques to achieve scalable, reliable, and restartable end-to-end replication. New techniques for ensuring secure transfer and consistency of information during end-to-end replication are also described.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit and priority under 35 U.S.C. 119(e) of U.S. Provisional Patent Application No. 63 / 352,992, filed on June 16, 2022; U.S. Provisional Patent Application No. 63 / 357,526, filed on June 30, 2022; U.S. Provisional Patent Application No. 63 / 412,243, filed on September 30, 2022; and U.S. Provisional Patent Application No. 63 / 378,486, filed on October 5, 2022, and claims priority to U.S. Non - Provisional Patent Application No. 17 / 991,688, entitled "SCALABLE AND SECURE CROSS REGION AND OPTIMIZED FILE SYSTEM DELTA TRANSFER FOR CLOUD SCALE", filed on November 21, 2022. The disclosures of these applications are hereby incorporated by reference in their entireties for all purposes.

[0002] Field The present disclosure generally relates to file systems. More particularly, but not by way of limitation, techniques for end - to - end file storage replication and security between file systems within different cloud infrastructure regions are described.

Background Art

[0003] Background Today, enterprise businesses place important data in file systems and need to protect important data from unplanned power outages and operational incidents and ensure business continuity because power outages always have a large impact when they occur. Therefore, disaster recovery and business continuity are becoming increasingly important.

Summary of the Invention

[0004] Brief Summary This disclosure generally relates to file systems. More particularly, techniques for end-to-end file storage replication and security between file systems within different cloud infrastructure regions are disclosed. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like. **Means for Solving the Problem**

[0005] In one embodiment, a technique including a method is provided, the method including generating a snapshot difference between a first snapshot and a second snapshot of a source file system located in a source region, the snapshot difference being the difference between the first snapshot and the second snapshot, the second snapshot being created later than the first snapshot, the method further including encrypting the snapshot difference using a session encryption key valid during a session, and uploading the encrypted snapshot difference from the source file system to a first object storage located in a target region, the source region and the target region being different regions, the method further including detecting at least a portion of the encrypted snapshot difference reaching the first object storage and, simultaneously before the source file system completes uploading all of the snapshot difference, downloading the encrypted snapshot difference from the first object storage to a target file system, decrypting the snapshot difference encrypted using the session encryption key by the target file system, and applying the decrypted snapshot difference to a third snapshot of the target file system to create a fourth snapshot of the target file system, the fourth snapshot being a replica of the second snapshot.

[0006] In yet another embodiment, the session starts when a second snapshot is created in the source file system and ends when a fourth snapshot is created in the target file system.

[0007] In yet another embodiment, the differences between the first snapshot and the second snapshot include the keys and values of a binary tree (B-tree) and the associated file data.

[0008] In yet another embodiment, the method further includes transferring a session encryption key from the source file system to the target file system via a protected encryption process.

[0009] In yet another embodiment, uploading the encrypted snapshot difference includes uploading a first portion of the encrypted snapshot difference by a first upload thread in the source file system and uploading a second portion of the encrypted snapshot difference by a second upload thread in the source file system, and the first upload thread and the second upload thread operate independently.

[0010] In yet another embodiment, the method further includes, when a failure of the first upload thread in the source file system is detected, the third upload thread in the source file system taking over from the first upload thread and continuing the upload process.

[0011] In yet another embodiment, downloading the encrypted snapshot difference includes downloading a first portion of the encrypted snapshot difference by a first download thread in the target file system and downloading a second portion of the encrypted snapshot difference by a second download thread in the target file system.

[0012] In yet another embodiment, the method further includes reversing the roles of the source file system and the target file system.

[0013] In yet another embodiment, reversing the roles includes uploading a new snapshot delta from the target file system to a second object storage located in the source region, and downloading the new snapshot delta from the second object storage to the source file system.

[0014] In yet another embodiment, generating the snapshot delta and uploading the encrypted snapshot delta are in different pipeline stages.

[0015] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the one or more methods disclosed herein.

[0016] In various embodiments, the non-transitory computer-readable medium stores computer-executable instructions that, when executed by one or more processors, cause one or more processors of a computer system to perform one or more of the methods disclosed herein.

[0017] In various embodiments, a computer program product includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0018] The techniques described above and below can be implemented in a number of ways and in a number of situations. Referring to the following figures, which are described in more detail below, a number of exemplary implementations and situations are provided. However, the following implementations and situations are only a part of many implementations and situations.

[0019] The features, embodiments, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

[0021] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of an embodiment. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and the description are not intended to be restrictive. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0022] Explanation of Terms in a Certain Embodiment In a certain embodiment, "Recovery Time Objective" (RTO) refers to the period of time after a failure occurs within the availability domain (AD) of the primary (or source) region, whether the failure is planned or unplanned, during which a user needs to make the replication available within the secondary (or target) region.

[0023] In a certain embodiment, "Recovery Point Objective" (RPO) refers to the maximum allowable range with respect to the time of data loss between a failure in the primary region (usually due to an unplanned outage) and the availability of the secondary region.

[0024] In certain embodiments, a "replicator" may refer to a component (e.g., a virtual machine (VM)) within the data plane of a file system that uploads differences to a remote object store (i.e., an object storage service) when the component is located in a source region, or downloads differences from the object storage for applying the differences when the component is located in a target region. The replicator is formed as a fleet (i.e., a plurality of VMs or replicator threads) called a replicator fleet and can execute the inter-region (or cross-region) replication process (e.g., uploading differences to the target region) in parallel.

[0025] In certain embodiments, a "delta generator" (DG) may refer to a component within the data plane of a file system that extracts the differences (i.e., changes) between the keys and values of two snapshots when the component is located in a source region, or applies differences to the latest snapshot within the B-tree of the file system when the component is located in a target region. The delta generator in the source region can use multiple threads (referred to as delta generator threads or range threads for multiple split B-tree key ranges) to execute the extraction of differences (or B-tree traversal) in parallel. The delta generator in the target region can use multiple threads to apply the downloaded differences to the latest snapshot in parallel.

[0026] For the purposes of the present disclosure, in certain embodiments, a "shared database" (SDB) may refer to a key-value store that components (e.g., a replicator fleet) within both the control plane and the data plane of a file system can read from and write to in order to communicate with each other. In certain embodiments, the SDB may be part of a B-tree.

[0027] In some embodiments, a "file system communicator" (FSC) may refer to a file manager layer that runs on a storage node within the data plane of a file system. This service is useful for file creation requests, deletion requests, read requests, and write requests, and it works with an FNS server (e.g., Orca) to service I / O to clients. The replicator fleet can communicate with a number of storage nodes, thereby distributing file system data read / write operations across the storage nodes.

[0028] In some embodiments, a "blob" may refer to a data type for storing information (e.g., a formatted binary file) in a database. Blobs are generated during replication by a source region and uploaded to an object store (i.e., object storage) within a target region. A blob may include a binary tree (B-tree) key and value as well as file data. A blob within an object store is called an object. The B-tree key and value pairs and the data associated with them are packed together into the blob that is uploaded to the object store within the target region.

[0029] In certain embodiments, a "manifest" may refer to information transmitted by a file system within a source region (referred to herein as the source file system) to a file system within a target region (referred to herein as the target file system) to facilitate the inter-region replication process. There are two types of manifest files: a master manifest and a checkpoint manifest. A range manifest file (or master manifest file) is created by the source file system at the start of the replication process and describes information (e.g., B-tree key ranges) required by the target file system. A checkpoint manifest file is created after a checkpoint within the source file system and notifies the target file system of the number of blobs included in the checkpoint and uploaded to the object store. In response, the target file system can then download that number of blobs.

[0030] In certain embodiments, a "difference" may refer to the differences identified between two specific snapshots after a replicator has recursively visited all nodes of a B-tree (also referred to herein as scanning the B-tree). A difference generator identifies key-value pairs of the B-tree with respect to the differences, traverses the B-tree nodes, and retrieves the file data associated with the B-tree keys. The difference between two snapshots may include multiple blobs. The term "difference" may include blobs and manifests when used in the context of uploading information by the source file system to the object store and downloading by the target file system from the object store.

[0031] In certain embodiments, an "object" may refer to a partial set of information representing the totality of differences during a region - to - region replication cycle, and is stored in an object store. An object can be on the order of several megabytes in size and is stored at a specific location within a bucket of the object store. An object can contain many differences (i.e., blobs and manifests). A blob that is uploaded and stored in the object store is called an object.

[0032] In certain embodiments, a "bucket" may refer to a container that stores objects in compartments within an object storage namespace (tenancy). In the present disclosure, a bucket is used by a source replicator to store differences protected using server - side encryption (SSE), and is also used by a target replicator to download changes and apply them to a snapshot.

[0033] In certain embodiments, "difference application" may refer to the process by which a target file system applies the differences downloaded to the latest snapshot to create a new snapshot. Difference application can include analyzing a manifest file, applying snapshot metadata, inserting B - tree keys and values into a B - tree, and storing the data associated with the B - tree keys (i.e., the file data or the data portion of a blob) in local storage. Snapshot metadata is created and applied at the start of a replication cycle.

[0034] In certain embodiments, a "region" may refer to a logical abstraction corresponding to a geographical area. Each region can include one or more connected data centers. A region is independent of other regions and can be separated by vast distances.

[0035] Introduction This disclosure generally relates to file systems. More particularly, this document discloses new techniques for end-to-end file storage replication and security between file systems within different cloud infrastructure regions. In certain embodiments, a file storage service generates differences between snapshots within a source file system and, upon disaster recovery, transfers the differences and associated data via high-throughput object storage to recreate new snapshots in a target file system located in a different region. The file storage service utilizes new techniques to achieve scalable, reliable, and restartable end-to-end replication. New techniques for ensuring secure transfer and consistency of information during end-to-end replication are also described.

[0036] Current disaster recovery practices can include taking periodic snapshots and resynchronizing those snapshots to another file system within a different availability domain (AD) or region. A region is a geographically located group of ADs of a cloud service provider (e.g., Oracle Cloud), independent from all other regions and having no dependencies between regions. An AD can be a physically separated data center that does not share resources (e.g., power or cooling, etc.) with another AD within the same region for fault tolerance. ADs within a region can be interconnected with a network. Resynchronization is manageable and maintained by a customer but lacks a user interface for displaying progress and is a slow serialized process that is not easy to manage as data grows over time.

[0037] Therefore, different approaches are needed to address these and other issues. The file storage replication of a cloud service provider (e.g., Oracle Cloud Infrastructure (OCI)) disclosed in this disclosure is based on incremental snapshots and propagates the differences in changed data from the primary AD within a region to a secondary AD within the same or a different region, providing a consistent point-in-time view of the entire file system. As used herein, the primary site (or source side) may refer to the location where the file system is located and where the replication process for disaster recovery is initiated (e.g., an AD or region). The secondary site (or target side) may refer to the location where the file system receives information from the file system within the primary site during the replication process and becomes the new operational file system after disaster recovery (e.g., an AD or region). The file system located at the primary site is called the source file system, and the file system located at the secondary site is called the target file system. Therefore, the primary site, source side, source region, primary file system, or source file system (referring to one of the file systems on the source side) may be used interchangeably. Similarly, the secondary site, target side, target region, secondary file system, or target file system (referring to one of the file systems on the target side) may be used interchangeably.

[0038] The file storage service (FSS) of the present disclosure supports complete disaster recovery for failover or failback with minimal management effort. Failover is a series of actions to make the secondary site / target site the primary / source (i.e., start providing services for the workload), which may include planned failover and / or unplanned failover. Planned failover (sometimes called planned migration) is initiated by the user to perform a planned failover from the source side (e.g., source region) to the target side (e.g., target region) without data loss. Unplanned failover occurs, for example, when the source side unexpectedly stops due to a disaster and the user needs to start using the target side because the source side is lost. Failback is to restore the primary side / source side to become the primary / source again before the failover. Failback may occur when the user wants to reuse the source side as the primary AD by reversing the failover process after the planned failover or unplanned failover and the trigger event (e.g., power outage) ends. The user can resume either from the last point in time on the source side before the trigger event or from the latest changes on the target side. The replication process described in the present disclosure can maintain the identity of the file system after round-trip replication. In other words, the source file system can resume providing services for the workload again after performing a failover and then a failback.

[0039] The techniques disclosed in this disclosure (e.g., methods, computer-readable media, and systems) use consistent snapshot information to replicate differences between snapshots from a source region to multiple remote (or target) regions, and then scan (or recursively visit) all keys and values within one or more file trees (e.g., B-trees) of a source file system (referred to herein as "scanning the B-tree" or "scanning keys") to construct consistent information (e.g., differences or discrepancies between the keys and values of two snapshots created at different times), including region-to-region replication of file system data and / or metadata. The constructed consistent information is in blob form and transferred to the remote side (e.g., the target region) using an object interface, such as an object store (described later), so that the target file system on the remote side can immediately detect the information transferred via the object interface and start downloading and applying it. This process is implemented using a control plane and can be scaled to thousands of file systems and hundreds of replication machines. Both the source file system and the target file system can operate simultaneously and asynchronously. Operating simultaneously means that the data upload process by the source file system and the data download process by the target file system can occur concurrently. Operating asynchronously means that the source file system and the target file system can each operate at their own pace without waiting for each other at all stages, e.g., with different start times, end times, processing speeds, etc.

[0040] In one embodiment, multiple file systems may exist in the same region and are represented by the same B-tree. Each of these file systems within the same region can be replicated independently across regions. For example, file system A may have a set of parallel execution replicator threads that scan the B-tree to perform replication of file system A. File system B, represented by the same B-tree, may have another set of such parallel execution replicator threads that scan the same B-tree to perform replication of file system B.

[0041] Regarding security, cross-region replication is completely secure. Information is transferred securely and applied securely. The disclosed technology provides separation between the source region and the target region such that keys are not shared between the two without being encrypted. Thus, if the source key is involved, the target is not affected. Further, the disclosed technology includes ways to read keys, convert those keys into a certain format, and upload and download those keys securely. Since different keys are created and used in different regions, separate keys are created at the target and applied to the information with a target-centric security mechanism. For example, FSS generates a session key that is only valid during one replication cycle or session to encrypt data uploaded from the source region to the object store and decrypt data downloaded from the object store to the target region. Separate keys are used locally within the source region and the target region.

[0042] In the disclosed technology, each upload process and download process through the object store during replication has different pipeline stages. For example, the upload process has multiple pipeline stages including scanning the B-tree to generate differences, accessing storage I / O, and uploading data (or blobs) to the object store. The download process has multiple pipeline stages including downloading data, applying differences to the snapshot, and storing the data in storage. Each of these pipelines also includes parallel processing threads to improve the throughput and performance of the replication process. Further, the parallel processing threads can take over from a failed processing thread and resume the replication process from the point of failure without restarting from the beginning. Thus, the replication process is highly scalable and reliable.

[0043] Figure 1 shows an exemplary concept of the target recovery point in time (RPO) and target recovery time (RTO) for an unplanned failover according to an embodiment. The RPO is the maximum allowable range of data loss between a primary site failure and the availability of a secondary site (usually specified in minutes). As shown in Figure 1, the primary site A102 encounters an unplanned incident at time 110 and triggers the failover replication process by copying the latest snapshot and its delta to the secondary site B104. The information first copied reaches the secondary site B104 at time 112. The primary site A102 completes the copy of information to the secondary site B104 at time 114, and the secondary site B104 completes the replication process at time 116. Thus, the secondary site B104 becomes fully operational at time 116. As a result, the user's data is not accessible within the primary site A110 from point 110 until the point 116 where the data becomes available again. Thus, the RPO is the time between point 110 and point 116. For example, if there is data equivalent to 10 minutes that the user is not interested in, the RPO is 10 minutes. If the data loss exceeds 10 minutes, the RPO is not met. An RPO of 0 means synchronous replication.

[0044] RTO is the time it takes for the secondary to become fully operational after a failure, so that the user can access the data again (usually specified in minutes). RTO is considered from the perspective of the secondary site. Referring again to Figure 1, the primary site A102 starts the failover replication process at time 120. However, the secondary site B104 remains operational until time 122 when it recognizes the incident (or power outage) at the primary site A102. Therefore, the secondary site B104 stops its service at time 122. The secondary site B104 becomes fully operational at time 126 using the same failover replication process as described for RPO. Therefore, RTO is the time between 122 and 126. Here, the secondary site B104 can take over the role of the primary site. However, for customers using the primary site A102, the service loss is between times 120 and 126.

[0045] The primary (or source) site is where the action is taking place, and the secondary (or target) site is inactive and not available until a disaster occurs. However, customers may be provided with a point in time to continue using for test-related activities at the secondary site. This relates to how the customer sets up replication, how the customer can start using the target if something goes wrong, and how the customer can return to the source after the source fails over.

[0046] End-to-end inter-region replication architecture Figure 2 is a simplified block diagram showing an architecture for inter-region remote replication according to an embodiment. In Figure 2, the end-to-end replication architecture shown includes two regions: a source region 290 and a target region 292. Each region may include one or more file systems. In one embodiment, the end-to-end replication architecture includes data planes 202 and 212, control planes (only control APIs 208a-n and 218a-n are shown), local storages 204 and 214, an object store 260, and a key management service (KMS) 250 for both the source region 290 and the target region 292. Figure 2 shows only one file system 280 in the source region 290 and one file system 282 in the target region 292 for simplicity. If there are two or more file systems in one region, the same replication architecture is applied to each pair of the source file system and the target file system. File systems within a region may share resources. For example, the KMS 250, the object store 260, and certain resources in the data plane may be shared by multiple file systems within the same region depending on the implementation.

[0047] The data plane within the architecture includes local storage nodes 204a - n and 214a - n, as well as replicators (or replicator fleets) 206a - n and 216a - n. The control API hosts within each region perform all orchestration between different regions. The FSS receives a request from a customer to set up replication between a source file system 280 and a target file system 282 where the customer's data will be moved. The control plane 208 obtains the request, performs resource allocation, and notifies the replicator fleet 206a - n within the source data plane 202 to start uploading data 230a from different snapshots to the object storage 260 (or sometimes called that only the differences are uploaded). An API is available to assist the customer in setting the target time and recovery time objective (RTO) for replication. The replication model disclosed in this disclosure is a "push - based" model based on snapshot differences, that is, the source region initiates the replication.

[0048] As used herein, the data 230a and 230b transferred between the source file system 280 and the target file system 282 are general terms and may include an initial snapshot, keys and values of different B - trees between two snapshots, file data (e.g., fmap), snapshot metadata (i.e., a set of B - tree keys of snapshots reflecting various snapshots obtained within the source file system), and other information (e.g., manifest files) that helps facilitate the replication process.

[0049] Regarding the data plane of the inter-region replication architecture, the replicator is a component within the data plane of the file system. The replicator performs either differential generation or differential application on the file system according to the region where the file system is located. For example, the replicator fleet 206 within the file system 280 of the source region performs the generation and replication of the differential 230a. The replicator fleet 216 within the file system 282 of the target region downloads the differentials 230b and applies them to the latest snapshot within the file system 282 of the target region. The file system 282 of the target region can also use the control plane and workflow to guarantee end-to-end transfer.

[0050] All incremental operations are based on snapshots, which are existing resources within file storage as a service. A snapshot is a point in time, data point, or image of what is happening within the file system and is executed periodically within the file system 280 of the source region. In the very first replication (e.g., where replication has not been obtained before), FSS obtains a base snapshot, which is a snapshot of all the content of the source file system, and transfers all that content to the target system. In other words, the replicator reads from the storage layer of that specific file system and stores all the data in the object storage bucket.

[0051] After the data plane 202 of the source file system 280 uploads all data 230a to the object storage (or object store) 260, the source-side control plane 208 notifies the target-side control plane 218 that there is new work to be done on the target side, and then this notification is relayed to the target-side replicator. Thereafter, the target-side replicators 216a - n begin to download objects (e.g., initial snapshots and deltas) from the object storage bucket 260 and apply the deltas captured on the source side.

[0052] For a base copy (e.g., the entire file system content up to a point in time ranging from the past 5 days to 5 years), the upload process may take time. To assist in meeting service level goals regarding time and performance, the source system 280 can take replication snapshots at specific intervals such as one hour. The source side 280 can then transfer all data within that one hour to the target side 282 and take new snapshots every hour. If there is some cache with many changes, the replication can be set to a shorter replication interval.

[0053] To illustrate the above, consider a situation where a first snapshot is created on a file system within a source region (referred to as the source file system). Replication is performed periodically, and thus the first snapshot is replicated to a file system within a target region (referred to as the target file system). Subsequently, when some update is performed within the source file system, a second snapshot is created. If an unplanned power outage occurs after the second snapshot has been created, the source file system attempts to replicate the second snapshot to the target file system. During failover, the source file system may well identify the difference (i.e., the delta) between the first snapshot and the second snapshot, which includes the keys and values of the B-tree, and the file data associated therewith, within the B-tree representing both the first snapshot and the second snapshot. Next, the deltas 230a and 230b (and other related information, e.g., snapshot metadata) are transferred from the source file system to the target file system via the object store 260 within the target region, and the target file system re-creates a replica of the second snapshot by applying the delta to the first snapshot previously established within the target region. When a replica of the second snapshot is created in the target file system, the failover replication process is complete and the target file system is ready to operate.

[0054] Regarding the control plane and its Application Programming Interfaces (APIs), the control plane provides instructions for the data plane that includes a replicator as an executor to execute instructions. Storage (204 and 214) and replicator fleets (206 and 216) are both within the data plane. The control plane is not shown in Figure 2. As used herein, a "cycle" may refer to a period that starts when the source file system 280 begins to transfer data 230a to the target file system 282 and ends when the target file system 282 has received all of the data 230b and completed the application of the received data. Data 230a - b is captured on the source side and then applied on the target side. When all changes on the target side are applied to the cycle, the source file system 280 takes another snapshot and starts another cycle.

[0055] The control APIs (208a - n and 218a - n) are a set of hosts within the overall architecture of the control plane and execute the configuration of the file system. The control APIs are responsible for communicating state information between different regions. State machines that track various state activities within a region, such as the progress of a job, the location of keys, and future tasks to be executed, are distributed across multiple regions. All of this information is stored in the control plane of each region and communicated between regions via the control APIs. In other words, the state information relates to the details of the life cycle, the details of the differences, and the life cycle of the resources. The state machine can also be useful for tracking the progress of replication and cooperating with the data plane to estimate the time taken for replication. Therefore, the state machine can provide the user with status regarding whether the replication is proceeding as planned and the normality of the job.

[0056] Furthermore, the communication between the control APIs (208a - n) of the source file system 280 and the control APIs (218a - n) of the target file system 218 in a different region includes the transfer of snapshots and metadata for creating an accurate copy from the source to the target. For example, when a customer periodically takes snapshots within the source file system, the control plane can ensure that snapshots of the same user, including metadata tracking, transfer, and recreation, are created in the target file system.

[0057] The object store 260 in FIG. 2 (also referred to as an "object" in this specification) is an object storage service (e.g., Oracle's object storage service) that enables reading blobs and writing files for archival purposes. The advantages of using an object store are, firstly, it is easy to configure, secondly, it is easy to stream data to the object store, and thirdly, it has the advantage of security streaming as a reliable repository for maintaining information, all of which are because there is no network loss, data can be immediately downloaded, and it exists permanently. Direct communication between replicators within the source region and the target region is possible, but direct communication requires the configuration of an inter-region network, which is not scalable and difficult to manage.

[0058] For example, if there is a large amount of data being moved from a source to a target, the source can upload the data to the object store 260, and the target 282 does not need to wait for all the information uploaded to the object store 260 to start downloading. Thus, both the source 280 and the target 282 can operate continuously and simultaneously. The use of the object store enables the system to scale and achieve higher throughput. Further, the key management service (KMS) 250 can control access to the object store 260 to ensure security. In other words, the source tries to move the data out of the source region as fast as possible and hold the data somewhere so that the data is not lost before it can be applied to the target.

[0059] Compared to using a network pipe with packet loss and recovery issues, the use of the object store 260 between the source region and the target region enables continuous data streaming where hundreds of file systems can be written from the source region to the object store, and at the same time, the target region can apply hundreds of files simultaneously. Thus, data streaming via the object store can achieve high throughput. Further, both the source region and the target region can operate at their own speeds for uploading and downloading.

[0060] Whenever a user changes some data in the source file system 280, a snapshot is taken and the difference before and after the change is updated. These changes are accumulated in the source file system 280 and can be streamed to the object store 260. The target file system 282 can detect that the data is available in the object store 260 and immediately download the changes and apply them to its file system. In some embodiments, only the differences are uploaded to the object storage after the base snapshot.

[0061] In some embodiments, the replicator can communicate with many different regions (e.g., from Phoenix to Ashburn and further to other remote regions), and the file system can manage many different endpoints on the replicator. Each replicator 206 within the source file system 280 can maintain a cache of these object storage endpoints, and further, in cooperation with the KMS 250, generate a transfer key (e.g., a session key) for encrypting the data address of the data in the object storage 260 (e.g., server-side encryption or SSE) to protect the data stored in the bucket. There is one master bucket for each AD within the target region. A bucket is a container that stores objects in a compartment within the object storage namespace (tenancy). Since all remote clients can communicate with the bucket and write information in a specific format, the information of each file system can be uniquely identified, preventing the mixing of data from different customers or file systems.

[0062] The object store 260 is a high-throughput system, and the techniques disclosed in this disclosure can utilize the object store. In one embodiment, the replication process includes multiple pipeline stages, a B-tree scan within the source file system 280, storage IO access, data upload to the object store 260, data download from the object store 260, and differential application within the target file system 282. Each stage includes parallel processing threads that participate in improving the performance of data streaming from the source region 290 to the target region 292 via the object store 260.

[0063] In one embodiment, each file system within the source region may include a set of replicator threads 206a - n that are executed in parallel to upload the differences to the object store 260. Each file system within the target region may also include a set of replicator threads 216a - n that are executed in parallel to download the differences from the object store 260. Since both the source side and the target side operate asynchronously at the same time, the source can upload as fast as possible, while the target can start downloading after detecting that the differences are available in the object store. Thereafter, the target file system applies the differences to the latest snapshot and deletes the differences in the object store after the application. Thus, the FSS consumes little space in the object store, and the object store has a very high throughput (e.g., gigabyte - scale transfers).

[0064] In one embodiment, multiple threads are also executed in parallel for storage I / O access (e.g., DASD) 204a - n and 214a - n. Thus, all processes related to the replication process, including accessing storage, uploading the snapshot and data 230a from the source file system 280 to the object store 260, and downloading the snapshot and data 230b to the target file system 282, include multiple threads that are executed in parallel to perform data streaming.

[0065] File storage is a local service of the AD. When a file system is created, that file system is within a specific AD. When a customer transfers or replicates data from one file system to another file system in the same or a different region, artifact (also called manifest) transfer may need to be used.

[0066] As an alternative to using an object store to transfer data, a network connection between remote machines (e.g., between source and target replicator nodes) can be set up, and VCN peering can be used to use Classless Inter-Domain Routing (CIDR) for each region.

[0067] Referring again to Figure 2, the Key Management System (KMS) 250 provides security for replication and provides a storage service to a cloud service provider (e.g., OCI). In certain embodiments, the file systems 280 on the source (or primary) side and the target (or secondary) side use separate KMS keys, and key management is hierarchical. The reason for using separate keys is that if the source is compromised, an unauthorized actor cannot decrypt the target using the same key. FSS has a three-tier key architecture. Since the source and target use different keys during data transfer, the source first decrypts the data, re-encrypts it using an intermediate key, and then re-encrypts the data on the target side. FSS defines a session, and each session is one data cycle. A key for transferring data in that session is created. In other words, a new key is used for each new session. In other embodiments, a key can be used for two or more sessions (e.g., two or more data transfers) before creating another key. The key is not transferred via the object store 260, and the key is only available on the source side and is not visible from outside the source for security reasons.

[0068] The replication cycle (also called a session) is periodic. For example, the replicators (206a~n and 216a~n) perform replication once every hour. The cycle starts when a new snapshot is created on the source side 280 and ends when all the differences 230b have been applied to the target side 282 (i.e., the target has reached the DONE state). Each session is completed before another session starts. Thus, there is always only one session and no overlap between sessions.

[0069] Secret management (i.e., replication using the KMS) processes the transfer of confidential materials between the source (primary) file system 290 and the target (or secondary) file system 292 using the KMS250. The source file system 280 calculates the differences, reads the file data, and then decrypts the file data in cooperation with the key management service using the encryption key of the local file system. Next, the source file system 280 generates a session key (referred to as a delta encryption key (DEK)), encrypts it to become an encrypted session key (referred to as a delta transfer key (DTK)), and transfers the DTK to the target file system 282 via the respective control planes 208 and 218. The source file system 280 further encrypts the data 230a using the DEK and uploads the encrypted data 230a to the object store 260 via the Transport Layer Security (TLS) protocol. Next, the object store 260 uses server-side encryption (SSE) to ensure the security for the storage of the data (e.g., differences, manifests, and metadata) 230a.

[0070] The target file system 282 securely obtains the encrypted session key DTK via the control plane 218 (using HTTPS via inter-region API communication), decrypts the session key DTK via the KMS 250 to obtain the DEK, and places the DEK at a location within the target region 292. When a replication job is scheduled within the target file system 282, the DEK is provided to a replicator (one of the replication fleets 216a - n), and the replicator uses this key to decrypt the data (e.g., the delta including file data) 230b downloaded from the object store 260 for application and re-encrypts the file data using the local file system key.

[0071] Replication between the source file system 280 and the target file system 282 is a parallel process, and both the source file system 280 and the target file system 282 operate at their own paces. When the source side completes the upload (which can occur before the target download process), the source side cleans up the memory and removes all keys. The target, when it completes the application of the delta to the latest snapshot, similarly cleans up the memory and removes all keys. The FSS service also releases the KMS key. In other words, there are two copies of the session key, one within the source file system 280 and another within the target file system 282. Both copies are deleted at the end of each session, and a new session key is generated in the next replication cycle. This process ensures that the same key is not used for different purposes. Further, the session key is encrypted by the file system key, creating a double layer of protection. This is to ensure that only a specific file system can use this session key.

[0072] System Design Figure 3 is a simplified schematic diagram of components involved in inter-region remote replication according to an embodiment. In one embodiment, components called the differential generator (DG) 310 within the source region A 302 and 330 within the target region B 304 are part of the replicator fleet 318 and operate on thousands of storage nodes within the fleet. The replicator 318 within the source region A makes remote procedural calls (RPCs: Remote Procedural Call) to the differential generator 310 (e.g., obtaining a set of keys and values, locking a block, etc.), and collects the keys, values, and data pages of the B-tree from the direct-access storage device (DASD: Direct-Access Storage Device) 314, which is a replication storage service for accessing storage and is regarded as a data server. The DG 310 within the source region A is a helper to the replicator 318, divides the key range of the difference, and packs all the keys / values within a specific range into a blob to be returned to the replicator 318. There are a plurality of storage nodes 322 and 342 connected to the DASD 314 and 334 in both regions, and each node contains a large number of disks (e.g., 10TB or more).

[0073] In one embodiment, the file system communicators (FSC) 312 and 332 in both regions are metadata servers that help update the source file system for user updates to the system. The FSC 312 and 332 are used for file system communication, and the differential generator 310 is used for replication. Both the DG 310 and 330 and the FSC 312 and 332 are metadata servers. User traffic passes through the FSC 312 and 332 and the DASD 314 and 334, while replication traffic passes through the DG. In an alternative embodiment, the function of the FSC can be merged with the function of the DG.

[0074] In one embodiment, the shared databases (SDBs) 316 and 336 of both regions are key-value stores, and through these components, both the control plane and the data plane (e.g., the replicator fleet) can read and write for each other to communicate. The control planes 320 and 340 of both regions can place new jobs into queues within their respective shared databases 316 and 336, and the replicator fleets 318 and 338 continuously read the queues within the shared databases 316 and 336, and when the replicator fleets 318 and 338 detect a job request, they can initiate file system replication. In other words, the shared databases 316 and 336 are conduits between the replicator fleet and the control plane. Further, the shared databases 316 and 336 are resources distributed across different regions, and the IO traffic between the shared databases 316 and 336 should be minimized. Similarly, the IO traffic with the DASD needs to be minimized so as not to affect the user's performance. However, the replication process may be adjusted as it is a secondary service compared to the primary service.

[0075] The replicator fleet 318 within the source region A can cooperate with the DG310 to start scanning the B-tree in the file system within the source region A, collect keys and values, and convert those keys and values into flat files or blobs to be uploaded to the object store. Once the data blobs (including keys and values and the actual data) are uploaded, the target can immediately apply those data blobs without waiting for a large number of blobs to be present in the object store 360. The object store 360 is located in the target region B for disaster recovery reasons. The goal is to push from the source to the target region B as quickly as possible and keep the data safe.

[0076] Optimize space by utilizing lower-cost machines with smaller footprint, and schedule as many replications as possible while ensuring fair bandwidth allocation among those machines, there are a number of replicators to replicate thousands of file systems. Replicator fleets 318 and 338 in both regions are run on virtual machines that can be automatically scaled up and down to build the entire fleet for running replications. Replicators and replication services can dynamically adapt based on capacity to support each job. If the load on one replicator is high, another replicator can be selected to share the load. Different replicators in the fleet can balance the load among each other to ensure that jobs can continue and are not stopped due to overloading individual replicators.

[0077] High-level process flow FIG. 4 is a simplified flowchart showing steps executed during inter-region remote replication, according to an embodiment.

[0078] Step S1: When a customer sets up a replication, the customer provides a source (or primary) file system (A) 402, a target (or secondary) file system (B) 404, and an RPO. The file systems are uniquely identified by file system identification information (e.g., Oracle Cloud ID or OCID), which is a globally unique identifier for the file system. The data is stored in a file storage service (「FSS」) control plane database.

[0079] Step S2: The source (A) control plane (CP-A) 410 adjusts to periodically create system snapshots at regular intervals (less than RPO), and notifies the data plane (including the replicator / uploader 412) of the latest snapshot and the last snapshot successfully copied to the target (B) file system 404.

[0080] Step S3: CP-A 410 notifies the replicator 412 (or uploader), which is a component within the data plane, to copy the latest snapshot from the source file system to the target file system.

[0081] S3a: The replicator 412 within the source (A) scans the B-tree to calculate and generate the difference between two specific snapshots, i.e., the previous snapshot (e.g., the first snapshot) and the latest snapshot (e.g., the second snapshot). The existing key infrastructure is used to decrypt the file system data.

[0082] S3b: The data 414 containing the difference is encrypted using a valid session key (DEK) during one session and uploaded to the object store 430 within the target (B) region (this data can be compressed and / or deduplicated during the copy). This upload can be executed in parallel by multiple replicator threads 412.

[0083] Step S4: CP-A 410 notifies the target (B) control plane (CP-B) 450 of the completion of the data upload and subsequent upload. CP-A may transfer the DEK to CP-B.

[0084] Step S5: CP-B450 calls the target replicator B452 (or downloader) to apply the differences. Since the upload from the source file system and the download to the target file system can operate simultaneously, steps S4 and S5 do not need to occur consecutively. In other words, when step S5 starts, step S4 (e.g., CP-A notifies the completion of the upload) may occur.

[0085] S5a: Replicator B452 downloads the encrypted data 454 (including the differences) from the object store 430. The target file system (i.e., CP-B and replicator B) can start downloading data when it detects at least a part of the encrypted data reaching the object store 430 as indicated by the checkpoint CM file, before the source file system (i.e., CP-A and replicator A) completes uploading all the data, as indicated by the final CM file.

[0086] S5b: Replicator B452 uses the DEK (received from the source file system) to decrypt the data and applies these differences to the latest snapshot (e.g., the third snapshot) of the target file system (B) to create a new snapshot.

[0087] Step S6: After the difference application is completed, CP-A410 is notified of the new snapshot (e.g., the fourth snapshot) now available at the target (B). In some embodiments, the new snapshot (e.g., the fourth snapshot) in the target file system is a replica (i.e., copy) of the latest snapshot (e.g., the second snapshot) in the source file system.

[0088] Step 7: The inter-region remote replication process repeats from step S2 to step S6.

[0089] FIG. 5 is a simplified diagram showing a high-level concept of B-tree traversal according to an embodiment. The B-tree structure can be used within a file system. The difference generator traverses the B-tree and ensures the consistency of the traversal. In other words, the traversal confirms that the keys and values are as expected at the end of the traversal so that data corruption cannot occur, and captures all information between any two snapshots. The file system is a transactional file system that may be changed, and since another user may update the same transaction or data, the user needs to be aware of the changes and re-do the transaction.

[0090] Keys, values, and snapshots are immutable (i.e., they cannot be changed except that the garbage collector can remove them). As shown in Figure 5, there are many snapshots (Snapshot 1 to Snapshot N) in the file system. When the diff generator scans the B-tree keys (510 to 560) in the source file system, the garbage collector 580 may come in and clean up the keys of the snapshots considered garbage, so the snapshots may be removed. When the diff generator scans the B-tree keys, the diff generator needs to ensure that the keys associated with the remaining snapshots (e.g., keys not removed by the garbage collector) are copied. When keys, such as 540 and 550, are removed by the garbage collector 580, the B-tree page can be shrunk, for example, from 2 pages before garbage collection to 1 page after garbage collection. A way for the diff generator to ensure consistency when scanning B-tree keys is for the garbage collector 580 to confirm that it has not changed or deleted any keys in the page (or section between two snapshots) that the diff generator has just scanned (e.g., between two keys). Once consistency is confirmed, the diff generator collects the keys and sends them to the replicator for processing and uploading.

[0091] The B-tree key can indicate what has changed. The techniques disclosed in this disclosure can determine which B-tree keys are new and what has been updated between two snapshots. The diff generator can collect the metadata part, keys and values, and related data, and then send them to the target. The target can understand that the received information is within the range of two snapshots and applies to the target file system. The diff generator (or a thread of the diff generator) scans the section between two keys, checks its consistency, and then uses the last end key as the next start key for the next scan. This process is repeated until all keys are checked, and the diff generator collects related data each time the consistency is confirmed.

[0092] For example, when a file is changed within a file system (e.g., created, deleted, and then recreated), this process creates multiple versions of the corresponding file directory entry. During the replication process, the garbage collector may clean up (or remove) the version of the file directory entry corresponding to the deleted file, which may cause a consistency issue called a whiteout. A whiteout occurs when there is a mismatch between the source file system and the target file system, because the target file system may fail to reconstruct the original snapshot chain that includes the changed file. The disclosed technique can detect whiteout files (i.e., changed files affected by the garbage collector) during B-tree scanning, extract the version that is not affected by the changed file, and provide related information to the target file system within the same replication cycle to ensure the consistency between the source file system and the target file system by properly reconstructing the correct snapshot chain.

[0093] Figures 6A and 6B are diagrams showing the pipeline stages of inter-region replication according to an embodiment. The inter-region replication of the source file system disclosed in the present disclosure includes four pipeline stages, namely, the start of inter-region replication, the B-tree scan within the source file system (i.e., the differential generation pipeline stage), the storage IO access for retrieving data (i.e., the data read pipeline stage), and the data upload to the object store (i.e., the data upload pipeline stage), which are included within the source file system. The target file system includes four pipeline stages in a similar but reverse order, namely, the preparation for inter-region replication, the download of data from the object store, the application of the difference within the target file system, and the storage IO access for storing the data. Figure 6A shows the four pipeline stages within the source file system, and the same concept applies to the target file system as well. Figure 6B shows the processes involved in the pipeline stages and the interactions between components. These pipeline stages can all operate in parallel. Each pipeline stage operates independently and can pass information to the next pipeline stage when the processing at the current stage is completed. Each pipeline stage receives a portion of the total bandwidth and is guaranteed not to use more than necessary. In other words, resources are fairly allocated among all jobs. If no other jobs are operating within the system, the operating job can acquire as many resources as possible.

[0094] Threads within each pipeline stage also execute tasks (i.e., replication jobs) independently of each other in parallel (or simultaneously) within the same pipeline stage (i.e., if a thread fails, it does not affect other threads). Further, the tasks (or replication jobs) executed by threads in each pipeline stage are restartable, i.e., if a thread fails, a new thread can take over the failed thread and continue the original task from the last successful point.

[0095] In some embodiments, the B-tree scan can be performed using parallel processing threads within the source file system 280. The B-tree can be divided into a plurality of key ranges between the first key and the last key in the file system. The number of key ranges can be determined by the customer. For each file system, a plurality of (e.g., about 8 to 16) range threads can be used for the B-tree scan. One range thread can perform a B-tree scan of one key range, and all range threads operate in parallel simultaneously. The number of threads used varies depending on factors such as the size of the file system, resource availability, and bandwidth for balancing resource and traffic congestion. Usually, the number of key ranges is more than the number of available range threads for fully utilizing the range threads. Therefore, the B-tree scan is scalable and can be processed by simultaneous parallel scans (e.g., using multiple threads).

[0096] After the difference generator scans the pages, if some keys are missing and thus some keys are inconsistent, the system can remove the ongoing uncommitted transactions and return to the starting point for rescan. During the repetition of the B-tree scan due to the inconsistency, the difference generator can ignore the missing keys and their related data in order to minimize the amount of information to be processed or uploaded to the target side, as these related data are regarded as garbage and thus not collected. Therefore, the B-tree scan and data transfer can be made more efficient. Further, the difference generator does not need to wait for the garbage collector to remove the information to be deleted before scanning the B-tree keys. For example, keys have dependencies on each other. If a key or iNode points to a block that has been deleted by the garbage collector or should be deleted, the system (or the difference generator) can itself understand that a particular block is garbage and the difference generator does not need to carry it.

[0097] The differential generator usually makes no changes on the source side (e.g., does not delete keys or blocks of data considered as garbage), and simply does not copy them to the target side. The B-tree scanning process and garbage collection are asynchronous processes. For example, when the block of data pointed to by a key no longer exists, the file system can flag the key as garbage, notify that the key should not be changed (e.g., is immutable), and only the garbage collector can remove the key. The differential generator can continue scanning the next key without waiting for the garbage collector. In other words, the differential generator and the garbage collector can proceed at their own pace.

[0098] In FIG. 6A, when the source region starts an inter-region replication process that can include multiple file systems, the main threads 610a - n select a replication job (one job per file system). The main thread (e.g., 610a or 610 for later use) of the file system within the source region (i.e., the source file system) communicates with the differential generator 620 (shown in FIG. 6B) to obtain the number of key ranges requested by the customer and updates the corresponding record in the SDB 622. Once the main thread 610 of the source file system knows the number of key ranges required, it further creates a set of range threads 612a - n based on the number of key ranges required. These range threads 612a - n are executed by the differential generator 620. These range threads 612a - n initialize the GETKEYVAL buffer 640 (shown in FIG. 6B), update the checkpoint record 642 in the SDB 622 (shown in FIG. 6B), and perform storage I / O access 644 by interacting with the DASD I / O threads 614a - n.

[0099] In one embodiment, each main thread 610 is responsible for monitoring all range threads 612a - n that it creates. During replication, the main thread 610 may generate a master manifest file that outlines the entire replication. The range threads 612a - n generate a range manifest file that includes the number of key ranges (i.e., the subdivision of the entire replication), and then generate a checkpoint manifest (CM) file for each range to provide updates to the target file system regarding the number of blobs per checkpoint, where the checkpoints are created during B - tree traversal. One checkpoint is created by the range thread 612. When the main thread 610 determines that all range threads 612a - n are complete, it creates a final checkpoint manifest (CM) file that includes an end - of - file marker, and then uploads the CM file to the object store so that the target file system can understand the progress within the source file system. The CM file includes an overview of all individual ranges, such as the number of ranges, the final state of the checkpoint records, and other information.

[0100] Range threads 612a~n are used for parallel processing to significantly reduce the time of B-tree traversal of a large source file system. In one embodiment, the B-tree keys are divided into ranges of approximately equal size. One range thread can perform a B-tree traversal of one key range. The number of range threads 612a~n used varies depending on factors such as the size of the file system, resource availability, and bandwidth to balance resources, the amount of data generated, and traffic congestion. Usually, the number of key ranges is about two to four times more than the number of available range threads 612a~n to fully utilize the range threads. Each of the range threads 612a~n has a dedicated buffer (GETKEYVAL) 640 that contains jobs available for work. Each range thread 612 operates independently of other range threads and periodically updates checkpoint records 642 in the SDB622.

[0101] Range threads 612a~n may need to collect file data (e.g., FMAP) associated with the B-tree keys and request IO access 644 to storage when traversing the B-tree (i.e., when recursively visiting all nodes of the B-tree). These IO requests are queued by each range thread 612 so that the DASD IO threads 614a~n (i.e., the data read pipeline stage) can handle those IO requests. These DASD IO threads 614a~n are common threads shared by all range threads 612a~n. After the DASD IO threads 614a~n obtain the requested data, the data is placed in the output buffer 646 to serialize the data into a blob so that the replica object threads 616a~n (i.e., the data upload pipeline stage) can upload it to the object store located in the target region. Each object thread selects an upload job that can include a portion of all the data to be uploaded, and all object threads execute the uploads in parallel.

[0102] Data plane FIG. 7 is a diagram showing a hierarchical structure in an FSS data plane according to an embodiment. In FIG. 7, the replicator fleet 710 includes four layers: a job layer 712, a delta generator client 714, an encryption / DASD IO 716, and an object 718. The replicator fleet 710 is a single process that serves to exchange information with a storage fleet 720, a KMS 730, and an object storage 740. In one embodiment, the job layer 712 polls the SDB 704 for a job 706 that is queued as either an upload job or a download job. The replicator fleet 710 includes a VM (or thread) that selects enqueue replication jobs up to the maximum capacity. A replicator thread may own part of a replication job, but coordinates with another replicator thread that owns the remaining part of the same replication job to complete the entire replication job simultaneously. The replication job executed by the replicator fleet 710 is restartable in that if a replicator thread fails during replication, another replicator thread can take over and continue from the last successful point to complete the job that the failed replicator thread originally owned. If a strayed replicator thread (e.g., a replicator thread that fails and restarts) competes with another replicator thread, the FSS can avoid the conflict by using a mechanism called a generation number to cause both replicator threads to update different records.

[0103] The differential generator client layer 714 performs a B-tree scan by accessing the differential generator server 724 in which the B-tree exists within the storage fleet 720. The encryption / DASD IO layer 716 assumes the roles of security and storage access. After the B-tree scan, the replicator fleet 710 may request IO access via the encryption / DASD IO layer 716 to access the DASD range 722 of the file data associated with the differences identified during the B-tree scan. The replicator fleet 710 and the storage fleet 720 both periodically update the status of the control API 702 (e.g., the checkpoint and lease of the replicator fleet 710) via the SDB 704, enabling the control API 702 to trigger an alarm or execute an action if necessary.

[0104] During the inter-region replication process, the encryption / DASD IO layer 716 exchanges information with the KMS and the FSK fleet 730 on the target side to create a session key (or snapshot encryption key) and uses the FSK for encryption and decryption of the session key. Finally, the object layer 718 is responsible for uploading differential and file data from the source file system to the object store 740 and downloading them from the object store 740 to the target file system.

[0105] The data plane of the FSS is responsible for differential generation. The data plane stores FSS data using a B-tree, which includes various types of key-value pairs including, but not limited to, a leader block, a superblock, an iNode, a file name key, a cookie map (cookies associated with directory entries), and a block map (also called FMAP in the case of file content data).

[0106] These B-tree keys are processed together by the replicator and the differential generator within the data plane. An algorithm for calculating the pairs of keys and values (i.e., part of the difference) changed between two specific snapshots within the file system continuously reads the keys, returns the keys to the replicator using the transaction budget, and finally ensures that the transaction is confirmed to obtain a consistent pair of keys and values for processing.

[0107] In other embodiments, the difference generation and calculation may be extensible. An extensible approach can calculate the difference (i.e., the change in the pairs of keys and values) between two snapshots by utilizing multiple threads to divide the B-tree into many key ranges. A pool of threads (i.e., differential generators) can execute the scan of the B-tree (i.e., the traversal of the B-tree) and calculate the differences in parallel.

[0108] Figure 8 shows a simplified exemplary binary large object (BLOB) format according to an embodiment. A BLOB is a data type for storing information (e.g., binary data) in a database. A BLOB is generated by the source region during replication and uploaded to the object store. The target region needs to download and apply the BLOB. BLOBs and objects can be used interchangeably in the same sense depending on the context.

[0109] During the B-tree scan, when the difference generator encounters the iNode of a specific file (i.e., data content) and its block map (also called the data associated with the B-tree key, FMAP), the difference generator, in cooperation with the replicator, traverses all the pages within the blocks (FMAP blocks) within the DASD range pointed to by the FMAP, reads them into the data buffer, decrypts the data using the local encryption file key, puts it into the output buffer, and serializes it into a blob for the replicator to upload to the object store. In other words, the difference generator needs to collect all the FMAPs of the identified differences in order to obtain all the data related to the differences between the two snapshots.

[0110] The snapshot differences stored in the object store may span multiple blobs (or objects if stored in the object store). The blob format of these blobs includes a key, a value, and, if present, the data associated with the key. For example, in Figure 7, the snapshot difference includes at least three blobs 802, 804, and 806. The first blob 802 includes a prefix 810 indicating the key and value types, the key length, and the value length, followed by a key 812 (key 1) and a value 814 (value 1). The second blob 804 includes a prefix 820 (key and value types, key length, and value length), a key 822 (key 2), a value 824 (value 2), a data length 826, and data 828 (data 2). In the prefix 820 of this second blob 804, since this blob includes additional data 828 associated with the key 822, the key and value types are fmap. The third blob 830 includes a format similar to that of the first blob 810, for example, a prefix 830, a key 832 (key 3), and a value 834 (value 3).

[0111] The data is decrypted, collected, and then written to a blob. All processes are executed in parallel. Multiple blobs can be processed and updated simultaneously. When all processes are complete, the data is written in blob format (shown in Figure 8) and can then be uploaded to an object store in the format or path name (shown in Figure 9).

[0112] Figure 9 shows an exemplary replication bucket format according to an embodiment. A "bucket" can refer to a container that stores objects in a compartment within an object storage namespace. In one embodiment, a bucket is used by a source replicator to store data protected using server-side encryption (SSE) technology and is also used by a target replicator to download changes and apply them to a snapshot. Replication data for all file systems in a target region can share a bucket within that region.

[0113] The data layout of a bucket in the object store has a directory structure that includes, but is not limited to, a file system ID (e.g., Oracle Cloud ID), a delta including a start snapshot number and an end snapshot number, a manifest that describes the content of the information within the layout of the object, and blobs. For example, the bucket in FIG. 9 includes two objects 910 and 930. The first object 910 includes two deltas 912 and 920. This object begins with a path name 911 (e.g., ocid1.filesystem.oc1.iad...) using the source file system ID as a prefix, followed by a first delta 912 generated from snapshot 1 and snapshot 2, and a second snapshot 920 generated from snapshot 2 and snapshot 3. Each delta includes one or more blobs that represent the content of that delta. The first delta 912 stores two blobs 914 and 916 in the order of generation. The second delta 920 includes only one blob 922. Each delta also includes a manifest that describes the content of the information within the layout of this delta, e.g., manifest 918 of the first delta 912 and manifest 924 of the second delta 920. The manifest within the bucket is content that describes the delta, such as the file system number and snapshot range. The manifest can be a master manifest, a range manifest, or a checkpoint manifest depending on the stage of the replication process.

[0114] The second object 930 also includes two deltas 932 and 940 in a similar format starting with path name 931. The two objects 910 and 930 within the bucket come from different source regions, namely IAD for object 910 and PHX for object 930. After the blobs are applied, the corresponding information within the layout can be removed to reduce space utilization.

[0115] The final manifest object (i.e., the checkpoint manifest, CM file) is uploaded from the source region to the object store, and the source file system indicates to the target region that the upload of the snapshot delta of a specific object has been completed. The source CP communicates this event to the target CP, and the target CP can notify the target DP via the SDB to trigger the download process of that object by the target replicator.

[0116] Control Plane State Machine The control plane within the source region or target region coordinates all of the replication workflow and drives the replication of data. The control plane performs the following functions: (1) creates the underlying system snapshot for creating deltas, (2) determines when such snapshots need to be created, (3) initiates replication based on the snapshots, (4) monitors the replication, (5) triggers the download of deltas by the secondary (or target side), and (6) indicates to the primary (or source) side that the snapshot has reached the secondary.

[0117] The file system has several operations for processing its resources, including but not limited to creating, reading, updating, and deleting (CRUD). These operations are usually synchronized within the same region, starting a workflow when the file system receives an HTTPS request from the API server, making changes in the backend to create a resource, and returning a response to the customer. The resources are divided into a source region and a target region. The state is maintained for the same resource between the source region and the target region. Therefore, there is asynchronous communication between the source region and the target region. A customer can create or update a resource by contacting the source region, and these creations or updates can be automatically reflected in secondary or auxiliary resources within the target region. The state machine in the control plane also targets recovery in many aspects, including but not limited to fleet failures, key management failures, disk failures, and object failures.

[0118] Regarding the application programming interfaces (APIs) within the control plane, there are various APIs for users to configure replication. The control API for any new resource only functions within the region where the object was created. In the target file system, a field called "IsTargetable" can be set in the API to ensure that the target file system receiving replication cannot be accidentally used by consumers. In other words, setting this field to false means that consumers can view the target file system, but no one can export the target file system or access any data within the live system. Since export is not a read-only permission but a read / write permission for export, any export can potentially modify the data. Therefore, during the replication process, exports are not permitted to prevent any changes to the target file system. Consumers can only access the data within the old snapshots that have already been replicated. Any newly created or replicated file system can have this field set to true. The reason is that the target can only obtain data from a single source. Otherwise, conflicts may occur when data is written or deleted. The system needs to know whether the target file system being used is already part of some replication. Setting the "IsTargetable" field to "true" means that replication is not in progress, and setting it to "false" means that the target file system cannot be used.

[0119] Regarding inter-region communication between components of the control plane, the primary resource on the source file system is called an application, and the auxiliary (or secondary) source on the target file system is called an application target. When source and target objects are created, they have a single replication relationship. Both objects can be updated only from the source side, including changes to compartments, editing of details, or deletion. If the user wants to delete the target side, the replication itself can be deleted. In the case of a planned failover, it is possible to delete the source side, and both the source and target replications are deleted. In the case of an unplanned failover, the source side is not available, so only the target replication can be deleted. In other words, there are two resources for a single replication, and those resources should be kept in a synchronized state. There are various workflows for updating metadata on both the source and target sides. Additionally, inter-region APIs for retry, fault handling, and failover are also part of the inter-region communication process.

[0120] When creating the necessary security and other related artifacts, the source uploads the security and artifacts to the object store, starts a job at the target (i.e., notifies the target that the job is available), and the target can start downloading the artifacts (e.g., snapshots or deltas). Then, the target continues to look for an end-of-file marker (also referred to herein as a checkpoint manifest (CM) file) within the object store. The CM file is used as a mechanism for the source side and the target side to communicate the completion of the upload of the object during the replication process. At every checkpoint, the source side uploads this CM file containing information such as the number of blobs uploaded up to this checkpoint, enabling the target side to download this number of blobs and apply them to the current snapshot. This CM file is a mechanism for the source side to communicate to the target side that the upload of the object to the object store is complete, allowing the target to start working on that object. In other words, the target continues to download until there are no more objects in the object storage. Thus, this approach enables concurrent processing on both the source side and the target side.

[0121] Figure 10 is a flowchart showing a state machine for simultaneous source upload and target download according to an embodiment. As previously explained, both the source file system and the target file system can perform replication simultaneously and thus can each have its own state machine. In one embodiment, each file system can have its own state machine while sharing some common job-level states. In Figure 10, the source file system has states 1030 - 1034 for session key generation and transfer in addition to states 1002 - 1018 for performing data upload. The target file system has states 1050 - 1068 related to data download. The session key can be generated at any time within the source file system while differences are being uploaded to the object storage. Thus, the session key transfer has its own state sequence 1030 - 1034. In Figure 10, the target file system cannot start the replication download process (i.e., Ready_to_Reconcile state 1050) until it receives an indication that at least an object has been uploaded to the object storage by the source file system (i.e., Mainfest_Copied state 1014) and that it is ready to download the session key (i.e., Copied_DTK state 1034).

[0122] In the source file system, multiple functional blocks such as a snapshot generator, a control API, and a delta monitor are part of the CP. The replicator fleet is part of the DP. The snapshot generator is responsible for periodically generating snapshots. The delta monitor periodically monitors the progress of the replicator in replication-related tasks, including the creation of snapshots and the replication schedule. When the delta monitor detects that the replicator has completed a replication job, it transitions the state to a copied state on the source side (e.g., Manifest_Copied state 1014) or a replicated state on the target side (e.g., Replicated state 1058). In certain embodiments, multiple file systems can simultaneously perform replication from a source region to a target region.

[0123] Referring to FIG. 10, in certain embodiments, in the source file system, in the simultaneous mode state machine, after creating a snapshot signal to the delta monitor indicating that a snapshot has been generated, the snapshot generator. The delta monitor that executes the CP replication state (CpRpSt) workflow is responsible for starting the upload of snapshot metadata to the object store on the target side. The CpRpSt workflow sets the Ready_to_Copy_Metadata state 1002 for the replicator fleet to start copying the metadata. When the replicator acquires a replication job, it creates a copy of the snapshot metadata (i.e., Snapshot_Metadata_Copying state 1004) and uploads those copies to the object store. When all replicators have completed the upload of the snapshot metadata, the state is set to the Snapshot_Metadata_Copied state 1006. Thereafter, the CpRpSt workflow continues to poll the source SDB for the session key.

[0124] Here, the CpRtSt workflow returns control to the differential monitor to monitor the differential upload process that transitions to the Ready_to_Copy state 1008 indicating that the differential calculation is scheduled. Next, the source CP API sends a request to the replicator to start the next stage of replication by uploading the differential and creating a copy of the manifest. The replicator that selects the replication job can start creating a copy of the manifest (i.e., the Mainfest_Copying state 1010). When the source file system completes the copy of the manifest, it transitions to the Manifest_Copied state 1014 and simultaneously notifies the target file system that it can start the internal state (the Ready_to_Reconcile state 1050).

[0125] As described above, the session key can be generated by the source file system during the upload of data. The replicator of the source file system communicates with the target KMS vault to obtain the master key that can be provided by the customer and creates a session key (referred to herein as the differential encryption key or DEK). Next, the replicator encrypts the session key using the local file system key (FSK: file system key) (which becomes the encrypted DEK, also referred to herein as the differential transfer key or DTK). Thereafter, the DTK is stored in the SDB within the source region and reused by the replicator thread during the replication cycle. The state machine transitions to the Ready_to_Copy_DTK state 1030.

[0126] The source file system transfers the resource identification information of DTK and KMS to the target API. Then, the target API puts those resource identification information into the SDB within the target region. During this transfer process, the state machine is set to the Copying_DTK state 1032. When the CpRpSt workflow in the source file system finishes polling the source SDB for the session key, the target file system downloads the session key (DTK) and sends a notification to the target side indicating that it is ready to use this session key to decrypt the downloaded differences for application. Then, the state machine migrates to the Copied_DTK state 1034. The target-side replicator retrieves the DTK from the SDB and requests the KMS API to decrypt this DTK into the plaintext DEK (i.e., the decrypted session key). Next, this DEK is stored in the local cache for reuse by the target replicator thread during the replication cycle.

[0127] When the source file system completes uploading the data for a specific replication cycle including session key transfer, the difference monitor notifies the target control API of the status such as validity confirmation information and migrates to the X-region_Copied_Done state 1016. This can occur before the target file system finishes downloading and applying the data. The source file system further cleans up the memory and removes all keys. Then, the source file system migrates to the Awaiting_Target_Response state 1018, waits for a response from the target file system, and starts a new replication cycle.

[0128] As described above, the target file system cannot start the replication download process until it receives an indication that at least the object has been uploaded to the object storage by the source file system (i.e., the Mainfest_Copied state 1014), and that it is ready to download the session key (i.e., the Copied_DTK state 1034). When these two conditions are met, the state machine transitions to the Ready_To_Reconcile state 1050. Next, in the Reconciling state 1052, the target file system starts an adjustment process with the source side, such as synchronizing snapshots of the source file system and the target file system, takes snapshots, and also performs some internal CP management operations including generating statistics. This internal state includes communication within the target file system between the delta monitor and the CP API.

[0129] After the adjustment process is completed, the replication job is passed to the target replicator (i.e., the Ready_to_Replicate state 1054). The target replicator monitors the checkpoint manifest (CM) file uploaded by the source file system. The CM file is marked by the target. Then, the target replicator thread starts to download the manifest and apply the downloaded and decrypted deltas (i.e., the Replicating state 1056). The target replicator thread also reads the FMAP data blocks from the blobs downloaded from the object store, communicates with the local FSK service to obtain the file system key FSK, and the FSK is used to re-encrypt each FMAP data block and store it in local storage.

[0130] When the source file system finishes uploading data, it updates the final CM file by setting the end-of-file (eof) field to true and uploads it to the object store. As soon as the target file system detects this final CM file, it finishes downloading the blobs and applies them, and the state machine transitions to the Replicated state 1058. In other words, the source file system and the target file system operate simultaneously. The target file system can start downloading the encrypted snapshot differences before the source file system finishes uploading all the snapshot differences (i.e., the final CM file with the eof field set to true) when the target file system detects that the checkpoint CM file in the object store is available (i.e., at least a part of the encrypted snapshot differences is available).

[0131] After the target file system applies all the differences (or blobs), it continues to download the snapshot metadata from the object store and inputs the information of the source file system's snapshot into the target file system's snapshot (i.e., the Snapshot_metadata_Populating state 1060). When the target file system's snapshot is input, the state machine transitions to the Snapshot_Metadata_Populated state 1062.

[0132] In the Snapshot_Deleting state 1064, the target file system deletes all blobs in the object store for the blobs that have been downloaded and applied to the latest snapshot. After that, when the blobs in the object store are deleted, the target control API notifies the target difference monitor and proceeds to the Snapshot_Deleted state 1066. The target file system further cleans up the memory and removes all keys. The FSS service also releases the KMS key.

[0133] After the target DP finishes the difference application and cleanup, it uses the target control API to verify the validity regarding the status of the source file system and whether it has received the X-region_Copied_Done notification from the source file system. If the notification has been received, the target difference monitor transitions to the X-region DONE state 1068 and sends the X-region DONE notification to the source file system. In some embodiments, the target file system can also detect whether the source file system has completed the upload by checking whether there is an end-of-file for all key ranges and all upload processing threads because all objects uploaded to the object store have special markers such as end-of-file markers in the CM file.

[0134] Referring to the state machine of the source file system again, while in the Awaiting_Target_Response state 1018, the source file system checks whether the status of the target CP has changed to completed, indicating that all differences downloaded by the target have been applied and the file data has been stored locally. If the status of the target CP has changed to completed, this marks the end of the replication cycle.

[0135] The source side and the target side operate asynchronously. When the source file system completes the replication upload, it notifies the target control API of the X-region_Copied_Done notification. Then, when the target file system completes the replication process, the differential monitor target communicates in the reverse direction with the source control API using the X-region DONE notification. The source file system returns to the Ready_to_Copy_Metadata state 1002 and starts another replication cycle.

[0136] Information exchange between the data plane and the control plane Figure 11 is an exemplary flowchart showing information exchange between the data plane and the control plane within a source region according to an embodiment. The data plane components and the control plane components communicate with each other using a shared database (SDB), such as 1106. The SDB is a key-value store that both the control plane components and the data plane components can read from and write to. The data plane components include a replicator and a differential generator. Information exchange between components within the source region A1101 and the target region B1102 is also shown.

[0137] In FIG. 11, in step S1, the source control plane (CPa) 1103 requests the object store within the target region B (OSb) 1112 to create a bucket. In step S2, the source replicator (REPLICATORa) 1108 periodically updates the heartbeat status to the source SDB (SDBa) 1106. A heartbeat is a concept used to track the progress of replication executed by the replicator. The heartbeat uses a mechanism called lease, where the heartbeat is continuously updated each time the replicator works on a job, enabling the control plane to recognize the entire release information. For example, the byte count is continuously moving on the job. If the replicator fails to function properly, the heartbeat may become stale, and then another replicator can detect and take over, continuing the work on the remaining jobs. Therefore, if the system crashes midway, the system can accurately start from the last point based on the checkpoint mechanism. A checkpoint helps the system know where the last point of progress was and enables it to continue from that point without re-executing the entire work.

[0138] In step S3, CPa1103 further requests the File System Service Workflow (FSW_CPa) 1104 to create snapshots periodically. In step S4, FSW_CPa1104 notifies CPa1103 about the new snapshot. In step S5, next, CPa1103 stores the snapshot information in SDBa1106. In step S6, REPLICATORa1108 polls SDB1106 for any changes to the existing snapshots. If a change is detected, in step S7, it retrieves the job specification. When REPLICATORa1108 detects a change to the snapshot in step S8, this initiates the replication process. In step S8, REPLICATORa1108 provides information about two snapshots (SNa and SNb) including the changes between the snapshots to the Difference Generator (DGa) 1110. In step S9, REPLICATORa1108 enters work item information such as the number of key ranges into SDBa1106. In step 10, REPLICATORa1108 checks the replication job queue in SDBa1106 to obtain work items. In step S11, it assigns those work items to the Difference Generator (DGa) 1110 to scan the B-tree keys of the snapshot (i.e., traverse the B-tree) and calculate the differences and the corresponding key-value pairs. In step 12, REPLICATORa1108 decrypts the file data associated with the identified B-tree keys and packs them together with the key-value pairs into blobs. In step 13, REPLICATORa1108 encrypts the blobs using the session key and uploads them as objects to OSb1112. In step S14, REPLICATORa performs a checkpoint and stores the checkpoint record in SDBa1106. This replication process repeats (as a loop) until all differences are identified and the data is uploaded to OSb1112.In step S15, REPLICATORa1108 then notifies SDBa1106 of the details of the replication job, which are then passed to CPa1103 in step S16 and further relayed to CPb1114 as the final CM file in step S17. In step S18, CPb1114 stores the job details in SDBb1116.

[0139] The exchange of information between the data plane and the control plane within target region B is similar. At the end of applying the delta to the target file system, the control plane within target region B notifies the control plane within source region A that the snapshot has been successfully applied. Thereby, the control plane within source region A can start over using the new snapshot.

[0140] Authentication is performed for all components. There is an authentication mechanism that uses the replication ID and file system number from the replicator to the file system key (FSK). The key can be given to the replicator only if the replicator provides appropriate content. Thus, the authentication mechanism can prevent fraudsters from obtaining the decryption key. Other security mechanisms include blocking network ports. A component called the file system key server (FSKS) is a gatekeeper for properly checking the requester by checking metadata such as the job the requester is executing and other information. For example, assume the replicator is trying to request the key to the file system. In that case, FSKS can check whether the replicator is associated with a specific job (e.g., whether the replication is actually associated with that file system) to confirm the validity of the requester.

[0141] Availability and Durability Availability addresses situations where a machine can automatically restart after going down or where services remain available while software deployment is in progress. For example, since all replicators are stateless, losing a replicator is transparent to customers because another replicator can take over and continue the job's work. The job's state is maintained not locally but in a shared database and other reliable locations. The shared database is a service like the one used by the control plane to maintain information about the file system and is based on a B-tree.

[0142] The system has thousands of storage nodes that allow any storage node to perform differential replication, so storage availability in the FSS of this disclosure is high. By using many machines that can take over from each other in case of some failure, the availability of the control plane is increased. For example, the progress of replication is not simply hindered by the failure of a single control plane. Thus, there is no single point of failure. Network access availability uses congestion management, including various types of throttling, to prevent source nodes from becoming overloaded.

[0143] The replication state is written to the shared database, and replication is durable by using checkpointing where replicators are stateless. The replication process is idempotent. Idempotency can refer to deterministic reapplication where, in case of an operation failure, retrying the same operation, for example, using the same key, upload process, or scan process, should result in the same outcome.

[0144] Operations within multiple regions should be idempotent. In the control plane, the actions taken should be stored. For example, if an HTTP request itself is repeated, an idempotency cache can be useful in remembering that a particular operation has been performed and that it is the same operation. For example, in the data plane, when a block is allocated, the block and the file system's file map key are written together. Thus, if the block is allocated again, the block can be identified. If the block is sealed, the write operation fails. The idempotency mechanism can know that the block has been sealed in the past and the write operation need not be retried. In yet another example, the idempotency mechanism stores a chain of steps that need to be performed for a particular key and value processing. In other words, the idempotency mechanism allows all operations to be checked to see if they are in the correct state. Thus, the system can simply proceed to the next step without repeating.

[0145] Atomic replay enables the application of differences to start as soon as the first difference object reaches the object store when a snapshot is rolled back, for example, when going from snapshot 10 to snapshot 5. To make the replay atomic, the entire difference needs to be maintained in the object store before the differences can be applied.

[0146] Expansion Regarding the expansion of replicators, the FSS of the present disclosure enables adding the number of replication machines (e.g., replicator virtual machines (“VMs”)) required to support many file systems. The number of replicators can be dynamically increased or decreased by considering the bandwidth requirements and availability of resources. Regarding the expansion of storage, thousands of storages can be used to parallelize the process and improve the working speed. Regarding the inter-region bandwidth, the bandwidth allocation is automatically adjusted, such as adjusting all inter-region bandwidths by grasping the increase in latency and reducing the required speed, to ensure that each workload is not overused or does not exceed a predefined throughput limit. All replicator processors (or threads) have this function.

[0147] In the expansion of checkpoint storage, the uploader and downloader checkpoint the progress to persistent storage, and the shared storage is used as a work queue for splitting key ranges. If the checkpoint workload overly burdens the shared database, the checkpoint storage function can be added to the differential generator for expansion purposes. The current workload of the shared database may consume less than 10 IOPs.

[0148] Failback mode FIG. 12 is a schematic diagram showing the failback mode according to an embodiment. The failback mode enables restoring the primary side / source side to become the primary / source again before failover. As shown in FIG. 12, the primary AD 1202 includes the source file system 1206, and the secondary AD 1204 includes the target file system 1208. The secondary AD 1204 may exist in the same region or a different region from the region of the primary AD 1202.

[0149] In FIG. 12, snapshot 1 1220 and snapshot 2 1222 in the source file system 1206 exist before a failover due to a power outage event. Similarly, snapshot 1 1240 and snapshot 2 1242 of the target file system 1208 exist before the failover. When a power outage occurs in snapshot 3 1224 in the primary AD 1202, the FSS performs an unplanned failover 1250, and snapshot 3 1224 in the source file system 1206 is replicated to the target file system 1208 and becomes the new snapshot 3 1224. After the target file system 1208 is up and running, the customer can make changes to create snapshot 4 1246 for the target file system 1208.

[0150] If the customer decides to reuse the source file system again, the FSS service may perform a failback. When the user performs a failback, the user has two options: (1) the last point in time in the source file system before the trigger event 1251, or (2) the latest change in the target file system 1252.

[0151] In the case of the first option, the user can resume from the last point in the source file system 1206 prior to the trigger event (i.e., snapshot 3 1224). In other words, since snapshot 3 1224 has previously successfully failed over to the target file system 1208, it becomes the snapshot for use after failback. To execute the failback 1251, the state of the source file system 1206 is changed to inaccessible. Next, the FSS service identifies snapshot 3 1224, the last point in the source file system 1206 before the failover was successful. The FSS may execute a clone of snapshot 3 1224 (i.e., a replication within the same region) in the primary AD 1202. Now, the primary AD 1202 returns to its initial settings prior to the power outage, and the user can reuse the source file system 1206 again. Since snapshot 3 1224 already exists in the file system being used, no data transfer from the secondary AD 1204 to the primary AD 1202 is required.

[0152] In the case of the second option, the user wants to reuse the source file system with the latest changes in the target file system 1208. In other words, since snapshot 4 1246 in the target file system 1208 was the latest change in the target file system 1208, it becomes the snapshot for use after failback. The failback process 1252 for this option includes reverse replication (i.e., reversing the roles of the source file system and the target file system for the replication process), and the FSS executes the following steps.

[0153] Step 1. The state of the source file system 1206 is changed to inaccessible. Step 2. Next, the FSS service identifies the latest snapshot in the successfully replicated target file system 1208, for example, snapshot 3 1244.

[0154] Step 3. The FSS service also detects the corresponding snapshot 3 1224 in the source file system 1206 and performs a clone (i.e., a copy within the same region).

[0155] Step 4. The FSS service starts reverse replication 1252 in a process similar to that described in connection with FIG. 4, but in the reverse direction. In other words, both the source file system 1206 and the target file system 1208 need to be synchronized, after which the target file system 1208 can upload the differences to the object store in the primary AD 1202. The source file system 1206 can download the differences from the object store, complete the application to snapshot 3 1224, and create a new snapshot 4 1226.

[0156] Here, the primary AD 1202 returns to its initial settings before the power outage, and the user can reuse the source file system 1206 again without transferring the data that already exists in both the source file system 1206 and the target file system 1208, for example, snapshots 1 to 3 (1220 to 1224) in the source file system 1206. This saves time and prevents unnecessary bandwidth.

[0157] Infrastructure Architecture as an Exemplary Service As described above, infrastructure as a service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services that arise in connection with those infrastructure components (examples of services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, since these services can be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.

[0158] In some cases, IaaS customers may access resources and services via a wide area network (WAN) such as the Internet and use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and install enterprise software on the VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application problems, monitoring performance, managing disaster recovery, etc.

[0159] In most cases, cloud computing models require the participation of a cloud provider. A cloud provider can be a third-party service that specializes in providing (e.g., offering, lending, selling) IaaS, but it doesn't have to be. An entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0160] In some examples, the deployment of IaaS is the process of placing a new application or a new version of an application on a prepared application server, etc. This process may include the process of preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by a cloud provider under a hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, a customer may play a role in handling the deployment of an (OS), middleware, and / or application on top of (e.g., a self-service virtual machine that can be spun up on demand).

[0161] In some examples, the provisioning of IaaS can also refer to acquiring computers or virtual hosts for use and installing the required libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning may need to be done first.

[0162] In some cases, there are two different challenges in IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is executed. Second, after everything is provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two challenges can be addressed by enabling the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., which components are needed and how those components exchange information) can be defined by one or more configuration files. In this way, the entire infrastructure topology (e.g., which resources depend on which other resources and how they interact with each other) can be described declaratively. In some cases, after the topology is defined, a workflow for creating and / or managing the various components described in the configuration file can be generated.

[0163] In some examples, the infrastructure can include many interconnected elements. For example, there can be one or more virtual private clouds (VPCs), also known as core networks (e.g., a configurable and / or shared pool of computing resources, possibly on-demand), which can be provisioned. In some examples, there can be one or more inbound traffic / outbound traffic group rules provisioned to define how the inbound traffic and / or outbound traffic of the network is set up, and there can also be one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. can be provisioned. As more infrastructure elements are desired and / or added, the infrastructure can evolve gradually.

[0164] In some cases, continuous deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Further, the techniques described can enable infrastructure management within these environments. In some examples, a service team may describe code that is desired to be deployed to one or more, but often many, different production environments (e.g., across various geographical locations and sometimes globally). However, in some examples, the infrastructure to which the code is deployed must be initially provisioned. In some cases, provisioning can be done manually, provisioning tools may be utilized to provision resources, and / or deployment tools may be utilized to deploy the code after the infrastructure has been provisioned.

[0165] FIG. 13 is a block diagram 1300 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1302 can be communicatively coupled to a secure host tenancy 1304 that can include a virtual cloud network (VCN) 1306 and a secure host subnet 1308. In some examples, the service operator 1302 may use one or more client computing devices, which can be portable handheld devices (e.g., iPhone (registered trademark), mobile phone, iPad (registered trademark), computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass head-mounted display) that run software such as Microsoft Windows Mobile (registered trademark) and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and have Internet, email, short message service (SMS), BlackBerry (registered trademark), or other communication protocols enabled. Alternatively, the client computing device can be a general-purpose personal computer, including, by way of example, personal computers and / or laptop computers running various versions of Microsoft Windows (registered trademark), Apple Macintosh (registered trademark), and / or Linux (registered trademark) operating systems. The client computing device can be a workstation computer running any of various commercially available UNIX (registered trademark) or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google (registered trademark) Chrome OS.Alternatively or in addition, the client computing device can be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect (registered trademark) gesture input device), and / or a personal messaging device, that can communicate via a network and / or the Internet that has access to the VCN 1306.

[0166] The VCN 1306 can include an LPG 1310 that can be communicatively coupled to an SSH VCN 1312 via a local peering gateway (LPG) 1310 included in a secure shell (SSH) VCN 1312. The SSH VCN 1312 can include an SSH subnet 1314 and can be communicatively coupled to a control plane VCN 1316 via an LPG 1310 included in the control plane VCN 1316. Also, the SSH VCN 1312 can be communicatively coupled to a data plane VCN 1318 via the LPG 1310. The control plane VCN 1316 and the data plane VCN 1318 can be included in a service tenancy 1319 that can be owned and / or operated by an IaaS provider.

[0167] The control plane VCN 1316 can include a control plane demilitarized zone (DMZ) layer 1320 that functions as a border network (e.g., a part of a corporate network between a corporate intranet and an external network). Servers based on the DMZ can have limited responsibilities and can help contain intrusions. Further, the DMZ layer 1320 can include one or more load balancer (LB) subnets 1322, a control plane application layer 1324 that can include an application subnet 1326, and a control plane data layer 1328 that can include a database (DB) subnet 1330 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1322 included in the control plane DMZ layer 1320 can be communicatively coupled to the application subnet 1326 included in the control plane application layer 1324 that can be included in the control plane VCN 1316 and to the Internet gateway 1334, and the application subnet 1326 can be communicatively coupled to the DB subnet 1330 included in the control plane data layer 1328 as well as to the service gateway 1336 and the network address translation (NAT) gateway 1338. The control plane VCN 1316 can include the service gateway 1336 and the NAT gateway 1338.

[0168] The control plane VCN 1316 can include a data plane mirror app layer 1340 that can include an app subnet 1326. The app subnet 1326 included in the data plane mirror app layer 1340 can include a virtual network interface controller (VNIC) 1342 that can execute a compute instance 1344. The compute instance 1344 can communicatively couple the app subnet 1326 of the data plane mirror app layer 1340 to an app subnet 1326 that can be included in the data plane app layer 1346.

[0169] The data plane VCN 1318 can include a data plane app layer 1346, a data plane DMZ layer 1348, and a data plane data layer 1350. The data plane DMZ layer 1348 can include an LB subnet 1322 that can be communicatively coupled to the app subnet 1326 of the data plane app layer 1346 and to the Internet gateway 1334 of the data plane VCN 1318. The app subnet 1326 can be communicatively coupled to the service gateway 1336 and the NAT gateway 1338 of the data plane VCN 1318. The data plane data layer 1350 can also include a DB subnet 1330 that can be communicatively coupled to the app subnet 1326 of the data plane app layer 1346.

[0170] The Internet gateways 1334 of the control plane VCN 1316 and of the data plane VCN 1318 can be communicatively coupled to a metadata management service 1352 that can be communicatively coupled to the public Internet 1354. The public Internet 1354 can be communicatively coupled to the NAT gateways 1338 of the control plane VCN 1316 and of the data plane VCN 1318. The service gateways 1336 of the control plane VCN 1316 and of the data plane VCN 1318 can be communicatively coupled to a cloud service 1356.

[0171] In some examples, the service gateway 1336 of the control plane VCN 1316 or the data plane VCN 1318 can make application programming interface (API) calls to the cloud service 1356 without going through the public Internet 1354. The API call from the service gateway 1336 to the cloud service 1356 can be unidirectional, and the service gateway 1336 can make an API call to the cloud service 1356, and the cloud service 1356 can send the requested data to the service gateway 1336. However, the cloud service 1356 does not have to initiate an API call to the service gateway 1336.

[0172] In some examples, the secure host tenancy 1304 can be directly connected to the service tenancy 1319, or otherwise can be isolated. The secure host subnet 1308 can communicate with the SSH subnet 1314 via the LPG 1310, and the LPG 1310 can enable two-way communication on a separated system if not. Connecting the secure host subnet 1308 to the SSH subnet 1314 can give the secure host subnet 1308 access to other entities within the service tenancy 1319.

[0173] The control plane VCN 1316 may enable users of service tenancy 1319 to set or otherwise provision desired resources. Desired resources provisioned within the control plane VCN 1316 may be deployed or otherwise used in the data plane VCN 1318. In some examples, the control plane VCN 1316 may be separable from the data plane VCN 1318, and the data plane mirror app layer 1340 of the control plane VCN 1316 may communicate with the data plane app layer 1346 of the data plane VCN 1318 via VNICs 1342 that may be included in the data plane mirror app layer 1340 and the data plane app layer 1346.

[0174] In some examples, a user or customer of the system may perform requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1354 that can communicate requests to the metadata management service 1352. The metadata management service 1352 can communicate requests to the control plane VCN 1316 via the internet gateway 1334. The request may be received by the LB subnet 1322 included in the control plane DMZ layer 1320. The LB subnet 1322 may determine that the request is valid, and in response, the LB subnet 1322 can send the request to the app subnet 1326 included in the control plane app layer 1324. If the validity of the request is confirmed and the request requires a call to the public internet 1354, the call to the public internet 1354 may be sent to the NAT gateway 1338 that can make the call to the public internet 1354. Metadata that may desirably be stored by the request may be stored within the DB subnet 1330.

[0175] In some examples, the data plane mirror application layer 1340 can facilitate direct communication between the control plane VCN 1316 and the data plane VCN 1318. For example, it may be desirable for changes, updates, or other appropriate modifications to the configuration to be applied to the resources included in the data plane VCN 1318. Through the VNIC 1342, the control plane VCN 1316 can communicate directly with the resources included in the data plane VCN 1318, thereby enabling changes, updates, or other appropriate modifications to the configuration of the resources.

[0176] In some embodiments, the control plane VCN 1316 and the data plane VCN 1318 may be included in the service tenant 1319. In this case, the user or customer of the system does not have to own or operate either the control plane VCN 1316 or the data plane VCN 1318. Instead, the IaaS provider may own or operate both the control plane VCN 1316 and the data plane VCN 1318, which may both be included in the service tenancy 1319. This embodiment can enable network separation that can prevent a user or customer from exchanging information with the resources of other users or other customers. Also, this embodiment can enable a user or customer of the system to privately store a database without relying on the public Internet 1354, which may not have the desired level of threat prevention for storage.

[0177] In other embodiments, the LB subnet 1322 included in the control plane VCN 1316 can be configured to receive signals from the service gateway 1336. In this embodiment, the control plane VCN 1316 and the data plane VCN 1318 can be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1354. A customer of the IaaS provider may desire this embodiment because the databases used by the customer may be controlled by the IaaS provider and may be stored in a service tenancy 1319 that can be isolated from the public Internet 1354.

[0178] FIG. 14 is a block diagram 1400 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1402 (e.g., service operator 1302 of FIG. 13) can be communicatively coupled to a secure host tenancy 1404 (e.g., secure host tenancy 1304 of FIG. 13) that can include a virtual cloud network (VCN) 1406 (e.g., VCN 1306 of FIG. 13) and a secure host subnet 1408 (e.g., secure host subnet 1308 of FIG. 13). The VCN 1406 can include an LPG 1410 (e.g., LPG 1310 of FIG. 13) that can be communicatively coupled to a secure shell (SSH) VCN 1412 (e.g., SSH VCN 1312 of FIG. 13) via an LPG 1310 included in the SSH VCN 1412. The SSH VCN 1412 can include an SSH subnet 1414 (e.g., SSH subnet 1314 of FIG. 13), and the SSH VCN 1412 can be communicatively coupled to a control plane VCN 1416 (e.g., control plane VCN 1316 of FIG. 13) via an LPG 1410 included in the control plane VCN 1416. The control plane VCN 1416 can be included in a service tenancy 1419 (e.g., service tenancy 1319 of FIG. 13), and the data plane VCN 1418 (e.g., data plane VCN 1318 of FIG. 13) can be included in a customer tenancy 1421 that can be owned or operated by a user or customer of the system.

[0179] The control plane VCN 1416 can include a control plane DMZ layer 1420 (e.g., the control plane DMZ layer 1320 of FIG. 13) that can include an LB subnet 1422 (e.g., the LB subnet 1322 of FIG. 13), a control plane application layer 1424 (e.g., the control plane application layer 1324 of FIG. 13) that can include an application subnet 1426 (e.g., the application subnet 1326 of FIG. 13), and a control plane data layer 1428 (e.g., the control plane data layer 1328 of FIG. 13) that can include a database (DB) subnet 1430 (e.g., similar to the DB subnet 1330 of FIG. 13). The LB subnet 1422 included in the control plane DMZ layer 1420 is communicatively coupled to the application subnet 1426 included in the control plane application layer 1424 that can be included in the control plane VCN 1416, and to an Internet gateway 1434 (e.g., the Internet gateway 1334 of FIG. 13). The application subnet 1426 is communicatively coupled to the DB subnet 1430 included in the control plane data layer 1428, and to a service gateway 1436 (e.g., the service gateway 1336 of FIG. 13) and a network address translation (NAT) gateway 1438 (e.g., the NAT gateway 1338 of FIG. 13). The control plane VCN 1416 can include the service gateway 1436 and the NAT gateway 1438.

[0180] The control plane VCN 1416 can include a data plane mirror app layer 1440 (e.g., the data plane mirror app layer 1340 of FIG. 13) that can include an app subnet 1426. The app subnet 1426 included in the data plane mirror app layer 1440 can include a virtual network interface controller (VNIC) 1442 (e.g., the VNIC 1342) that can execute a compute instance 1444 (e.g., similar to the compute instance 1344 of FIG. 13). The compute instance 1444 can facilitate communication between the app subnet 1426 of the data plane mirror app layer 1440 and an app subnet 1426 that can be included in the data plane app layer 1446 (e.g., the data plane app layer 1346 of FIG. 13) via the VNIC 1442 included in the data plane mirror app layer 1440 and the VNIC 1442 included in the data plane app layer 1446.

[0181] The internet gateway 1434 included in the control plane VCN 1416 can be communicatively coupled to a metadata management service 1452 (e.g., the metadata management service 1352 of FIG. 13) that can be communicatively coupled to the public internet 1454 (e.g., the public internet 1354 of FIG. 13). The public internet 1454 can be communicatively coupled to the NAT gateway 1438 included in the control plane VCN 1416. The service gateway 1436 included in the control plane VCN 1416 can be communicatively coupled to a cloud service 1456 (e.g., the cloud service 1356 of FIG. 13).

[0182] In some examples, the data plane VCN 1418 may be included in the customer's tenancy 1421. In this case, the IaaS provider may provide a control plane VCN 1416 for each customer, and the IaaS provider may configure the specific compute instances 1444 included in the service tenancy 1419 for each customer. Each compute instance 1444 may enable communication between the control plane VCN 1416 included in the service tenancy 1419 and the data plane VCN 1418 included in the customer's tenancy 1421. The compute instance 1444 may enable the resources provisioned within the control plane VCN 1416 included in the service tenancy 1419 to be deployed or otherwise used in the data plane VCN 1418 included in the customer's tenancy 1421.

[0183] In other examples, a customer of an IaaS provider may have a database that persists in the customer's tenancy 1421. In this example, the control plane VCN 1416 can include a data plane mirror app layer 1440 that can include an app subnet 1426. The data plane mirror app layer 1440 can exist in the data plane VCN 1418, but the data plane mirror app layer 1440 does not have to persist in the data plane VCN 1418. That is, the data plane mirror app layer 1440 can have access rights to the customer's tenancy 1421, but the data plane mirror app layer 1440 does not have to exist in the data plane VCN 1418 and does not have to be owned or operated by the customer of the IaaS provider. The data plane mirror app layer 1440 can be configured to make calls to the data plane VCN 1418, but does not have to be configured to make calls to any entity included in the control plane VCN 1416. The customer may wish to deploy or otherwise use resources within the data plane VCN 1418 that are provisioned within the control plane VCN 1416, and the data plane mirror app layer 1440 can facilitate the desired deployment or other use of the customer's resources.

[0184] In some embodiments, a customer of an IaaS provider can apply a filter to the data plane VCN 1418. In this embodiment, the customer can determine which data plane VCN 1418s are accessible, and the customer can restrict access from the data plane VCN 1418 to the public Internet 1454. The IaaS provider may not be able to apply a filter or otherwise control access of the data plane VCN 1418 to any external network or database. Applying filters and controls by the customer to the data plane VCN 1418 included in the customer's tenancy 1421 can help to isolate the data plane VCN 1418 from other customers and from the public Internet 1454.

[0185] In some embodiments, cloud service 1456 can be invoked by service gateway 1436 to access services that may not exist on either public internet 1454, control plane VCN 1416, or data plane VCN 1418. The connection between cloud service 1456 and control plane VCN 1416 or data plane VCN 1418 need not be operational or continuous. Cloud service 1456 can exist on a different network owned or operated by an IaaS provider. Cloud service 1456 can be configured to receive calls from service gateway 1436 and can be configured not to receive calls from public internet 1454. Some cloud services 1456 can be isolated from other cloud services 1456, and control plane VCN 1416 can be isolated from cloud services 1456 that may not exist in the same region as control plane VCN 1416. For example, control plane VCN 1416 may be located in "Region 1", and "Deployment 13" of the cloud service may be located in Region 1 and "Region 2". When a call to Deployment 13 is made by service gateway 1436 included in control plane VCN 1416 located in Region 1, this call can be sent to Deployment 13 within Region 1. In this example, control plane VCN 1416, or Deployment 13 within Region 1, need not be communicatively coupled to, or otherwise communicate with, Deployment 13 within Region 2.

[0186] FIG. 15 is a block diagram 1500 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1502 (e.g., the service operator 1302 of FIG. 13) can be communicatively coupled to a secure host tenancy 1504 (e.g., the secure host tenancy 1304 of FIG. 13) that can include a virtual cloud network (VCN) 1506 (e.g., the VCN 1306 of FIG. 13) and a secure host subnet 1508 (e.g., the secure host subnet 1308 of FIG. 13). The VCN 1506 can be communicatively coupled to an SSH VCN 1512 (e.g., the SSH VCN 1312 of FIG. 13) via an LPG 1510 (e.g., the LPG 1310 of FIG. 13) included in the SSH VCN 1512. The SSH VCN 1512 can include an SSH subnet 1514 (e.g., the SSH subnet 1314 of FIG. 13), and the SSH VCN 1512 can be communicatively coupled to a control plane VCN 1516 (e.g., the control plane VCN 1316 of FIG. 13) via an LPG 1510 included in the control plane VCN 1516 and to a data plane VCN 1518 (e.g., the data plane 1318 of FIG. 13) via an LPG 1510 included in the data plane VCN 1518. The control plane VCN 1516 and the data plane VCN 1518 can be included in a service tenancy 1519 (e.g., the service tenancy 1319 of FIG. 13).

[0187] The control plane VCN 1516 can include a control plane DMZ layer 1520 (e.g., the control plane DMZ layer 1320 of FIG. 13) that can include a load balancer (LB) subnet 1522 (e.g., the LB subnet 1322 of FIG. 13), a control plane application layer 1524 (e.g., similar to the control plane application layer 1324 of FIG. 13) that can include an application subnet 1526 (e.g., similar to the application subnet 1326 of FIG. 13), and a control plane data layer 1528 (e.g., the control plane data layer 1328 of FIG. 13) that can include a DB subnet 1530. The LB subnet 1522 included in the control plane DMZ layer 1520 can be communicatively coupled to the application subnet 1526 included in the control plane application layer 1524 that can be included in the control plane VCN 1516, and to an Internet gateway 1534 (e.g., the Internet gateway 1334 of FIG. 13). The application subnet 1526 can be communicatively coupled to the DB subnet 1530 included in the control plane data layer 1528, as well as to a service gateway 1536 (e.g., the service gateway of FIG. 13) and a network address translation (NAT) gateway 1538 (e.g., the NAT gateway 1338 of FIG. 13). The control plane VCN 1516 can include the service gateway 1536 and the NAT gateway 1538.

[0188] The data plane VCN 1518 can include a data plane application layer 1546 (e.g., the data plane application layer 1346 in FIG. 13), a data plane DMZ layer 1548 (e.g., the data plane DMZ layer 1348 in FIG. 13), and a data plane data layer 1550 (e.g., the data plane data layer 1350 in FIG. 13). The data plane DMZ layer 1548 can include a reliable application subnet 1560 and an unreliable application subnet 1562 of the data plane application layer 1546 and an LB subnet 1522 communicatively coupled to the Internet gateway 1534 included in the data plane VCN 1518. The reliable application subnet 1560 can be communicatively coupled to the service gateway 1536 included in the data plane VCN 1518, the NAT gateway 1538 included in the data plane VCN 1518, and the DB subnet 1530 included in the data plane data layer 1550. The unreliable application subnet 1562 can be communicatively coupled to the service gateway 1536 included in the data plane VCN 1518 and the DB subnet 1530 included in the data plane data layer 1550. The data plane data layer 1550 can include a DB subnet 1530 communicatively coupled to the service gateway 1536 included in the data plane VCN 1518.

[0189] The untrusted application subnet 1562 can include one or more primary VNICs 1564(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1566(1)-(N). Each tenant VM 1566(1)-(N) can be communicatively coupled to respective application subnets 1567(1)-(N) that can be included in respective container egress VCNs 1568(1)-(N) that can be included in respective customer tenancies 1570(1)-(N). Each secondary VNIC 1572(1)-(N) can facilitate communication between the untrusted application subnet 1562 included in the data plane VCN 1518 and the application subnets included in the container egress VCNs 1568(1)-(N). Each container egress VCN 1568(1)-(N) can include a NAT gateway 1538 communicatively coupled to the public internet 1554 (e.g., the public internet 1354 of FIG. 13).

[0190] The internet gateway 1534 included in the control plane VCN 1516 and the data plane VCN 1518 can be communicatively coupled to a metadata management service 1552 (e.g., the metadata management system 1352 of FIG. 13) communicatively coupled to the public internet 1554. The public internet 1554 can be communicatively coupled to the NAT gateway 1538 included in the control plane VCN 1516 and the data plane VCN 1518. The service gateway 1536 included in the control plane VCN 1516 and the data plane VCN 1518 can be communicatively coupled to cloud services 1556.

[0191] In some embodiments, the data plane VCN 1518 may be integrated with the customer's tenancy 1570. This integration may in some cases be useful or desirable for the IaaS provider's customers, such as when they may want support when running code. A customer may provide code to run that may be disruptive, communicate with other customers' resources, or otherwise cause undesirable effects. In response, the IaaS provider may determine whether to execute code provided to the IaaS provider by the customer.

[0192] In some examples, a customer of an IaaS provider may grant the IaaS provider temporary network access rights and request a function connected to the data plane application layer 1546. The code for executing this function may be executed in VMs 1566(1) to (N), and this code need not be configured to execute elsewhere on the data plane VCN 1518. Each of VMs 1566(1) to (N) may be connected to the tenancy 1570 of one customer. Each container 1571(1) to (N) included in VMs 1566(1) to (N) may be configured to execute the code. In this case, there can be a two-fold separation (e.g., the containers 1571(1) to (N) executing the code, the containers 1571(1) to (N) may be included in at least VMs 1566(1) to (N) included in the untrusted application subnet 1562), which can help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging the networks of different customers. The containers 1571(1) to (N) may be communicatively coupled to the customer's tenancy 1570 and may be configured to send or receive data with the customer's tenancy 1570. The containers 1571(1) to (N) need not be configured to send or receive data with any other entity within the data plane VCN 1518. Upon completion of the execution of the code, the IaaS provider may force-terminate or otherwise discard the containers 1571(1) to (N).

[0193] In some embodiments, the trusted application subnet 1560 can execute code that can be owned or operated by an IaaS provider. In this embodiment, the trusted application subnet 1560 may be communicatively coupled to the DB subnet 1530 and may be configured to perform CRUD operations within the DB subnet 1530. The untrusted application subnet 1562 may be communicatively coupled to the DB subnet 1530, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 1530. The containers 1571(1)-(N) that can execute customer code, which may be included in each customer's VMs 1566(1)-(N), do not have to be communicatively coupled to the DB subnet 1530.

[0194] In other embodiments, the control plane VCN 1516 and the data plane VCN 1518 do not have to be directly communicatively coupled. In this embodiment, there does not have to be a direct communication between the control plane VCN 1516 and the data plane VCN 1518. However, communication can occur indirectly by at least one method. The LPG 1510 may be established by the IaaS provider, which can facilitate communication between the control plane VCN 1516 and the data plane VCN 1518. In another example, the control plane VCN 1516 or the data plane VCN 1518 can make calls to the cloud service 1556 via the service gateway 1536. For example, a call from the control plane VCN 1516 to the cloud service 1556 can include a request for a service that can communicate with the data plane VCN 1518.

[0195] FIG. 16 is a block diagram 1600 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1602 (e.g., service operator 1302 of FIG. 13) can be communicatively coupled to a secure host tenancy 1604 (e.g., secure host tenancy 1304 of FIG. 13) that can include a virtual cloud network (VCN) 1606 (e.g., VCN 1306 of FIG. 13) and a secure host subnet 1608 (e.g., secure host subnet 1308 of FIG. 13). The VCN 1606 can include an LPG 1610 (e.g., LPG 1310 of FIG. 13) and can be communicatively coupled to an SSH VCN 1612 (e.g., SSH VCN 1312 of FIG. 13) via the LPG 1610 included in the SSH VCN 1612. The SSH VCN 1612 can include an SSH subnet 1614 (e.g., SSH subnet 1314 of FIG. 13), and the SSH VCN 1612 can be communicatively coupled to a control plane VCN 1616 (e.g., control plane VCN 1316 of FIG. 13) via the LPG 1610 included in the control plane VCN 1616 and to a data plane VCN 1618 (e.g., data plane 1318 of FIG. 13) via the LPG 1610 included in the data plane VCN 1618. The control plane VCN 1616 and the data plane VCN 1618 can be included in a service tenancy 1619 (e.g., service tenancy 1319 of FIG. 13).

[0196] The control plane VCN 1616 can include a control plane DMZ layer 1620 (e.g., the control plane DMZ layer 1320 of FIG. 13) that can include an LB subnet 1622 (e.g., the LB subnet 1322 of FIG. 13), a control plane application layer 1624 (e.g., the control plane application layer 1324 of FIG. 13) that can include an application subnet 1626 (e.g., the application subnet 1326 of FIG. 13), and a control plane data layer 1628 (e.g., the control plane data layer 1328 of FIG. 13) that can include a DB subnet 1630 (e.g., the DB subnet 1530 of FIG. 15). The LB subnet 1622 included in the control plane DMZ layer 1620 can be communicatively coupled to the application subnet 1626 included in the control plane application layer 1624 that can be included in the control plane VCN 1616, and to an Internet gateway 1634 (e.g., the Internet gateway 1334 of FIG. 13). The application subnet 1626 can be communicatively coupled to the DB subnet 1630 included in the control plane data layer 1628, and to a service gateway 1636 (e.g., the service gateway of FIG. 13) and a network address translation (NAT) gateway 1638 (e.g., the NAT gateway 1338 of FIG. 13). The control plane VCN 1616 can include the service gateway 1636 and the NAT gateway 1638.

[0197] The data plane VCN 1618 can include a data plane application layer 1646 (e.g., the data plane application layer 1346 of FIG. 13), a data plane DMZ layer 1648 (e.g., the data plane DMZ layer 1348 of FIG. 13), and a data plane data layer 1650 (e.g., the data plane data layer 1350 of FIG. 13). The data plane DMZ layer 1648 can include a trusted application subnet 1660 (e.g., the trusted application subnet 1560 of FIG. 15) and an untrusted application subnet 1662 (e.g., the untrusted application subnet 1562 of FIG. 15) of the data plane application layer 1646, and an LB subnet 1622 communicatively coupled to an Internet gateway 1634 included in the data plane VCN 1618. The trusted application subnet 1660 can be communicatively coupled to a service gateway 1636 included in the data plane VCN 1618, a NAT gateway 1638 included in the data plane VCN 1618, and a DB subnet 1630 included in the data plane data layer 1650. The untrusted application subnet 1662 can be communicatively coupled to a service gateway 1636 included in the data plane VCN 1618, and a DB subnet 1630 included in the data plane data layer 1650. The data plane data layer 1650 can include a DB subnet 1630 communicatively coupled to a service gateway 1636 included in the data plane VCN 1618.

[0198] The untrusted application subnet 1662 can include primary VNICs 1664(1) to (N) communicatively coupled to tenant virtual machines (VMs) 1666(1) to (N) present within the untrusted application subnet 1662. Each tenant VM 1666(1) to (N) can execute code within respective containers 1667(1) to (N) and can be communicatively coupled to an application subnet 1626 that can be included in a data plane application layer 1646 that can be included in a container egress VCN 1668. Each secondary VNIC 1672(1) to (N) can facilitate communication between the untrusted application subnet 1662 included in the data plane VCN 1618 and the application subnet included in the container egress VCN 1668. The container egress VCN can include a NAT gateway 1638 communicatively coupled to a public internet 1654 (e.g., the public internet 1354 of FIG. 13).

[0199] The internet gateway 1634 included in the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to a metadata management service 1652 (e.g., the metadata management system 1352 of FIG. 13) communicatively coupled to a public internet 1654. The public internet 1654 can be communicatively coupled to a NAT gateway 1638 included in the control plane VCN 1616 and the data plane VCN 1618. The service gateway 1636 included in the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to a cloud service 1656.

[0200] In some examples, the pattern shown by the architecture of block diagram 1600 of FIG. 16 may be considered an exception to the pattern shown by the architecture of block diagram 1500 of FIG. 15 and may be desirable for customers of an IaaS provider when the IaaS provider cannot communicate directly with a customer (e.g., a disconnected region). Each container 1667(1)-(N) included in VM1666(1)-(N) for each customer may be accessed in real time by the customer. Containers 1667(1)-(N) may be configured to make calls to respective secondary VNICs 1672(1)-(N) included in app subnet 1626 of data plane app layer 1646 that may be included in container egress VCN 1668. Secondary VNICs 1672(1)-(N) may be able to send the calls to NAT gateway 1638, and NAT gateway 1638 may be able to send the calls to public internet 1654. In this example, containers 1667(1)-(N) that may be accessed in real time by the customer can be separated from control plane VCN 1616 and may be separated from other entities included in data plane VCN 1618. Containers 1667(1)-(N) may also be separated from resources of other customers.

[0201] In another example, a customer can call cloud service 1656 using containers 1667(1) to (N). In this example, the customer may execute the code within containers 1667(1) to (N) that requests a service from cloud service 1656. Containers 1667(1) to (N) can send this request to secondary VNICs 1672(1) to (N), and secondary VNICs 1672(1) to (N) can send this request to a NAT gateway, and the NAT gateway can send this request to public internet 1654. Public internet 1654 can send this request to the LB subnet 1622 included in control plane VCN 1616 via internet gateway 1634. In response to determining that this request is valid, the LB subnet can send this request to app subnet 1626, and app subnet 1626 can send this request to cloud service 1656 via service gateway 1636.

[0202] It should be understood that the IaaS architectures 1300, 1400, 1500, 1600 shown in the figures may include components other than those shown. Further, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may include more or fewer components than those shown in the figures, combine two or more components, or may have different configurations or arrangements of components.

[0203] In one embodiment, the IaaS system described herein may include the provision of a series of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0204] FIG. 17 shows an exemplary computer system 1700 in which various embodiments may be implemented. System 1700 may be used to implement any of the computer systems described above. As shown in the figure, computer system 1700 includes a processing unit 1704 that communicates with a plurality of peripheral subsystems via a bus subsystem 1702. These peripheral subsystems may include a processing acceleration unit 1706, an I / O subsystem 1708, a storage subsystem 1718, and a communication subsystem 1724. Storage subsystem 1718 includes tangible computer-readable storage media 1722 and system memory 1710.

[0205] The bus subsystem 1702 provides a mechanism for the various components and subsystems of the computer system 1700 to communicate with each other as intended. Although the bus subsystem 1702 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1702 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus that uses any of a variety of bus architectures. For example, such architectures can include an ISA (Industry Standard Architecture) bus, an MCA (Micro Channel Architecture) bus, an EISA (Enhanced ISA) bus, a VESA (Video Electronics Standards Association) local bus, and a PCI (Peripheral Component Interconnect) bus implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard.

[0206] The processing unit 1704, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1700. One or more processors may be included in the processing unit 1704. These processors can include single-core processors or multi-core processors. In certain embodiments, the processing unit 1704 may be implemented as one or more independent processing units 1732 and / or 1734, with a single-core processor or multi-core processor included in each processing unit. In other embodiments, the processing unit 1704 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0207] In various embodiments, the processing unit 1704 can execute various programs according to program code and can maintain a plurality of simultaneously executing programs or processes. At any given time, some or all of the program code being executed can be present in the processor 1704 and / or the storage subsystem 1718. With appropriate programming, the processor 1704 can provide the various functions described above. The computer system 1700 can further include a processing acceleration unit 1706 that can include a digital signal processor (DSP), an application specific processor, and / or the like.

[0208] The I / O subsystem 1708 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated in a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. The user interface input devices may enable a user to control input devices such as a Microsoft Xbox (registered trademark) 360 game controller and exchange information via a natural user interface using gestures and spoken commands, and may include motion detection devices and / or gesture recognition devices such as a Microsoft Kinect (registered trademark) motion sensor. The user interface input devices may include gesture recognition devices such as a Google Glass (registered trademark) blink detector that detects a user's eye activity (e.g., a "blink" when taking a photo and / or selecting a menu) and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Further, the user interface input devices may include a voice recognition detection device that enables a user to interact with a voice recognition system (e.g., a Siri (registered trademark) navigator) via a voice command.

[0209] The user interface input device may include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, gamepad, and graphic tablet, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser distance meters, and eye tracking devices. Further, the user interface input device may include medical image input devices such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasonic examination devices. The user interface input device may include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0210] The user interface output device may include, among others, visual displays other than display subsystems, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as a flat panel device using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, a touch screen, or the like. Generally, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1700 to the user or another computer. For example, the user interface output device may include, but is not limited to, various display devices for visually communicating text information, graphics information, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0211] The computer system 1700 may comprise a storage subsystem 1718 that includes software elements as shown currently within system memory 1710. The system memory 1710 may store data generated during the execution of these programs in addition to program instructions that are readable and executable by the processing unit 1704.

[0212] Depending on the configuration and type of the computer system 1700, the system memory 1710 can be volatile (such as random-access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM is typically immediately accessible by the processing unit 1704 and / or contains data and / or program services that are currently being operated on and executed. In some implementations, the system memory 1710 can include multiple different types of memory, such as static random access memory (SRAM) or dynamic random-access memory (DRAM). In some implementations, a basic input / output system (BIOS) that includes basic routines useful for transferring information between elements within the computer system 1700, such as during startup, can typically be stored in ROM. By way of example and not limitation, the system memory 1710 also shows an application program 1712 that can include a client application, a web browser, a middle-tier application, relational database management systems (RDBMS), etc., program data 1714, and an operating system 1716.Examples of the operating system 1716 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0213] The storage subsystem 1718 may provide a tangible computer-readable storage medium for storing the basic programming and data configurations that provide the functionality of some embodiments. Software (programs, code services, instructions) that provides the aforementioned functionality when executed by a processor may be stored in the storage subsystem 1718. These software services or instructions may be executed by the processing unit 1704. The storage subsystem 1718 may provide a repository for storing data used in accordance with the present disclosure.

[0214] The storage subsystem 1700 may include a computer-readable storage medium reader 1720 that may be further connected to a computer-readable storage medium 1722. In combination with the system memory 1710, together, optionally, the computer-readable storage medium 1722 may comprehensively represent a storage medium for temporarily and / or more persistently containing, storing, transmitting, and retrieving computer-readable information, in addition to remote storage devices, local storage devices, fixed storage devices, and / or removable storage devices.

[0215] A computer-readable storage medium 1722 that includes code or a portion of code can include any suitable medium known in or used in the art, including storage media and communication media implemented in any way or technology for storing and / or transmitting information, such as volatile and non-volatile, removable and non-removable media, but not limited to these. The computer-readable storage medium 1722 can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer-readable media. The computer-readable storage medium 1722 can also include non-tangible computer-readable media such as data signals, data transmissions, or any other medium that can be used to transmit the desired information and can be accessed by the computing system 1700.

[0216] As an example, computer-readable storage medium 1722 may include a hard disk drive that reads from or writes to a removable non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk such as a CD ROM, a DVD, and a Blu-ray (registered trademark) disk, or other optical media. Computer-readable storage medium 1722 may include, but is not limited to, a Zip (registered trademark) drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. Computer-readable storage medium 1722 may include SSDs based on non-volatile memory such as flash memory-based semiconductor drives (SSDs), enterprise flash drives, semiconductor ROMs, SSDs based on volatile memory such as semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program services, and other data of computer system 1700.

[0217] The communication subsystem 1724 provides an interface to other computer systems and networks. The communication subsystem 1724 functions as an interface for receiving data from other systems of the computer system 1700 and for transmitting data to other systems. For example, the communication subsystem 1724 may enable the computer system 1700 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1724 can include components of a radio frequency (RF) transceiver for accessing wireless voice and / or data networks (such as cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (registered trademark) (using IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof), components of a global positioning system (GPS) receiver, and / or other components. In some embodiments, the communication subsystem 1724 can provide a wired network connection (such as Ethernet (registered trademark)) in addition to, or instead of, the wireless interface.

[0218] In some embodiments, the communication subsystem 1724 may receive input communications in the form of structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc., on behalf of one or more users who may use the computer system 1700.

[0219] As an example, the communication subsystem 1724 can be configured to receive the data feed 1726 in real time from users of social networks such as Twitter (registered trademark) feeds, Facebook (registered trademark) updates, and / or other communication services, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0220] Furthermore, the communication subsystem 1724 may be configured to receive data in the form of a continuous data stream, which can include an event stream 1728 of real-time events and / or event updates 1730 that have no explicit end and are essentially continuous or borderless. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0221] The communication subsystem 1724 may be configured to output structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc. to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 1700.

[0222] The computer system 1700 can be one of various types, including handheld portable devices (e.g., iPhone (registered trademark) mobile phones, iPad (registered trademark) computing tablets, PDAs), wearable devices (e.g., Google Glass (registered trademark) head-mounted displays), PCs, workstations, mainframes, ticket vending machines, server racks, or any other data processing system.

[0223] Due to the constantly changing nature of computers and networks, the description of the computer system 1700 shown in the figures is merely intended to be a specific example. Many other configurations are possible that include more or fewer components than the system shown in the figures. For example, customized hardware may be used and / or certain elements may be implemented in hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other methods and / or ways to implement various embodiments.

[0224] Although specific embodiments have been described, various modifications, changes, alternative structures, and equivalents are also encompassed within the scope of the present disclosure. Embodiments are not limited to operating within a particular data processing environment and can operate freely within multiple data processing environments. Further, although embodiments have been described using a particular series of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the series of transactions and steps described. The various features and aspects of the foregoing embodiments may be used individually or together.

[0225] Furthermore, although embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented using only hardware, only software, or combinations thereof. The various processes described herein may be implemented on the same processor or different processors in any combination. Thus, when a component or service is described as being configured to perform an operation, such a configuration may be realized, for example, by designing an electronic circuit to perform this operation, by programming a programmable electronic circuit (such as a microprocessor) to perform this operation, or by any combination thereof. Processes can communicate using a variety of techniques including, but not limited to, prior art techniques for interprocess communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0226] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is clear that additions, deletions, omissions, and other modifications and changes may be made without departing from the broader ideas and scope as set forth in the claims. Thus, while specific embodiments of the disclosure have been described, these are not intended to be limiting. Various changes and equivalents are within the scope of the appended claims.

[0227] The use of the terms "a", "an", and "the" and similar referents in the context of describing the disclosed embodiments (in particular, in the context of the appended claims) should be construed to cover both the singular and the plural unless specifically indicated otherwise herein or clearly contradicted by the context. The terms "comprising", "having", "including", and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" should be construed to mean either internally or partially or fully contained within, connected to, or joined together with, even if there is something intervening. The recitation of a range of values herein is merely intended to provide a convenient way of referring individually to each separate value falling within the range, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The use of any examples, or exemplary language (e.g., "such as") provided herein is merely intended to better illuminate the embodiments and does not impose a limitation on the scope of the disclosure unless otherwise claimed. No language in this specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0228] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally intended, unless otherwise explicitly stated, to convey in context that items, conditions, etc. can be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended, and should not be taken, to mean that a particular embodiment requires the presence of at least one of each of at least one of X, at least one of Y, or at least one of Z.

[0229] In this specification, preferred embodiments of the disclosure are described, including the best mode known to the applicant for carrying out the disclosure. Variations of such preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to adopt such variations as appropriate, and the disclosure may be practiced otherwise than as specifically described herein. Accordingly, the disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Further, any combination of the foregoing elements in all possible variations of the embodiments is included in the disclosure unless otherwise specifically indicated herein.

[0230] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference in their entirety, to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0231] In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments of the present specification. Those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or together. Furthermore, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the present specification and drawings are to be regarded as illustrative rather than restrictive.

[0232] Embodiments may be implemented by using a computer program product that includes computer programs / instructions, which, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

Claims

1. A method performed by a computer, The method includes generating a snapshot difference between a first snapshot and a second snapshot of a source file system located in a source region, wherein the snapshot difference is the difference between the first snapshot and the second snapshot, the second snapshot being created after the first snapshot, and the method further includes The snapshot difference is encrypted using a session encryption key that is valid during the session, The method further includes uploading the encrypted snapshot difference from the source file system to a first object storage located in the target region, wherein the source region and the target region are different regions, and the method further includes Simultaneously with detecting at least a portion of the encrypted snapshot differences that reach the first object storage, the encrypted snapshot differences are downloaded from the first object storage to the target file system before the source file system completes uploading all of the snapshot differences. The target file system decrypts the encrypted snapshot difference using the session encryption key, A method performed by a computer, comprising applying the decrypted snapshot difference to a third snapshot of the target file system to create a fourth snapshot of the target file system, wherein the fourth snapshot is a copy of the second snapshot.

2. The method performed by the computer according to claim 1, wherein the session begins when the second snapshot is created on the source file system and ends when the fourth snapshot is created on the target file system.

3. The computer method according to claim 1, wherein the difference between the first snapshot and the second snapshot includes the keys and values ​​of a binary tree (B-tree) and associated file data.

4. A method performed by a computer according to claim 1, further comprising transferring the session encryption key from the source file system to the target file system via a protected encryption process.

5. Uploading the encrypted snapshot difference comprises uploading a first portion of the encrypted snapshot difference by a first upload thread in the source file system and uploading a second portion of the encrypted snapshot difference by a second upload thread in the source file system, wherein the first upload thread and the second upload thread operate in parallel and independently, a method performed by the computer according to claim 1.

6. The method performed by a computer according to claim 5, further comprising the action of a third upload thread in the source file system taking over from the first upload thread and continuing the upload process when the first upload thread in the source file system detects a failure of the first upload thread in the source file system.

7. The method performed by the computer according to claim 1, wherein downloading the encrypted snapshot difference includes downloading a first portion of the encrypted snapshot difference by a first download thread in the target file system and downloading a second portion of the encrypted snapshot difference by a second download thread in the target file system, the first download thread and the second download thread operating in parallel and independently.

8. The method performed by a computer according to claim 1, further comprising reversing the roles of the source file system and the target file system.

9. The computer method according to claim 8, wherein reversing the aforementioned roles includes uploading a new snapshot difference from the target file system to a second object storage located in the source region, and downloading the new snapshot difference from the second object storage to the source file system.

10. A program for causing a computer to perform the method described in any one of Claims 1 to 9.

11. It is a system, One or more processors, The system comprises one or more non-temporary computer-readable media storing executable instructions, wherein when an instruction is executed by one or more processors, the system... The command causes the system to generate a snapshot difference between a first snapshot and a second snapshot of the source file system located in the source region, wherein the snapshot difference is the difference between the first snapshot and the second snapshot, the second snapshot being created after the first snapshot, and the command further causes the system to: The snapshot difference is encrypted using a session encryption key that is valid during the session, The system is instructed to upload the encrypted snapshot difference from the source file system to a first object storage located in the target region, wherein the source region and the target region are different regions, and the instruction further instructs the system to perform the following actions: Simultaneously with detecting at least a portion of the encrypted snapshot differences that reach the first object storage, the encrypted snapshot differences are downloaded from the first object storage to the target file system before the source file system completes uploading all of the snapshot differences. The target file system decrypts the encrypted snapshot difference using the session encryption key, The system performs the following actions: apply the decrypted snapshot difference to the third snapshot of the target file system to create a fourth snapshot of the target file system, wherein the fourth snapshot is a copy of the second snapshot.

12. The system according to claim 11, wherein the session begins when the second snapshot is created on the source file system and ends when the fourth snapshot is created on the target file system.

13. The system according to claim 11 or 12, further comprising transferring the session encryption key from the source file system to the target file system via a protected encryption process.

14. The system according to claim 11 or 12, wherein uploading the encrypted snapshot difference includes uploading a first portion of the encrypted snapshot difference by a first upload thread in the source file system and uploading a second portion of the encrypted snapshot difference by a second upload thread in the source file system, the first upload thread and the second upload thread operate in parallel and independently.

15. The system according to claim 11 or 12, further comprising reversing the roles of the source file system and the target file system by uploading a new snapshot difference from the target file system to a second object storage located in the source region, and downloading the new snapshot difference from the second object storage to the source file system.

Images