Hierarchical key management for cross-region replication

Abstract

A technique for performing hierarchical key management is described, involving end-to-end file storage replication between different cloud infrastructure regions. Hierarchical key management includes three distinct keys: a first security key for the source region, a session key valid only within the session for transferring data between the two different regions, and a second security key for the target region. A novel technique for using different file keys for different files in the file system within each region is also described.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. non-provisional application No. 18 / 094,302, filed January 6, 2023, entitled “HIERARCHICAL KEY MANAGEMENT FOR CROSS-REGION REPLICATION,” which claims the benefit and priority of U.S. provisional application No. 63 / 352,992, filed June 16, 2022; U.S. provisional application No. 63 / 357,526, filed June 30, 2022; U.S. provisional application No. 63 / 412,243, filed September 30, 2022; and U.S. provisional application No. 63 / 378,486, filed October 5, 2022, the disclosures of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This disclosure generally relates to file systems. More specifically, but not limited to, it describes key management techniques for end-to-end file storage replication between different cloud infrastructure regions. Background Technology

[0004] Business operations involve critical data. While a robust disaster recovery solution is important, security is also an essential aspect. Critical data needs to be secured both when it is at rest in the data center and during transit between data centers during failover and recovery. Therefore, security is a critical requirement in disaster recovery. Summary of the Invention

[0005] This disclosure generally relates to file systems. More specifically, but not limited to, it describes key management techniques for end-to-end file storage replication between different cloud infrastructure regions.

[0006] In some embodiments, a technique including a method is provided, comprising: generating, by a computing system, at least partially based on a first master key, a first security key associated with a source file system for encrypting and decrypting multiple file keys in the source file system, the source file system being configured to send snapshot increments during copy processing, the snapshot increments being identified between two snapshots of the source file system; generating, by the computing system, at least partially based on a second master key, a second security key associated with a target file system for encrypting and decrypting multiple file keys in the target file system, the target file system being configured to receive snapshot increments during copy processing; and generating, by the computing system, at least partially based on a third master key, a session key for encrypting and decrypting snapshot increments transmitted between the source file system and the target file system during copy processing, the session key being valid for a session; wherein the first master key, the second master key, and the third master key are different keys.

[0007] In yet another embodiment, a session is the duration between the start of a copy process in the source file system and the end of a copy process in the target file system.

[0008] In yet another embodiment, the snapshot increment transferred between the source file system and the target file system further includes an object storage device configured to receive the snapshot increment from the source file system and forward the snapshot increment to the target file system.

[0009] In yet another embodiment, the method further includes encrypting the snapshot increment with a session key before transmitting the snapshot increment to the object storage device, and decrypting the snapshot increment with the session key after transmitting the snapshot increment to the target file system.

[0010] In yet another embodiment, the method further includes transferring a session key from the control plane of the source file system to the control plane of the target file system.

[0011] In yet another embodiment, the session key is associated with a globally unique resource identifier.

[0012] In yet another embodiment, each of a plurality of file keys in the source file system is associated with a specific file in the source file system and is used to encrypt and decrypt file data of the specific file in the source file system, and each of a plurality of file keys in the target file system is associated with a specific file in the target file system and is used to encrypt and decrypt file data of the specific file in the target file system.

[0013] In yet another embodiment, the method further includes authenticating a key requester in the source file system who requests to use the first security key; and authenticating a key requester in the target file system who requests to use the second security key.

[0014] In yet another embodiment, authenticating the key requester in the source file system includes checking the identifier of the copy process and the identifier of the source file system.

[0015] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the one or more methods disclosed herein.

[0016] In various embodiments, a non-transitory computer-readable medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors of a computer system to perform one or more methods disclosed herein.

[0017] In various embodiments, a computer program product includes a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0018] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts represent only a small fraction of the many implementations and contexts. Attached Figure Description

[0019] The features, embodiments, and advantages of this disclosure can be better understood when reading the following detailed description with reference to the accompanying drawings.

[0020] Figure 1 Example concepts of recovery time objective (RTO) and recovery point objective (RPO) according to certain embodiments are depicted.

[0021] Figure 2 This is a simplified block diagram illustrating a cross-regional remote replication architecture according to certain embodiments.

[0022] Figure 3 This is a simplified schematic diagram of the components involved in cross-regional remote replication according to certain embodiments.

[0023] Figure 4 This is a simplified flowchart of the steps performed during cross-regional remote replication according to certain embodiments.

[0024] Figure 5 This is a simplified diagram illustrating a high-level concept of B-tree traversal according to certain embodiments.

[0025] Figure 6A This is a diagram illustrating the pipeline stages of cross-regional replication according to certain embodiments.

[0026] Figure 6B This is a diagram illustrating the pipeline stages of cross-region replication according to certain embodiments.

[0027] Figure 7 This is a diagram illustrating the hierarchical structure in the data plane of a file storage service (FSS) according to certain embodiments.

[0028] Figure 8 A simplified example binary large object (BLOB) format according to certain embodiments is described.

[0029] Figure 9 An example replication bucket format according to certain embodiments is described.

[0030] Figure 10 This is a flowchart illustrating a state machine for concurrent source upload and target download according to certain embodiments.

[0031] Figure 11 This is an example flowchart illustrating the interaction between the data plane and the control plane in the source region according to certain embodiments.

[0032] Figure 12 This is a simplified diagram illustrating a failback mode according to certain embodiments.

[0033] Figure 13 This is a flowchart illustrating key management for cross-regional replication according to certain embodiments.

[0034] Figure 14 This is a flowchart illustrating the session key generation steps according to certain embodiments.

[0035] Figure 15 A simplified table format (or schema) for storing encrypted session keys in a shared database (SDB) according to certain embodiments is described.

[0036] Figure 16 This is a flowchart illustrating an end-to-end key management process for cross-regional replication according to certain embodiments.

[0037] Figure 17 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0038] Figure 18 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0039] Figure 19This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0040] Figure 20 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0041] Figure 21 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0042] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it is clear that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The term “exemplary” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0043] This disclosure generally relates to file systems. More specifically, but not limited to, it describes key management techniques for end-to-end file storage replication between different cloud infrastructure regions.

[0044] In some embodiments, the File Storage Service (FSS) disclosed herein utilizes a three-tier key architecture for cross-region replication. Three distinct keys may be involved during the replication process: a source file system key (FSKa), a session key, and a destination file system key (FSKb). Within each source and destination region, the file system keys are also hierarchically structured to encrypt local files, and a master key from the client is used to securely store the file system keys.

[0045] In some embodiments, the source file system has three keys: a source file key (FKa) for each file, a source file system key (FSKa), and a master key (a client-managed key). The source file system has multiple files, each with its own file key FKa for encryption and decryption. The target file system has three keys: a target file key (FKb) for each file, a target file system key (FSKb), and a master key (a client-managed key). In some embodiments, the master key of the source file system is different from the master key of the target file system.

[0046] Explanation of terms in some embodiments

[0047] In some embodiments, the "recovery time objective" (RTO) refers to the time a user requires their replica to become available in the secondary (or target) region after a failure in the availability domain (AD) of the primary (or source) region, regardless of whether the failure was planned or unplanned.

[0048] In some embodiments, the "recovery point objective" (RPO) refers to the maximum acceptable tolerance for data loss in terms of time between a primary area failure (typically due to unplanned failure) and the availability of a secondary area.

[0049] In some embodiments, a "replicator" may refer to a component (e.g., a virtual machine (VM)) in the file system's data plane that serves the purpose of: uploading increments to a remote object repository (i.e., an object storage service) if the component is located in a source region, or downloading increments from an object storage device for incremental application if the component is located in a target region. Replicators may be organized as queues (i.e., multiple VMs or replicator threads), referred to as replicator queues, to perform cross-region (or X-region) replication processing (e.g., uploading increments to a target region) in parallel.

[0050] In some embodiments, an "increment generator" (DG) may refer to a component in the file system's data plane that is used to: extract the increment (i.e., change) between the key values ​​of two snapshots if the component is located in a source region, or apply the increment to the latest snapshot in the file system's B-tree if the component is located in a target region. The increment generator in the source region can use several threads (referred to as increment generator threads or range threads for multiple partition B-tree key ranges) to perform increment extraction (or B-tree visits) in parallel. The increment generator in the target region can use several threads to apply the downloaded increment in parallel to its latest snapshot.

[0051] For the purposes of this disclosure and in some embodiments, a “shared database” (SDB) may refer to a key-value store through which components in the control plane and data plane of the file system (e.g., replicator queues) can read and write to communicate with each other.

[0052] In some embodiments, the SDB may be part of a B-tree.

[0053] In some embodiments, a "File System Communicator" (FSC) can refer to a file manager layer running on storage nodes in the file system's data plane. This service facilitates file creation, deletion, read, and write requests and collaborates with NFS servers (e.g., Orca) to provide I / O services to clients. Replicator queues can communicate with numerous storage nodes, thereby distributing the work of reading / writing file system data across the storage nodes.

[0054] In some embodiments, "blob" can refer to a data type used to store information (e.g., a formatted binary file) in a database. A blob is generated from the source region during replication and uploaded to the object repository (i.e., object storage device) in the target region. A blob can include binary tree (B-tree) keys and values ​​as well as file data. A blob in the object repository is referred to as an object. B-tree key-value pairs and their associated data are packaged together in a blob for uploading to the object repository in the target region.

[0055] In some embodiments, a "manifest" can refer to information communicated by the file system in the source region (referred to herein as the source file system) to the file system in the target region (referred to herein as the target file system) to facilitate cross-region replication processing. There are two types of manifest files: the master manifest and the checkpoint manifest. The range manifest file (or master manifest file) is created by the source file system at the start of replication processing and describes the information expected by the target file system (e.g., B-tree key ranges). The checkpoint manifest file is created after a checkpoint in the source file system and informs the target file system of the number of blobs included in the checkpoint and uploaded to the object repository, allowing the target file system to download that number of blobs accordingly.

[0056] In some embodiments, "increment" can refer to the difference identified between two given snapshots after the replicator recursively visits each node of the B-tree (also referred to herein as a B-tree traversal). The increment generator identifies pairs of differing B-tree key values ​​and traverses the B-tree nodes to obtain the file data associated with the B-tree keys. The increment between two snapshots can contain multiple blobs. When used in the context of a source file system uploading information to an object repository and a target file system downloading it from the object repository, the term "increment" can include both blobs and manifests.

[0057] In some embodiments, an "object" may refer to a partial collection of information representing an entire increment during a cross-region replication cycle and is stored in an object repository. An object can be several MB in size and stored in a specific location within a bucket of the object repository. An object can contain many increments (i.e., blobs and manifests). A blob uploaded to and stored in the object repository is called an object.

[0058] In some embodiments, a "bucket" may refer to a container that stores objects in a compartment within an object storage namespace (lease). In this disclosure, the bucket is used by the source replicator to store secure increments using server-side encryption (SSE) and is also used by the target replicator to download changes to apply to snapshots.

[0059] In some embodiments, "incremental application" can refer to the process of applying incremental downloads to the target file system to its latest snapshot to create a new snapshot. This may include analyzing the manifest file, applying snapshot metadata, inserting B-tree keys and values ​​into its B-tree, and storing the data associated with the B-tree keys (i.e., the file data or the data portion of the blob) to its local storage. Snapshot metadata is created and applied at the start of the replication cycle.

[0060] In some embodiments, "region" may refer to a logical abstraction corresponding to a geographical location. Each region may include one or more connected data centers. Regions are independent of other regions and may be geographically distant.

[0061] End-to-end cross-region replication architecture

[0062] The end-to-end cross-region replication architecture provides novel techniques for end-to-end file storage replication and security between file systems in different cloud infrastructure regions. In some embodiments, the file storage service generates increments between snapshots in the source file system and transfers the increments and associated data via a high-throughput object storage device to recreate new snapshots in target file systems located in different regions during disaster recovery. The file storage service leverages novel techniques to achieve scalable, reliable, and restartable end-to-end replication. Novel techniques for ensuring the secure delivery and consistency of information during end-to-end replication are also described.

[0063] In the context of the cloud, a region refers to a local geographical area that includes one or more connected data centers. Regions are independent of other regions and can be geographically distant—spanning countries or even continents. A Realm is a logical collection of one or more regions. Realms are typically isolated from each other and do not share data. Within a Realm, data centers within a Realm can be organized into one or more Availability Domains (ADs). Availability Domains are isolated from each other, fault-tolerant, and unlikely to fail simultaneously. ADs are configured such that a failure at one AD within a Realm is unlikely to affect the availability of other ADs within the same Realm.

[0064] Current disaster recovery practices may involve periodically taking snapshots and resynchronizing them to another file system in a different availability domain (AD) or zone. While resynchronization can be managed and maintained by the customer, it lacks a user interface for viewing progress, is a slow and serialized process, and becomes difficult to manage as data grows over time.

[0065] Therefore, different approaches are needed to address these and other challenges. The cloud service provider (e.g., Oracle Cloud Infrastructure (OCI)) file storage replication disclosed in this disclosure is based on incremental snapshots to provide a consistent point-in-time view of the entire file system by propagating incremental changes to data from the primary Active Directory (AD) in a region to secondary ADs located in the same or different regions. As used herein, the primary site (or source side) can refer to a location (e.g., AD or region) where the file system is located and initiates replication processing for disaster recovery. The secondary site (or target side) can refer to a location (e.g., AD or region) where the file system receives information from the file system in the primary site during replication processing to become the new operating file system after disaster recovery. The file system located in the primary site is referred to as the source file system, and the file system located in the secondary site is referred to as the target file system. Therefore, the terms primary site, source side, source region, primary file system, or source file system (referring to one of the file systems on the source side) can be used interchangeably. Similarly, the terms secondary site, target side, target region, secondary file system, or target file system (referring to one of the file systems on the target side) can be used interchangeably.

[0066] This File Storage Service (FSS) provides full disaster recovery with minimal administrative effort for failover or failback. Failover is a series of actions that make a secondary / target site the primary / source (i.e., begin serving the workload) and can include planned and / or unplanned failovers. A planned failover (also known as a planned migration) is initiated by the user to perform a planned failover from the source side (e.g., source region) to the target side (e.g., target region) without data loss. An unplanned failover occurs when the source side unexpectedly stops due to, for example, a disaster, and the user needs to start using the target side because the source side is lost. Failback is restoring the primary / source side to its previous state before the failover to become the primary / source again. Failback may occur when, after a planned or unplanned failover and the triggering event (e.g., an interruption) has ended, a user wishes to reuse the source side as their primary Active Directory by reversing the failover process. Users can recover from the last point in time on the source side before the triggering event, or from the latest changes on the target side. The replication process described in this disclosure can preserve the file system identity after round-trip replication. In other words, the source file system can be served again for workloads after a failover and subsequent failure recovery.

[0067] The techniques (e.g., methods, computer-readable media, and systems) disclosed in this disclosure include cross-region replication of file system data and / or metadata by: copying incremental values ​​between snapshots from a source region to multiple remote (or target) regions using consistent snapshot information, and then traversing (or recursively traversing) all keys and values ​​(sometimes referred to herein as "traversing a B-tree" or "traversing keys") in one or more file trees (e.g., B-trees) of the source file system to construct consistent information (e.g., the increment or difference between the keys and values ​​of two snapshots created at different times). The constructed consistent information is placed into a blob format and transmitted to the remote side (e.g., the target region) using an object interface (e.g., an object repository (described later)), such that the target file system on the remote side can immediately download and begin applying the information once it detects the transmission on the object interface. This process is accomplished using a control plane and can be scaled to thousands of file systems and hundreds of replication machines. The source and target file systems can operate concurrently and asynchronously. Concurrent operation means that data upload processing of the source file system and data download processing of the target file system can occur simultaneously. Asynchronous operation means that the source file system and the target file system can operate at their own pace without waiting for each other at each stage, such as different start times, end times, processing speeds, etc.

[0068] In some embodiments, multiple file systems may exist in the same region and be represented by the same B-tree. Each of these file systems in the same region can be copied independently across the region. For example, file system A may have a set of replicator threads running in parallel, which traverse the B-tree to perform a copy of file system A. File system B, represented by the same B-tree, may have another set of such parallel replicator threads, which traverse the same B-tree to perform a copy of file system B.

[0069] Regarding security, cross-region replication is completely secure. Information is securely transmitted and applied. The disclosed technology provides isolation between the source and target regions, ensuring that keys are not shared between them in an unencrypted manner. Therefore, if the source key is compromised, the target remains unaffected. Furthermore, the disclosed technology includes how to read keys, how to convert them to certain formats, and how to securely upload and download them. Different keys are created and used in different regions, thus creating a separate key on the target and applying it to information within a target-centric security mechanism. For example, FSS generates session keys valid only within a single replication cycle or session to encrypt data to be uploaded from the source region to the object repository and decrypt data downloaded from the object repository to the target region. Different keys are used locally in the source and target regions.

[0070] In the disclosed technology, each upload and download process through the object repository during replication has different pipeline stages. For example, the upload process has several pipeline stages, including traversing the B-tree to generate an increment, accessing storage I / O, and uploading the data (or blob) to the object repository. The download process has several pipeline stages, including downloading the data, applying the increment to a snapshot, and storing the data in the storage device. Each of these pipelines also has parallel processing threads to increase the throughput and performance of the replication process. Furthermore, the parallel processing threads can take over any failed processing threads and recover the replication process from the point of failure without having to start from scratch. Therefore, the replication process has high scalability and reliability.

[0071] Figure 1 Example concepts of Recovery Point Objective (RPO) and Recovery Time Objective (RTO) for unplanned failover, according to certain embodiments, are described. RPO is the maximum tolerance (typically specified in minutes) for data loss between primary site failure and secondary site availability. Figure 1As shown, primary site A 102 encounters an unplanned incident at time 110, triggering a failover replication process by copying the latest snapshot and its increments to secondary site B 104. The initially copied information arrives at secondary site B 104 at time 112. Primary site A 102 completes its copying of information to secondary site B 104 at time 114, and secondary site B 104 completes its replication process at time 116. Therefore, secondary site B 104 becomes fully operational at time 116. Thus, from point 110 until point 116 when the data is available again, user data is inaccessible at primary site A 110. Therefore, the Recovery Point Objective (RPO) is the time between point 110 and point 116. For example, if there are 10 minutes of data that users don't care about, then the RPO is 10 minutes. If data is lost for more than 10 minutes, then the RPO is not met. Zero RPO means synchronous replication.

[0072] RTO is the time (usually specified in minutes) required for a secondary site to become fully operational again, allowing users to access data once a failure has occurred. This is from the perspective of the secondary site. (Reference) Figure 1 Primary site A 102 initiates failover replication processing at time 120. However, secondary site B 104 remains operational until time 122 when it becomes aware of the incident (or outage) at primary site A 102. Therefore, secondary site B 104 ceases its service at time 122. Using a similar failover replication process described for RPO, secondary site B 104 becomes fully operational at time 126. Therefore, RTO is the time between 122 and 126. Secondary site B 104 can now assume the role of primary site. However, for customers using primary site A 102, the service loss occurred between times 120 and 126.

[0073] The primary (or source) site is where action is taking place, while the secondary (or target) site is inactive and unavailable until a disaster occurs. However, it's possible to provide clients with a point in time where they can continue using the secondary site for test-related activities. This involves how clients set up replication, how they start using the target when issues arise, and how they revert to the source once their source fails over.

[0074] Figure 2 This is a simplified block diagram illustrating an architecture for cross-regional remote replication according to certain embodiments. Figure 2The end-to-end replication architecture shown has two regions: a source region 290 and a destination region 292. Each region may contain one or more file systems. In some embodiments, the end-to-end replication architecture includes data planes 202 and 212, a control plane (only control APIs 208a-n and 218a-n are shown), local storage devices 204 and 214, an object repository 260, and a key management service (KMS) 250 for both the source region 290 and the destination region 292. For simplicity, Figure 2 The diagram only illustrates one file system 280 in source region 290 and one file system 282 in target region 292. If multiple file systems exist in the regions, the same replication architecture applies to each pair of source and target file systems. File systems in a region can share resources. For example, depending on the implementation, certain resources in KMS 250, object repository 260, and the data plane can be shared by many file systems in the same region.

[0075] The data plane in this architecture includes local storage nodes 204a-n and 214a-n, and replicators (or replicator queues) 206a-n and 216a-n. A control API host in each region handles all coordination between the different regions. The FSS receives requests from clients to establish replication between a source file system 280 and a target file system 282 to which the client wants to move their data. The control plane 208 receives the request, allocates resources, and notifies the replicator queues 206a-n in the source data plane 202 to begin uploading data 230a (or what may be referred to as incremental uploads) from different snapshots to object storage device 260. The API can be used to help clients set replication time targets and recovery time targets (RTOs). The replication model disclosed in this disclosure is a "push-based" model based on snapshot increments, meaning that replication is initiated by the source region.

[0076] As used herein, the data 230a and 230b transferred between the source file system 280 and the target file system 282 are general terms and may include the initial snapshot, the different keys and values ​​in the B-tree between the two snapshots, file data (e.g., fmap), snapshot metadata (i.e., a set of snapshot B-tree keys reflecting the individual snapshots taken in the source file system), and other information that helps facilitate the copying process (e.g., manifest file).

[0077] Moving to the data plane of the cross-region replication architecture, the replicator is a component within the file system's data plane. Depending on the region where the file system resides, it performs incremental generation or incremental application for that file system. For example, replicator queue 206 in the source region file system 280 performs incremental generation and replication 230a. Replicator queue 216 in the target region file system 282 downloads incremental 230b and applies it to the latest snapshot in the target region file system 282. The target region file system 282 can also use its control plane and workflow to ensure end-to-end delivery.

[0078] All incremental work is based on snapshots, which are existing resources in File Storage as a Service (FSS). A snapshot is a point in time, a data point, or a snapshot of what is happening in a file system, and is performed periodically in the source zone file system 280. For the first copy, FSS takes a base snapshot (e.g., one that has never been copied before) as a snapshot of all the contents of the source file system and transfers all of that content to the target system. In other words, the replicator reads from the storage layer of that particular file system and puts all the data into the object storage bucket.

[0079] Once the data plane 202 of the source file system 280 has uploaded all data 230a to the object storage device (or object repository) 260, the source-side control plane 208 notifies the target-side control plane 218 that there is new work to be done on the target side. This is then relayed to the replicators on the target side. The target-side replicators 216a-n then begin downloading objects (e.g., initial snapshots and increments) from the object storage bucket 260 and applying the increments captured on the source side.

[0080] If it's a base copy (e.g., the entire filesystem content up to a certain point in time, such as five days to five years), then the upload process could take a long time. To help achieve service level goals regarding time and performance, the source system 280 can take replication snapshots over a specific duration (such as one hour). The source side 280 can then transfer all the data within that one hour to the target side 282, and take a new snapshot every hour. If some caches have a large number of changes, then replication can be set to a lower replication interval.

[0081] To illustrate the above discussion, consider a scenario where a first snapshot is created in the file system in the source region (referred to as the source file system). Replication is performed periodically; therefore, the first snapshot is copied to the file system in the target region (referred to as the target file system). When some updates are subsequently performed in the source file system, a second snapshot is created. If an unplanned interruption occurs after the creation of the second snapshot, the source file system will attempt to copy the second snapshot to the target file system. During this failover, the source file system can identify the differences between the first and second snapshots (i.e., the increments), which include the B-tree keys and values ​​in the B-tree representing both the first and second snapshots and their associated file data. Increments 230a and 230b are then transferred from the source file system to the target file system via the object repository 260 in the target region so that the target file system can recreate the second snapshot by applying that increment to its previously created first snapshot in the target region. Once the second snapshot is created in the target file system, the failover replication process is complete, and the target file system is ready.

[0082] Moving to the control plane and its application programming interface (“API”), the control plane provides instructions to the data plane, which includes replicators as executors of these instructions. Both the storage devices (204 and 214) and the replicator queues (206 and 216) reside in the data plane. Figure 2 The control plane is not shown. As used herein, a "cycle" can refer to the duration from when the source file system 280 begins transferring data 230a to the target file system 282 until the target file system 282 receives all data 230b and completes its application of the received data. Data 230a-b is captured on the source side and then applied to the target side. Once all changes for one cycle have been applied on the target side, the source file system 280 takes another snapshot and begins another cycle.

[0083] The control APIs (208a-n and 218an) are a set of hosts in the overall architecture of the control plane and perform file system configuration. The control APIs are responsible for transferring state information between different zones. State machines, which track various state activities within a zone (such as job progress, key location, and future tasks to be performed), are distributed across multiple zones. All this information is stored in the control plane of each zone and transferred between zones via the control APIs. In other words, the state information concerns lifecycle details, increment details, and resource lifecycles. The state machines can also track replication progress and collaborate with the data plane to help estimate replication time. Therefore, the state machines can provide users with information about whether replication is on schedule and whether the job is healthy.

[0084] Furthermore, communication between the control API (208a-n) of the source file system 280 and the control API (218a-n) of the target file system 218 in different regions includes the transfer of snapshots and metadata to enable an accurate copy from source to target. For example, when a customer periodically takes snapshots in the source file system, the control plane can ensure that the same user snapshots are created on the target file system, including metadata tracking, transfer, and re-creation.

[0085] Figure 2 The object repository 260 (also referred to as "object" in this document) is an object storage service (e.g., Oracle's Object Storage Service) that allows reading blobs and writing them to files for archiving. The benefits of using an object repository are: first, it is easy to configure; second, it is easy to stream data to the object repository; and third, it offers the benefit of secure streaming, serving as a reliable repository for storing information; all because there is no network loss, so data can be downloaded immediately and permanently stored. While direct communication between replicators in the source and target areas is possible, it requires a cross-region network setup, which is neither scalable nor easy to manage.

[0086] For example, if a large amount of data needs to be moved from the source to the target, the source can upload it to object repository 260, and the target 282 does not have to wait for all the information to be uploaded to object repository 260 before starting to download. Therefore, source 280 and target 282 can operate concurrently and continuously. Using an object repository allows the system to scale and achieve faster throughput. Furthermore, the Key Management Service (KMS) 250 can control access to object repository 260 to ensure security. In other words, the source attempts to move data out of the source region as quickly as possible and keeps it somewhere before applying it to the target so that the data is not lost.

[0087] Compared to using network pipelining (which suffers from packet loss and recovery issues), using an object repository 260 between the source and destination regions enables continuous data streaming, allowing hundreds of file systems from the source region to write to the object repository while hundreds of files can be applied concurrently in the destination region. Therefore, high throughput can be achieved through data streaming via the object repository. Furthermore, both the source and destination regions can operate at their own rates for uploading and downloading.

[0088] Whenever a user modifies some data in the source file system 280, a snapshot is taken, and the increments before and after the change are updated. Changes can accumulate on the source file system 280 and stream to the object repository 260. The target file system 282 can detect that data is available in the object repository 260 and immediately download the changes and apply them to its file system. In some embodiments, only the increments are uploaded to the object storage device after the base snapshot.

[0089] In some embodiments, the replicator can communicate with many different regions (e.g., from Phoenix to Ashburn to other remote regions), and the file system can manage many different endpoints on the replicator. Each replicator 206 in the source file system 280 can maintain a cache of these object storage endpoints and can also cooperate with KMS 250 to generate transfer keys (e.g., session keys) to encrypt data addresses in the object repository 260 (e.g., server-side encryption or SSE) to protect data stored in buckets. A master bucket is used for each AD in the target region. A bucket is a container that stores objects in compartments within an object storage namespace (lease). All remote clients can communicate with the bucket and write information in a specific format so that information for each file system can be uniquely identified to avoid confusing data from different clients or file systems.

[0090] Object repository 260 is a high-throughput system, and the techniques disclosed in this disclosure can utilize object repository. In some embodiments, the replication process has several pipelined stages: B-tree visits in source file system 280, storage I / O access, uploading data to object repository 260, downloading data from object repository 260, and incremental application in target file system 282. Each stage involves parallel processing threads to improve the performance of streaming data from source region 290 to target region 292 through object repository 260.

[0091] In some embodiments, each file system in the source region may have a set of replicator threads 206a-n running in parallel to upload increments to the object repository 260. Each file system in the target region may also have a set of replicator threads 216a-n running in parallel to download increments from the object repository 260. Because the source and target sides operate concurrently and asynchronously, the source can upload as quickly as possible, while the target can begin downloading as soon as it detects that an increment is available in the object repository. The target file system then applies the increment to the latest snapshot and deletes it after applying the increment to the object repository. Therefore, the FSS occupies very little space in the object repository, and the object repository has very high throughput (e.g., gigabyte transfers).

[0092] In some embodiments, multiple threads also run in parallel for storage I / O access (e.g., DASD) 204a-n and 214a-n. Therefore, all processes related to the replication process, including accessing storage devices, uploading snapshots and data 230a from the source file system 280 to the object repository 260, and downloading snapshots and data 230b to the target file system 282, have multiple threads running in parallel to perform data streaming.

[0093] File storage is an Active Directory (AD) local service. When a file system is created, it resides within a specific AD. For customers transferring data from one file system or copying it to another file system in the same or different regions, artifacts (also known as manifests) may be required for the transfer.

[0094] As an alternative to using object repositories to transfer data, VCN peering can be used to establish network connections between remote machines (e.g., between replicator nodes of source and destination) and use classless inter-domain routing (“CIDR”) for each region.

[0095] Refer again Figure 2 The Key Management System (KMS) 250 is for replication security and provides storage services to a cloud service provider (e.g., OCI). In some embodiments, the source (or primary) and target (or secondary) file systems 280 use different KMS keys, and key management is hierarchical. The reason for using different keys is that if the source is compromised, the compromiser cannot use the same key to decrypt the target. The FSS has a three-tier key architecture. Because the source and target use different keys when transmitting data, the source needs to decrypt the data first, re-encrypt it with an intermediate key, and then re-encrypt the data on the target side. The FSS defines a session, and each session is a data cycle. A key is created for the session to transmit data. In other words, a new key is used for each new session. In other embodiments, a key can be used for more than one session (e.g., more than one data transmission) before another key is created. No key is transmitted through the object repository 260, and the key is only available on the source side and is not visible outside the source for security reasons.

[0096] The replication cycle (also known as a session) is periodic and adjustable. For example, the replicators (206a-n and 216a-n) perform replication once per hour. A cycle begins when a new snapshot is created on the source side 280 and ends when all increments 230b have been applied on the target side 282 (i.e., the target reaches the DONE state). Each session completes before another session begins. Therefore, only one session exists at any given time, and there is no overlap between sessions.

[0097] Secret management (i.e., replication using KMS) utilizes KMS 250 to handle the transfer of secret materials between the source (primary) file system 290 and the target (or secondary) file system 292. The source file system 280 calculates the increment, reads the file data, and then uses a local file system encryption key, collaborating with a key management service to decrypt the file data. The source file system 280 then generates a session key (called the Incremental Encryption Key (DEK)), encrypts it to become an encrypted session key (called the Incremental Transfer Key (DTK)), and transmits the DTK to the target file system 282 via its respective control planes 208 and 218. The source file system 280 also uses the DEK to encrypt data 230a and uploads it to the object storage repository 260 via the Transport Layer Security (TLS) protocol. The object storage repository 260 then uses Server-Side Encryption (SSE) to ensure the security of the stored data (e.g., increments, manifests, and metadata) 230a.

[0098] The target file system 282 securely obtains the encrypted session key DTK via its control plane 218 (using HTTPS via cross-region API communication), decrypts it via KMS 250 to obtain the DEK, and places it at a location in the target region 292. When a replication job is scheduled in the target file system 282, the DEK is provided to the replicator (one of replication queues 216a-n), and the replicator uses the key to decrypt data (e.g., incremental data including file data) 230b downloaded from the object repository 260 for application, and re-encrypts the file data using its local file system key.

[0099] The replication between the source file system 280 and the target file system 282 is processed concurrently, with each operating at its own pace. When the source side completes the upload (which may occur before the target's download processing), it cleans up its storage and removes all keys. When the target completes applying the incremental update to its latest snapshot, it also cleans up its storage and removes all keys. The FSS service also releases the KMS key. In other words, there are two copies of the session key, one in the source file system 280 and the other in the target file system 282. Both copies are removed at the end of each session, and a new session key is generated in the next replication cycle. This process ensures that the same key is not used for different purposes. Furthermore, the session key is encrypted with a file system key to create double protection. This is to ensure that only a specific file system can use the session key.

[0100] Figure 3This is a simplified schematic diagram of the components involved in cross-region remote replication according to certain embodiments. In some embodiments, a component called Increment Generator (DG) 310 in source region A 302 and a component called 330 in target region B 304 are part of a replicator queue 318 and operate on thousands of storage nodes in the queue. Replicator 318 in source region A makes remote procedure calls (RPCs) to Increment Generator 310 (e.g., fetching key-value sets, locking blocks, etc.) to collect B-tree keys and values, as well as data pages, from Direct Access Storage Device (DASD) 314, which is a replication storage service for accessing storage devices and is considered a data server. DG 310 in source region A is an assistant to replicator 318 for breaking down the key range of the increment and packing all keys / values ​​of a given range into a blob to send back to replicator 318. Multiple storage nodes 322 and 342 are attached to DASDs 314 and 334 in both regions, with each node having many disks (e.g., 10TB or more).

[0101] In some embodiments, File System Communicators (FSCs) 312 and 332 in the two zones are metadata servers that help update the source file system so that users can update the system. FSCs 312 and 332 are used for file system communication, and Increment Generator 310 is used for replication. DGs 310 and 330 and FSCs 312 and 332 are both metadata servers. User traffic passes through FSCs 312 and 332 and DASDs 314 and 334, while replication traffic passes through the DGs. In alternative embodiments, the functionality of the FSCs can be incorporated into the functionality of the DGs.

[0102] In some embodiments, shared databases (SDBs) 316 and 336 in the two regions are key-value stores through which components of the control plane and data plane (e.g., replicator queues) can read and write to communicate with each other. Control planes 320 and 340 in the two regions can queue new jobs into their respective shared databases 316 and 336, and replicator queues 318 and 338 can continuously read queues from shared databases 316 and 336, initiating file system replication once a job request is detected. In other words, shared databases 316 and 336 act as a conduit between the replicator queues and the control plane. Furthermore, shared databases 316 and 336 are distributed resources across different regions, and IO traffic to / from shared databases 316 and 336 should be minimized. Similarly, IO traffic to / from DASD needs to be minimized to avoid impacting user performance. However, replication processing may occasionally be constrained because it is a secondary service compared to the primary service.

[0103] Replicator queue 318 in source region A can collaborate with DG 310 to begin traversing the B-tree in the file system of source region A to collect key-value pairs and transform them into flat files or blobs to be uploaded to the object repository. Once the data blobs (including key-value pairs and actual data) are uploaded, the target can apply them immediately without waiting for a large number of blobs to exist in object repository 360. Object repository 360 is located in target region B for disaster recovery purposes. The goal is to push data from the source to target region B as quickly as possible and keep the data secure.

[0104] Numerous replicators optimize space by utilizing low-cost, low-footprint machines and scheduling as many replicas as possible while ensuring fair bandwidth sharing among them, thus replicating thousands of file systems. Replicator queues 318 and 338 in the two regions run on virtual machines that can automatically expand and shrink to build the entire queue for performing replication. Replicators and replication services can be dynamically adjusted based on their capacity to support each job. If one replicator is overloaded, another can take over to share the load. Different replicators in the queue can balance the load among themselves to ensure jobs can continue and do not stop due to individual replicators being overloaded.

[0105] Figure 4 This is a simplified flowchart illustrating the steps performed during cross-regional remote replication according to certain embodiments.

[0106] Step S1: When a client establishes replication, the client provides the source (or primary) file system (A) 402, the target (or secondary) file system (B) 404, and the RPO. The file system is uniquely identified by a file system identifier (e.g., Oracle Cloud ID or OCID), which is a globally unique identifier for the file system. Data is stored in the File Storage Service (“FSS”) control plane database.

[0107] Step S2: The source (A) control plane (CP-A) 410 periodically coordinates the creation of system snapshots at regular intervals (less than RPO) and notifies the data plane (including the replicator / uploader 412) of the latest snapshot and the last snapshot successfully copied to the target (B) file system 404.

[0108] Step S3: CP-A 410 notifies the component replicator 412 (or uploader) in the data plane to copy the latest snapshot:

[0109] S3a: Replicator 412 in source (A) visits the B-tree to compute the increment between two given snapshots. Existing key infrastructure is used to decrypt file system data.

[0110] S3b: These increments 414 are uploaded to the object repository 430 in the target (B) region (during the copy, the data may be compressed and / or deduplicated). This upload can be performed in parallel by multiple replicator threads 412.

[0111] Step S4: CP-A 410 Notification of Upload Completion Target (B) Control Plane (CP-B) 450.

[0112] Step S5: CP-B 450 calls the target replicator-B 452 (or downloader) to apply the increment:

[0113] Step S5a: Replicator-B 452 downloads data 454 from object repository 430.

[0114] Step S5b: Replicator-B 452 applies these increments to the target file system (B).

[0115] Step S6: After the incremental application is complete, notify CP-A 410 target (B) that a new snapshot is now available.

[0116] Step 7: Cross-regional remote copying process repeats steps S2 to S6.

[0117] Figure 5 This is a simplified diagram illustrating a high-level concept of B-tree traversal according to certain embodiments. The B-tree structure can be used in file systems. An incremental generator traverses the B-tree and guarantees the consistency of the traversal. In other words, the traversal ensures that the key-value pairs are expected at the end of the traversal and captures all information between any two snapshots, preventing data corruption. The file system is a transactional file system that can be modified, and users need to be aware of modifications and redo transactions because another user might update the same transaction or data.

[0118] Key-value pairs and snapshots are immutable (e.g., they cannot be modified unless a garbage collector can remove them). Figure 5As shown, the file system contains many snapshots (snapshot 1-snapshot N). While the incremental generator is visiting B-tree keys (510-560) in the source file system, a snapshot might be removed because garbage collector 580 might intervene to clean up keys of snapshots considered garbage. When the incremental generator visits B-tree keys, it needs to ensure that keys associated with the remaining snapshots (e.g., those not removed by the garbage collector) are copied. When garbage collector 580 removes keys (e.g., 540 and 550), the B-tree pages might shrink, for example, from two pages before garbage collection to one page after garbage collection. The way the incremental generator can ensure consistency while visiting B-tree keys is to confirm that garbage collector 580 has not modified or deleted any keys on the pages (or portions between two snapshots) that the incremental generator just visited (e.g., between two keys). Once consistency is confirmed, the incremental generator collects the keys and sends them to the replicator for processing and uploading.

[0119] B-tree keys provide a picture of the changes that have occurred. The techniques disclosed in this disclosure can determine which B-tree keys are new and which have been updated between two snapshots. An incremental generator can collect metadata portions, keys and values, and associated data, and then send them to the target. The target can determine that the received information falls within the scope of the two snapshots and apply it to the target file system. After the incremental generator (or incremental generator thread) traverses the portion between two keys and confirms their consistency, it uses the last ending key as the next starting key for its next traversal. This process is repeated until all keys have been checked, and the incremental generator collects the associated data after each consistency confirmation.

[0120] For example, in a file system, when a file is modified (e.g., created, deleted, and then recreated), the process creates several versions of the corresponding file directory entry. During the copy process, the garbage collector may clean up (or remove) the versions of the file directory entries corresponding to deleted files, leading to a consistency problem known as whiteout. Whiteout occurs if there is an inconsistency between the source and target file systems because the target file system may be unable to reconstruct the original snapshot chain involving the modified files. The disclosed technique ensures consistency between the source and target file systems by detecting whitened files (i.e., modified files affected by the garbage collector) during B-tree visits, retrieving unaffected versions of the modified files, and providing relevant information to the target file system during the same copy cycle to properly reconstruct the correct snapshot chain.

[0121] Figure 6A and Figure 6BThis diagram illustrates the pipeline stages of cross-region replication according to certain embodiments. The cross-region replication of the source file system disclosed in this disclosure has four pipeline stages: initiating cross-region replication in the source file system, performing a B-tree visit in the source file system (i.e., the incremental generation pipeline stage), storage I / O access for retrieving data (i.e., the data read pipeline stage), and uploading data to the object repository (i.e., the data upload pipeline stage). The target file system has similar four pipeline stages, but in reverse order: preparing for cross-region replication, downloading data from the object repository, performing incremental application in the target file system, and storage I / O access for storing data. Figure 6A The diagram illustrates the four pipeline stages in the source file system, but a similar concept applies to the target file system. Figure 6B The diagram illustrates the interactions between processes and components involved in the pipeline stages. All these pipeline stages can operate in parallel. Each pipeline stage can operate independently, handing over information to the next pipeline stage when processing in the current stage is complete. It ensures that each pipeline stage shares the entire bandwidth and does not use more bandwidth than necessary. In other words, resources are allocated fairly among all jobs. If no other jobs are working in the system, then the working job receives as many resources as possible.

[0122] Threads within each pipeline stage also execute their tasks in parallel (or concurrently) and independently of each other within the same pipeline stage (i.e., if a thread fails, it does not affect other threads). Furthermore, the tasks (or copy jobs) executed by threads within each pipeline stage are restartable, meaning that when a thread fails, a new thread (also called a replacement thread) can take over the failed thread to continue the original task from the last successful point.

[0123] In some embodiments, B-tree visits can be performed using parallel processing threads in the source file system 280. The B-tree in the file system can be divided into multiple key ranges between the first and last keys. The number of key ranges can be determined by the client. Multiple range threads (e.g., approximately 8 to 16) per file system can be used for B-tree visits. A range thread can perform a B-tree visit against one key range, and all range threads operate concurrently and in parallel. The number of threads to use depends on factors such as the size of the file system, resource availability, and bandwidth to balance resource and traffic congestion. The number of key ranges is typically greater than the number of available range threads to fully utilize them. Therefore, B-tree visits can be scalable and can be handled through concurrent visits (e.g., using multiple threads).

[0124] If, after the incremental generator visits a page, some keys become inconsistent due to their absence, the system can abandon ongoing, uncommitted transactions and return to the starting point to revisit the page. During repeated B-tree visits due to inconsistencies, the incremental generator can minimize the amount of information to process or upload to the target side by ignoring missing keys and their associated data, as this associated data is considered garbage. Therefore, B-tree visits and data transfers can be more efficient. Furthermore, the incremental generator does not need to wait for the garbage collector to remove information to be deleted before visiting B-tree keys. For example, if keys are dependent on each other, if a key or iNode points to a block that has been deleted or should be deleted by the garbage collector, the system (or the incremental generator) can determine that the specific block is garbage, and the incremental generator does not need to carry it.

[0125] Incremental generators typically don't modify anything on the source side (e.g., they don't delete keys or data blocks considered garbage); they simply don't copy them to the target side. B-tree visits and garbage collection are asynchronous processes. For example, when a key points to a data block that no longer exists, the file system can mark the key as garbage and indicate that it shouldn't be modified (e.g., immutable), but only the garbage collector can remove it. The incremental generator can then move on to the next key without waiting for the garbage collector. In other words, the incremental generator and the garbage collector can proceed at their own pace.

[0126] exist Figure 6A In this process, when the source region initiates cross-region copy processing that may involve many file systems, the main threads 610a-n take over the copy job, one job per file system. The main thread (e.g., 610a or 610 used later) of the file system in the source region (i.e., the source file system) works in conjunction with the incremental generator 620 (e.g., ...). Figure 6B (As shown in the diagram) communication to obtain the number of key ranges requested by the client and update the corresponding records in SDB 622. Once the main thread 610 of the source file system determines the required number of key ranges, it further creates a set of range threads 612a-n based on the required number of key ranges. These range threads 612a-n are executed by the increment generator 620. They initialize their GETKEYVAL buffer 640 ( Figure 6B As shown in the figure), update the checkpoint record 642 in its SDB 622 ( Figure 6B (as shown in the diagram), and performs storage I / O access 644 by interacting with DASD I / O threads 614a-n.

[0127] In some embodiments, each main thread 610 is responsible for overseeing all range threads 612a-n it creates. During replication, the main thread 610 may generate a master manifest file summarizing the entire replication. Range threads 612a-n generate range manifest files including the number of key ranges (i.e., the subdivisions of the entire replication), and then generate a checkpoint manifest (CM) file for each range to provide the target file system with an update on the number of blobs at each checkpoint, where checkpoints are created during B-tree visits. A checkpoint is created by range thread 612. Once the main thread 610 determines that all range threads 612a-n have completed, it creates a final checkpoint manifest (CM) file with an end-of-file marker and then uploads the CM file to the object repository so that the target file system can determine the progress in the source file system. The CM file contains a summary of all the individual ranges, such as range counts, the final state of the checkpoint records, and other information.

[0128] Range threads 612a-n are used for parallel processing to significantly reduce the time required for B-tree visits in large source file systems. In some embodiments, the B-tree keys are partitioned into ranges of approximately equal size. A range thread can perform a B-tree visit against a range of keys. The number of range threads 612a-n to be used depends on factors such as the size of the file system, resource availability and bandwidth for balancing resources, the amount of data to be generated, and traffic congestion. The number of key ranges is typically greater than the number of available range threads 612a-n (approximately 2 to 4 times) to fully utilize the range threads. Each of the range threads 612a-n has a dedicated buffer (GETKEYVAL) 640 containing available jobs to be worked on. Each range thread 612 operates independently of the other range threads and periodically updates its checkpoint record 642 in SDB 622.

[0129] While range threads 612a-n are traversing the B-tree (i.e., recursively visiting each node of the B-tree), they may need to collect file data (e.g., FMAP) associated with the B-tree keys and request I / O access 644 to the storage device. These I / O requests are queued by each range thread 612 to allow DASD I / O threads 614a-n (i.e., the data read pipeline stage) to work on them. These DASD I / O threads 614a-n are common threads shared by all range threads 612a-n. After the DASD I / O threads 614a-n obtain the requested data, the data is placed in an output buffer 646 to serialize it into a blob for the replicator's object threads 616a-n (i.e., the data upload pipeline stage) to upload to the object repository located in the target region. Each object thread selects an upload job that may contain a portion of all the data to be uploaded, and all object threads perform the upload in parallel.

[0130] Figure 7 This is a diagram illustrating the hierarchical structure in the FSS data plane according to certain embodiments. Figure 7 In this configuration, the replicator queue 710 has four layers: job layer 712, incremental generator client 714, cryptographic / DASD IO 716, and objects 718. The replicator queue 710 is a single process responsible for interacting with the storage queue 720, KMS 730, and object repository 740. In some embodiments, the job layer 712 polls the SDB 704 to find queued jobs 706, whether upload or download jobs. The replicator queue 710 includes VMs (or threads) that select queued replication jobs according to their maximum capacity. Sometimes, a replicator thread may own a portion of a replication job, but it will cooperate with another replicator thread that owns the remainder of the same replication job to complete the entire replication job concurrently. Replication jobs executed by the replicator queue 710 are restartable because if a replicator thread fails midway through replication, another replicator thread can take over and continue from the last successful point to complete the job originally owned by the failed replicator thread. If a detached replicator thread (e.g., fails and is woken up again) conflicts with another replicator thread, the FSS can use a mechanism called generation numbering to avoid the conflict by having the two replicator threads update different records.

[0131] The incremental generator client layer 714 performs a B-tree visit by accessing the incremental generator server 724 where the B-tree in storage queue 720 resides. The encryption / DASD IO layer 716 is responsible for security and storage access. After the B-tree visit, replicator queue 710 can request IO access through the encryption / DASD IO layer 716 to access the DASD extension 722 to retrieve file data associated with the incremental data identified during the B-tree visit. Both replicator queue 710 and storage queue 720 periodically update their status (e.g., checkpoints and leases of replicator queue 710) to control API 702 via SDB 704, allowing control API 702 to trigger alarms or take action as necessary.

[0132] The encryption / DASD IO layer 716 interacts with the KMS and FSK queues 730 on the target side to create session keys (or snapshot encryption keys) during cross-region replication processing and encrypts and decrypts the session keys using FSK. Finally, the object layer 718 is responsible for uploading incremental and file data from the source file system to the object repository 740 and downloading them from the object repository 740 to the target file system.

[0133] The FSS data plane is responsible for incremental generation. The data plane uses B-trees to store FSS data, and the B-trees have different types of key-value pairs, including but not limited to leading blocks, superblocks, iNodes, filename keys, cookie maps (cookies associated with directory entries), and block maps (for file content data, also known as FMAPs).

[0134] These B-tree keys are processed jointly by the replicator and the increment generator in the data plane. The algorithm used to calculate the changed key-value pairs (i.e., a portion of the increment) between two given snapshots in the file system can continuously read the keys, return the keys to the replicator using a transaction budget, and ensure that the transaction is finally confirmed to obtain consistent key-value pairs for processing.

[0135] In other embodiments, increment generation and computation can be scalable. A scalable approach can leverage multiple threads to compute the increment (i.e., the change in key-value pairs) between two snapshots by dividing the B-tree into multiple key ranges. A pool of threads (i.e., increment generators) can perform scans of the B-tree (i.e., visits to the B-tree) and compute the increment in parallel.

[0136] Figure 8A simplified example of a binary large object (BLOB) format according to certain embodiments is described. A blob is a data type used to store information (e.g., binary data) in a database. A blob is generated by the source region and uploaded to the object repository during replication. The target region needs to download and apply the blob. Depending on the context, blobs and objects can be used interchangeably.

[0137] During a B-tree traversal, when the incremental generator encounters an iNode of a given file (i.e., the data content) and its block mapping (also known as the FMAP, the data associated with the B-tree key), the incremental generator collaborates with the replicator to traverse all pages within the DASD-scoped blocks (FMAP blocks) pointed to by the FMAP, read them into a data buffer, decrypt the data using a local encrypted file key, and place it into an output buffer to serialize it into a blob for the replicator to upload to the object repository. In other words, the incremental generator needs to collect all the FMAPs of identified increments to obtain all the data related to the difference between the two snapshots.

[0138] Snapshot increments stored in an object repository may span many blobs (or objects if stored in the object repository). These blobs have a blob format with keys, values, and associated data (if any) for the keys. For example, in... Figure 8 In the snapshot increment 800, at least three blobs 802, 804, and 806 are included. The first blob 802 has a prefix 810 indicating the key-value type, key length, and value length, followed by its key 812 (key1) and value 814 (val1). The second blob 804 has a prefix 820 (key-value type, key length, and value length), a key 822 (key2), a value 824 (val2), a data length 826, and data 828 (data2). In the prefix 820 of this second blob 804, its key-value type is fmap because this blob has additional data 828 associated with key 822. The third blob 830 has a similar format to the first blob 810, for example, a prefix 830, a key 832 (key3), and a value 834 (val3).

[0139] Data is decrypted, collected, and then written to a blob. All processing is performed in parallel. Multiple blobs can be processed and updated simultaneously. Once all processing is complete, the data can be written in blob format. Figure 8 (as shown in the image), then upload to the object repository using the format or path name ( Figure 9 (See diagram in the middle).

[0140] Figure 9An example replication bucket format according to certain embodiments is described. A "bucket" can refer to a container that stores objects in a compartment within an object storage namespace. In some embodiments, the source replicator uses buckets to store secure data using server-side encryption (SSE) technology, and the target replicator also uses buckets to download data to apply changes to snapshots. Replicated data across all file systems in the target region can share buckets within that region.

[0141] The data layout of buckets in the object repository has a directory structure, which includes, but is not limited to, file system IDs (e.g., Oracle cloud IDs), increments with start and end snapshot numbers, a list describing the information content in the object layout, and blobs. For example, Figure 9 The bucket contains two objects, 910 and 930. The first object, 910, has two increments, 912 and 920. It begins with a pathname 911 prefixed with the source filesystem ID (e.g., ocid1.filesystem.oc1.iad....). The first increment, 912, is generated from snapshots 1 and 2, and the second increment, 920, is generated from snapshots 2 and 3. Each increment has one or more blobs representing the contents of that increment. For the first increment, 912, it has two blobs, 914 and 916, stored in the order they were generated. For the second increment, 920, it has only one blob, 922. Each increment also has a list of contents describing information in the layout of that increment, such as list 918 for the first increment, 912, and list 924 for the second increment, 920. The lists in the bucket describe the contents of the increment (e.g., filesystem number and snapshot range). The lists can be master lists, range lists, or checkpoint lists, depending on the stage of the replication process.

[0142] The second object 930 also contains two increments 932 and 940 with a similar format, starting with the path name 931. The two objects 910 and 930 in the bucket come from different source regions; object 910 is an IAD, and object 930 is a PHX. Once the blob is applied, the corresponding information in this layout can be removed to reduce space usage.

[0143] The final manifest object (i.e., the checkpoint manifest, CM file) is uploaded from the source region to the object repository to indicate to the target region that the source file system has completed an incremental snapshot upload of a specific object. The source CP will communicate this event to the target CP, which can then notify the target DP via SDB to trigger the target replicator to download the object.

[0144] The control plane in the source or target region coordinates all replication workflows and drives data replication. The control plane performs the following functions: 1) creates a system snapshot as the basis for creating increments; 2) determines when such a snapshot needs to be created; 3) initiates replication based on the snapshot; 4) monitors replication; 5) triggers the download of increments on the secondary (or target) side; and 6) indicates to the primary (or source) side that the snapshot has reached the secondary side.

[0145] The file system has several operations to manage its resources, including but not limited to create, read, update, and delete (CRUD). These operations are typically synchronous within the same region and consume workflows when the file system receives HTTPS requests from the API server, makes changes in the backend to create resources, and returns responses to clients. Resources are split between source and target regions. The state of the same resource is maintained between the source and target regions. Therefore, asynchronous communication exists between the source and target regions. Clients can contact the source region to create or update resources, which can be automatically reflected in secondary or backup resources in the target region. The state machine in the control plane also covers recovery from many aspects, including but not limited to failures in the queue, key management failures, disk failures, and object failures.

[0146] Moving to the Application Programming Interface (API) in the control plane, there are different APIs for users to configure replication. The API for controlling any new resource only works in the areas where objects are created. In the target filesystem, a field called "IsTargetable" in its API can be set to ensure that the target filesystem undergoing replication is not accidentally used by consumers. In other words, setting this field to false means that while consumers can see the target filesystem, no one can export the target filesystem or access any data in the live system. Any export could potentially change data because an export is a read / write operation, not a read-only operation. Therefore, exports are not allowed to prevent any changes to the target filesystem during replication processing. Consumers can only access data in older snapshots that have already been replicated. All newly created or cloned filesystems can have this field set to true. The reason is that the target can only obtain data from a single source. Otherwise, conflicts could occur when writing or deleting data. The system needs to know whether the target filesystem being used is already part of a replication. A "true" setting for the "IsTargetable" field means there is no ongoing replication, and a "false" setting means the target filesystem cannot be used.

[0147] Regarding cross-region communication between control plane components, the primary resource on the source file system is called the application, and the alternate (or secondary) source on the target file system is called the application target. When the source and target objects are created, they have a single replication relationship. Both objects can only be updated from the source side, including changing compartments, editing, or deleting details. When a user wants to delete the target side, the replication can delete itself. For planned failovers, the source side can be deleted, and both the source and target replications are deleted. For unplanned failovers, the source side becomes unavailable, so only the target replication can be deleted. In other words, there are two resources for a single replication, and they should remain synchronized. Various workflows exist for updating metadata on both the source and target sides. Furthermore, retries, fault handling, and cross-region APIs for failover are also part of the cross-region communication process.

[0148] When the source creates the necessary security and other relevant artifacts, it uploads the security and artifacts to the object repository and initiates a job on the target (i.e., notifies the target that a job is available), and the target can begin downloading artifacts (e.g., a snapshot or incremental download). The target then continues to look for end-of-file markers (also referred to herein as checkpoint manifest (CM) files) in the object repository. The CM file serves as a mechanism for both the source and target sides to communicate that an object upload is complete during the replication process. At each checkpoint, the source side uploads the CM file, which contains information such as the number of blobs uploaded up to this checkpoint, allowing the target side to download that number of blobs to apply to its current snapshot. The CM file is the mechanism by which the source side communicates to the target side that the upload of an object to the object repository is complete so that the target can begin working on that object. In other words, the target will continue downloading until there are no more objects in the object storage. Therefore, this scheme enables concurrent processing on both the source and target sides.

[0149] Figure 10 This is a flowchart illustrating the state machines of concurrent source upload and target download according to certain embodiments. As discussed earlier, both the source file system and the target system can perform replication concurrently and therefore have their own state machines. In some embodiments, each file system may have its own state machine while sharing some common job-level states. Figure 10 In the process, the source file system has states 1002 to 1018 for performing data upload and states 1030 to 1034 for session key generation and transmission. The target file system has states 1050 to 1068 for data download. When incrementally uploading to the object storage device, the session key can be generated at any time in the source file system. Therefore, session key transmission has its own state sequence 1030 to 1034. Figure 10In this process, the target file system can only initiate the copy download process after it receives the following indications (i.e., Ready_to_Reconcile state 1050): at least one object has been uploaded to the object storage device by the source file system (i.e., Manifest_Copied state 1014) and the session key is ready for download (i.e., DTK Copied state 1034).

[0150] Within the source file system, several functional blocks, such as the snapshot generator, control API, and incremental monitor, are part of the CP (Content Processing). The replicator queue is part of DP (Dual Processing). The snapshot generator is responsible for periodically generating snapshots. The incremental monitor periodically monitors the replicator's progress on replication-related tasks, including snapshot creation and replication scheduling. Once the incremental monitor detects that the replicator has completed its replication job, it moves the status to either the copied state on the source side (e.g., copied manifest state 1014) or the replicated state on the target side (e.g., replicated state 1058). In some embodiments, several file systems can simultaneously perform replication from the source region to the target region.

[0151] refer to Figure 10 In some embodiments, the source file system resides in a concurrent mode state machine where the snapshot generator signals to the incremental monitor that a snapshot has been generated after creating it. The incremental monitor running the CP replication state (CpRpSt) workflow is responsible for initiating the upload of snapshot metadata to the target-side object repository. Snapshot metadata may include snapshot type, snapshot identification information, snapshot time, etc. The CpRpSt workflow sets the Ready_to_Copy_Metadata state 1002 so that the replicator queue begins copying metadata. When a replicator acquires a replication job, it makes a copy of the snapshot metadata (i.e., copying snapshot metadata state 1004) and uploads these copies to the object repository. When all replicators have completed uploading the snapshot metadata, the state is set to the Snapshot_Metadata_Copied state 1006. The CpRpSt workflow then continues to poll the source SDB for the session key.

[0152] Now, the CpRtSt workflow returns control to the incremental monitor to monitor incremental upload processing, thus entering the Ready_to_Copy state 1008, indicating that incremental computation has been scheduled. The source CP API then sends a request to the replicator to initiate the next stage of replication by making a copy of the manifest and uploading the incremental value. The replicator taking over the replication job can begin making a copy of the manifest (i.e., copying the manifest in the Manifest_Copying state 1010). When the source filesystem completes the manifest copy, it moves to the Manifest_Copied state 1014, simultaneously notifying the target filesystem that it can initiate its internal state (Ready to Reconcile state 1050).

[0153] As discussed above, the source file system can generate a session key while data upload is in progress. The source file system's replicator communicates with the target KMS vault to obtain the master key (which can be provided by the client) to create a session key (referred to herein as the incremental encryption key or DEK). The replicator then encrypts this session key using the local file system key (FSK) (now becoming the encrypted DEK, also referred to herein as the incremental transfer key (DTK)). The DTK is then stored in the SDB in the source region for reuse by the replicator thread during the replication cycle. The state machine moves to state 1030, Ready_to_Copy_DTK.

[0154] The source file system transmits the resource identifiers of the DTK and KMS to the target API, which then places them into the SDB in the target region. During this transmission process, the state machine is set to the Copying_DTK state 1032. When the CpRpSt workflow in the source file system finishes polling the source SDB to obtain the session key, it sends a notification to the target side, signaling that the session key (DTK) is ready for the target file system to download and use to decrypt its downloaded increments for application. The state machine then moves to the Copied_DTK state 1034. The target-side replicator retrieves the DTK from its SDB and requests the KMS API to decrypt it into a plaintext DEK (i.e., the decrypted session key).

[0155] When the source file system completes the data upload (including session key transfer) for a specific replication cycle, its incremental monitor notifies the target control API of status such as verification information and enters the X-region_Copied_Done state 1016. This may occur before the target file system completes data download and application. The source file system also cleans up its storage and removes all keys. The source file system then enters the Awaiting_Target_Response state 1018 to wait for a response from the target file system to begin a new replication cycle.

[0156] As mentioned earlier, the target file system can only initiate the copy download process after receiving indications that the source file system has uploaded at least one object to the object storage device (i.e., copy manifest state 1014) and that the session key is ready for download (i.e., copy DTK state 1034). Once these two conditions are met, the state machine moves to the ready-to-reconciliation state 1050. Then, in the reconciling state 1052, the target file system begins reconciliation processes with the source side, such as synchronizing snapshots of the source and target file systems, while performing some internal CP management tasks, including taking snapshots and generating statistics. This internal state involves communication between the target file system's internal incremental monitor and the CP API.

[0157] After the reconciliation process is complete, the replication job is passed to the target replicator (i.e., Ready to Replicate state 1054). The target replicator monitors the checkpoint manifest (CM) file to be uploaded by the source filesystem. The CM file is tagged by the target. The target replicator thread then begins downloading the manifest and applying the downloaded and decrypted increments (i.e., Replicating state 1056). The target replicator thread also reads FMAP data blocks from the blob downloaded from the object repository and communicates with the local FSK service to obtain the filesystem key FSK, which is used to re-encrypt each FMAP data block and store it on its local storage device.

[0158] If the source file system has completed the data upload, it will update the final CM file and upload it to the object repository by setting the end-of-file (eof) field to true. Once the target file system detects the final CM file, it will complete the download of blobs, apply them, and the state machine will move to the copied state 1058.

[0159] After all increments (or blobs) have been applied to the target file system, it continues to download snapshot metadata from the object repository and populate the target file system's snapshot with information from the source file system's snapshot (i.e., populating snapshot metadata (Snapshot_metadata_Populating) state 1060). Once the target file system's snapshot is populated, the state machine moves to the populated snapshot metadata (Snapshot_Metadata_Populated) state 1062.

[0160] In the Snapshot_Deleting state 1064, the target filesystem deletes all blobs in the object repository that have been downloaded and applied to its latest snapshot. Once the blobs in the object repository have been deleted, the target control API notifies the target incremental monitor and proceeds to the Snapshot_Deleted state 1066. The target filesystem also cleans up its storage and removes all keys. The FSS service also releases the KMS keys.

[0161] When the target DP completes incremental application and cleanup, it verifies the status of the source file system with the target control API and whether it has received a cross-region copy complete (X-region_Copied_Done) notification from the source file system. If the notification has been received, the target incremental monitor enters the cross-region complete (X-region DONE) state 1068 and sends a cross-region complete notification to the source file system. In some embodiments, the target file system can also detect whether the source file system has completed the upload by checking whether the end of the file exists in all key ranges and all upload processing threads, because each object uploaded to the object repository has a special marker, such as an end-of-file marker in a CM file.

[0162] Referring again to the source file system state machine, when the source file system is in state 1018 (waiting for target response), it checks whether the target CP's state has changed to complete, indicating that the target has applied all downloaded increments and the file data has been stored locally. If so, then the copy cycle ends.

[0163] The source and target sides operate asynchronously. When the source file system completes its copy upload, it notifies the target control API using a cross-region copy completion notification. When the target file system subsequently completes its copy processing, its incremental monitoring target sends an X-region completion notification back to the source control API. The source file system returns a ready-to-copy metadata status 1002 to begin another copy cycle.

[0164] Figure 11This is an example flowchart illustrating the interaction between the data plane and the control plane in a source region according to certain embodiments. The data plane components and the control plane components communicate with each other using a shared database (SDB) (e.g., 1106). The SDB is a key-value store that both the control plane components and the data plane components can read from and write to. The data plane components include a replicator and an incremental generator. The interaction between components in source region A 1101 and target region B 1102 is also illustrated.

[0165] exist Figure 11 In step S1, the source control plane (CPa) 1103 requests the object repository in the target region B (OSb) 1112 to create buckets. In step S2, the source replicator (replicator a) 1108 periodically updates its heartbeat status to the source SDB (SDBa) 1106. A heartbeat is a concept used to track the replication progress performed by a replicator. It uses a mechanism called leasing, where a replicator can continuously update its heartbeat whenever it works on a job, allowing the control plane to understand the entire lease information; for example, the byte count moves continuously on the job. If a replicator fails to work properly, the heartbeat may become stale, and another replicator can detect and take over to continue working on the legacy job. Therefore, if the system crashes midway, it can start precisely from the last point in time based on a checkpoint mechanism. Checkpoints help the system know where the last point in progress is, allowing it to continue from that point without re-executing the entire job.

[0166] At step S3, CPa 1103 also requests the File System Service Workflow (FSW_CPa) 1104 to periodically create snapshots, and at step S4, FSW_CPa 1104 notifies CPa 1103 of the new snapshot. At step S5, CPa 1103 then stores the snapshot information in SDB 1106. At step S6, Replicator a 1108 polls SDB 1106 for any changes to the existing snapshot, and if a change is detected, retrieves the job specification at step S7. At step S8, once Replicator a 1108 detects a change to the snapshot, this triggers the replication process. At step S8, Replicator a 1108 provides the Incremental Generator (DGa) 1110 with information about the two snapshots (SNa and SNb) and the changes between them. At step S9, Replicator a 1108 puts work item information (such as the number of key ranges) into SDB 1106. At step S10, replicator a 1108 checks the replication job queue in SDBa 1106 to obtain work items, and at step S11, assigns them to the increment generator (DGa) 1110 to scan the B-tree keys of the snapshot (i.e., visit the B-tree) to compute the increment and the corresponding key-value pair. At step S12, replicator a 1108 decrypts the file data associated with the identified B-tree keys and packages them together with the key-value pairs into a blob. At step S13, replicator a 1108 encrypts the blob using the session key and uploads it as an object to OSb 1112. At step S14, replicator a performs a checkpoint and stores the checkpoint record in SDBa 1106. This replication process repeats (as a loop) until all increments have been identified and the data has been uploaded to OSb 1112. At step S15, copier a 1108 then notifies SDBa 1106 of the replication job details, and then at step S16 the replication job details are passed to CPa 1103, and further relayed to CPb 1114 as the final CM file at step S17. At step S18, CPb 1114 stores the job details in SDBb 1116.

[0167] The interaction between the data plane and control plane in target region B is similar. At the end of applying the incremental snapshot to the target file system, the control plane in target region B notifies the control plane in source region A that the snapshot has been successfully applied. This allows the control plane in source region A to restart for the new snapshot.

[0168] Authentication is performed on each component. From the replicator to the File System Key (FSK), there is an authentication mechanism using the replication ID and file system number. The key is only provided to the replicator if the replicator provides the correct content. Therefore, the authentication mechanism prevents imposters from obtaining the decryption key. Other security mechanisms include blocking network ports. A component called the File System Key Server (FSKS) is a gatekeeper used to verify the requester by examining metadata such as the job the requester will execute and other information. For example, suppose a replicator attempts to request a key for a file system. In this case, the FSKS can verify the requester by checking whether the replicator is associated with a specific job (e.g., a replication actually associated with that file system).

[0169] Availability addresses situations where machines can automatically restart after downtime or where service remains available during software deployment. For example, all replicators are stateless, so the loss of a replicator is transparent to clients because another replicator can take over and continue working on the job. Job state is maintained in a shared database and other reliable locations, not locally. The shared database is a database-like service used by the control plane to maintain information about the file system, and it is based on a B-tree.

[0170] Storage availability is high in the FSS disclosed herein because the system has thousands of storage nodes, allowing any storage node to perform incremental replication. Control plane availability is high by utilizing numerous machines that can take over from each other in the event of any failure. For example, replication progress will not be hindered by the failure of just one control plane. Therefore, there is no single point of failure. Network access availability utilizes congestion management involving various types of throttling to ensure that source nodes are not overloaded.

[0171] Replication achieves persistence by utilizing checkpoints, where the replication state is written to the shared database, and the replicator is stateless. Replication processing is idempotent. Idempotency can refer to deterministic reapplication, meaning that when an operation fails, retries of the same operation should be effective and result in the same outcome by using, for example, the same key, upload processing, or access processing.

[0172] Operations in several domains are idempotent. In the control plane, it's necessary to remember actions already taken. For example, if an HTTP request is repeated, an idempotent cache can help remember that a specific operation has been performed and is the same operation. In the data plane, for instance, when a block is allocated, the block and its file system mapping key are written together. Therefore, when the block is allocated again, it can be recognized. If the block has been sealed, the write operation will fail. An idempotent mechanism can know that the block was sealed in the past, and the write operation does not need to be redone. In yet another example, an idempotent mechanism remembers the chain of steps required to process a specific key-value pair. In other words, an idempotent mechanism allows each operation to be checked to see if it is in the correct state. Therefore, the system can move directly to the next step without repetition.

[0173] When rolling back a snapshot, such as from snapshot 10 to snapshot 5, atomic replay allows incremental application to begin immediately upon the arrival of the first incremental object in the object repository. For replay to be atomic, the entire incremental object must be held in the object repository before it can be applied.

[0174] Regarding replicator expansion, the FSS disclosed herein allows for the addition of as many replicator machines (e.g., replicator virtual machines (“VMs”)) as needed to support numerous file systems. The number of replicators can be dynamically increased or decreased, taking into account bandwidth requirements and resource availability. Regarding storage expansion, thousands of storage devices can be used to parallelize processing and increase workload speed. Regarding inter-region bandwidth, bandwidth allocation ensures that each workload does not overuse or exceed its predefined throughput limits through automatic throttling (such as throttling all inter-region bandwidth by determining latency increases and slowing down requests). This capability is available to all replicator processors (or threads).

[0175] For checkpoint storage expansion, the uploader and downloader record their progress as checkpoints to persistent storage, and the shared storage is used as a work queue for split key ranges. If checkpoint workloads overwhelm the shared database, checkpoint storage functionality can be added to the incremental generator for scaling purposes. The current shared database workload may consume less than 10 IOPs.

[0176] Figure 12 This is a simplified diagram illustrating fault recovery modes according to certain embodiments. Fault recovery modes allow the primary / source side to be restored to its previous state before failover and to become primary / source again. Figure 12 As shown, the primary AD 1202 includes a source file system 1206, and the secondary AD 1204 includes a target file system 1208. The secondary AD 1204 may be located in the same region or a different region as the primary AD 1202.

[0177] exist Figure 12 In the source file system 1206, snapshots 1220 and 2222 existed before the failover caused by the interruption event. Similarly, snapshots 1240 and 2242 in the target file system 1208 existed before the failover. When the primary AD 1202 failed at snapshot 3224, the FSS performed an unplanned failover 1250, and snapshot 3224 in the source file system 1206 was copied to the target file system 1208 to become the new snapshot 3224. After the target file system 1208 came online, a customer may have made changes to the target file system 1208, which created snapshot 4246.

[0178] If the client decides to reuse the source file system, the FSS service can perform fault recovery. The user has two options when performing fault recovery: 1) the last point in time in the source file system before the event was triggered (1251), or 2) the latest changes in the target file system (1252).

[0179] For the first option, the user can recover from the last point in time in the source file system 1206 before the event was triggered (i.e., snapshot 3 1224). In other words, snapshot 3 1224 will be the snapshot to be used after the failover, since it was previously successfully failovered to the target file system 1208. To perform failover 1251, the state of the source file system 1206 is changed to inaccessible. The FSS service then identifies the last point in time in the source file system 1206 before the successful failover, i.e., snapshot 3 1224. The FSS can then perform a clone of snapshot 3 1224 (i.e., a copy in the same region) in the primary AD 1202. Now, the primary AD 1202 returns to its initial settings before the interruption, and the user can reuse the source file system 1206 again. Since snapshot 3 1224 is already in the file system to be used, there is no need to transfer data from the secondary AD 1204 to the primary AD 1202.

[0180] For the second option, the user wants to reuse the source file system with the latest changes in the target file system 1208. In other words, snapshot 4 1246 in the target file system 1208 will be the snapshot to be used after failure recovery, because it contains the latest changes in the target file system 1208. The failure recovery process 1252 for this option involves reverse copying (i.e., reversing the roles of the source and target file systems for copying), and FSS performs the following steps:

[0181] Step 1. The status of source file system 1206 is changed to inaccessible.

[0182] Step 2. Then, the FSS service identifies the latest snapshot that has been successfully copied in the target file system 1208, for example, snapshot 3 1244.

[0183] Step 3. The FSS service also finds the corresponding snapshot 3 1224 in the source file system 1206 and performs a clone (i.e., a copy in the same region).

[0184] Step 4. FSS service to communicate with about Figure 4 A similar process (but in the opposite direction) discussed in section 1252 initiates reverse replication. In other words, the source file system 1206 and the target file system 1208 need to be synchronized, and then the target file system 1208 can upload the increment to the object repository in the master AD 1202. The source file system 1206 can then download the increment from the object repository to complete the application of snapshot 3 1224, thereby creating a new snapshot 4 1226.

[0185] The main AD 1202 has now returned to its initial settings before the interruption, and the user can reuse the source file system 1206 without transferring existing data in both the source file system 1206 and the target file system 1208, such as snapshots 1-3 (1220-1224) in the source file system 1206. This saves time and avoids unnecessary bandwidth.

[0186] Hierarchical Key Management Architecture

[0187] For the purposes of this application, in some examples, the KMS vault is a key management service that stores and manages master encryption keys and secrets used for secure access to resources. The KMS vault enables the secure storage of master encryption keys (or master keys) and secrets that would otherwise be stored in configuration files or code. The vault service integrates with FSS to support the use of customer-managed keys. The master encryption key obtained from the KMS vault can be used to generate other encryption keys, such as file system keys (FSKs) and / or incremental encryption keys (DEKs, also referred to herein as session keys), for encrypting / decrypting other keys or data. For security purposes, FSKs and DEKs may also be referred to herein as security keys. Each vault master key can be automatically assigned a key version. The vault service supports keys as cloud infrastructure and assigns a globally unique resource identifier (e.g., an Oracle Cloud ID (OCID)) to each key.

[0188] In some embodiments, the File Storage Service (FSS) disclosed herein utilizes a three-tier key architecture for cross-region replication. Three distinct keys may be involved during the replication process: a source file system key (FSKa), a session key, and a destination file system key (FSKb). Within each source and destination region, the file system keys are also hierarchically structured to encrypt local files, and a master key from the client is used to securely store the file system keys.

[0189] In some embodiments, the source file system has three keys: a source file key (FKa) for each file, a source file system key (FSKa), and a master key (based on a client-managed key). The source file system has multiple files, each with its own file key FKa for encryption and decryption. Each FKa is encrypted by the FSKa and stored in the iNode of the file's B-tree. The unique FSKa for each file system is generated and encrypted / decrypted by a client-provided file master key (which may be referred to herein as the source-side master key) and stored in the metadata of the data plane. A similar scheme is used in the target area. In the target file system, it has three keys: a target file key (FKb) for each file, a target file system key (FSKb), and a master key (based on a client-managed key), which may be referred to herein as the target-side master key. In some embodiments, the source file system master key is different from the target file system master key.

[0190] During cross-region replication, the three keys mentioned for cross-region replication can be used to decrypt and encrypt data during transfers between regions. FSS defines a session, and each session is a data cycle (or a replication cycle). A session key is created only for that session to transfer data. First, the source region decrypts its file data using FSKa (via FKa) during incremental generation. Second, it encrypts the data (e.g., blob and manifest files) using the session key from the source region and uploads it for storage in the object repository. Third, the session key is transferred from the source region's CP to the target region's CP using the Key Management Service (KMS). Fourth, the target region downloads the data from the object repository and decrypts it using the session key. Finally, the target region re-encrypts its data using FSKb (via FKb) and stores it locally in a B-tree.

[0191] Existing key management for file system replication may use a single encryption key for more than one region and transfer data between different regions. If the security of a local region is compromised, this increases the risk of cross-region data integrity corruption. Therefore, different approaches are needed to address these challenges related to security risks and other risks.

[0192] The three-tier key architecture disclosed in this disclosure avoids any exposure of the FSKa or customer management key (or master key) outside the source region by encrypting data (e.g., blobs and lists) before uploading it to the object repository using session keys. Furthermore, instead of sharing filesystem-level keys among multiple filesystems in one or more regions, FSS's key infrastructure has a unique key (e.g., FSKa) for each filesystem, adding an extra layer of protection for each filesystem. Finally, the session key is transmitted via the CP API (requiring KMS assistance before reuse), rather than being transmitted with the data through the object repository, providing a third level of protection.

[0193] In some embodiments, all keys involved in the replication process are stored in encrypted form. For example, customer-managed keys (or service keys) are securely stored in a KMS vault. In some embodiments, FSKa and FKa are encrypted and stored locally in the source B-tree, while FSKb and FKb are encrypted and stored locally in the destination B-tree. To further ensure security, authentication processes exist in both the source and destination regions to verify the identity of the key requester. For example, the control plane (including a file system key server (FSK service)) may check the replication ID (i.e., the replication job the key requester will perform) and file system number (i.e., which file system the key requester belongs to) provided by the key requester before granting the file system key (FSK).

[0194] Placing the KMS vault in the target region while simultaneously creating a session key in the source region for replication offers the following advantages: In the event of an unexpected disaster in the source region, the session key remains in the target region because the target file system can still download data (or objects) from the object repository and decrypt the data for applications. In some embodiments, for security reasons, the session key is temporarily stored in the source region to encrypt data to be uploaded to the object repository.

[0195] Figure 13 This is a flowchart illustrating key management for cross-regional replication according to certain embodiments. Figure 13 In the source area 1301, the components involved in key management include a snapshot monitor 1310, a shared database (SDB) 1312, a replicator 1314, and a file system key (FSK) service 1316. In the target area 1302, the components involved in key management include a key management service (KMS) 1350, an application programming interface (API) 1330, an SDB 1332, an object storage device 1360, a replicator 1334, and an FSK service 1336. When replication is initiated, the control plane of the source file system can cooperate with the data plane to start. Figure 13 The key management process is shown in the figure.

[0196] exist Figure 13 In step S1, the source replicator 1314 communicates with the target KMS vault 1350 to obtain a master key (different from the master key of the source FSK discussed above), which may be referred to herein as the cross-region master key, to generate a new session key (i.e., DEK, plaintext key) for this replication cycle. In some cases, after another failed replicator has previously generated a session key, a replicator (or replicator thread, also referred to herein as the alternative replicator) can take over the replication job from that other failed replicator to restart replication. In some embodiments, the alternative replicator first checks the source SDB 1312 (refer to step S3 discussed later) to obtain the locally stored session key (i.e., decrypt the existing DTK). If, for some rare cases, the source SDB loses its record of the session key, then the alternative replicator will need to communicate with the target KMS vault using CGO code to call the KMS API and provide the KMS resource ID to obtain the master key to generate a new DEK, thereby restarting the replication job (i.e., encrypting the remaining unfinished increments left before).

[0197] At step S2, the source replicator 1314 communicates with the FSK service 1316 to obtain the source file system key (FSKa). The FSK service 1316 also authenticates the key requester and the source replicator 1314 by checking the replication ID and file system number (FSNum) to ensure that the key requester is indeed associated with a specific file system and a specific replication process. At step S3, the source replicator 1314 uses the FSKa to encrypt the session key DEK (the encrypted session key is referred to herein as the incremental transfer key (DTK)) and stores it in the source SDB 1312 for use by other replicator threads (including alternative threads) later in the replication cycle, thus avoiding the need to communicate with KMS again via cross-region calls. At step S4, the source replicator 1314 caches the FSKa locally. At step S5, in addition to storing the DTK in the source SDB 1312, the source replicator 1314 also caches the DTK locally. At step S6, during the replication cycle, each replicator thread handling the replication job (i.e., a part of the overall replication) checks the source SDB 1312 to reuse the session key (i.e., DEK) by decrypting the encrypted session key (i.e., DTK) using the locally cached FSKa. The source replicator 1314 also uses the FSKa to decrypt its local FKa, which in turn is used to decrypt file data collected during B-tree visits. Up to this point, the above steps have been performed within the source region 1301.

[0198] From steps S7 to S9, the key management process enters the cross-region phase. At step S7, in some embodiments, the source replicator 1314 then decrypts the data using FSKa (as discussed above) and re-encrypts the data (e.g., B-tree key values, file data, manifest files, and metadata) using the session key, and uploads it as an object in the target region 1302 to the object repository 1360. In some embodiments, a portion of the data (e.g., internal metadata for internal management purposes) may not be encrypted.

[0199] At step S8, the source replicator 1314 notifies the state machine to change to the Manifest_Copied state by marking a record in the source SDB 1312. In step S9, the snapshot monitor in the source region 1301 transmits the encrypted session key (i.e., DTK) and KMS key resource ID to the target API 1330. Although the flowchart shows that the step of transmitting data objects to the object repository (S7) occurs before the step of transmitting the DTK to the target API (S9), the source region does not need to wait until all data objects have been transmitted before starting step S9. The sooner the source file system can transmit the DTK to the target file system, the sooner the target file system can download and decrypt the data.

[0200] At step S10, the target API 1330 stores the received encrypted session key (i.e., DTK) and KMS key resource ID in the target SDB 1332 for reuse during the replication cycle. At step S11, the target replicator 1334 (or replicator thread) obtains the encrypted session key and KMS key resource ID from the target SDB 1332. The target replicator 1334 also checks if the DTK has expired. If it has expired, the target replicator 1334 triggers an alarm and signals a hard fault. Otherwise, it moves to the next step. At step S12, the target replicator 1334 sends the encrypted session key and KMS key resource ID to the KMS vault 1350 and requests the decryption of the DTK (i.e., becoming the DEK) for use. The KMS vault 1350 also authenticates the key requester and the target replicator 1334 to ensure that the DTK is used for the session associated with that particular replication cycle. At step S13, the target replicator 1334 caches the session key (i.e., DEK) locally. At step S14, the target replicator 1334 downloads data from the object repository 1360 and decrypts the data using the session key. At step S15, the target replicator 1334 communicates with the target FSK service 1336 to obtain the target file system key (FSKb). The target FSK service 1336 also authenticates the key requester and the source replicator 1334 by checking the replication ID and file system number to ensure that the key requester is indeed associated with a specific file system and a specific replication process. At step S16, the target replicator 1334 caches the FSKb locally for reuse. At step S17, the target replicator 1334 uses the FSKb to decrypt the target file key (FKb) (this key is used to re-encrypt the data after applying the data to its latest snapshot), and then stores the data locally in a B-tree. At step S18, the target replicator 1334 notifies the state machine to change to a replicated state by marking a record in the target SDB 1332.

[0201] When the source side completes the upload (which may occur before the target's download processing), the source side cleans up its storage and removes all keys. When the target completes applying the increment to its latest snapshot, it cleans up its storage and also removes all keys. The FSS service also releases the KMS master key. Therefore, the key is only valid for the current replication cycle.

[0202] Figure 14 This is a flowchart illustrating the session key generation steps according to certain embodiments. Figure 14 In the source area 1401, the components involved in generating the session key (i.e., DEK) may be the source SDB 1410, the source Control Plane (CP) API 1412, the source replicator 1414, and the source FSK service 1416. The components in the target area 1402 involve the target KMS vault 1418.

[0203] In some embodiments, at step S1, the source replicator 1414 receives a request from the CP to prepare session key generation for cross-region replication purposes and performs initialization to communicate with the KMS and object repository in the target region. At step S2, the source replicator 1414 communicates with the target KMS 1418 to list all vaults in a specific compartment. At step S3, the source replicator 1414 obtains all keys for a specific vault whose vault prefix matches the desired prefix, since both the key and the vault have unique prefix names. At step S4, the source replicator 1414 caches the master key resource ID for future use by all ADs. Since the master key vault and key are unique for each AD, caching the resource IDs of other ADs in the same source region while processing the key for that AD can help reduce future communication with the KMS. At step S5, the source replicator 1414 generates a session key (i.e., DEK) using the Additional Authentication Data (AAD) as the snapshot resource ID. AAD is metadata provided to KMS during encryption / decryption to help FSS verify whether a specific session key is associated with a snapshot resource. This provides an additional layer of protection. At step S6, as discussed above, all keys can be stored in encrypted form. Therefore, source replicator 1414 encrypts the session key using the source file system key FSK 1316 (not shown) and stores the encrypted session key (i.e., DTK) and KMS source ID in the source SDB 1410 for reuse by other replicator threads in the current replication cycle. The key and its KMS resource ID need to be stored together to be presented to the target KMS vault for later decryption. At step S7, source replicator (worker replicator thread) 1414 caches the DEK for its use, and at step S8, the data to be uploaded to the object repository is re-encrypted using the session key DEK.

[0204] As discussed earlier regarding the pipeline stages in Figure 6, multiple replicator threads in the replicator queue can each handle a portion of the overall replication process. For each replicator thread performing the replication process, steps 9 through 11 can be repeated. At step S9, the new source replicator thread 1414 obtains the encrypted session key DTK and KMS resource ID from the source SDB 1410. At step S10, the new replicator thread 1414 communicates with the KMS vault using the CGO call with the snapshot resource ID of the source region to obtain the decrypted session key DEK. At step S11, the new replicator thread 1414 re-encrypts the data for uploading to the object repository.

[0205] When all replicator threads complete their replication tasks for the current replication cycle, at step S12, the source CP API 1412 deletes the encrypted session key DTK and KMS resource ID stored in the source SDB 1410. At step 13, all replicator threads 1414 clear the cached session key DEK from their memory. Therefore, this ensures that the session key is valid only within one replication cycle (i.e., one session) and cannot be reused in another replication cycle.

[0206] Figure 15 A simplified table format (or schema) for a shared database (SDB) for storing encrypted session keys (DTKs) is described according to certain embodiments. As discussed above, each region has an SDB for storing the encrypted session keys (DTKs). In some embodiments, the SDB may be a key-value store. Therefore, the schema has two parts: a key part and a value part. The information in this table can be used for communication between the DP and CP in each region. Figure 15 In the key portion 1510, there are key type 1511, object type 1512, object number (i.e., incremental transfer ID or replication number) 1514, and snapshot number 1516. The value portion 1520 includes version 1522, KMS key resource ID 1524, encryption key 1526, target file system number 1528, target queue 1530, and target availability domain (AD) 1532.

[0207] In some embodiments, the key portion 1510 of architecture 1500 begins with key type 1511, which indicates the type of metadata in the B-tree to aid in efficient B-tree key parsing. For example, the key type of metadata may include iNode, file mapping (i.e., fmap), etc. Object type 1512 can be used to distinguish the purpose of a key entry. For example, setting object type 1512 to 10 may indicate that the entry is written by the source replicator in the source region and read by the CP in the source region to transmit the encrypted session key 1526 to the target region. On the other hand, setting object type 1512 to 11 may indicate that the entry is written by the target CP after receiving the encrypted session key 1526 from the source region and read by the target replicator to decrypt incremental / data downloaded from the target object repository.

[0208] In some embodiments, object number 1514 is used to identify which replication process (e.g., a specific replication cycle) is associated with the encrypted session key DTK 1526. Snapshot number 1516 is used to identify a specific snapshot involved in that replication.

[0209] In some embodiments, the value portion 1520 of architecture 1500 begins with a version 1522 field, which indicates the version of the resource ID. The KMS key resource ID 1524 is the resource ID of the KMS master key in the target region. In alternative embodiments, the KMS resource ID 1524 may be a customer-managed key. The encryption key 1526 contains an octet of ciphertext, a vault prefix, and a count to indicate the length of the byte array for the key. The target file system number 1528 is used to communicate with the FSK service to determine which target file system requires FSK for authentication / verification purposes. The target queue 1530 is used to identify the replicator queue in the target DP. The target AD 1532 is used for authentication / verification purposes to confirm the region and AD of the target file system.

[0210] Figure 16 This is a flowchart illustrating the end-to-end key management process for cross-region replication according to certain embodiments. In some embodiments, the FSS three-layer key architecture may have three distinct system-level security keys—the source file system key (FSKa), the target file system key (FSKb), and the session key (DEK). A security key can refer to a key used for security purposes and can encrypt / decrypt another key or data. FSKa is used for the source file system. FSKb is used for the target file system. The session key is used between the source and target file systems. Each of these three system-level security keys can be generated from three distinct master keys. A first version of the master key from the KMS vault can be used to generate the FSKa in the source file system. A second version of the master key from the KMS vault can be used to generate the FSKb in the target file system. A third version of the master key from the KMS vault can be used to generate the DEK in the source file system. In addition, there are two distinct file-level security keys—the source file key (FKa) and the target file key (FKb).

[0211] In some embodiments, a first version of the FSKa and master key is kept within the source region. A second version of the FSKa and master key is kept within the destination region. A third version of the master key may also be kept within the source region. Only the DEK can be transmitted across the source and destination regions and can be used to encrypt / decrypt data transmitted between the source and destination regions. Therefore, the exposure of the key outside the region where it was generated can be minimized.

[0212] At step 1610, the FSS can generate a source file system key (FSKa) in the source file system based on the source-side master key, which can be used to encrypt / decrypt file keys (e.g., FKa) in the source file system. At step 1612, the source file system can use the source file key (FKa) to decrypt the file data of its corresponding file previously encrypted by the FKa. At step 1614, the source replicator (e.g., Figure 13 Step 1314) can communicate with the target KMS vault to obtain a cross-regional master key to generate a new session key (i.e., DEK, plaintext key) for that replication cycle (or session). At step 1616, during the incremental generation process in the source file system, the source file system can use the DEK to encrypt the generated snapshot increment (including file data and manifest file).

[0213] At step 1630, the source file system can upload encrypted snapshot increments to the object repository for download by the target file system. At step 1632, the source file system can use FSKa to encrypt the session key to generate an encrypted session key (i.e., DTK), and then transfer it from the source CP to the target CP.

[0214] At step 1650, the target file system can decrypt the DTK using KMS to obtain the DEK, and then use the DEK to decrypt the downloaded snapshot increment for incremental application. At step 1652, the FSS can generate a target file system key (FSKb) in the target file system based on the target-side master key, which can be used to encrypt / decrypt file keys (e.g., FKb) in the target file system. At step 1654, the target file system can use the target file key (FKa) during incremental application to encrypt the file data in the downloaded snapshot increment for storage in its corresponding file.

[0215] Example cloud architecture

[0216] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0217] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0218] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0219] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0220] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0221] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0222] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provisioned to define how inbound / outbound traffic to the network and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are expected and added.

[0223] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0224] Figure 17 This is a block diagram 1700 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service provider 1702 may be communicatively coupled to secure host lease 1704, which may include a virtual cloud network (VCN) 1706 and a secure host subnet 1708. In some examples, service provider 1702 may use one or more client computing devices, which may be portable handheld devices (e.g., Cellular phone Computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google) Head-mounted displays), running software (such as Microsoft Windows) ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and support the Internet, email, and short message service (SMS). Or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, including, for example, those running various versions of Microsoft... Apple Personal computers and / or laptops running Linux operating systems. Client computing devices can be running various commercially available operating systems. Or a workstation computer operating a UNIX-like operating system, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., with or without...). A Microsoft Xbox game console with gesture input devices, and / or a personal messaging device capable of communicating via a network that can access VCN1706 and / or the Internet.

[0225] VCN 1706 may include a local peering gateway (LPG) 1710, which may be communicatively coupled to a secure shell (SSH) VCN 1712 via LPG 1710 included in SSH VCN 1712. SSH VCN 1712 may include an SSH subnet 1714, and SSH VCN 1712 may be communicatively coupled to a control plane VCN 1716 via LPG 1710 included in control plane VCN 1716. Furthermore, SSH VCN 1712 may be communicatively coupled to a data plane VCN 1718 via LPG 1710. Control plane VCN 1716 and data plane VCN 1718 may be contained within a service lease 1719 that may be owned and / or operated by an IaaS provider.

[0226] The control plane VCN 1716 may include a control plane demilitarized zone (DMZ) layer 1720 that acts as a peripheral network (e.g., part of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 1720 may include one or more load balancer (LB) subnets 1722, a control plane application layer 1724 that may include one or more application subnets 1726, and a control plane data layer 1728 that may include one or more database (DB) subnets 1730 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1722 contained in the control plane DMZ layer 1720 may be communicatively coupled to one or more application subnets 1726 contained in the control plane application layer 1724 and an Internet gateway 1734 that may be contained in the control plane VCN 1716. The application subnets 1726 may be communicatively coupled to one or more DB subnets 1730 contained in the control plane data layer 1728, as well as a service gateway 1736 and a Network Address Translation (NAT) gateway 1738. The control plane VCN 1716 may include the service gateway 1736 and the NAT gateway 1738.

[0227] The control plane VCN 1716 may include a data plane mirror application layer 1740, which may include one or more application subnets 1726. The one or more application subnets 1726 included in the data plane mirror application layer 1740 may include a virtual network interface controller (VNIC) 1742 capable of executing a compute instance 1744. The compute instance 1744 may communicatively couple the one or more application subnets 1726 of the data plane mirror application layer 1740 to the one or more application subnets 1726 that may be included in the data plane application layer 1746.

[0228] Data plane VCN 1718 may include data plane application layer 1746, data plane DMZ layer 1748, and data plane data layer 1750. Data plane DMZ layer 1748 may include one or more LB subnets 1722 communicatively coupled to one or more application subnets 1726 of data plane application layer 1746 and Internet gateway 1734 of data plane VCN 1718. One or more application subnets 1726 may be communicatively coupled to service gateway 1736 and NAT gateway 1738 of data plane VCN 1718. Data plane data layer 1750 may also include one or more DB subnets 1730 communicatively coupled to one or more application subnets 1726 of data plane application layer 1746.

[0229] The Internet gateway 1734 of the control plane VCN 1716 and data plane VCN 1718 can be communicatively coupled to the metadata management service 1752, which can be communicatively coupled to the public Internet 1754. The public Internet 1754 can be communicatively coupled to the NAT gateway 1738 of the control plane VCN 1716 and data plane VCN 1718. The service gateway 1736 of the control plane VCN 1716 and data plane VCN 1718 can be communicatively coupled to the cloud service 1756.

[0230] In some examples, the service gateway 1736 of the control plane VCN 1716 or data plane VCN 1718 can make application programming interface (API) calls to the cloud service 1756 without traversing the public internet 1754. API calls from the service gateway 1736 to the cloud service 1756 can be unidirectional: the service gateway 1736 can make API calls to the cloud service 1756, and the cloud service 1756 can send requested data to the service gateway 1736. However, the cloud service 1756 may not initiate API calls to the service gateway 1736.

[0231] In some examples, secure host lease 1704 can be directly connected to service lease 1719, which would otherwise be isolated. Secure host subnet 1708 can communicate with SSH subnet 1714 via LPG 1710, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 1708 to SSH subnet 1714 allows secure host subnet 1708 to access other entities within service lease 1719.

[0232] Control plane VCN 1716 allows users of service lease 1719 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 1716 can be deployed or otherwise used in data plane VCN 1718. In some examples, control plane VCN 1716 can be isolated from data plane VCN 1718, and the data plane mirror application layer 1740 of control plane VCN 1716 can communicate with the data plane application layer 1746 of data plane VCN 1718 via VNIC 1742, which can be included in both the data plane mirror application layer 1740 and the data plane application layer 1746.

[0233] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1754, which can transmit requests to the metadata management service 1752. The metadata management service 1752 can transmit requests to the control plane VCN 1716 via internet gateway 1734. Requests can be received by one or more LB subnets 1722 contained in the control plane DMZ layer 1720. The LB subnets 1722 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 1726 contained in the control plane application layer 1724. If the request is validated and requires a call to the public internet 1754, the call to the public internet 1754 can be transmitted to a NAT gateway 1738 that can make calls to the public internet 1754. The request may expect stored metadata to be stored in one or more DB subnets 1730.

[0234] In some examples, the data plane mirroring application layer 1740 can facilitate direct communication between the control plane VCN 1716 and the data plane VCN 1718. For example, it might be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 1718. Through VNIC 1742, the control plane VCN 1716 can communicate directly with the resources contained in the data plane VCN 1718, and thus can perform configuration changes, updates, or other appropriate modifications.

[0235] In some embodiments, the control plane VCN 1716 and data plane VCN 1718 may be included in service lease 1719. In this case, the system's users or customers may not own or operate the control plane VCN 1716 or data plane VCN 1718. Alternatively, the IaaS provider may own or operate both the control plane VCN 1716 and data plane VCN 1718, and both planes may be included in service lease 1719. This embodiment can enable the isolation of networks that might prevent users or customers from interacting with resources of other users or customers. Furthermore, this embodiment can allow users or customers of the system to privately store databases without relying on the public Internet 1754, which may not have the desired level of threat prevention for storage.

[0236] In other embodiments, one or more LB subnets 1722 included in the control plane VCN 1716 may be configured to receive signals from the service gateway 1736. In this embodiment, the control plane VCN 1716 and the data plane VCN 1718 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1754. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on service lease 1719, which may be isolated from the public internet 1754.

[0237] Figure 18 This is a block diagram 1800 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1802 (e.g., Figure 17 The service provider 1702) can communicatively couple to the secure host lease 1804 (e.g., Figure 17 Secure hosting lease 1704), the secure hosting lease 1804 may include a virtual cloud network (VCN) 1806 (e.g., Figure 17 VCN 1706) and Secure Host Subnet 1808 (e.g., Figure 17 The secure host subnet 1708). VCN 1806 may include a local peering gateway (LPG) 1810 (e.g., Figure 17 The LPG 1710, which can be communicatively coupled to the Secure Shell (SSH) VCN 1812 (e.g., LPG 1710) contained in the SSH VCN 1812, can be used to communicate with the Secure Shell (SSH) VCN 1812. Figure 17 SSH VCN 1712). SSH VCN 1812 can include SSH subnet 1814 (e.g., Figure 17 SSH subnet 1714), and SSH VCN 1812 can be communicatively coupled to control plane VCN 1816 via LPG 1810 included in control plane VCN 1816 (e.g., Figure 17 Control plane VCN 1716). Control plane VCN 1816 may be included in service lease 1819 (e.g., Figure 17 In the service lease 1719), and the data plane VCN 1818 (e.g., Figure 17 The data plane VCN 1718 may be included in a customer lease 1821 that may be owned or operated by the system’s users or customers.

[0238] Control plane VCN 1816 may include control plane DMZ layer 1820 (e.g., Figure 17 The control plane DMZ layer 1720 may include one or more LB subnets 1822 (e.g., Figure 17 (one or more) LB subnets 1722), may include (one or more) application subnets 1826 (e.g., Figure 17 The control plane application layer 1824 of (one or more) application subnets 1726 (e.g., Figure 17 The control plane application layer 1724 may include one or more database (DB) subnets 1830 (e.g., similar to...). Figure 17 The control plane data layer 1828 of (one or more) DB subnets 1730 (e.g., Figure 17 The control plane data layer 1728). One or more LB subnets 1822 contained in the control plane DMZ layer 1820 can be communicatively coupled to one or more application subnets 1826 contained in the control plane application layer 1824 and an Internet gateway 1834 that can be contained in the control plane VCN 1816 (e.g., Figure 17 Internet gateway 1734), and application subnet(s) 1826 can communicatively couple to DB subnet(s) 1830 contained in control plane data layer 1828 and service gateway 1836 (e.g., Figure 17 Service gateway 1736) and Network Address Translation (NAT) gateway 1838 (e.g., Figure 17(NAT gateway 1738). The control plane VCN 1816 may include the service gateway 1836 and the NAT gateway 1838.

[0239] The control plane VCN 1816 may include a data plane mirror application layer 1840 that may contain one or more application subnets 1826 (e.g., Figure 17 The data plane mirror application layer 1740). One or more application subnets 1826 contained in the data plane mirror application layer 1840 may include computational instances 1844 (e.g., similar to...). Figure 17 The virtual network interface controller (VNIC) 1842 (e.g., the VNIC of 1742) of the computing instance 1744. The computing instance 1844 can facilitate the mirroring of the application subnet(s) 1826 of the application layer 1840 in the data plane and can be included in the application layer 1846 in the data plane (e.g., Figure 17 Communication between one or more application subnets 1826 in the data plane application layer 1746 via VNIC 1842 contained in the data plane mirror application layer 1840 and VNIC 1842 contained in the data plane application layer 1846.

[0240] The Internet gateway 1834, included in the control plane VCN 1816, can be communicatively coupled to the metadata management service 1852 (e.g., Figure 17 Metadata management service 1752), which can communicatively couple to the public Internet 1854 (e.g., Figure 17 The public internet 1854 can communicatively couple to a NAT gateway 1838 contained in a control plane VCN 1816. The service gateway 1836 contained in the control plane VCN 1816 can communicatively couple to a cloud service 1856 (e.g., ...). Figure 17 Cloud services (1756).

[0241] In some examples, data plane VCN 1818 may be included in customer lease 1821. In this case, the IaaS provider may provide control plane VCN 1816 for each customer, and the IaaS provider may set up a unique compute instance 1844 for each customer, included in service lease 1819. Each compute instance 1844 may allow communication between control plane VCN 1816 included in service lease 1819 and data plane VCN 1818 included in customer lease 1821. Compute instance 1844 may allow resources provisioned in control plane VCN 1816 included in service lease 1819 to be deployed or otherwise used in data plane VCN 1818 included in customer lease 1821.

[0242] In other examples, an IaaS provider's customer may have a database residing in customer lease 1821. In this example, control plane VCN 1816 may include data plane mirror application layer 1840, which may include one or more application subnets 1826. Data plane mirror application layer 1840 may reside in data plane VCN 1818, but may not reside in data plane VCN 1818. That is, data plane mirror application layer 1840 may have access to customer lease 1821, but may not reside in data plane VCN 1818 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 1840 may be configured to invoke data plane VCN 1818, but may not be configured to invoke any entity contained in control plane VCN 1816. Customers may expect to deploy or otherwise use resources provided in the control plane VCN 1816 in the data plane VCN 1818, and the data plane mirroring application layer 1840 can facilitate customers' expected deployments or other uses of resources.

[0243] In some embodiments, an IaaS provider's customer may apply filters to data plane VCN 1818. In this embodiment, the customer may determine what data plane VCN 1818 can access, and the customer may restrict access from data plane VCN 1818 to the public Internet 1854. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1818's access to any external networks or databases. Applying filters and controls to data plane VCN 1818, which is included in customer lease 1821, can help isolate data plane VCN 1818 from other customers and the public Internet 1854.

[0244] In some embodiments, cloud service 1856 may be invoked by service gateway 1836 to access services that may not exist on public internet 1854, control plane VCN 1816, or data plane VCN 1818. The connection between cloud service 1856 and control plane VCN 1816 or data plane VCN 1818 may not be real-time or continuous. Cloud service 1856 may reside on different networks owned or operated by an IaaS provider. Cloud service 1856 may be configured to receive calls from service gateway 1836 and may be configured not to receive calls from public internet 1854. Some cloud services 1856 may be isolated from other cloud services 1856, and control plane VCN 1816 may be isolated from cloud services 1856 that may not be in the same region as control plane VCN 1816. For example, control plane VCN 1816 may be located in "Region 1," and cloud service "Deployment 17" may be located in both "Region 1" and "Region 2." If the service gateway 1836, contained in the control plane VCN 1816 located in region 1, makes a call to deployment 17, then that call can be transmitted to deployment 17 in region 1. In this example, the control plane VCN 1816 or deployment 17 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 17 in region 2.

[0245] Figure 19 This is a block diagram 1900 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1902 (e.g., Figure 17 The service provider 1702) can communicatively couple to the secure host lease 1904 (e.g., Figure 17 Secure hosting lease 1704), the secure hosting lease 1904 may include a virtual cloud network (VCN) 1906 (e.g., Figure 17 VCN 1706) and Secure Host Subnet 1908 (e.g., Figure 17 The secure host subnet 1708). VCN 1906 can include LPG 1910 (e.g., Figure 17 The LPG 1710), which can be communicatively coupled to the SSH VCN 1912 via the LPG 1910 included in the SSH VCN 1912 (e.g., Figure 17 SSH VCN 1712). SSH VCN 1912 can include SSH subnet 1914 (e.g., Figure 17 SSH subnet 1714), and SSH VCN 1912 can be communicatively coupled to control plane VCN 1916 via LPG 1910 contained in control plane VCN 1916 (e.g., Figure 17The control plane VCN 1716) and coupled to the data plane VCN 1918 via the LPG 1910 contained in the data plane VCN 1918 (e.g., Figure 17 Data plane 1718). Control plane VCN 1916 and data plane VCN 1918 may be included in service lease 1919 (e.g., Figure 17 The service rental (1719) is included.

[0246] The control plane VCN 1916 may include a subnet 1922 that may contain one or more load balancer (LB) subnets (e.g., Figure 17 The control plane DMZ layer 1920 of (one or more) LB subnets 1722) (e.g., Figure 17 The control plane DMZ layer 1720 may include one or more application subnets 1926 (e.g., similar to...). Figure 17 The control plane application layer 1924 of (one or more) application subnets 1726 (e.g., Figure 17 The control plane application layer 1724), and may include (one or more) DB subnets 1930, and the control plane data layer 1928 (e.g., Figure 17 The control plane data layer 1728). One or more LB subnets 1922 contained in the control plane DMZ layer 1920 can be communicatively coupled to one or more application subnets 1926 contained in the control plane application layer 1924 and an Internet gateway 1934 that can be contained in the control plane VCN 1916 (e.g., Figure 17 Internet gateway 1734), and application subnet(s) 1926 can communicatively couple to DB subnet(s) 1930 contained in control plane data layer 1928 and service gateway 1936 (e.g., Figure 17 The service gateway) and Network Address Translation (NAT) gateway 1938 (e.g., Figure 17 (NAT gateway 1738). The control plane VCN 1916 may include the service gateway 1936 and the NAT gateway 1938.

[0247] Data plane VCN 1918 may include data plane application layer 1946 (e.g., Figure 17 Data plane application layer 1746), data plane DMZ layer 1948 (e.g., Figure 17 Data plane DMZ layer 1748), and data plane data layer 1950 (e.g., Figure 17The data plane data layer 1750. The data plane DMZ layer 1948 may include one or more trusted application subnets 1960 and one or more untrusted application subnets 1962 that are communicatively coupled to the data plane application layer 1946, and one or more LB subnets 1922 of the Internet gateway 1934 contained in the data plane VCN 1918. The one or more trusted application subnets 1960 may be communicatively coupled to the service gateway 1936 contained in the data plane VCN 1918, the NAT gateway 1938 contained in the data plane VCN 1918, and one or more DB subnets 1930 contained in the data plane data layer 1950. The one or more untrusted application subnets 1962 may be communicatively coupled to the service gateway 1936 contained in the data plane VCN 1918 and the one or more DB subnets 1930 contained in the data plane data layer 1950. The data plane data layer 1950 may include one or more DB subnets 1930 that can be communicatively coupled to the service gateway 1936 contained in the data plane VCN 1918.

[0248] One or more untrusted application subnets 1962 may include one or more primary VNICs 1964(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1966(1)-(N). Each tenant VM 1966(1)-(N) may be communicatively coupled to a corresponding application subnet 1967(1)-(N) that may be contained in a corresponding container egress VCN 1968(1)-(N), which may be contained in a corresponding customer lease 1970(1)-(N). A corresponding secondary VNIC 1972(1)-(N) may facilitate communication between one or more untrusted application subnets 1962 contained in data plane VCN 1918 and application subnets contained in container egress VCN 1968(1)-(N). Each container exit VCN 1968(1)-(N) may include a NAT gateway 1938, which can communicatively couple to the public Internet 1954 (e.g., Figure 17 The public internet (1754).

[0249] The Internet gateway 1934, contained in the control plane VCN 1916 and the data plane VCN 1918, can communicatively couple to the metadata management service 1952 (e.g., Figure 17A metadata management system 1752 is provided, which can communicatively couple to the public Internet 1954. The public Internet 1954 can communicatively couple to a NAT gateway 1938 contained in a control plane VCN 1916 and a data plane VCN 1918. A service gateway 1936 contained in a control plane VCN 1916 and a data plane VCN 1918 can communicatively couple to a cloud service 1956.

[0250] In some embodiments, the data plane VCN 1918 can be integrated with the customer lease 1970. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer to the IaaS provider.

[0251] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request functionality attached to the data plane application layer 1946. The code running this functionality can execute in VMs 1966(1)-(N), and this code may not be configured to run anywhere else on the data plane VCN 1918. Each VM 1966(1)-(N) can be connected to a customer lease 1970. A corresponding container 1971(1)-(N) contained within VMs 1966(1)-(N) can be configured to run the code. In this scenario, dual isolation can exist (e.g., container 1971(1)-(N) runs the code, where container 1971(1)-(N) may be contained in at least one VM 1966(1)-(N) within an untrusted application subnet 1962), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Container 1971(1)-(N) may be communicatively coupled to Customer Lease 1970 and may be configured to transmit or receive data from Customer Lease 1970. Container 1971(1)-(N) may not be configured to transmit or receive data from any other entity in Data Plane VCN 1918. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of Container 1971(1)-(N).

[0252] In some embodiments, one or more trusted application subnets 1960 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1960 may be communicatively coupled to one or more database subnets 1930 and configured to perform CRUD operations in one or more database subnets 1930. One or more untrusted application subnets 1962 may be communicatively coupled to one or more database subnets 1930, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1930. Containers 1971(1)-(N) that may be contained in each customer's VM 1966(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1930.

[0253] In other embodiments, the control plane VCN 1916 and the data plane VCN 1918 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1916 and the data plane VCN 1918. However, communication can occur indirectly through at least one method. The LPG 1910 may be established by the IaaS provider, which can facilitate communication between the control plane VCN 1916 and the data plane VCN 1918. In another example, either the control plane VCN 1916 or the data plane VCN 1918 may invoke the cloud service 1956 via the service gateway 1936. For example, an invocation of the cloud service 1956 from the control plane VCN 1916 may include a request for a service that can communicate with the data plane VCN 1918.

[0254] Figure 20 This is a block diagram 2000 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2002 (e.g., Figure 17 Service providers (1702) can communicatively couple to secure host rental (2004) (e.g., Figure 17 Secure hosting lease 1704), the secure hosting lease 2004 may include Virtual Cloud Network (VCN) 2006 (e.g., Figure 17 VCN 1706) and Secure Host Subnet 2008 (e.g., Figure 17 The secure host subnet 1708). VCN 2006 can include LPG 2010 (e.g., Figure 17 LPG 1710), which can be used via SSH VCN 2012 (e.g., LPG 2010), Figure 17The LPG 2010 in SSH VCN 1712 is communicatively coupled to SSH VCN 2012. SSH VCN 2012 may include SSH subnet 2014 (e.g., Figure 17 SSH subnet 1714), and SSH VCN 2012 can be communicatively coupled to control plane VCN 2016 via LPG 2010 included in control plane VCN 2016 (e.g., Figure 17 The control plane VCN 1716) and coupled to the data plane VCN 2018 via the LPG 2010 contained in the data plane VCN 2018 (e.g., Figure 17 Data plane 1718). Control plane VCN 2016 and data plane VCN 2018 can be included in service lease 2019 (e.g., Figure 17 The service rental (1719) is included.

[0255] The control plane VCN 2016 may include one or more LB subnets 2022 (e.g., Figure 17 The control plane DMZ layer of (one or more) LB subnets 1722) 2020 (e.g., Figure 17 The control plane DMZ layer 1720 may include (one or more) application subnets 2026 (e.g., Figure 17 The control plane application layer 2024 of (one or more) application subnets 1726 (e.g., Figure 17 The control plane application layer 1724) may include (one or more) DB subnets 2030 (e.g., Figure 19 The control plane data layer of (one or more) DB subnets 1930) 2028 (e.g., Figure 17 The control plane data layer 1728). One or more LB subnets 2022 contained in the control plane DMZ layer 2020 can be communicatively coupled to one or more application subnets 2026 contained in the control plane application layer 2024 and an Internet gateway 2034 that can be contained in the control plane VCN 2016 (e.g., Figure 17 Internet gateway 1734), and application subnet(s) 2026 can communicatively couple to DB subnet(s) 2030 contained in control plane data layer 2028 and service gateway 2036 (e.g., Figure 17 The service gateway) and Network Address Translation (NAT) gateway 2038 (e.g., Figure 17 (NAT gateway 1738). The control plane VCN 2016 may include service gateway 2036 and NAT gateway 2038.

[0256] Data plane VCN 2018 may include data plane application layer 2046 (e.g., Figure 17 Data plane application layer 1746), data plane DMZ layer 2048 (e.g., Figure 17 Data plane DMZ layer 1748), and data plane data layer 2050 (e.g., Figure 17 The data plane data layer 1750). The data plane DMZ layer 2048 may include one or more trusted application subnets 2060 that can be communicatively coupled to the data plane application layer 2046 (e.g., Figure 19 (one or more) trusted application subnets 1960) and (one or more) untrusted application subnets 2062 (e.g., Figure 19 The data plane VCN 2018 may include one or more untrusted application subnets 1962 and one or more LB subnets 2022 of Internet gateway 2034. One or more trusted application subnets 2060 may communicatively couple to service gateway 2036, NAT gateway 2038, and DB subnets 2030 in data plane VCN 2018. One or more untrusted application subnets 2062 may communicatively couple to service gateway 2036 and DB subnets 2030 in data plane VCN 2018. Data plane VCN 2050 may include one or more DB subnets 2030 that may communicatively couple to service gateway 2036 in data plane VCN 2018.

[0257] One or more untrusted application subnets 2062 may include a primary VNIC 2064(1)-(N) communicatively coupled to tenant virtual machines (VMs) 2066(1)-(N) residing within one or more untrusted application subnets 2062. Each tenant VM 2066(1)-(N) may run code in a corresponding container 2067(1)-(N) and is communicatively coupled to an application subnet 2026 that may be contained in a data plane application layer 2046 contained in a container egress VCN 2068. A corresponding secondary VNIC 2072(1)-(N) may facilitate communication between one or more untrusted application subnets 2062 contained in a data plane VCN 2018 and the application subnets contained in a container egress VCN 2068. The container egress VCN may include a public internet 2054 (e.g., Figure 17 The public internet (1754) uses NAT gateway 2038.

[0258] Internet gateway 2034, included in control plane VCN 2016 and data plane VCN 2018, can communicatively couple to metadata management service 2052 (e.g., Figure 17 A metadata management system 1752 is provided, which can communicatively couple to the public internet 2054. The public internet 2054 can communicatively couple to a NAT gateway 2038 contained in a control plane VCN 2016 and a data plane VCN 2018. A service gateway 2036 contained in a control plane VCN 2016 and a data plane VCN 2018 can communicatively couple to a cloud service 2056.

[0259] In some examples, Figure 20 The architecture shown in the block diagram 2000 can be considered as... Figure 19 This is an exception to the pattern shown in the architecture diagram 1900, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 2067(1)-(N) contained in each customer's VM 2066(1)-(N). Container 2067(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 2072(1)-(N) contained in one or more application subnets 2026 of the data plane application layer 2046, which may be contained in a container egress VCN 2068. The auxiliary VNIC 2072(1)-(N) can transmit the call to a NAT gateway 2038, which can then transmit the call to the public internet 2054. In this example, containers 2067(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 2016 and from other entities contained in the data plane VCN 2018. Containers 2067(1)-(N) can also be isolated from resources from other clients.

[0260] In other examples, a client can use container 2067(1)-(N) to invoke cloud service 2056. In this example, the client can run code within container 2067(1)-(N) requesting a service from cloud service 2056. Container 2067(1)-(N) can then forward the request to auxiliary VNIC 2072(1)-(N), which can then forward the request to a NAT gateway, which can forward the request to public internet 2054. Public internet 2054 can then forward the request via internet gateway 2034 to one or more LB subnets 2022 contained in control plane VCN 2016. In response to determining that the request is valid, one or more LB subnets can forward the request to one or more application subnets 2026, which can then forward the request to cloud service 2056 via service gateway 2036.

[0261] It should be recognized that the IaaS architectures 1700, 1800, 1900, and 2000 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0262] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0263] Figure 21 An example computer system 2100, in which various embodiments can be implemented, is illustrated. System 2100 can be used to implement any of the computer systems described above. As shown, computer system 2100 includes a processing unit 2104 that communicates with a plurality of peripheral subsystems via a bus subsystem 2102. These peripheral subsystems may include a processing acceleration unit 2106, an I / O subsystem 2108, a storage subsystem 2118, and a communication subsystem 2124. Storage subsystem 2118 includes a tangible computer-readable storage medium 2122 and system memory 2110.

[0264] Bus subsystem 2102 provides a mechanism for allowing various components and subsystems of computer system 2100 to communicate with each other as intended. While bus subsystem 2102 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2102 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0265] A processing unit 2104, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 2100. One or more processors may be included in the processing unit 2104. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2104 may be implemented as one or more independent processing units 2132 and / or 2134, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 2104 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0266] In various embodiments, processing unit 2104 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 2104 and / or storage subsystem 2118. With appropriate programming, processor(s) 2104 can provide the various functions described above. Computer system 2100 may additionally include processing acceleration unit 2106, which may include digital signal processor (DSP), dedicated processor, etc.

[0267] The I / O subsystem 2108 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, keyboards, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as those from Microsoft… Motion sensors enable users to control devices such as Microsoft products via a natural user interface using gestures and voice commands. The 360 ​​game controller's input device interacts with it. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making a menu selection) and translating the eye gestures into the input device (e.g., Google). Google input in ) Blink detector. Additionally, the user interface input device may include enabling the user to interact with a voice recognition system (e.g., ...) via voice commands. Voice recognition sensing devices for interaction with navigators.

[0268] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0269] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 2100 to a user or other computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0270] Computer system 2100 may include storage subsystem 2118, which provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 2104, provide the aforementioned functionality. Storage subsystem 2118 may also provide a repository for storing data used according to this disclosure.

[0271] like Figure 21 As illustrated in the example, storage subsystem 2118 may include various components, including system memory 2110, computer-readable storage medium 2122, and computer-readable storage medium reader 2120. System memory 2110 may store program instructions that can be loaded and executed by processing unit 2104. System memory 2110 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 2110, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0272] System memory 2110 may also store operating system 2116. Examples of operating system 2116 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Phone OS OS and A mobile operating system of an OS. In some implementations of computer system 2100 that execute one or more virtual machines, the virtual machine and its guest operating system (GOS) may be loaded into system memory 2110 and executed by one or more processors or cores of processing unit 2104.

[0273] System memory 2110 can be configured differently depending on the type of computer system 2100. For example, system memory 2110 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, system memory 2110 may include a basic input / output system (BIOS), which contains basic routines that facilitate the transfer of information between components within computer system 2100, such as during startup.

[0274] Computer-readable storage medium 2122 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 2104 of the computer system 2100) for use by the computer system 2100.

[0275] Computer-readable storage medium 2122 may include any suitable medium known or used in the art, including storage and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0276] As an example, computer-readable storage medium 2122 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a disk drive that reads or writes to a removable non-volatile magnetic disk, and a removable non-volatile optical disc drive (such as CD ROM, DVD, and...). An optical disc drive that reads from or writes to a disk or other optical medium. Computer-readable storage medium 2122 may include, but is not limited to, Disk drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVDs, digital audio tapes, and so on. Computer-readable storage media 2122 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 2100.

[0277] Machine-readable instructions executable by one or more processors or cores of processing unit 2104 may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0278] The communication subsystem 2124 provides an interface to other computer systems and networks. The communication subsystem 2124 serves as an interface for receiving data from other systems and transmitting data from computer system 2100 to other systems. For example, the communication subsystem 2124 enables computer system 2100 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2124 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technologies, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 2124 may provide a wired network connection (e.g., Ethernet).

[0279] In some embodiments, the communication subsystem 2124 may also represent one or more users who may use the computer system 2100 to receive input communications in the form of structured and / or unstructured data feeds 2126, event streams 2128, event updates 2130, etc.

[0280] As an example, the communication subsystem 2124 can be configured to receive data feeds 2126 from users of social networks and / or other communication services in real time, such as... feed, Updates, web feeds such as rich site summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0281] Furthermore, the communication subsystem 2124 can also be configured to receive data in the form of a continuous data stream, which may include event streams 2128 and / or event updates 2130 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0282] The communication subsystem 2124 can also be configured to output structured and / or unstructured data feeds 2126, event streams 2128, event updates 2130, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 2100.

[0283] Computer system 2100 can be one of various types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), and wearable devices (e.g., Glass head-mounted displays, PCs, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0284] Due to the ever-evolving nature of computers and networks, the description of the computer system 2100 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0285] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0286] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0287] Accordingly, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made therein without departing from the broader spirit and scope set forth in the claims. Therefore, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0288] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as encompassing both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0289] Exclusion language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such exclusion language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0290] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0291] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0292] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

[0293] The embodiments may be implemented using a computer program product comprising a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure.

Claims

1. A method comprising: A first security key associated with the source file system is generated by a computing device based at least in part on a first master key for use in encrypting and decrypting multiple file keys in the source file system, the source file system being configured to send snapshot increments during the copying process, and the snapshot increments being identified between two snapshots of the source file system; A second security key associated with the target file system is generated by a computing device, at least in part based on a second master key, for use in encrypting and decrypting multiple file keys in the target file system, which is configured to receive snapshot increments during the copying process; as well as A session key is generated by a computing device based at least in part on a third master key for encrypting and decrypting snapshot increments transferred between the source and target file systems during the copying process, the session key being valid for the session; The snapshot increment is sent from the source file system to the object storage device, and then forwarded from the object storage device to the target file system. The session key is encrypted using a first security key associated with the source file system, and the encrypted session key is transmitted from the control plane of the source file system to the control plane of the target file system. The first master key, the second master key, and the third master key are different keys.

2. The method of claim 1, wherein the session is the duration between the start of the copy process in the source file system and the end of the copy process in the target file system.

3. The method of claim 1, further comprising encrypting the snapshot increment using a session key before transmitting the snapshot increment to the object storage device, and decrypting the snapshot increment using the session key after transmitting the snapshot increment to the target file system.

4. The method of claim 1, wherein the session key is associated with a globally unique resource identifier.

5. The method of claim 1, wherein each of the plurality of file keys in the source file system is associated with a specific file in the source file system and is used to encrypt and decrypt file data of the specific file in the source file system, and each of the plurality of file keys in the target file system is associated with a specific file in the target file system for encrypting and decrypting file data of the specific file in the target file system.

6. The method of claim 1, further comprising: Authenticate the key requester in the source file system who requests to use the first security key; as well as Authenticate the key requester in the target file system who requests to use the second security key.

7. The method of claim 6, wherein authenticating the key requester in the source file system includes checking the identifier of the copy process and the identifier of the source file system.

8. A non-transitory computer-readable medium storing computer-executable instructions, said instructions, when executed by one or more processors, causing said one or more processors to perform operations, comprising: A first security key associated with the source file system is generated by a computing device based at least in part on a first master key for use in encrypting and decrypting multiple file keys in the source file system, the source file system being configured to send snapshot increments during the copying process, and the snapshot increments being identified between two snapshots of the source file system; A second security key associated with the target file system is generated by a computing device, at least in part based on a second master key, for use in encrypting and decrypting multiple file keys in the target file system, which is configured to receive snapshot increments during the copying process; as well as A session key is generated by a computing device based at least in part on a third master key for encrypting and decrypting snapshot increments transferred between the source and target file systems during the copying process, the session key being valid for the session; The snapshot increment is sent from the source file system to the object storage device, and then forwarded from the object storage device to the target file system. The session key is encrypted using a first security key associated with the source file system, and the encrypted session key is transmitted from the control plane of the source file system to the control plane of the target file system. The first master key, the second master key, and the third master key are different keys.

9. The non-transitory computer-readable medium of claim 8, wherein the session is the duration between the start of a copy process in the source file system and the end of a copy process in the target file system.

10. The non-transitory computer-readable medium of claim 8, further comprising encrypting the snapshot increment with a session key before transmitting the snapshot increment to the object storage device, and decrypting the snapshot increment with the session key after transmitting the snapshot increment to the target file system.

11. The non-transitory computer-readable medium of claim 8, wherein each of the plurality of file keys in the source file system is associated with a specific file in the source file system and is used to encrypt and decrypt file data of the specific file in the source file system, and each of the plurality of file keys in the target file system is associated with a specific file in the target file system for encrypting and decrypting file data of the specific file in the target file system.

12. A system comprising: One or more processors; as well as One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the system to perform: A first security key associated with the source file system is generated by a computing device based at least in part on a first master key for use in encrypting and decrypting multiple file keys in the source file system, the source file system being configured to send snapshot increments during the copying process, and the snapshot increments being identified between two snapshots of the source file system; A second security key associated with the target file system is generated by a computing device, at least in part based on a second master key, for use in encrypting and decrypting multiple file keys in the target file system, which is configured to receive snapshot increments during the copying process; as well as A session key is generated by a computing device based at least in part on a third master key for encrypting and decrypting snapshot increments transferred between the source and target file systems during the copying process, the session key being valid for the session; The snapshot increment is sent from the source file system to the object storage device, and then forwarded from the object storage device to the target file system. The session key is encrypted using a first security key associated with the source file system, and the encrypted session key is transmitted from the control plane of the source file system to the control plane of the target file system. The first master key, the second master key, and the third master key are different keys.

13. The system of claim 12, wherein the session is the duration between the start of a copy process in the source file system and the end of a copy process in the target file system.

14. The system of claim 12, wherein the session key is associated with a globally unique resource identifier.

15. The system of claim 12, further comprising: Authenticate the key requester in the source file system who requests to use the first security key; as well as Authenticate the key requester in the target file system who requests to use the second security key.

16. The system of claim 15, wherein authenticating the key requester in the source file system includes checking the identifier of the copy process and the identifier of the source file system.

Images