Method, system and medium for capturing snapshots of key management data across nodes in a cloud infrastructure

Abstract

The present disclosure relates to capturing snapshots of key management data and storing the snapshots so that the key management data can be efficiently recreated in the event of an outage at one or more nodes. A snapshot orchestrator can request a snapshot instance from each node in a series of nodes across one or more regions in a cloud infrastructure service. Each snapshot instance can provide multiple modifications to multiple client keys maintained by each node in the series of nodes. The snapshot orchestrator can obtain the snapshot instance and corresponding metadata and verify the snapshot instance received from the series of nodes. The snapshot orchestrator can store the verified snapshot instance and corresponding metadata at a storage node. This can allow subsequent retrieval of the snapshot instance and re-creation of log records and modifications to multiple keys at any node in the series of nodes.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 194,023, filed May 27, 2021, and entitled “SNAPSHOTTING HARDWARE SECURITY MODULES AND DISK METADATA STORES,” and U.S. Non-Provisional Application No. 17 / 719,010, filed April 12, 2022. The entire contents of the above-referenced applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The disclosed technology relates to key management in a cloud environment. More specifically, the disclosed technology relates to capturing snapshots of key management data and corresponding metadata across a series of nodes in a cloud infrastructure system, and storing the snapshots to efficiently recreate the key management data in the event of an outage at one or more nodes. Background Art

[0004] Cloud infrastructure (CI) systems can perform multiple functions, such as storing data across nodes (e.g., servers) in the CI system and allowing querying of the stored data. In addition, the CI system can store and maintain large amounts of client data. In many cases, the client data can be encrypted using a key so that the client data can only be accessed and / or modified in response to providing the appropriate key. The key used for the client can be used to encrypt / decrypt portions of the client data across the CI system.

[0005] Additionally, multiple keys may be modified when client data is added or modified in the CI system. For example, a new key may be added to encrypt a new data set, or a key may be removed from a plurality of keys in response to removing another data set from the CI system. Modifications to multiple keys associated with a client may be logged and maintained by one or more nodes in the CI system. For example, a key management node may log and store all modifications to multiple keys and associated metadata on one or more security modules in the CI system as keys are added, removed, modified, etc. The key management node may be used to retrieve keys across the CI system and decrypt encrypted client data. Summary of the Invention

[0006] The present embodiments relate to capturing snapshots of key management data and storing the snapshots to efficiently recreate the key management data in the event of an outage at one or more nodes. A first exemplary embodiment relates to a method for capturing snapshots of key management data across a series of nodes. The method may include requesting, by a snapshot orchestrator, a snapshot instance from each node in the series of nodes across one or more regions in a cloud infrastructure service. Each snapshot instance may provide multiple modifications to multiple client keys maintained by each node in the series of nodes. In addition, each modification may correspond to an entry in a log record.

[0007] The method may also include obtaining, by the snapshot orchestrator, a snapshot instance and corresponding metadata from each node in the series of nodes. The method may also include verifying the snapshot instance received from the series of nodes. The method may also include storing the snapshot instance and corresponding metadata at the storage node in response to verifying the snapshot instance. This may allow subsequent retrieval of the snapshot instance and re-creation of log records and modifications to the plurality of keys at any node in the series of nodes.

[0008] Another example embodiment relates to a snapshot management system. The snapshot management system may include a processor and a non-transitory computer-readable medium. The non-transitory computer-readable medium includes instructions that, when executed by the processor, cause the processor to request a snapshot instance from each node in a series of nodes. Each snapshot instance may provide multiple modifications to multiple client keys maintained by each node in the series of nodes. Each modification may correspond to a log sequence record in an append-only log record.

[0009] The instructions may further cause the processor to obtain a snapshot instance and corresponding metadata from each node in the series of nodes. The instructions may further cause the processor to verify the snapshot instance received from the series of nodes based on the entropy value of each of the identified snapshot instances. The instructions may further cause the processor to store the snapshot instance and corresponding metadata at the storage node in response to verifying the snapshot instance. The snapshot instance and corresponding metadata may be configured to be used to recreate the append-only log record and the modification of the plurality of keys at any node in the series of nodes.

[0010] Another example embodiment relates to a non-transitory computer-readable medium. The non-transitory computer-readable medium may include a sequence of instructions stored thereon that, when executed by a processor, cause the processor to perform a process. The process may include requesting a snapshot instance from each node in a series of nodes across one or more regions in a cloud infrastructure service. Each snapshot instance may specify multiple modifications to multiple client keys maintained by each node in the series of nodes. Additionally, each modification may correspond to an entry in a log record.

[0011] The process may also include obtaining a snapshot instance and corresponding metadata from each node in the series of nodes. The process may also include verifying the snapshot instance received from the series of nodes by determining that the entropy value of each received snapshot instance is within a threshold similarity. The process may also include storing the snapshot instance and corresponding metadata at the storage node in response to verifying the snapshot instance. This may allow subsequent retrieval of the snapshot instance and re-creation of the log record and modification of the plurality of keys at any node in the series of nodes.

[0012] Furthermore, the embodiments may be implemented using a computer program product comprising a computer program / instructions that, when executed by a processor, causes the processor to perform any of the methods / techniques described in the disclosure herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example snapshot management system in accordance with at least one embodiment.

[0014] Figure 2 Illustrated is a block diagram of an example of a system for capturing a snapshot of key management data in accordance with at least one embodiment.

[0015] Figure 3 A block diagram of an example snapshot orchestrator is illustrated in accordance with at least one embodiment.

[0016] Figure 4 is an example bitmap for a snapshot instance according to at least one embodiment.

[0017] Figure 5 Illustrated is a block diagram of an example snapshot file format bitmap in accordance with at least one embodiment.

[0018] Figure 6 is a block diagram of an example method for capturing a snapshot of key management data across a series of nodes in accordance with at least one embodiment.

[0019] Figure 7 is a block diagram illustrating one mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0020] Figure 8 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0021] Figure 9 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0022] Figure 10is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0023] Figure 11 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION

[0024] A cloud infrastructure (CI) service may include one or more storage modules (or "vaults") for securely managing and storing cryptographic keys across various CI services and applications. A CI vault may include a secure and resilient managed service that allows for secure data encryption while maximizing efficiency in hardware provisioning and / or software patching, and with high availability.

[0025] In the CI service, security-sensitive workloads can be encrypted using cryptographic keys stored in hardware security modules (HSMs). The CI service can use HSMs to provide key management services. Client keys can be stored in a vault, and with cross-region replication of keys, clients can recover from regional failures by replicating keys across regions in the CI service. In response to enabling replication for a client, existing and new keys can be replicated with associated metadata with accurate replication status.

[0026] The CI Key Management Service can use HSMs to store customer keys. Each HSM can include several partitions, which provide logical and security boundaries for the keys stored in each HSM. In response to a client creating a vault, the vault can include the associated partition and all keys within the vault created in that partition. The contents of one partition may not be shared with another partition unless they share the same trust chain, thereby maintaining a security blast radius local to a single partition.

[0027] HSMs can be modeled as replicated state machines (RSMs). Each client-facing vault can include an associated write-ahead log (WAL). The WAL can record operations on its vault, including but not limited to the creation and deletion of keys and metadata updates in an ordered manner. The WAL can be stored in an internal CI persistent key-value store that supports transactional reads and writes. Similar to the calculator described above, a vault can be completely recreated by applying the entries in the WAL associated with the vault.

[0028] Additionally, when a vault is created for a client, the associated partitions in the HSM are replicated by default across hosts in data centers within the region. By enabling cross-region replication on the vault, the WAL can be relayed to the destination region and applied to all hosts in that region. By default, client keys are replicated across hosts distributed across data centers. The KMS service within a region can run behind a firewall, without access to external networks (e.g., the internet). To enable cross-region communication, a high-performance RESTful service (e.g., a WAL service) can run in each region, which can carry WAL traffic from the KMS service to another region within the domain. Cross-region calls from one region to the WAL service in another region can pass through an access control proxy at the edge of the source region. Any service can incorporate Representational State Transfer (REST) ​​technology, be geographically distributed, and execute with one or more load balancers. Each write-ahead log (WAL) can include an append-only log that contains modifications to keys over time. Each WAL can include a series of entries, each with an associated log sequence number (LSN). The LSN can be monotonically dense and increments for each WAL entry.

[0029] In addition, the CI KMS can include multiple distributed services working together across multiple regions. However, cross-region processing can cause various problems (e.g., problems caused by Byzantine failures). For example, in such a distributed system including multiple regions, an incident may occur that causes partial or complete loss of a host, disk, data repository, HSM, or network problems that cause data corruption, partitioning, or out-of-order message delivery during transmission. In addition, this may cause errors or accidents in manipulating data in the WAL or other data center failures. In response, the WAL can include the entirety of various system states related to the customer vault, so that the service can be stateless. Each WAL entry can encode both client data and metadata about the CI service.

[0030] For example, a node hosting a key management service (e.g., a hardware security module (HSM)) may fail or lose functionality for any of a variety of reasons. For example, in the event of a large-scale outage, the device hosting the HSM may lose power or functionality, which may result in the loss of data in the HSM. In the event of a key management service data loss, multiple keys (and the key tracking logs) may be lost, resulting in the inability to access encrypted client data.

[0031] In some cases, CI KMS can be a sharded system with multiple regional shards. A regional shard identifier can encode the combination of CI realm, region, and KMS shard as an integer. A regional shard identifier can include a 32-bit integer representing each shard across regions / realms. This can allow for efficient computation to generate WAL membership.

[0032] Additionally, multiple WALs can be compared to determine if the WALs were identical up to a specified point. This can be performed by computing an incremental checksum for each log entry and comparing the actual entropy value to the calculated entropy value during recalculation. Performing anti-entropy checks can help identify bit flips or resolve conflicts between WALs and converge WALs between regions.

[0033] In addition, each WAL can include a large number (e.g., millions) of entries. Therefore, applying multiple WALs each time a host needs to be booted or cross-region replication needs to be set up for a client can be inefficient. In order to replicate WAL data across regions, WAL snapshots can be taken periodically and replicated across regions. Each snapshot can include an entropy value. Snapshots can be taken on each individual state machine at the same point in time, and a centralized snapshot orchestrator can compare entropy values ​​and promote snapshots to valid after reaching a consensus. In addition, the data can be processed by performing a checksum of the entire snapshot to determine whether a bit flip has occurred.

[0034] In many cases, different snapshot methods can be utilized to capture snapshots for RSM. A first snapshot method can include a stop-the-world snapshot, where no changes are allowed to the state of the RSM while the snapshot is taken. Another example snapshot method can include a concurrent snapshot, where a snapshot is taken while new operations are being processed, where WAL records can be processed while the snapshot is being taken. Additionally, a full snapshot can contain the entire replication state associated with an LSN. Alternatively, an incremental snapshot can include a series of snapshots where one snapshot depends on a previous snapshot. Incremental snapshots can be used to reduce the size of snapshots and the time required to create a snapshot.

[0035] This embodiment involves capturing snapshots of key management service data (e.g., WALs for multiple keys). A snapshot orchestrator can initiate snapshots across multiple hosts at specific instances (e.g., at a specified log sequence number (LSN) of the WAL) and synchronize the resulting snapshots. LSNs can include monotonically dense, increasing sequence numbers that can be used to sort WAL entries. The saved snapshots can include resilient replication of log data that can be efficiently restored by hosts in the event of a CI system outage.

[0036] The present embodiment may implement a snapshot orchestrator that may periodically orchestrate snapshots of data based on various thresholds. The snapshot orchestrator may implement a verification (anti-entropy) process to confirm and verify snapshot content across hosts in a cluster of host devices. Additionally, the snapshot orchestrator may obtain content with metadata from one or more HSMs so that client data does not leave the device or system boundaries according to one or more standards / protocols. Snapshot data may be encrypted and checksummed to ensure the security of the snapshot data. For example, a portion of the snapshot data may be exported from the snapshot data and compared to the snapshot data to detect errors in the snapshot data or to verify the integrity of the snapshot data. The snapshot data may be stored across multiple devices (e.g., stored across regions). Additionally, the payload structure of the snapshot data and the truncation of the snapshot data may enable efficient recovery of the snapshot data without an exponential increase in memory resources.

[0037] The snapshot orchestrator can include log record handlers that handle individual log records of application-specific types. A WAL handler can process WALs and dispatch them to one or more log record handlers. As WAL records are applied, the WAL handler can maintain the current state (e.g., the current LSN). The WAL manager can manage the WAL handlers based on WALs being added / removed from the system.

[0038] In some embodiments, a method for obtaining a snapshot of key management data is provided. A snapshot orchestrator service executing on a device in a CI system can perform the method as described herein. The snapshot orchestrator can send requests for snapshot data from multiple host devices. The snapshot captured by the host device can include key management data (e.g., a modified WAL that provides a string of multiple keys for encrypting client data) and any metadata associated with the snapshot data (e.g., identifying the host device, the LSN instance at which the snapshot data was obtained). The multiple host devices can include HSM nodes and / or database nodes deployed across one or more regions in the CI system.

[0039] A request for snapshot data may be sent to a series of host devices. A snapshot service may obtain a snapshot instance at each host device and provide data (e.g., an entropy value of the snapshot data) to a snapshot orchestration device. The snapshot service may obtain the snapshot data in response to the snapshot data being in a specified state (e.g., a specified LSN instance in a write append log (WAL)). In some cases, the snapshot orchestrator may compare the obtained entropy values ​​received from the snapshot service and, upon success, may upload the set of snapshot data to an external persistent storage repository.

[0040] The snapshot orchestrator device can obtain snapshot instances from multiple host devices and verify the snapshot data. This can include performing verification or anti-entropy processes to verify the snapshot data. In some cases, the WAL can be truncated at the point where the snapshot is taken to remove the portion of the WAL before the specified LSN instance.

[0041] The snapshot orchestrator can store snapshot data on storage nodes. In some cases, the primary storage node can serve as the point of truth for snapshot data instances, enabling coordination and synchronization of snapshot data between nodes in cross-region storage. In the event of an outage or loss of critical management data, snapshot data can be retrieved to efficiently serve key data to clients.

[0042] A. Overview of Snapshot Management System

[0043] The system described herein can store client keys in one or more hardware security modules (HSMs) and store associated metadata in on-disk repositories across one or more regions. This can be accomplished through an internal replication system that keeps data synchronized across these hosts in different regions. In addition, the system described herein can periodically take snapshots of these HSMs and associated metadata repositories to back up client key data and compress the replication log. This allows for efficient retrieval of key data in the event of a failure or loss of key data in one or more regions.

[0044] Figure 1 is a block diagram of an example snapshot management system 100. The snapshot management system 100 can provide a snapshot infrastructure that efficiently captures snapshot instances and associated metadata. The snapshot management system 100 can schedule snapshots periodically based on various configurable thresholds. The snapshot management system 100 can also verify the obtained snapshot instances (e.g., by performing anti-entropy checks).

[0045] In some cases, the snapshot management system 100 can encrypt or perform a checksum on the snapshot instance to increase the security of the snapshot instance. In addition, the snapshot management system 100 can allow cross-region storage of snapshot instances. The snapshot payload structure can allow for efficient recovery of snapshot instances without memory bloat.

[0046] like Figure 1 As shown in FIG, snapshot management system 100 may include snapshot orchestrator 102. Snapshot orchestrator 102 may request snapshot instances from multiple host nodes (e.g., 106A-B) across one or more regions. Snapshot orchestrator 102 may further process the obtained snapshot instances (e.g., WAL log records) to verify the snapshot instances. In response to the verification of the snapshot instances, the instances may be stored for subsequent retrieval.

[0047] Snapshot orchestrator 102 may include a snapshot retrieval subsystem 104. Snapshot retrieval subsystem 104 may initiate a request for a snapshot instance from host nodes 106A-B. For example, snapshot retrieval subsystem 104 may add an entry (e.g., at an LSN instance) to the WAL requesting that a snapshot be taken at the specified LSN instance. As another example, snapshot retrieval subsystem 104 may send a message to host nodes 106A-B requesting that a snapshot be taken by each host node 106A-B.

[0048] Snapshot orchestrator 102 can run within the cluster manager and initiate a snapshot on a host node by adding a new entry to the WAL. Snapshot orchestrator 102 can perform an anti-entropy check. For example, if an anti-entropy check fails, if a portion of the host nodes have a common anti-entropy value, the node on which the snapshot failed can be removed and allowed to catch up. Alternatively, an alert can be issued.

[0049] The snapshot management system 100 may include multiple host nodes (e.g., 106A-B). The host nodes 106A-B may include HSMs deployed across multiple regions in the CI service. Each host node 106A-B may maintain a separate log of all keys created / removed for a client. In addition, each host node 106A-B may include a snapshot generation subsystem 108A-B to detect requests to capture snapshots, interact with a snapshotter 110A-B to capture snapshots, and provide snapshot instances to the snapshot orchestrator 102. A snapshotter may be a worker node running within each replica / service instance that is configured to perform tasks related to snapshot taking (e.g., performing snapshot taking, local storage of snapshots, pulling / pushing remote snapshots, etc.).

[0050] Each snapshotter 110A-B may be executed within each host node instance 106A-B. Snapshotters 110A-B may be part of a write-ahead log manager and may be responsible for local snapshotting and storage of snapshots. Snapshotters 110A-B may expose a set of application programming interfaces (APIs) that may be called by snapshot orchestrator 102. In some cases, multiple previous snapshots may be stored remotely at each snapshotter 110A-B.

[0051] The snapshot retrieval subsystem 104 can obtain the snapshot instances 112A-B provided by each host node 106A-B. The snapshot instances 112A-B can be provided to the snapshot verification subsystem 114 for verification of the snapshot instances 112A-B. For example, the snapshot verification subsystem 114 can compare the snapshot instances 112A-B to identify any differences in the instances 112A-B, truncate the snapshot instances 112A-B, perform anti-entropy checks, etc.

[0052] The verified snapshot instance 116 can be stored at the storage module 118. The storage module 118 can maintain multiple snapshot instances for subsequent retrieval in the event of a failure of any host node 106A-B. For example, in response to a failure of the host node 106a, the most recent snapshot instance can be provided to the host node 106a to replicate the WAL for the client.

[0053] Figure 2 A block diagram 200 illustrates an example of a system for capturing a snapshot of key management data. Figure 2 As shown in , the snapshot orchestrator 102 can request snapshots from a series of snapshot services executed on the host device. Figure 2 As shown in , write-ahead log 202 may include an append-only log, which may include multiple instances (e.g., 204A-B). Additionally, snapshot orchestrator 102 may add an entry (e.g., 206) to WAL 202 that includes an LSN instance instructing host node 106A-B to capture a snapshot. Host node 106A-B may, in turn, identify WAL instance 206 in WAL 202 and initiate the capture of a snapshot instance, as described herein.

[0054] The snapshot orchestrator 102 can orchestrate the snapshot process across multiple host nodes, which can be responsible for pushing the snapshots to the storage module 118 for long-term storage. Each region can include a pair of regions to achieve cross-region redundancy. The snapshot orchestrator 102 can push the snapshots to the storage module 118 for each region. The storage module 118 can delete the snapshots associated with the WAL after a preconfigured tombstoning interval (e.g., after 30 days).

[0055] Snapshot orchestrator 102 can track WAL snapshots and associated metadata in a table (e.g., a database such as a key-value store). Snapshot orchestrator 102 can run as a lease-based daemon and periodically request snapshots based on a time interval. Snapshot orchestrator 102 can expose an API to execute snapshots on demand by writing to WAL 202. Additionally, snapshot orchestrator 102 daemon can perform WAL truncation based on the most recent snapshot LSN. Snapshots can be initiated in response to one or more conditions being met. Example conditions can be based on the most recent snapshot creation time being greater than the snapshot interval and the state of the most recent snapshot in progress. Another example condition can be based on the most recent LSN in the WAL being greater than the maximum allowed record in the WAL. In response to any of the conditions being met, a log entry can be written to the WAL and a snapshot metadata record can be added to the metadata table for the snapshot in progress.

[0056] In some cases, multiple key partition backups can capture snapshots of partitions for the HSM. This can provide an encrypted snapshot for each partition on the HSM. Any backup functionality provided by the HSM (configuration, keys, and users) can be included in an encrypted blob (binary large object). The backup can be verified and the accuracy of the data can be checked to ensure accuracy in the snapshot (for example, using a cyclic redundancy check (CRC), such as a CRC32 error detection algorithm, etc.). This verification can include running a checksum as the partition data is read from the card. The backup can be stored in a designated storage device, and when the restore function is initialized, the restore function can be run on a new temporary partition that has passed the integrity check.

[0057] In some cases, when a snapshot entry is detected in the WAL, all host nodes can run a backup on their HSM cards. However, because the backups are encrypted, it can be difficult to compare backups to each other. Therefore, all data for each backup type can be downloaded and returned along with an encrypted blob of data. The orchestrator can then use metadata to sort the data and compare the backups to determine if they point to the same objects.

[0058] Each host node 106A-B can implement a snapshotter 110A-B. Snapshotter 110A-B can run as part of a write-ahead log manager. Snapshotter 110A-B can capture snapshots and save them locally. Snapshotter 110A-B can also expose an API that allows returning the state of the snapshotter associated with a particular LSN, listing locally existing snapshots for an LSN, and returning snapshots that can be written directly to local / remote storage modules.

[0059] Snapshotters 110A-B can capture new local snapshots. Requests for snapshots can flow through the WAL for each host node 106A-B using a specific log record type. The WAL manager can then call the WAL handler to obtain the snapshot, where the snapshot writer is responsible for writing the snapshot and associated metadata.

[0060] In some cases, snapshots can be stored as a collection of "mini" snapshots within the overall snapshot for the WAL. Each logging handler processor can be responsible for versioning its data in the snapshot associated with the specific logging handler. The snapshot writer can be responsible for versioning the entire snapshot file and performing integrity checks. The integrity check can be signed with the snapshot's cryptographic key value via a private key available on each individual machine.

[0061] Downloaded snapshots and / or locally saved snapshots are stored in subdirectories within the WAL manager. Snapshots and the state of each corresponding device can be managed by the snapshot manager. Each snapshot can reside in a storage subdirectory, named after the specified LSN, and with a marker file indicating whether the snapshot is complete and verified.

[0062] B. Snapshot Orchestrator

[0063] As described above, the snapshot orchestrator can obtain snapshots from one or more host nodes and maintain the snapshots (and corresponding metadata) to recreate the WAL in the event of the loss of any host node. For example, in response to a failure in a region that includes a host node, the snapshot orchestrator can provide the snapshot and associated metadata to the host node to recreate the WAL that provides client key management data.

[0064] Figure 3 A block diagram 300 of an example snapshot orchestrator 302 is illustrated. Snapshot orchestrator 302 can obtain snapshot data (WALs) from one or more host nodes, as described herein. Snapshot orchestrator 302 can poll the obtained WALs and perform additional processing (e.g., logical deletion). Snapshots received from host nodes and associated metadata can be maintained in tables. The metadata can include the host node's name, LSN instance, snapshot status, time the record was created, and the like. The snapshot(s) and corresponding metadata maintained in the tables can be stored locally at WAL repository 304.

[0065] Snapshot orchestrator 302 can truncate the WAL via WAL truncation 306. For example, a snapshot captured by each host node can be truncated at a specified LSN instance. Truncating the snapshot can reduce the size of each snapshot while also allowing for an accurate representation of the WAL for recreating the WAL at a given host node.

[0066] The snapshots and corresponding metadata may be stored at storage module 310. Storage module 310 may include one or more interconnected storage nodes, allowing efficient storage and retrieval of snapshot data in response to requests from orchestrator 302 for snapshots.

[0067] The cross-region snapshot replicator 308 may include a component responsible for transporting backups and existing snapshots between regions. If any differences exist between nodes in different regions, the cross-region snapshot replicator 308 may move through the snapshot metadata table and replicate the snapshot data across regions. The cross-region snapshot replicator 308 may include a lease-based daemon that can connect to each region individually. For each snapshot whose status indicates a successful snapshot capture, the snapshot may be replicated to the cross-region snapshot replicator 308. The data replication handler of the cross-region snapshot replicator 308 may be responsible for applying the data replication WAL record to the local database. Each local database may include fewer than a threshold number of entries. In some cases, snapshotting may only block writes to data, and the database may be read by the snapshot orchestrator's data plane while the snapshot is in progress. The local database may include an array of key-value pairs. The key-value pairs may be sorted in the database by LSN value and calculated CRC32 value. When a data record is updated, the CRC32 value may be incrementally modified. The CRC32 function converts a variable-length string into an 8-character string, which is a textual representation of the hexadecimal value of a 32-bit binary sequence. Therefore, as the data becomes more accurate, the CRC32 value can improve.

[0068] C. Snapshot data bitmap

[0069] Figure 4 4 is an example bitmap 400 for snapshot instances. Snapshot data may include snapshot instances obtained from a host node, as described herein. Snapshot data may include multiple handler snapshots of variable length. Handler snapshots may include individual snapshot instances obtained by each snapshot service.

[0070] The bitmap for snapshot data 400 may include any of the handler snapshot fields 402, 404, the snapshot header field 406, the CRC32 field 408, and the version field 410. For example, a first snapshot instance (e.g., an instance from an HSM) may be included in the first handler snapshot field 402, and a second snapshot instance (e.g., an instance from a Berkeley database (BDB), etc.) may be included in the second handler snapshot field 404. The snapshot instances may be combined by the snapshot orchestrator to form a snapshot payload that may be subsequently parsed and restored by a node (e.g., an HSM, a BDB node). The handler snapshot fields 402, 404 may include a variable length (vlen).

[0071] The snapshot data may also include a snapshot header 406 that identifies the size of the snapshot in bytes. The snapshot data may be encrypted using asymmetric keys using Public Key Cryptography Standard (PKCS) encapsulation encryption. The snapshot data may also include a cyclic redundancy check (CRC32) field 408 and a version field 410 that specifies the version of the node providing the snapshot data. The CRC32 field 408 and the version field 410 may be unencrypted to allow confirmation of the validity of each snapshot instance without having to decrypt the snapshot instance or access the public key.

[0072] A snapshot instance can be associated with a metadata file that specifies metadata (e.g., name and offset) for the handler snapshot. For example, this can include the LSN associated with the snapshot, the write-ahead log name, handler snapshot metadata for each handler snapshot, the number of handler snapshots, the length of the snapshot including the header (in bytes), the version of the snapshot, the log record type, the entropy value, etc.

[0073] Figure 5 Illustrated is a block diagram of an example snapshot file format bitmap 500. The snapshot file format bitmap 500 may include a version field 502, a header CRC32 field 504, a header field 506, and a plurality of table fields (eg, 508, 510, 512).

[0074] The version field 502 may provide the version of the snapshot. The length of the snapshot may include a signed integer (e.g., 4 bytes, etc.) that limits the snapshot size to 2GB. The header CRC32 field 504 may include 4 bytes that specify a CRC of the remaining bytes comprising the header. The header field 506 may contain metadata required to index into each table. The file format may include multiple tables (e.g., Table 1, Table 2, Table 3) of variable length (e.g., in Table fields 508, 510, 512).

[0075] D. Flow process for capturing and managing snapshots of key management data

[0076] As described above, the snapshot management system can periodically capture snapshots of key management data across a series of nodes and store the snapshots for subsequent re-creation of the key management data. For example, in response to an outage at a first node, the most recent snapshot can be provided to the first node to re-create the key management data at that node. This can allow for efficient re-creation of key management data (e.g., modifications to client keys over time) in response to an outage event (e.g., a failure) at one or more nodes. Figure 66 is a block diagram of an example method 600 for capturing a snapshot of key management data across a series of nodes. For example, the method as described herein may be performed by a snapshot orchestrator (e.g., 102) interacting with host nodes (e.g., 106A-B) as part of a snapshot management system (e.g., 100).

[0077] At 602, the method may include requesting a snapshot instance from a series of nodes. Each node in the series of nodes may include a snapshotter capable of capturing a snapshot of key management data on each node. The key management data may include modifications (e.g., additions / removals) to any of a plurality of keys specific to a client. Each modification to the plurality of keys may be associated with an entry in a log record (e.g., a specified LSN instance in a write append log (WAL)). For example, in some cases, the series of nodes may be deployed across one or more regions in a cloud infrastructure service (such as across different data centers in different geographic regions). In this example, each modification to any of the plurality of client keys identified at a first node in a first region is synchronized across other nodes in other regions of the cloud infrastructure service via a cross-region snapshot replicator.

[0078] In some cases, requesting a snapshot includes adding an entry to a log record specifying a request to capture a snapshot instance. In response, each node in the series of nodes can capture a snapshot instance in response to identifying the addition of an entry to the log record. In some cases, a snapshot instance can be requested in response to expiration of a threshold duration (e.g., every 1 minute, every 10 minutes) or detection of a threshold number of entries being added to the log record (e.g., every 100 entries, every 1000 entries).

[0079] At 604, the snapshot orchestrator may obtain a snapshot instance and corresponding metadata from the series of nodes. The snapshot instance may include a snapshot of the log (e.g., WAL) on each node in the series of nodes. The metadata for each snapshot instance may include various aspects of the snapshot and / or the node that captured the snapshot. For example, the metadata may specify the log sequence number in the WAL that triggered the snapshot, the snapshot state, an identifier of the host node that captured the snapshot, an entropy value, etc. In some cases, the metadata includes a key for accessing a specified partition of the first node, each partition of the first node independently maintaining log records for the client. The snapshot orchestrator may use the key to access the partition to provide the stored snapshot instance and corresponding metadata to the partition of the first node.

[0080] At 606, the snapshot orchestrator may verify the set of snapshot data. This may include identifying that the set of snapshot data includes intelligible data that can be used to restore the state of the client's key at the specified instance. Additionally, this may include performing a verification or anti-entropy process to verify the snapshot data.

[0081] As an example, the snapshot orchestrator can identify an entropy value specific to each snapshot instance and each corresponding node. As used herein, an entropy value can include a value (or a series of values) derived from one or more characteristics, such as the LSN in which the snapshot is requested, a checksum value of the number of bytes in the header of the specified snapshot instance or the snapshot instance itself, the time when the snapshot was captured, etc. In other words, as described and used herein, in general, an entropy value is not necessarily a numerical measure of the uncertainty of the result, as may be used in the industry. Instead, according to the techniques described herein, an entropy value can be generated based on the above characteristics, and the entropy value can be specific to each snapshot instance and / or each corresponding node. The snapshot orchestrator can determine whether the identified entropy values ​​for each received snapshot instance are within a threshold similarity to each other. The snapshot instance can be verified in response to the entropy values ​​being within the threshold similarity. For example, if all entropy values ​​are the same or within a threshold similarity, then the snapshot instance can be verified.

[0082] In some cases, the snapshot orchestrator may truncate each snapshot instance to remove modifications to multiple client keys and the corresponding entries in the log record that precede the specified entry in the log record. For example, if each snapshot instance includes 1000 entries, the snapshot instance may be truncated to only the 100 most recent entries to improve the efficiency of storing the snapshot instance while still allowing all relevant client keys to be recreated.

[0083] In some cases, in response to verifying a snapshot instance, a table may be updated to include metadata corresponding to the snapshot instance. The metadata may relate to any of the following: a specified log sequence number for append-only log records, the state of the snapshot instance captured, and a timestamp at which the request for the snapshot instance was verified.

[0084] At 608, the snapshot orchestrator can store the snapshot data in the storage node and a series of host nodes (e.g., in a cross-region storage device). In some cases, the primary storage node can serve as a point of truth for the snapshot data instance, coordinating / synchronizing the snapshot data across nodes that independently store the WAL. In the event of an outage or loss of key management data, the snapshot data can be retrieved to efficiently provide key data to the client.

[0085] As an example, the snapshot orchestrator may obtain an outage notification at the first node. This may be due to an outage at the first node (e.g., a power outage) or another failure at the first node. The snapshot orchestrator may periodically check for outages at any node. In other cases, the node (or corresponding alarm system) may send an outage notification in response to detecting an outage at any node. The snapshot orchestrator may retrieve the stored snapshot instance and corresponding metadata from the storage node. In addition, the snapshot orchestrator may provide the stored snapshot instance and corresponding metadata to the first node. The first node may use the stored snapshot instance and corresponding metadata to recreate a log record specifying modifications to multiple client keys. In addition, various modifications and equivalent forms include relevant, appropriate combinations of the features disclosed in the embodiments.

[0086] E.IaaS Overview

[0087] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provide various services to accompany these infrastructure components (e.g., billing, monitoring, logging, load balancing, and clustering, etc.). Therefore, since these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0088] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise software into the VM. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, and more.

[0089] In most cases, the cloud computing model will require the involvement of a cloud provider. A cloud provider may, but need not, specialize in providing (e.g., provisioning, renting, selling) IaaS third-party services. An entity may also choose to deploy a private cloud, thereby becoming its own infrastructure service provider.

[0090] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server or the like. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling the deployment of the application (e.g., the OS), middleware, and / or (e.g., on self-service virtual machines (e.g., that can be launched on demand)).

[0091] In some examples, IaaS provisioning can refer to obtaining computers or virtual hosts for use, and even installing required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0092] In some cases, there are two distinct challenges with IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is running. Second, once everything has been provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both of these challenges can be addressed by enabling configuration that defines the infrastructure in a declarative manner. In other words, the infrastructure (e.g., which components are required and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which other resources, and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, workflows can be generated that create and / or manage the different components described in the configuration files.

[0093] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also referred to as core networks. In some examples, one or more inbound / outbound traffic group rules can also be provided to define how inbound / outbound traffic for the network will be set up, as well as one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., can also be provided. The infrastructure can evolve gradually as more and more infrastructure elements are desired and / or added.

[0094] In some cases, continuous deployment techniques can be employed to enable deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is intended to be deployed to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes across the world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, a provisioning tool can be utilized to provision resources, and / or once the infrastructure is provisioned, a deployment tool can be utilized to deploy the code.

[0095] Figure 7 7 is a block diagram 700 illustrating an example schema for an IaaS architecture according to at least one embodiment. A service operator 702 can be communicatively coupled to a secure host lease 704 that can include a virtual cloud network (VCN) 706 and a secure host subnet 708. In some examples, the service operator 702 can use one or more client computing devices (the client computing devices can be portable handheld devices (e.g., Cellular phones, computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google Head-mounted display)), running software (such as Microsoft Windows ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 9, Palm OS, etc.), and supports Internet, email, short message service (SMS), Alternatively, the client computing device may be a general-purpose personal computer, including, for example, a computer running various versions of Microsoft Apple The client computing device may be a personal computer and / or laptop computer running various commercially available or a workstation computer running any operating system in the UNIX or UNIX-like operating systems, including but not limited to various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., with or without gesture input device), and / or a personal messaging device capable of communicating over a network that has access to VCN 706 and / or the Internet.

[0096] VCN 706 may include a local peer gateway (LPG) 710, which may be communicatively coupled to a secure shell (SSH) VCN 712 via the LPG 710 contained in a SSH VCN 712. SSH VCN 712 may include an SSH subnet 714, and SSH VCN 712 may be communicatively coupled to a control plane VCN 716 via the LPG 710 contained in a control plane VCN 716. Furthermore, SSH VCN 712 may be communicatively coupled to a data plane VCN 718 via the LPG 710. The control plane VCN 716 and the data plane VCN 718 may be contained in a service lease 719, which may be owned and / or operated by an IaaS provider.

[0097] The control plane VCN 716 may include a control plane demilitarized zone (DMZ) layer 720 that serves as a perimeter network (e.g., a portion of a company's network between an internal company network and an external network). DMZ-based servers can have limited liability and help control breaches. Furthermore, the DMZ layer 720 may include one or more load balancer (LB) subnets 722, a control plane application layer 724 that may include (one or more) application (app) subnets 726, and a control plane data layer 728 that may include (one or more) database (DB) subnets 730 (e.g., (one or more) front-end DB subnets and / or (one or more) back-end DB subnets). LB subnet(s) 722 included in control plane DMZ layer 720 may be communicatively coupled to application subnet(s) 726 included in control plane application layer 724 and internet gateway 734 that may be included in control plane VCN 716, and application subnet(s) 726 may be communicatively coupled to DB subnet(s) 730 included in control plane data layer 728, as well as service gateway 736 and network address translation (NAT) gateway 738. Control plane VCN 716 may include service gateway 736 and NAT gateway 738.

[0098] The control plane VCN 716 may include a data plane mirror application layer 740, which may include application subnet(s) 726. The application subnet(s) 726 contained in the data plane mirror application layer 740 may include a virtual network interface controller (VNIC) 742 that may execute a compute instance 744. The compute instance 744 may communicatively couple the application subnet(s) 726 of the data plane mirror application layer 740 to the application subnet(s) 726 that may be contained in the data plane application layer 746.

[0099] The data plane VCN 718 may include a data plane application layer 746, a data plane DMZ layer 748, and a data plane data layer 750. The data plane DMZ layer 748 may include (one or more) LB subnets 722, which may be communicatively coupled to the application subnet(s) 726 of the data plane application layer 746 and the internet gateway 734 of the data plane VCN 718. The application subnet(s) 726 may be communicatively coupled to the service gateway 736 of the data plane VCN 718 and the NAT gateway 738 of the data plane VCN 718. The data plane data layer 750 may also include (one or more) DB subnet(s) 730, which may be communicatively coupled to the application subnet(s) 726 of the data plane application layer 746.

[0100] The internet gateway 734 of the control plane VCN 716 and the data plane VCN 718 can be communicatively coupled to a metadata management service 752, which can be communicatively coupled to the public internet 754. The public internet 754 can be communicatively coupled to the NAT gateway 738 of the control plane VCN 716 and the data plane VCN 718. The service gateway 736 of the control plane VCN 716 and the data plane VCN 718 can be communicatively coupled to a cloud service 756.

[0101] In some examples, service gateway 736 of control plane VCN 716 or data plane VCN 718 can make application programming interface (API) calls to cloud services 756 without going through public internet 754. API calls from service gateway 736 to cloud services 756 can be one-way: service gateway 736 can make API calls to cloud services 756, and cloud services 756 can send requested data to service gateway 736. However, cloud services 756 may not initiate API calls to service gateway 736.

[0102] In some examples, secure host lease 704 can be directly connected to service lease 719, which can otherwise be isolated. Secure host subnet 708 can communicate with SSH subnet 714 via LPG 710, which can enable two-way communication on otherwise isolated systems. Connecting secure host subnet 708 to SSH subnet 714 can enable secure host subnet 708 to access other entities within service lease 719.

[0103] Control plane VCN 716 can allow users of service leases 719 to set up or otherwise provision desired resources. The desired resources provisioned in control plane VCN 716 can be deployed or otherwise used in data plane VCN 718. In some examples, control plane VCN 716 can be isolated from data plane VCN 718, and a data plane mirror application layer 740 of control plane VCN 716 can communicate with a data plane application layer 746 of data plane VCN 718 via a VNIC 742, which can be included in both data plane mirror application layer 740 and data plane application layer 746.

[0104] In some examples, a user or customer of the system may make a request, such as a create, read, update, or delete (CRUD) operation, over the public internet 754, which may transmit the request to the metadata management service 752. The metadata management service 752 may transmit the request to the control plane VCN 716 via the internet gateway 734. The request may be received by the LB subnet(s) 722 contained in the control plane DMZ layer 720. The LB subnet(s) 722 may determine that the request is valid, and in response to this determination, the LB subnet(s) 722 may transmit the request to the application subnet(s) 726 contained in the control plane application layer 724. If the request is validated and requires a call to the public internet 754, the call to the public internet 754 may be transmitted to the NAT gateway 738, which may make the call to the public internet 754. The storage that the request may desire to be stored may be stored in the DB subnet(s) 730.

[0105] In some examples, data plane mirror application layer 740 can facilitate direct communication between control plane VCN 716 and data plane VCN 718. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in data plane VCN 718. Control plane VCN 716, via VNIC 742, can communicate directly with resources contained in data plane VCN 718 and, thereby, can perform configuration changes, updates, or other appropriate modifications to these resources.

[0106] In some embodiments, control plane VCN 716 and data plane VCN 718 may be included in service lease 719. In this case, a user or customer of the system may not own or operate control plane VCN 716 or data plane VCN 718. Alternatively, the IaaS provider may own or operate control plane VCN 716 and data plane VCN 718, both of which may be included in service lease 719. This embodiment may enable the isolation of networks that may prevent a user or customer from interacting with the resources of other users or other customers. Furthermore, this embodiment may allow users or customers of the system to store databases privately without relying on the public internet 754, which may not have the desired level of threat protection, for storage.

[0107] In other embodiments, the LB subnet(s) 722 contained in the control plane VCN 716 can be configured to receive signals from the service gateway 736. In this embodiment, the control plane VCN 716 and the data plane VCN 718 can be configured to be called by customers of the IaaS provider without calling the public internet 754. Customers of the IaaS provider may desire this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on the service lease 719, which can be isolated from the public internet 754.

[0108] Figure 8 is a block diagram 800 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 802 (e.g., Figure 7 The service operator 702) can be communicatively coupled to the secure host lease 804 (e.g., Figure 7 ), which may include a virtual cloud network (VCN) 806 (e.g., Figure 7 VCN 706) and secure host subnet 808 (e.g., Figure 7 The VCN 806 may include a local peer gateway (LPG) 810 (e.g., Figure 7 LPG 710), the VCN 806 can be accessed via a secure shell (SSH) VCN 812 (e.g., Figure 7 The LPG 710 in the SSH VCN 712 of FIG. 812 is communicatively coupled to the SSH VCN 812. The SSH VCN 812 may include an SSH subnet 814 (e.g., Figure 7 SSH subnet 714), and SSH VCN 812 can be accessed via a control plane VCN 816 (e.g., Figure 7The LPG 810 in the control plane VCN 716 of the embodiment of the present invention is communicatively coupled to the control plane VCN 816. The control plane VCN 816 can be included in the service lease 819 (e.g., Figure 7 service lease 719), and the data plane VCN 818 (e.g., Figure 7 The data plane VCN 718) may be contained in a customer lease 821 that may be owned or operated by a user or customer of the system.

[0109] The control plane VCN 816 may include a control plane DMZ layer 820 (e.g., Figure 7 The control plane DMZ layer 720) may include (one or more) LB subnets 822 (e.g., Figure 7 (one or more) LB subnets 722), may include (one or more) application subnets 826 (e.g., Figure 7 (one or more) application subnets 726) of the control plane application layer 824 (e.g., Figure 7 The control plane application layer 724 of FIG. 1 may include (one or more) database (DB) subnets 830 (e.g., similar to Figure 7 (one or more) DB subnet 730) of the control plane data layer 828 (e.g., Figure 7 828). The LB subnet(s) 822 contained in the control plane DMZ layer 820 may be communicatively coupled to the application subnet(s) 826 contained in the control plane application layer 824 and the internet gateway 834 (e.g., Figure 7 734), and the application subnet(s) 826 may be communicatively coupled to the DB subnet(s) 830 and the service gateway 836 (e.g., Figure 7 Serving gateway) and Network Address Translation (NAT) gateway 838 (e.g., Figure 7 The control plane VCN 816 may include a serving gateway 836 and a NAT gateway 838.

[0110] The control plane VCN 816 may include a data plane mirror application layer 840 (e.g., Figure 7 The data plane image application layer 740). The application subnet(s) 826 included in the data plane image application layer 840 may include a computing instance 844 (e.g., similar to Figure 7The compute instance 844 can facilitate the application subnet(s) 826 of the data plane mirroring application layer 840 and can be included in the data plane application layer 846 (e.g., Figure 7 Communication between (one or more) application subnets 826 in the data plane application layer 746) via the VNIC 842 contained in the data plane mirror application layer 840 and the VNIC 842 contained in the data plane application layer 846.

[0111] The internet gateway 834 included in the control plane VCN 816 can be communicatively coupled to the metadata management service 852 (e.g., Figure 7 , which can be communicatively coupled to a public Internet 854 (e.g., Figure 7 The public internet 854 can be communicatively coupled to a NAT gateway 838 included in the control plane VCN 816. The service gateway 836 included in the control plane VCN 816 can be communicatively coupled to cloud services 856 (e.g., Figure 7 Cloud service 756).

[0112] In some examples, the data plane VCN 818 can be contained in a customer lease 821. In this case, the IaaS provider can provision a control plane VCN 816 for each customer, and the IaaS provider can provision each customer with a unique compute instance 844 contained in a service lease 819. Each compute instance 844 can enable communication between the control plane VCN 816 contained in the service lease 819 and the data plane VCN 818 contained in the customer lease 821. The compute instance 844 can enable resources provisioned in the control plane VCN 816 contained in the service lease 819 to be deployed or otherwise used in the data plane VCN 818 contained in the customer lease 821.

[0113] In another example, a customer of the IaaS provider may have a database that resides in customer tenancy 821. In this example, control plane VCN 816 may include a data plane mirror application layer 840, which may include application subnet(s) 826. Data plane mirror application layer 840 may reside in data plane VCN 818, but data plane mirror application layer 840 may not reside in data plane VCN 818. In other words, data plane mirror application layer 840 may access customer tenancy 821, but data plane mirror application layer 840 may not reside in data plane VCN 818 or be owned or operated by the customer of the IaaS provider. Data plane mirror application layer 840 may be configured to make calls to data plane VCN 818, but may not be configured to make calls to any entity contained in control plane VCN 816. A customer may desire to deploy or otherwise use resources provisioned in control plane VCN 816 in data plane VCN 818, and data plane mirror application layer 840 may facilitate the customer's desired deployment or other use of resources.

[0114] In some embodiments, a customer of an IaaS provider can apply filters to data plane VCN 818. In this embodiment, the customer can determine what data plane VCN 818 can access, and the customer can restrict access from data plane VCN 818 to the public internet 854. The IaaS provider may not apply filters or otherwise control access to any external networks or databases by data plane VCN 818. The customer applying filters and controls to data plane VCN 818 contained in a customer lease 821 can help isolate data plane VCN 818 from other customers and the public internet 854.

[0115] In some embodiments, cloud services 856 can be called by service gateway 836 to access services that may not exist on the public internet 854, control plane VCN 816, or data plane VCN 818. The connection between cloud services 856 and control plane VCN 816 or data plane VCN 818 may not be real-time or continuous. Cloud services 856 may exist on different networks owned or operated by the IaaS provider. Cloud services 856 can be configured to receive calls from service gateway 836 and not to receive calls from the public internet 854. Some cloud services 856 may be isolated from other cloud services 856, and control plane VCN 816 may be isolated from cloud services 856 that may not be in the same region as control plane VCN 816. For example, control plane VCN 816 may be located in "Region 1," and cloud service "Deployment 7" may be located in both Region 1 and "Region 2." If service gateway 836 contained in control plane VCN 816 located in region 1 makes a call to deployment 7, the call may be transmitted to deployment 7 in region 1. In this example, control plane VCN 816 or deployment 7 in region 1 may not be communicatively coupled or otherwise in communication with deployment 7 in region 2.

[0116] Figure 9 is a block diagram 900 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 902 (e.g., Figure 7 The service operator 702) can be communicatively coupled to the secure host lease 904 (e.g., Figure 7 ), which may include a virtual cloud network (VCN) 906 (e.g., Figure 7 VCN 706) and secure host subnet 908 (e.g., Figure 7 The VCN 906 may include an LPG 910 (e.g., Figure 7 LPG 710), the VCN 906 can be connected to the SSH VCN 912 (e.g., Figure 7 The LPG 910 in the SSH VCN 712 of FIG. 7 is communicatively coupled to the SSH VCN 912. The SSH VCN 912 may include an SSH subnet 914 (e.g., Figure 7 SSH subnet 714), and SSH VCN 912 can be accessed via a control plane VCN 916 (e.g., Figure 7 The LPG 910 in the control plane VCN 716 of the embodiment of the present invention is communicatively coupled to the control plane VCN 916 and is connected to the control plane VCN 916 via a data plane VCN 918 (e.g., Figure 7The LPG 910 in the data plane 718 of the control plane is communicatively coupled to the data plane VCN 918. The control plane VCN 916 and the data plane VCN 918 can be included in the service lease 919 (e.g., Figure 7 Service lease 719).

[0117] The control plane VCN 916 may include a load balancer (LB) subnet 922 (e.g., Figure 7 (one or more) LB subnet 722) of the control plane DMZ layer 920 (e.g., Figure 7 The control plane DMZ layer 720 may include (one or more) application subnets 926 (e.g., similar to Figure 7 (one or more) application subnets 726) of the control plane application layer 924 (e.g., Figure 7 The control plane application layer 724 of FIG. 1 ), the control plane data layer 928 (eg, Figure 7 928). The LB subnet(s) 922 contained in the control plane DMZ layer 920 may be communicatively coupled to the application subnet(s) 926 contained in the control plane application layer 924 and the internet gateway 934 (e.g., Figure 7 ), and the application subnet(s) 926 may be communicatively coupled to the DB subnet(s) 930 and the service gateway 936 (e.g., Figure 7 Serving gateway) and Network Address Translation (NAT) gateway 938 (e.g., Figure 7 The control plane VCN 916 may include a serving gateway 936 and a NAT gateway 938.

[0118] The data plane VCN 918 may include a data plane application layer 946 (e.g., Figure 7 Data plane application layer 746), data plane DMZ layer 948 (e.g., Figure 7 Data plane DMZ layer 748), and data plane data layer 950 (e.g., Figure 7960 and the data plane data layer 750). The data plane DMZ layer 948 may include trusted application subnet(s) 960 and untrusted application subnet(s) 962 that may be communicatively coupled to the data plane application layer 946, and LB subnet(s) 922 of the internet gateway 934 included in the data plane VCN 918. The trusted application subnet(s) 960 may be communicatively coupled to the service gateway 936 included in the data plane VCN 918, the NAT gateway 938 included in the data plane VCN 918, and the DB subnet(s) 930 included in the data plane data layer 950. The untrusted application subnet(s) 962 may be communicatively coupled to the service gateway 936 included in the data plane VCN 918 and the DB subnet(s) 930 included in the data plane data layer 950. The data plane data layer 950 may include DB subnet(s) 930 that may be communicatively coupled to the service gateway 936 included in the data plane VCN 918.

[0119] The untrusted application subnet(s) 962 may include one or more primary VNICs 964(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N). Each tenant VM 966(1)-(N) may be communicatively coupled to a respective application subnet 967(1)-(N) that may be contained in a respective container egress VCN 968(1)-(N), which may be contained in a respective customer lease 970(1)-(N). The respective secondary VNICs 972(1)-(N) may facilitate communication between the untrusted application subnet(s) 962 contained in the data plane VCN 918 and the application subnets contained in the container egress VCNs 968(1)-(N). Each container egress VCN 968(1)-(N) may include a NAT gateway 938 that may be communicatively coupled to the public Internet 954 (e.g., Figure 7 public Internet 754).

[0120] The internet gateway 934 included in the control plane VCN 916 and included in the data plane VCN 918 can be communicatively coupled to the metadata management service 952 (e.g., Figure 7952 can be communicatively coupled to a public internet 954. The public internet 954 can be communicatively coupled to a NAT gateway 938 included in the control plane VCN 916 and in the data plane VCN 918. The service gateway 936 included in the control plane VCN 916 and in the data plane VCN 918 can be communicatively coupled to a cloud service 956.

[0121] In some embodiments, the data plane VCN 918 can be integrated with the customer lease 970. In some cases, such integration may be useful or desirable for the IaaS provider's customers, such as when support may be desired when executing code. A customer may provide code to be executed that may be destructive, may communicate with other customer resources, or may otherwise cause undesirable effects. In response, the IaaS provider may determine whether to execute the code provided by the customer to the IaaS provider.

[0122] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider and request functionality attached to a data plane layer application 946. The code running the functionality may be executed in a VM 966(1)-(N), and the code may not be configured to run anywhere else on the data plane VCN 918. Each VM 966(1)-(N) may be connected to a customer lease 970. The corresponding container 971(1)-(N) contained in the VM 966(1)-(N) may be configured to run the code. In this case, there may be double isolation (e.g., the container 971(1)-(N) runs the code, where the container 971(1)-(N) may be contained in at least one VM 966(1)-(N) contained in the untrusted application subnet(s) 962), which may help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging a different customer's network. Containers 971(1)-(N) may be communicatively coupled to customer lease 970 and may be configured to transmit or receive data from customer lease 970. Containers 971(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 918. After running the code, the IaaS provider may terminate or otherwise dispose of containers 971(1)-(N).

[0123] In some embodiments, the trusted application subnet(s) 960 may run code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet(s) 960 may be communicatively coupled to the DB subnet(s) 930 and configured to perform CRUD operations in the DB subnet(s) 930. The untrusted application subnet(s) 962 may be communicatively coupled to the DB subnet(s) 930, but in this embodiment, the untrusted application subnet(s) may be configured to perform read operations in the DB subnet(s) 930. Containers 971(1)-(N), which may be included in each customer's VM 966(1)-(N) and may run code from the customer, may not be communicatively coupled to the DB subnet(s) 930.

[0124] In other embodiments, control plane VCN 916 and data plane VCN 918 may not be directly communicatively coupled. In this embodiment, direct communication may not exist between control plane VCN 916 and data plane VCN 918. However, communication may occur indirectly through at least one method. LPG 910 may be established by an IaaS provider, which may facilitate communication between control plane VCN 916 and data plane VCN 918. In another example, control plane VCN 916 or data plane VCN 918 may make a call to cloud service 956 via service gateway 936. For example, a call from control plane VCN 916 to cloud service 956 may include a request for a service that can communicate with data plane VCN 918.

[0125] Figure 10 is a block diagram 1000 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 1002 (e.g., Figure 7 The service operator 702) can be communicatively coupled to the secure host lease 1004 (e.g., Figure 7 ), which may include a virtual cloud network (VCN) 1006 (e.g., Figure 7 VCN 706) and secure host subnet 1008 (e.g., Figure 7 The VCN 1006 may include an LPG 1010 (e.g., Figure 7 LPG 710), the VCN 1006 can be connected to the VCN 1012 via the SSH VCN 1012 (e.g., Figure 7 The LPG 1010 in the SSH VCN 712 is communicatively coupled to the SSH VCN 1012. The SSH VCN 1012 may include an SSH subnet 1014 (e.g., Figure 7 SSH subnet 714), and SSH VCN 1012 can be accessed via a control plane VCN 1016 (e.g., Figure 7 The LPG 1010 in the control plane VCN 716 of the embodiment of the present invention is communicatively coupled to the control plane VCN 1016 and is communicated to the control plane VCN 1016 via a LPG 1010 contained in the data plane VCN 1018 (e.g., Figure 7 The LPG 1010 in the data plane 718 of the control plane is communicatively coupled to the data plane VCN 1018. The control plane VCN 1016 and the data plane VCN 1018 can be included in the service lease 1019 (e.g., Figure 7 Service lease 719).

[0126] The control plane VCN 1016 may include (one or more) LB subnets 1022 (e.g., Figure 7 (one or more) LB subnet 722) of the control plane DMZ layer 1020 (e.g., Figure 7 The control plane DMZ layer 720 may include (one or more) application subnets 1026 (e.g., Figure 7 (one or more) application subnets 726) of the control plane application layer 1024 (e.g., Figure 7 The control plane application layer 724 of FIG. 10 may include (one or more) DB subnets 1030 (e.g., Figure 9 (one or more) DB subnet 930) of the control plane data layer 1028 (e.g., Figure 7 1028). The LB subnet(s) 1022 contained in the control plane DMZ layer 1020 may be communicatively coupled to the application subnet(s) 1026 contained in the control plane application layer 1024 and the internet gateway 1034 (e.g., Figure 7 1024 ), and the application subnet(s) 1026 may be communicatively coupled to the DB subnet(s) 1030 and the service gateway 1036 (e.g., Figure 7 Serving gateway) and Network Address Translation (NAT) gateway 1038 (e.g., Figure 7 The control plane VCN 1016 may include a service gateway 1036 and a NAT gateway 1038.

[0127] The data plane VCN 1018 may include a data plane application layer 1046 (e.g., Figure 7 Data plane application layer 746), data plane DMZ layer 1048 (e.g., Figure 7Data plane DMZ layer 748)), and data plane data layer 1050 (e.g., Figure 7 The data plane DMZ layer 1048 may include trusted application subnet(s) 1060 (e.g., Figure 9 (one or more) trusted application subnets 960) and (one or more) untrusted application subnets 1062 (e.g., Figure 9 1018) and the LB subnet(s) 1022 of the internet gateway 1034 contained in the data plane VCN 1018. The trusted application subnet(s) 1060 can be communicatively coupled to the service gateway 1036 contained in the data plane VCN 1018, the NAT gateway 1038 contained in the data plane VCN 1018, and the DB subnet(s) 1030 contained in the data plane data layer 1050. The untrusted application subnet(s) 1062 can be communicatively coupled to the service gateway 1036 contained in the data plane VCN 1018 and the DB subnet(s) 1030 contained in the data plane data layer 1050. The data plane data layer 1050 can include the DB subnet(s) 1030 that can be communicatively coupled to the service gateway 1036 contained in the data plane VCN 1018.

[0128] The untrusted application subnet(s) 1062 may include primary VNICs 1064(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1066(1)-(N) residing within the untrusted application subnet(s) 1062. Each tenant VM 1066(1)-(N) may run code in a corresponding container 1067(1)-(N) and may be communicatively coupled to the application subnet 1026 that may be contained in the data plane application layer 1046, which may be contained in the container egress VCN 1068. Corresponding secondary VNICs 1072(1)-(N) may facilitate communications between the untrusted application subnet(s) 1062 contained in the data plane VCN 1018 and the application subnets contained in the container egress VCN 1068. The container egress VCN may include a secondary VNIC that may be communicatively coupled to the public internet 1054 (e.g., Figure 7 NAT gateway 1038 of the public Internet 754).

[0129] The internet gateway 1034 included in the control plane VCN 1016 and included in the data plane VCN 1018 can be communicatively coupled to the metadata management service 1052 (e.g., Figure 710), which can be communicatively coupled to a public internet 1054. The public internet 1054 can be communicatively coupled to a NAT gateway 1038 included in the control plane VCN 1016 and included in the data plane VCN 1018. The service gateway 1036 included in the control plane VCN 1016 and included in the data plane VCN 1018 can be communicatively coupled to a cloud service 1056.

[0130] In some examples, Figure 10 The architecture model shown in block diagram 1000 can be considered as Figure 9 900 , and may be desirable to customers of the IaaS provider in situations where the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access each customer's respective container 1067(1)-(N) contained in a VM 1066(1)-(N) in real time. The container 1067(1)-(N) can be configured to make calls to a respective secondary VNIC 1072(1)-(N) contained in the application subnet(s) 1026 of the data plane application layer 1046, which can be contained in the container egress VCN 1068. The secondary VNIC 1072(1)-(N) can transmit the call to the NAT gateway 1038, which can transmit the call to the public internet 1054. In this example, containers 1067(1)-(N), which may be accessed by customers in real time, may be isolated from control plane VCN 1016 and may be isolated from other entities contained in data plane VCN 1018. Containers 1067(1)-(N) may also be isolated from resources from other customers.

[0131] In another example, a customer can use containers 1067(1)-(N) to call cloud service 1056. In this example, the customer can run code in containers 1067(1)-(N) that requests a service from cloud service 1056. Containers 1067(1)-(N) can transmit the request to secondary VNICs 1072(1)-(N), which can transmit the request to a NAT gateway, which can transmit the request to the public internet 1054. The public internet 1054 can transmit the request to LB subnet(s) 1022 contained in control plane VCN 1016 via internet gateway 1034. In response to determining that the request is valid, the LB subnet(s) can transmit the request to application subnet(s) 1026, which can transmit the request to cloud service 1056 via service gateway 1036.

[0132] It should be appreciated that the IaaS architectures 700, 800, 900, and 1000 depicted in the figures may have other components in addition to those depicted. Furthermore, the embodiments shown in the figures are merely examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have a different arrangement or configuration of components.

[0133] In certain embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0134] Figure 11 An example computer system 1100 is shown in which various embodiments may be implemented. System 1100 may be used to implement any of the computer systems described above. As shown, computer system 1100 includes a processing unit 1104 that communicates with multiple peripheral subsystems via a bus subsystem 1102. These peripheral subsystems may include a processing acceleration unit 1106, an I / O subsystem 1108, a storage subsystem 1118, and a communication subsystem 1124. Storage subsystem 1118 includes tangible computer-readable storage media 1122 and system memory 1110.

[0135] The bus subsystem 1102 provides a mechanism for allowing the various components and subsystems of the computer system 1100 to communicate with each other by intention. Although the bus subsystem 1102 is schematically shown as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. The bus subsystem 1102 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of various bus architectures. For example, this architecture can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a Mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0136] The processing unit 1104, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1100. One or more processors may be included in the processing unit 1104. These processors may include single-core or multi-core processors. In certain embodiments, the processing unit 1104 may be implemented as one or more independent processing units 1132 and / or 1134, each of which includes a single-core or multi-core processor. In other embodiments, the processing unit 1104 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0137] In various embodiments, the processing unit 1104 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in (one or more) processors 1104 and / or in the storage subsystem 1118. Through appropriate programming, (one or more) processors 1104 can provide the various functions described above. The computer system 1100 can additionally include a processing acceleration unit 1106, which can include a digital signal processor (DSP), a special-purpose processor, or the like.

[0138] I / O subsystem 1108 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices such as Microsoft Motion sensors that enable users to control devices such as The user interface input devices may also include eye gesture recognition devices, such as detecting eye activity from the user (e.g., a "wink" when taking a picture and / or making a menu selection) and translating eye gestures to input devices (e.g., Google ) In addition, the user interface input device may include an input device that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0139] The user interface input device may also include, but is not limited to, a three-dimensional (3D) mouse, a joystick or pointing stick, a game board and a drawing board, and audio / video equipment, such as a loudspeaker, a digital camera, a digital camcorder, a portable media player, a webcam, an image scanner, a fingerprint scanner, a barcode reader 3D scanner, a 3D printer, a laser rangefinder, and a sight tracking device. In addition, the user interface input device may include, for example, a medical imaging input device, such as computer tomography, magnetic resonance imaging, positron emission tomography, a medical ultrasound device. The user interface input device may also include, for example, an audio input device, such as a MIDI keyboard, a digital musical instrument, etc.

[0140] User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1100 to a user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0141] Computer system 1100 may include a storage subsystem 1118 containing software elements, shown presently located within system memory 1110. System memory 1110 may store program instructions executable and loadable on processing unit 1104, as well as data generated during execution of these programs.

[0142] Depending on the configuration and type of computer system 1100, system memory 1110 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated and executed by processing unit 1104. In some embodiments, system memory 1110 may include a variety of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some embodiments, a basic input / output system (BIOS), which contains basic routines that help transfer information between elements within computer system 1100, such as during startup, may typically be stored in ROM. By way of example and not limitation, system memory 1110 also illustrates application programs 1112, which may include client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc., program data 1114, and an operating system 1116. By way of example, operating system 1116 may include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or such as iOS, Phone, OS, 11OS and OS operating system mobile operating system.

[0143] The storage subsystem 1118 may also provide a tangible computer-readable storage medium for storing the basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provide the above-described functionality when executed by the processor may be stored in the storage subsystem 1118. These software modules or instructions may be executed by the processing unit 1104. The storage subsystem 1118 may also provide a repository for storing data used in accordance with the present disclosure.

[0144] The storage subsystem 1100 may also include a computer-readable storage media reader 1120 that may be further connected to computer-readable storage media 1122. Together with the system memory 1110, and optionally in conjunction therewith, the computer-readable storage media 1122 may comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing, storing, transferring, and retrieving computer-readable information.

[0145] The computer-readable storage medium 1122 containing the code or portions of the code may also include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other tangible computer-readable medium. This may also include non-tangible computer-readable media such as data signals, data transmissions, or any other medium that can be used to transmit the desired information and can be accessed by the computing system 1100.

[0146] By way of example, computer-readable storage media 1122 may include a hard drive that reads from or writes to non-removable nonvolatile magnetic media, a magnetic disk drive that reads from or writes to removable nonvolatile magnetic disks, and a magnetic disk drive that reads from or writes to removable nonvolatile optical disks such as CD ROMs, DVDs, and optical drives. Computer readable storage media 1122 may include, but is not limited to: The computer system 1100 may include a disk drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1122 may also include a solid-state drive (SSD) based on non-volatile memory (such as an SSD based on flash memory, an enterprise flash drive, a solid-state ROM, etc.), an SSD based on volatile memory (such as an SSD based on solid-state RAM, dynamic RAM, static RAM, DRAM), a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of a DRAM-based SSD and a flash memory-based SSD. The disk drive and its associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1100.

[0147] The communication subsystem 1124 provides an interface to other computer systems and networks. The communication subsystem 1124 is used as an interface for receiving data from other systems and transmitting data from the computer system 1100 to other systems. For example, the communication subsystem 1124 can enable the computer system 1100 to be connected to one or more devices via the Internet. In some embodiments, the communication subsystem 1124 may include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technology such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 902.11 series standards), or other mobile communication technology, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communication subsystem 1124 can provide a wired network connection (e.g., Ethernet) as an addition or alternative to a wireless interface.

[0148] In some embodiments, the communication subsystem 1124 may also receive incoming communications in the form of structured and / or unstructured data feeds 1126 , event streams 1128 , event updates 1130 , and the like on behalf of one or more users who may use the computer system 1100 .

[0149] For example, the communication subsystem 1124 may be configured to receive data feeds 1126 in real time from users of social networking and / or other communication services, such as feed, Updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0150] Additionally, the communication subsystem 1124 may also be configured to receive data in the form of continuous data streams, which may include event streams 1128 and / or event updates 1130, which may be continuous or unbounded in nature, real-time events without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0151] The communication subsystem 1124 can also be configured to output structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1100.

[0152] Computer system 1100 can be of various types, including a handheld portable device (e.g., Cellular phones, computing tablets, PDAs), wearable devices (e.g., Google head-mounted display), PC, workstation, mainframe, kiosk, server rack, or any other data processing system.

[0153] Due to the ever-changing nature of computers and networks, the description of the computer system 1100 depicted in the figure is intended only as a specific example. Many other configurations with more or fewer components than the system depicted in the figure are possible. For example, customized hardware may also be used and / or specific elements may be implemented with hardware, firmware, software (including applets) or a combination thereof. In addition, connections to other computing devices such as network input / output devices may also be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods of implementing the various embodiments.

[0154] While specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are encompassed within the scope of this disclosure. The embodiments are not limited to operating within a particular data processing environment, but rather can operate freely within a variety of data processing environments. Furthermore, while the embodiments have been described using a specific series of transactions and steps, it will be apparent to those skilled in the art that the scope of this disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above-described embodiments may be used individually or in combination.

[0155] In addition, although embodiments have been described using specific combinations of hardware and software, it will be appreciated that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors. Thus, where a component or module is described as being configured to perform certain operations, such configuration may be accomplished by, for example, designing an electronic circuit to perform the operation, programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0156] The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense. However, additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, while specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0157] The use of the terms "a" and "an" and "the" and similar designations in the context of describing the disclosed embodiments (especially in the context of the following claims) should be interpreted as covering both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms "include", "have", "include" and "contain" should be interpreted as open terms (i.e., meaning "including but not limited to"). The term "connected" should be interpreted as being partially or entirely included, attached to or connected together, even if there is something in the middle. Unless otherwise indicated herein, the description of the value range herein is only intended to be used as a shorthand method of individually quoting each individual value falling within the range, and each individual value is incorporated into the specification as if it were described separately in this article. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. Unless otherwise stated, the use of any and all examples or exemplary language (e.g., "such as") provided herein is only intended to better illustrate the embodiments and does not limit the scope of the present disclosure. Any language in the specification should not be interpreted as indicating that any unclaimed element is essential for the practice of the present disclosure.

[0158] Unless expressly stated otherwise, disjunctive language (such as the phrase "at least one of X, Y, or Z") is intended to be understood in context as serving generally to indicate that an item, term, etc. may be one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present.

[0159] Preferred embodiments of the present disclosure are described herein, including the best mode known for implementing the present disclosure. Variations of these preferred embodiments will become apparent to those of ordinary skill in the art after reading the foregoing description. Those of ordinary skill should be able to appropriately adopt such variations and may practice the present disclosure in a manner different from that specifically described herein. Thus, the present disclosure includes all modifications and equivalent forms of the subject matter recited in the appended claims where permitted by applicable law. In addition, unless otherwise indicated herein, the present disclosure includes any combination of the above-mentioned elements in all possible variations thereof.

[0160] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0161] In the foregoing description, various aspects of the present disclosure are described with reference to their specific embodiments, but it will be appreciated by those skilled in the art that the present disclosure is not limited thereto. Each feature and aspect disclosed above can be used individually or in combination. In addition, without departing from the broader spirit and scope of this description, the embodiments can be used in any number of environments and applications other than those described herein. Thus, this description and the accompanying drawings should be considered to be illustrative rather than restrictive.

Claims

1. A method for capturing a snapshot of key management data across a series of nodes in a cloud infrastructure, the method comprising: The snapshot orchestrator requests multiple snapshot instances from multiple nodes in the cloud infrastructure service, where: The cloud infrastructure service stores keys across cloud infrastructure components and maintains a log of keys. For each of the multiple snapshot instances: corresponds to the corresponding entry in the log record, and comprising modifying a plurality of client keys maintained by corresponding nodes of the plurality of nodes, and The plurality of client keys decrypt client data stored across the cloud infrastructure service; Obtained from multiple nodes by the snapshot orchestrator: the plurality of snapshot instances, and metadata corresponding to the plurality of snapshot instances; verifying that the plurality of snapshot instances received from the plurality of nodes include intelligible data to restore a state of the plurality of client keys; and In response to verifying the plurality of snapshot instances, the plurality of snapshot instances and metadata are stored at a storage node, thereby enabling subsequent retrieval of the plurality of snapshot instances and re-creation of modifications to the plurality of client keys at any of the plurality of nodes.

2. The method of claim 1 , wherein requesting a snapshot comprises adding an entry to a log record specifying a request to capture the plurality of snapshot instances, wherein each of the plurality of nodes captures the plurality of snapshot instances in response to identifying the addition of the entry to the log record. 3 . The method of claim 2 , wherein the plurality of snapshot instances is requested in response to expiration of a threshold duration or detection of adding a threshold number of entries to a log record.

4. The method of claim 1 , wherein each modification to any client key of the plurality of client keys identified at a first node in a first region is synchronized across other nodes in other regions of the cloud infrastructure service via a cross-region snapshot replicator.

5. The method of claim 1 , wherein verifying the plurality of snapshot instances comprises: identifying, for each of the received snapshot instances, an entropy value specific to each snapshot instance and each corresponding node; as well as A determination is made as to whether the entropy values ​​identified for each of the received snapshot instances are within a threshold similarity of each other, wherein the plurality of snapshot instances are verified in response to the entropy values ​​being within the threshold similarity.

6. The method of claim 1, wherein: Each of the modifications corresponds to an entry in a log record; as well as The method also includes truncating, by a snapshot orchestrator, the respective snapshot instances, the truncating removing modifications to the plurality of client keys and corresponding entries in the log record that precede a specified entry in the log record.

7. The method of claim 1 , further comprising: obtaining an interrupt notification at a first node among the plurality of nodes; Retrieving a first snapshot instance and corresponding metadata from the plurality of snapshot instances from a storage node; as well as providing the first snapshot instance and the corresponding metadata to the first node, The first node recreates modifications to the plurality of client keys using the first snapshot instance and the corresponding metadata.

8. The method of claim 7, wherein the metadata includes a key for accessing a specified partition of the first node, each partition of the first node independently maintaining a log record of the client, and wherein the snapshot orchestrator uses the key to access the partition to provide the stored snapshot instance and corresponding metadata to the partition of the first node.

9. A snapshot management system, comprising: processor; as well as A non-transitory computer-readable medium comprising instructions that, when executed by the processor, cause the processor to: Request multiple snapshot instances from multiple nodes in a cloud infrastructure service, where: The cloud infrastructure service stores keys across components of the cloud infrastructure, and Maintaining a log record of keys, For each of the multiple snapshot instances: corresponds to the corresponding entry in the log record, and comprising modifying a plurality of client keys maintained by corresponding nodes of the plurality of nodes, and The plurality of client keys decrypt client data stored across the cloud infrastructure service; Obtained from the multiple nodes the plurality of snapshot instances; and metadata corresponding to the plurality of snapshot instances; verifying that the plurality of snapshot instances received from the plurality of nodes include intelligible data to restore a state of the plurality of client keys; and In response to verifying the plurality of snapshot instances, the plurality of snapshot instances and metadata are stored at a storage node, wherein the snapshot instances and the corresponding metadata allow re-creation of modifications to the plurality of client keys at any of the plurality of nodes.

10. The snapshot management system according to claim 9, wherein: Each modification corresponds to a log sequence record in the append-only log record; and Requesting the plurality of snapshot instances includes adding a specified log sequence number to an append-only log record, the specified log sequence number specifying a request to capture the plurality of snapshot instances, wherein each of the plurality of nodes captures the plurality of snapshot instances in response to identifying the specified log sequence number added in the append-only log record.

11. The snapshot management system of claim 10, wherein the processor is further configured to: In response to verifying the plurality of snapshot instances, updating a table to include metadata corresponding to the plurality of snapshot instances, wherein the metadata relates to any of: a specified log sequence number to which only log records are appended, a state in which the plurality of snapshot instances are captured, and a timestamp at which a request for the plurality of snapshot instances was initiated.

12. The snapshot management system of claim 10, wherein the processor is further configured to: identifying an entropy value for each of the plurality of snapshot instances, wherein each entropy value is based on at least a designated log sequence number of an addition of an append-only log record; and A determination is made as to whether the entropy values ​​identified for each of the received snapshot instances are within a threshold similarity of each other, wherein the plurality of snapshot instances are verified in response to the entropy values ​​being within the threshold similarity.

13. The snapshot management system according to claim 10, wherein: The processor is further configured to truncate the respective snapshot instances, the truncation removing modifications to the plurality of client keys and corresponding entries in the append-only log record before a specified log sequence number added in the append-only log record.

14. The snapshot management system of claim 9, wherein the processor is further configured to: obtaining an interrupt notification at a first node among the plurality of nodes; Retrieving a first snapshot instance and corresponding metadata from the plurality of snapshot instances from a storage node; and The first snapshot instance and the corresponding metadata are provided to the first node, wherein the first node recreates the modifications to the plurality of client keys using the first snapshot instance and the corresponding metadata.

15. A non-transitory computer-readable medium comprising a sequence of instructions stored thereon, which, when executed by a processor, causes the processor to perform operations comprising: The snapshot orchestrator requests multiple snapshot instances from multiple nodes in the cloud infrastructure service, where: The cloud infrastructure service stores keys across cloud infrastructure components and maintains a log of keys. For each of the multiple snapshot instances: corresponds to the corresponding entry in the log record, and comprising modifying a plurality of client keys maintained by corresponding nodes of the plurality of nodes, and The plurality of client keys decrypt client data stored across the cloud infrastructure service; Obtain the multiple snapshot instances and corresponding metadata from the multiple nodes: the plurality of snapshot instances; as well as metadata corresponding to the plurality of snapshot instances; verifying that the plurality of snapshot instances received from the plurality of nodes include intelligible data to restore a state of the plurality of client keys; as well as In response to verifying the plurality of snapshot instances, the plurality of snapshot instances and metadata are stored at a storage node, thereby enabling subsequent retrieval of the plurality of snapshot instances and re-creation of modifications to the plurality of client keys at any of the plurality of nodes.

16. The non-transitory computer readable medium of claim 15, wherein the process further comprises: obtaining an interrupt notification at a first node among the plurality of nodes; Retrieving a first snapshot instance and corresponding metadata from the plurality of snapshot instances from a storage node; as well as providing the first snapshot instance and the corresponding metadata to the first node, The first node recreates modifications to the plurality of client keys using the first snapshot instance and the corresponding metadata.

17. The non-transitory computer readable medium of claim 15, wherein the operations further comprise: determining an entropy value of the received snapshot instance; as well as determining whether the entropy value of each of the received snapshot instances is within a threshold similarity, The entropy value represents the number of bits in each snapshot instance.

18. The non-transitory computer readable medium of claim 15, wherein: The modification corresponds to an entry in the log record; and The plurality of snapshot instances is requested in response to expiration of a threshold duration or detection of adding a threshold number of entries to a corresponding log record.

19. The non-transitory computer readable medium of claim 15, wherein: Each of the modifications corresponds to an entry in a log record; as well as The process also includes truncating the respective snapshot instances, the truncating removing modifications to the plurality of client keys and corresponding entries in the log record that precede the specified entry in the log record.

20. The non-transitory computer readable medium of claim 15, wherein Requesting a snapshot includes adding a log sequence number specifying a request to capture the plurality of snapshot instances to a log record, wherein each of the plurality of nodes captures the plurality of snapshot instances in response to identifying the added log sequence number to the log record.

Images