Techniques for key management services in an overlay network

Abstract

Techniques for implementing a key management service in a reduced footprint data center are disclosed. A cryptographic service can be executed at a computing device of a reduced footprint data center. The cryptographic service can generate a master encryption key and encrypt the master encryption key using a secure component of the computing device to produce an encrypted master encryption key. The encrypted master encryption key can be stored in a block storage volume communicatively connected to the computing device. The cryptographic service can transmit the master encryption key to a host region data center and receive an encapsulated master encryption key. The cryptographic service can store the encapsulated master encryption key in a database of the reduced footprint data center.

Description

[0001] Cross-reference to related applications

[0002] This international application claims priority and benefits from the following applications, the entire contents of which are hereby incorporated herein by reference for all purposes:

[0003] 1. U.S. Provisional Patent Application No. 63 / 564,195, filed on March 12, 2024, entitled “SCALABLE FOOTPRINT FOR DEDICATED CLOUDTECHNIQUES”;

[0004] 2. U.S. Provisional Patent Application No. 63 / 568,061, filed on March 21, 2024, entitled “NETWORKING FOR A SCALABLE DEDICATED CLOUD FOOTPRINT”;

[0005] 3. U.S. Provisional Patent Application No. 63 / 568,234, filed on March 21, 2024, entitled “SCALABLE FOOTPRINT FOR DEDICATED CLOUDTECHNIQUES”;

[0006] 4. U.S. Provisional Patent Application No. 63 / 633,966, filed on April 15, 2024, entitled “SCALABLE FOOTPRINT FOR DEDICATED CLOUDTECHNIQUES”;

[0007] 5. U.S. Provisional Patent Application No. 63 / 637,691, filed on April 23, 2024, entitled “SCALABLE FOOTPRINT FOR DEDICATED CLOUDTECHNIQUES”;

[0008] 6. U.S. Provisional Patent Application No. 63 / 660,377, filed on June 14, 2024, entitled “DRCC ARCHITECTURE UPDATE”;

[0009] 7. U.S. Provisional Patent Application No. 63 / 688,247, filed August 28, 2024, entitled “TECHNIQUES FOR A KEY MANAGEMENT SERVICE IN AN OVERLAY NETWORK”; and

[0010] 8. U.S. Patent Application No. 19 / 075,720, filed on March 10, 2025, entitled “TECHNIQUES FOR A KEY MANAGEMENT SERVICE IN AN OVERLAY NETWORK”. Technical Field

[0011] This disclosure generally relates to data centers. More specifically, this disclosure relates to data centers in which key management services are implemented in the overlay network of the data center. Background Technology

[0012] Cloud service providers (CSPs) can use resources across several data centers to provision computing infrastructure to customers. As cloud computing demand increases, CSPs can improve the availability of cloud resources by scaling data centers. However, scaling can result in a large data center footprint and a large number of computing devices, requiring a corresponding amount of resources to operate and reserving significant computing resources for the effective management of the cloud resources themselves. Summary of the Invention

[0013] Embodiments of this disclosure relate to cloud computing networks. More specifically, this disclosure describes an architecture, infrastructure, and related technologies for implementing key management services in a space-saving data center. A typical CSP can provide cloud services to multiple customers. Each customer may have the ability to customize and configure the supplied infrastructure to support the cloud resources allocated to them. To manage the infrastructure provisioning for multiple customers, the CSP may reserve computing resources within the data center to provide certain “core” services to both customers and other services operated by the CSP. For example, services such as block storage, object storage, identity and access management, and key management and secret services are implemented within a “service enclave” of the data center. The service enclave may be connected via a substrate network of computing devices (virtual machines and / or bare metal instances) hosted within the data center. The substrate network may be part of the data center’s “underlay network,” which includes a physical network connecting the bare metal devices, smart network interface cards (SmartNICs) for the computing devices, and networking infrastructure such as top-of-rack switches. In contrast, CSP customers are supplied with infrastructure in an “overlay network”, which includes one or more VCNs of virtualized environments to provide resources (e.g., compute, storage, etc.) to customers.

[0014] Service enclaves exist on dedicated hardware within a data center. Therefore, services hosted within a service enclave are difficult to scale. While additional racks and servers can be implemented within the data center to expand the resources available to CSP customers, the dedicated compute resources used for service enclaves typically have a fixed size, dependent on the maximum projected size of the data center. Expanding a service enclave can require complex compute resource additions, which can impact the availability of core services delivered to customers. Furthermore, unused resources within a service enclave (e.g., if the service enclave is resized to be too large for customer demand from the data center) cannot be easily made available to customers because service enclaves typically do not allow network access from the customer's overlay network.

[0015] Even with increasing demand for cloud services, CSPs may want to deploy data centers with the smallest possible initial physical footprint to meet this need. This smaller footprint improves the ease of deploying physical components and configuring the initial infrastructure, while still allowing the data center to scale to meet customer demands. Instead of dedicating a portion of computing hardware to the service enclave within the reduced footprint, the "core services" hosted in the service enclave can be implemented in the overlay network. By doing so, the core services can be scaled as the data center footprint expands. The computing equipment used to construct the reduced footprint data center can be homogenized, improving the initial configuration and making it easy to scale the footprint when adding additional homogenized equipment. Furthermore, by eliminating the base network, flexible overlay network shapes become available for both the CSP's core services and its customers.

[0016] One aspect of reducing data center footprint is eliminating dedicated hardware devices for specific operations, including management keys (e.g., encryption keys, signing keys, etc.) and other secrets. In a typical data center, a Key Management Service (KMS) communicates with a Dedicated Hardware Security Module (HSM) to perform encryption and decryption operations, execute digital signatures, and store authenticated hardware keys. The HSM can be configured with a root key for encryption and signing operations during a key signing ceremony involving the physical transport of the HSM key (e.g., via a memory stick) to a secure facility that holds the CSP trust root key on an isolated computer. Once the HSM root key is signed during the key signing ceremony, the HSM can be implemented in the data center to provide key services bound to the CSP trust root. In a space-saving data center, the HSM may not exist. Instead, bare-metal computing devices can each have a Trusted Platform Module (TPM) for storing a limited number of authenticated keys. In order to establish a trust chain with the security root key maintained by the CSP, a reduced-footprint data center can be configured to connect to an existing data center and use the HSM of that data center to establish a trust chain for the KMS implemented in the reduced-footprint data center.

[0017] The embodiments described herein relate to methods, systems, and computer-readable media for implementing KMS in an overlay network of a space-reduced data center. One method may include performing a cryptographic service at a computing device in the space-reduced data center. The method may further include: generating a master encryption key by the cryptographic service, and encrypting the master encryption key to produce an encrypted master encryption key. The cryptographic service may use a security component of the computing device to encrypt the master encryption key. The method may further include storing the encrypted master encryption key in a block storage volume communicatively connected to the computing device. The method may further include: transmitting the master encryption key to a host area data center by the cryptographic service, and receiving a wrapped master encryption key in response. The method may further include storing the wrapped master encryption key in a database of the space-reduced data center.

[0018] Another embodiment relates to a computing system including one or more processors and one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the computing system to perform the methods described above.

[0019] Another embodiment relates to a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the computing system to perform the methods described above.

[0020] Additional aspects included in this disclosure are provided in the following examples: Example 1: A method for performing cryptographic operations in a space-reduced data center using a key management service. The method may include: receiving a request for a cryptographic operation from a key management service; in response to the request, obtaining an encrypted master encryption key from a block storage volume communicatively connected to the computing device by the key management service, which is executed at a computing device in the space-reduced data center; decrypting the encrypted master encryption key using a security component of the computing device to generate a master encryption key; obtaining an encrypted intermediate encryption key from a database; decrypting the encrypted intermediate encryption key using the master encryption key to generate an intermediate encryption key; obtaining an encrypted software key from the database, the encrypted software key being identified using the request; decrypting the encrypted software key using the intermediate encryption key to generate a software key; and performing the cryptographic operation using the software key.

[0021] Example 1.1: The method of Example 1, wherein the request corresponds to a tenant in a data center that reduces its footprint.

[0022] Example 1.2: The method of Example 1 further includes obtaining software key metadata from an additional database and at least in part based on the request, the software key metadata being used by a key management service to identify the software key in the database.

[0023] Example 1.3: The method of Example 1, where the cryptographic operation includes the encryption of tenant data.

[0024] Example 1.4: The method of Example 1, where the cryptographic operations include signing the tenant data.

[0025] Example 1.5: The method of Example 1 further includes transmitting the output of a cryptographic operation to a tenant application executing in a space-reduced data center, the request originating from the tenant application.

[0026] Example 1.6: The method of Example 1, wherein the security component includes a trusted platform module of the computing device.

[0027] Example 2: A method for migrating a key management service executed in a space-reduced data center. The method may include connecting a bare-metal host device to the space-reduced data center. The bare-metal host device may include a Hardware Security Module (HSM) configured using a key signing ceremony. The method may further include: executing an HSM agent process at the bare-metal host device; transferring a master encryption key from the key management service to the HSM agent process; and having the HSM agent process encapsulate the master encryption key to generate an encapsulated encryption key.

[0028] Example 2.1: The method of Example 2, wherein encapsulating the master encryption key includes encrypting the master encryption key using the root encryption key maintained by the HSM to produce an encrypted master encryption key.

[0029] Example 2.2: The method of Example 2.1, wherein encapsulating the master encryption key also includes signing the encrypted master encryption key using a root signing key maintained by the HSM.

[0030] Example 2.3: The method of Example 2, where the HSM is configured via a key signing ceremony to store the root encryption key and the root signing key.

[0031] Example 2.4: The method of Example 2 further includes: obtaining the encrypted master encryption key from a block storage volume of a computing device communicatively connected to a data center with reduced footprint by a key management service before transmitting the master encryption key; and decrypting the encrypted master encryption key using a security component of the computing device to generate a master encryption key.

[0032] Example 2.5: The method of Example 2 also includes storing the encapsulated encryption key at the HSM.

[0033] Example 2.6: The method of Example 2 further includes storing one or more software encryption keys at the HSM by the key management service, which are obtained from the database by the key management service.

[0034] Example 3: A computing system includes one or more processors and one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the computing system to perform any of the methods described in Examples 1-2.6 above.

[0035] Example 4: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform any of the methods described in Examples 1-1.6 above. Furthermore, embodiments may be implemented using a computer program product comprising a computer program / instructions that, when executed by a processor, cause the processor to perform the methods of Examples 1 to 2.6. Attached Figure Description

[0036] Figure 1 This is a block diagram illustrating an example system architecture for a space-reduced data center, including an initialization device, according to some embodiments.

[0037] Figure 2A This is a block diagram illustrating a typical data center that includes multiple server racks reserved for specific functions.

[0038] Figure 2B This is a block diagram illustrating a space-reducing data center according to some embodiments, in which services are located in an overlay network.

[0039] Figure 3 This is a block diagram illustrating a space-saving expansion of a data center according to some embodiments.

[0040] Figure 4 This is a block diagram illustrating the networking connection between the overlay network and the underlying network in a data center with reduced footprint according to some embodiments.

[0041] Figure 5 This is a block diagram illustrating an example data center architecture with reduced footprint, based on some embodiments of a managed key management service.

[0042] Figure 6 This is a block diagram illustrating an example architecture and simplified process for a key signing ceremony in a data center key management service for reducing footprint, according to some embodiments.

[0043] Figure 7 This is a block diagram illustrating an example architecture and simplified process for host-to-host key transfer in a data center with reduced footprint, according to some embodiments.

[0044] Figure 8 This is a block diagram illustrating a simplified process for performing cryptographic operations using a key management service, according to some embodiments.

[0045] Figure 9 This is a block diagram illustrating a simplified process for migrating key management services from trusted platform module support to hardware security module support, according to some embodiments.

[0046] Figure 10 This is a flowchart of an example process for implementing a key management service in a data center with a reduced footprint, according to some embodiments.

[0047] Figure 11 This is a flowchart of an example process for performing cryptographic operations using a key management service in a data center with a reduced footprint, according to some embodiments.

[0048] Figure 12 This is a flowchart of an example process for migrating a key management service in a data center with a reduced footprint, according to some embodiments.

[0049] Figure 13 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0050] Figure 14 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0051] Figure 15 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0052] Figure 16 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0053] Figure 17 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0054] The adoption of cloud services has increased rapidly in recent years. Various types of cloud services are now offered by a wide variety of cloud service providers (CSPs). The term cloud service is generally used to refer to services or functions that a CSP provides to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that constitute the CSP's infrastructure and are used to provide cloud services to customers are separate from the customer's own on-premise servers and systems. Therefore, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for these services. Cloud services are designed to provide subscribers with easy, scalable, and on-demand access to applications and computing resources without requiring customers to invest in the infrastructure used to provide these services or functions. Various types or models of cloud services can be provided, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc. Customers can subscribe to one or more cloud services provided by a CSP. Customers can be any entity, such as individuals, organizations, enterprises, etc.

[0055] As described above, the CSP is responsible for providing the infrastructure and resources used to deliver cloud services to subscribers. The resources provided by the CSP can include both hardware and software resources. These resources may include, for example, computing resources (e.g., virtual machines, containers, applications, processors), storage resources (e.g., databases, data repositories), networking resources (e.g., routers, host machines, load balancers), identity, and other resources. In some implementations, the resources provided by the CSP for providing a set of cloud services are organized into a data center. A data center can be configured to provide a specific set of cloud services. The CSP is responsible for equipping the data center with the infrastructure and resources used to provide that specific set of cloud services. The CSP may build one or more data centers.

[0056] The following definitions are useful for parts of a data center built by a CSP: The underlying network is the physical network located beneath the overlay network and its Virtual Cloud Network (VCN). The existing base network is part of the underlying network. ILOM ports, management, and SmartNIC base addresses are also part of the underlying network.

[0057] Overlay network – A network environment available for services and applications (including virtualized environments) that provide data center functionality to both customers and CSPs. An overlay network may include one or more VCNs, virtualized environments, and networking connections from these VCNs in a space-saving data center to other cloud computing services offered by the CSP (e.g., services provided in other data center environments). The following is about… Figures 9-13 Provides specific details about network virtualization and VCN as part of Infrastructure as a Service.

[0058] The base network is a portion of the underlying network that contains host devices (e.g., bare-metal computing devices and / or VMs) that run only base services. In existing environments, these host devices may not have SmartNICs. The host devices can be managed by a service team responsible for one or more base services within the base services.

[0059] Baseline services – A list of services currently running in the baseline network. While most of these services (block storage, object storage, identity services, etc.) operate within service enclaves, some baseline services are active outside of them. Currently, baseline services are a mix of services that must communicate with the underlying network (e.g., network monitoring) and services that reside in service enclaves due to historical reasons (e.g., object storage). With the elimination of dedicated baseline hosts, we expect baseline services to converge to those that only need to communicate with the underlying network.

[0060] SmartNIC – a computing component that combines a network interface card with additional functionality for network virtualization to create a network abstraction layer that can run on top of physical networking components, such as the underlying network. A SmartNIC may include a processor and memory capable of performing computational operations to provide additional functionality.

[0061] One or more BIOS devices – computing devices or multiple computing devices on server racks in a space-saving data center. One or more BIOS devices can be designed to enable independent and resilient operation during various boot scenarios and network outages. One or more BIOS devices can be configured to facilitate initial boot processing in a space-saving data center, provide essential services during recovery, and ensure the stability of the area, especially in power-constrained environments. The BIOS device hosts a range of functions that allow the area to operate autonomously. These functions may include, for example, DNS resolution, NTP synchronization, DHCP / ZTP configuration, and various security and provisioning services. By providing these capabilities, the BIOS device ensures that the rack can bootstrap itself without relying on external resources, recover from power or network-related events, and maintain essential connectivity and management functions. For example, each server rack may have one BIOS device, or there may be two or three BIOS devices. In various embodiments, the BIOS device may have similar hardware specifications (e.g., number of processors, amount of memory, number of attached storage devices) to other server devices on the rack.

[0062] Reduced footprint data centers can have new architectures for regions where the initial network footprint is as small as possible (e.g., six racks, four racks, or even just one rack for server equipment) while still providing core cloud services and scalability to meet customer needs. Specifically, reduced footprint data centers may not isolate resources for service enclaves (SEs) from resources for customer enclaves (CEs). Instead, the Butterfly region places SE services (e.g., block storage, object storage, identity, key management, secrets) that primarily operate in the base network into the overlay network. This means the Butterfly region will not have dedicated hosts for the base network but will require specific solutions to connect to the base services now in the overlay. Additionally, a small portion of the basic bootstrap service is needed to ensure initial routing configuration for services in the overlay during startup and / or recovery.

[0063] Figures 1-4 It provides an overview of the concepts embodied in data centers that reduce footprint.

[0064] Figure 1 This is a block diagram illustrating an example system architecture for a space-reduced data center 100, including initialization device 102. (See diagram for example.) Figure 1 As shown, the space-reduced data center 100 may include six racks of server equipment. These racks may be referred to as "butterfly" racks. The space-reduced data center 100 may include butterfly racks 110, 120, 130, 140, 150, and 160. In some embodiments, these racks may be identical. For example, butterfly rack 110 may include the same number of computing and / or networking devices as each of the other butterfly racks 120-160.

[0065] The Butterfly rack 110 may include two top-of-rack (TOR) switches 106 and 108. Each of the TOR switches 106 and 108 may include one or more networking switches configured to provide network communication between server equipment and other computing devices within the Butterfly rack 110, and to provide one or more networking connections to other Butterfly racks 120-160 and other networks (including customer network 114).

[0066] The Butterfly rack 110 may also include one or more BIOS devices 102. The BIOS device may be a server device configured to execute one or more processes to provide a set of "core services" within the reduced footprint data center 100 during boot / startup processing. The one or more BIOS devices 102 may configure one or more components of the reduced footprint data center 100 during boot. For example, the one or more BIOS devices 102 may send network configuration information to networking devices within the Butterfly racks 110-160. These networking devices may be SmartNICs attached to server devices within the Butterfly racks 110-160. As another example, the one or more BIOS devices 102 may send network configuration information to a base access VCN. The base access VCN may be deployed to one or more hosts within the reduced footprint data center 100. For example, a VM running in the Butterfly racks 110-160 may be configured as a base access VCN. The base access VCN can be configured to provide networking routing between one or more other VCNs (e.g., customer VCNs) and networking devices (e.g., SmartNICs), as well as other networking components that now perform base services in their own VCNs within the overlay layer.

[0067] In some embodiments, one or more BIOS devices 102 may also be configured to host one or more services, such as a Key Exchange Service (KeS), a Device Encryption Key (DEK) service, or other core services. Furthermore, one or more BIOS devices 102 may include boot volumes for booting VMs on host devices in a space-reduced data center 100. For example, one or more BIOS devices 102 may provide boot volumes for VMs on hypervisors hosted on one or more server devices 104.

[0068] The butterfly rack 110 may include one or more additional server devices 104. Each of the server devices 104 may include one or more processors and one or more memories, which together can store and execute instructions for implementing computing services as described herein, including, for example, computing, storage, VMs, CSP services, customer service, and / or applications. Figure 1 As shown, each of the butterfly racks 110-160 can include the same number of server devices(s) and TORs. In some embodiments, while each of the butterfly racks 110-160 can include a BIOS device, the techniques described herein can be implemented using only a single BIOS device within a space-saving data center 100. Each of the server devices(s) 104 can include a Trusted Platform Module (TPM). The TPM on each device can be a microcontroller or other processor(s) and a storage area for performing cryptographic operations such as hashing, encryption / decryption, key and key pair generation, and key storage. TPMs can typically conform to standards characterizing such devices, such as ISO / IEC 11889.

[0069] The reduced footprint data center 100 may also include a networking rack 112. The networking rack 112 may include one or more networking devices, including switches, gateways, routers, etc., for communicatively coupling the butterfly racks 110-160 to each other and to a customer network 114. The customer network 114 may include an on-premises network connected to the reduced footprint data center 100. In some embodiments, the customer network 114 may provide network connectivity to a public network, including the Internet. As described below with respect to Figure 2, the networking rack 112 may not be part of the initial “small” reduced footprint data center and may be added to support scaling of the reduced footprint data center 100.

[0070] Figure 2AThis is a block diagram illustrating a conventional data center 200 that includes multiple server racks reserved for specific functions. In the conventional data center 200, each of the multiple server racks may include multiple server devices, as well as networking equipment (e.g., TOR) and power and distribution equipment. Figure 2A The typical data center 200 shown can have a standard footprint of 13 server racks as shown, but larger data centers may have additional server racks.

[0071] To provide network isolation between CSP data and customer data for CSP services running in a conventional data center 200, a portion of the server racks can be reserved as a service enclave. This allows computing devices on these server racks to be hosted and provide CSP services within the conventional data center 200, without simultaneously hosting customer data. For example... Figure 2A As shown, server racks 1-4 can be included as service enclave racks 202.

[0072] Similarly, a portion of the server racks can be provided as customer enclaves, allowing the computing devices on these server racks to host customer services, applications, and associated customer data. Racks 5 through 7 can be part of customer enclave racks 204 within a conventional data center 200.

[0073] Isolation between service enclaves and customer enclaves can be enforced through software-defined perimeters, which define the edge devices and / or software within the enclave to distinguish them from hardware / software elements outside the enclave. Access to and from each enclave can be controlled, monitored, and / or policy-driven. For example, access to a service enclave can be authorized and restricted to authorized CSP clients. Such access can be based on one or more credentials provided to the enclave.

[0074] The conventional data center 200 may also include database racks 206 (racks 8-9) and networking racks 208 (racks 10-13). Database racks 206 may include computing and storage devices within the conventional data center 200 that provide storage and management for databases, data repositories, object storage, and similar data persistence technologies. Networking racks 208 may include networking devices that provide connectivity to computing devices within the conventional data center 200 and to other networks (e.g., customer networks, the Internet, etc.).

[0075] Figure 2B This is a block diagram illustrating a space-reduced data center 210 according to some embodiments, in which services reside within an overlay network. The space-reduced data center 210 may be... Figure 1An example of a space-reducing data center 100 includes six butterfly racks, each with multiple server devices, networking devices, and power distribution equipment.

[0076] Unlike a conventional data center 200 (where specific server racks are reserved as service enclave racks 202 and customer enclave racks 204), a reduced-footprint data center 210 can have an overlay network 212 spanning computing devices across all server racks. For example, server devices on Butterfly Rack 1 and Butterfly Rack 6 can host VMs for VCNs within the overlay network 212. The overlay network 212 can then include both core service 214 and customer service 216. Core service 214 can include one or more VCNs for CSP services that would otherwise be hosted within a service enclave of the conventional data center 200 (e.g., on service enclave rack 202). In the reduced-footprint data center 210, core service 214 can reside on overlay network 212 across any one or more server devices within a Butterfly Rack. Similarly, customer service 216 can reside on overlay network 212 across host devices on any Butterfly Rack. In some embodiments, core service 214 can be hosted on a specific device within the reduced-footprint data center. For example, core service 214 can be hosted on butterfly racks 1-3, while customer service 216 can be hosted on butterfly racks 4-6. In other embodiments, core service 214 and customer service 216 can be hosted on any butterfly rack, such as... Figure 2B The image is drawn in the middle.

[0077] Figure 3 This is an extended block diagram illustrating a space-reduced data center 300 according to some embodiments. The space-reduced data center 300 may include a plurality of space-reduced server racks 302. Each of the plurality of space-reduced server racks 302 may be as described above. Figure 1 An example of one of the butterfly racks described, 110-160.

[0078] Multiple footprint-reduced server racks 302 can be connected in a ring network 304 using directional network connections between each TOR switch on each server rack. For example, a first TOR switch at each rack can be connected to the first TOR switches of two adjacent server racks, allowing data communication from the server racks to flow in one direction. A second TOR switch at each rack can be connected to the second TOR switches of two adjacent server racks, providing data communication between racks in opposite directions. The first and second TOR switches at each rack can be interconnected and connected to each server device on the rack, providing multiple redundant network paths from any server device in any server rack to another server device in another server rack. Therefore, the ring network 304 can allow low-latency and highly available network connectivity between resources hosted on any computing device (e.g., server device) in the multiple footprint-reduced server racks 302.

[0079] To scale the reduced footprint data center 300 from the initial footprint provided by the reduced footprint server rack 302, additional server racks can be connected to the reduced footprint server rack 302. A networking rack 306 can be implemented at the reduced footprint data center 300. The networking rack 306 can be as described above regarding... Figure 1 An example of the described networking rack 112. Networking rack 306 may include multiple networking ports that can be used to connect to one or more of a plurality of footprint-reduced server racks. For example, networking rack 306 may be connected to a first footprint-reduced server rack using connection 308. Networking rack 306 may be, for example, a dual-chassis system with four LCs, each LC having 34 x 400G ports for a total of 384 x 100G links in each chassis.

[0080] Once the networked rack 306 is implemented and connected, additional server racks 310 can be installed in the reduced footprint data center 300. The additional server rack 310 may differ from the reduced footprint server racks in the multiple reduced footprint server racks 302. For example, the additional server rack 310 may include a different number of server devices, each including a different number of computing and / or storage resources (e.g., processors, processing cores, dynamic memory, non-volatile memory, etc.). Once the additional server rack 310 is connected to the networked rack 306, cloud services hosted on the multiple reduced footprint server racks 302 can be scaled to utilize the computing resources of the additional server rack 310. As an example, a cloud service (e.g., computing) hosted on the multiple reduced footprint server racks 302 may have a portion of its data plane provisioned to one of the server devices in the additional server rack 310, thereby allowing the computing service to instantiate VMs on the additional server rack 310.

[0081] Figure 4 This is a block diagram illustrating an example network architecture for networking connections between one or more VCNs (baseline service VCNs) in an overlay network 402 and an underlying network 422 in a space-reduced data center 400 according to some embodiments. The space-reduced data center 400 may be an example of other space-reduced data centers described herein, including... Figure 1 A data center with a reduced footprint of 100.

[0082] In the space-reduced data center 400, CSP services previously implemented in the SE (e.g., hosted on service enclave rack 202 of Figure 2) can now be executed in one or more base service VCNs 404-408. For example, base service VCN-1 404 could be a VCN for the compute service control plane, base service VCN-2 406 could be a VCN for PKI services, and base service VCN-N 408 could be a VCN for block storage services. The SE service control and data planes can be separated into different VCNs. Base service VCNs 404-408 can reside in overlay network 402. Overlay network 402 may also include one or more client VCNs 416, whose connectivity to the underlying network 422 may be restricted.

[0083] Each base service VCN can have its own routing table, which defines network traffic routing rules for forwarding network traffic within the space-saving data center 400 network. For example... Figure 4As shown, base service VCN-1 may have VCN-1 routing table 410, base service VCN-2 may have VCN-2 routing table 412, and base service VCN-N 408 may have VCN-N routing table 414. The routing information for each of the base services VCNs 404-408 can be initially configured when the space-saving data center 400 is first built, so that network traffic to / from the core SE service can be routed between the overlay network 402 and the underlying network 422.

[0084] The underlying network 422 may include various devices and other networking endpoints connected via physical networking components of the space-reduced data center 400. For example... Figure 4 As shown, the underlying network 422 may include, but is not limited to, one or more ILOMs 424, bastion hosts 426, one or more NTP servers 428, BIOS services 430, and one or more VNICs 432. One or more ILOMs 424 may be computing devices and network targets that provide access to server equipment in the reduced footprint data center 400 for both in-band and out-of-band management. For example, one or more ILOMs 424 may allow remote management of associated server equipment within server racks in the reduced footprint data center 400, separate from the networking pathway defined for the region. The bastion host 426 may be a service that performs on server equipment in the reduced footprint data center 400, provides network access via the underlying network 422, and has no public network address. The bastion host 426 may provide remote access to computing resources within the reduced footprint data center 400 in conjunction with the bastion service operating on the underlying network 422. The Bastion service may be an SE service on the overlay network 402 that has not been moved to the reduced footprint data center 400. Similarly, a Network Time Protocol (NTP) server 428 may operate in the underlying network 422 to provide accurate timing for devices and services within the reduced footprint data center 400. The BIOS service 430 may include services on one or more initialization devices hosted on server racks within the reduced footprint data center 400. For example, the BIOS service 430 may include a key encryption service that can be used to encrypt / decrypt data on server devices in the reduced footprint data center 400 during initial boot processing. As another example, the BIOS service 430 may include a network configuration service that can provide initial network configuration for devices within the reduced footprint data center 400. One or more VNICs 432 may include network interfaces defined by SmartNICs connected to server devices within the reduced footprint data center 400.

[0085] As the SE service is moved to overlay network 402, it may still require network connectivity to the underlying network 422 to function properly. To provide this connectivity, a base access VCN 418 can be implemented within a space-saving data center 400. The base access VCN 418 may include a dynamic routing gateway (DRG) that allows communication between base service VCNs 404-408 and the underlying network 422. The base access VCN 418 may then have a DRG routing table 420, which can define a single routing rule for reaching the underlying network 422 from the base service VCNs 404-408.

[0086] To avoid circular dependencies during the initial construction of the space-reducing Data Center 400 or recovery from an outage, the equipment is initialized (e.g., Figure 1 The BIOS device 102 can be used to configure network addresses and routes for the base access VCN 418, dynamic routing gateways within the base access VCN 418, and / or one or more SmartNICs for the underlying network 422. When booting server equipment for each reduced-footprint data center 400 server rack, the base access VCN 418 can be deployed to communicatively connect one or more base service VCNs 404-408 to the underlying network 422. The initialization device can send network configuration information to the base access VCN 418 to configure the DRG routing table 420, thereby providing an initial network address (e.g., an IP address) for each endpoint of the base service VCNs 404-408 in the overlay network 402 until DHCP services and other networking services are available in their respective base service VCNs.

[0087] In addition, the initialization device can send networking configuration information to define one or more static routes for the dynamic routing gateway as part of the DRG routing table 420. Static routes can characterize the underlying network 422 (including the SmartNICs of each server device connected to the space-saving data center, e.g., ...). Figure 1 Network connection between (one or more) server devices 104) and each base service VCN 404-408.

[0088] Finally, the initialization device can send network configuration information to each SmartNIC to provide each SmartNIC with a network address (e.g., the network address of the endpoint of the SmartNIC in the underlying network 422). The initialization device can complete the configuration of each of these components in the reduced footprint data center 400 in response to an indication that the corresponding component has been promoted to an active state (e.g., the SmartNIC is powered on and accessible via the underlying network 422, the base access VCN 418 is deployed to one or more hosts within the reduced footprint data center 400, etc.).

[0089] Key management services in overlay networks

[0090] One of the core services could be a Key Management Service (KMS), configured to provide digitally signed keys to applications within a space-saving data center. In a typical data center, KMS can communicate with a Hardware Security Module (HSM), which can be configured with a signed root key that establishes a trust chain with the CSP, using manual processing performed by the CSP. When operating in an overlay, KMS will run only on VMs. Unlike bare-metal hosts, the VMs used for KMS in an overlay may not support HSMs because space-saving data centers may have standardized physical resources (e.g., server equipment) without dedicated hardware to support the "core" services.

[0091] Instead of HSMs, KMS implemented in space-saving data centers can use Trusted Platform Modules (TPMs) as the physical security boundary. Compared to HSMs, TPMs may have limited storage space for keys. For example, a typical TPM on a computing device might be able to store approximately 10 keys for various cryptographic operations, while an HSM could store over 100,000 keys. Therefore, an HSM can provide customers with hardware-protected keys that are specific to each customer and securely stored on the HSM. Due to the limited space for keys, KMS in space-saving data centers can provide customers with software keys that are encrypted and stored in a database, where this encryption is ultimately protected by keys maintained by the TPM.

[0092] In a typical data center, the HSM (Host Smart Provider) is also configured using a Key Signing Ceremony (KSC), where the HSM's public key is physically (e.g., on a portable flash drive) taken to the CSP's security facility for signing to generate a root certificate authority (CA) certificate for the HSM. The resulting signed root CA certificate is then physically returned and loaded onto the HSM. To establish this same root trust in a butterfly zone using a TPM (Trusted Provider Terminal), an automated KSC can be used, where an existing regional data center of the CSP with an HSM (with a root trust established via a traditional KSC) can be used as a secure repository of the root private key for the reduced-footprint data center. Partitions can be created on the HSM for the corresponding reduced-footprint data center, and the root key from the HSM can be used to sign the master encryption key (MEK) generated by the KMS (Knowledge Management System) in the reduced-footprint data center. Since the TPM has limited key storage space, the TPM key can be used to encrypt the signed MEK from the HSM in the host zone. This encrypted, signed MEK can then be stored in a block volume accessible to the KMS VM host.

[0093] The communication channel between the space-saving data center and the host zone can be implemented by using an application principal in the host zone (an instance authenticated and authorized by the Identity and Access Management (IAM) service of the host zone) for the AutoKSC application in the space-saving data center. The AutoKSC application can communicate with the KMS in the host zone to support the encapsulation / decapsulation (e.g., signing and / or encryption) of the MEK at the HSM using the HSM agent.

[0094] To support the use of TPM, a new cryptographic module service can be implemented in a KMS within a reduced-footprint data center, replacing the functionality of the HSM agent in a typical CSP data center. A software intermediate encryption key (IEK) can be used to encrypt all client keys by the KMS in the reduced-footprint data center. The IEK can be stored encrypted via MEK. To support cryptographic operations for clients, the reduced-footprint data center KMS can use the TPM to decrypt the stored MEK and load it into memory, then use the MEK to decrypt the stored IEK and load it into memory, and finally use the IEK to decrypt the client key to perform the requested operation for the client.

[0095] If the reduced-footprint data center scales to a full-size data center, the TPM-based KMS described above can be migrated to an HSM-based KMS. A bare-metal host with an HSM storing keys signed using a traditional KSC can be added to the reduced-footprint data center. The HSM root key can be used to encrypt / decrypt the MEK of the KMS operating in the reduced-footprint data center, thus transitioning from the TPM key. If the customer wants the hardware key for the reduced-footprint data center, then the customer's software key can then be imported into the HSM. Otherwise, the KMS can operate using the HSM root key to decrypt the MEK as needed.

[0096] Removing services from the base offers numerous advantages. Service teams can eliminate redundant work (e.g., networking configuration for underlying and overlay connections), CSPs can completely eliminate some services, compute and storage capacity becomes interchangeable across all services in the overlay, and service connectivity is greatly simplified. Service teams can operate without needing to know the configuration of their services. If a host is provisioned to communicate with the base's access VCN, the service functions like any other customer service. Importantly, a reduced-footprint data center does not dedicate a large portion of its compute resources to CSP services in an immutable manner from the outset. If CSP services can be scaled down to meet customer demand, the freed-up resources can be offered to customers without physical scaling up. Complementarily, CSP services can also scale up in the same way as customer services, since the CSP services now reside in the CE overlay network. In the specific case of KMS, the techniques described in this paper allow KMS to retain the same security posture and root trust as a regular data center without cumbersome manual key signing processing and by omitting dedicated HSM equipment in a reduced-footprint data center.

[0097] Now turn to the attached image. Figure 1 This is a block diagram illustrating an example system architecture for a space-reduced data center 100, including initialization device 102. (See diagram for example.) Figure 1 As shown, the space-reduced data center 100 may include six racks of server equipment. These racks may be referred to as "butterfly" racks. The space-reduced data center 100 may include butterfly racks 110, 120, 130, 140, 150, and 160. In some embodiments, these racks may be identical. For example, butterfly rack 110 may include the same number of computing and / or networking devices as each of the other butterfly racks 120-160.

[0098] The Butterfly rack 110 may include two top-of-rack (TOR) switches 106 and 108. Each of the TOR switches 106 and 108 may include one or more networking switches configured to provide network communication between server equipment and other computing devices within the Butterfly rack 110, and to provide one or more networking connections to other Butterfly racks 120-160 and other networks (including customer network 114).

[0099] The Butterfly rack 110 may also include one or more BIOS devices 102. The BIOS device may be a server device configured to execute one or more processes to provide a set of "core services" within the reduced footprint data center 100 during boot / startup processing. The one or more BIOS devices 102 may configure one or more components of the reduced footprint data center 100 during boot. For example, the one or more BIOS devices 102 may send network configuration information to networking devices within the Butterfly racks 110-160. These networking devices may be SmartNICs attached to server devices within the Butterfly racks 110-160. As another example, the one or more BIOS devices 102 may send network configuration information to a base access VCN. The base access VCN may be deployed to one or more hosts within the reduced footprint data center 100. For example, a VM running in the Butterfly racks 110-160 may be configured as a base access VCN. The base access VCN can be configured to provide networking routing between one or more other VCNs (e.g., customer VCNs) and networking devices (e.g., SmartNICs), as well as other networking components that now perform base services in their own VCNs within the overlay layer.

[0100] In some embodiments, BIOS device 102 may also be configured to host one or more services, such as a Key Exchange Service (KeS), a Device Encryption Key (DEK) service, or other core services. Furthermore, BIOS device 102 may include a boot volume for booting VMs on host devices in a space-saving data center 100. For example, BIOS device 102 may provide a boot volume for VMs hosted on hypervisors hosted on one or more server devices 104.

[0101] The butterfly rack 110 may include one or more additional server devices 104. Each of the server devices 104 may include one or more processors and one or more memories, which together can store and execute instructions for implementing computing services as described herein, including, for example, computing, storage, VMs, CSP services, customer service, and / or applications. Figure 1As shown, each of the butterfly racks 110-160 can include the same number of server devices(s) and TORs. In some embodiments, while each of the butterfly racks 110-160 can include a BIOS device, the techniques described herein can be implemented using only a single BIOS device within a space-saving data center 100. Each of the server devices(s) 104 can include a Trusted Platform Module (TPM). The TPM on each device can be a microcontroller or other processor(s) and a storage area for performing cryptographic operations such as hashing, encryption / decryption, key and key pair generation, and key storage. TPMs can typically conform to standards characterizing such devices, such as ISO / IEC 11889.

[0102] The space-reduced data center 100 may also include a networked rack 112. The networked rack 112 may include one or more networking devices, including switches, gateways, routers, etc., for communicatively coupling the butterfly racks 110-160 to each other and to a customer network 114. The customer network 114 may include an on-premises network connected to the space-reduced data center 100. In some embodiments, the customer network 114 may provide network connectivity to a public network, including the Internet.

[0103] Figure 5 This is a block diagram illustrating an example architecture of a space-reduced data center 500 for a managed key management service 502 according to some embodiments. The KMS 502 can be hosted on VMs (not shown), which in turn reside on one or more computing devices 504 (e.g., Figure 1 The computing device 504 may also include a TPM 506 and a block storage device 520. The block storage device 520 may be one or more storage devices, such as a non-volatile memory fast (NVMe) solid-state drive, that are communicatively connected to the computing device 504.

[0104] KMS 502 can be configured to perform cryptographic service 503. Cryptographic service 503 can be one or more processes running on one or more VMs hosting KMS 502. Cryptographic service 503 can be configured to perform cryptographic operations, such as encrypting and decrypting keys and other data in the runtime environment. The memory allocated for cryptographic service 503 may not be persistent outside the instance of cryptographic service 503, so that any keys decrypted by cryptographic service 503 will not be maintained in a decrypted state by KMS 502.

[0105] The space-reduced data center 500 can communicate with host area 510. Host area 510 can be one or more data centers of the CSP. Host area 510 can be a conventionally configured data center with relatively large racks and server equipment. In particular, host area 510 can include HSM 512, which works in conjunction with KMS of host area 510. The communication channel between the space-reduced data center 500 and host area 510 can be a secure channel established for cross-realm communication between different data centers. HSM 512 can store root key 514. As briefly described above, root key 514 may have been signed by the CSP's root authority in a physical key signing ceremony, thereby establishing a chain of trust between root key 514 and the CSP's root authority.

[0106] To establish a chain of trust in the space-saving data center 500, KMS 502 can generate a master encryption key (MEK) 518. MEK 518 can then be transmitted to the host area 510 KMS for signing and encapsulation operations performed by HSM 512 using root key 514. As used herein, the term "encapsulation" describes a process by which a key or similar block of data is encrypted using another key, requiring a wrapping key to decrypt the wrapped key. Figure 5 In the example depicted, two distinct encapsulation / encryption operations can occur. The first is encapsulation performed by HSM 512 in host zone 510. MEK 518 can be encapsulated using root key 514, allowing the encapsulated MEK to be transmitted back to the reduced-footprint data center 500. This encapsulated (encrypted and signed) MEK can then be stored in the database of the reduced-footprint data center 500 (in...). Figure 5 (Not shown in the image) This is used for data recovery and subsequent retrieval and use by KMS 502 (e.g., if device 504 fails). A second encapsulation operation occurs using TPM 506 at the reduced footprint data center 500. The signed MEK 518 can be transmitted back to KMS 502, whereby TPM 506 can encapsulate MEK 518 to produce an encrypted MEK 522. The encrypted MEK 522 can be stored in block storage device 520. In addition to the data recovery operation, KMS 502 can perform cryptographic operations using the encrypted MEK 522 stored in block storage device 520 in response to a request received in the reduced footprint data center 500.

[0107] When cryptographic operations require it (e.g., to sign a certificate in response to a certificate signing request received from a process executing in the reduced footprint data center 500), KMS 502 can retrieve the encrypted MEK 522 from block storage device 520. The encrypted MEK 522 can be decapsulated by TPM 506 using encapsulation key 516 to obtain MEK 518, which can be maintained in the process memory of cryptographic service 503. Cryptographic operations may require the use of a client key. Because the reduced footprint data center 500 does not include an HSM, the client key may not be a hardware-based key stored at the HSM, but rather a software-based key that can be stored in database 524 as one of the client's encrypted keys 528. For security reasons, the encrypted key 528 can be encrypted using an intermediate encryption key (IEK) 526, which itself is encrypted using MEK 518 when it is stored in database 524. In order to use the client key for cryptographic operations, KMS 502 can retrieve the encrypted IEK 526, use MEK 518 in memory to decrypt the encrypted IEK 526 to generate IEK 530, and then use IEK 530 to decrypt one or more of the encrypted keys 528 to generate key 532, which is also maintained in an unencrypted manner only in the volatile memory of cryptographic service 503 during cryptographic operations.

[0108] Figure 6 This is a block diagram illustrating an example architecture and simplified process for a key signing ceremony in a key management service within a data center 600 for reducing footprint, according to some embodiments. The key management service (KMS) can be as described above... Figure 5An example of the described KMS 502. The KMS may be hosted on one or more VMs in a reduced-footprint data center 600, including a KMS host 602. The bare-metal device hosting the VMs may include a TPM that stores encapsulation keys used to encrypt (“encapsulate”) various cryptographic keys used in cryptographic operations performed by the KMS. For example, a TPM 606 may store an encapsulation key 608 that can be used by the KMS host 602. In some examples, the encapsulation key 608 may correspond to a key pair generated by the TPM 606, which includes both a public encapsulation key and a private encapsulation key. The private encapsulation key may be stored on the TPM 606 and never transmitted from the TPM 606, while the public encapsulation key may be the encapsulation key 608 that is described herein and provided to processes, applications, and / or other hosts within the reduced-footprint data center 600 or another host area for use when performing cryptographic operations. The KMS host 602 may execute a KSC application 604 to perform operations for a key signing ceremony with an existing host area 614.

[0109] As mentioned above Figure 5 The existing host area 614 discussed may include an HSM and related resources for generating key pairs and establishing a root of trust for a space-saving data center 600. Host area 614 may include an HSM 616 with a key vault 618 for storing cryptographic secrets, including a KMS root key 622 and an HSM encapsulation key 624.

[0110] In conjunction with HSM 616, Host Zone 614 may also host KMS instance 620, which may include one or more processes executed on host devices within Host Zone 614 and configured to perform operations for establishing root trust at Footprint Data Center 600. Operations for establishing the root trust and obtaining the MEK for Footprint Data Center 600 may be performed in response to requests from Footprint Data Center 600 (e.g., from KSC application 604). For example, KMS instance 620 may generate a KMS root key 622 stored in a key vault 618 of HSM 616. The public key of KMS root key 622 may be used to generate a KMS certificate 626, which may be sent to Footprint Data Center 600 and used as a root trust with KMS host 602. KMS instance 620 may then transfer the KMS certificate 626 to KSC application 604 in Footprint Data Center 600. KMS certificate 626 can be used to verify a signature generated from KMS root key 622 (e.g., the private key of the KMS root key pair) from HSM 616 to establish root trust within the reduced footprint data center 600. Furthermore, KMS instance 620 can be configured to generate MEK 628 and perform cryptographic encapsulation and decapsulation operations on behalf of the reduced footprint data center 600. After generating MEK 628, KMS instance 620 can use HSM encapsulation key 624 to encrypt MEK 630, producing HSM-encrypted MEK 630, an encrypted copy of the original MEK 628 accessible only via HSM 616. The HSM-encrypted MEK 630 can then be stored in database 632 of host zone 614 for subsequent use by KMS instance 620.

[0111] KMS instance 620 can receive encapsulation key 608 (e.g., the public encapsulation key of TPM 606 in the space-saving data center 600) from KSC application 604. KMS instance 620 can then encrypt MEK 628 to produce encrypted MEK 634. KMS instance 620 can also use KMS root key 622 to sign MEK 628 to produce signed MEK 636, which can be a cryptographic signature that can be verified using KMS certificate 626. Encapsulated MEK 634 and signed MEK 636 can be transmitted to KSC application 604. After receiving KMS certificate 626, encrypted MEK 634, and signed MEK 636, KSC application 604 can store KMS certificate 626, encrypted MEK 634, and signed MEK 636 on host disk 610 of KMS host 602 and in database 612 of space-saving data center 600.

[0112] Figure 6 The example architecture described above is... Figure 5 The KMS described herein provides additional details. Specifically, cross-domain communication channels 640 and 650 are shown for transmitting encapsulation key 608 from KMS host 602 to KMS instance 620 and for transmitting KMS certificate 626 to KMS host 602 and for providing signed MEK 636 and encrypted MEK 634 to a space-reduced data center 600. Cross-domain communication channels 640 and 650 can provide secure communication channels for transmitting various cryptographic materials to the space-reduced data center 600.

[0113] Figure 7 The illustration is based on some embodiments for use above regarding Figure 6 The diagram describes an example architecture and simplified process for host-to-host key transfer in a space-reduced data center 600. After KMS host 602 in the space-reduced data center obtains KMS certificate 626, encrypted MEK 634, and signed MEK 636, KMS host 602 can transfer these cryptographic materials to another KMS host (e.g., KMS host 712) in the space-reduced data center 600 in a manner that preserves the root trust established with KMS certificate 626 and allows KMS host 712 to encrypt the key using the encapsulation key 718 provided within a TPM 716 accessible to KMS host 712.

[0114] To perform key transfer operations, KMS host 602 can use KSC application 604 and KMS host 712 can use KSC application 714 to perform encryption / decryption operations (e.g., encapsulation / decapsulation) on existing keys at KMS host 602, and provide them to KMS host 712 for verification using KMS certificate 626 and encryption using encapsulation key 718. KSC application 714 can obtain encapsulation key 718 (the public key in the encapsulation key pair) from TPM 716 and provide it to KSC application 604. Similarly, KSC application 604 can obtain encapsulation key 608 (the public key in the encapsulation key pair) from TPM 606 and provide it to KSC application 714. KSC application 604 can then retrieve the encrypted MEK 634 and decrypt it using encapsulation key 608 to recover the plaintext MEK 628. KSC application 604 can then use the encapsulation key 718 (from KMS host 712) to encrypt MEK 628, producing an encrypted MEK 730, which can only be decrypted by TPM 716 at KMS host 712. KSC application 604 can then include the encrypted MEK 730 in a payload 732 that also contains KMS certificate 626 and a signed MEK 636. Payload 732 can then be signed using the private key from the encapsulation key pair in TPM 606, producing a payload signature 738, which can then be verified using the encapsulation key 608 (the public key from the encapsulation key pair). KSC application 604 can then transmit payload 732 and payload signature 738 to KSC application 714 at KMS host 712.

[0115] Once payload 732 and payload signature 738 are received by KSC application 714, KSC application 714 can verify payload 732 using encapsulation key 608. Payload 732 can first be verified by verifying payload signature 738 using encapsulation key 608. If this verification is successful, then KSC application 714 can then verify the signed MEK 636 using KMS certificate 626 and encapsulation key 718 (used to retrieve MEK 628 data) from payload 732. If verification is successful, then KSC application 714 can store KMS certificate 626, encrypted MEK 730, and signed MEK 636 in host disk 720 and database 740 of data center 600 with reduced footprint. In some examples, database 740 can be connected to... Figure 6 The database is the same as 612.

[0116] Figure 8 This is a block diagram illustrating a simplified process for performing cryptographic operations 800 using a key management service, according to some embodiments. KMS can be... Figure 5 An example of KMS 502. KMS can be hosted by VMs, including in data centers with reduced footprint (e.g., Figure 6 The KMS VM 802 is located within a space-reduced data center 600. The KMS may include a KMS data plane 810 and a KMS cryptographic service 804. The KMS cryptographic service 804 may be one or more processes configured to perform cryptographic operations (including decapsulating one or more keys) in response to requests from other services in the space-reduced data center.

[0117] A request for a cryptographic operation 814 can be received at the KMS data plane 810 in the KMS DP runtime memory 812. For example, the cryptographic operation 814 could be a request to encrypt data using a software key managed by the KMS (e.g., software key 834). In response to this request, the KMS data plane 812 can retrieve software key metadata 818 that identifies the software key 834 corresponding to the request. For example, the request could include information identifying a CSP customer, allowing the KMS data plane 810 to retrieve software key metadata 818 corresponding to that customer. The KMS cryptographic service 804 can then retrieve the software key metadata 818 from the host device storage device 824 (e.g., ...). Figure 6 The KMS cryptographic service 804 retrieves the encrypted MEK 826 from the KMS cryptographic service database 806 and has the TPM 828 decrypt it to generate MEK 830. The operation of decrypting the key to a plaintext key can be performed in the KMS cryptographic service runtime memory 808, so that the plaintext key is not stored in a non-volatile storage device in a space-saving data center. The KMS cryptographic service 804 can then retrieve the encrypted IEK 820 from the KMS cryptographic service database 806 and use MEK 830 to decrypt the encrypted IEK 820 to generate IEK 832. The KMS cryptographic service 804 can then determine the encrypted software key 822 corresponding to the software key metadata 818 in the KMS cryptographic service database 806 and use IEK 832 to decrypt the encrypted software key 822 to generate software key 834. Once software key 834 is available, the KMS cryptographic service 804 can perform the requested cryptographic operation 814 and provide the output to the requesting entity.

[0118] Figure 9 The illustration is for a key management service (e.g., according to some embodiments) Figure 5 A block diagram illustrating a simplified process for migrating KMS (502) from Trusted Platform Module support to Hardware Security Module support. KMS can be hosted in a space-saving data center (e.g., Figure 6 This is for one or more VMs (including KMS VM902) in a data center (600) that require less space. As mentioned above... Figure 9 As described, the KMS VM 902 may include a KMS cryptographic service 904 configured to perform cryptographic operations of KMS. The KMS cryptographic service 904 may include both runtime memory (e.g., KMS runtime memory 906) and persistent memory (KMS database 912). The KMS runtime memory 906 may store plaintext keys generated during cryptographic operations when decapsulating the various keys described herein.

[0119] If the reduction in data center footprint is scaled to the point where hardware-based key management is feasible, then KMS in the reduced-footprint data center can be improved by using TPM (e.g., Figure 5 The migration of software keys protected by a TPM 506 to a hardware-based key protection and storage system (HSM) is performed. To perform the migration, a bare-metal host device 924, including an HSM 926, can be connected to a space-reduced data center. The HSM 926 can be configured with an HSM root key 928, which is signed by the CSP's root issuing authority using the manual key signing process described above. Once the HSM 926 and bare-metal host device 924 are connected, the KMS VM 902 can obtain MEK 908 by decrypting the encrypted MEK 922 using the TPM and then encrypting it again using the HSM root key 928. MEK 908 can then be used to decapsulate the encrypted IEK 918 to produce IEK 910, which can then be used to encapsulate / decapsulate software keys, including the encrypted software key 920. The keys can be stored in the key vault 914 of the KMS database 912, including the encapsulation key 916, the encrypted IEK 918, and the encrypted software key 920. In this way, the HSM 926 replaces the TPM function at the KMS host device. If the customer chooses, the encrypted software key 920 can be migrated to the HSM 926 for storage. For example, the encrypted software key 920 can be transferred to the HSM 926 and stored in the key vault of the HSM 926.

[0120] Figure 10 This is a flowchart of an example process 1000 for implementing a key management service in a space-reducing data center, according to some embodiments. This process can be implemented by a space-reducing data center (e.g., Figure 1 The components of a space-saving data center (100) include one or more computing devices, such as... Figure 1 One or more server devices 104. KMS can be Figure 5 Example of KMS 502.

[0121] Processing 1000 can begin at box 1002, where computing devices in a space-reduced data center perform cryptographic services. For example, the cryptographic service could be a process initiated by a KMS running in the space-reduced data center. The cryptographic process can be configured to perform one or more cryptographic operations on data, including but not limited to encryption, decryption, hashing, generating keys or key pairs, and generating digital signatures. The cryptographic service can be performed in response to a request received by the KMS or other triggering events.

[0122] At box 1004, the cryptographic service can generate a master encryption key (MEK). The MEK can be a cryptographic key that can be used for cryptographic operations including key encapsulation (e.g., encryption).

[0123] At box 1006, the cryptographic service can encrypt the master encryption key to generate an encrypted master encryption key. For example, the cryptographic service can use a wrapper key stored by a security component to encrypt MEK. In some embodiments, the security component is a Trusted Platform Module (TPM). The encryption operation can be performed at the TPM.

[0124] At box 1008, the cryptographic service can store the encrypted MEK in a block storage volume communicatively connected to the computing device. For example, the encrypted MEK can be stored on an NVMe SSD attached to the computing device.

[0125] At box 1010, the cryptographic service may transfer the MEK (generated in the operation at box 1004) to a host area data center. The host area data center may be an existing data center associated with a CSP. The host area data center may include a hardware security module (HSM) configured to perform cryptographic operations using one or more root keys stored at the HSM and signed with a root authority key protected by the CSP (e.g., in a physically secure location). The host area data center may host its own KMS, which uses the HSM as a secure repository for its root keys. The host area data center KMS may sign and / or encrypt (e.g., encapsulate) the MEK. In some embodiments, transferring the MEK to the host area data center includes transmitting the MEK using a cross-domain communication channel established using identity data corresponding to the cryptographic service. The identity data may be generated within the host area data center.

[0126] At box 1012, the cryptographic service can receive an encapsulated master encryption key from the host area data center. The encapsulated MEK can be encrypted using a root key maintained at the host area data center. In some embodiments, the encapsulated MEK is generated by a hardware security module in the host area data center by: encrypting the master encryption key with the root encryption key to generate a second encrypted master encryption key, and signing the second encrypted master encryption key with the root signing key.

[0127] At box 1014, the cryptographic service can store the encapsulated MEK in a database with a reduced data center footprint.

[0128] In some embodiments, the cryptographic service may determine that the second computing device in the space-reducing data center does not have a MEK. The cryptographic service can then obtain the encrypted MEK and decrypt it using a security component of the computing device. The cryptographic service can then transmit the MEK to the second computing device using a secure communication channel. The cryptographic service performed at the second computing device can encrypt the MEK using a second security component of the second computing device to produce a second encrypted MEK. The second encrypted MEK can be stored in a second storage volume communicatively connected to the second computing device. In some embodiments, the secure communication channel may include a bidirectional transport layer security (mTLS) channel.

[0129] In some embodiments, the cryptographic service may receive an indication that data recovery processing has commenced. In response to this indication, the cryptographic service may retrieve the encapsulated master encryption key from a database and transmit the encapsulated master encryption key to the host data center. The host data center KMS may use the host data center's hardware security components to authenticate and decrypt the encapsulated MEK to generate a new MEK. The cryptographic service can then receive the MEK from the host data center.

[0130] Figure 11 This is a flowchart of an example process 1100 for performing cryptographic operations using a key management service in a space-reduced data center, according to some embodiments. Process 1100 can be performed by a space-reduced data center (e.g., Figure 1 The components of a space-saving data center (100) include one or more computing devices, such as... Figure 1 One or more server devices 104.

[0131] Processing 1100 can begin at box 1102, where the key management service (e.g., Figure 5The KMS (502) receives requests for cryptographic operations. For example, a customer of a CSP using a reduced-footprint data center may request a certificate signed using a customer key stored and maintained by the KMS in the reduced-footprint data center. In some embodiments, the cryptographic operation includes encryption operations on tenant data (e.g., encrypting the tenant data received with the request). In some embodiments, the cryptographic operation includes signing operations on the tenant data (e.g., generating a digital signature using the tenant data received with the request).

[0132] At box 1104, KMS can obtain an encrypted MEK from a block storage volume. The block storage volume can be communicatively connected to a computing device. For example, the block storage volume can be a volume on an NVMe storage device (e.g., a solid-state drive) of the computing device. KMS can obtain the encrypted MEK in response to a request.

[0133] At box 1106, the KMS can decrypt the encrypted MEK to generate another MEK. The KMS can use a secure component of the computing device to decrypt the encrypted MEK. For example, the secure component could be a TPM that stores the encapsulation key. The TPM can use the encapsulation key to decrypt the encrypted MEK (e.g., in the case where the MEK was encrypted using the encapsulation key to generate the encrypted MEK).

[0134] At box 1108, KMS can obtain the encrypted IEK. The encrypted IEK can be stored in a database accessible to KMS. The IEK can be another encryption key that KMS can use to encrypt / decrypt software keys maintained by KMS. The encrypted IEK can be stored in the same database used to store other software keys maintained by KMS, including encrypted client keys. The encrypted IEK can be encrypted using MEK.

[0135] At box 1110, KMS can use MEK to decrypt the encrypted IEK to generate IEK.

[0136] At box 1112, the KMS can obtain an encrypted software key from a database. The KMS can use this request to identify the encrypted software key. For example, the KMS can use this request to obtain metadata associated with the tenant transmitting the request (e.g., a customer in a data center reducing its footprint). The metadata can associate the stored encrypted software key with the customer and the request, allowing the KMS to determine the specific encrypted software key to retrieve from the database. In some embodiments, the KMS can obtain software key metadata from an additional database based on the request.

[0137] At box 1114, KMS can decrypt an encrypted software key to generate a software key. KMS can use IEK to decrypt an encrypted software key.

[0138] At box 1116, the KMS can then perform cryptographic operations using a software key. For example, the KMS can use the software key to generate a digital signature for data received as part of a request. In some embodiments, the KMS can transmit the output of the cryptographic operations to a tenant application running in a space-saving data center. For example, the tenant application could be a process running on a compute VM in a space-saving data center. The tenant application can send a request for a digital signature (e.g., a certificate signing request) for use within the tenant application. The cryptographic operations can then include the KMS signing the data using the tenant software key and transmitting that signature to the tenant application.

[0139] Figure 12 This is a flowchart of an example process for migrating a key management service in a space-reducing data center, according to some embodiments. Process 1200 can be performed by a space-reducing data center (e.g., Figure 1 The components of a space-saving data center (100) include one or more computing devices, such as... Figure 1 One or more server devices 124 and / or Figure 5 KMS 502.

[0140] Processing 1200 can begin at box 1202, where a bare-metal host device is connected to a reduced-footprint data center. This bare-metal host device may include a Hardware Security Module (HSM). For example, as part of a footprint expansion operation, a new server device including an HSM may be installed in a server rack within the reduced-footprint data center to scale out the available resources of the reduced-footprint data center to support additional cloud service capabilities. The HSM can be configured using a key signing ceremony. For example, the HSM may include a root encryption key and a root signing key, which are signed using a root issuing authority protected by a CSP.

[0141] At box 1204, the bare metal host device can execute an HSM agent process. The HSM agent process can be a program or application executing on the bare metal computing device or the HSM itself. The HSM agent process can be configured to use the HSM's processor and memory to perform operations. For example, the HSM agent process can use the HSM to perform cryptographic operations using the HSM's cryptographic processor and stored keys.

[0142] At box 1206, KMS can transfer the MEK to the HSM agent process. For example, KMS executed at one or more host devices in a space-saving data center can establish a secure connection to the bare metal host device (e.g., via the underlying network) to transfer the MEK.

[0143] At box 1208, the HSM agent can encapsulate the MEK to produce an encapsulated MEK. The encapsulated MEK can be an encrypted MEK generated by performing encryption of the MEK using the HSM's root key. In some embodiments, the encapsulated master encryption key is used to sign the encrypted master encryption key with the root signing key maintained by the HSM.

[0144] In some embodiments, before transmitting the MEK, the KMS can obtain the encrypted MEK from a block storage volume of a computing device communicatively connected to a data center with a reduced footprint. The KMS can then use the security components of the computing device to decrypt the encrypted MEK.

[0145] In some embodiments, the HSM agent process may store encapsulated encryption keys. In some embodiments, the KMS may store one or more software encryption keys at the HSM. These one or more software encryption keys may be obtained by the KMS from a database.

[0146] Example Infrastructure as a Service Architecture

[0147] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany these infrastructure components (e.g., example services include billing software, monitoring software, logging software, load balancing software, and clustering software, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0148] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0149] In most cases, cloud computing models may require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0150] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0151] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0152] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0153] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provisioned to define how inbound / outbound traffic to the network and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are expected and added.

[0154] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, it may be necessary to first set up the infrastructure on which the code will be deployed. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0155] Figure 13This is a block diagram 1300 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 1302 may be communicatively coupled to a secure host lease 1304, which may include a virtual cloud network (VCN) 1306 and a secure host subnet 1308. In some examples, service operator 1302 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. Client computing devices can be workstation computers running a variety of commercially available UNIX® or UNIX-like operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, client computing devices can be any other electronic device, such as thin client computers, internet-enabled gaming systems (e.g., Microsoft Xbox game consoles with or without Kinect® gesture input devices), and / or personal messaging devices capable of communicating over a network that can access VCN 1306 and / or the internet.

[0156] VCN 1306 may include a local peering gateway (LPG) 1310, which may be communicatively coupled to a secure shell (SSH) VCN 1312 via an LPG 1310 contained in an SSH VCN 1312. SSH VCN 1312 may include an SSH subnet 1314, and SSH VCN 1312 may be communicatively coupled to a control plane VCN 1316 via an LPG 1310 contained in a control plane VCN 1316. Furthermore, SSH VCN 1312 may be communicatively coupled to a data plane VCN 1318 via an LPG 1310. Control plane VCN 1316 and data plane VCN 1318 may be contained within a service lease 1319 that may be owned and / or operated by an IaaS provider.

[0157] The control plane VCN 1316 may include a control plane demilitarized zone (DMZ) layer 1320 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control security vulnerabilities. Furthermore, the DMZ layer 1320 may include one or more load balancer (LB) subnets 1322, a control plane application layer 1324 that may include one or more application subnets 1326, and a control plane data layer 1328 that may include one or more database (DB) subnets 1330 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1322 contained in the control plane DMZ layer 1320 may be communicatively coupled to one or more application subnets 1326 contained in the control plane application layer 1324 and an Internet gateway 1334 that may be contained in the control plane VCN 1316. The application subnets 1326 may be communicatively coupled to one or more DB subnets 1330 contained in the control plane data layer 1328, as well as a service gateway 1336 and a Network Address Translation (NAT) gateway 1338. The control plane VCN 1316 may include the service gateway 1336 and the NAT gateway 1338.

[0158] The control plane VCN 1316 may include a data plane mirror application layer 1340, which may include one or more application subnets 1326. The one or more application subnets 1326 included in the data plane mirror application layer 1340 may include a virtual network interface controller (VNIC) 1342 capable of executing a compute instance 1344. The compute instance 1344 may communicatively couple the one or more application subnets 1326 of the data plane mirror application layer 1340 to the one or more application subnets 1326 that may be included in the data plane application layer 1346.

[0159] Data plane VCN 1318 may include data plane application layer 1346, data plane DMZ layer 1348, and data plane data layer 1350. Data plane DMZ layer 1348 may include one or more LB subnets 1322 communicatively coupled to one or more application subnets 1326 of data plane application layer 1346 and Internet gateway 1334 of data plane VCN 1318. One or more application subnets 1326 may be communicatively coupled to service gateway 1336 and NAT gateway 1338 of data plane VCN 1318. Data plane data layer 1350 may also include one or more DB subnets 1330 communicatively coupled to one or more application subnets 1326 of data plane application layer 1346.

[0160] The Internet gateway 1334 of control plane VCN 1316 and data plane VCN 1318 can be communicatively coupled to metadata management service 1352, which can be communicatively coupled to public Internet 1354. Public Internet 1354 can be communicatively coupled to NAT gateway 1338 of control plane VCN 1316 and data plane VCN 1318. Service gateway 1336 of control plane VCN 1316 and data plane VCN 1318 can be communicatively coupled to cloud service 1356.

[0161] In some examples, the service gateway 1336, either the control plane VCN 1316 or the data plane VCN 1318, can make application programming interface (API) calls to the cloud service 1356 without traversing the public internet 1354. API calls from the service gateway 1336 to the cloud service 1356 can be unidirectional: the service gateway 1336 can make API calls to the cloud service 1356, and the cloud service 1356 can send requested data to the service gateway 1336. However, the cloud service 1356 may not initiate API calls to the service gateway 1336.

[0162] In some examples, secure host lease 1304 can be directly connected to service lease 1319, which would otherwise be isolated. Secure host subnet 1308 can communicate with SSH subnet 1314 via LPG 1310, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 1308 to SSH subnet 1314 allows secure host subnet 1308 to access other entities within service lease 1319.

[0163] Control plane VCN 1316 may allow users of service lease 1319 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 1316 may be deployed or otherwise used in data plane VCN 1318. In some examples, control plane VCN 1316 may be isolated from data plane VCN 1318, and the data plane mirror application layer 1340 of control plane VCN 1316 may communicate with the data plane application layer 1346 of data plane VCN 1318 via VNIC 1342, which may be included in both the data plane mirror application layer 1340 and the data plane application layer 1346.

[0164] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1354, which can transmit requests to the metadata management service 1352. The metadata management service 1352 can transmit requests to the control plane VCN 1316 via internet gateway 1334. Requests can be received by one or more LB subnets 1322 contained in the control plane DMZ layer 1320. The LB subnets 1322 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 1326 contained in the control plane application layer 1324. If the request is validated and requires a call to the public internet 1354, the call to the public internet 1354 can be transmitted to a NAT gateway 1338 that can make calls to the public internet 1354. The request may expect stored metadata to be stored in one or more DB subnets 1330.

[0165] In some examples, the data plane mirroring application layer 1340 can facilitate direct communication between the control plane VCN 1316 and the data plane VCN 1318. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 1318. Through VNIC 1342, the control plane VCN 1316 can communicate directly with the resources contained in the data plane VCN 1318, and thus can perform configuration changes, updates, or other appropriate modifications.

[0166] In some embodiments, the control plane VCN 1316 and data plane VCN 1318 may be included in service lease 1319. In this case, the system's users or customers may not own or operate the control plane VCN 1316 or data plane VCN 1318. Alternatively, the IaaS provider may own or operate the control plane VCN 1316 and data plane VCN 1318, both of which may be included in service lease 1319. This embodiment can enable the isolation of networks that might prevent users or customers from interacting with resources of other users or customers. Furthermore, this embodiment can allow users or customers of the system to privately store databases without relying on the public Internet 1354, which may not have the desired level of threat prevention for storage.

[0167] In other embodiments, one or more LB subnets 1322 included in the control plane VCN 1316 may be configured to receive signals from the service gateway 1336. In this embodiment, the control plane VCN 1316 and the data plane VCN 1318 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1354. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on service lease 1319, which may be isolated from the public internet 1354.

[0168] Figure 14 This is a block diagram 1400 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1402 (e.g., Figure 13 The service provider 1302 can communicatively couple to the secure host lease 1404 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1404 may include a virtual cloud network (VCN) 1406 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1408 (e.g., Figure 13 The secure host subnet 1308). VCN 1406 may include a local peering gateway (LPG) 1410 (e.g., Figure 13 The LPG 1310, which can be communicatively coupled to the Secure Shell (SSH) VCN 1412 (e.g., LPG 1310 contained in the SSH VCN 1412) Figure 13 SSH VCN 1312). SSH VCN 1412 can include SSH subnet 1414 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1412 can be communicatively coupled to control plane VCN 1416 via LPG 1410 contained in control plane VCN 1416 (e.g., Figure 13 Control plane VCN 1316). Control plane VCN 1416 may be included in service lease 1419 (e.g., Figure 13 In the service lease 1319), and the data plane VCN 1418 (e.g., Figure 13 The data plane VCN 1318 may be included in a customer lease 1421 that may be owned or operated by a user or customer of the system.

[0169] Control plane VCN 1416 may include control plane DMZ layer 1420 (e.g., Figure 13 The control plane DMZ layer 1320), which may include one or more LB subnets 1422 (e.g., Figure 13 (one or more) LB subnets 1322), may include (one or more) application subnets 1426 (e.g., Figure 13 The control plane application layer 1424 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324 may include one or more database (DB) subnets 1430 (e.g., similar to...). Figure 13 The control plane data layer 1428 of (one or more) DB subnets 1330 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1422 contained in the control plane DMZ layer 1420 can be communicatively coupled to one or more application subnets 1426 contained in the control plane application layer 1424 and an Internet gateway 1434 that can be contained in the control plane VCN 1416 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1426 can communicatively couple to DB subnet(s) 1430 contained in control plane data layer 1428 and service gateway 1436 (e.g., Figure 13 Service gateway 1336) and Network Address Translation (NAT) gateway 1438 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0170] The control plane VCN 1416 may include a data plane mirror application layer 1440 that may contain one or more application subnets 1426 (e.g., Figure 13 The data plane mirror application layer 1340). One or more application subnets 1426 contained in the data plane mirror application layer 1440 may include computational instances 1444 (e.g., similar to...). Figure 13 The virtual network interface controller (VNIC) 1442 (e.g., the VNIC of 1342) of the computing instance 1344. The computing instance 1444 may facilitate the mirroring of the application subnet(s) 1426 of the application layer 1440 in the data plane and may be included in the application layer 1446 in the data plane (e.g., Figure 13 Communication between one or more application subnets 1426 in the data plane application layer 1346 via VNIC 1442 contained in the data plane mirror application layer 1440 and VNIC 1442 contained in the data plane application layer 1446.

[0171] The Internet gateway 1434, included in the control plane VCN 1416, can be communicatively coupled to the metadata management service 1452 (e.g., Figure 13 Metadata management service 1352), which can communicatively couple to the public Internet 1454 (e.g., Figure 13 The public internet 1454 can communicatively couple to a NAT gateway 1438 contained in a control plane VCN 1416. The service gateway 1436 contained in the control plane VCN 1416 can communicatively couple to a cloud service 1456 (e.g., ...). Figure 13 Cloud services (1356).

[0172] In some examples, data plane VCN 1418 may be included in customer lease 1421. In this case, the IaaS provider may provide control plane VCN 1416 for each customer, and the IaaS provider may set up a unique compute instance 1444 for each customer, included in service lease 1419. Each compute instance 1444 may allow communication between control plane VCN 1416 included in service lease 1419 and data plane VCN 1418 included in customer lease 1421. Compute instance 1444 may allow resources provisioned in control plane VCN 1416 included in service lease 1419 to be deployed or otherwise used in data plane VCN 1418 included in customer lease 1421.

[0173] In other examples, an IaaS provider's customer may have a database residing in customer lease 1421. In this example, control plane VCN 1416 may include data plane mirror application layer 1440, which may include one or more application subnets 1426. Data plane mirror application layer 1440 may reside in data plane VCN 1418, but may not reside in data plane VCN 1418. That is, data plane mirror application layer 1440 may have access to customer lease 1421, but may not reside in data plane VCN 1418 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 1440 may be configured to invoke data plane VCN 1418, but may not be configured to invoke any entity contained in control plane VCN 1416. Customers may expect to deploy or otherwise use resources provided in the control plane VCN 1416 in the data plane VCN 1418, and the data plane mirroring application layer 1440 can facilitate customers' expected deployments or other uses of resources.

[0174] In some embodiments, an IaaS provider's customer may apply filters to data plane VCN 1418. In this embodiment, the customer may determine what data plane VCN 1418 can access, and the customer may restrict access from data plane VCN 1418 to the public internet 1454. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1418's access to any external networks or databases. Applying filters and controls to data plane VCN 1418 contained in customer lease 1421 helps isolate data plane VCN 1418 from other customers and the public internet 1454.

[0175] In some embodiments, cloud service 1456 may be invoked by service gateway 1436 to access services that may not exist on public internet 1454, control plane VCN 1416, or data plane VCN 1418. The connection between cloud service 1456 and control plane VCN 1416 or data plane VCN 1418 may not be real-time or continuous. Cloud service 1456 may reside on different networks owned or operated by an IaaS provider. Cloud service 1456 may be configured to receive calls from service gateway 1436 and may be configured not to receive calls from public internet 1454. Some cloud services 1456 may be isolated from other cloud services 1456, and control plane VCN 1416 may be isolated from cloud services 1456 that may not be in the same region as control plane VCN 1416. For example, control plane VCN 1416 may be located in "Region 1," and cloud service "Deployment 13" may be located in both "Region 1" and "Region 2." If the service gateway 1436, contained in the control plane VCN 1416 located in region 1, makes a call to deployment 13, then that call can be transmitted to deployment 13 in region 1. In this example, the control plane VCN 1416 or deployment 13 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 13 in region 2.

[0176] Figure 15 This is a block diagram 1500 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1502 (e.g., Figure 13 The service provider 1302) can communicatively couple to the secure host lease 1504 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1504 may include a virtual cloud network (VCN) 1506 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1508 (e.g., Figure 13 The secure host subnet 1308). VCN 1506 can include LPG 1510 (e.g., Figure 13 The LPG 1310), which can be communicatively coupled to the SSH VCN 1512 via the LPG 1510 included in the SSH VCN 1512 (e.g., Figure 13 SSH VCN 1312). SSH VCN 1512 can include SSH subnet 1514 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1512 can be communicatively coupled to control plane VCN 1516 via LPG 1510 included in control plane VCN 1516 (e.g., Figure 13 The control plane VCN 1316) and coupled to the data plane VCN 1518 via the LPG 1510 contained in the data plane VCN 1518 (e.g., Figure 13 Data plane 1318). Control plane VCN 1516 and data plane VCN 1518 may be included in service lease 1519 (e.g., Figure 13 In the service rental (1319).

[0177] The control plane VCN 1516 may include a subnet 1522 that may contain one or more load balancer (LB) subnets (e.g., Figure 13 The control plane DMZ layer 1520 of (one or more) LB subnets 1322) (e.g., Figure 13 The control plane DMZ layer 1320 may include one or more application subnets 1526 (e.g., similar to...). Figure 13 The control plane application layer 1524 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324), and may include (one or more) DB subnets 1530, and the control plane data layer 1528 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1522 contained in the control plane DMZ layer 1520 can be communicatively coupled to one or more application subnets 1526 contained in the control plane application layer 1524 and an Internet gateway 1534 that can be contained in the control plane VCN 1516 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1526 can communicatively couple to DB subnet(s) 1530 contained in control plane data layer 1528 and service gateway 1536 (e.g., Figure 13 The service gateway) and Network Address Translation (NAT) gateway 1538 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1516 may include the service gateway 1536 and the NAT gateway 1538.

[0178] Data plane VCN 1518 may include data plane application layer 1546 (e.g., Figure 13 Data plane application layer 1346), data plane DMZ layer 1548 (e.g., Figure 13 Data plane DMZ layer 1348), and data plane data layer 1550 (e.g., Figure 13 The data plane data layer 1350. The data plane DMZ layer 1548 may include one or more trusted application subnets 1560 and one or more untrusted application subnets 1562 that can be communicatively coupled to the data plane application layer 1546, and one or more LB subnets 1522 of the Internet gateway 1534 contained in the data plane VCN 1518. One or more trusted application subnets 1560 may be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518, the NAT gateway 1538 contained in the data plane VCN 1518, and one or more DB subnets 1530 contained in the data plane data layer 1550. One or more untrusted application subnets 1562 may be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518 and the one or more DB subnets 1530 contained in the data plane data layer 1550. The data plane data layer 1550 may include one or more DB subnets 1530 that can be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518.

[0179] One or more untrusted application subnets 1562 may include one or more primary VNICs 1564(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1566(1)-(N). Each tenant VM 1566(1)-(N) may be communicatively coupled to a corresponding application subnet 1567(1)-(N) that may be contained in a corresponding container egress VCN 1568(1)-(N), which may be contained in a corresponding customer lease 1570(1)-(N). A corresponding secondary VNIC 1572(1)-(N) may facilitate communication between one or more untrusted application subnets 1562 contained in data plane VCN 1518 and application subnets contained in container egress VCN 1568(1)-(N). Each container exit VCN 1568(1)-(N) may include a NAT gateway 1538, which can communicatively couple to the public Internet 1554 (e.g., Figure 13 (Public Internet 1354).

[0180] Internet gateway 1534, contained in control plane VCN 1516 and data plane VCN 1518, can be communicatively coupled to metadata management service 1552 (e.g., Figure 13 A metadata management system 1352 is provided, which can communicatively couple to the public internet 1554. The public internet 1554 can communicatively couple to a NAT gateway 1538 contained in a control plane VCN 1516 and a data plane VCN 1518. A service gateway 1536 contained in a control plane VCN 1516 and a data plane VCN 1518 can communicatively couple to a cloud service 1556.

[0181] In some embodiments, the data plane VCN 1518 may be integrated with the customer lease 1570. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer to the IaaS provider.

[0182] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request that functionality be attached to a data plane layer application 1546. The code running this functionality may execute in VMs 1566(1)-(N) and may not be configured to run anywhere else on the data plane VCN 1518. Each VM 1566(1)-(N) may be connected to a customer lease 1570. The corresponding container 1571(1)-(N) contained in VMs 1566(1)-(N) may be configured to run the code. In this case, there may be dual isolation (e.g., container 1571(1)-(N) running the code, where container 1571(1)-(N) may be contained in at least one or more untrusted application subnets 1562 containing VMs 1566(1)-(N)), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1571(1)-(N) may be communicatively coupled to customer lease 1570 and may be configured to transmit or receive data from customer lease 1570. Containers 1571(1)-(N) may not be configured to transmit or receive data from any other entity in the data plane VCN 1518. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of containers 1571(1)-(N).

[0183] In some embodiments, one or more trusted application subnets 1560 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1560 may be communicatively coupled to one or more database subnets 1530 and configured to perform CRUD operations in one or more database subnets 1530. One or more untrusted application subnets 1562 may be communicatively coupled to one or more database subnets 1530, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1530. Containers 1571(1)-(N) that may be contained in each customer's VM 1566(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1530.

[0184] In other embodiments, the control plane VCN 1516 and the data plane VCN 1518 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1516 and the data plane VCN 1518. However, communication can occur indirectly through at least one method. The LPG 1510 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1516 and the data plane VCN 1518. In another example, either the control plane VCN 1516 or the data plane VCN 1518 may invoke the cloud service 1556 via the service gateway 1536. For example, an invocation of the cloud service 1556 from the control plane VCN 1516 may include a request for a service that can communicate with the data plane VCN 1518.

[0185] Figure 16 This is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1602 (e.g., Figure 13 The service provider 1302) can communicatively couple to the secure host lease 1604 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1604 may include a virtual cloud network (VCN) 1606 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1608 (e.g., Figure 13 The secure host subnet 1308). VCN 1606 can include LPG 1610 (e.g., Figure 13 The LPG 1310), the LPG 1610 can be accessed via SSH VCN 1612 (e.g., LPG 1310), Figure 13The LPG 1610 in SSH VCN 1312 is communicatively coupled to SSH VCN 1612. SSH VCN 1612 may include SSH subnet 1614 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1612 can be communicatively coupled to control plane VCN 1616 via LPG 1610 included in control plane VCN 1616 (e.g., Figure 13 The control plane VCN 1316) and coupled to the data plane VCN 1618 via the LPG 1610 contained in the data plane VCN 1618 (e.g., Figure 13 Data plane 1318). Control plane VCN 1616 and data plane VCN 1618 may be included in service lease 1619 (e.g., Figure 13 In the service rental (1319).

[0186] The control plane VCN 1616 may include one or more LB subnets 1622 (e.g., Figure 13 The control plane DMZ layer 1620 of (one or more) LB subnets 1322) (e.g., Figure 13 The control plane DMZ layer 1320 may include one or more application subnets 1626 (e.g., Figure 13 The control plane application layer 1624 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324 may include (one or more) DB subnets 1630 (e.g., Figure 15 The control plane data layer 1628 of (one or more) DB subnets 1530 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1622 contained in the control plane DMZ layer 1620 can be communicatively coupled to one or more application subnets 1626 contained in the control plane application layer 1624 and an Internet gateway 1634 that can be contained in the control plane VCN 1616 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1626 can communicatively couple to DB subnet(s) 1630 contained in control plane data layer 1628 and service gateway 1636 (e.g., Figure 13 The service gateway) and Network Address Translation (NAT) gateway 1638 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0187] The data plane VCN 1618 may include the data plane application layer 1646 (e.g., Figure 13 Data plane application layer 1346), data plane DMZ layer 1648 (e.g., Figure 13 Data plane DMZ layer 1348), and data plane data layer 1650 (e.g., Figure 13 The data plane data layer 1350). The data plane DMZ layer 1648 may include one or more trusted application subnets 1660 that can be communicatively coupled to the data plane application layer 1646 (e.g., Figure 15 (one or more) trusted application subnets 1560 and (one or more) untrusted application subnets 1662 (e.g., Figure 15 The data plane includes one or more untrusted application subnets 1562 and one or more LB subnets 1622 of an Internet gateway 1634 contained in data plane VCN 1618. One or more trusted application subnets 1660 may communicatively couple to a service gateway 1636 contained in data plane VCN 1618, a NAT gateway 1638 contained in data plane VCN 1618, and one or more DB subnets 1630 contained in data plane data layer 1650. One or more untrusted application subnets 1662 may communicatively couple to a service gateway 1636 contained in data plane VCN 1618 and one or more DB subnets 1630 contained in data plane data layer 1650. Data plane data layer 1650 may include one or more DB subnets 1630 that may communicatively couple to a service gateway 1636 contained in data plane VCN 1618.

[0188] One or more untrusted application subnets 1662 may include a primary VNIC 1664(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1666(1)-(N) residing within one or more untrusted application subnets 1662. Each tenant VM 1666(1)-(N) may run code in a corresponding container 1667(1)-(N) and is communicatively coupled to an application subnet 1626 that may be contained in a data plane application layer 1646 contained in a container egress VCN 1668. A corresponding secondary VNIC 1672(1)-(N) may facilitate communication between one or more untrusted application subnets 1662 contained in a data plane VCN 1618 and the application subnet contained in a container egress VCN 1668. The container egress VCN may include a public internet 1654 (e.g., Figure 13 The public internet (1354) uses NAT gateway 1638.

[0189] Internet gateway 1634, contained in control plane VCN 1616 and data plane VCN 1618, can be communicatively coupled to metadata management service 1652 (e.g., Figure 13 A metadata management system 1352 is provided, which can communicatively couple to the public internet 1654. The public internet 1654 can communicatively couple to a NAT gateway 1638 contained in a control plane VCN 1616 and a data plane VCN 1618. A service gateway 1636 contained in a control plane VCN 1616 and a data plane VCN 1618 can communicatively couple to a cloud service 1656.

[0190] In some examples, Figure 16 The architecture shown in the block diagram 1600 can be considered as... Figure 15 This is an exception to the pattern shown in the architecture of block diagram 1500, and this pattern may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1667(1)-(N) contained in each customer's VM 1666(1)-(N). Container 1667(1)-(N) can be configured to invoke the corresponding auxiliary VNIC 1672(1)-(N) contained in one or more application subnets 1626 of the data plane application layer 1646, which may be contained in the container egress VCN 1668. The auxiliary VNIC 1672(1)-(N) can transmit the invocation to NAT gateway 1638, which can transmit the invocation to the public Internet 1654. In this example, containers 1667(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1616 and from other entities contained in the data plane VCN 1618. Containers 1667(1)-(N) can also be isolated from resources from other clients.

[0191] In other examples, a client can use containers 1667(1)-(N) to invoke cloud service 1656. In this example, the client can run code within containers 1667(1)-(N) requesting a service from cloud service 1656. Container 1667(1)-(N) can then forward the request to a secondary VNIC 1672(1)-(N), which can then forward the request to a NAT gateway, which can then forward the request to the public internet 1654. The public internet 1654 can then forward the request via internet gateway 1634 to one or more LB subnets 1622 contained in control plane VCN 1616. In response to determining that the request is valid, one or more LB subnets can then forward the request to one or more application subnets 1626, which can then forward the request to cloud service 1656 via service gateway 1636.

[0192] It should be recognized that the IaaS architectures 1300, 1400, 1500, and 1600 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0193] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0194] Figure 17 An example computer system 1700 in which various embodiments of the present disclosure may be implemented is illustrated. System 1700 can be used to implement any of the computer systems described above. As shown, computer system 1700 includes a processing unit 1704 that communicates with a plurality of peripheral subsystems via a bus subsystem 1702. These peripheral subsystems may include a processing acceleration unit 1706, an I / O subsystem 1708, a storage subsystem 1718, and a communication subsystem 1724. Storage subsystem 1718 includes a tangible computer-readable storage medium 1722 and system memory 1710.

[0195] Bus subsystem 1702 provides a mechanism for allowing various components and subsystems of computer system 1700 to communicate with each other as intended. While bus subsystem 1702 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1702 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0196] A processing unit 1704, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1700. One or more processors may be included in the processing unit 1704. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1704 may be implemented as one or more independent processing units 1732 and / or 1734, each including a single-core or multi-core processor. In other embodiments, the processing unit 1704 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0197] In various embodiments, processing unit 1704 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1704 and / or storage subsystem 1718. With appropriate programming, processor(s) 1704 can provide the various functions described above. Computer system 1700 may additionally include processing acceleration unit 1706, which may include digital signal processor (DSP), dedicated processor, etc.

[0198] I / O subsystem 1708 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, keyboards, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input in an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0199] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0200] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1700 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0201] Computer system 1700 may include storage subsystem 1718, which provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 1704, provide the aforementioned functionality. Storage subsystem 1718 may also provide a repository for storing data used according to this disclosure.

[0202] like Figure 17 As illustrated in the example, storage subsystem 1718 may include various components, including system memory 1710, computer-readable storage medium 1722, and computer-readable storage medium reader 1720. System memory 1710 may store program instructions that can be loaded and executed by processing unit 1704. System memory 1710 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1710, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0203] System memory 1710 may also store operating system 1716. Examples of operating system 1716 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations of computer system 1700 that execute one or more virtual machines, the virtual machine and its guest operating system (GOS) may be loaded into system memory 1710 and executed by one or more processors or cores of processing unit 1704.

[0204] Depending on the type of computer system 1700, system memory 1710 can have different configurations. For example, system memory 1710 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, system memory 1710 may include a basic input / output system (BIOS) containing basic routines that facilitate, for example, the transfer of information between components within computer system 1700 during startup.

[0205] Computer-readable storage medium 1722 may represent a remote, local, fixed and / or removable storage device and a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1704 of the computer system 1700) for use by the computer system 1700.

[0206] Computer-readable storage medium 1722 may include any suitable medium known or used in the art, including storage and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0207] As an example, computer-readable storage medium 1722 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, disk drives that read from or write to removable non-volatile magnetic disks, and optical disc drives that read from or write to removable non-volatile optical discs (such as CD ROMs, DVDs, and Blu-ray® discs or other optical media). Computer-readable storage medium 1722 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital video tapes, and so on. Computer-readable storage medium 1722 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program services and other data for computer system 1700.

[0208] Machine-readable instructions executable by one or more processors or cores of the processing unit 1704 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0209] The communication subsystem 1724 provides an interface to other computer systems and networks. The communication subsystem 1724 serves as an interface for receiving data from other systems and sending data from computer system 1700 to other systems. For example, the communication subsystem 1724 enables computer system 1700 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1724 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1724 may provide a wired network connection (e.g., Ethernet).

[0210] In some embodiments, the communication subsystem 1724 may also represent one or more users who can use the computer system 1700 to receive input communications in the form of structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc.

[0211] As an example, the communication subsystem 1724 can be configured to receive data feeds 1726 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0212] Furthermore, the communication subsystem 1724 can also be configured to receive data in the form of a continuous data stream, which may include event streams 1728 and / or event updates 1730 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quotation machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0213] The communication subsystem 1724 can also be configured to output structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1700.

[0214] The computer system 1700 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0215] Due to the ever-evolving nature of computers and networks, the description of the computer system 1700 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0216] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0217] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0218] Accordingly, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made therein without departing from the broader spirit and scope set forth in the claims. Therefore, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0219] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as encompassing both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0220] Disjunctive language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, each individually.

[0221] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0222] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0223] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A method comprising: Perform cryptographic services on computing devices in data centers that minimize their footprint; The master encryption key is generated by the cryptographic service; The master encryption key is encrypted using the security components of the computing device to generate an encrypted master encryption key; The encrypted master encryption key is stored in a block storage volume that is communicatively connected to the computing device; The cryptographic service transmits the master encryption key to the host regional data center; The cryptographic service receives the encapsulated master encryption key from the host region data center; as well as The encapsulated master encryption key is stored in the database of the space-reduced data center.

2. The method as described in claim 1, wherein, The encapsulated master encryption key is generated by the hardware security module of the host area data center by encrypting the master encryption key with a root encryption key to generate a second encrypted master encryption key and signing the second encrypted master encryption key with a root signing key.

3. The method as described in claim 1, wherein, Transmitting the master encryption key to the host regional data center includes: transmitting the master encryption key using a cross-domain communication channel, the cross-domain communication channel being established using identity data corresponding to the cryptographic service, the identity data being generated in the host regional data center.

4. The method of claim 1, wherein, The computing device is a first computing device, the security component is a first security component, and the method further includes: It was determined that the second computing device in the data center with reduced footprint did not possess the master encryption key; The encrypted master encryption key is obtained by the cryptographic service at the first computing device; The encrypted master encryption key is decrypted using the first security component to generate the master encryption key; The master encryption key is transmitted to the second computing device using a secure communication channel; The master encryption key is encrypted using a second security component of the second computing device to generate a second encrypted master encryption key; and The second encrypted master encryption key is stored in a second storage volume that is connected to the second computing device.

5. The method of claim 4, wherein, The secure communication channel includes a bidirectional transport layer security (mTLS) channel.

6. The method of claim 1, further comprising: Receive an instruction that data recovery processing has been initiated; In response to the instruction, the encapsulated master encryption key is retrieved from the database; The encapsulated master encryption key is transmitted to the host data center; as well as The master encryption key is received from the host data center, wherein the encapsulated master encryption key is authenticated and decrypted by the hardware security component of the host data center to generate the master encryption key.

7. The method of claim 1, wherein, The security component includes a trusted platform module.

8. A computing system, comprising: One or more processors; as well as One or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the computing system to at least: Perform cryptographic services at the computing devices of the computing system; The master encryption key is generated by the cryptographic service; The master encryption key is encrypted using the security components of the computing device to generate an encrypted master encryption key; The encrypted master encryption key is stored in a block storage volume that is communicatively connected to the computing device; The cryptographic service transmits the master encryption key to the host regional data center; The cryptographic service receives the encapsulated master encryption key from the host region data center; as well as The encapsulated master encryption key is stored in the database of the computing system.

9. The computing system of claim 8, wherein, The encapsulated master encryption key is generated by the hardware security module of the host area data center by encrypting the master encryption key with a root encryption key to generate a second encrypted master encryption key and signing the second encrypted master encryption key with a root signing key.

10. The computing system of claim 8, wherein, Transmitting the master encryption key to the host regional data center includes: transmitting the master encryption key using a cross-domain communication channel, the cross-domain communication channel being established using identity data corresponding to the cryptographic service, the identity data being generated in the host regional data center.

11. The computing system of claim 8, wherein, The computing device is a first computing device, the security component is a first security component, and wherein the one or more memories store additional computer-executable instructions that, when executed by the one or more processors, cause the computing system to further: It is determined that the second computing device of the computing system does not have the master encryption key; The encrypted master encryption key is obtained by the cryptographic service at the first computing device; The encrypted master encryption key is decrypted using the first security component to generate the master encryption key; The master encryption key is transmitted to the second computing device using a secure communication channel; The master encryption key is encrypted using a second security component of the second computing device to generate a second encrypted master encryption key; as well as The second encrypted master encryption key is stored in a second storage volume that is connected to the second computing device.

12. The computing system of claim 11, wherein, The secure communication channel includes a bidirectional transport layer security (mTLS) channel.

13. The computing system of claim 8, wherein, The one or more memories store additional computer-executable instructions that, when executed by the one or more processors, cause the computing system to further: Receive an instruction that data recovery processing has been initiated; In response to the instruction, the encapsulated master encryption key is retrieved from the database; The encapsulated master encryption key is transmitted to the host data center; as well as The master encryption key is received from the host data center, wherein the encapsulated master encryption key is authenticated and decrypted by the hardware security component of the host data center to generate the master encryption key.

14. The computing system of claim 8, wherein, The security component includes a trusted platform module.

15. A non-transitory computer-readable medium storing instructions, which, when executed by one or more processors of a computing system, cause the computing system to at least: Perform cryptographic services at the computing device of the computing system; The master encryption key is generated by the cryptographic service; The master encryption key is encrypted using the security components of the computing device to generate an encrypted master encryption key; The encrypted master encryption key is stored in a block storage volume that is communicatively connected to the computing device; The cryptographic service transmits the master encryption key to the host regional data center; The cryptographic service receives the encapsulated master encryption key from the host region data center; as well as The encapsulated master encryption key is stored in the database of the computing system.

16. The non-transitory computer-readable medium of claim 15, wherein, The encapsulated master encryption key is generated by the hardware security module of the host area data center by encrypting the master encryption key with a root encryption key to generate a second encrypted master encryption key and signing the second encrypted master encryption key with a root signing key.

17. The non-transitory computer-readable medium of claim 15, wherein, Transmitting the master encryption key to the host regional data center includes: transmitting the master encryption key using a cross-domain communication channel, the cross-domain communication channel being established using identity data corresponding to the cryptographic service, the identity data being generated in the host regional data center.

18. The non-transitory computer-readable medium of claim 15, wherein, The computing device is a first computing device, the security component is a first security component, and wherein the one or more memories store additional computer-executable instructions that, when executed by the one or more processors, cause the computing system to further: It is determined that the second computing device of the computing system does not have the master encryption key; The encrypted master encryption key is obtained by the cryptographic service at the first computing device; The encrypted master encryption key is decrypted using the first security component to generate the master encryption key; The master encryption key is transmitted to the second computing device using a secure communication channel; The master encryption key is encrypted using a second security component of the second computing device to generate a second encrypted master encryption key; as well as The second encrypted master encryption key is stored in a second storage volume that is connected to the second computing device.

19. The non-transitory computer-readable medium of claim 18, wherein, The secure communication channel includes a bidirectional transport layer security (mTLS) channel.

20. The non-transitory computer-readable medium of claim 15, wherein the one or more memories store additional computer-executable instructions, which, when executed by the one or more processors, cause the computing system to further: Receive an instruction that data recovery processing has been initiated; In response to the instruction, the encapsulated master encryption key is retrieved from the database; The encapsulated master encryption key is transmitted to the host data center; as well as The master encryption key is received from the host data center, wherein the encapsulated master encryption key is authenticated and decrypted by the hardware security component of the host data center to generate the master encryption key.

Images