Techniques for image-based region construction

Abstract

Techniques are disclosed for building regional data centers using image-based resource deployment. A manager service executing in a distributed computing system can deploy software resources to a first set of physical resources within a data center and generate a set of images for the first set of physical resources. The set of images can include a software image corresponding to each physical resource in the first set of physical resources. The manager service can also determine whether a second set of physical resources is compatible with the set of images and deploy the set of images to the second set of physical resources. The manager service can then configure identifiers associated with the software resources of the set of images deployed to the second set of physical resources.

Description

[0001] Cross-reference to related applications

[0002] This international application claims priority and benefit to U.S. nonprovisional application No. 18 / 417,179, filed January 19, 2024, entitled “TECHNIQUES FOR IMAGE-BASEDREGION BUILD,” the entire contents of which are incorporated herein by reference for all purposes.

[0003] This application is also related to the following applications, the entire contents of which are incorporated herein by reference for all purposes:

[0004] (1) U.S. non-provisional application No. 18 / 122,674, filed on March 16, 2023, entitled “TECHNIQUES FOR BUILDING CLOUD REGIONSAT A PREFAB FACTORY”, agent file number 088325-1307191 (344000US);

[0005] (2) U.S. non-provisional application No. 18 / 122,675, filed on March 16, 2023, entitled “TECHNIQUES FOR VALIDATING CLOUDREGIONS BUILT AT A PREFAB FACTORY”, agent file number 088325-1373430 (344040US);

[0006] (3) U.S. non-provisional application No. 18 / 215,632, filed on June 28, 2023, entitled “TECHNIQUES FOR ROTATING NETWORKADDRESSES IN PREFAB REGIONS”, agent file number 088325-1307193 (344100US); and

[0007] (4) U.S. non-provisional application No. 18 / 382,885, filed on October 23, 2023, entitled “TECHNIQUES FOR ROTATING SERVICEENDPOINTS IN PREFAB REGIONS”, agent file number 088325-1307192 (344200US);

[0008] (5) U.S. non-provisional application No. 18 / 520,510, filed on November 27, 2023, entitled “TECHNIQUES FOR ROTATING RESOURCEIDENTIFIERS IN PREFAB REGIONS”, agent file number 088325-1327651 (347900US). Technical Field

[0009] This disclosure relates to cloud computing data centers. More specifically, this disclosure describes techniques for constructing regional data centers using software images of previously constructed template regional data centers. Background Technology

[0010] Cloud infrastructure providers can operate one or more data centers in geographic regions around the world. A "region" is a logical abstraction of a collection of computing, storage, and networking resources surrounding a data center in a given geographic area used to provide cloud computing infrastructure. Building a new region may include provisioning computing resources, configuring infrastructure, and deploying code to those resources (typically via network connectivity to the data center). However, building a region with physical resources located at the final destination data center site requires significant preparation work at the data center, which can complicate the logistics and scheduling of completing the region's construction. Summary of the Invention

[0011] Embodiments of this disclosure relate to the automated construction of regions using a prefabrication plant. A prefabrication plant can be a facility specifically designed for configuring computing devices, networking devices, and other physical resources for delivery to a destination site (e.g., a destination region—one or more data centers, customer facilities, etc. within a geographic area). Operations for constructing a region may include bootstrapping (e.g., provisioning and / or deployment) resources (e.g., infrastructure components, artifacts, etc.) so that the region can provide any suitable number of services upon delivery to the destination. Once the physical resources are configured at the prefabrication plant, they can be transported to the destination site, installed at the destination data center, and the final configuration and other software resources can be deployed to the physical resources. Resources used for bootstrapping (e.g., software artifacts, software images, etc.) can be provided in a boot environment within an existing region (e.g., one or more data centers in a host region). The host region can be selected based on network proximity to the prefabrication plant, and, complementaryly, the prefabrication plant can be located to have high-performance network connectivity with one or more host regions to support the boot environment. A build region can be orchestrated by one or more cloud-based services that manage the inventory of physical computing devices used to build the region in the prefabrication plant, generate and specify the configuration of the region to be built in the prefabrication plant, manage the bootstrapping of the region, configure the region for transfer to the destination site, and test and verify the physical resources after they have been installed at the destination site. Prefabricated regions can be built to meet specific customer configuration preferences (build-to-order) or according to a generic specification that is further customized during installation at a specific customer's site (build-to-stock).

[0012] One embodiment relates to a computer-implemented method for performing image-based deployment of a regional network at a prefabrication plant or at a destination site. The method may be performed by a manager service running on one or more computing devices of a distributed computing system hosting the regional network. The method may include deploying software resources to a first set of physical resources within a data center. The first set of physical resources may be characterized by a first network topology and a first hardware configuration. The method may further include generating an image set for the first set of physical resources. The image set may include a software image corresponding to each physical resource in the first set of physical resources, and each software image may include at least one software resource deployed to the first set of physical resources. The method may further include the manager service determining whether a second set of physical resources is compatible with the image set, and if so, deploying the image set to the second set of physical resources. The second set of physical resources may be characterized by a second network topology and a second hardware configuration. The method may further include the manager service configuring identifiers associated with the software resources of the image set deployed to the second set of physical resources.

[0013] Another embodiment relates to a distributed computing system including one or more processors and instructions that, when executed by the one or more processors, cause the distributed computing system to perform the methods described above.

[0014] Another embodiment relates to a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a distributed computing system, cause the distributed computing system to perform the methods described above. Furthermore, embodiments can be implemented using a computer program product comprising a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure. Attached Figure Description

[0015] To facilitate identification of any discussion of a particular element or action, the highest one or more bits in the reference number refer to the figure number in which the element was first introduced.

[0016] Figure 1 This is a block diagram illustrating a prefabrication plant for constructing a region and preparing regional computing devices for transmission to a target data center, according to at least one embodiment.

[0017] Figure 2 This is a block diagram illustrating a prefabrication plant connected to services provided by a CSP for building areas, according to at least one embodiment.

[0018] Figure 3 This is a diagram illustrating a network configuration for managing computing resources in an area being built in a prefabrication plant using a manager service and a network service, according to at least one embodiment.

[0019] Figure 4 This is a diagram illustrating the testing and evaluation of an area after delivery to a destination site using a manager service and a test service, according to at least one embodiment.

[0020] Figure 5 This is a block diagram illustrating the creation of an image set for a regional device according to at least one embodiment.

[0021] Figure 6 This is a block diagram illustrating an image provider for deploying image sets to regions in a data center, according to at least one embodiment.

[0022] Figure 7 This is a block diagram illustrating the deanonymization of a regional device image installed at a destination site according to at least one embodiment.

[0023] Figure 8This is a flowchart of an example process according to at least one embodiment for building an image set for a first set of physical resources and deploying the region set to a second set of physical resources.

[0024] Figure 9 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0025] Figure 10 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0026] Figure 11 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0027] Figure 12 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0028] Figure 13 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0029] Example of automated data center construction (regional construction) infrastructure

[0030] The adoption of cloud services has increased rapidly in recent years. Currently, various cloud service providers (CSPs) offer a wide range of cloud services. The term "cloud service" generally refers to a service or functionality provided by a CSP to a user or customer on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that constitute the CSP's infrastructure and are used to provide cloud services to customers are separate from the customer's own on-premises servers and systems. Therefore, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with simple, scalable, and on-demand access to applications and computing resources without requiring customers to invest in the infrastructure used to provide the service or functionality. Various types or models of cloud services can be provided, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc. Customers can subscribe to one or more cloud services provided by a CSP. Customers can be any entity, such as individuals, organizations, enterprises, government entities, etc.

[0031] As indicated above, the CSP is responsible for providing the infrastructure and resources used to deliver cloud services to subscribers. The resources provided by the CSP can include both hardware and software resources. These resources can include, for example, computing resources (e.g., virtual machines, containers, applications, processors, bare-metal computers), storage resources (e.g., databases, data repositories), networking resources (e.g., routers, host machines, load balancers), identity, and other resources. In some implementations, the resources provided by the CSP for delivering a set of cloud services are organized into a data center. The data center can be configured to provide a specific set of cloud services. The CSP is responsible for equipping the data center with the infrastructure and resources for delivering that specific set of cloud services. The CSP can build one or more data centers.

[0032] Data centers provided by a CSP can be hosted in different regions. A region is a local geographic area and can be identified by its name. Regions are typically independent of each other and can be geographically distant, such as spanning countries or even continents. Regions are grouped into domains. Examples of regions for a CSP could include the western United States, the eastern United States, eastern Australia, southeastern Australia, etc.

[0033] A region can include one or more data centers located within a specific geographic area corresponding to that region. As an example, a data center within a region could be located in a city within that region. For instance, for a specific CSP, a data center in the Western United States region could be located in San Jose, California; a data center in the Eastern United States region could be located in Ashburn, Virginia; a data center in the Eastern Australia region could be located in Sydney, Australia; a data center in the Southeastern Australia region could be located in Melbourne, Australia; and so on.

[0034] Data centers within a region can be organized into one or more availability domains for high availability and disaster recovery purposes. An availability domain can include one or more data centers within the region. Availability domains within a region are isolated from and fault-tolerant of each other, and are built in a way that makes it extremely unlikely for data centers in multiple availability domains to fail simultaneously. For example, availability domains within a region can be built in a way that makes it unlikely for a failure in one availability domain within the region to affect the availability of data centers in other availability domains within the same region.

[0035] When a customer or subscriber subscribes to or registers for one or more services provided by a CSP, the CSP creates a tenancy for the customer. A tenancy is like an account created for the customer. In some implementations, a customer's tenancy exists within a single domain and has access to all areas belonging to that domain. The customer's user can then access the services they have subscribed to under that tenancy.

[0036] As indicated above, CSPs build or deploy data centers to provide cloud services to their customers. As a CSP's customer base grows, it typically builds new data centers in new regions or increases the capacity of existing data centers to serve the increasing needs of its customers and better serve them. Preferably, the data center is built in a location geographically close to the location of the customers served by that data center. Geographical proximity between the data center and the customers served by that data center results in lower latency, leading to more efficient use of resources and faster, more reliable service to customers. Accordingly, CSPs typically build new data centers in new regions within geographically close to the customers served by the data center. For example, for a growing customer base in Germany, a CSP might build one or more data centers in new regions within Germany.

[0037] Building and configuring data centers (or multiple data centers) within a region to provide cloud services is sometimes referred to as building a region. The term "region building" is used to refer to constructing one or more data centers within a region. Building a region involves provisioning or creating a new set of resources that are required or used to provide a set of services that the data centers are configured to offer. The end result of the region building process is the creation of a region in which data centers, along with their contained hardware and software resources, are capable of providing a set of services intended for use in that region, and include a set of resources for providing that set of services.

[0038] Building new regions is a highly complex activity, requiring large-scale coordination among various bootstrapping activities. At a high level, this involves performing and coordinating a variety of tasks, such as: identifying a set of services to be provided by the data center; identifying the various resources required to provide that set of services; creating, provisioning, and deploying the identified resources; properly cabling the underlying hardware so that it can be used in the intended way; and so on. Each of these tasks further has sub-tasks that require coordination, which further increases the complexity. Due to this complexity, currently, region building involves several manually initiated or manually controlled tasks that require careful human coordination. As a result, the task of building new regions (i.e., building one or more data centers within a region and configuring the hardware and software in each data center to provide the necessary cloud services) is very time-consuming. Region building can take, for example, many months. Furthermore, this process is highly error-prone, sometimes requiring several iterations before the desired configuration of the region is achieved, which further increases the time spent building regions (e.g., deploying hardware and software resources). These limitations and problems severely restrict the ability of CSPs to increase computing resources in a timely manner in response to growing customer demand.

[0039] Recent innovations allow CSPs to shorten build times, reduce wasted computing resources, and mitigate risks associated with build regions. CSPs can use orchestration services to bootstrap services to new regions. Orchestration services can be cloud services hosted in regions separate from the target region (e.g., orchestration regions). To bootstrap services to the target region, the orchestration service can create a bootstrap environment to host instances of one or more cloud services. The orchestration service can then use the services in the bootstrap environment to support the deployment of services to the target region.

[0040] More recent innovations allow CSPs to centralize region building operations into one or more facilities that can act as “factories” to produce partially or fully configured physical infrastructure for subsequent delivery to destination sites. Instead of waiting for the construction of the target region data center and the installation of physical components (e.g., servers, network switches, power supplies, etc.) at the data center before services are bootstrapped to the target region, CSPs can build regions in a prefabrication plant, ship the configured physical components (such as racks) to the destination data center, and then finalize and verify the region’s components once the racks arrive at the destination site. A prefabrication plant can build multiple regions simultaneously. Each region built in a prefabrication plant can have a separate configuration, network topology, and services. By building regions in a prefabrication plant, the complexity of scheduling and logistics associated with preparing destination facilities, delivering physical components to destination facilities, and managing bootstrapping resources within cloud services can be significantly reduced, as regions can be built and maintained in advance until the destination site is ready.

[0041] Prefabrication plants can also be used to build computing components that are to be integrated into customers’ on-premises solutions, for example, when customers control and manage their own data center environments.

[0042] This disclosure relates to a prefabrication plant in which automated region building is performed using one or more prefabrication services. A prefabrication manager service can orchestrate the overall building of regions within the prefabrication plant. The manager service can work in conjunction with one or more additional prefabrication services to manage an inventory of physical components used to build regions in the prefabrication plant, configure networks (e.g., endpoints, network topology, addresses, and / or other identifiers for components within the region), bootstrap services onto the region infrastructure, prepare components for region transfer (including encrypting data volumes to provide security during transfer), verify the region after delivery and installation at the destination site, and finalize the region configuration, including performing any remaining bootstrap or update operations on services previously deployed to the region infrastructure in the prefabrication plant.

[0043] Specifically, prefabrication services can perform the deployment of software resources to physical components using device images. For example, a template regional network can be built, where individual infrastructure and software components are deployed as orchestrated by a manager service. Once built, each device in the template regional network can be imaged to create an image set corresponding to the network topology and hardware configuration of that template regional network. The manager service can orchestrate the creation of the image set and its deployment to a target regional network, which can be built after installation in a prefabrication facility or at the destination site data center. Advantageously, when the target hardware is compatible with the image set, deploying images to devices can occur significantly faster than deploying services individually to each node in the regional network. Accordingly, service providers supplying regional network construction operations according to the techniques described herein can even substantially reduce computing resource expenditures beyond the improvements gained from automating regional network construction operations.

[0044] Some definitions

[0045] A “region” is a logical abstraction corresponding to a collection of computing, storage, and networking resources associated with a geographical location. A region can include any suitable number of one or more execution targets. A region can be associated with one or more data centers. A “prefabricated region” describes a region built in a prefabricated factory environment before being delivered to a corresponding geographical location. In some embodiments, execution targets may correspond to a destination data center rather than a prefabricated factory data center.

[0046] An "execution target" is the smallest unit of change used to perform a release. A "release" is an expression of intent to orchestrate specific changes to a service (e.g., deploying version 8, "adding internal DNS records," etc.). For most services, an execution target represents an "instance" of the service or an instance of changes to be applied to the service. A single service can be directed to each of one or more execution targets. Execution targets can be associated with a set of devices (e.g., a data center).

[0047] "Guidance" for a single service refers to the collective task associated with the provisioning and deployment of any appropriate number of resources (e.g., infrastructure components, artifacts, etc.) corresponding to that single service. "Guidance area" refers to the collective task associated with each service to be included in the guidance within that area.

[0048] A "service" refers to functionality provided by a set of resources, typically in the form of an API that a client can invoke to achieve some useful result. The set of resources for a service includes any suitable combination of infrastructure, platforms, or software (e.g., applications) hosted by a cloud provider and configured to provide the functionality. Services can be made available to users via the Internet.

[0049] "Artifacts" refer to code deployed to infrastructure components or Kubernetes engine clusters, which can include software (e.g., applications), configuration information for infrastructure components (e.g., configuration files), credentials, etc.

[0050] IaaS provisioning (or "provisioning") refers to acquiring a computer or virtual host for use, and even installing necessary libraries or services on it. The phrase "provisioning device" refers to the state in which a device is evolved to the point where an end user can use it for their specific purpose. A device that has undergone provisioning can be called a "provisioned device." Preparing a provisioned device (installing libraries and daemons) can be part of provisioning; this preparation is different from deploying a new application or a new version of an application to a prepared device. In most cases, deployment does not include provisioning, and provisioning may need to be performed first. Once ready, the device can be called an "infrastructure component."

[0051] IaaS deployment (or "deployment") refers to the process of providing and / or installing new applications or new versions of applications on provisioned infrastructure components. Once the infrastructure components have been provisioned (e.g., acquired, assigned, prepared, etc.), additional software can be deployed (e.g., provided to and installed on the infrastructure components). After provisioning and deployment are complete, the infrastructure components may be referred to as "resources" or "software resources." Examples of resources may include, but are not limited to, virtual machines, databases, object storage, block storage, load balancers, etc.

[0052] A Virtual Boot Environment (ViBE) is a virtual cloud network provisioned within the coverage of an existing zone (e.g., a "host zone"). Once provisioned, the ViBE connects to the new zone using a communication channel (e.g., an IPSec tunnel VPN). Certain essential core services (or "seed" services), such as deployment orchestrators, public key infrastructure (PKI) services, dynamic host configuration protocol (DHCP) services, and domain name services (DNS), can be provisioned in the ViBE. These services provide the capabilities needed to bring hardware online, establish a chain of trust to the new zone, and deploy the remaining services in the new zone. Utilizing a Virtual Boot Environment can prevent circular dependencies between boot resources by leveraging the resources of the host zone. These services can be staged and tested in the ViBE before a pre-provisioned zone (e.g., a target zone) becomes available.

[0053] "Manager service" can refer to a service configured to manage the provisioning and deployment operations of any appropriate number of services as part of a prefabricated region build. The manager service can be used in conjunction with one or more additional prefabrication services to orchestrate region builds in a prefabrication plant and to manage how prefabricated regions are installed and configured at their destination data center after they have been built and shipped. The manager service and other prefabrication services can be hosted in an existing region of the CSP.

[0054] "Host Zone" refers to the zone that hosts the Virtual Boot Environment (ViBE). The host zone can be used to boot the ViBE.

[0055] A "target area" refers to the area being built in the prefabrication plant. During the construction of the prefabricated area, the target area is associated with the physical space, power, and cooling provided by the prefabrication plant. After guidance, once the prefabricated area has been transported to its destination data center, it becomes associated with the destination data center where it is installed.

[0056] Prefabricated area construction

[0057] In some examples, this paper describes technologies for building regions at a prefabrication plant. As briefly described above, such technologies may include one or more prefabrication services (e.g., manager services, network services, inventory services, testing services, deployment orchestration systems) hosted by a CSP, which can manage and bootstrap infrastructure components (e.g., provisioning and deployment software) for one or more regions within the prefabrication plant. The prefabrication plant can be configured to support multiple region builds simultaneously. For example, the physical resources (e.g., server racks, network switches, etc.) of a first prefabrication region may be installed at one location within the prefabrication plant, while the physical resources of a second prefabrication region may be installed at a second location within the prefabrication plant. Each prefabrication region can connect to a dedicated network fabric within the prefabrication plant to provide independent networking connectivity to each region, enabling each region to communicate with prefabrication services and / or other cloud services to support region builds. Based on the build request (regional specifications, such as the number of server racks for the region, the number of computing devices, the number and types of services to be hosted in the region, the region's network topology, etc.), the prefabrication service can generate instructions to install (e.g., by factory personnel) the corresponding physical infrastructure in the prefabrication plant. This can include networking physical devices together on their racks, positioning the racks at locations within the prefabrication plant, and establishing a static network structure for connecting devices to the prefabrication plant. The manager service can then orchestrate the provisioning of the regional infrastructure and software resources to the deployment of the prefabricated regional infrastructure, configure the prefabricated regional infrastructure for transport, manage (e.g., schedule and monitor) the transports within the prefabricated region, and perform testing and verification of the prefabricated region once it reaches its destination site.

[0058] Prefabrication plants can centralize region building processes for more efficient use of compute and networking resources that support region building. For example, a prefabrication plant can be located “proximity” (e.g., with low-latency and high-data-rate networking connectivity) to a host region that includes prefabrication services and / or ViBE. Multiple regions can be built using the improved performance of network connectivity to the host region, thus avoiding potentially poor performance when performing region building on newly constructed data center sites for typical region building. Because the CSP can control the prefabrication plant and its network connectivity, the prefabrication plant also provides improved physical and compute security for equipment during region building.

[0059] Furthermore, the prefabrication plant improves the management of physical component inventory. The manager service can determine which computing devices are needed for building a specific region, and these devices can be stored at or near the prefabrication plant. As regions are built and shipped, the infrastructure for new regions can be quickly moved to the prefabrication plant and installed, thus improving efficiency.

[0060] Now turn to the attached diagram. Figure 1 This is a block diagram illustrating a prefabrication system 100 according to at least one embodiment. The prefabrication system 100 includes a prefabrication plant 102 for constructing regions (e.g., prefabricated region 106A, prefabricated region 106B, prefabricated region 106C) and preparing region computing devices for delivery to a target data center (e.g., data center 108, data center 110). Each region constructed in the prefabrication plant 102 may include one or more devices forming the computing environment of the data center. The prefabrication plant 102 can be used to construct multiple regions simultaneously. For example, the prefabrication plant 102 can construct all of prefabricated regions 106A, 106B, and 106C simultaneously. In some examples, the devices of the regions can be installed and temporarily stored in the prefabrication plant 102 prior to the commencement of infrastructure provisioning and software deployment operations.

[0061] Prefabrication plant 102 can be a data center-like facility, including sufficient power, cooling, and networking infrastructure to support the construction of one or more regions. Prefabrication plant 102 can be located near the existing computing infrastructure of a CSP (e.g., CSP 104). For example, CSP 104 can operate an existing data center for one or more regions. Prefabrication plant 102 can be located near, or even adjacent to, the existing data center in the host region to provide high-data-rate network connectivity between the CSP's cloud services and the computing equipment in the regions being built within prefabrication plant 102. Additionally or alternatively, prefabrication plant 102 can be positioned to improve logistics operations, including delivery from regions to destination data centers.

[0062] The prefabricated areas constructed in prefabrication plant 102 can include any suitable number of physical resources, including computing devices (e.g., servers, multi-server racks, etc.), storage devices (e.g., block storage devices, object storage devices, etc.), networking devices (e.g., switches, routers, gateways, etc.). Each area may have different physical resources depending on the specific requirements of the destination area and data center. For example, prefabrication area 106A may include 100 racks, each with 40 computing devices, while prefabrication area 106B may include 20 racks, each with 30 computing devices. Each rack of computing devices may include one or more networking devices that are communicatively connected to the server devices on the rack and configured to connect to the networking infrastructure of prefabrication plant 102 to form a network with other computing devices in the prefabrication area. Each rack may also include power and cooling equipment to support the operation of the computing devices on the rack.

[0063] Prefabrication plant 102 may include any suitable number of networking devices to support the installation and connection of one or more computing devices in the prefabricated area being built. For example, prefabrication plant 102 may include any suitable number of leaf-ridge switches to support the connection of computing devices on multiple racks to form a network for the prefabricated area. Similarly, prefabrication plant 102 may include network cables installed in the facility that provide network connectivity to the networking infrastructure of prefabrication plant 102. Network cables may be positioned to terminate at locations within prefabrication plant 102 where racks for installing computing devices for the prefabricated area can be installed during the area building operation. Additional details regarding the networking infrastructure and configuration of the prefabrication plant are provided below. Figures 9 to 11 supply.

[0064] Prefabrication plant 102 can connect to services provided by CSP 104 via one or more networks. During the region build operation, CSP 104 can provision infrastructure components on the physical resources of the prefabrication region and deploy software resources, configurations, and / or other artifacts to the supplied infrastructure components. For example, CSP 104 can provision computing devices in prefabrication region 106A to host one or more virtual machines, provide hostnames, network addresses, and other network configurations for the supplied physical and virtual devices, and then deploy one or more services to execute on the supplied infrastructure. This can bring the prefabrication region to a state close to the final production state of the equipment when it is installed at the destination facility.

[0065] Once the prefabrication area has been constructed, physical resources can be configured for transfer / transportation to the destination facility. As used herein, the term "transfer" can be used synonymously with the term "transportation" in the context of moving physical resources associated with a prefabrication area from the prefabrication plant to the destination site. Configuring a prefabrication area for transfer may include obtaining a "snapshot" of the current network configuration of the computing devices in the prefabrication area, storing the snapshot, providing each computing device with a snapshot portion including the identifiers of each device and its neighboring devices within the network, encrypting the data volumes of the computing devices, and configuring the devices to boot into a test state upon power-up after the transfer. In addition to network snapshots, CSP 104's prefabrication service can also capture device snapshots, which are disk images of the fully configured individual switches, computing devices, and smart NICs in the various racks to be transported to the destination site. Device snapshots enable rapid replacement of any device in the transported rack if it is non-functional upon arrival and must be replaced. Transport to the destination facility can be achieved through one or more methods, including by truck 112 or by airplane 114. For example, prefabricated area 106B can be configured to be delivered to data center 108 by truck 112, while prefabricated area 106C can be configured to be delivered to data center 110 by aircraft 114.

[0066] Once the computing equipment in the prefabricated area arrives at the destination facility, it can be installed there according to the facility's configuration. The destination facility may be a data center already built to host the prefabricated area equipment, with networking, power, cooling, and other infrastructure provided according to the prefabricated area's configuration. The data center may have a network connection to CSP 104. The installation of the prefabricated area may include manual operations and other related tasks for connecting racks and their computing equipment to the data center's network infrastructure. Once the physical connection is complete, the equipment in the prefabricated area can be powered on, which may initiate one or more test operations based on the configuration performed at the prefabrication plant 102 prior to transmission. The prefabricated area may also be connected to CSP 104 via one or more network connections to the data center to communicate with the prefabrication service. For example, prefabricated area 106B may be connected to CSP 104 via connection 118, while prefabricated area 106C may be connected to CSP 104 via connection 116. The prefabrication service may deploy the final configuration for the installed equipment, deploy updates to software resources on the installed equipment, and perform additional testing and verification operations on the prefabricated area at the destination data center.

[0067] Figure 2 This is a block diagram illustrating a prefabrication system 200 according to at least one embodiment, the prefabrication system 200 including a prefabrication plant 202 connected to a prefabrication service 210 for building areas provided by a CSP 204. The prefabrication plant 202 may be... Figure 1 An example of prefabrication plant 102, and CSP 204 can be Figure 1 An example of CSP 104. Prefabrication plant 202 may interface with CSP 204 via network 208, which may be a public network (such as the Internet), a private network, or another network. Prefabrication service 210 may include manager service 212, inventory service 214, testing service 216, orchestration service 218, and network service 220. Prefabrication service 210 may perform operations corresponding to building prefabricated region 206 in prefabrication plant 202, including managing boot environments (e.g., ViBE 222), supplying infrastructure components in prefabricated region 206, deploying software resources to prefabricated region 206, configuring the network of prefabricated region 206, testing the prefabricated region at various points during the build process, and managing the physical inventory (e.g., physical inventory 224) of computing devices used to build prefabricated region 206 and other prefabricated regions being built in prefabrication plant 202.

[0068] Manager service 212 can perform tasks for coordinating the operations of prefabrication service 210, including scheduling prefabrication area construction operations of other prefabrication services 210, generating physical build requests and corresponding instructions, initiating the delivery of prefabrication area 206 to the destination site, and managing the provisioning and deployment of resources in prefabrication area 206 at both the prefabrication plant 202 and the destination site. Physical build requests can specify the quantity and type of physical resources to be used in prefabrication area 206. Physical build requests can also include a set of instructions for personnel to install the corresponding physical resources in prefabrication plant 202. For example, manager service 212 can generate a physical build request specifying: the number of racks and server devices for prefabrication area 206, the number of networking devices that can be used to connect the server devices to form a network for prefabrication area 206, and determining a connection plan for the specified server devices, networking devices, and the existing networking infrastructure of prefabrication plant 202. The physical build request may also include instructions to enable personnel to obtain physical equipment from an associated location (e.g., physical inventory 224), and instructions to install the equipment at a specified location in the prefabrication plant 202. In some embodiments, the operation of the physical build request may be performed by an automated system under the control of the manager service 212. For example, obtaining racks of server equipment from physical inventory 224 and installing the racks at the prefabrication plant 202 may be performed by a robotic system configured to move physical racks from the site to the site.

[0069] Inventory service 214 can be configured to track and monitor physical devices corresponding to one or more regions (e.g., one or more data centers within a region). Inventory service 214 can also track physical devices in one or more prefabrication zones (e.g., prefabrication zone 206) within prefabrication plant 202. Tracking and monitoring physical devices may include maintaining an inventory of devices based on device identifiers (e.g., serial numbers, device names, etc.) and the association of devices with data centers. Inventory service 214 can provide inventory information to other prefabrication services 210 (including manager service 212) for use in the prefabrication zone construction process. For example, inventory service 214 can determine whether a physical device is located at prefabrication plant 202 or at a destination site. Inventory service 214 can query devices via a network (e.g., network 208) to determine their location and / or association with a region, prefabrication zone, or data center. Inventory service 214 can also maintain a physical inventory (e.g., physical inventory 224) of devices stored for use in the prefabrication zone construction operation. For example, inventory service 214 can track physical devices as they are received at physical inventory 224 and then retrieved from physical inventory 224 for use as part of a prefabrication area at prefabrication plant 202. In some examples, inventory service 214 can provide inventory information to manager service 212, which can be used to generate a physical build request for prefabrication area 206, including instructions for obtaining physical resources from physical inventory 224 and installing the physical resources at prefabrication plant 202.

[0070] Physical inventory 224 may be a warehouse or storage facility for storing physical resources (e.g., computing devices) used in prefabrication area construction operations. Physical inventory 224 may be located near prefabrication plant 202 to facilitate the retrieval of physical resources based on physical construction requests. For example, physical inventory 224 may be a building adjacent to the building used for prefabrication plant 202. In some examples, physical inventory 224 may be located within prefabrication plant 202. Personnel associated with the CSP and prefabrication plant 202 may place physical resources into and retrieve them from physical inventory 224. In some cases, during prefabrication area construction operations, physical resources may be retrieved and installed from physical inventory 224 using instructions provided by the physical construction request via robots, automated guided vehicles, or other similar autonomous or semi-autonomous systems.

[0071] Orchestration service 218 can be configured to perform bootstrapping operations to provision infrastructure components and deploy software resources to prefabrication zone 206. Orchestration service 218 can also construct boot environments (e.g., ViBE 222) for use when bootstrapping resources to prefabrication zone 206. Orchestration service 218 can be an example of the deployment orchestrator described above. In some examples, orchestration service 218 can be configured to boot (e.g., provision and deploy) services to prefabrication zones (e.g., prefabrication zone 206) based on predefined profiles that identify resources (e.g., infrastructure components and software to be deployed) used to implement a given change to the prefabrication zone. Orchestration service 218 can parse and analyze the profiles to identify dependencies between resources. Orchestration service 218 can generate specific data structures based on the analysis and can use these data structures to drive operations and manage the order in which services are bootstrapped to zones. The orchestration service 218 can use these data structures to identify when it can bootstrap services, when bootstrap is blocked, and / or when bootstrap operations associated with previously blocked services can be resumed.

[0072] In some embodiments, orchestration service 218 may include components configured to perform bootstrapping tasks associated with a single service in a prefabricated region. Orchestration service 218 may maintain current state data indicating any suitable aspects of the current state of resources associated with the service. In some embodiments, desired state data may include a configuration of the desired state of resources associated with the service, declared (e.g., via declarative statements). In some embodiments, orchestration service 218 may identify changes required for one or more resources by comparing desired state data with current state data. For example, orchestration service 218 may determine any suitable changes required to resources of the service to procure one or more infrastructure components, one or more deployed artifacts, or to bring the state of resources of the service into alignment with the desired state. Specific details regarding particular implementations of orchestration service 218 are provided in U.S. Patent Application No. 17 / 016,754, entitled “Techniques for Deploying Infrastructure Resources with a Declarative Provisioning Tool,” the entire contents of which are incorporated herein by reference for all purposes.

[0073] ViBE 222 can be an example of a bootstrap environment that can be used to deploy resources to prefabrication areas within prefabrication plant 202. ViBE can include a virtual cloud network (e.g., a network of cloud resources) implemented within a suitable area of ​​a CSP (e.g., CSP 204). ViBE can have one or more nodes (e.g., compute nodes, storage nodes, load balancers, etc.) to support the operation of services hosted by orchestration service 218. ViBE services can then be used to support the deployment of services to prefabrication area 206. For example, orchestration service 218 can deploy instances of one or more component services of orchestration service 218 to a bootstrap environment (e.g., an instance of orchestration service 218), which can then be used to deploy resources from ViBE 222 to prefabrication area 206. Because ViBE is implemented as a virtual cloud network within an existing area, any suitable amount of regional infrastructure can be provisioned to support services deployed within ViBE (compared to the fixed hardware resources of a seed server). In addition to deploying software resources to ViBE 222, orchestration service 218 can also be configured to supply infrastructure resources (e.g., virtual machines, compute instances, storage devices, etc.) to ViBE 222. ViBE 222 can simultaneously support boot operations for more than one prefabrication zone in prefabrication plant 202.

[0074] When prefab zone 206 is available to support bootstrapping operations, ViBE 222 can connect to prefab zone 206, allowing services in ViBE 222 to interact with services and / or infrastructure components in prefab zone 206. This enables the deployment of production-grade services, rather than deploying self-contained seed services as in previous systems, and will require internet connectivity to the target zone. Typically, seed services are deployed as part of a collection of containers and used to bootstrap the dependencies required to build the zone. Using the existing zone's infrastructure / toolchain, resources can be bootstrapped into ViBE 222 and connected to prefab zone 206 to provision hardware and deploy services until prefab zone 206 becomes self-sufficient (e.g., self-sufficient with respect to services hosted within prefab zone 206). Utilizing ViBE 222 allows the establishment of the dependencies and services needed to provision / prepare infrastructure and deploy software, while using resources from the host zone to break circular dependencies in core services.

[0075] Test service 216 can be configured to perform one or more test or verification operations on prefabricated region 206 after resource provisioning and / or deployment. Test operations can be part of user-accepted testing used to determine whether the behavior of the prefabricated region conforms to the build specifications. For example, test service 216 can perform tests that interact with instances of services deployed to prefabricated region 206 to verify the expected operation of the queried service. As another example, test service 216 can perform networking tests to obtain the hostname, networking address, and / or other identifiers of components in prefabricated region 206 for comparison with the expected identifiers of components as specified in the build request or other specifications for prefabricated region 206. Test service 216 can perform test operations both during the prefabricated region build process at prefabricated plant 202 and after prefabricated region 206 is delivered to the destination site. Test operations performed at prefabricated plant 202 can be the same as or different from test operations performed after prefabricated region 206 is delivered to the destination site.

[0076] Manager service 212 can obtain inventory information from inventory service 214 for use when generating physical build requests. For example, manager service 212 can use the inventory information to determine which physical resources to install in the prefabrication plant 202 for the prefabrication area corresponding to the physical build request.

[0077] Figure 3 This is a diagram illustrating a CSP system 300, according to at least one embodiment, for managing the network configuration of computing resources for a prefabrication zone 330 under construction in a prefabrication plant 302 using a manager service 312 and a network service 320. The prefabrication plant 302 and prefabrication zone 330 may be examples of other prefabrication plants and prefabrication zones described herein, including... Figure 2 The prefabrication plant 202 and prefabrication area 206. Prefabrication services 310 can be provided by CSP and can be as described above. Figure 2 Examples of prefabricated services 210 described include, as Figure 2 The example of Manager Service 212 and Manager Service 312 as Figure 2 Example of network service 220 is network service 320.

[0078] As mentioned above Figure 2As described, the manager service 312 can perform tasks for coordinating the operations of the prefabrication service 310, including scheduling prefabrication zone construction operations performed by other prefabrication services 310, generating physical construction requests and corresponding instructions, and configuring prefabrication zone 206 for delivery to the destination site. The physical construction request can specify the quantity and type of physical resources to be used in prefabrication zone 206. The network service 320 can use configuration information from the construction request to determine the network topology of devices (e.g., servers, networking devices, racks of servers and networking devices, etc.). After the provisioning of infrastructure components in prefabrication zone 330, the network service 320 can also determine the network configuration of the devices in prefabrication zone 330.

[0079] In some examples, network service 320 may store a snapshot of the network configuration of a prefabricated area (e.g., prefabricated area 330). The snapshot may include information about the network topology of the prefabricated area at a specific point in time, including network identifiers of devices in the prefabricated area (e.g., network addresses, hostnames, etc.), current network connections between devices, physical networking interfaces between devices and the networking infrastructure 338 of the prefabricated plant 302, and network settings for the devices (e.g., port configurations, gateway configurations, etc.). As an example, server device 336 may be a computing device in server rack 332A of prefabricated area 330. Server device 336 may have a networking connection 340 to a switch 334 of server rack 332. Thus, the network configuration of prefabricated area 330 may include information associating server device 336 with switch 334, including specifying the type of network connection 340, the port of switch 334 to which server device 336 is connected, and the settings of server device 336 and switch 334 corresponding to the networking connection 340 between server device 336 and switch 334. Furthermore, the network configuration may include information associating server device 336 with “neighboring” devices in prefabricated area 330 that have networking connections 342, 344 between them. Network connections 342 and 344 may be via switch 334 so that server device 336 can communicate with other devices in server rack 332A via network connections 342, 344. In some examples, “neighboring” devices for a given device in prefabricated area 330 may include each computing device on the same server rack. Additionally, switch 334 may have network connections to one or more other switches within prefabricated area 330 (e.g., network connection 346 to a switch in server rack 332B).

[0080] Network snapshots can be used to verify the physical installation (e.g., physical networking connectivity) of prefabricated area 330 after devices are installed at the destination site. For example, network service 320 can provide a network snapshot (or a portion of a snapshot) to each device in prefabricated area 330 as part of configuring prefabricated area 330 for transport to the destination site. For example, network service 320 can provide network snapshot 326 to server device 336 for storage at server device 336. Network snapshot 326 can be a portion of a network snapshot corresponding to the network configuration of the entire prefabricated area 330. Network snapshot 326 may include an identifier of server device 336 (e.g., network address, hostname, etc.) and information associating server device 336 with one or more other devices in prefabricated area 330. Information associating server device 336 with neighboring devices may include identifiers of the neighboring devices and information about network connections between them. For example, server device 336 can use network snapshot 326 to identify neighboring devices and communicate with them via network connections.

[0081] Network service 320 can also maintain the network configuration of the network structure of prefabrication plant 302. For example, prefabrication plant 302 may have networking infrastructure to support the simultaneous construction of multiple separate prefabrication areas. Prefabrication plant 302 may have multiple dedicated locations for placing server racks for the prefabrication areas being constructed. Each location may have a set of networking cables for the networking infrastructure, which terminates at a location where a server rack can be connected. Based on the equipment placed at that location, specific cables from that set of networking cables can be connected to the equipment (e.g., to a switch at the top of the rack) to connect the equipment to other equipment in the prefabrication area using a portion of the network structure of prefabrication plant 302. For example, server rack 332A may be placed at a location within prefabrication plant 302 and connected to networking infrastructure 338 using switch 334, while server rack 332B may be placed at a second location and connected to networking infrastructure 338.

[0082] In addition to operations for storing the network configuration of prefabricated area 330, configuring prefabricated area 330 for transport to the destination site may also include manager service 312 configuring each device to enter a test state during subsequent power-on of the device, encrypting the device's data volume with an encryption key, storing the encryption key at a device that can act as a key server for prefabricated area 330 during initialization at the destination site, and configuring one of these devices to act as a Dynamic Host Configuration Protocol (DHCP) server during the initialization of prefabricated area 330 at the destination site. Manager service 312 may also generate instructions for use by personnel or robotic systems associated with prefabrication plant 302 for packaging devices for transport. Manager service 312 may also generate instructions for use by personnel associated with the destination facility for installing and connecting devices at the destination facility.

[0083] In some embodiments, the device configuring prefabrication area 330 may also include operations for capturing device snapshots of each device. Device snapshots may include software images of one or more disk drives or other storage of a computing device, which can be used to copy the device's software configuration to a replacement device. Manager service 312 may generate device snapshots in conjunction with one or more of the prefabrication services 310. Device snapshots may be stored in a database or data repository (e.g., one or more snapshots 324) along with network snapshots(s). As a specific example, manager service 312 may generate device snapshot 352 of server device 350 of prefabrication area 330 at prefabrication plant 302. Device snapshot 352 may be used to image another physical device with the same or similar physical configuration as server device 350 to create a copy of the server device in the event of failure of server device 350 (e.g., damage or loss during transport to the destination site).

[0084] Figure 4 This is a diagram illustrating a CSP system 400, according to at least one embodiment, for testing and evaluating a prefabricated area 330 after delivery to a destination site 402 using a manager service 412 and a test service 416. The destination site 402 may be a data center facility located corresponding to a new area deployed for the CSP using the computing resources of the prefabricated area 430. The prefabrication service 410 may be provided by the CSP and may be similar to... Figure 2 The pre-built service 210 includes a manager service 412 as an example of a manager service 212, a test service 416 as an example of a test service 216, and a pre-built service 416 as an example of a test service 216. Figure 2 The example of orchestration service 218 is orchestration service 418.

[0085] Transporting the prefabricated area 330 to the destination site 402 may include powering off each device, disconnecting the devices from the prefabrication plant's networking infrastructure, and appropriately packaging the devices for transport as needed. Server racks (e.g., server racks 332A, 332B) can be transported intact without disconnecting the individual devices within the racks. Once delivered to the destination site 402, the server racks can be placed in the destination site 402 according to the resulting data center's physical layout and connected to the destination site's networking infrastructure 438. For example, a networking connection can be established between the networking infrastructure 438 and the switches of server racks 332A, 332B by connecting one or more networking cables to a switch (e.g., switch 334).

[0086] As described above, the devices in prefabrication zone 330 may have been configured to boot into test mode upon first power-on at destination site 402. In some embodiments, the devices may have a dedicated boot volume to support test mode during initialization at destination site 402. In other embodiments, the boot volume may be configured on an external device connected to each device in prefabrication zone 330. For example, each server device (e.g., server device 336) may be connected to a SmartNIC that provides a low-overhead boot volume that can be used to boot the server device into test mode. Because the boot volume may be used solely to support test mode, the data on the boot volume may not need to be encrypted like the data volume on the server device.

[0087] The test mode can be configured to allow each computing device to verify its connectivity with other devices in prefabricated area 330. This verification determines whether the physical network connection between the device and the networking infrastructure 438 at destination site 402 is correctly established. To verify the connection, devices in test mode can use stored network configurations or network services (e.g., Figure 3 Network service 320) identifies and stores a portion of the network configuration at each device. For example, server device 336 can use network snapshot 326 to identify nearby computing devices that are communicatively connected to server device 336 via network connection 342. To verify network connection 342, server device 336 can send a verification request to the nearby computing device. If network connection 342 is intact, the server device can receive a verification indication from the nearby computing device indicating that the verification request was successfully received at the nearby computing device. Server device 336 can verify all connections specified in network snapshot 326. Similarly, devices on a server rack (e.g., server rack 332A) can verify connections to every other server rack (e.g., server rack 332B) in prefabricated area 330.

[0088] In some embodiments, a device in prefabricated area 330 may be configured to act as a DHCP server (e.g., DHCP server 446). DHCP server 446 may provide network addresses or other identifiers to devices in prefabricated area 330 during initialization. For example, during test mode, each device may verify its connection to DHCP server 446 and then receive an address, identifier, or other network configuration information from DHCP server 446. The device may compare the received identifier with an identifier included in the network configuration generated by a network service during the prefabricated area construction operation at the prefabrication plant. For example, server device 336 may receive an identifier from DHCP server 446 and then compare the received identifier with an identifier in network snapshot 326. Because prefabricated area 330 should not undergo any component changes during transport, the network configuration of prefabricated area 330 at destination site 402 should remain unchanged, including the configuration information from DHCP server 446. That is, the server device in the prefabricated area should receive the same network address from DHCP server 446 after the device is installed at destination site 402. If the network configuration changes, the server device may indicate that the network configuration of prefabricated area 330 may be incorrect.

[0089] In some embodiments, if any device is damaged during transport and no longer functions properly, an operator at the destination site can replace the damaged device with a new replacement device and configure the new device using a snapshot of the device taken before transport, thus allowing successful verification after on-site installation even in the event of hardware failure during transport. For example, server device 350 may be damaged during transport to destination site 402. A non-functional state of server device 350 may be discovered during test operations used to verify the network configuration of prefabrication area 330. To restore functionality, manager service 412 can generate instructions to replace server device 350 with the same physical device located at the same location on server rack 332B. Once the replacement device is installed, manager service 412 can deploy device snapshot 352, generated during the prefabrication area build operation in prefabrication plant 302. Deploying device snapshot 352 may include imagerizing one or more disk drives or other storage devices of the replacement server device to make the replacement server device have the same software configuration as server device 350 in prefabrication area 330 before transport to destination site 402. Other devices, including networking devices such as switch 334, can be similarly replaced and restored using captured device snapshots.

[0090] DHCP server 446 can perform test mode verification operations similar to those of other devices within prefabricated area 330. If DHCP server 446 can successfully verify the network connection between itself and a neighboring device, then DHCP server 446 can exit test mode and begin operating as a DHCP server for other devices within prefabricated area 330. In some embodiments, DHCP server 446 can complete its test mode verification operation before other devices in prefabricated area 330 complete their own test mode verification operations. For example, server device 336 can boot into test mode and attempt to verify its network connection with DHCP server 446 before verifying its own network connection 342 or network connection 344 with a neighboring computing device. DHCP server 446 may not send a verification instruction to server device 336 until DHCP server 446 has completed its own test mode verification operation. Server device 336 can then wait a predetermined amount of time and retry the verification request to DHCP server 446. Similarly, other computing devices performing test mode verification operations can wait and retry verification requests until DHCP server 446 is operational.

[0091] As described above, the data volumes of devices in prefabricated area 330 can be encrypted before being transported to destination site 402. The encryption key used to encrypt the data volume of each device can be associated with that specific device. Encryption key 444 can be stored at one of the computing devices in prefabricated area 330 configured to act as a key server for prefabricated area 330 during initialization (e.g., stored at key server 442). Encryption key 444 itself can be encrypted by a master key. In some embodiments, encryption key 444 can be protected by a hardware security module (e.g., a Trusted Platform Module (TPM)). This hardware security module can be part of key server 442 or part of another device connected to key server 442 (e.g., a SmartNIC, an external security device, etc.). In some embodiments, the master key or external security device can be delivered separately from prefabricated area 330 to destination site 402 (e.g., by an operator) and provided to key server 442 or installed at key server 442 as part of the installation operation for prefabricated area 330. Key server 442 can perform test mode verification operations similar to those performed on other computing devices in prefabricated area 330. If the test mode verification operation completes successfully, the key server 442 can begin providing encryption key 444 to other computing devices in the prefabricated area to decrypt the data volume. For example, the key server 442 can receive a key request from server device 336. In response, the key server 442 can decrypt the data volume storing encryption key 444 (e.g., via a master key, via a hardware security module), retrieve the encryption key corresponding to server device 336, and send that encryption key to server device 336.

[0092] Once prefabricated region 330 has been installed and initialized at destination site 402 (e.g., the device boots into normal operating mode, the data volume is decrypted, and services deployed during the prefabricated region construction operation at the prefabrication plant are executing), test service 416 can perform one or more acceptance tests. Acceptance tests may include verifying that all services are functioning as expected. For example, test service 416 may interact with the services executing at prefabricated region 330 to verify that the service is operating according to the requirements defined for the acceptance test. Test service 416 may provide the results of the acceptance tests to manager service 412, indicating that the prefabricated region construction is complete.

[0093] During the transport of prefabricated area 330 to destination site 402, updates or other changes can be specified for one or more infrastructure components and / or software resources already supplied and / or deployed to prefabricated area 330 at the prefabrication plant. For example, services may be updated to a newer version during transport. Before the prefabrication area construction operation is completed, orchestration service 418 can deploy the updated software resources to prefabricated area 330 at destination site 402. Deploying updated software resources can occur similarly to deploying software resources to prefabricated area 330 in the prefabrication plant.

[0094] Image-based region building

[0095] Using a prefabrication plant to provision infrastructure components and deploy software resources within a regional network allows for a high degree of flexibility and customization in building each individual regional network. In particular, the deployment of orchestration services, virtual machines, databases, object storage devices, compute instances, load balancers, etc., using Manager Services ensures correct dependencies and networking integrity of resources deployed when a region is built. In some cases, deploying software resources in an image-based manner may be desirable. Regions built at the prefabrication plant can have similar or identical physical resources (e.g., server devices, network devices, etc.) from one region to the next, making it possible to deploy a single software image to a physical device that saves significant time and computational resources compared to deploying and configuring software resources individually to each physical resource. For example, a region may include a predefined arrangement of server racks, each rack having a predefined number of server devices with identical configurations of processing power, memory, storage devices, etc. Therefore, when building a new region in the prefabrication plant, capturing a software image for each physical device and then deploying those images to similar devices can be more efficient than deploying software resources individually to each device.

[0096] As used herein, the term "image" refers to a collection of software and firmware resources corresponding to a physical computing device (e.g., a server device, a networking device, etc.). For example, an image of a server device may include an operating system, deployed applications, and configuration data stored in the server device's persistent storage. Therefore, an image may also include data on the state of virtual machines hosted by the server device and persisted in the server device's storage at the time the image is captured. An image may also include the server device's networking configuration and identifiers for the server device and its software resources (e.g., hostname, network address, Domain Name System (DNS) name, certificates, etc.).

[0097] Software resources deployed to a prefabrication zone during prefabrication operations in a prefabrication plant can include several services (e.g., storage, compute, networking, etc.), depending on the service architecture, each of which can consist of smaller microservices. The interdependencies of various services and microservices on other services hosted within the zone can complicate naive image-based deployments. For example, a software resource for a service might be hosted on one device within the zone but depends on another software resource for a different service hosted on a second device within the zone. Proper networking configuration between the one device and the second device allows the services to communicate and operate correctly. However, without properly considering the network topology of the zone where the images are created or the network topology of a new zone being built in an image-based manner, deploying an image of each device to two other devices in the new zone may not recreate the correct networking relationship between the two devices. The techniques described in this disclosure can build a set of images of physical resources that can be deployed to the zone being built while preserving the dependencies between services and software resources on each device.

[0098] The manager service can be configured to create image sets from regions. For example, template regions can be built in a prefabrication plant using the techniques described above. Once infrastructure components and software resources are provisioned and deployed to the template regions, the manager service can obtain images of each device in the template regions, as well as the network topology of the network regions. The network topology used for the template regions can specify the arrangement of server devices in server racks, identify the networking devices in each server rack, and specify the networking connections between each server rack. For example, the network topology can identify the fourth server device on server rack two, its connection to the top rack switch of server rack two, and the connections of the top rack switch to other networking infrastructure within the template regions. Using the network topology, the image of each device can be deployed to the region being built according to the topology.

[0099] The template region may include the same number of physical resources (e.g., server devices, server racks, networking devices, etc.) as the target region to be built using the image-based region building operations described herein. In some embodiments, the size of the template region may be adjusted to fit one or more “standard” region sizes. For example, a region built in a prefabrication plant may typically include 20 server racks, each with ten server devices, such that the template region may also have 20 server racks, each with ten devices. The images created from the template region can then be easily deployed in the prefabrication plant to common configurations of devices used for the region. In most cases, the template region may be topologically identical to the target region built using the image-based region building operations (e.g., having the same network topology). In some embodiments, the template region may be topologically different from the target region. For example, the target region may be larger than the template region due to additional server racks. The set of images from the template region can be deployed to devices in the target region corresponding to the network topology of the template, while the additional server racks are provisioned separately (e.g., using generic machine images, using orchestrated region building operations) and integrated into the region network as part of the final configuration.

[0100] When a template region is built, the deployed resources can be configured with identifiers corresponding to the template region. For example, a template region may include hostnames, network addresses, certificates, region names, and other identifiers used to implement network connectivity within the template region's network. However, these identifiers may not be suitable for use in the target region. It may be necessary to make the images obtained for the template region agnostic to the template region's network configuration before they are deployed to the target region's physical resources. The manager service can be configured to anonymize images in an image set by replacing region-specific identifiers (e.g., numeric identifiers, globally unique identifiers (“UUIDs”), human-readable labels, region-specific network addresses, region-specific hostnames, etc.) with generic identifiers (e.g., generic hostnames). In some embodiments, template regions can be built using generic or anonymized identifiers, eliminating the need for separate anonymization of images when building the image set. When a target region is built using an image-based region build operation, the manager service can orchestrate late bindings of identifiers corresponding to the target region. Specific details regarding the rotation of regional network addresses, resource identifiers, and service endpoints can be found in U.S. Patent Application No. 18 / 215,632, entitled “TECHNIQUES FORROTATING NETWORK ADDRESSES IN PREFAB REGIONS”; U.S. Patent Application No. 18 / 382,885, entitled “TECHNIQUES FOR ROTATING SERVICE ENDPOINTS IN PREFAB REGIONS”; and U.S. Patent Application No. 18 / 520,510, entitled “TECHNIQUES FOR ROTATING RESOURCE IDENTIFIERS IN PREFAB REGIONS”; the contents of these applications are incorporated herein by reference in their entirety for all purposes.

[0101] Image-based region building operations can offer many advantages over conventional resource deployment operations. As an example, copying a device image from a repository to a target device can happen significantly faster than deploying all individual software resources to the same device. If some or all devices in the target region are built from the corresponding image, the speed and computational efficiency of the entire region build process can be greatly improved. As another example, customization of software resources deployed in an image can be transferred from the prefab factory to the destination site. Because some configuration operations at the prefab factory may overlap with those at the destination site, performing an image-based region build with an anonymous image and then later binding a region-specific identifier can avoid redundant configuration tasks between the prefab factory and the destination site.

[0102] Figure 5 This is a block diagram illustrating the creation of an image set 502 for region devices (e.g., devices 504, 506, 508) in a template region 500 according to at least one embodiment. The template region 500 may be a prefabrication plant (e.g., Figure 3 The prefabrication area constructed at the prefabrication plant 302 (e.g., Figure 3 Example of a prefabricated region 330. In some embodiments, template region 500 may be a collection of computing devices and networking devices configured as a regional network for the purpose of constructing an image set according to the techniques described herein.

[0103] like Figure 5 As shown, devices 504-508 can be computing devices (e.g., server devices, server racks, etc.), networking devices (e.g., network switches, gateways, etc.), or other suitable physical resources for template area 500. Devices 504-508 can be connected together to form a local area network. For example, device 504 can be connected to devices 506 and 508 in network arrangement 516. Network arrangement 516 can include additional connections to other devices, including networking devices and other networking infrastructure.

[0104] Devices 504-508 may be associated with one or more identifiers 518-522. As briefly described above, physical resources within a local area network may host software resources that include identifiers. For example, identifiers may be numeric identifiers, globally unique identifiers (“UUIDs”), human-readable labels, or resource names (e.g., the “type” of a resource, such as VM for virtual machine). In some examples, identifiers 518-522 may include device-specific or software resource-specific information, such as network addresses, hostnames, etc.

[0105] To build the image set 502, the manager service (e.g., Figure 4 The device manager service 412 can be configured to generate device images from each device in template area 500. A device image can be a copy of one or more storage devices for each device. For example, device image 510 can be a copy of data stored at device 504. Similarly, device image 512 can be a copy of data stored at device 506, and device image 514 can be a copy of data stored at device 508. Device images can include operating systems, deployed software resources, services and / or applications persisted at each device, and other data stored at the device, such that when the image is deployed (e.g., copied) to a target device with a similar or identical physical configuration, the target device can be used as a copy of the device from which the device image is obtained. The operations for generating device images can be related to the above regarding... Figure 3The operation described as part of the region building operation for generating device snapshots is similar. Furthermore, each device 504-508 may have a corresponding hardware configuration. The hardware configuration may specify the device's processing, memory, storage capacity, and / or networking capabilities. For example, device 504 may be a server device with two central processing units, 32 GB of dynamic memory, and 1 TB of non-volatile (e.g., NVMe) storage, while device 506 may be another server device with four processing units, 64 GB of dynamic memory, and 1 TB of non-volatile storage. Template region 500 may be associated with a hardware configuration that includes hardware configuration information for all devices (e.g., devices 504-508) within the template region. This hardware configuration information may be stored along with image set 502 and / or combined with network topology 524 to characterize the devices to which specific device images can be deployed from image set 502.

[0106] The manager service can also determine the network layout 516 between devices 504-508. For example, with network services (e.g., Figure 3 The network service 320, combined with the manager service, can store network configurations corresponding to the network layout 516 between devices 504-508. This stored network configuration can be a network topology 524 specifying the layout from each device in template area 500 to each other device. For example, network topology 524 can specify that device 504 is the fourth server device in the second rack of the server equipment and is directly connected to device 506, which is the top rack network switch of the second rack of the server equipment. More generally, network topology 524 can include network identifiers (e.g., network addresses, hostnames, etc.) for devices 504-508, current network connections between devices 504-508, physical networking interfaces between devices and the prefabrication plant's networking infrastructure, and network settings for the devices (e.g., port configurations, gateway configurations, etc.). Network topology 524 can be used to identify the devices to which each device image 510-514 will subsequently be deployed.

[0107] To generate an image set 502 that is agnostic to the target device during subsequent deployment of device images, the manager service can be configured to anonymize each device image by removing identifiers 518-522 when generating device images 510-514. For example, if template region 500 is built in a prefabrication plant, then device 504 can be associated with identifier 518, which includes a resource name identifying the region and domain of the prefabrication plant. When generating the image, the manager service can remove or modify the region-specific portion of identifier 518. In some embodiments, template region 500 can be built and configured using anonymized identifiers 518-522. For example, template regions can be configured using dummy values ​​for regions / domains in identifiers 518-522 that allow the template region 500 service to function correctly but can be rotated after the device image is deployed to the target physical resource and installed at the destination site.

[0108] Figure 6 This is a block diagram illustrating an image provider 614 for deploying an image set to a region 630 in a data center 602, according to at least one embodiment. The region network 630 may be... Figure 3 An example of a prefabricated area 330. Data center 602 may be a facility suitable for hosting area network 630. For example, data center 602 may be... Figure 3 An example of a prefabrication plant 302. In some embodiments, data center 602 may be a data center in a production area, wherein an image provider can provide image-based region building operations for physical resources within a regional network 630, such as when deploying images to those physical resources after they have been installed at a destination site. Prefabrication service 610 may be an example of other prefabrication services described herein, including Figure 4 Prefabricated service 410. Prefabricated service 610 may include manager service 612. Manager service 612 may be an example of other manager services described herein, including... Figure 2 The manager service 212 can be configured to orchestrate the aforementioned prefabrication operations and obtain and / or generate device images from the template area (e.g., Figure 5 The device images 510-514 are used to coordinate the deployment of those device images to physical resources in region 630, as described below. Prefabricated service 610 can be connected to regional network 630 in data center 602 via network 608, which can be... Figure 2 Example of network 208.

[0109] like Figure 6As shown, regional network 630 can be built during a prefabrication regional construction operation at data center 602 (e.g., a prefabrication plant). When using image-based regional construction technology, infrastructure components and deployment software resources are not supplied separately (as mentioned above). Figure 1-4 Instead of the described image set, the manager service 612 can orchestrate the deployment of device-specific images from image sets to physical resources in region 630. Image set 616 can represent an image set of one or more template regions for deployment to region 630. For example, image set 616 may include an image set having device images of each device in server racks 632A and 632B for region 630, and image sets for regions with different numbers of server racks and devices. Manager service 612 can determine which image set from image set 616 should be deployed to region 630. For example, manager service 612 can determine the network topology for region 630 and use the network topology of the storage corresponding to the image set (e.g., Figure 5 The network topology 524 is used to determine whether the topology is identical or otherwise suitable for deploying the image set. If the topology is suitable, the manager service can select the image set to deploy using the image provider. The network topology of region 630 can include the arrangement of physical resources and their network connections within region 630 via networking infrastructure 637. For example, the first server device 634 can be the second server device in server rack 632A, and the second server device 636 can be the second server device in server rack 632B, such as... Figure 6 As depicted in the text.

[0110] Image provider 614 can be configured in conjunction with manager service 612 to deploy device images to physical resources in region 630 using a network topology corresponding to the image set. For example, image provider 614 can identify device image 638 corresponding to a first server device 634 in server rack 632 and deploy device image 638 to the first server device 634. Similarly, image provider 614 can identify device image 640 corresponding to a second server device 636 in server rack 632B. Image provider 614 can then deploy device image 638 to the first server device 634 and device image 640 to the second server device 636. Because the first server device 634 and the second server device 636 are part of server racks 632A and 632B respectively, device images 638 and 640 may be different.

[0111] Figure 7This is a block diagram illustrating the deanonymization of a region device image installed at destination site 702 according to at least one embodiment. Destination site 702 may be a data center different from data center 602 (e.g., a prefabrication plant) where image-based region building operations are performed. As briefly described above, manager service 612 can coordinate and perform image deployments for both physical resources in the prefabrication plant and at the destination site (e.g., destination site 702). Thus, in some examples, physical resources of region 630 may be installed at destination site 702, and manager service 612 can then deploy device images from the image set to the physical resources (e.g., first server device 634, second server device 636), and then the manager service can deanonymize the device images.

[0112] To deanonymize device images, manager service 612 can be configured to apply the correct identifier to each device in region 630. For example, after device image 638 has been deployed on first server device 634, manager service can orchestrate the binding of identifier 742 to the software resource at first server device 634. Similarly, after device image 640 has been deployed on second server device 636, manager service can orchestrate the binding of identifier 744. As described in detail in related application U.S. Application No. 18 / 520,510 entitled “TECHNIQUES FOR ROTATING RESOURCE IDENTIFIERS IN PREFAB REGIONS”, identifiers 742 and 744 can be provided to the software resource by a service that can be configured to create, look up, retrieve, and return identifiers. This identity service can be configured to support application programming interface (“API”) calls to generate guaranteed unique identifiers using attributes and attribute values ​​associated with the corresponding software resource. The identity service can be configured to use one or more algorithms to generate identifiers. The algorithm may include methods for constructing namespace strings that identify software resources, and methods for hashing the namespace strings to generate hash identifiers. Therefore, the manager service 612 may, as appropriate (e.g., correct zone name, correct domain name, etc.), orchestrate the configuration of software resources in deployed device images 638, 640 for the local area network 630 at destination site 702.

[0113] Figure 8 This is a flowchart of an example method 800 for constructing an image set for a first set of physical resources and deploying that region set to a second set of physical resources, according to at least one embodiment. Method 800 can be executed by one or more components of a distributed computing system, including an execution manager service (e.g., Figure 2 Manager service 212 Figure 6 The CSP (e.g., the manager service 612) Figure 2 One or more components of a distributed computing system (CSP 204). The operations of method 800 can be performed in any suitable order, and method 800 can include more than Figure 8 The operations described herein may be more or fewer operations.

[0114] Some or all of method 800 (or any other processing and / or method described herein, or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed jointly on one or more processors via hardware or a combination thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable storage medium may be non-transitory.

[0115] Method 800 may begin at block 802, wherein a manager service deploys software resources to a first set of physical resources within a data center. The first set of physical resources may include devices for a template area used to initially configure the software resources for the image set. In some embodiments, the data center may be a prefabrication plant in which the template area is built. The first set of physical resources may be characterized by a first network topology and a first hardware configuration. The first network topology may include information specifying the networking arrangement of the physical resources in the first set of physical resources. For example, the network topology may specify that a first server device is located at location two of a set of 20 server racks in the template area, and that the first server device is connected to port four of the network switch of the fourth server rack. The first network topology may also include network configuration information for each physical resource. In some embodiments, the first network topology may include information specifying a first arrangement of network connections within the first set of physical resources.

[0116] The first hardware configuration may include a first parameter specifying the hardware capabilities of the first group of physical resources. The first parameter may specify the processor performance, amount of dynamic memory, or storage capacity of each physical resource in the first group. For example, the first hardware configuration may specify that the first server device has a specific number and type of processing units, a specific amount of dynamic memory, and a specific amount of non-volatile storage devices. The deployment of software resources to physical resources can be as described above. Figure 2 It happened as described.

[0117] At box 804, the manager service can generate an image set for the first group of physical resources. The image set may include a software image corresponding to each physical resource in the first group of physical resources. For example, the manager service may obtain a copy of the data stored in the non-volatile storage device of each physical resource, which may include the operating system, deployed software resources (including services and microservice components that can be executed on the physical resources), the status and configuration of VMs on the physical resources, etc.

[0118] At box 806, the manager service can determine whether the second set of physical resources is compatible with the image set. The second set of physical resources may include computing devices and / or networking devices to be installed in a target area at the destination site. The target area may be being constructed at a prefabrication plant or similar data center facility. The target area may also be installed in a data center at the destination site. The second set of physical resources may be characterized by a second network topology and a second hardware configuration. The second network topology may include information specifying the networking arrangement of the physical resources in the second set of physical resources. In some embodiments, the second network topology includes information specifying a second arrangement of network connections within the second set of physical resources.

[0119] The second hardware configuration may include a second parameter specifying the hardware capabilities of the second set of physical resources. This second parameter may specify the processor performance, amount of dynamic memory, or storage capacity of each physical resource in the second set.

[0120] At box 808, the manager service can deploy the image set to a second set of physical resources. For example, the manager service can orchestrate an image provider service to copy each device image of the image set to the corresponding physical resource in the second set of physical resources. The network topology information of the image set can be used to determine the corresponding physical resource. For example, a device image generated from a server device in the second location of the fourth server rack in the first set of physical resources can be deployed to another server device in the second location of the fourth server rack in the second set of physical resources. Deploying the image set to the second set of physical resources can be based in part on determining the compatibility of the second set of physical resources with the image set.

[0121] In some embodiments, determining whether the second set of physical resources is compatible with the image set includes determining whether the first network topology is the same as the second network topology. For example, the second set of physical resources may include the same number of server devices, networking devices, and server racks as the first set of physical resources, wherein each server rack has the same number of server devices, and each server device is connected to the networking devices in the same arrangement.

[0122] In some embodiments, determining whether a second set of physical resources is compatible with the image set includes determining whether a first parameter of the first hardware configuration is less than or equal to a second parameter of the second hardware configuration. For example, the server devices of the second set of physical resources may have a larger non-volatile storage device than each server device of the first set of physical resources. Therefore, a device image generated from the first set of physical resources may be suitable for deployment at the second set of physical resources because the additional non-volatile storage device is sufficient to accommodate the device image and works correctly when the corresponding physical resources execute the corresponding software of the device image.

[0123] At box 810, the manager service can configure identifiers associated with software resources of the image set deployed to the second set of physical resources. For example, the manager service can orchestrate identity rotation for software resources of the deployed image set by instructing the identity service to generate identifiers for software resources corresponding to the region, domain, and network of the destination site.

[0124] In some embodiments, the image set for the first set of physical resources can be maintained together with additional image sets for other sets of physical resources. For example, different template regions can be constructed, each with its own network topology and hardware configuration, allowing image sets to be built and anonymized for each template region. If the manager service determines that the second set of physical resources is incompatible with the image set, the manager service can select a compatible image set from the additional image sets. The manager service can then deploy the compatible image set to the second set of physical resources.

[0125] In some embodiments, before generating the image set, the manager service may anonymize the identifiers associated with the software resources of the image set deployed to the first set of physical resources. Anonymizing the identifiers may include replacing the identifiers with virtual identifiers.

[0126] In some embodiments, the manager service can deploy updated software resources to a second set of physical resources. Because the image set may have been generated from the template area some time before it was deployed to the second set of physical resources, the software resources may have been modified or updated during that period. After the second set of physical resources has been configured for its destination site and has a suitable identifier for the destination site network, the manager service can deploy the updated software resources corresponding to the changes (e.g., updated services, updated microservices, updated applications).

[0127] Example Infrastructure as a Service architecture

[0128] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, and clustering software, etc.). Therefore, because these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0129] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the internet and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0130] In most cases, cloud computing models may require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0131] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be started on demand), etc.

[0132] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0133] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning a set of initial infrastructure before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0134] In some examples, the infrastructure can have many interconnected components. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., a potential on-demand pool of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provided to define how inbound / outbound traffic to the network and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provided. The infrastructure can evolve incrementally as more and / or more infrastructure elements are expected and added.

[0135] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which code may need to be deployed is first set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0136] Figure 9This is a block diagram 900 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 902 may be communicatively coupled to a secure host lease 904, which may include a virtual cloud network (VCN) 906 and a secure host subnet 908. In some examples, service operator 902 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. Client computing devices can be workstation computers running a variety of commercially available UNIX® or UNIX-like operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, client computing devices can be any other electronic device, such as thin client computers, internet-enabled gaming systems (e.g., Microsoft Xbox game consoles with or without Kinect® gesture input devices), and / or personal messaging devices capable of communicating over a network that can access VCN 906 and / or the internet.

[0137] VCN 906 may include a local peering gateway (LPG) 910, which may be communicatively coupled to a secure shell (SSH) VCN 912 via LPG 910 contained in SSH VCN 912. SSH VCN 912 may include an SSH subnet 914, and SSH VCN 912 may be communicatively coupled to a control plane VCN 916 via LPG 910 contained in control plane VCN 916. Furthermore, SSH VCN 912 may be communicatively coupled to a data plane VCN 918 via LPG 910. Control plane VCN 916 and data plane VCN 918 may be contained within a service lease 919 that may be owned and / or operated by an IaaS provider.

[0138] The control plane VCN 916 may include a control plane demilitarized zone (DMZ) layer 920 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 920 may include one or more load balancer (LB) subnets 922, a control plane application layer 924 that may include one or more application subnets 926, and a control plane data layer 928 that may include one or more database (DB) subnets 930 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 922 contained in the control plane DMZ layer 920 may be communicatively coupled to one or more application subnets 926 contained in the control plane application layer 924 and an Internet gateway 934 that may be contained in the control plane VCN 916. The application subnets 926 may be communicatively coupled to one or more DB subnets 930 contained in the control plane data layer 928, as well as a service gateway 936 and a Network Address Translation (NAT) gateway 938. The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0139] The control plane VCN 916 may include a data plane mirror application layer 940, which may include one or more application subnets 926. The one or more application subnets 926 included in the data plane mirror application layer 940 may include a virtual network interface controller (VNIC) 942 capable of executing a compute instance 944. The compute instance 944 may communicatively couple the one or more application subnets 926 of the data plane mirror application layer 940 to the one or more application subnets 926 that may be included in the data plane application layer 946.

[0140] Data plane VCN 918 may include data plane application layer 946, data plane DMZ layer 948, and data plane data layer 950. Data plane DMZ layer 948 may include one or more LB subnets 922 communicatively coupled to one or more application subnets 926 of data plane application layer 946 and Internet gateway 934 of data plane VCN 918. One or more application subnets 926 may be communicatively coupled to service gateway 936 and NAT gateway 938 of data plane VCN 918. Data plane data layer 950 may also include one or more DB subnets 930 communicatively coupled to one or more application subnets 926 of data plane application layer 946.

[0141] The Internet gateway 934 of the control plane VCN 916 and data plane VCN 918 can be communicatively coupled to the metadata management service 952, which can be communicatively coupled to the public Internet 954. The public Internet 954 can be communicatively coupled to the NAT gateway 938 of the control plane VCN 916 and data plane VCN 918. The service gateway 936 of the control plane VCN 916 and data plane VCN 918 can be communicatively coupled to the cloud service 956.

[0142] In some examples, the service gateway 936 of the control plane VCN 916 or data plane VCN 918 can make application programming interface (API) calls to the cloud service 956 without traversing the public internet 954. API calls from the service gateway 936 to the cloud service 956 can be unidirectional: the service gateway 936 can make API calls to the cloud service 956, and the cloud service 956 can send requested data to the service gateway 936. However, the cloud service 956 may not initiate API calls to the service gateway 936.

[0143] In some examples, secure host lease 904 can be directly connected to service lease 919, which would otherwise be isolated. Secure host subnet 908 can communicate with SSH subnet 914 via LPG 910, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 908 to SSH subnet 914 allows secure host subnet 908 to access other entities within service lease 919.

[0144] Control plane VCN 916 allows users of service lease 919 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 916 can be deployed or otherwise used in data plane VCN 918. In some examples, control plane VCN 916 can be isolated from data plane VCN 918, and the data plane mirror application layer 940 of control plane VCN 916 can communicate with the data plane application layer 946 of data plane VCN 918 via VNIC 942, which can be included in both the data plane mirror application layer 940 and the data plane application layer 946.

[0145] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 954, which can transmit requests to the metadata management service 952. The metadata management service 952 can transmit the request to the control plane VCN 916 via internet gateway 934. The request can be received by one or more LB subnets 922 contained in the control plane DMZ layer 920. The LB subnets 922 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 926 contained in the control plane application layer 924. If the request is validated and requires a call to the public internet 954, the call to the public internet 954 can be transmitted to a NAT gateway 938 that can make calls to the public internet 954. The request may expect the storage to be located in one or more DB subnets 930.

[0146] In some examples, the data plane mirroring application layer 940 can facilitate direct communication between the control plane VCN 916 and the data plane VCN 918. For example, it might be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 918. Through VNIC 942, the control plane VCN 916 can communicate directly with the resources contained in the data plane VCN 918, and thus can perform configuration changes, updates, or other appropriate modifications.

[0147] In some embodiments, the control plane VCN 916 and data plane VCN 918 may be contained within a service lease 919. In this case, the system's users or customers may not own or operate the control plane VCN 916 or data plane VCN 918. Alternatively, the IaaS provider may own or operate both the control plane VCN 916 and data plane VCN 918, both of which may be contained within the service lease 919. This embodiment enables the isolation of networks that might prevent users or customers from interacting with resources from other users or customers. Furthermore, this embodiment allows users or customers of the system to privately store databases without relying on the public internet 954, which may not have the desired level of threat protection for storage.

[0148] In other embodiments, one or more LB subnets 922 included in the control plane VCN 916 may be configured to receive signals from the service gateway 936. In this embodiment, the control plane VCN 916 and the data plane VCN 918 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 954. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 919, which can be isolated from the public internet 954.

[0149] Figure 10 This is a block diagram 1000 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1002 (e.g., Figure 9 The service provider 902 can communicatively couple to the secure host lease 1004 (e.g., Figure 9 Secure hosting lease 904), the secure hosting lease 1004 may include a virtual cloud network (VCN) 1006 (e.g., Figure 9 VCN 906) and Secure Host Subnet 1008 (e.g., Figure 9 The secure host subnet 908). VCN 1006 may include a local peering gateway (LPG) 1010 (e.g., Figure 9 The LPG 910, which can be communicatively coupled to the Secure Shell (SSH) VCN 1012 (e.g., via the LPG 910 contained in the SSH VCN 1012) Figure 9 SSH VCN 912). SSH VCN 1012 can include SSH subnet 1014 (e.g., Figure 9 SSH subnet 914), and SSH VCN 1012 can be communicatively coupled to control plane VCN 1016 via LPG 1010 included in control plane VCN 1016 (e.g., Figure 9 Control plane VCN 916). Control plane VCN 1016 may be included in service lease 1019 (e.g., Figure 9 In the service lease 919), and the data plane VCN1018 (e.g., Figure 9 The data plane VCN 918 can be included in a customer lease 1021 that can be owned or operated by the system's users or customers.

[0150] The control plane VCN 1016 may include one or more LB subnets 1022 (e.g., Figure 9 The control plane DMZ layer 1020 of (one or more) LB subnets 922) (e.g., Figure 9The control plane DMZ layer 920), may contain one or more application subnets 1026 (e.g., Figure 9 The control plane application layer 1024 of (one or more) application subnets 926 (e.g., Figure 9 The control plane application layer 924) may contain one or more database (DB) subnets 1030 (e.g., similar to...). Figure 9 The control plane data layer 1028 of (one or more) DB subnets 930 (e.g., Figure 9 The control plane data layer 928). One or more LB subnets 1022 contained in the control plane DMZ layer 1020 can be communicatively coupled to one or more application subnets 1026 contained in the control plane application layer 1024 and an Internet gateway 1034 that can be contained in the control plane VCN 1016 (e.g., Figure 9 Internet gateway 934), and application subnet(s) 1026 can communicatively couple to DB subnet(s) 1030 contained in control plane data layer 1028 and service gateway 1036 (e.g., Figure 9 The service gateway) and Network Address Translation (NAT) gateway 1038 (e.g., Figure 9 (NAT gateway 938). The control plane VCN 1016 may include the service gateway 1036 and the NAT gateway 1038.

[0151] The control plane VCN 1016 may include a data plane mirror application layer 1040 that may contain one or more application subnets 1026 (e.g., Figure 9 The data plane mirror application layer 940). One or more application subnets 1026 contained in the data plane mirror application layer 1040 may include compute instances 1044 (e.g., similar to...). Figure 9 The virtual network interface controller (VNIC) 1042 (e.g., the VNIC of 942) of the computing instance 944. The computing instance 1044 may facilitate the mirroring of the application subnet(s) 1026 of the application layer 1040 in the data plane and may be included in the application layer 1046 in the data plane (e.g., Figure 9 Communication between one or more application subnets 1026 in the data plane application layer 946 via VNIC 1042 contained in the data plane mirror application layer 1040 and VNIC 1042 contained in the data plane application layer 1046.

[0152] The Internet gateway 1034, included in the control plane VCN 1016, can be communicatively coupled to the metadata management service 1052 (e.g., Figure 9Metadata management service 952), which can communicatively couple to the public Internet 1054 (e.g., Figure 9 The public internet 1054 can communicatively couple to a NAT gateway 1038 contained in a control plane VCN 1016. The service gateway 1036 contained in the control plane VCN 1016 can communicatively couple to a cloud service 1056 (e.g., ...). Figure 9 Cloud services (956).

[0153] In some examples, data plane VCN 1018 may be included in customer lease 1021. In this case, the IaaS provider may provide control plane VCN 1016 for each customer, and the IaaS provider may set up a unique compute instance 1044 for each customer, included in service lease 1019. Each compute instance 1044 may allow communication between control plane VCN 1016 included in service lease 1019 and data plane VCN 1018 included in customer lease 1021. Compute instance 1044 may allow resources provisioned in control plane VCN 1016 included in service lease 1019 to be deployed or otherwise used in data plane VCN 1018 included in customer lease 1021.

[0154] In other examples, an IaaS provider's customer may have a database residing in customer lease 1021. In this example, control plane VCN 1016 may include data plane mirror application layer 1040, which may include one or more application subnets 1026. Data plane mirror application layer 1040 may reside in data plane VCN 1018, but may not reside in data plane VCN 1018. That is, data plane mirror application layer 1040 may have access to customer lease 1021, but may not reside in data plane VCN 1018 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 1040 may be configured to invoke data plane VCN 1018, but may not be configured to invoke any entity contained in control plane VCN 1016. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 1016 in the data plane VCN 1018, and the data plane mirror application layer 1040 can facilitate the customer's desired deployment or other use of resources.

[0155] In some embodiments, an IaaS provider's customer can apply filters to data plane VCN 1018. In this embodiment, the customer can determine what data plane VCN 1018 can access, and the customer can restrict access from data plane VCN 1018 to the public internet 1054. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1018's access to any external networks or databases. Applying filters and controls to data plane VCN 1018, which is included in customer lease 1021, can help isolate data plane VCN 1018 from other customers and the public internet 1054.

[0156] In some embodiments, cloud service 1056 may be invoked by service gateway 1036 to access services that may not exist on public internet 1054, control plane VCN 1016, or data plane VCN 1018. The connection between cloud service 1056 and control plane VCN 1016 or data plane VCN 1018 may not be real-time or continuous. Cloud service 1056 may reside on different networks owned or operated by an IaaS provider. Cloud service 1056 may be configured to receive calls from service gateway 1036 and may be configured not to receive calls from public internet 1054. Some cloud services 1056 may be isolated from other cloud services 1056, and control plane VCN 1016 may be isolated from cloud services 1056 that may not be in the same region as control plane VCN 1016. For example, control plane VCN 1016 may be located in "Region 1," and cloud service "Deployment 9" may be located in both "Region 1" and "Region 2." If the service gateway 1036, contained in the control plane VCN 1016 located in region 1, makes a call to deployment 9, then that call can be transmitted to deployment 9 in region 1. In this example, the control plane VCN 1016 or deployment 9 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 9 in region 2.

[0157] Figure 11 This is a block diagram 1100 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1102 (e.g., Figure 9 The service provider 902 can communicatively couple to the secure host lease 1104 (e.g., Figure 9 Secure hosting lease 904), the secure hosting lease 1104 may include a virtual cloud network (VCN) 1106 (e.g., Figure 9 VCN 906) and Secure Host Subnet 1108 (e.g., Figure 9 The secure host subnet 908). VCN 1106 may include LPG1110 (e.g., Figure 9The LPG 910), which can be communicatively coupled to the SSH VCN 1112 via the LPG 1110 included in the SSH VCN 1112 (e.g., Figure 9 SSH VCN 912). SSH VCN 1112 can include SSH subnet 1114 (e.g., Figure 9 SSH subnet 914), and SSH VCN 1112 can be communicatively coupled to control plane VCN 1116 via LPG 1110 contained in control plane VCN 1116 (e.g., Figure 9 The control plane VCN 916) and coupled to the data plane VCN 1118 via the LPG 1110 contained in the data plane VCN 1118 (e.g., Figure 9 Data plane 918). Control plane VCN 1116 and data plane VCN 1118 can be included in service lease 1119 (e.g., Figure 9 In the service rental (919).

[0158] The control plane VCN 1116 may include one or more load balancer (LB) subnets 1122 (e.g., Figure 9 The control plane DMZ layer 1120 of (one or more) LB subnets 922) (e.g., Figure 9 The control plane DMZ layer 920 may contain one or more application subnets 1126 (e.g., similar to...). Figure 9 The control plane application layer 1124 of (one or more) application subnets 926 (e.g., Figure 9 The control plane application layer 924), and the control plane data layer 1128, which may contain one or more DB subnets 1130, for example, Figure 9 The control plane data layer 928). One or more LB subnets 1122 contained in the control plane DMZ layer 1120 can be communicatively coupled to one or more application subnets 1126 contained in the control plane application layer 1124 and an Internet gateway 1134 that can be contained in the control plane VCN 1116 (e.g., Figure 9 Internet gateway 934), and application subnet(s) 1126 can communicatively couple to DB subnet(s) 1130 contained in control plane data layer 1128 and service gateway 1136 (e.g., Figure 9 The service gateway) and Network Address Translation (NAT) gateway 1138 (e.g., Figure 9 (NAT gateway 938). The control plane VCN 1116 may include the service gateway 1136 and the NAT gateway 1138.

[0159] Data plane VCN 1118 may include data plane application layer 1146 (e.g., Figure 9 Data plane application layer 946), data plane DMZ layer 1148 (e.g., Figure 9 Data plane DMZ layer 948), and data plane data layer 1150 (e.g., Figure 9 The data plane data layer 950). The data plane DMZ layer 1148 may include one or more trusted application subnets 1160 and one or more untrusted application subnets 1162 that are communicatively coupled to the data plane application layer 1146, and one or more LB subnets 1122 of the Internet gateway 1134 contained in the data plane VCN 1118. The one or more trusted application subnets 1160 may be communicatively coupled to the service gateway 1136 contained in the data plane VCN 1118, the NAT gateway 1138 contained in the data plane VCN 1118, and one or more DB subnets 1130 contained in the data plane data layer 1150. The one or more untrusted application subnets 1162 may be communicatively coupled to the service gateway 1136 contained in the data plane VCN 1118 and the one or more DB subnets 1130 contained in the data plane data layer 1150. The data plane data layer 1150 may include one or more DB subnets 1130 that can be communicatively coupled to the service gateway 1136 contained in the data plane VCN 1118.

[0160] One or more untrusted application subnets 1162 may include one or more primary VNICs 1164(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1166(1)-(N). Each tenant VM 1166(1)-(N) may be communicatively coupled to a corresponding application subnet 1167(1)-(N) that may be contained in a corresponding container egress VCN 1168(1)-(N), which may be contained in a corresponding customer lease 1170(1)-(N). A corresponding secondary VNIC 1172(1)-(N) may facilitate communication between one or more untrusted application subnets 1162 contained in data plane VCN 1118 and application subnets contained in container egress VCN 1168(1)-(N). Each container exit VCN 1168(1)-(N) may include a NAT gateway 1138, which can communicatively couple to the public Internet 1154 (e.g., Figure 9 (Public Internet 954).

[0161] Internet gateway 1134, contained in control plane VCN 1116 and data plane VCN 1118, can be communicatively coupled to metadata management service 1152 (e.g., Figure 9 A metadata management system 952 is provided, which can communicatively couple to the public internet 1154. The public internet 1154 can communicatively couple to a NAT gateway 1138 contained in a control plane VCN 1116 and a data plane VCN 1118. A service gateway 1136 contained in a control plane VCN 1116 and a data plane VCN 1118 can communicatively couple to a cloud service 1156.

[0162] In some embodiments, the data plane VCN 1118 may be integrated with the customer lease 1170. Such integration may be useful or desired by the IaaS provider's customer in certain situations, such as when support may be expected during code execution. A customer may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer to the IaaS provider.

[0163] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1146. The code running this functionality may execute in VMs 1166(1)-(N) and may not be configured to run anywhere else on the data plane VCN 1118. Each VM 1166(1)-(N) may be connected to a customer lease 1170. The corresponding container 1171(1)-(N) contained in VMs 1166(1)-(N) may be configured to run the code. In this case, dual isolation may exist (e.g., container 1171(1)-(N) runs the code, where container 1171(1)-(N) may be contained in at least one or more untrusted application subnets 1162 containing VMs 1166(1)-(N)), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1171(1)-(N) may be communicatively coupled to customer lease 1170 and may be configured to transmit or receive data from customer lease 1170. Containers 1171(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 1118. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of containers 1171(1)-(N).

[0164] In some embodiments, one or more trusted application subnets 1160 may run code that may be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1160 may be communicatively coupled to one or more database subnets 1130 and configured to perform CRUD operations in one or more database subnets 1130. One or more untrusted application subnets 1162 may be communicatively coupled to one or more database subnets 1130, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1130. Containers 1171(1)-(N) that may be contained in each customer's VM 1166(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1130.

[0165] In other embodiments, the control plane VCN 1116 and the data plane VCN 1118 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1116 and the data plane VCN 1118. However, communication may occur indirectly through at least one method. The LPG 1110 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1116 and the data plane VCN 1118. In another example, either the control plane VCN 1116 or the data plane VCN 1118 may invoke the cloud service 1156 via the service gateway 1136. For example, an invocation of the cloud service 1156 from the control plane VCN 1116 may include a request for a service that can communicate with the data plane VCN 1118.

[0166] Figure 12 This is a block diagram 1200 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1202 (e.g., Figure 9 The service provider 902 can communicatively couple to the secure host lease 1204 (e.g., Figure 9 The secure hosting lease 904), the secure hosting lease 1204 may include a virtual cloud network (VCN) 1206 (e.g., Figure 9 VCN 906) and Secure Host Subnet 1208 (e.g., Figure 9 The secure host subnet 908). VCN 1206 can include LPG1210 (e.g., Figure 9 The LPG 910), the LPG 1210 can be accessed via SSH VCN 1212 (e.g., LPG 910), Figure 9The LPG 1210 in SSH VCN 1212 is communicatively coupled to SSH VCN 1212. SSH VCN 1212 may include SSH subnet 1214 (e.g., Figure 9 SSH subnet 914), and SSH VCN 1212 can be communicatively coupled to control plane VCN 1216 via LPG 1210 contained in control plane VCN 1216 (e.g., Figure 9 The control plane VCN 916) and coupled to the data plane VCN 1218 via the LPG 1210 contained in the data plane VCN 1218 (e.g., Figure 9 Data plane 918). Control plane VCN 1216 and data plane VCN 1218 may be contained in service lease 1219 (e.g., Figure 9 In the service rental (919).

[0167] The control plane VCN 1216 may include one or more LB subnets 1222 (e.g., Figure 9 The control plane DMZ layer 1220 of (one or more) LB subnets 922) (e.g., Figure 9 The control plane DMZ layer 920), may contain one or more application subnets 1226 (e.g., Figure 9 The control plane application layer 1224 of (one or more) application subnets 926 (e.g., Figure 9 The control plane application layer 924) may contain one or more DB subnets 1230 (e.g., Figure 11 The control plane data layer 1228 of (one or more) DB subnets 1130 (e.g., Figure 9 The control plane data layer 928). One or more LB subnets 1222 contained in the control plane DMZ layer 1220 can be communicatively coupled to one or more application subnets 1226 contained in the control plane application layer 1224 and an Internet gateway 1234 that can be contained in the control plane VCN 1216 (e.g., Figure 9 Internet gateway 934), and application subnet(s) 1226 can communicatively couple to DB subnet(s) 1230 contained in control plane data layer 1228 and service gateway 1236 (e.g., Figure 9 The service gateway) and Network Address Translation (NAT) gateway 1238 (e.g., Figure 9 (NAT gateway 938). The control plane VCN 1216 may include the service gateway 1236 and the NAT gateway 1238.

[0168] Data plane VCN 1218 may include data plane application layer 1246 (e.g., Figure 9 Data plane application layer 946), data plane DMZ layer 1248 (e.g., Figure 9 Data plane DMZ layer 948), and data plane data layer 1250 (e.g., Figure 9 The data plane data layer 950). The data plane DMZ layer 1248 may include one or more trusted application subnets 1260 that can be communicatively coupled to the data plane application layer 1246 (e.g., Figure 11 (one or more) trusted application subnets 1160 and (one or more) untrusted application subnets 1262 (e.g., Figure 11 The data plane VCN 1250 may include one or more untrusted application subnets 1162 and one or more LB subnets 1222 of Internet gateway 1234 contained in data plane VCN 1218. One or more trusted application subnets 1260 may communicatively couple to service gateway 1236, NAT gateway 1238 contained in data plane VCN 1218, and one or more DB subnets 1230 contained in data plane data layer 1250. One or more untrusted application subnets 1262 may communicatively couple to service gateway 1236 contained in data plane VCN 1218 and one or more DB subnets 1230 contained in data plane data layer 1250. Data plane data layer 1250 may include one or more DB subnets 1230 that may communicatively couple to service gateway 1236 contained in data plane VCN 1218.

[0169] One or more untrusted application subnets 1262 may include a primary VNIC 1264(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1266(1)-(N) residing within one or more untrusted application subnets 1262. Each tenant VM 1266(1)-(N) may run code in a corresponding container 1267(1)-(N) and is communicatively coupled to an application subnet 1226 that may be contained in a data plane application layer 1246 contained in a container egress VCN 1268. A corresponding secondary VNIC 1272(1)-(N) may facilitate communication between one or more untrusted application subnets 1262 contained in a data plane VCN 1218 and the application subnets contained in a container egress VCN 1268. The container egress VCN may include a public internet 1254 (e.g., Figure 9 The public internet (954) uses NAT gateway 1238.

[0170] Internet gateway 1234, contained in control plane VCN 1216 and data plane VCN 1218, can be communicatively coupled to metadata management service 1252 (e.g., Figure 9 A metadata management system 952 is provided, which can communicatively couple to the public internet 1254. The public internet 1254 can communicatively couple to a NAT gateway 1238 contained in a control plane VCN 1216 and a data plane VCN 1218. A service gateway 1236 contained in a control plane VCN 1216 and a data plane VCN 1218 can communicatively couple to a cloud service 1256.

[0171] In some examples, Figure 12 The architecture shown in block diagram 1200 can be considered as... Figure 11 This is an exception to the pattern shown in the architecture of block diagram 1100, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1267(1)-(N) contained in each customer's VM 1266(1)-(N). Container 1267(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 1272(1)-(N) contained in one or more application subnets 1226 of the data plane application layer 1246, which may be contained in the container egress VCN 1268. The auxiliary VNIC 1272(1)-(N) can transmit the call to a NAT gateway 1238, which can then transmit the call to the public internet 1254. In this example, containers 1267(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1216 and from other entities contained in the data plane VCN 1218. Containers 1267(1)-(N) can also be isolated from resources from other clients.

[0172] In other examples, a client may use containers 1267(1)-(N) to invoke cloud service 1256. In this example, the client may run code within containers 1267(1)-(N) requesting a service from cloud service 1256. Container 1267(1)-(N) may transmit the request to a secondary VNIC 1272(1)-(N), which may transmit the request to a NAT gateway, which may then transmit the request to the public internet 1254. The public internet 1254 may then transmit the request via internet gateway 1234 to one or more LB subnets 1222 contained in control plane VCN 1216. In response to determining that the request is valid, one or more LB subnets may transmit the request to one or more application subnets 1226, which may then transmit the request to cloud service 1256 via service gateway 1236.

[0173] It should be recognized that the IaaS architectures 900, 1000, 1100, and 1200 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0174] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0175] Figure 13 An example computer system 1300, in which various embodiments can be implemented, is illustrated. System 1300 can be used to implement any of the computer systems described above. As shown, computer system 1300 includes a processing unit 1304 that communicates with a plurality of peripheral subsystems via a bus subsystem 1302. These peripheral subsystems may include a processing acceleration unit 1306, an I / O subsystem 1308, a storage subsystem 1318, and a communication subsystem 1324. Storage subsystem 1318 includes a tangible computer-readable storage medium 1322 and system memory 1310.

[0176] Bus subsystem 1302 provides a mechanism for allowing various components and subsystems of computer system 1300 to communicate with each other as intended. While bus subsystem 1302 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1302 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0177] A processing unit 1304, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1300. One or more processors may be included in the processing unit 1304. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1304 may be implemented as one or more independent processing units 1332 and / or 1334, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1304 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0178] In various embodiments, processing unit 1304 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1304 and / or storage subsystem 1318. With appropriate programming, processor(s) 1304 can provide the various functions described above. Computer system 1300 may additionally include processing acceleration unit 1306, which may include digital signal processor (DSP), dedicated processor, etc.

[0179] I / O subsystem 1308 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the motion sensor of Microsoft Kinect®, which enables users to control and interact with input devices such as game controllers for the Microsoft Xbox® 360 via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input in an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0180] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0181] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1300 to a user or other computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0182] Computer system 1300 may include storage subsystem 1318, which provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 1304, provide the aforementioned functionality. Storage subsystem 1318 may also provide a repository for storing data used according to this disclosure.

[0183] like Figure 13 As illustrated in the example, storage subsystem 1318 may include various components, including system memory 1310, computer-readable storage medium 1322, and computer-readable storage medium reader 1320. System memory 1310 may store program instructions that can be loaded and executed by processing unit 1304. System memory 1310 may also store data used during the execution of instructions and / or data generated during the execution of program instructions. Various types of programs may be loaded into system memory 1310, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0184] System memory 1310 may also store operating system 1316. Examples of operating system 1316 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations of computer system 1300 that execute one or more virtual machines, the virtual machine, along with the guest operating system (GOS), may be loaded into system memory 1310 and executed by one or more processors or cores of processing unit 1304.

[0185] System memory 1310 can be configured differently depending on the type of computer system 1300. For example, system memory 1310 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, system memory 1310 may include a basic input / output system (BIOS), which contains basic routines such as those that facilitate the transfer of information between components within computer system 1300 during startup.

[0186] Computer-readable storage medium 1322 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information for use by computer system 1300, including instructions executable by processing unit 1304 of computer system 1300.

[0187] Computer-readable storage medium 1322 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0188] As an example, computer-readable storage medium 1322 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a disk drive that reads or writes to a removable non-volatile magnetic disk, and an optical disc drive that reads or writes to a removable non-volatile optical disc (such as a CD-ROM, DVD, and Blu-ray® disc or other optical media). Computer-readable storage medium 1322 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and the like. Computer-readable storage medium 1322 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program services and other data for computer system 1300.

[0189] Machine-readable instructions executable by one or more processors or cores of the processing unit 1304 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0190] The communication subsystem 1324 provides an interface to other computer systems and networks. The communication subsystem 1324 serves as an interface for receiving data from other systems and sending data from computer system 1300 to other systems. For example, the communication subsystem 1324 enables computer system 1300 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1324 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1324 may provide a wired network connection (e.g., Ethernet).

[0191] In some embodiments, the communication subsystem 1324 may also represent one or more users who can use the computer system 1300 to receive input communications in the form of structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, etc.

[0192] As an example, the communication subsystem 1324 may be configured to receive data feeds 1326 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0193] Furthermore, the communication subsystem 1324 can also be configured to receive data in the form of a continuous data stream, which may include event streams 1328 and / or event updates 1330 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0194] The communication subsystem 1324 can also be configured to output structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1300.

[0195] The computer system 1300 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing system.

[0196] Due to the ever-evolving nature of computers and networks, the description of the computer system 1300 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0197] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0198] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0199] Accordingly, the specification and drawings are intended to be illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0200] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as covering both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0201] Disjunctive language, such as the phrase “at least one of X, Y, or Z”, is intended to be understood in the context in which items, terms, etc., can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise expressly stated. Therefore, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, each individually.

[0202] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0203] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0204] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A computer-implemented method, comprising: The manager service deploys software resources to the first set of physical resources within the data center, which are characterized by a first network topology and a first hardware configuration. The manager service generates an image set for a first group of physical resources, the image set including a software image corresponding to each physical resource in the first group of physical resources, and each software image including at least one software resource deployed to the first group of physical resources; The manager service determines whether the second set of physical resources is compatible with the image set. The second set of physical resources is characterized by the second network topology and the second hardware configuration. Based at least in part on the determination that the second set of physical resources is compatible with the image set, the manager service deploys the image set to the second set of physical resources; as well as Identifiers associated with the software resources of the image set configured by the manager service and deployed to the second set of physical resources.

2. The computer-implemented method as described in claim 1, further comprising: Maintain additional image sets for other groups of physical resources; Based at least in part on the determination that the second set of physical resources is incompatible with the image set, the manager service selects a compatible image set from the additional image set; as well as The manager service deploys the compatible image set to the second set of physical resources.

3. The computer-implemented method of claim 1, wherein the network topology includes information specifying a first arrangement of network connections within a first set of physical resources, and wherein the second network topology includes information specifying a second arrangement of network connections within a second set of physical resources.

4. The computer-implemented method of claim 3, wherein determining whether the second set of physical resources is compatible with the image set includes determining whether the first network topology is the same as the second network topology.

5. The computer-implemented method of claim 1, wherein the first hardware configuration includes a first parameter specifying the processor performance, number of dynamic memories, or storage capacity of each physical resource in the first group of physical resources, and wherein the second hardware configuration includes a second parameter specifying the processor performance, number of dynamic memories, or storage capacity of each physical resource in the second group of physical resources.

6. The computer-implemented method of claim 5, wherein determining whether the second set of physical resources is compatible with the image set includes determining whether a first parameter of the first hardware configuration is less than or equal to a second parameter of the second hardware configuration.

7. The computer-implemented method of claim 1, further comprising anonymizing, by a manager service, the identifier associated with the software resources of the image set deployed to the first set of physical resources prior to generating the image set.

8. The computer-implemented method of claim 1, further comprising deploying the updated software resources to a second set of physical resources by a manager service after configuring the identifier.

9. A distributed computing system, comprising: One or more processors; as well as One or more memories storing computer-executable instructions, which, when executed by the one or more processors, cause the distributed computing system to: The manager service deploys software resources to the first set of physical resources within the data center, which are characterized by a first network topology and a first hardware configuration. The manager service generates an image set for a first group of physical resources, the image set including a software image corresponding to each physical resource in the first group of physical resources, and each software image including at least one software resource deployed to the first group of physical resources; The manager service determines whether the second set of physical resources is compatible with the image set. The second set of physical resources is characterized by the second network topology and the second hardware configuration. Based at least in part on the determination that the second set of physical resources is compatible with the image set, the manager service deploys the image set to the second set of physical resources; as well as Identifiers associated with the software resources of the image set configured by the manager service and deployed to the second set of physical resources.

10. The distributed computing system of claim 9, wherein the one or more memories store additional instructions, which, when executed by the one or more processors, cause the distributed computing system to further: Maintain additional image sets for other groups of physical resources; Based at least in part on the determination that the second set of physical resources is incompatible with the image set, the manager service selects a compatible image set from the additional image set; as well as The manager service deploys the compatible image set to the second set of physical resources.

11. The distributed computing system of claim 9, wherein the network topology includes information specifying a first arrangement of network connections within a first set of physical resources, and wherein the second network topology includes information specifying a second arrangement of network connections within a second set of physical resources.

12. The distributed computing system of claim 11, wherein determining whether the second set of physical resources is compatible with the image set includes determining whether the first network topology is the same as the second network topology.

13. The distributed computing system of claim 9, wherein the first hardware configuration includes a first parameter specifying the processor performance, number of dynamic memories, or storage capacity of each physical resource in the first group of physical resources, and wherein the second hardware configuration includes a second parameter specifying the processor performance, number of dynamic memories, or storage capacity of each physical resource in the second group of physical resources.

14. The distributed computing system of claim 13, wherein determining whether the second set of physical resources is compatible with the image set includes determining whether a first parameter of the first hardware configuration is less than or equal to a second parameter of the second hardware configuration.

15. The distributed computing system of claim 9, wherein the one or more memories store additional instructions that, when executed by the one or more processors, cause the distributed computing system to further anonymize identifiers associated with software resources of the image set deployed to a first set of physical resources by a manager service prior to generating the image set.

16. The distributed computing system of claim 9, wherein the one or more memories store additional instructions that, when executed by the one or more processors, cause the distributed computing system to further deploy updated software resources to a second set of physical resources by a manager service after configuring an identifier.

17. A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors, cause a distributed computing system to: The manager service deploys software resources to the first set of physical resources within the data center, which are characterized by a first network topology and a first hardware configuration. The manager service generates an image set for a first group of physical resources, the image set including a software image corresponding to each physical resource in the first group of physical resources, and each software image including at least one software resource deployed to the first group of physical resources; The manager service determines whether the second set of physical resources is compatible with the image set. The second set of physical resources is characterized by the second network topology and the second hardware configuration. Based at least in part on the determination that the second set of physical resources is compatible with the image set, the manager service deploys the image set to the second set of physical resources; as well as Identifiers associated with the software resources of the image set configured by the manager service and deployed to the second set of physical resources.

18. The non-transitory computer-readable medium of claim 17, storing additional instructions that, when executed by the one or more processors, cause the distributed computing system to further: Maintain additional image sets for other groups of physical resources; Based at least in part on the determination that the second set of physical resources is incompatible with the image set, the manager service selects a compatible image set from the additional image set; as well as The manager service deploys the compatible image set to the second set of physical resources.

19. The non-transitory computer-readable medium of claim 17, wherein the network topology includes information specifying a first arrangement of network connections within a first set of physical resources, and wherein the second network topology includes information specifying a second arrangement of network connections within a second set of physical resources.

20. The non-transitory computer-readable medium of claim 19, wherein determining whether the second set of physical resources is compatible with the image set includes determining whether the first network topology is the same as the second network topology.

Images