Techniques for managing region-building dependencies

Abstract

Techniques are described for managing dependencies between components during automatic region construction. An orchestration service (e.g., a multi-flock orchestrator (MFO)) obtains configuration files corresponding to services to be bootstrapped into a region. The configuration files provide data from which bootstrapping tasks for bootstrapping a plurality of services within the region can be identified. The MFO 1) identifies dependencies between respective services based at least in part on parsing the plurality of configuration files, 2) generates a construction dependency graph (e.g., a data structure identifying configuration files and dependencies and indicating a corresponding order in which bootstrapping tasks are performed) based at least in part on the parsing, and 3) progressively instructs a provisioning and deployment manager to execute bootstrapping tasks based at least in part on traversing the construction dependency graph, thereby bootstrapping the services into the region according to the identified dependencies.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT application claims priority to U.S. Provisional Patent Application No. 63 / 308,003, entitled "Techniques for Bootstrapping a Region Build," filed February 8, 2022, U.S. Provisional Patent Application No. 63 / 312,814, entitled "Techniques for Implementing Virtual Data Centers," filed February 22, 2022, U.S. Provisional Patent Application No. 63 / 315,034, entitled "Techniques for Managing Region Build Dependencies," filed February 28, 2022, and U.S. Provisional Patent Application No. 18 / 163,219, entitled "Techniques for Managing Region Build Dependencies," filed February 1, 2023, the disclosures of which are incorporated by reference in their entireties herein for all purposes.

[0002] FIELD OF THEINVENTION The present disclosure relates to identifying and managing dependencies between components when bootstrapping a datacenter (also referred to as performing "region construction"). An orchestration service (e.g., a multi-flock orchestrator) can generate data structures (e.g., graphs, trees, etc.) from configuration files associated with resources (e.g., infrastructure components, services, etc.) to be provisioned and / or deployed within the datacenter. These data structures can be used to identify, track, and / or manage dependencies between components during region construction such that delays are avoided and efficiency of region construction is improved overall. [Background technology]

[0003] background Currently, cloud infrastructure services use many individual services to build a datacenter (e.g., to bootstrap various resources in a datacenter in a particular geographic region). In one example, a region is a logical abstraction that corresponds to a localized geographic area in which one or more datacenters are located (or will be located). Building a datacenter may include provisioning and configuring infrastructure resources and deploying code to those resources (e.g., for various services). The actions to build a datacenter may be collectively referred to as "building a region." A region may include any suitable number of datacenters, and thus building a region may include actions to build multiple datacenters. Traditional means to build a region require significant manual effort because the bootstrap action for one service may depend on other features and / or services of the region that may not yet be available. As the number of service teams and regions increases, the tasks to be performed to orchestrate provisioning and deployment increase significantly. Relying heavily on manual effort to bootstrap services and / or build a region may be time intensive, introduce risks, and may not scale well. Summary of the Invention [Means for solving the problem]

[0004] Quick Overview Embodiments of the present disclosure relate to performing automated region construction (e.g., bootstrapping (e.g., provisioning and / or deploying) resources (e.g., infrastructure components, artifacts, etc.) for any suitable number of services within a region (e.g., a geographic location associated with one or more data centers). A Multi-Flock Orchestrator (MFO) can automatically discover dependencies between the bootstrapping operations based at least in part on performing a static analysis (e.g., one or more parsing passes) of configuration files corresponding to the services. The MFO can orchestrate the bootstrapping operations according to the identified dependencies.

[0005] At least one embodiment is directed to a computer-implemented method. The method can include an orchestration service (e.g., a multi-flock orchestrator) of a cloud computing environment obtaining a plurality of configuration files corresponding to a plurality of services to be bootstrapped into a region. In one embodiment, the plurality of configuration files provide data from which a bootstrap task can be identified for bootstrapping the plurality of services within the region. The method can further include the orchestration service identifying one or more dependencies between respective services of the plurality of services based at least in part on performing operations to parse the plurality of configuration files. The method can further include the orchestration service generating a build dependency graph that maintains the identified one or more dependencies based at least in part on the parsing. In one embodiment, the build dependency graph can be a data structure that identifies the plurality of configuration files and the one or more dependencies and indicates a corresponding order in which the bootstrap tasks are performed. The method can further include the orchestration service progressively instructing a provisioning and deployment manager (e.g., CIOS Central of FIGS. 1-3 ) to perform the bootstrap tasks based at least in part on traversing the build dependency graph. In one embodiment, a bootstrap task is executable to bootstrap a number of services into a region according to one or more identified dependencies.

[0006] In an embodiment, identifying the one or more dependencies may further include identifying required capabilities of the first service on which the second service depends, based at least in part on the parsing. The generated construction dependency graph may indicate that the first service should be bootstrapped before the second service and that initiation of a respective bootstrap task for the second service should be delayed until a capability indicating that the first service is bootstrapped is published. In an embodiment, identifying the one or more dependencies may further include the orchestration service identifying optional capabilities of the first service on which the second service optionally depends, based at least in part on the parsing. The generated construction dependency graph may indicate that the second service may be bootstrapped before the first service. In an embodiment, the construction dependency graph may further indicate that if the second service is bootstrapped before the first service, an additional bootstrap task corresponding to the second service should be executed when the first service is available.

[0007] In an embodiment, the method may further include the orchestration service identifying a circular dependency between the two services. Identifying the circular dependency may cause the orchestration service to instruct the provisioning and deployment manager to perform at least one additional execution of a respective bootstrap task corresponding to at least one of the two services. In an embodiment, the at least one additional execution of the respective bootstrap task resolves the circular dependency between the two services.

[0008] In an embodiment, the method can further include the orchestration service identifying the orphaned service in a build dependency graph. The build dependency graph can indicate that the orphaned service is not bootstrapped due to an error in a corresponding configuration file that corresponds to the orphaned service.

[0009] In an embodiment, the method may further include the orchestration service generating a plurality of phase data structures that identify one or more phases associated with bootstrapping the region and an order in which the one or more phases are to be executed. Each phase data structure may be associated with bootstrapping one or more instances of a given service.

[0010] Another embodiment is directed to a computing device that hosts an orchestration service of a cloud computing system and includes one or more processors and instructions that, when executed by the one or more processors, cause the orchestration service and / or the computing device to perform the methods disclosed herein.

[0011] Another embodiment is directed to a cloud computing system including one or more processors and instructions that, when executed by the one or more processors, cause an orchestration service of the cloud computing system to perform the methods disclosed herein.

[0012] Yet another embodiment is directed to a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a cloud computing system, cause an orchestration service of the cloud computing system to perform the methods disclosed herein.

[0013] To facilitate identifying the description of any particular element or act, one or more of the most significant digits of a reference number may refer to the figure number in which that element first appears. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an environment in which a Cloud Infrastructure Orchestration Service (CIOS) can operate to dynamically provide bootstrap services in a region, according to at least one embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating an environment and method for building a virtual bootstrap environment (ViBE) according to at least one embodiment. [Diagram 3] 1 is a block diagram illustrating an environment and method for bootstrapping a service into a target region using ViBE, according to at least one embodiment. [Figure 4] 1 is a block diagram illustrating a region with a number of blocks (eg, services) associated therewith, in accordance with at least one embodiment. [Diagram 5] FIG. 2 is a block diagram illustrating the relationship between flocks, several phases, and a corresponding number of execution targets for each phase, in accordance with at least one embodiment. [Figure 6] FIG. 11 illustrates an example code segment illustrating several capability types, according to at least one embodiment. [Figure 7] FIG. 1 illustrates another example code segment from which one or more dependencies can be determined, in accordance with at least one embodiment. [Figure 8] FIG. 2 illustrates an example portion of a construction dependency graph in accordance with at least one embodiment. [Figure 9]FIG. 1 illustrates an example method for bootstrapping a number of services within a region based at least in part on one or more dependencies identified by performing a static analysis of one or more flock configuration files, according to at least one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 11] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment. [Figure 12] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment. [Figure 13] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment. [Figure 14] FIG. 1 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. It will be understood, however, that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0016] Example automated data center construction (region construction) infrastructure Recently, there has been an exponential increase in the adoption of cloud services. Currently, various types of cloud services are offered by various different cloud service providers (CSPs). The term cloud service is commonly used to refer to services or functionality provided on-demand (e.g., via a subscription model) by a CSP to a user or customer using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure and that are used to provide cloud services to customers are separate from the customer's own on-premise services and systems. Thus, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to give subscribing customers easy, scalable, on-demand access to applications and computing resources without the customer having to invest in procuring the infrastructure used to provide the service or functionality. Various different types or models of cloud services may be offered, such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), Infrastructure-as-a-Service (IaaS), etc. A customer may subscribe to one or more cloud services offered by a CSP. A customer may be any entity, such as an individual, an organization, or a business.

[0017] As mentioned above, a CSP is responsible for providing the infrastructure and resources used to provide cloud services to its subscribing customers. The resources provided by the CSP can include both hardware and software resources. These resources can include, for example, computational resources (e.g., virtual machines, containers, applications, processors), memory resources (e.g., databases, data stores), networking resources (e.g., routers, host machines, load balancers), identity and other resources. In one implementation, the resources provided by the CSP to provide a set of cloud services CSPs are organized as a data center. A data center can be configured to provide a particular set of cloud services. The CSP is responsible for equipping the data center with the infrastructure and resources used to provide that particular set of cloud services. A CSP can build one or more data centers.

[0018] Data centers offered by a CSP can be hosted in different regions. A region is a localized geographic area and can be identified by a region name. Regions are generally independent of each other and can be separated by great distances, such as across multiple countries or even continents. Regions are grouped into realms. Examples of CSP regions can include US West, US East, Australia East, Australia Southeast, etc.

[0019] A region may include one or more data centers, and the data centers may be located within a geographic area corresponding to the region. As an example, the data centers in a region may be located in a city within the region. For example, for a particular CSP, a data center in the US West region may be located in San Jose, California, a data center in the US East region may be located in Ashburn, Virginia, a data center in the Australia East region may be located in Sydney, Australia, a data center in the Australia Southeast region may be located in Melbourne, Australia, etc.

[0020] Data centers within a region can be organized into one or more availability domains, which are used for high availability and disaster recovery purposes. An availability domain can contain one or more data centers within a region. Availability domains within a region are isolated from each other, fault tolerant, and designed in a manner that makes it highly unlikely that data centers in more than one availability domain will fail at the same time. For example, availability domains within a region can be built in a manner that makes it highly unlikely that a failure in one availability domain within a region will affect the availability of data centers in other availability domains in the same region.

[0021] When a customer or subscriber subscribes or signs up for one or more services offered by a CSP, the CSP creates a tenancy for that customer. A tenancy is like an account created for a customer. In one implementation, a customer's tenancy exists in a single realm and has access to all regions that belong to that realm. The customer's users can then access the services to which the customer subscribes under this tenancy.

[0022] As described above, a CSP builds or deploys a data center to provide cloud services to the CSP's customers. As the CSP's customer base grows, the CSP typically builds a new data center in a new region or expands the capacity of an existing data center to accommodate the growing demand of the customers and to better serve the customers. Preferably, the data center is built in close geographic proximity to the location of the customers served by the data center. The geographical proximity of a data center to the customers served by the data center facilitates more efficient use of resources and faster and more reliable service to the customers. Thus, a CSP typically builds a new data center in a new region in a geographic area that is geographically close to the customers served by the data center. For example, for a growing customer base in Germany, the CSP may build one or more data centers in a new region in Germany.

[0023] Building a data center (or multiple data centers) in a region may also be referred to as building a region. The term "region construction" is used to refer to building one or more data centers in a region. Building a data center in a region involves provisioning or creating a new set of resources that are needed or used to provide the set of services that the data center is configured to provide. The end result of the region construction process is the creation of a data center in the region that is capable of providing the set of services targeted to that data center and includes the set of resources used to provide the set of services.

[0024] Building a new data center in a region is a highly complex task that requires extensive coordination between various bootstrapping activities. Broadly speaking, it requires the execution and coordination of various tasks, such as identifying the set of services to be provided by the data center, identifying the various resources required to provide the set of services, creating, provisioning and deploying the identified resources, and properly connecting the resources so that they can be used in the intended manner. Each of these tasks has further subtasks that require coordination, which further increases the complexity. Due to this complexity, currently, building a data center in a region involves several manual initiation or manual control tasks that require careful manual coordination. As a result, the task of building a new region (i.e., building one or more data centers in a region) is very time-consuming. It can take time, for example, several months, to build a data center. Moreover, this process is highly error-prone and may require several re-runs before the desired configuration of the data center is achieved, thereby further increasing the time required to build the data center. These limitations and problems significantly limit the ability of CSPs to scale computing resources in a timely manner to meet growing customer needs.

[0025] Technical effects This disclosure describes techniques that reduce construction time, reduce computing resource waste, and reduce risk associated with constructing one or more data centers in a region. Using the techniques described herein, new data centers can be constructed in a region in a relatively much shorter time than the weeks and months required to construct a traditional regional data center, while reducing the risk of error over traditional approaches.

[0026] Disclosed herein is a cloud infrastructure orchestration service (CIOS) configured to bootstrap (e.g., provision and deploy) services to a new datacenter based on a predefined configuration file that identifies resources (e.g., infrastructure components and software to be deployed) for implementing a given change to the datacenter. The CIOS can parse and analyze the configuration file (e.g., flock config) to identify dependencies between resources, execution targets, phases, and flocks. The CIOS can generate specific data structures from the analysis, which can be used to drive operations and manage the order in which services are bootstrapped into regions. The CIOS can use these data structures to identify when a service can be bootstrapped, when bootstrapping is blocked, and / or when bootstrapping operations associated with a previously blocked service can be resumed. Advantageously, the CIOS can identify circular dependencies in the data structures and perform operations to remove / resolve those circular dependencies prior to task execution. Using these techniques, the CIOS substantially reduces the risk of executing tasks before the resources on which they depend are available.

[0027] Using the techniques disclosed herein, CIOS can optimize the parallelism of executing changes to datacenters while ensuring that tasks are not initiated until the functionality they depend on is available in the region. In this way, CIOS allows region construction to be performed more efficiently, thereby significantly reducing the time it takes to construct a datacenter and the wasted computing resources seen in traditional approaches.

[0028] Disclosed herein is a multi-flock orchestrator (MFO) (e.g., an orchestration service). The MFO can be configured to use several version sets that identify different sets of configuration files used for testing and / or region construction. Performing a region construction can include performing any suitable operations to provision and / or deploy any suitable number of resources in one or more data centers corresponding to the region. A particular version set can be used to construct the data center. The version set can identify a particular set of configuration files from which bootstrap tasks (e.g., provisioning and deployment tasks) for the data center can be determined. The MFO can perform static analysis of the configuration files to identify dependencies between services and to identify and resolve circular dependencies prior to region construction. The MFO can progressively instruct the provisioning and deployment manager to perform bootstrap tasks while ensuring that dependent bootstrap tasks are not initiated until the resources on which they depend are available.

[0029] As various capabilities at the center become available, the MFO can identify and perform subsequent bootstrap tasks to be performed, thereby incrementally driving the build process to completion. The techniques of this disclosure allow for unit and / or integration testing to be performed on configuration files before they are used for region build, thereby increasing the likelihood of successful region build. Using the techniques of this disclosure, the MFO allows for automated region build while reducing the risk of error and the time required in conventional systems.

[0030] Certain definitions A "region" is a logical abstraction that corresponds to a geographic location. A region may include any suitable number of one or more execution targets. In an embodiment, an execution target may correspond to a data center.

[0031] An "Execution Target" is the smallest unit of change for executing a release. A "Release" is an expression of intent to orchestrate a particular change to a service (e.g., deploy version 8, "add internal DNS record", etc.). For most services, an execution target represents an "instance" of the service. A single service can be bootstrapped onto one or more execution targets each. An execution target can be associated with a set of devices (e.g., a data center).

[0032] "Bootstrapping" is intended to refer to the collective tasks associated with the provisioning and deployment of any suitable number of resources (e.g., infrastructure components, artifacts, etc.) that correspond to a single service.

[0033] "Service" refers to functionality provided by a set of resources. The set of resources for a service includes any suitable combination of infrastructure, platform, or software (e.g., applications) hosted by a cloud provider that can be configured to provide the functionality of the service. Services can be provided to users over the Internet.

[0034] "Artifact" refers to infrastructure components or code that is deployed to a Kubernetes engine cluster, which may include software (e.g., applications), configuration information for infrastructure components (e.g., configuration files), etc.

[0035] "Flock config" refers to a configuration file (or set of configuration files) that describes the set of all resources (e.g., infrastructure components and artifacts) associated with a single service. A Flock config can contain declarative statements that specify one or more aspects that correspond to the desired state of the service's resources.

[0036] "Service State" refers to a point-in-time snapshot of all resources (e.g., infrastructure resources, artifacts, etc.) associated with a service. The service state indicates the situation corresponding to the provisioning and / or deployment tasks associated with the service resources.

[0037] IaaS provisioning (or "provisioning") refers to obtaining a computer or virtual host for use and installing the necessary libraries or services on it. The phrase "provisioning a device" refers to making a device available for use by an end user for a specific use. A device that has undergone the provisioning process may be called a "provisioned device." Preparing a provisioned device (installing libraries and daemons) may be part of provisioning. This preparation is distinct from deploying a new application or a new version of an application to a prepared device. In most cases, deployment does not include provisioning, which may need to be done first. After preparation, the device may be called an "infrastructure component."

[0038] IaaS deployment (or "deployment") refers to the process of providing and / or installing an application or a new version of an application onto provisioned infrastructure components. After the infrastructure components are provisioned (e.g., acquired, allocated, prepared, etc.), additional software can be deployed (e.g., provisioned and installed onto the infrastructure components). After provisioning and deployment are complete, the infrastructure components can be referred to as "resources." Examples of resources can include, but are not limited to, virtual machines, databases, object storage, block storage, load balancers, etc.

[0039] A "capability" identifies a unit of functionality associated with a service. A unit may be some or all of the functionality provided by a service. For example, a capability may be exposed that indicates that a resource is available for authorization / authentication processing (e.g., a subset of the functionality provided by the resource). As another example, a capability may be exposed that indicates that the full functionality of the service is available. Capabilities may be used to identify functionality on which a resource or service depends and / or functionality of the resource or service that is available for use.

[0040] A "virtual bootstrap environment" (ViBE) refers to a virtual cloud network that is provisioned in an overlay (e.g., a "host region") of an existing region. Once provisioned, the ViBE is connected to the new region using a communication channel (e.g., an IPSec tunnel VPN). Certain important core services (or "seed" services) can be provisioned in the ViBE, such as a deployment orchestrator, public key infrastructure (PKI) services, etc. These services can provide the capabilities required to bring the hardware online, establish a chain of trust to the new region, and deploy the remaining services in the new region. The use of a virtual bootstrap environment can prevent circular dependencies between bootstrap resources by using resources in the host region. Services can be staged and tested in the ViBE before the physical region (e.g., the target region) is available.

[0041] "Cloud Infrastructure Orchestration Services" (CIOS) may refer to a system configured to manage the provisioning and deployment operations of any suitable number of services as part of a region construction.

[0042] A Multi-Flock Orchestrator (MFO) can be a computing component (e.g., services) that coordinates events among components of the CIOS to provision and deploy services to a target region (e.g., a new region). The MFO tracks relevant events for each service in the region build and takes actions in response to those events.

[0043] "Host Region" refers to a region that hosts a Virtual Bootstrap Environment (ViBE). A host region can be used to bootstrap a ViBE.

[0044] "Target region" refers to the region being constructed. "Exposing a capability" refers to providing an indication that a particular capability is available (or unavailable), through "publishing" as used in "publisher-subscriber" computing design, or otherwise. Capabilities are "exposed" (e.g., collected by a capability service, provided to a capability service, pushed, pulled, etc.) to provide an indication that functionality of a resource / service is available. In one embodiment, capabilities can be exposed / transmitted via events, notifications, data submissions, function calls, API calls, etc. An event (or other notification / data submission, etc.) indicating availability of a particular capability can be broadcast / addressed (e.g., published) to the capability service.

[0045] A "capability service" can be a configured flock that models dependencies between different flocks. Capability services can be provided within a cloud infrastructure orchestration service and can define what capabilities, services, and features are made available in a region.

[0046] A "Real-time Regional Data Distributor" (RRDD) can be a service or system configured to manage regional data that can be populated into a flock config to dynamically create execution targets for new regions.

[0047] Described herein, in certain examples, are techniques for implementing a cloud infrastructure orchestration service (CIOS). Such techniques can be configured to manage the bootstrapping (e.g., provisioning and deployment of software to infrastructure components) in a cloud environment (e.g., region), as briefly described above. In some cases, the CIOS can include computing components (e.g., CIOS central and CIOS regional, both of which are described in more detail below) that can be configured to manage the bootstrapping tasks (provisioning and deployment) for a given service and a multi-flock orchestrator (also described in more detail below) that is configured to initiate / manage a region build (e.g., a bootstrapping operation corresponding to multiple services).

[0048] CIOS enables region construction and worldwide infrastructure provisioning and code deployment with minimal manual run-time effort from service teams (e.g., beyond initial approval and / or physical transport of hardware, in some cases). High-level responsibilities of CIOS include, but are not limited to, coordinating region construction, giving users a view of the current state of resources managed by CIOS (e.g., in a region, across multiple regions, worldwide, etc.), and managing bootstrapping operations for bootstrapping resources within a region.

[0049] CIOS can provide view reconciliation, where a view of a desired state of a resource (e.g., a desired configuration) can be matched with the current / actual state of the resource (e.g., a current configuration). In some cases, view reconciliation can include obtaining state data to determine what resources are actually running and the current configuration and / or state of the resource. Reconciliation can occur at various granularities, such as at the service level.

[0050] The CIOS may perform plan generation, where differences between desired and current states of resources are identified. Part of plan generation may include identifying actions that would need to be performed to bring resources from their current state to the desired state. In some examples, the CIOS may present the generated plan to a user for approval. In these examples, the CIOS may mark the plan as approved or rejected based on user input from the user. Thus, because the plan is generated by a machine, the user may spend less time making judgments about the plan, and the plan may be more accurate. Although the plan is too detailed for human use, the CIOS may provide this data via an advanced user interface (UI).

[0051] In one example, the CIOS can handle change control execution by executing approved plans. Once an execution plan has been created and approved, engineers may not need to be involved in change control unless the CIOS initiates a rollback. The CIOS can handle rollbacks to a previous service version by generating a plan to revert the service to a previous (e.g., pre-release) state (e.g., if the CIOS detects a deterioration in the service health during execution).

[0052] CIOS can measure service health by monitoring alarms and running integration tests. CIOS can help teams quickly define rollback actions in case of service degradation, and CIOS can execute them. CIOS can generate and display plans, and track approvals. CIOS can combine provisioning and deployment functions into a single system that coordinates these tasks across region construction. CIOS also supports discovery of flocks (e.g., service resources such as flock configs corresponding to any suitable number of services), artifacts, resources, and dependencies. CIOS can discover dependencies between execution tasks at any level (e.g., resource level, execution target level, phase level, service level, etc.) by static analysis (e.g., including content parsing and processing) of one or more configuration files. Using these dependencies, CIOS can generate various data structures that can be used to drive task execution (e.g., tasks related to infrastructure resource provisioning and artifact deployment across regions) from these dependencies.

[0053] FIG. 1 is a block diagram of an environment 100 in which a cloud infrastructure orchestration (CIOS) 102 can operate to dynamically provide bootstrap services in a region, according to at least one embodiment. CIOS 102 can include, but is not limited to, the following components: a real-time regional data distributor (RRDD) 104, a multi-flock orchestrator (MFO) 106, a CIOS central 108, a CIOS regional 110, and a capability service 112. The specific functions of CIOS central 108 and CIOS regional 110 are described in more detail in U.S. patent application Ser. No. 17 / 016,754, entitled "Techniques for Deploying Infrastructure Resources with a Declarative Provisioning Tool," which is incorporated in its entirety for all purposes. In an embodiment, any suitable combination of the components of CIOS 102 can be provided as a service. In an embodiment, a portion of CIOS 102 can be deployed to a region (e.g., a data center represented by host region 103). In one embodiment, CIOS 102 may include any suitable number of cloud services (not shown in FIG. 1), as described in more detail in U.S. patent application Ser. No. 17 / 016,754 and with respect to FIGS. 2 and 3 below.

[0054] The real-time regional data distributor (RRDD) 104 can be configured to maintain and provide regional data that identifies realms, regions, execution targets, and availability domains. In some cases, the regional data can be in any suitable form (e.g., JSON format, data object / container, XML, etc.). The regional data maintained by the RRDD 104 can include any suitable number of subsets of data that can be individually referenced by a corresponding identifier. For example, an identifier "all_regions" can be associated with a data structure (e.g., a list, structure, object, etc.) that includes metadata for all defined regions. As another example, an identifier such as "realms" can be associated with a data structure that identifies metadata for a number of realms and a set of regions corresponding to each realm. In general, the regional data can maintain any suitable attributes, such as identifiers, DNS suffixes, state (e.g., region state), etc., for one or more realms, regions, availability domains (ADs), execution targets (ETs), etc. The RRDD 104 can be configured to manage regional state as part of the regional data. The regional state can include any suitable information indicative of a state of bootstrap in the region. For example, some example region states may include "initial," "under construction," "production," "suspended," or "to be decommissioned." The "initial" state may indicate a region that has not yet been bootstrapped. The "under construction" state may indicate that bootstrap of one or more flocks in the region has begun. The "production" state may indicate that bootstrap is complete and the region is ready for validation. The "suspended" state may indicate that CIOS Central 108 or CIOS Regional 110 has suspended internal interaction with the regional stack, possibly due to operational issues. The "decommissioned" state may indicate that the region has been decommissioned and is likely unavailable and / or will not be contacted again in the future.

[0055] CIOS central 108 may be configured to provide any suitable number of user interfaces through which a user (e.g., user 109) may interact with CIOS 102. For example, a user may make changes to region data through a user interface provided by CIOS central 108. CIOS central 108 may further provide various interfaces that allow a user to view changes made to flock configurations and / or artifacts, generate and view plans, approve / reject plans, and view the status of plan execution (e.g., corresponding to tasks involving infrastructure provisioning, deployment, region construction, and / or the desired state of any suitable number of resources managed by CIOS 102). CIOS central 108 may implement a control plane that may be configured to manage any suitable number of CIOS regional 110 instances. CIOS central 108 may provide one or more interfaces for presenting region data, thereby allowing user 109 to view and / or modify the region data. CIOS central 108 may be configured to invoke functions of RRDD 104 through any suitable number of interfaces. In general, CIOS central 108 can be configured to manage region data, either directly or indirectly (e.g., via RRDD 104). CIOS central 108 can be configured to edit the flock config to populate the region data as variables in the flock config.

[0056] Each instance of CIOS regional 110 may correspond to a module configured to perform bootstrapping tasks associated with a single service of the region. CIOS regional 110 may receive desired state data from CIOS central 108. In one embodiment, the desired state data may include a flock config that declares (e.g., by declarative statements) a desired state of the resources associated with the service. CIOS central 108 may maintain current state data that indicates any suitable aspect of the current state of the resources associated with the service. In one embodiment, CIOS regional 110 may identify that changes need to be made to one or more resources by comparing the desired state data to the current state data. For example, CIOS regional 110 may determine that one or more infrastructure components need to be provisioned, one or more artifacts need to be deployed, or any suitable changes need to be made to the resources of the service to match the desired state. When CIOS regional 110 performs the bootstrapping operation, it may expose data that indicates various capabilities of the resources as the capabilities become available. A "capability" identifies a unit of functionality associated with a service. A unit may be some or all of the functionality provided by a service. For example, a capability may be exposed to indicate that a resource is available for authorization / authentication processing (e.g., a subset of the functionality provided by the resource). As another example, a capability may be exposed to indicate that the full functionality of a service is available. Capabilities may be used to identify functionality on which a resource or service depends and / or functionality of the resource or service that is available for use.

[0057] The capability service 112 is configured to maintain capability data that indicates: 1) what capabilities of various services are currently available; 2) whether any resources / services are waiting for a particular capability; 3) what particular resources and / or services are waiting for a given capability; or any suitable combination of the above. The capability service 112 may provide an interface through which the capability data may be requested. The capability service 112 may provide one or more interfaces (e.g., application programming interfaces) that enable the capability service 112 to send capability data to the MFO 106 and / or the CIOS regional 110 (e.g., each instance of the CIOS regional 110). In an embodiment, any suitable component or module of the MFO 106 and / or the CIOS regional 110 may be configured to request the capability data from the capability service 112.

[0058] In one embodiment, a multi-flock orchestrator (MFO) 106 can be configured to drive region construction activities. In one embodiment, the MFO 106 can manage information describing what flocks / flock config versions and / or artifact versions to use to bootstrap a given service in a region (or to make a unit of change to a target region). In one embodiment, the MFO 106 can be configured to monitor (or otherwise be notified of) changes to the region data managed by the real-time regional data distributor 104. In one embodiment, region construction may be triggered by the MFO 106 upon receiving an indication that the region data has changed. In one embodiment, the MFO 106 may collect various flock configs and artifacts used in region construction. Some or all of the flock configs can be configured to be region agnostic. That is, the flock configs may not explicitly specify which flocks are to be bootstrapped into which regions. In one embodiment, the MFO 106 can trigger a data population process whereby the collected flock configs are recompiled (e.g., by the CIOS Central 108). During recompilation, operations may be performed (e.g., by the CIOS Central 108) to cause the region data maintained by the real-time regional data distributor 104 to be populated into the configuration files. The flock configs can reference the region data by variables / parameters, without requiring a hard-coded identity of the region data. The flock configs can use this data population to be dynamically modified at run time, without having the region data hard-coded and therefore more difficult to change.

[0059] The multi-flock orchestrator 106 may perform static flock analysis, in which the flock config is parsed to identify dependencies between resources, execution targets, phases, and flocks, specifically to identify circular dependencies that need to be removed. In an embodiment, the MFO 106 may generate any suitable number of data structures based on the identified dependencies. These data structures (e.g., directed acyclic graphs, linked lists, etc.) may be used by the cloud infrastructure orchestration service 102 to drive operations to perform region construction. For example, these data structures may collectively define the order in which services are bootstrapped within a region. Examples of such data structures are discussed in more detail below in connection with the construction dependency graph 338 of FIG. 3. If circular dependencies (e.g., service A requires service B and vice versa) exist and are identified by the static flock analysis and / or graph, the MFO may be configured to notify any appropriate service teams that changes need to be made to the corresponding flock config to correct these circular dependencies. The MFO 106 may be configured to traverse one or more data structures to manage the order in which services are bootstrapped into regions. The MFO 106 can identify capabilities available within a given region at any given time (e.g., using data obtained from the capabilities service 112). The MFO 106 can use this data to identify when the MFO 106 can bootstrap a service, when bootstrap is blocked, and / or when a bootstrap operation associated with a previously blocked service can be resumed. Based on this scanning, the MFO 106 can perform various releases in which instructions are sent by the MFO 106 to the CIOS central 108 to perform bootstrap operations corresponding to any suitable number of block configurations.In one example, the MFO 106 can be configured to identify that one or more flock configs may require multiple releases due to circular dependencies found in the graph, and as a result, the MFO 106 can send multiple sets of instructions to the CIOS central 108 for a given flock config to break the circular dependencies identified in the graph.

[0060] In one embodiment, a user may request that a new region (e.g., target region 114) be built. This may involve bootstrapping resources corresponding to various services. In one embodiment, the target region 114 may not be available for communication (and / or may not be secure) at the time the region build request is issued. Rather than deferring bootstrapping until the target region 114 is available and configured to perform the bootstrapping operation, CIOS 102 may initiate region build using a virtual bootstrap environment 116. The virtual bootstrap environment (ViBE) 116 may be an overlay network hosted by the host region 103 (an existing region that has previously been configured with a core set of services, is available for communication, and is secure). The MFO 106 may utilize the resources of the host region 103 to bootstrap resources to the ViBE 116 (commonly referred to as "building the ViBE"). For example, MFO 106 can provide instructions via CIOS central 108 to cause an instance of CIOS regional 110 in a host region (e.g., host region 103) to bootstrap another instance of CIOS regional in ViBE 116. After the CIOS regional in ViBE is available for processing, bootstrapping of services for the target region 114 can continue in ViBE 116. Services previously bootstrapped in ViBE 116 can be migrated to the target region 114 when the target region 114 is available to perform the bootstrap operation. Using these techniques, CIOS 102 can greatly increase the speed at which regions are built by greatly reducing the need for manual input and / or configuration to be provided.

[0061] 2 is a block diagram illustrating an environment 200 and method for building a virtual bootstrap environment (ViBE) 202 (an example of ViBE 116 of FIG. 1) according to at least one embodiment. ViBE 202 represents a virtual cloud network that is provisioned in an overlay of an existing region (e.g., hosted region 204, which is an example of hosted region 103 of FIG. 1 and in one embodiment is a hosted region service enclave). ViBE 202 represents an environment in which services can be staged for a target region (e.g., a region under construction, such as target region 114 of FIG. 1) before the target region is available.

[0062] To bootstrap a new region (e.g., target region 114 in FIG. 1), a core set of services can be bootstrapped. These core set of services exist in the host region 204 but do not yet exist in ViBE (nor in the target region). These basic core services provide the functionality required for provisioning devices, establishing a chain of trust to the new region, and deploying the remaining services (e.g., flocks) to the region. ViBE 202 may be a tenancy that is deployed in the host region 204. This can be thought of as a virtual region.

[0063] When the target region is available to provide the bootstrap operations, ViBE 202 can be connected to the target region such that services in the ViBE can interact with services and / or infrastructure components of the target region. This allows for the deployment of production-level services instead of a self-contained seed service as in previous systems, which would require connectivity over the Internet to the target region. Traditionally, a seed service would be deployed as part of a container collection and used to bootstrap the dependencies required to build the region. Using the existing region's infrastructure / tooling, resources can be bootstrapped (e.g., provisioned and deployed) to ViBE 202 until the target region is self-sufficient and can communicate directly, and it can connect to the service enclaves of the region (e.g., host region 204) to provision hardware and deploy services. The use of ViBE 202 allows for the maintenance of dependencies and services required to be able to provision / prepare infrastructure and deploy software while utilizing the host region's resources to break circular dependencies for core services.

[0064] A multi-flock orchestrator (MFO) 206 can be configured to perform operations to build (e.g., configure) ViBE 202. The MFO 206 can obtain appropriate flock configs corresponding to the various resources to be bootstrapped into a new region (in this case, the ViBE region, ViBE 202). For example, the MFO 206 can obtain a flock config (e.g., a "ViBE flock config") that specifies aspects of the bootstrap of capability services 208 and workers 210. As another example, the MFO 206 can obtain another flock config corresponding to the bootstrap of domain name services (DNS) 212 into ViBE 202.

[0065] In step 1, MFO 206 can instruct CIOS central 214 (e.g., an example of CIOS central 108 and CIOS central 214 of FIGS. 1 and 2, respectively). For example, MFO 206 can send a request (e.g., including a ViBE flock config) to request bootstrap of capability services 208 and workers 210 that do not yet exist in ViBE 202 at this point. In one embodiment, CIOS 214 has access to all flock configs. Thus, in one example, MFO 206 can send an identifier for a ViBE flock config rather than the file itself, and CIOS central 214 can retrieve it independently from storage (e.g., from DB 308 or Flock DB 312 of FIG. 3).

[0066] In step 2, CIOS central 214 can provide the ViBE Flock Config via a corresponding request to CIOS regional 216. CIOS regional 216 can parse the ViBE Flock Config to identify and execute specific infrastructure provisioning and deployment operations in step 3.

[0067] In one embodiment, CIOS regional 216 can use additional corresponding services for provisioning and deployment. For example, in step 4, CIOS regional 216 can instruct deployment orchestrator 218 (e.g., an example of a core service or other creation, build and deployment application software in host region 204) to execute instructions to bootstrap capability service 208 and worker 210 within ViBE 202.

[0068] In step 5, a capability may be sent to capability service 208 (from CIOS regional 216, deployment orchestrator 218 via worker 210, or otherwise) indicating that resources corresponding to the ViBE block are available. Capability service 208 may maintain this data. In one embodiment, capability service 208 adds this information to a list of available capabilities that capability service 208 maintains with ViBE. For example, the capability provided to capability service 208 in step 5 may indicate that capability service 208 and worker 210 are available for processing.

[0069] In step 6, the MFO 206 can determine that the capability service 208 and the worker 210 are available based on receiving or obtaining data (identifiers corresponding to capabilities) from the capability service 208.

[0070] In step 7, as a result of receiving / obtaining the data in step 6, MFO 206 can instruct CIOS Central 214 to bootstrap a DNS service (e.g., DNS 212) into ViBE 202. These instructions can specify or include a particular block config that corresponds to the DNS service.

[0071] In step 8, CIOS Central 214 can instruct CIOS Regional 216 to deploy DNS 212 to ViBe 202. In one embodiment, the DNS block config for DNS 212 is provided by CIOS Central 214.

[0072] In step 9, now that worker 210 is deployed to ViBE 202, worker 210 can be assigned by CIOS regional 216 to the task of deploying DNS 212. The worker can run a declarative infrastructure provisioner in the manner described above in connection with FIG. 3 to identify the set of operations that need to be performed to deploy DNS 212 (e.g., by comparing the flock config (desired state) with the current state of the resources associated with the flock (which currently do not exist)).

[0073] At step 10, deployment orchestrator 218 instructs worker 210 to deploy DNS 212 according to the actions identified at step 9. As shown, worker 210 proceeds to perform the actions to deploy DNS 212 to ViBE 202 at step 11. At step 12, worker 210 notifies capability service 208 that DNS 212 is available on ViBE 202. MFO 206 then determines that resources associated with the ViBE flock config and the DNS flock config are available, and any suitable number of additional resources can subsequently be bootstrapped into ViBE.

[0074] After steps 1 through 12 are completed, the process for building ViBE202 may be considered complete, and ViBE202 may be considered built.

[0075] FIG. 3 is a block diagram illustrating an environment 300 and method for bootstrapping a service into a target region using ViBE, according to at least one embodiment.

[0076] In step 1, a user 302 can use any suitable user interface provided by CIOS central 304 (e.g., an example of CIOS central 108 and CIOS central 214 of FIGS. 1 and 2, respectively) to modify region data. For example, a user 302 can create a new region into which several services are bootstrapped.

[0077] In step 2, CIOS central 304 may perform an operation to send the changes to RRDD 306 (e.g., an example of RRDD 104 in FIG. 1). In step 3, RRDD 306 may store the received region data in database 308, which is a data store configured to store region data including any suitable identifiers, attributes, states, etc., such as region, AD, realm, ET, etc. In one embodiment, updater 307 may be used to store the region data in database 308 or any suitable data store from which the updates may be accessible (by a service team). In one embodiment, updater 307 may be configured to notify (e.g., by any suitable electronic notification) of updates made to database 308.

[0078] In step 4, the MFO 310 (e.g., an example of the MFO 106 and MFO 206 of FIGS. 1 and 2, respectively) may detect the change in the region data. In one embodiment, the MFO 310 may be configured to poll the RRDD 306 for changes in the region data. In one embodiment, the RRDD 306 may be configured to publish or otherwise notify the MFO 310 of the region change.

[0079] In step 5, detection of a change in region data can trigger MFO 310 to retrieve a version set (e.g., a version set associated with a particular identifier, such as a "golden version set" identifier) ​​that identifies the specific version of each flock (e.g., service) to be bootstrapped into the new region and the specific version of each artifact corresponding to that flock. The version set can be retrieved from DB 312. As a flock evolves and changes, the corresponding config and artifact versions used for region construction can change. These changes can be maintained in flock DB 312 so that MFO 310 can identify which versions of flock config and artifacts to use for construction of the region (e.g., ViBE region, target region / non-ViBE region, etc.). Flock configs (e.g., all versions of flock configs) and / or artifacts (e.g., all versions of artifacts) can be stored in DB 308, DB 312, or any suitable data store accessible to CIOS Central 304 and / or MFO 310.

[0080] In step 6, MFO 310 can request CIOS Central 304 to recompile each of the flock configs associated with the version set with the current region data. In one embodiment, the request can indicate the version of each flock config and / or artifacts that correspond to those flock configs.

[0081] In step 7, CIOS Central 304 can obtain the current regional data from DB 308 (e.g., directly or via real-time regional data distributor 306) and retrieve any appropriate flock configurations and artifacts according to the version requested by MFO 310.

[0082] In step 8, CIOS central 304 can re-edit the flock config with the region data obtained in step 7 to populate the flock config with the current region data. CIOS central 304 can return the edited flock config to MFO 310. In one embodiment, CIOS central 304 can simply indicate that the edit is complete, and MFO 310 can access the re-edited flock config via RRDD 306.

[0083] In step 9, the MFO 310 may perform a static analysis of the recompiled flock config. As part of the static analysis, the MFO 310 may parse the flock config (e.g., using libraries associated with a declarative infrastructure provisioner (e.g., Terraform, etc.)) to identify dependencies between flocks. From this analysis and the identified dependencies, the MFO 310 may generate a build dependency graph 338. The build dependency graph 338 may be an acyclic directed graph that specifies the order in which flocks are bootstrapped (and / or changes indicated in the flock config are applied) into new regions. Each node in the graph may correspond to the bootstrap of any appropriate portion of a particular flock. A particular bootstrap order may be specified based at least in part on the dependencies. In an embodiment, the dependencies may be represented as attributes of the nodes and / or may be indicated by the edges of the graph connecting the nodes. The MFO 310 may traverse the graph (e.g., starting from the origin node) to drive the operation of region construction.

[0084] In an embodiment, MFO 310 can use a cycle detection algorithm to detect the presence of a cycle (e.g., service A depends on service B and vice versa). MFO 310 can identify orphan capability dependencies. For example, MFO 310 can identify orphan nodes in construction dependency graph 338 that do not connect to any other nodes. MFO 310 can identify capabilities that were published in error (e.g., when a capability was published prematurely and the corresponding functionality is not actually available yet). MFO 310 can detect from the graph that there are one or more instances of publication of the same capability. In an embodiment, any suitable number of such errors can be detected, and MFO 310 (or another suitable component, such as CIOS Central 304) can be configured to notify or otherwise present this information to a user (e.g., via an electronic notification, a user interface, etc.). In one embodiment, MFO 310 may be configured to force delete / recreate resources to break circular dependencies and may redirect CIOS Central 304 to perform bootstrap operations for those resources and / or the corresponding block configs.

[0085] The starting node may correspond to bootstrapping the ViBE flock, and the second node may correspond to bootstrapping the DNS. Steps 10-15 correspond to deployment (by deployment orchestrator 317, which is an example of deployment orchestrator 218 in FIG. 2) of the ViBE flock to ViBE 316 (e.g., an example of ViBE 116 and ViBE 202 in FIG. 1 and FIG. 2, respectively). That is, steps 10-15 of FIG. 3 generally correspond to steps 1-6 of FIG. 2. After being notified that capabilities exist corresponding to the ViBE flock to be deployed (e.g., indicating that capability service 318 and worker 320, which correspond to capability service 208 and worker 210 in FIG. 2, are available), MFO 310 resumes traversing build dependency graph 338 to identify the next operation to perform.

[0086] For example, MFO 310 may continue traversing construction dependency graph 338 to identify that a DNS block should be deployed. Steps 16-21 may be performed to deploy DNS 322 (an example of DNS 212 in FIG. 2). These operations may generally correspond to steps 7-12 in FIG. 2.

[0087] At step 21, a capability indicating that DNS 322 is available may be stored. Upon detecting this capability, MFO 310 may resume traversing build dependency graph 338. During this traversal, MFO 310 may identify that any appropriate portions of an instance of CIOS Regional (e.g., an instance of CIOS Regional 314) should be deployed to ViBE 316. In one embodiment, steps 16-21 may be substantially repeated with respect to the deployment of CIOS Regional (ViBE) 326 (an instance of CIOS Regional 314, CIOS Regional 110 in FIG. 1) and Worker 328 to ViBE 316. A capability that CIOS Regional (ViBE) 326 is available may be sent to capability service 318.

[0088] Upon detecting that CIOS Regional (ViBE) 326 is available, MFO 310 may resume traversing build dependency graph 338. During this traversal, MFO 310 may identify that a deployment orchestrator (e.g., deployment orchestrator 330, which is an example of deployment orchestrator 317) is deployed to ViBE 316. In one embodiment, steps 16-21 may be substantially repeated for the deployment of deployment orchestrator 330. Information may be sent to capability service 318 identifying capabilities indicating that deployment orchestrator 330 is available.

[0089] After the deployment orchestrator 330 is deployed, ViBE 316 can be considered available to process subsequent requests. Upon detecting that the deployment orchestrator 330 is available, MFO 310 can instruct that subsequent bootstrap requests are routed to the ViBE component without using the host region component (the component of the host region 332). Thus, MFO 310 can continue to traverse the build dependency graph 338 and at each node, instruct flock deployment to ViBE 316 via CIOS Central 304. CIOS Central 304 can request CIOS Regional (ViBE) 326 to deploy resources according to the flock configuration.

[0090] At some point during this process, the target region 334 may become available. An indication that the target region is available may be identifiable from region data for the target region 334 provided by the user 302 (e.g., as an update to the region data). The availability of the target region 334 may depend on the establishment of a network connection between the target region 334 and an external network (e.g., the Internet). The network connection may be supported over a public network (e.g., the Internet), but software security measures (e.g., IPSec) may be used to provide one or more encrypted tunnels (e.g., IPSec tunnels such as tunnel 336) from ViBE 316 to the target region 334. As used herein, "IPSec" refers to a protocol suite for authentication and encryption of network traffic over networks using the Internet Protocol (IP) and may include one or more available implementations of the protocol suite (e.g., Openswan, Libreswan, strongSwan, etc.). The network may connect ViBE 316 to a service enclave of the target region 334.

[0091] Prior to establishing the IPSec tunnel, the initial network connectivity to the target region 334 may be on a sufficient connection (e.g., an out-of-band VPN tunnel) to allow bootstrapping of networking services until an IPSec gateway can be deployed on an asset (e.g., a bare metal asset) in the target region 334. To bootstrap the network resources of the target region 334, the deployment orchestrator 330 can deploy an IPSec gateway on the asset in the target region 334. The deployment orchestrator 330 can then deploy a VPN host in the target region 334 configured to terminate the IPSec tunnel from ViBE 316. After a service in ViBE 316 (e.g., deployment orchestrator 330, service A, etc.) can establish an IPsec connection with a VPN host in the target region 334, the bootstrapping operation from ViBE 316 to the target region 334 can begin.

[0092] In an embodiment, the bootstrap operation may begin with a service in ViBE 316 provisioning resources in the target region 334 to support hosting instances of core services when deployed from ViBE 316. For example, a host provisioning service may provision a hypervisor on infrastructure (e.g., bare metal hosts) in the target region 334 to allocate computing resources to VMs. Once the host provisioning service completes the allocation of physical resources in the target region 334, the host provisioning service may publish information indicating a capability indicating that physical resources have been allocated in the target region 334. The capability may be published to the capability service 318 (e.g., by worker 328) via CIOS regional (ViBE) 326.

[0093] With the hardware allocation for the target region 334 established and notified to the capability service 318, CIOS regional (ViBE) 326 can orchestrate the deployment of instances of core services from ViBE 316 to the target region 334. This deployment may be similar to the process described above for building ViBE 316, but using ViBE components (e.g., CIOS regional (ViBE) 326, worker 328, deployment orchestrator 330) instead of host region 332 service enclave components. The deployment operations may generally correspond to steps 16-21 described above.

[0094] When a service is deployed from ViBE 316 to a target region 334, a DNS record associated with the service may correspond to an instance of the service in ViBE 316. The DNS record associated with the service can be updated later to complete the deployment of the service to the target region 334. In other words, the instance of the service in ViBE 316 can continue to receive traffic (e.g., requests) to the service until the DNS record is updated. The service can be partially deployed to the target region 334 and can publish information (e.g., to the capability service 318) indicating a capability that the service is partially deployed. For example, a service running in ViBE 316 can be deployed to the target region 334 along with corresponding compute instances, load balancers, and associated applications and other software, but may need to wait for database data to migrate to the target region 334 before being fully deployed. The DNS record (e.g., managed by DNS 322) can still be associated with the service in ViBE 316. Once the data migration for the service is complete, the DNS record can be updated to point to the operational service deployed in the target region 334. The deployed service in target region 334 can then receive traffic (eg, requests) for that service, while the instance of the service in ViBE 316 will no longer be able to receive traffic for that service.

[0095] Static analysis techniques for identifying build dependencies FIG. 4 is a block diagram illustrating a region 400 (one example of a target region 114) with several associated flocks, according to at least one embodiment. The region 400 may be associated with the bootstrap of any number of flocks (e.g., flocks 1-7, collectively referred to as "flock 402"). Each flock may correspond to resources (e.g., infrastructure components and software artifacts) that are bootstrapped within the region 400 for a service instance. In an embodiment, some aspects of the bootstrap of resources of a flock may depend on the availability of capabilities of other flocks. Several flocks may be bootstrapped simultaneously (e.g., because flocks 3, 4, and 5 may have common capability dependencies). Based on identifying these dependencies, an order of bootstrap of each flock within the region 400 may be determined. Techniques for doing this are described in more detail in conjunction with FIGS. 6-9.

[0096] FIG. 5 is a block diagram illustrating a relationship between a flock 500 (e.g., an example of flock 1 in FIG. 4), several phases (e.g., phase 1 and phase 2 as shown), and several corresponding execution targets for each phase, according to at least one embodiment. Each execution target in FIG. 5 indicates a different location to which resources corresponding to flock 500 are bootstrapped. For example, each execution target in FIG. 5 may indicate a different set of data centers or devices to which resources corresponding to flock 500 are bootstrapped. For any region build, the resources of a flock (e.g., resources corresponding to services) may be bootstrapped to any suitable number of execution targets. In an embodiment, the bootstrap operation for a set of execution targets may be associated with a "phase," and possibly multiple phases may be defined. In an embodiment, an order in which the bootstrap operation is performed among a set of execution targets (e.g., ET-1 through ET-7, collectively referred to as "execution targets 502") may be defined and associated with a phase (e.g., phase 1), while another set of execution targets (e.g., ET-8 through ET-13, collectively referred to as "execution targets 504") may be defined and associated with a different phase (e.g., phase 2). In an embodiment, there may be a specified order in which the phases are executed. Thus, a phase may depend on the completion of another phase, an execution target may depend on the completion of one or more other execution targets, etc. In an embodiment, a data structure (e.g., phase data structure 506) may be generated by CIOS 102 of FIG. 1 to identify a set of phases (e.g., one or more) in which bootstrap of a given service is performed across multiple execution targets, where each phase is associated with one or more execution targets. Any suitable number of data structures may be generated to identify the set of execution targets (e.g., execution target 502) and the order in which bootstrap of a service is performed across those sets of execution targets.

[0097] In one embodiment, CIOS 102 of Figure 1 may maintain any suitable number of data structures to maintain dependencies between flocks, phases, execution targets, etc. In one embodiment, CIOS 102 may generate a build dependency graph, such as the build dependency graph described in more detail in connection with Figure 8.

[0098] Figure 6 illustrates an example set of code segments illustrating several dependencies between services, according to at least one embodiment. The dependencies in Figure 6 can be determined from one or more flock configs. For example, any suitable number of dependencies discussed in connection with Figure 6 can be determined from the code segments of flock config 600.

[0099] Code segment 602 illustrates one type of dependency (called a "required capability"). The resource block of code segment 602 requests that a given source be read (e.g., "dns_a_record") and that the results be exported with a local name of "my dns". Although not shown, additional statements may be provided within the resource block of code segment 602. Code segment 602 illustrates a situation in which flock config 600 includes a dependency on a required capability. That is, code segment 602 does not provide a fallback plan if resource "dns_a_record" does not exist. Thus, when statically analyzed (e.g., parsed and evaluated with a set of predefined rules), code segment 602 can cause a multi-flock orchestrator (e.g., MFOs 106, 206, and 310 in FIGS. 1-3) to identify a dependency on "dns_a_record" (e.g., a resource associated with another flock config, not shown). Because this dependency indicates a required capability, MFO can be configured to ensure that the resource “dns_a_record” exists (e.g., using a node in the build dependency graph) before instructing the bootstrap task to occur using flock config 600.

[0100] Code segment 604 illustrates another type of dependency (referred to as an "optional capability"). To break a circular dependency (e.g., a load balancer service requires the functionality of a DNS service and vice versa), at least one of the corresponding flock configs can be generated and / or modified to enable its application and infrastructure to run without the other service. For example, a DNS flock config (e.g., flock config 600) can be defined and / or modified to enable the corresponding resource to be bootstrapped without a load balancer. If a load balancer is not present, the DNS can be configured to forward all traffic to a single host. However, if a load balancer is present, the DNS infrastructure and artifacts are redeployed, thereby enabling the final production configuration of the DNS service to be achieved. A flock config that provides an alternative bootstrap option when a dependent capability (e.g., a capability that the flock config depends on) is not present can be said to include an optional dependency on the corresponding resource (e.g., a capability of another service). A capability on which a resource in the flock config 600 depends may be called an "optional capability."

[0101] For example, code segment 604 defines a capability "dns_a_record". This code segment then requests that the resource "dns_a_record" be read and exported as "my dns". The MFO may identify code segment 604 as indicating an optional dependency based at least in part on identifying dns_a_record as an optional capability. In an embodiment, the MFO may be configured to identify dns_a_record as an optional capability based at least in part on identifying a tenancy and / or a conditional statement for creating that resource. For example, the MFO may be configured to identify line 606 as indicating an optional dependency on capability "dns_a_record" because a tenancy operator (e.g., operator "?") is used. In this case, the resource of code segment 604 may be identified as optional because the "count" (or for_each) field depends on the results of a capability data source. The code segment 604 specifies that if the capability "dns_a_record" exists then the resource of the code segment 604 is created, otherwise the resource of the code segment 604 is not created.

[0102] Code segment 608 illustrates another example dependency. Although code segment 608 may appear similar to the optional dependency discussed above in connection with code segment 604, code segment 608 may be identified as including a mandatory dependency. Code segment 608 illustrates an example of a non-counter. Code segment 608 indicates that a resource "dns_a_record" is to be created, and thus a mandatory dependency exists between code segment 608 and resource "dns_a_record." The load_balancer capability of code segment 608 may be identified as an optional dependency because code segment 608 tolerates a load balancer service that is not yet running by falling back to pointing DNS to a single host as shown in line 610.

[0103] In one embodiment, a conflicting dependency can be identified from a given flock configuration. For example, if a flock configuration includes a code segment that indicates a mandatory dependency on a given resource (e.g., a resource associated with another service) and a statement that specifies the ability to optionally handle the presence of that resource, the mandatory dependency can take precedence over the optional dependency. In one embodiment, the MFO can ensure that a code segment that corresponds to a given resource (e.g., a phase, an execution target, a flock, etc.) fully tolerates the absence of another resource / capability in order to specify the depended-on resource / capability as optional.

[0104] In some embodiments, MFO can be configured to enforce dependencies (e.g., global dependencies) that are not declared within a flock config. For example, there may be some important milestone (e.g., full connectivity to a region is established). In some cases, it may be assumed that each flock config in the system requires a corresponding capability without requiring that all flock configs explicitly or implicitly declare a dependency on that capability.

[0105] In an embodiment, a dependency can be specified implicitly or explicitly. For example, a flock config 600 may include a statement such as "depends_on=resourceA.name". Code segment 612 shows an explicit dependency at 614 where the dependency (e.g., a required dependency on capability 1234) is explicitly defined using the statement "depends_on". Any dependency that is specified that is not specified by an explicit dependency statement (e.g., using the identifier "depends_on") may be referred to as an "implicit dependency".

[0106] FIG. 7 illustrates another example set of code segments illustrating some statically determinable dependencies, according to at least one embodiment. A "statically determinable dependency" refers to a dependency (e.g., required, optional, implicit, explicit, etc.) that is determinable from the code prior to run time. Some statically determinable dependencies may be identified after edit time (e.g., after a flock config is edited as described in connection with steps 6-8 of FIG. 3) and prior to run time of region construction. Just as a dependency is statically determinable (e.g., determinable using static analysis (parsing and analysis) of one or more flock configs prior to run time), the value of the code segment from which the dependency is identified may be known locally (e.g., known within the scope of the applicable flock config).

[0107] For example, code segment 702 can be a code segment from which dependencies are statically determinable based at least in part on all values ​​(e.g., "my_capability" of parameter "name") being locally known. The value of parameter "name" is locally known within flock config 700 because the value is explicitly hard-coded within flock config 700. Thus, dependencies can be statically determinable from code segment 702 based at least in part on all variable / parameter values ​​being locally known.

[0108] Code segment 704 may declare a local variable "name" at 706 that is set to the value "my_capability." Because the variable "name" is declared locally at 706 (e.g., as a local variable with local scope to flock config 700), the corresponding value is known locally. In one embodiment, references to parameters defined in the "local" object (shown at 707) may be populated at edit time with the corresponding value specified in the local object. Thus, at edit time, references to "local.name" at 708 may be populated with the value "my_capability" (from 706), thereby allowing the identity of the dependency to be statically determined from code segment 704.

[0109] Code segment 710 may declare a local variable "names" at 712 that contains a list of values ​​(e.g., strings, identifiers, etc.) "my_capability1", "my_capability2", and "my_capability3". Because the variable "names" is declared locally at 713, the corresponding values ​​are known locally (e.g., within the scope of flock config 700). In one embodiment, references to parameters defined in a "local" object (as shown in 713) may be populated at edit time with the corresponding values ​​identified in the local object. Thus, references to "local.names" at 714 and 716 may be populated at edit time (from 713) with the values ​​"my_capability1", "my_capability2", and "my_capability3", thereby allowing the identities of dependencies from code segment 710 to be statically determinable.

[0110] 8 illustrates a portion of an example build dependency graph 800 according to at least one embodiment. Build dependency graph 800 may be an example of build dependency graph 338 of FIG. 3 generated by MFO 310.

[0111] The construction dependency graph 800 may be an acyclic directed graph. In an embodiment, the construction dependency graph 800 may include any suitable number of nodes (e.g., nodes 802-814). Each node of the construction dependency graph 800 may correspond to a flock config (or a portion of a flock config) or a capability. As shown, the nodes 802-814 correspond to a bootstrapping operation of a service (Service 1 in this example). Each of the nodes 802-814 may correspond to a "release" of a portion of Service 1 to which instructions may be sent to bootstrap at least a portion of the resources of Service 1. The nodes 802-814 and their corresponding order may be identified (e.g., by the MFO 310 of FIG. 3) based at least in part on performing a static analysis of the flock configs of the various services, including one or more flock configs associated with Service 1 (and the corresponding flock configs from which dependencies on capabilities 1-7 are identified). The construction dependency graph 800 shown in FIG. 8 may be a portion of a much larger graph. Nodes 802-814 may, among other things, define the order in which one or more resources (e.g., infrastructure components, software artifacts, etc.) are bootstrapped (e.g., provisioned, deployed, etc.) within a region (e.g., one or more data centers corresponding to the region). Nodes 802-814 are intended to correspond to particular phases (also identified within a flock config associated with service 1).

[0112] Capability nodes 1-7 (capabilities 1-7 shown in FIG. 8, each of which is an example of a required capability) may indicate capabilities upon which a particular bootstrap task of service 1 depends. Node 802 may indicate a starting node for bootstrapping service 1. In one embodiment, node 802 may depend on one or more capabilities being available (e.g., as indicated by capability nodes 1-3). In one embodiment, the dependency relationship between node 802 and capabilities 1-3 indicates that bootstrapping of service 1 cannot begin until capabilities 1-3 are available.

[0113] Once the MFO determines (e.g., from data provided by capability service 318 of FIG. 3) that capabilities 1-3 are available, the MFO may scan to node 804, where a subset of resources (e.g., infrastructure components) are identified as needing to be provisioned. In one embodiment, node 804 may be identified as depending on capability 6. Thus, the MFO may wait until it determines (from data provided by capability service 318) that capability 6 is available and then send instructions to CIOS central 304 of FIG. 3 to bootstrap the resources corresponding to node 804.

[0114] In one embodiment, the MFO may traverse up to nodes 806 and 812. Node 812 may indicate a dependency on capabilities 4 and 5. The MFO may wait for notification that capabilities 4 and 5 are available before proceeding past node 812.

[0115] Meanwhile, the MFO may instruct CIOS Central 304 to perform a bootstrap operation corresponding to node 806. In one embodiment, these tasks may include deploying a portion of Service 1 (e.g., Application 1). The MFO may then proceed to node 808. In one embodiment, node 808 may correspond to a task that includes deploying a portion of Service 1 (e.g., Application 2). Node 808 may indicate a dependency on Capability 7. The MFO may wait for notification that the capability is available before proceeding to node 810.

[0116] At node 810, the MFO may instruct CIOS Central 304 to perform a bootstrap operation corresponding to node 810. In one embodiment, node 810 may correspond to a task that includes deploying a portion of Service 1 (e.g., Application 3). The MFO may then proceed to node 814.

[0117] Node 814 may be dependent on the completion of operations corresponding to nodes 812 and 810. In one embodiment, node 814 may be associated with publishing a capability indicating that bootstrapping of service 1 (e.g., phase 1 bootstrapping) is complete. In one embodiment, when the MFO determines that operations corresponding to nodes 812 and 810 are complete, it may publish a capability to the capability service 318 indicating that bootstrap service 1 (in phase 1) is complete.

[0118] Nodes 802-814 illustrate an example where the MFO 106 has identified that a particular flock config (e.g., a flock config corresponding to service 1) requires multiple releases due to dependencies found in the graph. As a result, the MFO 106 can perform multiple releases corresponding to instructions for any suitable combination of nodes 802-814 to the CIOS central 304 for a given flock config to break the circular dependencies identified in the graph.

[0119] 8, defining any suitable number of bootstrap tasks corresponding to any suitable number of flocks, across any suitable number of execution targets, and according to any suitable number of phases. Using the static analysis techniques disclosed herein to generate the construction dependency graph 800, the MFO can ensure the order in which bootstrap tasks are performed by traversing the graph (e.g., between flocks, across execution targets, according to phase), thereby enabling the MFO to enforce any dependencies identified by the static analysis described herein.

[0120] The MFO can be configured to identify circular dependencies (e.g., service A needs service B and vice versa) via the build dependency graph 800. If cycles exist and are identified via static flock analysis and / or the graph, the MFO can be configured to notify any appropriate service teams that changes need to be made to the corresponding flock configurations to correct those circular dependencies. Although not shown, an orphaned node that does not connect to any other nodes and / or capabilities in the build dependency graph 800 can be identified as an orphaned node. The MFO 310 can identify capabilities that have been published in error (e.g., when a capability is published prematurely and the corresponding functionality is not actually available yet). The MFO 310 can detect from the graph that there are one or more instances of publication of the same capability. In an embodiment, any suitable number of such errors can be detected, and the MFO 310 (or another suitable component, such as CIOS Central 304) can be configured to notify or otherwise present this information to a user (e.g., via electronic notification, a user interface, etc.). In one embodiment, MFO 310 may be configured to force delete / recreate resources to break circular dependencies and may redirect CIOS Central 304 to perform bootstrap operations for those resources and / or the corresponding block configs.

[0121] FIG. 9 illustrates an example method 900 for bootstrapping a number of services within a region based at least in part on one or more dependencies identified by performing a static analysis of one or more flock configuration files, according to at least one embodiment. The method 900 may be performed by one or more components of the cloud infrastructure orchestration service 102 of FIG. 1 (e.g., the multi-flock orchestrator 106 of FIG. 1). A computer-readable storage medium includes computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform 900 800. The method 900 may be performed in any suitable order. It should be understood that the method 900 may include more or fewer steps than those shown in FIG. 9.

[0122] Method 900 may begin at 902 with obtaining a number of configuration files corresponding to a number of services to be bootstrapped into a region. In one embodiment, the number of configuration files provide data (e.g., resource definitions, capability definitions, phase definitions, execution target definitions, etc.) from which a bootstrap task can be identified for bootstrapping a number of services in the region.

[0123] At 904, one or more dependencies between respective services of the plurality of services may be identified based at least in part on performing operations for parsing the plurality of configuration files. For example, a static analysis (e.g., one or more parsing and analysis according to a set of predefined rules) may be performed on the configuration files to identify required and / or optional dependencies (any of which may be explicitly or implicitly defined) between flocks (e.g., between flocks / flock resources and / or between phases and / or execution targets).

[0124] At 906, a build dependency graph (e.g., build dependency graph 800 of FIG. 8 ) may be generated based at least in part on the parsing at 904. In an embodiment, the build dependency graph may maintain the identified dependency or dependencies. The build dependency graph may be a data structure that identifies multiple configuration files and one or more dependencies and indicates a corresponding order in which bootstrapping tasks are performed.

[0125] At 908, a provisioning and deployment manager (e.g., CIOS Central 304 of FIG. 3) can be progressively instructed (e.g., via multiple releases and corresponding sets of instructions) to execute a bootstrap task based at least in part on the traversal of the build dependency graph. In one embodiment, the bootstrap task can be executed such that multiple services are bootstrapped into a region according to one or more identified dependencies.

[0126] Cloud Service Infrastructure Architecture Examples As mentioned above, Infrastructure as a Service (IaaS) is one particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider can also provide various services (e.g., billing, monitoring, logging, load balancing, clustering, etc.) that accompany those infrastructure components. Thus, these services can be policy-driven, so that an IaaS user can enforce policies to drive load balancing to maintain application availability and performance.

[0127] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet, and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, and more.

[0128] In most cases, the cloud computing model will require the involvement of a cloud provider, which can be, but is not necessarily, a third-party service that specializes in providing IaaS (e.g., offering, renting, selling). An entity may choose to deploy a private cloud, thereby becoming its own provider of infrastructure services.

[0129] In one example, an IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider below the hypervisor layer (e.g., server, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling (OS), middleware, and / or application deployment (e.g., self-service virtual machines (which can be spun up on demand, etc.)).

[0130] In one example, IaaS provisioning refers to obtaining a computer or virtual host for use and then installing the required libraries or services on it. In many cases, deployment does not include provisioning, which may need to occur first.

[0131] In some cases, there are two different challenges in IaaS provisioning. First, there is the initial challenge of provisioning an initial set of instructions before anything can be put into production. Second, there is the challenge of evolving the existing infrastructure (e.g. adding new services, modifying services, removing services, etc.) after everything has been provisioned. In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, one or more configuration files can define the infrastructure (e.g. what components are needed and how they interact). Thus, the overall topology of the infrastructure (e.g. what resources depend on which resources and how each of them work together) can be defined declaratively. In some cases, after the topology is defined, workflows can be generated that create and / or manage the different components described in the configuration files.

[0132] In one example, an infrastructure can have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shared computing resources), also referred to as a core network. In one example, there may also be one or more inbound / outbound traffic group rules that are provisioned to define how the network's inbound / outbound traffic is configured and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., can also be provisioned. The infrastructure can evolve over time as more and more infrastructure elements are required and / or added.

[0133] In some cases, continuous deployment techniques can be employed to enable deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is required to be deployed to one or more, often many, different production environments (e.g., across various different geographic locations, sometimes spanning the globe). In some examples, however, the infrastructure onto which the code will be deployed must first be set up. In some cases, provisioning can be done manually, provisioning tools can be used to provision the resources, and / or deployment tools can be used to deploy the code once the infrastructure has been provisioned.

[0134] 10 is a block diagram 1000 illustrating an example platform of an IaaS architecture in accordance with at least one embodiment. A service operator 1002 can be communicatively coupled to a secure host tenancy 1004, which can include a virtual cloud network (VCN) 1006 and a secure host subnet 1008. In one example, the service operator 1002 can use one or more customer computing devices, which can be portable handheld devices (e.g., iPhones, mobile phones, iPads, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass head-mounted displays) running software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and capable of Internet, email, short message service (SMS), Blackberry, or other communications protocols. Alternatively, the customer computing devices may be general purpose personal computers, including, for example, personal and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The customer computing devices may be workstation computers running any of the various commercially available UNIX or UNIX-like operating systems, including, but not limited to, the various GNU / Linux operating systems, such as Google Chrome OS.Alternatively or additionally, the customer computing device may be any other electronic device, such as a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device), and / or a personal messaging device capable of communicating via VCN 1006 and / or an Internet-accessible network.

[0135] The VCN 1006 may include a local peering gateway (LPG) 1010 that may be communicatively coupled to an SSH VCN 1012 via an LPG 1010 that is included in a secure shell (SSH) VCN 1012. The SSH VCN 1012 may include an SSH subnet 1014, which may be communicatively coupled to a control plane VCN 1016 via an LPG 1010 that is included in the control plane VCN 1016. The SSH VCN 1012 may also be communicatively coupled to a data plane VCN 1018 via the LPG 1010. The control plane VCN 1016 and the data plane VCN 1018 may be included in a service tenancy 1019 that may be owned and / or operated by the IaaS provider.

[0136] The control plane VCN 1016 may include a control plane demilitarized zone (DMZ) tier 1020 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers may have limited duties and help thwart intrusions. Additionally, the DMZ tier 1020 may include one or more load balancer (LB) subnets 1022, a control plane application tier 1024 that may include application subnets 1026, a control plane data tier 1028 that may include a database (DB) subnet 1030 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1022 included in the control plane DMZ tier 1020 can be communicatively coupled to an application subnet 1026 included in the control plane application tier 1024 and an Internet gateway 1034 that can be included in the control plane VCN 1016, which can be communicatively coupled to a DB subnet 1030, a service gateway 1036 and a network address translation (NAT) gateway 1038 included in the control plane data tier 1028. The control plane VCN 1016 can include the service gateway 1036 and the NAT gateway 1038.

[0137] The control plane VCN 1016 can include a data plane mirrored application tier 1040 that can include an application subnet 1026. The application subnet 1026 included in the data plane mirrored application tier 1040 can include a virtual network interface controller (VNIC) 1042 capable of running a compute instance 1044. The compute instance 1044 can communicatively couple the application subnet 1026 of the data plane mirrored application tier 1040 to the application subnet 1026 that can be included in the data plane application tier 1046.

[0138] The data plane VCN 1018 can include a data plane application layer 1046, a data plane DMZ layer 1048, and a data plane data layer 1050. The data plane DMZ layer 1048 can include a LB subnet 1022 that can be communicatively coupled to an application subnet 1026 of the data plane application layer 1046 and an Internet gateway 1034 of the data plane VCN 1018. The application subnet 1026 can be communicatively coupled to a service gateway 1036 of the data plane VCN 1018 and a NAT gateway 1038 of the data plane VCN 1018. The data plane data layer 1050 can also include a DB subnet 1030 that can be communicatively coupled to the application subnet 1026 of the data plane application layer 1046.

[0139] The Internet gateways 1034 of the control plane VCN 1016 and the data plane VCN 1018 may be communicatively coupled to a metadata management service 1052, which may be communicatively coupled to the public Internet 1054. The public Internet 1054 may be communicatively coupled to NAT gateways 1038 of the control plane VCN 1016 and of the data plane VCN 1018. The service gateways 1036 of the control plane VCN 1016 and of the data plane VCN 1018 may be communicatively coupled to cloud services 1056.

[0140] In one example, a service gateway 1036 in the control plane VCN 1016 or in the data plane VCN 1018 can make application programming interface (API) calls to a cloud service 1056 without traversing the public Internet 1054. API calls from the service gateway 1036 to the cloud service 1056 can be one-way; that is, the service gateway 1036 can make API calls to the cloud service 1056 and the cloud service 1056 can send the requested data to the service gateway 1036. However, the cloud service 1056 cannot originate API calls to the service gateway 1036.

[0141] In one example, a secure host tenancy 1004 can be directly connected to an otherwise separable service tenancy 1019. A secure host subnet 1008 can communicate with an SSH subnet 1014 through an LPG 1010, which can enable bidirectional communication through otherwise separate systems. Connecting the secure host subnet 1008 to the SSH subnet 1014 can give the secure host subnet 1008 access to other entities in the service tenancy 1019.

[0142] The control plane VCN 1016 can enable users of the service tenancy 1019 to configure or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1016 can be deployed or otherwise used in the data plane VCN 1018. In one example, the control plane VCN 1016 can be separate from the data plane VCN 1018, and the data plane mirror application tier 1040 of the control plane VCN 1016 can communicate with the data plane application tier 1046 of the data plane VCN 1018 via a VNIC 1042, which can be included in the data plane mirror application tier 1040 and the data plane application tier 1046.

[0143] In one example, a user or customer of the system may make a request, for example, a create, read, update or delete (CRUD) operation, via the public Internet 1054, which may communicate the request to a metadata management service 1052. The metadata management service 1052 may communicate the request to the control plane VCN 1016 via an Internet gateway 1034. The request may be received by a LB subnet 1022 included in the control plane DMZ layer 1020. The LB subnet 1022 may determine that the request is valid, and in response to this determination, the LB subnet 1022 may send the request to an application subnet 1026 included in the control plane application layer 1024. If the validity of the request is verified and a call to the public Internet 1054 is required, the call to the public Internet 1054 may be sent to a NAT gateway 1038, which may make a call to the public Internet 1054. Memory that may be desired to be stored by the request may be stored in the DB subnet 1030.

[0144] In one example, a data plane mirror application layer 1040 can facilitate direct communication between the control plane VCN 1016 and the data plane VCN 1018. For example, it may be desired that configuration changes, updates, or other suitable modifications be applied to resources included in the data plane VCN 1018. The VNIC 1042 allows the control plane VCN 1016 to communicate directly with the resources included in the data plane VCN 1018, thereby enabling it to perform configuration changes, updates, or other suitable modifications to those resources.

[0145] In an embodiment, the control plane VCN 1016 and the data plane VCN 1018 may be included in the service tenancy 1019. In this case, a user or customer of the system may not own or operate the control plane VCN 1016 or the data plane VCN 1018. Instead, an IaaS provider may own or operate the control plane VCN 1016 and the data plane VCN 1018, both of which may be included in the service tenancy 1019. This embodiment may allow for network isolation that may prevent a user or customer from interacting with the resources of other users or other customers. This embodiment may also allow a user or customer of the system to store databases privately without having to rely on the public Internet 1054, which may not have the desired level of threat protection for storage.

[0146] In another embodiment, the LB subnet 1022 included in the control plane VCN 1016 can be configured to receive signals from the service gateway 1036. In this embodiment, the control plane VCN 1016 and the data plane VCN 1018 can be configured to be called by the IaaS provider's customers without calling the public Internet 1054. Customers of the IaaS provider may desire this embodiment because databases used by the customers can be controlled by the IaaS provider and stored on a service tenancy 1019 that is separable from the public Internet 1054.

[0147] 11 is a block diagram 1100 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1102 (e.g., service operator 1002 of FIG. 10) can be communicatively coupled to a secure host tenancy 1104 (e.g., secure host tenancy 1004 of FIG. 10), which can include a virtual cloud network (VCN) 1106 (e.g., VCN 1006 of FIG. 10) and a secure host subnet 1108 (e.g., secure host subnet 1008 of FIG. 10). The VCN 1106 can include a local peering gateway (LPG) 1110 (e.g., LPG 1010 of FIG. 10), which can be communicatively coupled to an SSH VCN 1112 (e.g., SSH VCN 1012 of FIG. 10) via an LPG 1010 included in a secure shell (SSH) VCN 1112. SSH VCN 1112 can include an SSH subnet 1114 (e.g., SSH subnet 1014 in FIG. 10), which can be communicatively coupled to a control plane VCN 1116 (e.g., control plane VCN 1016 in FIG. 10) via an LPG 1110 included in the control plane VCN 1116. The control plane VCN 1116 can be included in a service tenancy 1119 (e.g., service tenancy 1019 in FIG. 10), and the data plane VCN 1118 (e.g., data plane VCN 1018 in FIG. 10) can be included in a customer tenancy 1121, which can be owned or operated by a user or customer of the system.

[0148] The control plane VCN 1116 may include a control plane DMZ tier 1120 (e.g., control plane DMZ tier 1020 of FIG. 10) that may include a LB subnet 1122 (e.g., LB subnet 1022 of FIG. 10), a control plane application tier 1124 (e.g., control plane application tier 1024 of FIG. 10) that may include an application subnet 1126 (e.g., application subnet 1026 of FIG. 10), and a control plane data tier 1128 (e.g., control plane data tier 1028 of FIG. 10) that may include a database (DB) subnet 1130 (e.g., similar to DB subnet 1030 of FIG. 10). The LB subnet 1122 included in the control plane DMZ tier 1120 can be communicatively coupled to an application subnet 1126 included in the control plane application tier 1124 and to an Internet gateway 1134 (e.g., Internet gateway 1034 of FIG. 10) that may be included in the control plane VCN 1116, and the application subnet 1126 can be communicatively coupled to a DB subnet 1130 included in the control plane data tier 1128 and to a service gateway 1136 (e.g., service gateway 1036 of FIG. 10) and a network address translation (NAT) gateway 1138 (e.g., NAT gateway 1038 of FIG. 10). The control plane VCN 1116 can include the service gateway 1136 and the NAT gateway 1138.

[0149] The control plane VCN 1116 can include a data plane mirrored application tier 1140 (e.g., data plane mirrored application tier 1040 of FIG. 10 ), which can include an application subnet 1126. The application subnet 1126 included in the data plane mirrored application tier 1140 can include a virtual network interface controller (VNIC) 1142 (e.g., VNIC of 1042) that can run a compute instance 1144 (e.g., similar to compute instance 1044 of FIG. 10 ). The compute instance 1144 can facilitate communication between the application subnet 1126 of the data plane mirrored application tier 1140 and the application subnet 1126 that can be included in the data plane application tier 1146 (e.g., data plane application tier 1046 of FIG. 10 ) via the VNIC 1142 included in the data plane mirrored application tier 1140 and the VNIC 1142 included in the data plane application tier 1146.

[0150] The Internet gateway 1134 included in the control plane VCN 1116 can be communicatively coupled to a metadata management service 1152 (e.g., metadata management service 1052 of FIG. 10), which can be communicatively coupled to a public Internet 1154 (e.g., public Internet 1054 of FIG. 10). The public Internet 1154 can be communicatively coupled to a NAT gateway 1138 included in the control plane VCN 1116. The service gateway 1136 included in the control plane VCN 1116 can be communicatively coupled to cloud services 1156 (e.g., cloud services 1056 of FIG. 10).

[0151] In one example, the data plane VCN 1118 may be included in the customer tenancy 1121. In this case, the IaaS provider may provide each customer with a control plane VCN 1116, and the IaaS provider may configure a unique compute instance 1144 included in the service tenancy 1119 for each customer. Each compute instance 1144 may enable communication between the control plane VCN 1116 included in the service tenancy 1119 and the data plane VCN 1118 included in the customer tenancy 1121. The compute instance 1144 may enable resources provisioned in the control plane VCN 1116 included in the service tenancy 1119 to be deployed or otherwise used in the data plane VCN 1118 included in the customer tenancy 1121.

[0152] In another example, an IaaS provider customer may have a database that resides within customer tenancy 1121. In this example, control plane VCN 1116 may include a data plane mirrored application tier 1140 that may include application subnet 1126. The data plane mirrored application tier 1140 may reside in data plane VCN 1118, but the data plane mirrored application tier 1140 may not reside in data plane VCN 1118. That is, the data plane mirrored application tier 1140 may have access to customer tenancy 1121, but the data plane mirrored application tier 1140 may not reside in data plane VCN 1118 or be owned or operated by the IaaS provider customer. The data plane mirrored application tier 1140 may be configured to make calls to the data plane VCN 1118, but may not be configured to make calls to any entities included in the control plane VCN 1116. A customer may wish to deploy or otherwise use resources in the data plane VCN 1118 that have been provisioned in the control plane VCN 1116, and the data plane mirror application tier 1140 can facilitate the desired deployment or other use of the customer's resources.

[0153] In one embodiment, an IaaS provider's customer can filter the data plane VCN 1118. In this embodiment, the customer can determine what the data plane VCN 1118 can access, and the customer can limit access to the public internet 1154 from the data plane VCN 1118. It may not be possible for the IaaS provider to filter or otherwise control the access of the data plane VCN 1118 to any external networks or databases. The application of filters and controls by the customer to the data plane VCN 1118 contained in the customer tenancy 1121 can facilitate isolating the data plane VCN 1118 from other customers and from the public internet 1154.

[0154] In an embodiment, cloud services 1156 can be invoked by the service gateway 1136 to access services that may not be on the public internet 1154, on the control plane VCN 1116, or on the data plane VCN 1118. The connection between the cloud services 1156 and the control plane VCN 1116 or the data plane VCN 1118 may not be live or continuous. The cloud services 1156 can be on different networks owned or operated by the IaaS provider. The cloud services 1156 can be configured to receive calls from the service gateway 1136 and not receive calls from the public internet 1154. Some cloud services 1156 can be isolated from other cloud services 1156, and the control plane VCN 1116 can be isolated from cloud services 1156 that may not be in the same region as the control plane VCN 1116. For example, the control plane VCN 1116 may be located in "Region 1" and the cloud service "Deployment 10" may be located in Region 1 and Region 2. If a call is made to deployment 10 by a service gateway 1136 included in a control plane VCN 1116 located in region 1, the call can be sent to deployment 10 in region 1. In this example, control plane VCN 1116, or deployment 10 in region 1, cannot be communicatively coupled or otherwise communicate with deployment 10 in region 2.

[0155] 12 is a block diagram 1200 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1202 (e.g., service operator 1002 of FIG. 10) can be communicatively coupled to a secure host tenancy 1204 (e.g., secure host tenancy 1004 of FIG. 10), which can include a virtual cloud network (VCN) 1206 (e.g., VCN 1006 of FIG. 10) and a secure host subnet 1208 (e.g., secure host subnet 1008 of FIG. 10). VCN 1206 can include an LPG 1210 (e.g., LPG 1010 of FIG. 10), which can be communicatively coupled to an SSH VCN 1212 (e.g., SSH VCH 1012 of FIG. 10) via an LPG 1210 included in SSH VCN 1212. SSH VCN 1212 can include SSH subnet 1214 (e.g., SSH subnet 1014 in FIG. 10), which can be communicatively connected to a control plane VCN 1216 (e.g., control plane VCN 1016 in FIG. 10) via an LPG 1210 included in control plane VCN 1216 and to a data plane VCN 1218 (e.g., data plane 1018 in FIG. 10) via an LPG 1210 included in data plane VCN 1218. Control plane VCN 1216 and data plane VCN 1218 can be included in service tenancy 1219 (e.g., service tenancy 1019 in FIG. 10).

[0156] The control plane VCN 1216 may include a control plane DMZ tier 1220 (e.g., control plane DMZ tier 1020 of FIG. 10) that may include a load balancer (LB) subnet 1222 (e.g., LB subnet 1022 of FIG. 10), a control plane application tier 1224 (e.g., control plane application tier 1024 of FIG. 10) that may include an application subnet 1226 (e.g., similar to application subnet 1026 of FIG. 10), and a control plane data tier 1228 (e.g., control plane data tier 1028 of FIG. 10) that may include a DB subnet 1230. LB subnet 1222 included in control plane DMZ tier 1220 can be communicatively coupled to application subnet 1226 included in control plane application tier 1224 and to an Internet gateway 1234 (e.g., Internet gateway 1034 in FIG. 10) that may be included in control plane VCN 1216, and application subnet 1226 can be communicatively coupled to DB subnet 1230 included in control plane data tier 1228 and to a service gateway 1236 (e.g., service gateway in FIG. 10) and a network address translation (NAT) gateway 1238 (e.g., NAT gateway 1038 in FIG. 10). Control plane VCN 1216 can include service gateway 1236 and NAT gateway 1238.

[0157] The data plane VCN 1218 can include a data plane application layer 1246 (e.g., data plane application layer 1046 in FIG. 10), a data plane DMZ layer 1248 (e.g., data plane DMZ layer 1048 in FIG. 10), and a data plane data layer 1250 (e.g., data plane data layer 1050 in FIG. 10). The data plane DMZ layer 1248 can include a LB subnetwork 1222 that can be communicatively coupled to a trusted application subnetwork 1260 and an untrusted application subnetwork 1262 of the data plane application layer 1246 and an Internet gateway 1234 included in the data plane VCN 1218. The trusted application subnetwork 1260 can be communicatively coupled to a service gateway 1236 included in the data plane VCN 1218, a NAT gateway 1238 included in the data plane VCN 1218, and a DB subnetwork 1230 included in the data plane data layer 1250. The untrusted application subnet 1262 can be communicatively coupled to a service gateway 1236 included in the data plane VCN 1218 and a DB subnet 1230 included in the data plane data layer 1250. The data plane data layer 1250 can include a DB subnet 1230 that can be communicatively coupled to a service gateway 1236 included in the data plane VCN 1218.

[0158] The untrusted application subnet 1262 may include one or more primary VNICs 1264(1)-(N) communicatively coupled to tenant virtual machines (VMSs) 1266(1)-(N). Each tenant VM 1266(1)-(N) may be communicatively coupled to a respective application subnet 1267(1)-(N) that may be included in a respective container egress VCN 1268(1)-(N) that may be included in a respective customer tenancy 1270(1)-(N). Each secondary VNIC 1272(1)-(N) may facilitate communication between the untrusted application subnet 1262 included in the data plane VCN 1218 and the application subnet included in the container egress VCN 1268(1)-(N). Each container egress VCN 1268(1)-(N) may include a NAT gateway 1238 communicatively coupled to the public Internet 1254 (e.g., public Internet 1054 of FIG. 10).

[0159] An Internet gateway 1234 included in the control plane VCN 1216 and included in the data plane VCN 1218 may be communicatively coupled to a metadata management service 1252 (e.g., metadata management system 1052 of FIG. 10), which may be communicatively coupled to the public Internet 1254. The public Internet 1254 may be communicatively coupled to a NAT gateway 1238 included in the control plane VCN 1216 and included in the data plane VCN 1218. A service gateway 1236 included in the control plane VCN 1216 and included in the data plane VCN 1218 may be communicatively coupled to cloud services 1256.

[0160] In one embodiment, data plane VCN 1218 can be integrated with customer tenancy 1270. This integration can be useful or desirable for an IaaS provider's customer in some cases, such as when support is desired when executing code. A customer can provide code for execution that may be disruptive, communicate with other customer resources, or otherwise cause undesirable effects. In response, the IaaS provider can determine whether to execute the code provided to the IaaS provider by the customer.

[0161] In one example, an IaaS provider customer may be granted temporary network access to the IaaS provider and may request to add functionality to data plane application layer 1246. The code to perform that functionality may be executed in VMs 1266(1)-(N), and the code may not be configured to run anywhere else on data plane VCN 1218. Each VM 1266(1)-(N) may be connected to one customer tenancy 1270. Each container 1271(1)-(N) contained in VM 1266(1)-(N) may be configured to execute this code. In this case, there may be double isolation (e.g., container 1271(1)-(N) executes code, and container 1271(1)-(N) may be at least contained in VM 1266(1)-(N) contained in untrusted application subnet 1262), which may help prevent malformed or otherwise undesirable code from damaging the IaaS provider's network or from damaging a different customer's network. Containers 1271(1)-(N) can be communicatively coupled to customer tenancy 1270 and can be configured to send or receive respective data to or from customer tenancy 1270. Containers 1271(1)-(N) cannot be configured to send or receive data to or from any other entities in data plane VCN 1218. Upon completion of code execution, the IaaS provider can disable or otherwise discard containers 1271(1)-(N).

[0162] In one embodiment, trusted application subnet 1260 may execute code owned or operated by the IaaS provider. In this embodiment, trusted application subnet 1260 may be communicatively coupled to DB subnet 1230 and may be configured to perform CRUD operations on DB subnet 1230. Untrusted application subnet 1262 may be communicatively connected to DB subnet 1230, but in this embodiment, the untrusted application subnet may be configured to perform read operations on DB subnet 1230. Containers 1271(1)-(N) may be included in each customer's VMs 1266(1)-(N) and may execute code from the customer, but may not be communicatively coupled to DB subnet 1230.

[0163] In other embodiments, the control plane VCN 1216 and the data plane VCN 1218 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1216 and the data plane VCN 1218. However, communication may be indirect in at least one manner. An IaaS provider may configure an LPG 1210 that may facilitate communication between the control plane VCN 1216 and the data plane VCN 1218. In another example, the control plane VCN 1216 or the data plane VCN 1218 may make a call to a cloud service 1256 through a service gateway 1236. For example, a call from the control plane VCN 1216 to the cloud service 1256 may include a request for a service that may communicate with the data plane VCN 1218.

[0164] 13 is a block diagram 1300 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1302 (e.g., service operator 1002 of FIG. 10) can be communicatively connected to a secure host tenancy 1304 (e.g., secure host tenancy 1004 of FIG. 10), which can include a virtual cloud network (VCN) 1306 (e.g., VCN 1006 of FIG. 10), and a second secure host subnet 1308 (e.g., secure host subnet 1008 of FIG. 10). VCN 1306 can include an LPG 1310 (e.g., LPG 1010 of FIG. 10), which can be communicatively connected to an SSH VCN 1312 (e.g., SSH VCN 1012 of FIG. 10) via an LPG 1310 included in SSH VCN 1312. SSH VCN 1312 can include an SSH subnet 1314 (e.g., SSH subnet 1014 in FIG. 10), which can be communicatively connected to a control plane VCN 1316 (e.g., control plane VCN 1016 in FIG. 10) via an LPG 1310 included in control plane VCN 1316 and to a data plane VCN 1318 (e.g., data plane 1018 in FIG. 10) via an LPG 1310 included in data plane VCN 1318. Control plane VCN 1316 and data plane VCN 1318 can be included in service tenancy 1319 (e.g., service tenancy 1019 in FIG. 10).

[0165] The control plane VCN 1316 may include a control plane DMZ layer 1320 (e.g., the control plane DMZ layer 1020 of FIG. 10) which may include a LB subnet 1322 (e.g., the LB subnet 1022 of FIG. 10), a control plane application layer 1324 (e.g., the control plane application layer 1024 of FIG. 10) which may include an application subnet 1326 (e.g., the application subnet 1026 of FIG. 10), and a control plane data layer 1328 (e.g., the control plane data layer 1028 of FIG. 10) which may include a DB subnet 1330 (e.g., the DB subnet 1230 of FIG. 12). The LB subnet 1322 included in the control plane DMZ tier 1320 can be communicatively coupled to an application subnet 1326 included in the control plane application tier 1324 and an Internet gateway 1334 (e.g., Internet gateway 1034 in FIG. 10) that may be included in the control plane VCN 1316, and the application subnet 1326 can be communicatively coupled to a DB subnet 1330 included in the control plane data tier 1328 and a service gateway 1336 (e.g., service gateway in FIG. 10) and a network address translation (NAT) gateway 1338 (e.g., NAT gateway 1038 in FIG. 10). The control plane VCN 1316 can include the service gateway 1336 and the NAT gateway 1338.

[0166] The data plane VCN 1318 can include a data plane application layer 1346 (e.g., data plane application layer 1046 in FIG. 10), a data plane DMZ layer 1348 (e.g., data plane DMZ layer 1048 in FIG. 10), and a data plane data layer 1350 (e.g., data plane data layer 1050 in FIG. 10). The data plane DMZ layer 1348 can include a LB subnetwork 1322 that can be communicatively coupled to a trusted application subnetwork 1360 (e.g., trusted application subnetwork 1260 in FIG. 12) and an untrusted application subnetwork 1362 (e.g., untrusted application subnetwork 1262 in FIG. 12) of the data plane application layer 1346 and an Internet gateway 1334 included in the data plane VCN 1318. The trusted application subnet 1360 can be communicatively coupled to a service gateway 1336 included in the data plane VCN 1318, a NAT gateway 1338 included in the data plane VCN 1318, and a DB subnet 1330 included in the data plane data layer 1350. The untrusted application subnet 1362 can be communicatively coupled to the service gateway 1336 included in the data plane VCN 1318 and the DB subnet 1330 included in the data plane data layer 1350. The data plane data layer 1350 can include the DB subnet 1330 that can be communicatively coupled to the service gateway 1336 included in the data plane VCN 1318.

[0167] The untrusted application subnet 1362 may include primary VNICs 1364(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1366(1)-(N) that are in the untrusted application subnet 1362. Each tenant VM 1366(1)-(N) may execute code in a respective container 1367(1)-(N) and may be communicatively coupled to an application subnet 1326 that may be included in a data plane application tier 1346 that may be included in a container egress VCN 1368. Each secondary VNIC 1372(1)-(N) may facilitate communication between the untrusted application subnet 1362 included in the data plane VCN 1318 and the application subnet included in the container egress VCN 1368. The container egress VCN may include a NAT gateway 1338 that may be communicatively coupled to the public internet 1354 (e.g., public internet 1054 of FIG. 10).

[0168] An Internet gateway 1334 included in the control plane VCN 1316 and included in the data plane VCN 1318 may be communicatively coupled to a metadata management service 1352 (e.g., metadata management system 1052 of FIG. 10), which may be communicatively coupled to the public Internet 1354. The public Internet 1354 may be communicatively coupled to a NAT gateway 1338 included in the control plane VCN 1316 and included in the data plane VCN 1318. A service gateway 1336 included in the control plane VCN 1316 and included in the data plane VCN 1318 may be communicatively coupled to cloud services 1356.

[0169] In one example, the pattern illustrated by the architecture of block diagram 1300 of FIG. 13 can be considered an exception to the pattern illustrated by the architecture of block diagram 1200 of FIG. 12 and may be desirable for customers of an IaaS provider when the IaaS provider cannot communicate directly with the customer (e.g., disconnected region). Each container 1367(1)-(N) contained in a VM 1366(1)-(N) for each customer is accessible in real time by that customer. The containers 1367(1)-(N) can be configured to make calls to a respective secondary VNIC 1372(1)-(N) contained in an application subnet 1326 of a data plane application layer 1346 that can be contained in a container egress VCN 1368. The secondary VNIC 1372(1)-(N) can send the calls to a NAT gateway 1338 that can send the calls to the public Internet 1354. In this example, containers 1367(1)-(N) that are accessible to a customer in real time are segregated from control plane VCN 1316 and are segregated from other entities contained within data plane VCN 1318. Containers 1367(1)-(N) are also segregated from resources from other customers.

[0170] In another example, a customer can use container 1367(1)-(N) to invoke cloud service 1356. In this example, the customer can execute code in container 1367(1)-(N) that requests a service from cloud service 1356. Container 1367(1)-(N) can send the request to secondary VNIC 1372(1)-(N), which can send the request to a NAT gateway, which can send the request to public Internet 1354. Public Internet 1354 can send the request via Internet gateway 1334 to LB subnet 1322 included in control plane VCN 1316. In response to determining that the request is valid, the LB subnet can send the request to application subnet 1326, which can send the request via service gateway 1336 to cloud service 1356.

[0171] It should be understood that the IaaS architectures 1000, 1100, 1200, 1300 depicted in the figures may have components other than those depicted, and the depicted embodiments are only some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS systems may have more or fewer components than depicted in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0172] In one embodiment, the IaaS system described herein may include a set of application, middleware, and database service offerings that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. One example of such an IaaS system is Oracle Cloud Infrastructure (OCI), offered by the assignee of the present application.

[0173] 14 illustrates an exemplary computer system 1400 in which various embodiments may be implemented. The system 1400 may be used to implement any of the computer systems described above. As shown, the computer system 1400 includes a processing unit 1404 that communicates with several peripheral subsystems via a bus subsystem 1402. These peripheral subsystems may include a processing acceleration unit 1406, an I / O subsystem 1408, a storage subsystem 1418, and a communication subsystem 1424. The storage subsystem 1418 includes a tangible computer readable storage medium 1422 and a system memory 1410.

[0174] The bus subsystem 1402 provides a mechanism for allowing the various components and subsystems of the computer system 1400 to communicate with each other as desired. Although the bus subsystem 1402 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may use multiple buses. The bus subsystem 1402 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0175] A processing unit 1404, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1400. The processing unit 1404 may include one or more processors. These processors may include single-core or multi-core processors. In one embodiment, the processing unit 1404 may be implemented as one or more independent processing units 1432 and / or 1434, with each processing unit including a single-core or multi-core processor. In other embodiments, the processing unit 1404 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0176] In various embodiments, the processing unit 1404 may execute various programs in response to program code and may maintain multiple parallel executing programs or processes. At any given time, some or all of the program code being executed may reside within the processor 1404 and / or within the storage subsystem 1418. With appropriate programming, the processor 1404 may provide the various functions discussed above. The computer system 1400 may further include a processing acceleration unit 1406, which may include a digital signal processor (DSP), special purpose processor, and / or the like.

[0177] The I / O subsystem 1408 can include user interface input devices and user interface output devices. User interface input devices can include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can include motion sensing and / or gesture recognition devices, such as, for example, a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device, such as a Microsoft Xbox® 360 game controller, through a natural user interface using gestures and speech commands. User interface input devices can also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye activity from a user (e.g., "blinking" when taking pictures and / or selecting menus) and translates the eye gestures as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0178] User interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser distance meters, and eye-tracking systems. Additionally, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0179] User interface output devices may include a display subsystem, non-visual displays such as indicator lights or audio output devices. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), a liquid crystal display (LCD) or a plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include any type of possible device and mechanism for outputting information from computer system 1400 to a user or to another computer. For example, user interface output devices may include a variety of display devices that visually convey text, graphics and audio / video information, such as, but not limited to, monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices and modems.

[0180] Computer system 1400 may include a storage subsystem 1418 that provides a tangible, non-transitory computer-readable storage medium for storing software and data structures that provide functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of processing unit 1404, provide the functionality described above. Storage subsystem 1418 may also provide a repository for storing data used in accordance with the present disclosure.

[0181] As shown in Figure 14, the storage subsystem 1418 can include various components including a system memory 1410, a computer readable storage medium 1422, and a computer readable storage medium reader 1420. The system memory 1410 can store program instructions that are loadable and executable by the processing unit 1404. The system memory 1410 can also store data used during execution of the instructions and / or data generated during execution of the program instructions. A variety of different types of programs can be loaded into the system memory 1410, including, but not limited to, client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), virtual machines, containers, and the like.

[0182] The system memory 1410 may also store an operating system 1416. Examples of the operating system 1416 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including but not limited to the GNU / Linux operating system, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems. In certain implementations in which the computer system 1400 runs one or more virtual machines, the virtual machines, along with their guest operating systems (GOS), may be loaded into the system memory 1410 and executed by one or more processors or cores of the processing unit 1404.

[0183] The system memory 1410 may be provided in different configurations depending on the type of computer system 1400. For example, the system memory 1410 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, the system memory 1410 may include a basic input / output system (BIOS), which contains the basic routines that help to transfer information between elements within the computer system 1400, such as during start-up.

[0184] Computer-readable storage medium 1422 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information for use by computer system 1400, including instructions executable by processing unit 1404 of computer system 1400.

[0185] The computer readable storage medium 1422 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storage and / or transmission of information. This may include tangible computer readable storage media, such as RAM, ROM, Electrically Erasable Programmable ROM (EEPROM), Flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other tangible computer readable media.

[0186] For example, the computer readable storage medium 1422 may include hard disk drives that read or write non-removable non-volatile magnetic media, magnetic disk drives that read or write removable non-volatile magnetic disks, and optical disk drives that read or write removable non-volatile optical disks such as CD ROMs, DVDs and Blu-Ray disks or other optical media. The computer readable storage medium 1422 may include, but is not limited to, Zip drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. The computer readable storage medium 1422 may also include solid state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid state ROMs, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computer system 1400.

[0187] Machine-readable instructions executable by one or more processors or cores of the processing unit 1404 may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM or flash memory, hard drives, floppy drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0188] The communications subsystem 1424 provides an interface to other computer systems and networks. The communications subsystem 1424 serves as an interface for receiving and transmitting data to and from systems other than the computer system 1400. For example, the communications subsystem 1424 may enable the computer system 1400 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1424 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 family standard, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1424 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0189] In some embodiments, the communications subsystem 1424 may also receive incoming communications in the form of structured and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc., on behalf of one or more users who may use the computer system 1400.

[0190] For example, the communications subsystem 1424 can be configured to receive data feeds 1426 in real time from users of social networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0191] Additionally, the communications subsystem 1424 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1428 of real-time events and / or event updates 1430, which may be of a continuous or unbounded nature with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0192] The communications subsystem 1424 may also be configured to output structured and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc. to one or more databases in communication with one or more streaming data source computers coupled to the computer system 1400.

[0193] The computer system 1400 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0194] Due to the ever-changing nature of computers and networks, the description of the computer system 1400 shown in the figure is intended as an example only. Many other configurations having more or fewer components than the system shown in the figure are possible. For example, customized hardware may also be used, and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Also, connections to other computing devices, such as network input and / or output devices, may be employed. Based on this disclosure and the teachings presented herein, one of ordinary skill in the art will recognize other ways and / or methods of implementing various embodiments.

[0195] Although specific embodiments have been described, various modifications, variations, alternative constructions and equivalents are encompassed within the scope of the disclosure. The embodiments are not limited to operating in one particular data processing environment, but can freely operate in multiple data processing environments. Furthermore, while the embodiments have been described using a particular sequence of transactions and steps, it will be apparent to those skilled in the art that the scope of the disclosure is not limited to the sequence of transactions and steps described. Various features and aspects of the above-described embodiments can be used individually or in combination.

[0196] Also, while embodiments are described using a particular combination of hardware and software, it should be understood that other combinations of hardware and software are within the scope of the present disclosure. The embodiments can be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be performed on the same processor or on different processors in any combination. Thus, when a component or module is described as being configured to perform an operation, such configuration can be achieved, for example, by designing an electronic circuit to perform the operation, or by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or a combination thereof. Processes can communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0197] The embodiments may be implemented using a computer program product comprising a computer program / instructions which, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

[0198] Accordingly, the specification and drawings are to be regarded in an illustrative and not a restrictive sense. However, it will be apparent that additions, substitutions, deletions and other modifications and alterations may be made thereto without departing from the broader spirit and scope of the appended claims. Thus, although certain disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are also intended to be encompassed within the scope of the appended claims.

[0199] Use of the terms "a," "an," "the," and similar referents in the context of describing embodiments of the present disclosure (particularly in the context of the appended claims) should be construed to include both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended (i.e., meaning "including, but not limited to"), unless otherwise indicated. The term "connected" should be construed as being partly or wholly contained within, attached to, or joined to one another, even if there are intervening elements. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated herein as if it were individually set forth herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by context. The use of any examples and illustrative language (e.g., "etc.") described herein, unless otherwise required, is intended merely to facilitate a better understanding of the embodiments and does not limit the scope of the disclosure. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0200] Disjunctive phrases such as "at least one of X, Y, or Z" are intended to be interpreted as generally used within the context to indicate that an item, term, etc., can be either X, Y, or Z, or any combination thereof (e.g., X, Y and / or Z), unless specifically stated otherwise. Thus, such disjunctive phrases are generally not intended to, and should not, imply that an embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0201] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the present disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the above description. Those skilled in the art should be able to adopt such variations as appropriate, and the present disclosure may be carried out in a manner different from that specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Also, unless otherwise indicated herein, any combination of the above-described elements is encompassed by the present disclosure in all its possible variations.

[0202] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually and specifically incorporated by reference and was set forth in its entirety herein.

[0203] Although aspects of the disclosure have been described hereinabove with reference to specific embodiments thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the disclosure above can be used individually or in combination. Also, the embodiments can be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. 1. A computer-implemented method comprising: an orchestration service of the cloud computing environment obtaining a plurality of configuration files corresponding to a plurality of services to be bootstrapped into the region; the plurality of configuration files provide data identifiable therefrom for a bootstrap task for bootstrapping the plurality of services within the region; The method comprises: identifying, based at least in part on the orchestration service performing an operation to parse the plurality of configuration files, one or more dependencies between respective services of the plurality of services; generating a construction dependency graph that maintains the one or more identified dependencies based at least in part on the parsing; the build dependency graph is a data structure that identifies the plurality of configuration files and the one or more dependencies and indicates a corresponding order in which bootstrapping tasks are performed; The method comprises: the orchestration service progressively instructing a provisioning and deployment manager to perform a bootstrap task based at least in part on traversing the build dependency graph; The method, wherein the bootstrap task is executed to bootstrap the plurality of services into the region according to the one or more identified dependencies.

2. identifying the one or more dependencies further includes the orchestration service identifying, based at least in part on the parsing, required capabilities of a first service on which a second service depends; 2. The method of claim 1, wherein the generated construction dependency graph indicates that the first service should be bootstrapped before the second service and that initiation of a respective bootstrap task for the second service should be delayed until the first service publishes a capability indicating it is bootstrapped.

3. identifying the one or more dependencies further includes the orchestration service identifying, based at least in part on the parsing, optional capabilities of the first service on which the second service optionally depends; The method of claim 2 , wherein the generated construction dependency graph indicates that the second service may be bootstrapped before the first service.

4. 4. The method of claim 3, wherein the construction dependency graph further indicates that if the second service is bootstrapped before the first service, then an additional bootstrap task corresponding to the second service should be executed when the first service is available.

5. 5. The method of claim 1, further comprising: the orchestration service identifying a circular dependency between two services; wherein identifying the circular dependency causes the orchestration service to instruct the provisioning and deployment manager to perform at least one additional execution of a respective bootstrap task corresponding to at least one of the two services; and wherein performing the at least one additional execution of the respective bootstrap task resolves the circular dependency between the two services.

6. 5. The method of claim 1, further comprising: the orchestration service identifying an orphaned service in the construction dependency graph; and the construction dependency graph indicating that the orphaned service is not bootstrapped due to an error in a corresponding configuration file corresponding to the orphaned service.

7. 5. The method of claim 1, further comprising: the orchestration service generating a plurality of phase data structures that identify one or more phases associated with bootstrapping the region and an order in which the one or more phases are to be executed, each phase associated with bootstrapping one or more instances of a given service.

8. one or more processors; one or more memories that store computer-executable instructions, The computer-executable instructions, when executed by the one or more processors, cause an orchestration service of the computing system to: and causing a region to obtain a plurality of configuration files corresponding to a plurality of services to be bootstrapped, the plurality of configuration files providing data from which a bootstrap task can be identified for bootstrapping the plurality of services within the region, the computer-executable instructions further causing the orchestration service to: identifying one or more dependencies between respective services of the plurality of services based at least in part on performing an operation to parse the plurality of configuration files; generating a build dependency graph that maintains the one or more identified dependencies based at least in part on the parsing, the build dependency graph being a data structure that identifies the plurality of configuration files and the one or more dependencies and indicates a corresponding order in which bootstrapping tasks are performed; and incrementally instructing a provisioning and deployment manager to execute a bootstrap task based at least in part on traversing the build dependency graph, the bootstrap task being executed to bootstrap the plurality of services into the region according to the one or more identified dependencies.

9. Execution of the computer-executable instructions to identify the one or more dependencies further comprises causing the orchestration service to:

9. The computing system of claim 8, further comprising: identifying, based at least in part on the parsing, required capabilities of a first service on which a second service depends; and wherein the generated construction dependency graph indicates that the first service should be bootstrapped before the second service and that initiation of a respective bootstrap task for the second service should be delayed until the first service publishes a capability indicating that it is bootstrapped.

10. Execution of the computer-executable instructions to identify the one or more dependencies further comprises causing the orchestration service to:

10. The computing system of claim 9, further comprising: identifying, based at least in part on the parsing, optional capabilities of the first service on which the second service optionally depends; and wherein the generated construction dependency graph indicates that the second service may be bootstrapped before the first service.

11. 11. The computing system of claim 10, wherein the construction dependency graph further indicates that if the second service is bootstrapped before the first service, then an additional bootstrap task corresponding to the second service should be executed when the first service is available.

12. Execution of the computer-executable instructions further causes the orchestration service to:

12. The computing system of claim 8, further comprising: identifying a circular dependency between two services; wherein identifying the circular dependency causes the orchestration service to instruct the provisioning and deployment manager to perform at least one additional execution of a respective bootstrap task corresponding to at least one of the two services; and wherein performing the at least one additional execution of the respective bootstrap task resolves the circular dependency between the two services.

13. 12. The computing system of claim 8, wherein execution of the computer-executable instructions further causes the orchestration service to identify an orphaned service in the construction dependency graph, the construction dependency graph indicating that the orphaned service is not bootstrapped due to an error in a corresponding configuration file corresponding to the orphaned service.

14. 12. The computing system of claim 8, wherein execution of the computer-executable instructions further causes the orchestration service to generate a plurality of phase data structures that identify one or more phases associated with bootstrapping the region and an order in which the one or more phases are to be executed, each phase associated with bootstrapping one or more instances of a given service.

15. A computer-readable program having computer-executable instructions stored thereon, The computer-executable instructions, when executed by one or more processors, cause an orchestration service of a cloud computing system to: causing a region to obtain a plurality of configuration files corresponding to a plurality of services to be bootstrapped, the plurality of configuration files providing data identifiable therefrom to a bootstrap task for bootstrapping the plurality of services within the region; identifying one or more dependencies between respective services of the plurality of services based at least in part on performing an operation to parse the plurality of configuration files; generating a build dependency graph based at least in part on the parsing, the build dependency graph maintaining the identified one or more dependencies, the build dependency graph being a data structure identifying the plurality of configuration files and the one or more dependencies and indicating a corresponding order in which bootstrapping tasks are performed; a provisioning and deployment manager configured to progressively instruct a provisioning and deployment manager to execute a bootstrap task based at least in part on traversing the build dependency graph, the bootstrap task being executed to bootstrap the plurality of services into the region according to the one or more identified dependencies.

16. Execution of the computer-executable instructions to identify the one or more dependencies further comprises causing the orchestration service to:

16. The computer-readable program of claim 15, further comprising identifying, based at least in part on the parsing, required capabilities of a first service on which a second service depends, and wherein the generated construction dependency graph indicates that the first service should be bootstrapped before the second service and that initiation of a respective bootstrap task for the second service should be delayed until the first service publishes a capability indicating that it is bootstrapped.

17. Execution of the computer-executable instructions to identify the one or more dependencies further comprises causing the orchestration service to:

17. The computer-readable program of claim 16, further comprising identifying, based at least in part on the parsing, optional capabilities of the first service on which the second service optionally depends, and wherein the generated construction dependency graph indicates that the second service may be bootstrapped before the first service.

18. Execution of the computer-executable instructions further causes the orchestration service to:

18. The computer-readable program of claim 15, further comprising: identifying a circular dependency between two services; wherein identifying the circular dependency causes the orchestration service to instruct the provisioning and deployment manager to perform at least one additional execution of a respective bootstrap task corresponding to at least one of the two services; and wherein performing the at least one additional execution of the respective bootstrap task resolves the circular dependency between the two services.

19. 18. The computer-readable program of claim 15, wherein execution of the computer-executable instructions further causes the orchestration service to identify an orphaned service in the construction dependency graph, the construction dependency graph indicating that the orphaned service will not be bootstrapped due to an error in a corresponding configuration file corresponding to the orphaned service.

20. 18. The computer-readable program of claim 15, wherein execution of the computer-executable instructions further causes the orchestration service to generate a plurality of phase data structures that identify one or more phases associated with bootstrapping the region and an order in which the one or more phases are to be executed, each phase associated with bootstrapping one or more instances of a given service.

Images