Technology for building data centers in cloud regions

Abstract

Techniques for performing automated region construction are described. An orchestration service (e.g., a multi-flock orchestrator) of a cloud computing environment may obtain configuration files corresponding to services to be bootstrapped in a region corresponding to one or more data centers. Each of the services may be associated with a respective set of resources including at least one infrastructure component or corresponding software artifact. The orchestration service may identify dependencies between the services based at least in part on the configuration files. An order in which operations to bootstrap the services are performed may be determined based at least in part on the identified dependencies. The orchestration service may incrementally instruct a provisioning and deployment manager to perform corresponding operations to bootstrap the services according to the determined order.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT application claims priority to U.S. Provisional Patent Application No. 63 / 308,003, filed February 8, 2022, entitled "Techniques for Bootstrapping a Region Build," U.S. Provisional Patent Application No. 63 / 312,814, filed February 22, 2022, entitled "Techniques for Implementing Virtual Data Centers," U.S. Provisional Patent Application No. 63 / 315,005, filed February 28, 2022, entitled "Techniques for Building Data Centers in Cloud Regions," and U.S. Provisional Patent Application No. 18 / 076,238, filed December 6, 2022, entitled "Techniques for Building Data Centers in Cloud Regions," the disclosures of which are incorporated herein by reference in their entireties for all purposes.

[0002] FIELD OF THEINVENTION The present disclosure relates to techniques for building a data center in a cloud computing environment. More specifically, the present disclosure describes techniques for provisioning and deploying resources corresponding to various cloud computing components (e.g., services) within a data center in an automated manner. [Background technology]

[0003] background Today, cloud infrastructure services utilize many individual services to build datacenters (e.g., to bootstrap various resources into datacenters in a particular geographic region). In some examples, a region is a logical abstraction that corresponds to a local geographic area in which one or more datacenters are (or will be) located. Building a datacenter may include provisioning and configuring infrastructure resources (e.g., for various services) and deploying code to these resources. Operations to build a datacenter may be collectively referred to as performing a "region build." Any suitable number of datacenters may be included in a region, and thus a region build may include operations to build multiple datacenters. Traditional tools for building regions require significant manual effort. In addition, a bootstrap operation for one service may depend on other features and / or services of the region that may not yet be available. As the number of service teams and regions increases, the tasks performed to orchestrate provisioning and deployment increase dramatically. Substantial reliance on manual effort to bootstrap services and / or build regions may be very time-consuming, introduce risks, and may not scale well. Summary of the Invention [Problem to be solved by the invention]

[0004] Quick Overview Embodiments of the present disclosure relate to performing automated region builds (e.g., bootstrap (e.g., provisioning and / or deploying) resources (e.g., infrastructure components, artifacts, etc.) for any suitable number of services within a region (e.g., a geographic location associated with one or more data centers). The bootstrap operations can be coordinated and orchestrated by an orchestration service (e.g., a multi-flock orchestrator) based at least in part on automatic detection of dependencies between these operations. The multi-flock orchestrator may further maintain various versions of configuration files and / or software artifacts, thereby intelligently and automatically identifying the particular set of versions on which region builds are to be performed.

[0005] At least one embodiment relates to a computer-implemented method. The method may include a multi-floc orchestrator of a cloud computing environment obtaining a plurality of configuration files corresponding to a plurality of services to be bootstrapped within a region corresponding to one or more data centers. In some embodiments, each of the plurality of services is associated with a respective set of resources including infrastructure components and corresponding software artifacts. The method may further include the multi-floc orchestrator identifying one or more dependencies among the plurality of services based at least in part on the plurality of configuration files. The method may further include the multi-floc orchestrator determining an order in which operations for bootstrapping the plurality of services are performed based at least in part on the identified one or more dependencies. The method may further include the multi-floc orchestrator incrementally instructing a bootstrap controller to perform corresponding operations for bootstrapping the plurality of services according to the determined order.

[0006] Another embodiment relates to another computer-implemented method. The method may include a multi-flock orchestrator of a cloud computing environment maintaining a plurality of version sets identifying respective sets of configuration files of a plurality of configuration files associated with a plurality of services. The method may further include the multi-flock orchestrator determining a first version set identifying a first set of configuration files from the plurality of configuration files. The method may further include the multi-flock orchestrator performing a validation process to validate the first set of configuration files identified by the first version set. The method may further include the multi-flock orchestrator generating a second version set identifying a second set of flock configuration files. In some embodiments, the second set of configuration files may be identified from the first set of configuration files based at least in part on identifying the configuration files that pass the validation process. The method may further include the multi-flock orchestrator performing a region build utilizing the second set of configuration files identified by the second version set.

[0007] Another embodiment relates to a computing device including one or more processors and instructions that, when executed by the one or more processors, cause the computing device to perform the methods disclosed herein.

[0008] Yet another embodiment relates to a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing cluster, cause the computing cluster to perform the methods disclosed herein.

[0009] To readily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the number of the figure in which that element is first introduced. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of an environment in which a cloud infrastructure orchestration service (CIOS) 102 may operate to dynamically provide bootstrap services in a region, according to at least one embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating an environment and method for building a Virtual Bootstrap Environment (ViBE) according to at least one embodiment. [Diagram 3] 1 is a block diagram illustrating an environment and method for bootstrapping a service into a target region utilizing ViBE, according to at least one embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an example method for maintaining multiple version sets that identify which configuration file versions and corresponding artifacts are utilized to bootstrap a service into a target region, according to at least one embodiment. [Diagram 5] FIG. 1 is a block diagram illustrating an example flow for performing operations for provisioning and deploying multiple services in a region, according to at least one embodiment. [Figure 6] FIG. 1 is a block diagram illustrating an example interface representing information related to services associated with a region construction in accordance with at least one embodiment. [Figure 7] FIG. 1 is a block diagram illustrating an example interface that depicts information related to events associated with region construction in accordance with at least one embodiment. [Figure 8] FIG. 1 illustrates an example method for performing region construction (e.g., for modifying a region by incrementally performing a bootstrap operation) in accordance with at least one embodiment. [Figure 9]FIG. 1 illustrates an example method for utilizing a version set to perform region construction, according to at least one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 11] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 12] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 13] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 14] FIG. 1 is a block diagram illustrating an exemplary computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0012] An example of an automated data center (region) construction infrastructure The adoption of cloud services has shown a rapid increase in recent years. Now, various types of cloud services are offered by various different cloud service providers (CSPs). The term cloud service is generally used to refer to services or functions made available by a CSP to a user or customer on demand (e.g., via a subscription model) using the systems and infrastructure (cloud infrastructure) offered by the CSP. Typically, the servers and systems that make up the CSP's infrastructure used to provide cloud services to the customer are separate from the customer's own on-premise operated servers and systems. Thus, the customer can utilize the cloud services offered by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribing customers with easy, scalable and on-demand access to applications and computing resources without the customer having to invest in procuring the infrastructure used to provide the service or functionality. Various different types or models of cloud services may be offered, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and others. A customer can subscribe to one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business.

[0013] As indicated above, a CSP is responsible for providing the infrastructure and resources used to provide cloud services to subscribing customers. The resources provided by a CSP can include both hardware and software resources. These resources can include, for example, computational resources (e.g., virtual machines, containers, applications, processors), memory resources (e.g., databases, data stores), network resources (e.g., routers, host machines, load balancers), identities, and other resources. In one implementation, the resources provided by a CSP to provide a set of cloud services CSPs are organized into a datacenter. A datacenter may be configured to provide a particular set of cloud services. The CSP is responsible for outfitting the datacenter with the infrastructure and resources used to provide that particular set of cloud services. A CSP may build one or more datacenters.

[0014] Data centers provided by a CSP may be hosted in different regions. A region is a local geographic area and may be identified by a region name. Regions are generally independent of each other and can be separated by large distances, such as across countries or even continents. Regions are grouped into realms. Examples of regions for a CSP may include US West, US East, Australia East, Australia Southeast, etc.

[0015] A region may include one or more data centers, the data centers being located within a certain geographic area corresponding to the region. As an example, the data centers in a region may be located in cities within the region. For example, for a particular CSP, a data center in the US West region may be located in San Jose, California, a data center in the US East region may be located in Ashburn, Virginia, a data center in the Australia East region may be located in Sydney, Australia, a data center in the Australia Southeast region may be located in Melbourne, Australia, etc.

[0016] Data centers within a region may be organized into one or more availability domains, which are used for high availability and disaster recovery purposes. An availability domain may include one or more data centers within a region. Availability domains within a region are isolated from each other, fault tolerant, and designed to make it highly unlikely that data centers in multiple availability domains will fail simultaneously. For example, availability domains within a region may be constructed to make it highly unlikely that a failure in one availability domain within a region will affect the availability of data centers in other availability domains within the same region.

[0017] When a customer or subscriber subscribes to or contracts with one or more services provided by a CSP, the CSP creates a tenancy for the customer. A tenancy is like an account created for a customer. In one implementation, a tenancy for a customer exists in one realm and can access all regions that belong to that realm. Thus, the customer's users can access the services subscribed by the customer under this tenancy.

[0018] As indicated above, CSPs build or deploy data centers to provide cloud services to their customers. As the CSP's customer base grows, the CSP typically builds new data centers in new regions or increases the capacity of existing data centers to serve the growing demands of the customers and provide better service to the customers. Preferably, the data center is built in geographic proximity to the location of the customers served by the data center. The geographical proximity between the data center and the customers served by the data center leads to more efficient utilization of resources and faster and more reliable service to the customers. Thus, the CSP typically builds new data centers in new regions in geographic areas that are geographically close to the customers served by the data center. For example, for a growing customer base in Germany, the CSP may build one or more data centers in new regions in Germany.

[0019] Building a datacenter (or multiple datacenters) in a region is sometimes referred to as building a region. The term "region construction" is used to refer to building one or more datacenters in a region. Building a datacenter in a region requires provisioning or creating a new set of resources needed or used to provide the set of services that the datacenter is configured to provide. The end result of the region construction process is the creation of a datacenter in the region that is capable of providing the set of services intended for that datacenter and includes the set of resources used to provide the set of services.

[0020] Building a new datacenter in a region is a very complex activity that requires extensive coordination between various bootstrapping activities. At a high level, it requires the execution and coordination of various tasks, such as identifying the set of services to be provided by the datacenter, identifying the various resources required to provide the set of services, generating, provisioning, and deploying the identified resources, and properly wiring the resources so that they can be used in the intended manner. Each of these tasks has further subtasks that need to be coordinated, further increasing the complexity. Due to this complexity, currently, building a datacenter in a region requires several manually initiated or manually controlled tasks that require careful manual coordination. As a result, the task of building a new region (i.e., building one or more datacenters in a region) is very time-consuming. It can take time, e.g., months, to build a datacenter. In addition, the process is highly error-prone and sometimes requires several iterations before the desired configuration of the datacenter is achieved, which further increases the time it takes to build a datacenter. These limitations and problems severely limit the ability of CSPs to grow computing resources in a timely manner in response to growing customer needs.

[0021] Technical effects This disclosure describes techniques for shortening construction time, reducing computing resource waste, and reducing the risks associated with building one or more data centers in a region. Instead of the weeks and months required in the past to build a data center in a region, the techniques described herein can be used to build a new data center in a region in a relatively significantly shorter time with less risk of error than traditional approaches.

[0022] Disclosed herein is a cloud infrastructure orchestration service (CIOS) configured to bootstrap (e.g., provision and deploy) services to a new datacenter based on a predefined configuration file that identifies resources (e.g., infrastructure components and software to be deployed) for implementing a given change to the datacenter. The CIOS can parse and analyze the configuration file (e.g., flock configuration) to identify dependencies between resources, execution targets, phases, and flocks. The CIOS may generate specific data structures from the analysis and use these data structures to drive operations and manage the order in which services are bootstrapped into regions. The CIOS may utilize these data structures to identify when the CIOS can bootstrap services, when the bootstrap is blocked and / or when the bootstrap operations associated with previously blocked services can be resumed. Advantageously, the CIOS can identify circular dependencies in the data structures and perform operations to eliminate / resolve these circular dependencies prior to task execution. Using these techniques, the CIOS substantially reduces the risk of executing tasks prior to the availability of resources on which those tasks depend.

[0023] Using the techniques disclosed herein, CIOS can optimize parallel processing to perform changes to datacenters while ensuring that tasks are not initiated until the functionality they depend on is available in the regions, thereby allowing CIOS to perform region construction more efficiently, which significantly reduces the time required to construct a datacenter and the wasted computational resource usage seen in conventional approaches.

[0024] A multi-flock orchestrator (MFO) is disclosed herein. The MFO may be configured to utilize multiple version sets that identify different sets of configuration files to be utilized for testing and / or region construction (e.g., provisioning and deployment to a regional datacenter). A particular version set can be used to construct a regional datacenter. The version set can identify a particular set of configuration files from which bootstrap tasks (e.g., provisioning and deployment tasks) in the region can be determined. The MFO may identify dependencies between services by performing static analysis of configuration files, and identify and resolve circular dependencies prior to region construction. The MFO may incrementally command a provisioning and deployment manager (e.g., CIOS Central, discussed in Figures 1-3) to execute bootstrap tasks while ensuring that dependent bootstrap tasks are not initiated until the resources on which those tasks depend are available. As various capabilities of the region become available, the MFO may identify and implement subsequent bootstrap tasks to be executed, driving the construction process incrementally to completion. The disclosed techniques allow unit and / or integration tests to be run with configuration files before those files are utilized for region construction, which increases the likelihood of successful region construction. Using the disclosed techniques, MFO allows automated region construction to be performed while reducing the risk of error and the time required in conventional systems.

[0025] definition A "region" is a logical abstraction that corresponds to a geographic location. A region may include any suitable number of one or more execution targets. In some embodiments, an execution target may correspond to a data center.

[0026] An "Execution Target" refers to the smallest unit of change for executing a release. A "Release" refers to an expression of intent to orchestrate a particular change to a service (e.g., deploy version 8, "add internal DNS records", etc.). For most services, an execution target represents an "instance" of the service. A service can be bootstrapped into one or more execution targets. An execution target may be associated with a set of devices (e.g., a data center).

[0027] "Bootstrapping" is intended to mean the collective tasks associated with provisioning and deploying any suitable number of resources (e.g., infrastructure components, artifacts, etc.) corresponding to a service.

[0028] "Service" means functionality provided by a set of resources. The set of resources for a service includes any suitable combination of infrastructure, platform, or software (e.g., applications) hosted by a cloud provider that can be configured to provide the functionality of the service. A service can be made available to users over the Internet.

[0029] "Artifact" means an infrastructure component or code that is deployed to a Kubernetes engine cluster, which may include software (e.g., applications), configuration information for the infrastructure component (e.g., configuration files), etc.

[0030] "Flock configuration" means a configuration file (or set of configuration files) that describes the set of all resources (e.g., infrastructure components and artifacts) associated with a service. A flock configuration may contain declarative statements that specify one or more aspects that correspond to a desired state of the resources of the service.

[0031] "Service State" means a current snapshot of all resources (e.g., infrastructure resources, artifacts, etc.) associated with a service. The service state indicates the state corresponding to the provisioning and / or deployment tasks associated with the service resources.

[0032] IaaS provisioning (or "provisioning") means acquiring computers or virtual hosts for use, as well as installing the necessary libraries or services on them. The expression "provisioning a device" means developing a device into a state where it can be utilized by an end user for their specific use. A device that has undergone a provisioning process may be called a "provisioned device". Preparing a provisioned device (installing libraries and daemons) may be part of provisioning. This preparation is different from deploying a new application or a new version of an application on a prepared device. In most cases, deployment does not include provisioning, which may need to be performed first. Once prepared, the device may be called an "infrastructure component".

[0033] IaaS deployment (or "deployment") refers to the process of providing and / or installing a new application or a new version of an application on provisioned infrastructure components. Once the infrastructure components are provisioned (e.g., acquired, assigned, prepared, etc.), additional software may be deployed (e.g., provided and installed on the infrastructure components). After provisioning and deployment are complete, the infrastructure components may be referred to as "resources." Examples of resources may include, but are not limited to, virtual machines, databases, object storage, block storage, load balancers, etc.

[0034] A "capability" identifies a unit of functionality associated with a service. A unit can be part or all of the functionality provided by a service. For example, a capability can be published that indicates that a resource is available for authorization / authentication processing (e.g., a subset of the functionality provided by the resource). As another example, a capability can be published that indicates that the complete functionality of a service is available. Capabilities can be used to identify functionality on which a resource or service depends and / or features of the resource or service that are available for use.

[0035] "Virtual Bootstrap Environment" (ViBE) refers to a virtual cloud network that is provisioned in an overlay of an existing region (e.g., a "host region"). Once provisioned, the ViBE is connected to the new region using a communication channel (e.g., an IPsec tunnel VPN). Certain required core services (or "seed" services) can be provisioned in the ViBE, such as a deployment orchestrator, public key infrastructure (PKI) services, etc. These services can provide the capabilities required to bring hardware online, establish a chain of trust to the new region, and deploy the remaining services in the new region. Utilizing a virtual bootstrap environment can prevent circular dependencies between bootstrap resources by utilizing resources of the host region. Services can be staged and tested in the ViBE before the physical region (e.g., the target region) is available.

[0036] "Cloud Infrastructure Orchestration Services" (CIOS) may mean a system configured to manage provisioning and deployment operations for any suitable number of services as part of a region deployment.

[0037] A multi-flock orchestrator (MFO) can be a computing component (e.g., a service) that coordinates events among components of the CIOS to provision and deploy services to a target region (e.g., a new region). The MFO tracks relevant events for each service in the region configuration and takes action in response to these events.

[0038] "Host Region" means a region that hosts a Virtual Bootstrap Environment (ViBE). A host region may be used to bootstrap a ViBE.

[0039] "Target Region" means the region being constructed. "Publishing a capability" means "publishing" as used in a "publisher-subscriber" computing design or otherwise to provide an indication that a particular capability is available (or unavailable). Capabilities are "published" (e.g., collected by a capability service, provided to a capability service, pushed, pulled, etc.) to provide an indication that functionality of a resource / service is available. In some embodiments, capabilities may be published / sent via an event, notification, data transmission, function call, API call, etc. An event (or other notification / data transmission, etc.) indicating availability of a particular capability can be broadcast / addressed (e.g., published) to the capability service.

[0040] A "capability service" may be a flock configured to model dependencies between different flocks. Capability services may be provided within a cloud infrastructure orchestration service and may define which capabilities, services, and features are available in a region.

[0041] A "Real-time Regional Data Distributor" (RRDD) can be a service or system configured to manage regional data that can be injected into a flock configuration to dynamically generate execution targets for new regions.

[0042] In some examples, techniques for implementing a cloud infrastructure orchestration service (CIOS) are described herein. Such techniques can be configured to manage bootstrapping (e.g., provisioning and deploying software to) infrastructure components within a cloud environment (e.g., a region), as briefly described above. In some examples, the CIOS can include computing components (e.g., CIOS central and CIOS regional, both of which are described in more detail below) that can be configured to manage bootstrapping tasks (provisioning and deployment) for a given service, and a multi-flock orchestrator (also described in more detail below) configured to initiate / manage region construction (e.g., bootstrapping operations corresponding to multiple services).

[0043] CIOS enables region construction and worldwide infrastructure provisioning and code deployment with minimal manual runtime effort from service teams (e.g., beyond initial approval and / or physical transportation of hardware, in some examples). High-level responsibilities of CIOS include, but are not limited to, orchestrating region construction, providing users with a view of the current state of resources managed by CIOS (e.g., in a region, across regions, worldwide, etc.), and managing bootstrap operations to bootstrap resources within a region.

[0044] CIOS may provide view reconciliation, where a view of a desired state (e.g., desired configuration) of a resource may be reconciled with the current / actual state (e.g., current configuration) of the resource. In some examples, view reconciliation may include obtaining state data to identify which resources are actually operating and their current configuration and / or state. Reconciliation can be performed at various granularities, such as at the service level.

[0045] The CIOS can perform plan generation, where differences between desired and current states of resources are identified. Part of the plan generation can include identifying operations that need to be performed to bring resources from their current state to the desired state. In some examples, the CIOS can present the generated plant to a user for approval. In these examples, the CIOS can mark the plan as approved or rejected based on user input from the user. Thus, the user can spend less time judging the plan, and the plan is more accurate because it is machine generated. The plan is almost too detailed for human consumption. However, the CIOS can provide this data through a very advanced user interface (UI).

[0046] In some examples, the CIOS can handle change management execution by executing an approved plan. Once an execution plan is generated and approved, the engineer may no longer need to be involved in change management until the CIOS initiates the rollback. The CIOS can handle rollback to a previous service version by generating a plan to revert the service to a previous (e.g., pre-release) state (e.g., when the CIOS detects a service health degradation while executing).

[0047] CIOS can measure service health by monitoring alarms and running integration tests. CIOS can help teams quickly define rollback behavior in the event of service degradation and can execute the rollback behavior later. CIOS can generate and display plans and track approvals. CIOS can combine provisioning and deployment functions in one system that coordinates these tasks across region construction. CIOS also supports discovery of flocks (e.g., service resources such as flock configurations corresponding to any suitable number of services), artifacts, resources, and dependencies. CIOS can discover dependencies between execution tasks at all levels (e.g., resource level, execution target level, phase level, service level, etc.) by static analysis (e.g., including parsing and processing the contents) of one or more configuration files. Using these dependencies, CIOS can generate various data structures from these dependencies, which can be used to drive task execution (e.g., tasks related to provisioning infrastructure resources and deployment of artifacts across regions).

[0048] FIG. 1 is a block diagram of an environment 100 in which a cloud infrastructure orchestration service (CIOS) may operate to dynamically provide bootstrap services in a region, according to at least one embodiment. CIOS 102 may include, but is not limited to, the following components: real-time regional data distributor (RRDD) 104, multi-flock orchestrator (MFO) 106, CIOS central 108, CIOS regional 110, and capability service 112. Specific functionality of CIOS central 108 and CIOS regional 110 is provided in more detail in U.S. patent application Ser. No. 17 / 016,754, entitled "Techniques for Deploying Infrastructure Resources with a Declarative Provisioning Tool," the entire contents of which are incorporated in their entirety for all purposes. In some embodiments, any suitable combination of components of CIOS 102 may be provided as a service. In some embodiments, some portion of CIOS 102 may be deployed to a region (e.g., a data center represented by host region 103). In some embodiments, CIOS 102 may include any suitable number of cloud services (not shown in FIG. 1), as discussed in further detail in U.S. patent application Ser. No. 17 / 016,754 and below with respect to FIGS. 2 and 3.

[0049] The real-time regional data distributor (RRDD) 104 may be configured to maintain and provide region data identifying realms, regions, execution targets, and availability domains. In some cases, the region data may be in any suitable format (e.g., JSON format, data object / container, XML, etc.). The region data maintained by the RRDD 104 may include any suitable number of subsets of data that may be individually referenceable by a corresponding identifier. For example, an identifier such as "all_regions" may be associated with a data structure (e.g., a list, structure, object, etc.) that includes metadata for all defined regions. As another example, an identifier such as "realm" may be associated with a data structure that identifies metadata for a number of realms and a set of regions corresponding to each realm. In general, the region data may maintain any suitable attributes of one or more realms, regions, availability domains (ADs), execution targets (ETs), etc., such as identifiers, DNS extensions, states (e.g., state of the region), etc. The RRDD 104 may be configured to manage region state as part of the region data. The region state may include any suitable information indicative of a state of bootstrap within the region. For example, some example region states may include "initial," "construction," "production," "dormant," or "decommissioned." The "initial" state may indicate a region that has not yet been bootstrapped. The "construction" state may indicate that bootstrapping of one or more flocks in the region has begun. The "production" state may indicate that bootstrapping is complete and the region is being prepared for validation. The "dormant" state may indicate that CIOS Central 108 or CIOS Regional 110 has paused internal interaction with the regional stack, possibly due to operational issues. The "decommissioned" state may indicate that the region is being decommissioned and is likely to become unavailable and / or will not be contacted again.

[0050] CIOS central 108 may be configured to provide any suitable number of user interfaces by which a user (e.g., user 109) may interact with CIOS 102. For example, a user may make changes to region data through a user interface provided by CIOS central 108. CIOS central 108 may additionally provide various interfaces that allow a user to view changes made to flock configurations and / or artifacts, generate and view plans, approve / reject plans, view status in plan execution (e.g., corresponding to infrastructure provisioning, deployment, region construction, and / or desired states of any suitable number of resources managed by CIOS 102). CIOS central 108 may implement a control plane configured to manage any suitable number of CIOS regional 110 instances. CIOS central 108 may provide one or more user interfaces for representing region data, allowing user 109 to view and / or modify the region data. CIOS central 108 may be configured to invoke functions of RRDD 104 through any suitable number of interfaces. In general, CIOS central 108 may be configured to manage region data, either directly or indirectly (e.g., via RRDD 104). CIOS central 108 may be configured to compile a flock configuration to inject the region data as variables within the flock configuration.

[0051] Each instance of CIOS regional 110 may correspond to a module configured to perform bootstrapping tasks associated with one service of the region. CIOS regional 110 may receive desired state data from CIOS central 108. In some embodiments, the desired state data may include a flock configuration that declares (e.g., via a declarative statement) a desired state of resources associated with the service. CIOS central 108 may maintain current state data that indicates any suitable aspect of the current state of resources associated with the service. In some embodiments, CIOS regional 110 may identify that changes are required for one or more resources by comparison of the desired state data and the current state data. For example, CIOS regional 110 may determine that one or more infrastructure components need to be provisioned, one or more artifacts are deployed, or any suitable changes are required to the resources of the service to bring the state of those resources into agreement with the desired state. As CIOS regional 110 performs the bootstrapping operation, CIOS regional 110 may publish data that indicates various capabilities of the resources as they become available. A "capability" identifies a unit of functionality associated with a service. A unit can be part or all of the functionality provided by a service. For example, a capability can be published that indicates that a resource is available for authorization / authentication processing (e.g., a subset of the functionality provided by the resource). As another example, a capability can be published that indicates that the complete functionality of a service is available. Capabilities can be used to identify functionality on which a resource or service depends and / or features of the resource or service that are available for use.

[0052] The capability service 112 is configured to maintain capability data that indicates 1) which capabilities of various services are currently available, 2) whether any resources / services are waiting for a particular capability, 3) which particular resources and / or services are waiting for a given capability, or any suitable combination of the above. The capability service 112 may provide an interface by which the capability service may be requested. The capability service 112 may provide one or more interfaces (e.g., application programming interfaces) that allow the capability data to be sent to the MFO 106 and / or CIOS regional 110 (e.g., each instance of the CIOS regional 110). In some embodiments, any suitable component or module of the MFO 106 and / or CIOS regional 110 may be configured to request the capability data from the capability service 112.

[0053] In some embodiments, a multi-flock orchestrator (MFO) 106 may be configured to drive the region construction effort. In some embodiments, the MFO 106 may manage information describing which flocks / flock configuration versions and / or artifact versions are utilized to bootstrap a given service in a region (or to make a unit of change to a target region). In some embodiments, the MFO 106 may be configured to monitor (or otherwise be notified of) changes to the region data managed by the real-time regional data distributor 104. In some embodiments, region construction may be triggered by the MFO 106 upon receiving an indication that the region data has changed. In some embodiments, the MFO 106 may collect various flock configurations and artifacts used for region construction. Some or all of the flock configurations may be configured to be region agnostic; that is, the flock configuration may not explicitly identify which region the flock is bootstrapped to. In some embodiments, the MFO 106 may trigger a data injection process through which the collected flock configuration is recompiled (e.g., by the CIOS central 108). During the recompilation, an operation may be performed (e.g., by the CIOS central 108) to cause the region data maintained by the real-time regional data distributor 104 to be injected into the configuration file. The flock configuration can reference the region data through variables / parameters without requiring a hard-coded identification of the region data. The flock configuration can be dynamically modified at runtime with this data injection, rather than the region data being hard-coded, thus making it more difficult to change.

[0054] The multi-flock orchestrator 106 may perform static flock analysis, where the flock configuration is parsed to identify dependencies between resources, execution targets, phases, and flocks, and in particular to identify circular dependencies that need to be removed. In some embodiments, the MFO 106 may generate any suitable number of data structures based on the identified dependencies. These data structures (e.g., directed acyclic graphs, linked lists, etc.) may be utilized by the cloud infrastructure orchestration service 102 to drive operations to perform region construction. For example, these data structures may collectively define the order in which services are bootstrapped within a region. An example of such a data structure is discussed further below with respect to the construction dependency graph 338 of FIG. 3. When circular dependencies (e.g., service A requires service B and vice versa) exist and are identified through static flock analysis and / or graphs, the MFO may be configured to notify any appropriate service teams that changes are required to the corresponding flock configuration to correct these circular dependencies. The MFO 106 can be configured to traverse one or more data structures to manage the order in which services are bootstrapped into regions. The MFO 106 can identify (e.g., using data obtained from the capabilities service 112) the capabilities available within a given region at any given time. The MFO 106 can use this data to identify when a service can be bootstrapped, when bootstrap is blocked, and / or when a bootstrap operation associated with a previously blocked service can be resumed. Based on this traversal, the MFO 106 can execute various releases in which instructions are sent by the MFO 106 to the CIOS central 108 to execute a bootstrap operation corresponding to any suitable number of flocking configurations.In some examples, the MFO 106 may be configured to identify that one or more flock configurations may require multiple releases due to circular dependencies found in the graph, and as a result, the MFO 106 may send multiple sets of instructions to the CIOS central 108 for a given flock configuration to break the circular dependencies identified in the graph.

[0055] In some embodiments, a user may request that a new region (e.g., target region 114) be built. This may require bootstrapping resources corresponding to various services. In some embodiments, target region 114 may not be communicatively available (and / or secure) at the time the region build request is initiated. Rather than deferring bootstrapping until such time that target region 114 is available and configured to perform the bootstrap operation, CIOS 102 may initiate region build using a virtual bootstrap environment 116. The virtual bootstrap environment (ViBE) 116 may be an overlay network hosted by host region 103 (an existing region that has been previously configured with a core set of services and is communicatively available and secure). MFO 106 may leverage the resources of host region 103 to bootstrap resources to ViBE 116 (commonly referred to as "building a ViBE"). For example, MFO 106 can provide instructions through CIOS Central 108 to cause an instance of CIOS regional 110 in a host region (e.g., host region 103) to bootstrap another instance of CIOS regional in ViBE 116. Once the CIOS regional in ViBE becomes available for processing, bootstrapping of services for the target region 114 can continue in ViBE 116. When the target region 114 is available to perform the bootstrap operation, previously bootstrapped services in ViBE 116 may be moved to the target region 114. Utilizing these techniques, CIOS 102 can significantly increase the speed at which regions are built by dramatically reducing the need for any manual input and / or configuration to be provided.

[0056] 2 is a block diagram illustrating an environment 200 and method for building a virtual bootstrap environment (ViBE) 202 (an example of ViBE 116 of FIG. 1) according to at least one embodiment. ViBE 202 represents a virtual cloud network that is provisioned in an overlay of an existing region (e.g., hosted region 204, an example of hosted region 103 of FIG. 1 and in the embodiments is a hosted region service enclave). ViBE 202 represents an environment in which services can be staged for a target region (e.g., a region under construction such as target region 114 of FIG. 1) before the target region is available.

[0057] To bootstrap a new region (e.g., target region 114 of FIG. 1), a core set of services may be bootstrapped. These core set of services may exist in the host region 204, but do not yet exist in ViBE (nor in the target region). These required core services provide the functionality needed to provision devices, establish a chain of trust to the new region, and deploy the remaining services (e.g., negative locks) within the region. ViBE 202 may be a tenancy that is deployed in the host region 204. This can be thought of as a virtual region.

[0058] Once the target region is available to provide bootstrap operations, ViBE 202 can be connected to the target region, allowing services in ViBE to interact with services and / or infrastructure components of the target region. This allows for the deployment of production level services instead of a self-contained seed service as in previous systems, requiring a connection to the target region over the Internet. Traditionally, seed services are deployed as part of a container collection and are used to bootstrap the dependencies needed to build the region. Using the existing region's infrastructure / tooling, resources may be bootstrapped (e.g., provisioned and deployed) into ViBE 202 and connected to the service enclaves of the region (e.g., host region 204) to provision hardware and deploy services until the target region is self-sufficient and can be directly communicated with. Utilizing ViBE 202 allows for the dependencies and services needed to be able to provision / prepare infrastructure and deploy software while utilizing the host region's resources to break circular dependencies of core services.

[0059] A multi-flock orchestrator (MFO) 206 may be configured to perform operations to build (e.g., configure) ViBE 202. MFO 206 may obtain applicable flock configurations corresponding to various resources to be bootstrapped into a new region (in this case, a ViBE region, ViBE 202). For example, MFO 206 may obtain a flock configuration (e.g., a "ViBE flock configuration") that identifies aspects of bootstrapping capability services 208 and workers 210. As another example, MFO 206 may obtain another flock configuration corresponding to bootstrapping domain name services (DNS) 212 into ViBE 202.

[0060] In step 1, MFO 206 may instruct CIOS central 214 (e.g., an example of CIOS central 108 and CIOS central 214 of FIGS. 1 and 2, respectively). For example, MFO 206 may send a request (e.g., including a ViBE flock configuration) to request bootstrap of capability services 208 and workers 210 that do not yet exist in ViBE 202 at this time. In some embodiments, CIOS central 214 may have access to all flock configurations. Thus, in some examples, MFO 206 may send an identifier for a ViBE flock configuration rather than the file itself, which CIOS central 214 may independently obtain from storage (e.g., from DB 308 or flock DB 312 of FIG. 3).

[0061] In step 2, CIOS central 214 may provide the ViBE flock configuration via a corresponding request to CIOS regional 216. CIOS regional 216 may parse the ViBE flock configuration to identify and perform specific infrastructure provisioning and deployment operations in step 3.

[0062] In some embodiments, CIOS regional 216 may utilize additional corresponding services for provisioning and deployment. For example, in step 4, CIOS regional 216 CIOS regional may instruct deployment orchestrator 218 (e.g., an example of core services or other write, build and deploy application software of host region 204) to execute instructions that, in turn, bootstrap capability service 208 and worker 210 within ViBE 202.

[0063] In step 5, a capability may be sent to capability service 208 (from CIOS regional 216, deployment orchestrator 218 via worker 210, or otherwise) indicating that resources corresponding to the ViBE block are available. Capability service 208 may persist this data. In some embodiments, capability service 208 adds this information to a list it maintains of capabilities available with ViBE. For example, the capability provided to capability service 208 in step 5 may indicate that capability service 208 and worker 210 are available for processing.

[0064] In step 6, the MFO 206 may identify that the capability indicates that the capability service 208 and worker 210 are available based on receiving or obtaining data (an identifier corresponding to the capability) from the capability service 208.

[0065] In step 7, as a result of receiving / obtaining the data in step 6, MFO 206 may instruct CIOS Central 214 to bootstrap a DNS service (e.g., DNS 212) on ViBE 202. The instructions may identify or include a particular flock setting that corresponds to the DNS service.

[0066] In step 8, CIOS central 214 may instruct CIOS regional 216 to deploy DNS 212 to ViBE 202. In some embodiments, the DNS flocking configuration for DNS 212 is provided by CIOS central 214.

[0067] In step 9, worker 210, now deployed in ViBE 202, may be assigned by CIOS regional 216 to the task of deploying DNS 212. The worker may run a declarative infrastructure provisioner in the form described above in connection with FIG. 3 to identify the set of operations that need to be performed to deploy DNS 212 (e.g., by comparing the flock configuration (desired state) with the current state of the (now non-existent) resources associated with the flock).

[0068] At step 10, deployment orchestrator 218 may instruct worker 210 to deploy DNS 212 according to the operations identified at step 9. As shown, worker 210 proceeds to execute operations to deploy DNS 212 to ViBE 202 at step 11. At step 12, worker 210 notifies capability service 208 that DNS 212 is available on ViBE 202. MFO 206 may subsequently identify that the ViBE flock configuration and resources associated with the DNS flock configuration are available and may proceed to bootstrap any appropriate number of additional resources to ViBE.

[0069] After steps 1-12 are completed, the process for building ViBE202 can be considered complete, and ViBE202 can be considered built.

[0070] FIG. 3 is a block diagram illustrating an environment 300 and method for bootstrapping a service to a target region utilizing ViBE according to at least one embodiment.

[0071] In step 1, user 302 may utilize any suitable user interface provided by CIOS central 304 (an example of CIOS central 108 and CIOS central 214 of FIGS. 1 and 2, respectively) to modify region data. For example, user 302 may create a new region into which a number of services are bootstrapped.

[0072] In step 2, CIOS central 304 may perform an operation to send the changes to RRDD 306 (e.g., an example of RRDD 104 in FIG. 1). In step 3, RRDD 306 may store the received region data in database 308, a data store configured to store region data including any suitable identifiers, attributes, states, etc., such as region, AD, realm, ET, etc. In some embodiments, updater 307 may be utilized to store the region data in database 308 or any suitable data store, from which such updates may be accessible (e.g., to a service team). In some embodiments, updater 307 may be configured to notify (e.g., via any suitable electronic notification) of updates made to database 308.

[0073] In step 4, the MFO 310 (an example of MFOs 106 and 206 in FIGS. 1 and 2, respectively) may detect changes in the region data. In some embodiments, the MFO 310 may be configured to poll the RRDD 306 for changes in the region data. In some embodiments, the RRDD 306 may be configured to publish or otherwise notify the MFO 310 of the region changes.

[0074] In step 5, detecting a change in the region data may trigger MFO 310 to retrieve a version set (e.g., a version set associated with a particular identifier, such as a "golden version set" identifier) ​​to identify a specific version for each flock (e.g., service) to be bootstrapped into the new region and a specific version for each artifact corresponding to that flock. The version set may be retrieved from DB 312. As flocks develop and change, the versions for their corresponding settings and artifacts used for region construction may change. These changes may be persisted in flock DB 312 so that MFO 310 may identify which versions of flock settings and artifacts to use to use to build a region (e.g., ViBE region, target region / non-ViBE region, etc.). Flock settings (e.g., all versions of flock settings) and / or artifacts (e.g., all versions of artifacts) may be stored in DB 308, DB 312, or any suitable data store accessible to CIOS Central 304 and / or MFO 310.

[0075] In step 6, the MFO 310 may request the CIOS Central 304 to recompile each of the flock settings associated with the version set with the current region data. In some embodiments, the request may indicate versions for each flock setting and / or artifacts that correspond to those flock settings.

[0076] In step 7, the CIOS Central 304 may obtain the current regional data from the DB 308 (e.g., directly or via the real-time regional data distributor 306) and retrieve any appropriate flock settings and artifacts according to the version requested by the MFO 310.

[0077] In step 8, CIOS Central 304 may recompile the flock configuration with the region data obtained in step 7 to inject the flock configuration with the current region data. CIOS Central 304 may return the compiled flock configuration to MFO 310. In some embodiments, CIOS Central 304 may simply indicate that compilation has occurred, or MFO 310 may access the recompiled flock configuration via RRDD 306.

[0078] In step 9, the MFO 310 may perform a static analysis of the recompiled flock configuration. As part of the static analysis, the MFO 310 may parse the flock configuration (e.g., using libraries associated with a declarative infrastructure provisioner (e.g., Terraform, etc.)) to identify dependencies between flocks. From the identified analysis and dependencies, the MFO 310 may generate a build dependency graph 338. The build dependency graph 338 may be an acyclic, directed graph that identifies the order in which flocks are bootstrapped into new regions (and / or the order in which changes indicated in the flock configuration are applied). Each node in the graph may correspond to bootstrapping any appropriate portion of a particular flock. The particular bootstrap order may be identified based at least in part on the dependencies. In some embodiments, the dependencies may be represented as attributes of the nodes and / or may be indicated via edges of the graph connecting the nodes. The MFO 310 may traverse the graph (e.g., beginning at a start node) to drive the region build operation.

[0079] In some embodiments, MFO 310 may utilize a cycle detection algorithm to detect the existence of a cycle (e.g., service A depends on service B and vice versa). MFO 310 may identify orphaned capability dependencies. For example, MFO 310 may identify orphaned nodes in construction dependency graph 338 that are not connected to any other nodes. MFO 310 may identify erroneously issued capabilities (e.g., when a capability is issued too early and the corresponding functionality is not actually available yet). MFO 310 may detect from the graph that there are one or more instances that issue the same capability. In some embodiments, any suitable number of these errors may be detected, and MFO 310 (or another suitable component, such as CIOS Central 304) may be configured to notify or otherwise present this information to a user (e.g., via an electronic notification, a user interface, etc.). In some embodiments, MFO 310 may be configured to force the removal / recreation of resources to break circular dependencies, and may again provide instructions to CIOS Central 304 to perform bootstrap operations for those resources and / or corresponding block configurations.

[0080] The initiating node may correspond to bootstrapping the ViBE flock, and the second node may correspond to bootstrapping the DNS. Steps 10-15 correspond to deploying (via deployment orchestrator 317, an example of deployment orchestrator 218 in FIG. 2) the ViBE flock to ViBE 316 (e.g., an example of ViBE 116 and 202 in FIGS. 1 and 2, respectively). That is, steps 10-15 in FIG. 3 generally correspond to steps 1-6 in FIG. 2. Upon notification that capabilities corresponding to the ViBE flock to be deployed exist (e.g., indicating that capability service 318 and worker 320, corresponding to capability service 208 and worker 210 in FIG. 2, are available), MFO 310 resumes traversing construction dependency graph 338 to identify the next operation to be executed.

[0081] For example, MFO 310 may continue to traverse construction dependency graph 338 to identify a DNS block to be deployed. Steps 16-21 may be performed to deploy DNS 322 (an example of DNS 212 in FIG. 2). These operations may generally correspond to steps 7-12 in FIG. 2.

[0082] At step 21, a capability indicating that DNS 322 is available may be stored. Upon detecting this capability, MFO 310 may resume traversal of construction dependency graph 338. In this traversal, MFO 310 may inform ViBE 316 that any appropriate portions of an instance of CIOS regional (e.g., an instance of CIOS regional 314) are deployed to ViBE 316. In some embodiments, steps 16-21 may be substantially repeated with respect to deploying CIOS regional (ViBE) 326 (an instance of CIOS regional 314, CIOS regional 110 in FIG. 1) and worker 328 to ViBE 316. The capability may be sent to capability service 318 that CIOS regional (ViBE) 326 is available.

[0083] Upon detecting that CIOS Regional (ViBE) 326 is available, MFO 310 may resume traversal of construction dependency graph 338. In this traversal, MFO 310 may identify that a deployment orchestrator (e.g., deployment orchestrator 330, an example of deployment orchestrator 317) is deployed to ViBE 316. In some embodiments, steps 16-21 may be substantially repeated with respect to deploying deployment orchestrator 330. Information identifying capabilities may be sent to capability service 318 indicating that deployment orchestrator 330 is available.

[0084] Once the deployment orchestrator 330 is deployed, ViBE 316 may be considered available to process subsequent requests. Upon detecting that the deployment orchestrator 330 is available, MFO 310 may instruct subsequent bootstrap requests to be routed to the ViBE component rather than utilizing the host region component (the component of host region 322). Thus, MFO 310 may continue to traverse the construction dependency graph 338 at each node instructing flock deployment to ViBE 316 via CIOS Central 304. CIOS Central 304 may request CIOS Regional (ViBE) 326 to deploy resources according to the flock configuration.

[0085] At some point during this process, the target region 334 may become available. An indication that the target region is available may be discernible from region data for the target region 334 provided by the user 302 (e.g., as an update to the region data). The availability of the target region 334 may depend on establishing a network connection between the target region 334 and an external network (e.g., the Internet). The network connection may be supported over a public network (e.g., the Internet) but may use software security tools (e.g., IPsec) to provide one or more encrypted tunnels (e.g., IPsec tunnels such as tunnel 336) from ViBE 316 to the target region 334. As used herein, "IPsec" refers to a suite of protocols for authenticating and encrypting network traffic over networks that use the Internet Protocol (IP) and may include one or more available implementations of the suite of protocols (e.g., Openswan, Libreswan, strongSwan, etc.). The network may connect ViBE 316 to a service enclave of the target region 334.

[0086] Prior to establishing the IPsec tunnel, the initial network connection to the target region 334 may be on a sufficient connection (e.g., an out-of-band VPN tunnel) to allow bootstrapping of networking services until IPsec can be deployed on assets (e.g., bare metal assets) in the target region 334. To bootstrap the network resources of the target region 334, the deployment orchestrator 330 may deploy an IPsec gateway on the assets in the target region 334. The deployment orchestrator 330 may then deploy a VPN host in the target region 334 configured to terminate the IPsec tunnel from ViBE 316. A service in ViBE 316 (e.g., deployment orchestrator 330, Service A, etc.) can establish an IPsec connection with the VPN host in the target region 334, and the bootstrap operation from ViBE 316 to the target region 334 may begin.

[0087] In some embodiments, the bootstrap operation may begin by services in ViBE 316 provisioning resources in the target region 334 to support hosting instances of core services when deployed from ViBE 316. For example, a host provisioning service may provision a hypervisor on infrastructure (e.g., bare metal hosts) in the target region 334 to allocate computing resources for VMs. Once the host provisioning service completes the allocation of physical resources in the target region 334, the host provisioning service may publish information indicating a capability indicating that the physical resources in the target region 334 have been allocated. The capability may be published to the capability service 318 (e.g., by worker 328) via CIOS regional (ViBE) 326.

[0088] Once the hardware allocation for the target region 334 has been established and posted to the capability service 318, CIOS regional (ViBE) 326 can orchestrate the deployment of instances of core services from ViBE 316 to the target region 334. This deployment may be similar to the process described above for building ViBE 316, but using ViBE components (e.g., CIOS regional (ViBE) 326, worker 328, deployment orchestrator 330) instead of host region 332 service enclave components. The deployment operation may generally correspond to steps 16-21 described above.

[0089] As a service is deployed from ViBE 316 to target region 334, a DNS record associated with the service may correspond to an instance of the service in ViBE 316. The DNS record associated with the service may be updated later to complete the deployment of the service to target region 334. In other words, the instance of the service in ViBE 316 may continue to receive traffic (e.g., requests) to the service until the DNS record is updated. The service may be partially deployed into target region 334 and may publish information (e.g., to capability service 318) indicating the capabilities that the service is partially deployed in. For example, a service running in ViBE 316 may be deployed into target region 334 with corresponding compute instances, load balancers, and associated applications and other software, but may need to wait for database data to move to target region 334 before being fully deployed. The DNS record (e.g., managed by DNS 322) may still be associated with the service in ViBE 316. Once the data movement for the service is complete, the DNS record may be updated to point to the operational service deployed in target region 334. The deployed service in target region 334 may then receive traffic (eg, requests) for the service, while the instance of the service in ViBE 316 may no longer receive traffic for the service.

[0090] Multi-flock orchestration with version sets FIG. 4 is a block diagram illustrating an example method 400 for maintaining multiple version sets according to at least one embodiment. Method 400 may be performed by any suitable component of cloud infrastructure orchestration service 102 of FIG. 1. In some embodiments, method 400 may be performed by multi-flock orchestrator 106 of FIG. 1. A version set refers to a snapshot of all artifacts and configuration files (e.g., flock configuration files, referred to herein for short as "flock configurations") used to perform a region build. Flock configurations and / or artifacts provided in a version set may be associated with a version identifier that identifies a particular version of the artifact and / or flock configuration from other versions of that artifact or flock configuration. For example, a flock configuration may be associated with identifier "service 1" but has multiple versions (e.g., version 1, version 2, version 3.1, etc.). A multi-flock orchestrator (e.g., MFOs 106, 206, and 310 of Figures 1, 2, and 3, respectively) may be configured to manage multiple version sets corresponding to potentially different flock configurations and / or versions of flock configurations, or artifacts and / or versions of artifacts utilized. A version set may be used to perform region builds, whereas a different version set may be used for testing and integration efforts. By tracking different version sets, the multi-flock orchestrator can ensure that a stable version set (e.g., a version set that is likely to produce a successful region build) is used for region builds. A version set may be associated with an identifier (e.g., "us-ashburn-1") and a scope identifier by which the version set may be referenced.Exemplary scope identifiers may include, but are not limited to, "golden," "nightly," "break-glass," and "default," although others may be used. One scope identifier (e.g., golden) may indicate a version set used for region construction. Another scope identifier (e.g., nightly) may indicate a version set used for testing (e.g., unit tests run nightly). Another scope identifier (e.g., break-glass) may indicate a version set that can be used to recover from errors incurred during region construction. Another scope identifier (e.g., default) may indicate a version set of flocking configurations / artifacts that are tested in hopes of eventual inclusion in the region construction.

[0091] In step 1, each service team may publish flock configuration data (e.g., a flock configuration identifier and a corresponding version identifier that uniquely identifies a particular version of the flock configuration from other versions of the flock configuration). In some embodiments, publishing the flock configuration data may be performed by utilizing an application programming interface (API) call or another suitable function call. By executing this call, the flock configuration data (e.g., a flock configuration identifier and a version identifier) ​​may be added to a default version set (e.g., default version set 402). In some embodiments, default version set 402 may be stored in DB 312 of FIG. 3 or any suitable data store. As another example, a particular version of a flock configuration may be manually modified (e.g., with a label / tag) to indicate that the flock configuration should be included in the default version set. The multi-flock orchestrator may parse all flock configurations to identify the sets that are associated with the default version set. As shown in default version set 404, "A" through "D" correspond to flock setting identifiers associated with the respective flocks, whereas "8", "9" and "3" provided in default version set 402 refer to versions of flock settings. Thus, flock setting A, version 8, flock setting B, version 9, and flock setting C, version 2 may be added to default version set 404. The default version set may already include flock setting "D", version 9. The default version set maintained after step 1 is shown at 406.

[0092] In step 2, any suitable operation may be performed to determine a nightly version set for testing. For example, a set of flock configuration identifiers and corresponding version identifiers may be copied from a default version set (e.g., as shown at 406) to a nightly version set (e.g., nightly version set 408). In some embodiments, the nightly version set may be associated with a validation process (e.g., nightly unit and / or integration testing performed according to a predefined schedule). It should be appreciated that the nightly version set may correspond to any suitable validation process, and not necessarily to a validation process that is performed every night. The nightly version set may identify flock configuration identifiers and versions for each service of the region build. As shown at 408, when testing is initiated, the nightly version set may include flock configuration A, version 2, flock configuration "B", version 7, and flock configuration "C", version 3. Although different versions may still be used (as shown at 406), the default version set at the time testing is initiated is the set that is copied to the nightly version set. The service may be bootstrapped in a test environment using the flock configurations identified by the nightly version set (and the corresponding artifact versions identified within these flock configurations), and this version in the region may be validated.

[0093] In step 3, a validation process (including one or more unit and / or integration tests) may be performed to validate the bootstrapped services in the test environment. In some embodiments, the validation process may include any suitable predefined operations. These operations may be directed to testing various functions corresponding to the services identified by the nightly version set. A multi-flock orchestrator (MFO) may identify one or more errors that are provided as an output of the validation process. These errors may correspond to one or more services. In some embodiments, when an error corresponding to a particular service is identified, the MFO may be configured to restrict that version of the flock configuration from being promoted / copied to the golden version set maintained by the MFO and used for subsequent region builds.

[0094] In step 4, after completing the validation process in step 3, the MFO may be configured to determine a golden version set. If the service successfully passes the validation process (e.g., by generating no errors or at least by generating acceptable errors), the MFO may copy the corresponding flock configuration data (e.g., configuration identifier and version identifier) ​​to the golden version set. If the service fails the validation process (e.g., by generating errors or at least by generating unacceptable errors), the MFO may maintain the last known version of the flock configuration known to have passed the validation process. For example, if flock configuration "A", version 1 passes the validation process, it may be promoted to the golden version set. Thereafter, flock configuration "A", version 2 may be published to the default version set and later promoted to the nightly version set. However, if flock configuration "A", version 2 fails the validation process (e.g., generating one or more unacceptable errors), flock configuration "A", version 2 may not be promoted to the golden version set. Alternatively, the MFO may maintain block set "A", version 1 in the golden version set (eg, the golden version set shown at 410).

[0095] In step 5, upon initiating region construction (e.g., initiated by the MFO upon identifying new region data, or otherwise), the golden version set may be copied and / or retrieved by the MFO to be utilized for region construction. This process may generally correspond to step 5 of FIG. 5 described above.

[0096] The golden version set may be utilized for region builds. In some embodiments, the service team may identify an error in the flock configuration and provide an updated (or different) version of the flock configuration. In step 6, the updated flock configuration may be added to the Break Glass version set. This Break Glass version set may be used as an override of any appropriate flock configuration in the golden version set. For example, in step 6, flock configuration "A", version 9 may be added to the Break Glass version set. Version 9 of flock configuration "A" may differ from version 1 of the golden version set shown at 412. Adding version 9 of flock configuration "A" to the Break Glass version set may trigger an MFO to regenerate the build dependency graph 314 by performing a static analysis of the set of flock configurations in the Break Glass version set shown at 414, where flock configuration "A", version 1 is replaced with flock configuration "A", version 9. In some embodiments, any suitable data structures for the MFO to execute the construction dependency graph 314 and / or manage the bootstrap operation (e.g., one or more data structures generated and / or utilized by the declarative infrastructure provisioner 315 of FIG. 3) may be regenerated. Any regenerated data structures may be traversed in a manner similar to that described above where a node is visited. If a capability associated with the node (e.g., a capability on which the service corresponding to the node depends) is available (as identified by consulting with the capability service 112 of FIG. 1), the traversal may proceed to the next node for which the same evaluation may occur. This process may continue until a node is reached for which a capability on which the node depends is determined to be unavailable.At this point, the MFO may parse the traversal until it determines that the capabilities are available (e.g., by polling and / or by being notified by the capability service 112). For example, the MFO may send a request to the capability service 112 to be notified when capabilities on which the node depends are available, and the capability service 112 may be configured to provide a corresponding notification upon receiving data indicating that the capabilities are available. Alternatively, the MFO may send one or more requests on any suitable pre-defined schedule to request the status of the capabilities (or any suitable combination of capabilities, such as all capabilities) from the capability service 112. In some embodiments, the MFO may receive data from the capability service 112 at any suitable time indicating the status of the capabilities (e.g., that the capabilities are available or unavailable). In some embodiments, the capability service 112 may indicate the status of any suitable number of capabilities in any suitable transmission.

[0097] By utilizing the various version sets shown in Figure 4, the Multi-Flock Orchestrator (MFO) can reduce the number and / or likelihood of errors during region builds. This is achieved by allowing new flock configurations (potentially containing new artifact versions) to be named with the inclusion of the default version, running a validation process on nightly version sets generated from versions provided in the default version set, and promoting only those flock configuration versions that pass to the golden version set where subsequent region builds can be performed.

[0098] FIG. 5 is a block diagram illustrating an example method 500 for performing operations to bootstrap (e.g., provision and deploy) resources to a region, according to at least one embodiment. Method 500 may be performed by a multi-flock orchestrator 502 (an example of the multi-flock orchestrator 106 of FIG. 1). A capability service 504 may be an example of the capability service 112 of FIG. 1. Steps 1-9 of FIG. 5 may be performed prior to performing the operations of method 500. Thus, static analysis may be performed (e.g., by the multi-flock orchestrator 502). The term "static analysis" is intended to refer to an operation for identifying a set of dependencies between bootstrap operations from program code (e.g., configuration files such as flock configurations). Performing static analysis for a region build may include taking a set of configuration files (e.g., flock configurations corresponding to the golden version set, described above with respect to FIG. 4) and parsing these files to identify explicit and / or implicit dependencies between resources defined therein. In some embodiments, the configuration file may identify optional and / or required dependencies. Optional dependencies may identify dependencies that may be selectively ignored or enforced. Required dependencies indicate that the dependency is enforced without exception. In some embodiments, the multi-flock orchestrator 502 may generate one or more records to maintain and / or enforce these dependencies. For example, the multi-flock orchestrator 502 may generate a graph (e.g., the build dependency graph 314 of FIG. 3, a data structure that is an example of a record, etc.) based on the identified dependencies. Each node in the graph may correspond to a provisioning and deployment operation for the service. In some embodiments, separate graphs may be maintained for provisioning versus deployment operations.Exemplary techniques for generating and / or utilizing these records (e.g., data structures) are provided in U.S. Provisional Patent Application No. 63 / 315,034, entitled "Techniques for Managing Region Build Dependencies," the entire contents of which are incorporated by reference for all purposes. The following example utilizes a directed graph, however, any suitable record for maintaining dependencies between services / flocks may be utilized. In some embodiments, the method 500 may begin at least in part based on the multi-flock orchestrator (MFO) 502 identifying that a first node of a directed graph indicates a flock that has no dependencies.

[0099] It should be appreciated that a flock configuration may be used to encapsulate / identify a set of resources to be bootstrapped. In some cases, a flock configuration may identify all resources of a service, but it is not necessarily the case that all resources of a service are included in one flock configuration.

[0100] In step 1, the MFO 502 may execute instructions to provision infrastructure components according to a first flock configuration. For example, the MFO 502 may instruct and / or request a CIOS Central (e.g., CIOS Centrals 108, 214, and 304 of FIGS. 1-3, respectively) to bootstrap infrastructure components for Service 1. In some embodiments, the instructions / request may include the flock configuration to be utilized, or the request may indicate a version corresponding to the flock configuration to be utilized. Once the version is identified, the CIOS Central (not shown here) may retrieve (e.g., from DB 308 or 312 of FIG. 3, etc.) the flock configuration corresponding to the requested version. Subsequently, the CIOS Central may proceed to command / request provisioning of infrastructure resources for Service 1 from the CIOS Regional (e.g., an example of CIOS Regional 110, 216 and / or 314 of Figures 1-3, respectively) in a manner similar to that discussed above with respect to Figures 2 and 3 (e.g., after real-time regional data has been injected into the flock configuration).

[0101] In step 2, MFO 502 may instruct / request CIOS Central to deploy one or more artifacts to the infrastructure components provisioned in step 1. As in step 1, the request may include the flock configuration (or a version identifier for the flock configuration) to be utilized and / or a version identifier for each artifact to be deployed. The requests in steps 1 and 2 may each have a designator that identifies the request as corresponding to a provisioning request (e.g., a request to provision an infrastructure component) or a deployment request (e.g., a request to deploy various artifacts such as applications, configuration files, or the like). In some embodiments, MFO 502 may make one request to bootstrap service 1 indicating a request to provision infrastructure components for the service and a deployment request to deploy various artifacts to these infrastructure components. Thus, in some examples, steps 1 and 2 may be performed by one request from MFO 502 to CIOS Central. In other embodiments, each of steps 1 and 2 (or collectively steps 1 and 2 if issued together) may correspond to a "path" of MFO 502 through the graph. As a non-limiting example, a first pass through a graph may cause the operations of step 1 to be performed, while a second pass through the graph may cause the operations of step 2 to be performed.

[0102] In step 3, a capability (Capability 1) is published to a capability service 504 (e.g., an example of capability services 112, 208 and / or 308 of FIGS. 1-3, respectively). Publishing the capability may include calling an application programming interface (API) or another function call with an identifier for the capability (referred to as a “capability identifier”). The capability identifier may be of any suitable format, such as an alphanumeric value of any suitable length. The capability identifier may be associated with a particular function. In some embodiments, the capability may be published when one or more artifacts are deployed. The capability may be associated with a portion of the functionality provided by the service, or the capability may indicate that all functionality of the service is available. In some embodiments, multiple capabilities may be published in step 3 as part of the bootstrap service 1 (e.g., when infrastructure is provisioned for service 1 and / or when artifacts are deployed). The particular computing component that issues the capability may be any suitable component of CIOS 102 of FIG. 1 (e.g., CIOS regional 110, or, if ViBE is used, CIOS regional 216 or 314 of FIGS. 2 and 3, respectively, or CIOS regional (ViBE) 326 of FIG. 3, deployment orchestrator 218, 317, OS 330 of FIGS. 2 and 3, etc.).

[0103] In step 4, the MFO 502 may poll or otherwise request an indication of available capabilities. This request may be made at any suitable time, such as according to a predetermined periodicity, according to a predetermined schedule, etc. In some embodiments, the request may cause the capability service 504 to provide a list of all issued / received capability identifiers. In some embodiments, the request may include a particular capability identifier, and the response provided may indicate whether the capability identifier has been issued (e.g., whether the capability has been issued, whether the capability is available, etc.).

[0104] MFO 502 may evaluate whether other resources depend on the capabilities identified in step 4 via the graph (e.g., construction dependency graph 314) it is using to facilitate region construction. For example, the graph may indicate that service 2 depends on functionality associated with service 1. This dependency may be expressed as a capability. For example, the graph may maintain an association with service 2 indicating that service 1 cannot be bootstrapped until a capability (e.g., capability "1234") is issued indicating that service 1 is fully (or partially) bootstrapped / operational. Upon identifying that the required capability (e.g., capability "1234") is available via step 4, MFO 502 may continue traversing the graph and proceed to step 5.

[0105] In some embodiments, MFO 502 may identify that a capability is unavailable. In some embodiments, MFO 502 may issue an indication to capability service 504 that a capability is needed. MFO 502 may traverse the graph to identify that Service 2 depends on Service 1 (e.g., corresponding to capability "1234"). In some embodiments, MFO 502 may pause traversing the graph while waiting for a capability indicating that Service 1 is available. As various resources of the region become available, capabilities may be issued to capability service 504 indicating that these capabilities have become available. In some embodiments, once capability service 504 identifies that a particular capability has become available, capability service 504 may send data via any suitable electronic method, such as an application programming interface, a function call, etc., to notify MFO 502 of the same. Additionally or alternatively, the MFO 502 may periodically poll or request an identification of available capabilities (e.g., a particular set of one or more capabilities) and / or the capability service 504 may periodically publish the available capabilities. The MFO 702 may consider the capability unavailable and may refrain from further traversal of the graph (or traversal of a particular path in the graph) until the capability service 504 provides data indicating that the capability is available. Once the MFO 702 identifies that a capability has become available (e.g., based on data provided by the capability service 504), the MFO 702 may continue traversing the graph, whereupon additional services may be bootstrapped.

[0106] Steps 5, 6, and 7 may generally correspond to steps 1, 2, and 3 in terms of bootstrapping service 2. As described above, steps 5 and 6 may be performed separately or as one request. In step 7, capability 2 may be published, indicating that some or all of the functionality of service 2 is available. As described above, capability service 504 may persist this capability information.

[0107] In step 8, MFO 502 may determine that capabilities associated with Service 2 are available. As described above, MFO 502 may request this information (e.g., specifically by capability identifier or via obtaining a list of available capabilities) or capability service 504 may notify MFO 502 of the availability of the capabilities. In some embodiments, capability service 502 may specifically notify MFO 502, or the capabilities may be broadcast to one or more computing components (including MFO 502) that capabilities associated with Service 2 are available.

[0108] Steps 9, 10, 11 correspond to steps 1, 2 and 3, and / or 5, 6 and 7 in terms of bootstrapping service 3. Step 12 may correspond to steps 4 and 8.

[0109] The operations of method 500 may be performed any suitable number of times. For example, steps 1-4 may be performed any suitable number of times (e.g., for each service to be bootstrapped in a region) according to the traversal of the graph that MFO 502 is using to drive region construction. It should be appreciated that in some embodiments, any suitable portion of the steps of FIG. 5 may be performed in a parallel manner rather than the sequential manner depicted in FIG. 5. For example, if the resources of services 1-3 have no dependencies, or if all capabilities they depend on are available, it may be the case that at least a portion of the bootstrap operations performed for each service can be performed in a substantially parallel manner. Method 500 may be utilized to bootstrap services and / or resources directly into a ViBE (e.g., ViBE 524 of FIG. 5) or into a new region without using a ViBE (e.g., if the new region is available by communication).

[0110] FIG. 6 is a block diagram illustrating an example user interface 600 presenting information related to services associated with a region build, according to at least one embodiment. In some embodiments, the UI 600 may be hosted by any suitable component of the cloud infrastructure orchestration service 102 of FIG. 1 (e.g., CIOS Central 108, 214, and / or 304 of FIGS. 1-3, respectively). Any suitable data corresponding to a region build may be displayed in the user interface (UI) 600. In some embodiments, the UI 600 may include a UI element (e.g., UI element 602, a drop-down menu) for selecting a particular region build. Upon selecting a particular region build (e.g., Region X) from the UI element 602, data corresponding to the region build may be displayed in the UI 600.

[0111] The data displayed in UI 600 may initially correspond to a “Services” tab (e.g., Services tab 604). In some embodiments, Services tab 604 may include any suitable information corresponding to services that are bootstrapped in the selected region. As a non-limiting example, Services tab 604 may include columns 606-616 corresponding to a service identifier (column 606), a version set identifier (column 608), a change type (e.g., infrastructure or application / artifact) and status (e.g., success / failure) (column 610), consumed capabilities (column 612), created capabilities (column 614), and dependencies (column 616). In some embodiments, column 606 provides an identifier for each service that is bootstrapped in the selected region. In some embodiments, by clicking a service identifier (e.g., Service Identifier 617), the UI may filter the data provided to UI 600 in area 618 to only data associated with the selected service identifier. Selecting the version set identifier 619 may cause the UI 600 to display (eg, via a pop-up window) the version identifiers for the flock settings and / or artifacts associated with the given service and the version set identifier 619.

[0112] Column 608 displays a version set identifier (e.g., version set identifier 619, “Golden”) that indicates the currently active version set for the service. In some embodiments, an option may be selected to compare the current version set (e.g., Golden) with any other selected version set identifiers associated with the service so that a user may view and / or compare two version sets associated with the same service. In some embodiments, an icon (e.g., icon 628) may be displayed within column 608 for each service. Selecting the icon may display a popup or other suitable interface to display version set information. For example, an identifier for a flock set (e.g., a particular version used in the “Golden” version set) may be identified and / or one or more artifact identifiers and corresponding versions may be displayed. In some embodiments, selecting icon 628 may display version set information corresponding to any suitable number of version sets. For example, selecting icon 628 may provide a popup or other user interface in which information associated with the “Golden” version set may be provided. In some embodiments, if the service is associated with any other version sets (e.g., the Break Glass version set), information corresponding to the other version sets may be displayed. Thus, a user may access information corresponding to any appropriate version set (including, for example, a flock identifier that identifies a flock setting, a version identifier that corresponds to the flock setting, and data corresponding to any appropriate number of artifacts referenced by the flock setting). An instance of artifact data may include an identifier for the artifact (e.g., Application A) and a version identifier that corresponds to a version of the artifact (e.g., Version 1).In some embodiments, the version set information may include a release identifier that corresponds to a release associated with the service. Release. Column 610 displays information related to the provisioning of infrastructure and deployment of artifacts (e.g., applications, configuration files, etc.). In some embodiments, an identifier (e.g., a release identifier that identifies a particular release that corresponds to the infrastructure and / or artifacts) may be shown in column 610. A status may indicate the current status of the operation that corresponds to the release.

[0113] Column 612 displays an indicator that identifies which capabilities have been consumed. The consumed capabilities are capabilities on which the corresponding service that was processed depends. In other words, the consumed capabilities indicate capabilities that were issued in relation to other services on which the current service depends.

[0114] Column 614 displays indicators that identify which capabilities the service has generated. Generated capabilities are capabilities that have been issued in the process of bootstrapping a given service. Other services may depend on one or more of these generated capabilities.

[0115] In some embodiments, an icon may appear next to the capability identifier to indicate status. For example, icon 620 may indicate that a capability was successfully consumed / created. As shown in FIG. 6, icon 620 indicates that capability 2 was successfully created. Similarly, icon 621 indicates that a capability (e.g., capability 5) has not yet been created, as displayed in FIG. 6. Icon 623 may be used to indicate that a capability failed (e.g., was not created or was not consumed). Icon 625 may be used to indicate that a capability that the service depends on has failed. Icon 627 may be used to indicate a dependency on a capability that has not yet been created. Selecting any of the icons / capability indicators may cause a pop-up or other display to be displayed that shows the services that correspond to the selected capability. For example, selecting icon 627 (or an indicator corresponding to icon 627) may provide a pop-up window listing all services that depend on that capability (in this example, services 2 and 3).

[0116] UI 600 may include area 630. Area 630 may be configured to display any suitable information associated with the region. For example, area 630 may indicate a status of the region construction (none, building, creating, dormant, deprecated, etc.), a number of capabilities issued (e.g., 3), and a total number of capabilities expected to be issued (e.g., 123). In some embodiments, area 630 may display a progress indicator indicating the status of the capabilities. A portion 632 of the progress indicator may indicate capabilities that have been successfully issued. A portion 632 of the progress indicator may indicate capabilities that have been successfully issued. A portion 634 of the progress indicator may indicate capabilities that have not yet been issued. A portion 636 of the progress indicator may indicate capabilities that have failed.

[0117] UI 600 may include area 638. Area 638 may include any suitable UI element for filtering the data displayed in area 618. A user may define a set of filters associated with an identifier. These predefined filter sets may be displayed (e.g., by identifier) ​​in area 638, as shown at 640. In some embodiments, user input may be provided by selecting UI element 642. A user may enter any suitable filter via this element, which may filter the data displayed in area 618.

[0118] By selecting the Events tab 642, a user may navigate to the user interface (UI) 700 of FIG.

[0119] 7 is a block diagram illustrating an example user interface 700 that displays information related to events associated with a region build, according to at least one embodiment. The UI 700 may display a view of an events tab after selection via the UI 600. By selecting an events tab 702 (corresponding to events tab 642 of FIG. 6), data may be displayed in area 704. An event may correspond to a capability.

[0120] Events tab 702 may include columns 706-714 corresponding to an event identifier (column 706), a time / timestamp (column 708) corresponding to the time the event was received, a position (column 710) indicating the order in which capabilities are assumed to be issued, a type (column 712) indicating the type of event (e.g., capability), a name (column 714) identifying a particular service (flock configuration) with which the event is associated (e.g., issued by bootstrapping a service) or other identifier. Any suitable information associated with an event not displayed in area 704 may be displayed based on selection of option 716.

[0121] 8 illustrates an example method 800 for performing region construction (e.g., for modifying a region by incrementally performing a bootstrap operation) according to at least one embodiment. Method 800 may be performed by one or more components of cloud infrastructure orchestration service 102 of FIG. 1 (e.g., an orchestration service such as multi-flock orchestrator 106 of FIG. 1). A computer-readable storage medium including computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform method 800. Method 800 may be performed in any suitable order or in parallel. It should be appreciated that method 800 may include more or fewer steps than those shown in FIG. 8.

[0122] Method 800 may begin at 802, where a number of configuration files may be obtained that correspond to a number of services to be bootstrapped in a region corresponding to one or more data centers. In some embodiments, each of the multiple services may be associated with a respective set of resources including infrastructure components and corresponding software artifacts.

[0123] At 804, one or more dependencies between the multiple services may be identified based at least in part on the multiple configuration files. As described above, a static analysis may be performed, which may include parsing the configuration files to identify one or more dependencies (e.g., direct references to variables and / or identifiers associated with other services, indirect references to variables and / or identifiers associated with other services, etc.).

[0124] At 806, an order in which operations to bootstrap the multiple services are performed may be determined based at least in part on the one or more identified dependencies. As described above, a record, a data structure (e.g., a directed graph such as the construction dependency graph 314 of FIG. 5) may be generated to maintain knowledge of these dependencies.

[0125] At 808, a provisioning and deployment manager (e.g., CIOS Central 108 of FIG. 1) may be instructed to the increment to perform corresponding operations to bootstrap the multiple services according to the determined order. In some embodiments, such as when a data structure is used to manage dependencies, the data structure may be traversed in a manner similar to that described above in connection with FIG. 7 to drive bootstrap operations of any suitable number of services corresponding to the region construction to the increment.

[0126] FIG. 9 illustrates an example method 900 for utilizing a version set to perform a region build (e.g., bootstrap resources to one or more data centers in a region) according to at least one embodiment. Method 900 may be performed by one or more components of cloud infrastructure orchestration services 102 of FIG. 1 (e.g., an orchestration service such as multi-flock orchestrator 106 of FIG. 1). A computer-readable storage medium including computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform method 900. Method 900 may be performed in any suitable order or in parallel. Method 900 may include more or fewer steps than those illustrated in FIG. 900.

[0127] Method 900 may begin, at 902, where multiple version sets (e.g., the version sets of FIG. 6) may be maintained. Each of the multiple version sets may identify a respective set of configuration files for a plurality of configuration files associated with a plurality of services.

[0128] A first version set (e.g., a nightly version set) may be determined at 904. The first version set may identify a first set of configuration files from the plurality of configuration files.

[0129] At 906, a validation process may be performed to validate the first set of configuration files identified by the first version set.

[0130] At 908, a second version set (e.g., a golden version set) may be generated that identifies a second set of configuration files. In some embodiments, the second set of configuration files is identified from the first set of configuration files based at least in part on identifying flock configuration files that successfully passed the validation process.

[0131] At 910, a region build may be performed utilizing a second set of configuration files identified by the second version set.

[0132] Exemplary Cloud Service Infrastructure Architecture As mentioned above, Infrastructure as a Service (IaaS) is one particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may supply various services (e.g., billing, monitoring, logging, load balancing, and clustering, etc.) to accompany these infrastructure components. Thus, these services may be policy-driven, so that an IaaS user may be able to implement policies to drive load balancing to maintain application availability and performance.

[0133] In some examples, an IaaS customer may access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, and the like.

[0134] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but does not have to be, a third-party service that specializes in providing (offering, renting, selling) IaaS. An entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0135] In some examples, IaaS deployment is the process of adding a new application or a new version of an application to a provisioned application server, etc. It may also include the process of provisioning the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider below the hypervisor layer (e.g., server, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling the (OS), middleware, and / or application deployment (e.g., in self-service virtual machines (e.g., that can be spun up on demand)).

[0136] In some instances, IaaS provisioning may refer to acquiring a computer or virtual host for use and then installing required libraries or services on it. In most cases, deployment does not include provisioning, and provisioning may not need to be performed first.

[0137] In some cases, there are two distinct challenges for IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is operational. Second, there is the challenge of evolving the existing infrastructure once everything is provisioned (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges may be solved by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which components are required and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which, how they each work together) can be described declaratively. In some instances, once the topology is defined, workflows can be generated that generate and / or manage the different components described in the configuration files.

[0138] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potentially on-demand pools of configurable and / or shared computing resources), also known as a core network. In some examples, there may also be one or more inbound / outbound traffic groups and one or more virtual machines (VMs) that are provisioned to define how the network's inbound and / or outbound traffic is set up. Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. As more infrastructure elements are desired and / or added, the infrastructure may grow incrementally.

[0139] In some examples, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. In addition, the described techniques may enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, but often many, different production environments (e.g., across various different geographic locations, sometimes across the globe). However, in some examples, the infrastructure onto which the code will be deployed must first be set up. In some examples, provisioning may be done manually, provisioning tools may be utilized to provision the resources, and / or deployment tools may be utilized to deploy the code once the infrastructure is provisioned.

[0140] 10 is a block diagram 1000 illustrating an example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1002 can be communicatively coupled to a secure host tenancy 1004, which can include a virtual cloud network (VCN) 1006 and a secure host subnet 1008. In some examples, the service operator 1002 can use one or more client computing devices, which can be portable handheld devices (e.g., iPhone®, mobile phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display) running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and enabled for Internet, email, short message service (SMS), Blackberry®, or other communications protocols. Alternatively, the client computing devices can be general purpose personal computers including, for example, personal and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices can be, for example, workstation computers running any of the various commercially available UNIX or UNIX-like operating systems, including, without limitation, various GNU / Linux operating systems, such as Google Chrome OS.Alternatively, or in addition, the client computing devices may be any other electronic device, such as a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over a network that has access to the VCN 1006 and / or the Internet.

[0141] The VCN 1006 can include a local peering gateway (LPG) 1010 that can be communicatively coupled to a secure shell (SSH) VCN 1012 via an LPG 1010 included in the SSH VCN 1012. The SSH VCN 1012 can include an SSH subnet 1014, and the SSH VCN 1012 can be communicatively coupled to a control plane VCN 1016 via an LPG 1010 included in the control plane VCN 1016. The SSH VCN 1012 can also be communicatively coupled to a data plane VCN 1018 via the LPG 1010. The control plane VCN 1016 and the data plane VCN 1018 can be included in a service tenancy 1019, which can be owned and / or operated by the IaaS provider.

[0142] The control plane VCN 1016 can include a control plane demilitarized zone (DMZ) tier 1020 that serves as a perimeter network (e.g., the portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers may have limited responsibility and may help contain breaches. Additionally, the DMZ tier 1020 can include one or more load balancer (LB) subnets 1022, a control plane app tier 1024 that can include app subnets 1026, a control plane data tier 1028 that can include database (DB) subnets 1030 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1022 included in the control plane DMZ tier 1020 can be communicatively coupled to an app subnet 1026 included in the control plane app tier 1024 and an Internet gateway 1034 that can be included in the control plane VCN 1016, and the app subnet 1026 can be communicatively coupled to a DB subnet 1030, a service gateway 1036, and a network address translation (NAT) gateway 1038 included in the control plane data tier 1028. The control plane VCN 1016 can include the service gateway 1036 and the NAT gateway 1038.

[0143] The control plane VCN 1016 can include a data plane mirror Aplita 1040, which can include an app subnet 1026. The app subnet 1026 included in the data plane mirror Aplita 1040 can include a virtual network interface controller (VNIC) 1042 on which a compute instance 1044 can run. The compute instance 1044 can communicatively couple the app subnet 1026 of the data plane mirror Aplita 1040 to the app subnet 1026, which can be included in the data plane Aplita 1046.

[0144] The data plane VCN 1018 can include a data plane A-Tier 1046, a data plane DMZ Tier 1048, and a data plane Data Tier 1050. The data plane DMZ Tier 1048 can include a LB subnet 1022 that can be communicatively coupled to an app subnet 1026 of the data plane A-Tier 1046 and an Internet Gateway 1034 of the data plane VCN 1018. The app subnet 1026 can be communicatively coupled to a service gateway 1036 of the data plane VCN 1018 and a NAT gateway 1038 of the data plane VCN 1018. The data plane Data Tier 1050 can also include a DB subnet 1030 that can be communicatively coupled to the app subnet 1026 of the data plane A-Tier 1046.

[0145] The Internet gateways 1034 of the control plane VCNs 1016 and data plane VCNs 1018 can be communicatively coupled to a metadata management service 1052, which can be communicatively coupled to the public Internet 1054. The public Internet 1054 can be communicatively coupled to a NAT gateway 1038 of the control plane VCNs 1016 and data plane VCNs 1018. The service gateways 1036 of the control plane VCNs 1016 and data plane VCNs 1018 can be communicatively coupled to cloud services 1056.

[0146] In some examples, the service gateways 1036 of the control plane VCN 1016 and the data plane VCN 1018 can make application programming interface (API) calls to the cloud services 1056 without traversing the public Internet 1054. The API calls from the service gateway 1036 to the cloud services 1056 can be one-way. The service gateway 1036 can make the API calls to the cloud services 1056, and the cloud services 1056 can send the requested data to the service gateway 1036. However, the cloud services 1056 may not initiate the API calls to the service gateway 1036.

[0147] In some examples, the secure host tenancy 1004 can be directly connected to a service tenancy 1019 that may otherwise be isolated. The secure host subnet 1008 can communicate with the SSH subnet 1014 through an LPG 1010, which may allow bidirectional communication on an otherwise isolated system. Connecting the secure host subnet 1008 to the SSH subnet 1014 may give the secure host subnet 1008 access to other entities in the service tenancy 1019.

[0148] The control plane VCN 1016 may enable users of the service tenancy 1019 to set up or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1016 may be deployed or otherwise used in the data plane VCN 1018. In some examples, the control plane VCN 1016 can be isolated from the data plane VCN 1018, and the data plane mirror Apri tier 1040 of the control plane VCN 1016 can communicate with the data plane Apri tier 1046 of the data plane VCN 1018 via a VNIC 1042 that can be included in the data plane mirror Apri tier 1040 and the data plane Apri tier 1046.

[0149] In some examples, a user, or customer, of the system may make a request, e.g., create, read, update, or delete (CRUD) operations, over the public internet 1054, which may communicate the request to a metadata management service 1052. The metadata management service 1052 may communicate the request to the control plane VCN 1016 through an internet gateway 1034. The request may be received by a LB subnet 1022 included in the control plane DMZ tier 1020. The LB subnet 1022 may determine that the request is valid, and in response to this determination, the LB subnet 1022 may send the request to an app subnet 1026 included in the control plane app tier 1024. If the request is validated and requests a call to the public internet 1054, the call to the public internet 1054 may be sent to a NAT gateway 1038, which may make the call to the public internet 1054. Metadata that may be desired to be stored with the request may be stored in the DB subnet 1030.

[0150] In some examples, the data plane mirror ApriTia 1040 can facilitate direct communication between the control plane VCN 1016 and the data plane VCN 1018. For example, it may be desired that changes to the configuration, updates, or other appropriate modifications be applied to resources included in the data plane VCN 1018. Through the VNIC 1042, the control plane VCN 1016 can communicate directly with the resources included in the data plane VCN 1018, thereby enabling it to perform changes to the configuration, updates, or other appropriate modifications thereon. In some embodiments, the control plane VCN 1016 and the data plane VCN 1018 can be included in the service tenancy 1019. In this case, a user or customer of the system may not own or operate either the control plane VCN 1016 or the data plane VCN 1018. Instead, an IaaS provider may own or operate the control plane VCN 1016 and the data plane VCN 1018, both of which may be included in the service tenancy 1019. This embodiment may allow for network isolation that may prevent users or customers from interacting with other users' or other customers' resources. This embodiment may also allow users or customers of the system to store databases privately, without having to rely on the public Internet 1054 for storage, which may not have the desired level of threat protection.

[0151] In other embodiments, the LB subnet 1022 included in the control plane VCN 1016 can be configured to receive signals from the service gateway 1036. In this embodiment, the control plane VCN 1016 and the data plane VCN 1018 may be configured to be called by the IaaS provider's customers without calling the public Internet 1054. The IaaS provider's customers may desire this embodiment because databases used by the customers may be stored in the service tenancy 1019, which may be controlled by the IaaS provider and may be isolated from the public Internet 1054.

[0152] 11 is a block diagram 1100 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1102 (e.g., service operator 1002 of FIG. 10) can be communicatively coupled to a secure host tenancy 1104 (e.g., secure host tenancy 1004 of FIG. 10), which can include a virtual cloud network (VCN) 1106 (e.g., VCN 1006 of FIG. 10) and a secure host subnet 1108 (e.g., secure host subnet 1008 of FIG. 10). The VCN 1106 can include a local peering gateway (LPG) 1110 (e.g., LPG 1010 of FIG. 10), which can be communicatively coupled to a secure shell (SSH) VCN 1112 (e.g., SSH VCN 1012 of FIG. 10) via an LPG 1010 included in the SSH VCN 1112. SSH VCN 1112 can include an SSH subnet 1114 (e.g., SSH subnet 1014 in FIG. 10), and SSH VCN 1112 can be communicatively coupled to a control plane VCN 1116 (e.g., control plane VCN 1016 in FIG. 10) via an LPG 1110 included in the control plane VCN 1116. The control plane VCN 1116 can be included in a service tenancy 1119 (e.g., service tenancy 1019 in FIG. 10), and the data plane VCN 1118 (e.g., data plane VCN 1018 in FIG. 10) can be included in a customer tenancy 1121, which can be owned or operated by a user or customer of the system.

[0153] The control plane VCN 1116 may include a control plane DMZ tier 1120 (e.g., control plane DMZ tier 1020 of FIG. 10) which may include a LB subnet 1122 (e.g., LB subnet 1022 of FIG. 10), a control plane Aplica tier 1124 (e.g., control plane Aplica tier 1024 of FIG. 10) which may include an app subnet 1126 (e.g., app subnet 1026 of FIG. 10), and a control plane data tier 1128 (e.g., control plane data tier 1028 of FIG. 10) which may include a database (DB) subnet 1130 (e.g., similar to the DB subnet 1030 of FIG. 10). The LB subnet 1122 included in the control plane DMZ tier 1120 can be communicatively coupled to an app subnet 1126 included in the control plane app tier 1124 and an Internet gateway 1134 (e.g., Internet gateway 1034 of FIG. 10) that can be included in the control plane VCN 1116, and the app subnet 1126 can be communicatively coupled to a DB subnet 1130, a service gateway 1136 (e.g., service gateway 1036 of FIG. 10), and a network address translation (NAT) gateway 1138 (e.g., NAT gateway 1038 of FIG. 10) included in the control plane data tier 1128. The control plane VCN 1116 can include the service gateway 1136 and the NAT gateway 1138.

[0154] The control plane VCN 1116 can include a data plane mirror Apri tier 1140 (e.g., data plane mirror Apri tier 1040 of FIG. 10 ), which can include an app subnet 1126. The app subnet 1126 included in the data plane mirror Apri tier 1140 can include a virtual network interface controller (VNIC) 1142 (e.g., VNIC of 1042) on which a compute instance 1144 (e.g., similar to compute instance 1044 of FIG. 10 ) can run. The compute instance 1144 can facilitate communication between the app subnet 1126 of the data plane mirror Apri tier 1140 and a supplement subnet 1126 that can be included in the data plane Apri tier 1146 (e.g., data plane Apri tier 1046 of FIG. 10 ) via the VNIC 1142 included in the data plane mirror Apri tier 1140 and the VNIC 1142 included in the data plane Apri tier 1146.

[0155] An Internet gateway 1134 included in the control plane VCN 1116 can be communicatively coupled to a metadata management service 1152 (e.g., metadata management service 1052 of FIG. 10), which can be communicatively coupled to a public Internet 1154 (e.g., public Internet 1054 of FIG. 10). The public Internet 1154 can be communicatively coupled to a NAT gateway 1138 included in the control plane VCN 1116. A service gateway 1136 included in the control plane VCN 1116 can be communicatively coupled to cloud services 1156 (e.g., cloud services 1056 of FIG. 10).

[0156] In some examples, the data plane VCN 1118 can be included in the customer tenancy 1121. In this case, the IaaS provider may provide a control plane VCN 1116 for each customer, and the IaaS provider may set up a unique compute instance 1144 included in the service tenancy 1119 for each customer. Each compute instance 1144 can enable communication between the control plane VCN 1116 included in the service tenancy 1119 and the data plane VCN 1118 included in the customer tenancy 1121. The compute instance 1144 allows resources provisioned in the control plane VCN 1116 included in the service tenancy 1119 to be deployed or otherwise used in the data plane VCN 1118 included in the customer tenancy 1121.

[0157] In another example, an IaaS provider customer may have a database that resides in customer tenancy 1121. In this example, control plane VCN 1116 may include a data plane mirror Apri Tier 1140 that may include app subnet 1126. Data plane mirror Apri Tier 1140 may reside in data plane VCN 1118, but data plane mirror Apri Tier 1140 may not reside in data plane VCN 1118. That is, data plane mirror Apri Tier 1140 may have access to customer tenancy 1121, but data plane mirror Apri Tier 1140 may not reside in data plane VCN 1118 or be owned or operated by an IaaS provider customer. Data plane mirror Apri Tier 1140 may be configured to make calls to data plane VCN 1118, but may not be configured to make calls to any entities included in control plane VCN 1116. A customer may wish to deploy or otherwise use resources in a data plane VCN 1118 that are provisioned in a control plane VCN 1116, but the data plane mirror Aplita 1140 can facilitate the customer's desired deployment or other use of the resources.

[0158] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 1118. In this embodiment, the customer can determine which data plane VCNs 1118 can access, and the customer may limit access from the data plane VCN 1118 to the public Internet 1154. The IaaS provider may not be able to apply filters or otherwise control the access of the data plane VCN 1118 to any external networks or databases. Applying filters and controls by the customer to the data plane VCN 1118 included in the customer tenancy 1121 can help isolate the data plane VCN 1118 from other customers and the public Internet 1154.

[0159] In some embodiments, cloud services 1156 can be called by the service gateway 1136 to access services in the control plane VCN 1116 or data plane VCN 1118 that may not exist on the public Internet 1154. The connection between the cloud services 1156 and the control plane VCN 1116 or data plane VCN 1118 may not be live or continuous. The cloud services 1156 may exist on different networks owned or operated by the IaaS provider. The cloud services 1156 may be configured to receive calls from the service gateway 1136 and may be configured not to receive calls from the public Internet 1154. Some cloud services 1156 may be isolated from other cloud services 1156, and the control plane VCN 1116 may be isolated from cloud services 1156 that may not be in the same region as the control plane VCN 1116. For example, the control plane VCN 1116 may be located in "Region 1" and cloud service "Deployment 10" may be located in Region 1 and Region 2. When a call is made to deployment 10 by a service gateway 1136 included in a control plane VCN 1116 located in region 1, the call may be sent to the deployment 10 in region 1. In this example, the control plane VCN 1116, or the deployment 10 in region 1, may not be communicatively coupled to or otherwise in communication with the deployment 10 in region 2.

[0160] 12 is a block diagram 1200 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1202 (e.g., service operator 1002 of FIG. 10) can be communicatively coupled to a secure host tenancy 1204 (e.g., secure host tenancy 1004 of FIG. 10), which can include a virtual cloud network (VCN) 1206 (e.g., VCN 1006 of FIG. 10) and a secure host subnet 1208 (e.g., secure host subnet 1008 of FIG. 10). VCN 1206 can include an LPG 1210 (e.g., LPG 1010 of FIG. 10), which can be communicatively coupled to an SSH VCN 1212 (e.g., SSH VCN 1012 of FIG. 10) via an LPG 1210 included in SSH VCN 1212. SSH VCN 1212 can include an SSH subnet 1214 (e.g., SSH subnet 1014 in FIG. 10), which can be communicatively coupled to a control plane VCN 1216 (e.g., control plane VCN 1016 in FIG. 10) via an LPG 1210 included in the control plane VCN 1216, and to a data plane VCN 1218 (e.g., data plane 1018 in FIG. 10) via an LPG 1210 included in the data plane VCN 1218. The control plane VCN 1216 and the data plane VCN 1218 can be included in a service tenancy 1219 (e.g., service tenancy 1019 in FIG. 10).

[0161] The control plane VCN 1216 may include a control plane DMZ tier 1220 (e.g., control plane DMZ tier 1020 of FIG. 10 ) which may include a load balancer (LB) subnet 1222 (e.g., LB subnet 1022 of FIG. 10 ), a control plane Aplica tier 1224 (e.g., control plane Aplica tier 1024 of FIG. 10 ) which may include an App subnet 1226 (e.g., similar to App subnet 1026 of FIG. 10 ), and a control plane data tier 1228 (e.g., control plane data tier 1028 of FIG. 10 ) which may include a DB subnet 1230. The LB subnet 1222 included in the control plane DMZ tier 1220 can be communicatively coupled to an app subnet 1226 included in the control plane app tier 1224 and an Internet gateway 1234 (e.g., Internet gateway 1034 of FIG. 10) that can be included in the control plane VCN 1216, and the app subnet 1226 can be communicatively coupled to a DB subnet 1230, a service gateway 1236 (e.g., service gateway of FIG. 10), and a network address translation (NAT) gateway 1238 (e.g., NAT gateway 1038 of FIG. 10) included in the control plane data tier 1228. The control plane VCN 1216 can include the service gateway 1236 and the NAT gateway 1238.

[0162] The data plane VCN 1218 can include a data plane A-P-Tier 1246 (e.g., data plane A-P-Tier 1046 of FIG. 10), a data plane DMZ tier 1248 (e.g., data plane DMZ tier 1048 of FIG. 10), and a data plane data tier 1250 (e.g., data plane data tier 1050 of FIG. 10). The data plane DMZ tier 1248 can include a LB subnetwork 1222, which can be communicatively coupled to a trusted app subnetwork 1260 and an untrusted app subnetwork 1262 of the data plane A-P-Tier 1246, and an Internet gateway 1234 included in the data plane VCN 1218. The trusted app subnetwork 1260 can be communicatively coupled to a service gateway 1236 included in the data plane VCN 1218, a NAT gateway 1238 included in the data plane VCN 1218, and a DB subnetwork 1230 included in the data plane data tier 1250. The untrusted app subnet 1262 can be communicatively coupled to a service gateway 1236 included in the data plane VCN 1218 and a DB subnet 1230 included in the data plane data tier 1250. The data plane data tier 1250 can include a DB subnet 1230 that can be communicatively coupled to a service gateway 1236 included in the data plane VCN 1218.

[0163] The untrusted app subnet 1262 can include one or more primary VNICs 1264(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1266(1)-(N). Each tenant VM 1266(1)-(N) can be communicatively coupled to a respective app subnet 1267(1)-(N) that can be included in a respective container egress VCN 1268(1)-(N) that can be included in a respective customer tenancy 1270(1)-(N). Each secondary VNIC 1272(1)-(N) can facilitate communication between the untrusted app subnet 1262 included in the data plane VCN 1218 and the app subnet included in the container egress VCN 1268(1)-(N). Each container egress VCN 1268(1)-(N) may include a NAT gateway 1238 that may be communicatively coupled to the public Internet 1254 (e.g., the public Internet 1054 in FIG. 10).

[0164] An Internet gateway 1234 included in the control plane VCN 1216 and the data plane VCN 1218 can be communicatively coupled to a metadata management service 1252 (e.g., metadata management system 1052 of FIG. 10 ), which can be communicatively coupled to the public Internet 1254. The public Internet 1254 can be communicatively coupled to a NAT gateway 1238 included in the control plane VCN 1216 and the data plane VCN 1218. A service gateway 1236 included in the control plane VCN 1216 and the data plane VCN 1218 can be communicatively coupled to cloud services 1256.

[0165] In some embodiments, data plane VCN 1218 can be integrated with customer tenancies 1270. This integration may be useful or desirable for an IaaS provider's customer in some cases, such as when they may want support when executing code. The customer may provide code to execute that may be disruptive, may communicate with other customer resources, or may otherwise produce undesirable effects. In response, the IaaS provider may determine whether to execute the code provided to the IaaS provider by the customer.

[0166] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider and may request a function to be attached to data plane applitier 1246. Code to execute the function may be executed in VMs 1266(1)-(N), and the code may be configured to not execute anywhere else in data plane VCN 1218. Each VM 1266(1)-(N) may be connected to one customer tenancy 1270. Each container 1271(1)-(N) included in VM 1266(1)-(N) may be configured to execute code. In this case, there may be dual isolation (e.g., containers 1271(1)-(N) executing code, where containers 1271(1)-(N) may be contained in at least VMs 1266(1)-(N) contained in untrusted app subnet 1262), which may help prevent erroneous or otherwise unwanted code from damaging the IaaS provider's network or damaging a different customer's network. Containers 1271(1)-(N) may be communicatively coupled to customer tenancy 1270 and configured to send or receive data from customer tenancy 1270. Containers 1271(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 1218. Once the code execution is complete, the IaaS provider may kill or otherwise discard containers 1271(1)-(N).

[0167] In some embodiments, trusted app subnet 1260 may execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 1260 may be communicatively coupled to DB subnet 1230 and may be configured to perform CRUD operations on DB subnet 1230. Untrusted app subnet 1262 may be communicatively coupled to DB subnet 1230, but in this embodiment, untrusted app subnet 1262 may be configured to perform read operations on DB subnet 1230. Containers 1271(1)-(N), which may be included in each customer's VMs 1266(1)-(N) and may execute code from the customer, may not be communicatively coupled to DB subnet 1230.

[0168] In other embodiments, the control plane VCN 1216 and the data plane VCN 1218 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 1216 and the data plane VCN 1218. However, communication may occur indirectly through at least one method. An LPG 1210 may be established by an IaaS provider, which may facilitate communication between the control plane VCN 1216 and the data plane VCN 1218. In another example, the control plane VCN 1216 or the data plane VCN 1218 may make a call to a cloud service 1256 through a service gateway 1236. For example, a call from the control plane VCN 1216 to the cloud service 1256 may include a request for a service that may communicate with the data plane VCN 1218.

[0169] 13 is a block diagram 1300 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1302 (e.g., service operator 1002 of FIG. 10) can be communicatively coupled to a secure host tenancy 1304 (e.g., secure host tenancy 1004 of FIG. 10), which can include a virtual cloud network (VCN) 1306 (e.g., VCN 1006 of FIG. 10) and a secure host subnet 1308 (e.g., secure host subnet 1008 of FIG. 10). The VCN 1306 can include an LPG 1310 (e.g., LPG 1010 of FIG. 10), which can be communicatively coupled to an SSH VCN 1312 (e.g., SSH VCN 1012 of FIG. 10) via an LPG 1310 included in the SSH VCN 1312. SSH VCN 1312 can include an SSH subnet 1314 (e.g., SSH subnet 1014 in FIG. 10), which can be communicatively coupled to a control plane VCN 1316 (e.g., control plane VCN 1016 in FIG. 10) via an LPG 1310 included in the control plane VCN 1316, and to a data plane VCN 1318 (e.g., data plane VCN 1018 in FIG. 10) via an LPG 1310 included in the data plane VCN 1318. The control plane VCN 1316 and the data plane VCN 1318 can be included in a service tenancy 1319 (e.g., service tenancy 1019 in FIG. 10).

[0170] The control plane VCN 1316 may include a control plane DMZ tier 1320 (e.g., control plane DMZ tier 1020 of FIG. 10) which may include a LB subnet 1322 (e.g., LB subnet 1022 of FIG. 10), a control plane Aplica tier 1324 (e.g., control plane Aplica tier 1024 of FIG. 10) which may include an App subnet 1326 (e.g., App subnet 1026 of FIG. 10), and a control plane data tier 1328 (e.g., control plane data tier 1028 of FIG. 10) which may include a DB subnet 1330 (e.g., DB subnet 1230 of FIG. 12). The LB subnet 1322 included in the control plane DMZ tier 1320 can be communicatively coupled to an app subnet 1326 included in the control plane app tier 1324 and an Internet gateway 1334 (e.g., Internet gateway 1034 of FIG. 10) that can be included in the control plane VCN 1316, and the app subnet 1326 can be communicatively coupled to a DB subnet 1330, a service gateway 1336 (e.g., service gateway of FIG. 10), and a network address translation (NAT) gateway 1338 (e.g., NAT gateway 1038 of FIG. 10) included in the control plane data tier 1328. The control plane VCN 1316 can include the service gateway 1336 and the NAT gateway 1338.

[0171] The data plane VCN 1318 can include a data plane A-P-Tier 1346 (e.g., data plane A-P-Tier 1046 of FIG. 10), a data plane DMZ tier 1348 (e.g., data plane DMZ tier 1048 of FIG. 10), and a data plane data tier 1350 (e.g., data plane data tier 1050 of FIG. 10). The data plane DMZ tier 1348 can include a trusted app subnet 1360 (e.g., trusted app subnet 1260 of FIG. 12) and an untrusted app subnet 1362 (e.g., untrusted app subnet 1262 of FIG. 12) of the data plane A-P-Tier 1346, as well as a LB subnet 1322 that can be communicatively coupled to an Internet gateway 1334 included in the data plane VCN 1318. The trusted app subnet 1360 may be communicatively coupled to a service gateway 1336 included in the data plane VCN 1318, a NAT gateway 1338 included in the data plane VCN 1318, and a DB subnet 1330 included in the data plane data tier 1350. The untrusted app subnet 1362 may be communicatively coupled to a service gateway 1336 included in the data plane VCN 1318 and a DB subnet 1330 included in the data plane data tier 1350. The data plane data tier 1350 may include a DB subnet 1330 that may be communicatively coupled to a service gateway 1336 included in the data plane VCN 1318.

[0172] The untrusted app subnet 1362 can include primary VNICs 1364(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1366(1)-(N) that reside within the untrusted app subnet 1362. Each tenant VM 1366(1)-(N) can execute code in a respective container 1367(1)-(N) and can be communicatively coupled to an app subnet 1326 that can be included in a data plane app tier 1346 that can be included in a container egress VCN 1368. Each secondary VNIC 1372(1)-(N) can facilitate communication between the untrusted app subnet 1362 included in the data plane VCN 1318 and the app subnet included in the container egress VCN 1368. The container egress VCN can include a NAT gateway 1338 that can be communicatively coupled to the public Internet 1354 (e.g., public Internet 1054 of FIG. 10).

[0173] An Internet gateway 1334 included in the control plane VCN 1316 and the data plane VCN 1318 can be communicatively coupled to a metadata management service 1352 (e.g., metadata management service 1052 of FIG. 10 ), which can be communicatively coupled to the public Internet 1354. The public Internet 1354 can be communicatively coupled to a NAT gateway 1338 included in the control plane VCN 1316 and the data plane VCN 1318. A service gateway 1336 included in the control plane VCN 1316 and the data plane VCN 1318 can be communicatively coupled to cloud services 1356.

[0174] In some examples, the pattern illustrated by the architecture of block diagram 1300 of FIG. 13 may be considered an exception to the pattern illustrated by the architecture of block diagram 1200 of FIG. 12 and may be desirable for an IaaS provider's customer when the IaaS provider cannot directly communicate with the customer (e.g., disconnected region). Each container 1367(1)-(N) included in VM 1366(1)-(N) for each customer can be accessed in real time by the customer. The containers 1367(1)-(N) may be configured to make calls to a respective secondary VNIC 1372(1)-(N) included in app subnet 1326 of data plane app tier 1346 that may be included in container egress VCN 1368. The secondary VNIC 1372(1)-(N) can send the call to NAT gateway 1338, which may send the call to public Internet 1354. In this example, containers 1367(1)-(N) that may be accessed in real time by customers may be isolated from control plane VCN 1316 and may be isolated from other entities included in data plane VCN 1318. Containers 1367(1)-(N) may be isolated from resources from other customers.

[0175] In another example, a customer may use containers 1367(1)-(N) to call cloud service 1356. In this example, the customer may execute code in containers 1367(1)-(N) that requests a service from cloud service 1356. Containers 1367(1)-(N) can send the request to secondary VNICs 1372(1)-(N), which can send the request to a NAT gateway, which can send the request to public Internet 1354. Public Internet 1354 can send the request to LB subnet 1322 included in control plane VCN 1316 via Internet gateway 1334. In response to determining that the request is valid, LB subnet can send the request to app subnet 1326, which can send the request to cloud service 1356 via service gateway 1336.

[0176] It should be appreciated that the illustrated IaaS architectures 1000, 1100, 1200, 1300 may have components other than those shown. Additionally, the illustrated embodiments are only some examples of cloud infrastructure systems that may incorporate disclosed embodiments. In some other embodiments, the IaaS systems may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of components.

[0177] In one embodiment, the IaaS system described herein may include a suite of application, middleware, and database service offerings that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI), offered by the present assignee.

[0178] 14 illustrates an exemplary computer system 1400 upon which various embodiments may be implemented. The system 1400 may be used to implement any of the computer systems described above. As illustrated, the computer system 1400 includes a processing unit 1404 that communicates with a number of peripheral subsystems via a bus subsystem 1402. These peripheral subsystems may include a processing acceleration unit 1406, an I / O subsystem 1408, a storage subsystem 1418, and a communication subsystem 1424. The storage subsystem 1418 includes a tangible computer readable storage medium 1422 and a system memory 1410.

[0179] The bus subsystem 1402 provides a mechanism for allowing the various components and subsystems of the computer system 1400 to communicate with each other as intended. Although the bus subsystem 1402 is shown diagrammatically as one bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1402 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P 1386.1 standard.

[0180] Processing unit 1404, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of computer system 1400. One or more processors may be included in processing unit 1404. These processors may include single-core or multi-core processors. In some embodiments, processing unit 1404 may be implemented as one or more independent processing units 1432 and / or 1434 with single-core or multi-core processors included in each processing unit. In other embodiments, processing unit 1404 may be implemented as a quad-core processing unit formed by integrating two dual-core processors into one chip.

[0181] In various embodiments, the processing unit 1404 can execute various programs in response to program code and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code being executed may reside in the processor 1404 and / or in the memory subsystem 1418. With appropriate programming, the processor 1404 can provide various functions as described above. The computer system 1400 may additionally include a processing acceleration unit 1406, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0182] The I / O subsystem 1408 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include motion sensing and / or gesture recognition devices such as a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and voice instructions. User interface input devices may also include eye gesture recognition devices such as a Google Glass® blink detector that detects eye movements from a user (e.g., "blinking" when taking a picture and / or selecting a menu) and translates the eye gestures as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® Navigator) through voice commands.

[0183] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye gaze detection devices. In addition, user interface input devices may also include, for example, medical image input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0184] The user interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as a flat panel device using a liquid crystal display (LCD) or a plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1400 to a user or to another computer. For example, user interface output devices may include, without limitation, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0185] Computer system 1400 may include a storage subsystem 1418 that provides a tangible, non-transitory computer-readable storage medium for storing software and data structures that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of processing unit 1404, provide the functionality described above. Storage subsystem 1418 may also provide a repository for storing data used in accordance with the present disclosure.

[0186] 14, the storage subsystem 1418 can include various components including a system memory 1410, a computer readable storage medium 1422, and a computer readable storage medium reader 1420. The system memory 1410 may store program instructions loadable and executable by the processing unit 1404. The system memory 1410 may store data used during the execution of the instructions and / or data generated during the execution of the program instructions. A variety of different types of programs may be loaded into the system memory 1410 including, but not limited to, client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), virtual machines, containers, and the like.

[0187] The system memory 1410 may store an operating system 1416. Examples of operating systems 1416 may include various versions of Microsoft Window, Apple Macintosh, and / or Linux operating systems, various commercially available UNIX or UNIX-like operating systems (including, without limitation, various GNU / Linux operating systems, Google Chrome OS, etc.) and / or mobile operating systems, such as iOS, Windows Phone, Android OS, BlackBerry OS, and Palm OS operating systems. In certain implementations in which the computer system 1400 runs one or more virtual machines, the virtual machine with a guest operating system (GOS) may be loaded into the system memory 1410 and executed by one or more processors or cores of the processing unit 1404.

[0188] The system memory 1410 may be provided in different configurations depending on the type of computer system 1400. For example, the system memory 1410 may be volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., read only memory (ROM), flash memory, etc.). Different types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), and others. In some implementations, the system memory 1410 may include a basic input / output system (BIOS) containing the basic routines that help to transfer information between elements within the computer system 1400, such as during start-up.

[0189] Computer-readable storage media 1422 may represent remote, local, fixed, and / or removable storage devices and media for temporarily and / or more permanently containing and storing computer-readable information for use by computer system 1400, including instructions executable by processing unit 1404 of computer system 1400.

[0190] The computer readable storage medium 1422 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storing and / or transmitting information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other tangible computer readable media.

[0191] For example, the computer readable storage medium 1422 may include a hard disk drive that reads from or writes to a non-removable, non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable, non-volatile magnetic disk, an optical disk drive that reads from or writes to a removable, non-volatile optical disk or other optical medium, such as a CD-ROM, DVD, and Blu-Ray® disk. The computer readable storage medium 1422 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer readable storage medium 1422 may also include a flash memory-based SSD, an enterprise flash drive, a solid-state drive (SSD) based on a non-volatile memory such as a solid-state ROM, a solid-state RAM, a dynamic RAM, a static RAM, a DRAM-based SSD, an SSD based on a volatile memory such as a magnetoresistive RAM (MRAM) SSD, a hybrid SSD that uses a combination of DRAM and a flash memory SSD. The disk drives and their associated computer-readable media may provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for computer system 1400.

[0192] Machine-readable instructions executable by one or more processors or cores of the processing unit 1404 may be stored in a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0193] The communications subsystem 1424 provides an interface to other computer systems and networks. The communications subsystem 1424 serves as an interface for receiving data from other systems and transmitting data from the computer system 1400 to other systems. For example, the communications subsystem 1424 may enable the computer system 1400 to be connected to one or more devices via the Internet. In some embodiments, the communications subsystem 1424 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using advanced data network technologies such as cellular technology, 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components). In some embodiments, the communications subsystem 1424 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0194] In some embodiments, the communications subsystem 1424 may receive incoming communications in the form of structured and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc. on behalf of one or more users who may be using the computer system 1400.

[0195] For example, the communications subsystem 1424 may be configured to receive data feeds 1426 in real time from users of social networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0196] Additionally, the communications subsystem 1424 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1428 of real-time events and / or event updates 1430, which may be continuous or essentially open-ended. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0197] The communications subsystem 1424 may be configured to output structured and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc. to one or more databases that may communicate with one or more streaming data source computers coupled to the computer system 1400.

[0198] The computer system 1400 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0199] Due to the ever-changing nature of computers and networks, the illustrated description of computer system 1400 is intended as a specific example only. Many other configurations having more or fewer components than the illustrated system are possible. For example, customized hardware may be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods for implementing the various embodiments.

[0200] Although specific embodiments have been described, various modifications, variations, alternative configurations, and equivalents are within the scope of the disclosure. The embodiments are not limited to operation in one particular data processing environment, but are free to operate in multiple data processing environments. Although the embodiments have been described with a particular sequence of transactions and steps, it should be apparent to one skilled in the art that the scope of the disclosure is not limited to the described sequence of transactions and steps. Various features and aspects of the above-described embodiments may be used individually or in combination.

[0201] Furthermore, while embodiments are described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. The embodiments may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein can be implemented on the same processor or on different processors in any combination. Thus, where a component or module is described as being configured to perform a certain operation, such configuration can be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or in any combination thereof. The processors can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0202] The embodiments may be implemented by using a computer program product comprising computer programs / instructions which, when executed by a processor, cause the processor to perform any of the methods described in the disclosure.

[0203] Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, although certain disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are intended to be within the scope of the following claims.

[0204] In the context of describing the disclosed embodiments (particularly in the context of the claims that follow), the use of the terms "a," "an," and "the" and similar referents should be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise noted. The term "connected" should be construed as partially or wholly contained within, attached to, or joined to, even if there is something intervening. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of referring to each separate value contained within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by context. The use of any and all examples or exemplary language (e.g., "etc.") provided herein is intended merely to better illuminate the embodiments and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0205] Disjunctive language, such as the phrase "at least of X, Y, or Z," is intended to be understood in context as being generally used to indicate that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y and / or Z), unless specifically stated otherwise. Thus, such disjunctive language is not intended, and should not generally imply, that an embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0206] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the disclosure. Variations of these preferred embodiments may become apparent to those skilled in the art upon reading the above description. Those skilled in the art should be able to adopt such variations as appropriate, and the disclosure may be carried out in forms other than those specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure, unless otherwise indicated herein.

[0207] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0208] In the foregoing specification, aspects of the disclosure are described in relation to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or in combination. Moreover, the embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. 1. A computer-implemented method comprising: an orchestration service of a cloud computing environment including obtaining a plurality of configuration files corresponding to a plurality of services to be bootstrapped within a region corresponding to one or more data centers, each of the plurality of services being associated with a respective set of resources including at least one of infrastructure components or corresponding software artifacts; The method comprises: the orchestration service identifying one or more dependencies among the plurality of services based at least in part on the plurality of configuration files; determining, by the orchestration service, an order in which operations for bootstrapping the plurality of services are to be performed based at least in part on the identified one or more dependencies; the orchestration service incrementally instructing a provisioning and deployment manager to perform corresponding operations to bootstrap the plurality of services according to the determined order; The computer-implemented method further includes:

2. maintaining a record of said one or more dependencies; Identifying that a first service has a corresponding dependency on a capability of an available second service; identifying that the capability of the second service is unavailable; and sending data to a capability service indicating a request to be notified when said capability becomes available; The computer-implemented method of claim 1 further comprising:

3. 3. The computer-implemented method of claim 2, wherein identifying that the capability of the second service is unavailable includes determining that notification indicating that the capability of the second service is available has not yet been received from the capability service of the cloud computing environment.

4. instructing the provisioning and deployment manager to increment receiving a corresponding notification from the capability service indicating that the capability is available; instructing the provisioning and deployment manager to perform an operation to bootstrap resources corresponding to the second service within the region in response to receiving the notification that the capability is available; The computer-implemented method of claim 1 further comprising:

5. 2. The computer-implemented method of claim 1, wherein the provisioning and deployment manager provisions at least one infrastructure component and deploys one or more artifacts to the at least one infrastructure component by incrementally instructing the provisioning and deployment manager to perform the corresponding operations to bootstrap the plurality of services.

6. The computer-implemented method of claim 1 , wherein the one or more dependencies are identified based at least in part on performing a parse of one or more of the plurality of configuration files.

7. 2. The computer-implemented method of claim 1, further comprising: the orchestration service generating one or more directed graphs based at least in part on identifying the one or more dependencies and an order in which operations for bootstrapping the plurality of services are to be performed; and wherein instructing the provisioning and deployment manager incrementally to perform the corresponding operations for bootstrapping the plurality of services according to the determined order is based at least in part on one or more traversals of the one or more directed graphs.

8. 1. A cloud computing system, comprising: one or more processors; When executed by the one or more processors, the orchestration service of the cloud computing system obtaining a plurality of configuration files corresponding to a plurality of services to be bootstrapped within a region corresponding to one or more data centers, each of the plurality of services associated with a respective set of resources including infrastructure components and corresponding software artifacts; identifying one or more dependencies among the plurality of services based at least in part on the plurality of configuration files; determining an order in which operations for bootstrapping the plurality of services are performed based at least in part on the identified one or more dependencies; and instructing a provisioning and deployment manager to increment the order of the services to execute corresponding operations for bootstrapping the plurality of services according to the determined order. and one or more memories that store computer-executable instructions.

9. Executing the instructions further includes causing the orchestration service to: Identifying that a first service has a corresponding dependency on a capability of an available second service; identifying that the capability of the second service is unavailable; The cloud computing system of claim 8 , further comprising causing a capability service to transmit data indicating a request to be notified when the capability becomes available.

10. 10. The cloud computing system of claim 9, wherein executing the instructions that cause the orchestration service to identify that the capability of the second service is unavailable further causes the orchestration service to determine that notification indicating that the capability of the second service is available has not yet been received from the capability service of the cloud computing system.

11. Executing the instructions to instruct the provisioning and deployment manager to increment further includes instructing the orchestration service to: receiving a corresponding notification from the capability service indicating that the capability is available; 9. The cloud computing system of claim 8, further comprising: instructing the provisioning and deployment manager to perform operations to bootstrap resources corresponding to the second service within the region in response to receiving the notification that the capability is available.

12. 9. The cloud computing system of claim 8, wherein executing the instructions to incrementably instruct the provisioning and deployment manager to perform the corresponding operations to bootstrap the plurality of services causes the provisioning and deployment manager to provision at least one infrastructure component and deploy one or more artifacts to the at least one infrastructure component.

13. The cloud computing system of claim 8 , wherein the one or more dependencies are identified based at least in part on performing a parsing of one or more of the plurality of configuration files.

14. 14. The cloud computing system of claim 8, wherein executing the instructions further causes the orchestration service to generate one or more directed graphs based at least in part on identifying the one or more dependencies and an order in which operations for bootstrapping the plurality of services are to be performed, and wherein instructing the provisioning and deployment manager incrementally to perform the corresponding operations for bootstrapping the plurality of services according to the determined order is based at least in part on one or more traversals of the one or more directed graphs.

15. A program for causing a computer to execute the method described in any one of claims 1 to 7.

Images