Service plans and manifests for data center orchestration

Abstract

A cloud infrastructure orchestration service may maintain a service plan and manifest (SPAM) corresponding to services to be directed (e.g., provisioned and deployed) to a cloud computing environment (e.g., to a data center). The service plan may specify a deterministic order of release for performing processing of the complete boot service using one or more build milestones and one or more execution units, each execution unit specifying ordered steps for transitioning between build milestones. Each step may reference one or more execution target checkpoint transitions, which in turn reference an alias that defines the released configuration file. The manifest may be used to identify profiles and workpieces to be used by the release, and to validate the service plan. SPAM may be used to reduce / eliminate uncertain behavior of a previously orchestrated system and provide visualization of guided processing at different granularities.

Description

[0001] Cross Reference to Related Applications

[0002] This non-provisional application claims priority to U.S. Provisional Patent Application No. 63 / 503,145, filed May 18, 2023, entitled “Region Build Orchestration using Skills and Capabilities,” and U.S. Non-Provisional Patent Application No. 18 / 658,638, filed May 8, 2024, entitled “Service Plans and Manifests for Data Center Orchestration,” the disclosures of which are incorporated by reference herein in their entireties for all purposes. TECHNICAL FIELD

[0003] The present disclosure relates to service plans and manifests (SPAM) that leverage specifications for bootstrapping processes of individual services as individual services. Service plans and manifests (SPAM) provide a complete service build description that specifies releases that can be necessary (or required) to build a service and a deterministic / explicit order of these releases. The SPAM can include explicit expectations of progress expected for each transition (e.g., each release execution corresponds to a particular phase / execution goal). One or more services (e.g., all services to be bootstrapped in a region) can be associated with a corresponding SPAM. The information provided by these SPAMs can be used to eliminate various errors that can occur in a data center build process by identifying problems early in the build life cycle (e.g., upon submission of the SPAM) rather than at build time. BACKGROUND

[0004] Today, cloud infrastructure services leverage many individual services to build data centers (e.g., to bootstrap various resources in a data center in a particular geographic region). A region is a logical abstraction that corresponds to a local geographic area in which one or more data centers are (or will be) located. Building a data center (also referred to as performing a “region build”) can include provisioning and configuring infrastructure resources and deploying code to the resources (e.g., to implement various services). Any suitable number of data centers can be included in a region, so a region build can include operations for building multiple data centers. A bootstrapping operation of one service can depend on the availability of other functions and / or services of the region. As the number of service teams and regions grows, the tasks performed to orchestrate provisioning and deployment increase dramatically. Conventional tools for building a region require significant manual effort, or automated techniques suffer from deficiencies in overhead, accuracy, and ease of use. Improvements can be made. SUMMARY

[0005] Embodiments of the present disclosure relate to techniques for building data centers. Previous implementations do not derive a centralized description of the operations required to build a service from them. Instead, service build information is distributed across numerous configuration files. Current implementations lack a specification of how to build a service and include only implementation details. This leads to a lack of understanding of service builds and greatly increases the effort required to resolve problems during data center builds. The present disclosure is directed to utilizing service plans and manifests. A service plan refers to a description of how to build a service in a data center. A "service manifest" details which configuration files (e.g., "groups") and artifacts (e.g., software) are to be utilized to build a service.

[0006] At least one embodiment is directed to a computer-implemented method (referred to simply as "method"). The computer-implemented method can include maintaining, by a cloud infrastructure orchestration system, a service plan and a manifest corresponding to a service of a plurality of services to be launched within a cloud computing environment. In some embodiments, the service plan specifies a process for launching the service within the cloud computing environment. The process for launching the service is associated, within the service plan, with an ordered set of build milestones and an ordered set of releases. In some embodiments, the manifest identifies at least one of a set of configuration files or a set of artifacts with which to perform the process for launching the service. The method can include generating, by the cloud infrastructure orchestration system, a first representation that depicts the ordered set of build milestones associated with the process for launching the service within the cloud computing environment. The method can include generating, by the cloud infrastructure orchestration system, a second representation that depicts the ordered set of releases associated with the process for launching the service within the cloud computing environment. The method can include providing, by the cloud infrastructure orchestration system, the first representation that depicts the ordered set of build milestones associated with the process for launching the service within the cloud computing environment at a user interface. The method can include providing, by the cloud infrastructure orchestration system, the second representation that depicts the ordered set of releases associated with the process for launching the service within the cloud computing environment at the user interface.

[0007] In some embodiments, each build milestone of the ordered set of build milestones can be associated with a corresponding subset of the ordered set of releases.

[0008] In some embodiments, the first representation includes a plurality of nodes and a plurality of edges. Each node of the plurality of nodes can correspond to a build milestone of the ordered set of build milestones. Each edge of the plurality of edges can represent a transition between two build milestones of the ordered set of build milestones.

[0009] In some embodiments, a build milestone in the ordered set of build milestones can be associated with one or more external capability dependency relationships indicating a dependency on a first capability expected to be published by a second service during a second processing for bootstrapping the second service within the cloud computing environment.

[0010] In some embodiments, the service plan further includes an execution unit specifying a transition between two build milestones in the ordered set of build milestones. The execution unit can associate the transition with a subset of the ordered set of releases.

[0011] In some embodiments, the execution unit is associated with: 1) an external capability dependency relationship indicating a dependency on a first capability to be published by another one of the plurality of services; and 2) an internal capability indicating a dependency on a second capability expected to be published by the service.

[0012] In some embodiments, the first representation depicts the processing for bootstrapping the service within the cloud computing environment at a first granularity, and the second representation depicts the processing for bootstrapping the service within the cloud computing environment at a second granularity. In some embodiments, the second granularity can be different than the first granularity.

[0013] Another embodiment is directed to a cloud computing service including one or more processors and memory storing instructions that, when executed by the one or more processors, cause the cloud computing service to perform the method(s) disclosed herein.

[0014] Yet another embodiment is directed to a non-transitory computer readable medium storing computer executable instructions that, when executed by one or more processors of a cloud computing service, cause the cloud computing service to perform the method(s) disclosed herein. BRIEF DESCRIPTION OF DRAWINGS

[0015] To easily identify discussions of any particular element or action, the left-most number or numbers in a drawing figure correspondence with the first digit or digits in the left-most column of the corresponding figure description.

[0016] Figure 1 is a block diagram of an environment in which a cloud infrastructure orchestration system (CIOS) according to at least one embodiment can operate to dynamically bootstrap services in a region.

[0017] Figure 2 is a block diagram for illustrating an environment and method for constructing a virtual bootstrapping environment (ViBE) according to at least one embodiment.

[0018] Figure 3is a block diagram for illustrating an environment and method for directing services to target areas with ViBE, in accordance with at least one embodiment.

[0019] Figure 4 is a block diagram depicting a data model representing various metadata associated with skills, in accordance with at least one embodiment.

[0020] Figure 5 is a block diagram depicting a data model representing various metadata associated with SPAM, in accordance with at least one embodiment.

[0021] Figure 6 is a block diagram depicting construction milestone entities representing example service plans and manifests, in accordance with at least one embodiment.

[0022] Figure 7 depicts an example execution cell entity of an example service plan, in accordance with at least one embodiment.

[0023] Figure 8 depicts an example group configuration entity of an example service plan, in accordance with at least one embodiment.

[0024] Figure 9 is a block diagram depicting relationships between various parts of a service plan and manifest, in accordance with at least one embodiment.

[0025] Figure 10 depicts an example user interface for viewing aspects of service related information associated with area construction, in accordance with at least one embodiment.

[0026] Figure 11 is a schematic diagram depicting an example user 1100 presenting a graph corresponding to one or more services of area construction, in accordance with at least one embodiment.

[0027] Figure 12 is a flow diagram depicting an example method for operations for driving data center construction with service plans and manifests, in accordance with at least one embodiment.

[0028] Figure 13 is a block diagram illustrating one mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0029] Figure 14 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0030] Figure 15 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0031] Figure 16is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0032] Figure 17 is a block diagram illustrating an example computer system, in accordance with at least one embodiment. DETAILED DESCRIPTION

[0033] In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to limit the embodiments to the exact details shown. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0034] Example automated data center build (region build) infrastructure

[0035] Recently, there has been a rapid increase in the adoption of cloud services. Now, various cloud service providers (CSPs) provide various types of cloud services. The term “cloud service” is generally used to refer to a service or functionality provided by a CSP to a user or customer on-demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the infrastructure of the CSP and are used to provide cloud services to customers are separate from the customer’s own on-premise servers and systems. Thus, customers can leverage cloud services provided by a CSP without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide simple, scalable, and on-demand access to applications and computing resources for subscribing customers without the customers investing in purchasing infrastructure for providing the services or functionality. Various different types or models of cloud services can be provisioned, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), etc. A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, an organization, a business, etc.

[0036] As indicated above, a CSP is responsible for providing the infrastructure and resources for providing cloud services to subscribing customers. The resources provided by a CSP can include both hardware and software resources. These resources can include, for example, compute resources (e.g., virtual machines, containers, applications, processors), memory resources (e.g., databases, data stores), networking resources (e.g., routers, host machines, load balancers), identity and other resources. In certain implementations, the resources provided by a CSP for providing a collection of cloud services CSP are organized into data centers. A data center can be configured to provide a particular collection of cloud services. A CSP is responsible for equipping a data center with the infrastructure and resources for providing a particular collection of cloud services. A CSP can build one or more data centers.

[0037] A data center provided by a CSP can be hosted in different regions. A region is a local geographic area and can be identified by a region name. Regions are typically independent of each other and can be located at a significant distance apart, such as across a country or even a continent. Regions are grouped into realms. Examples of regions for a CSP can include US West, US East, Australia East, Australia Southeast, and so on.

[0038] A region can include one or more data centers, where a data center is located within a certain geographic area corresponding to the region. As an example, a data center in a region can be located in a city within the region. For instance, for a particular CSP, a data center for the US West region can be located in San Jose, California; a data center for the US East region can be located in Ashburn, Virginia; a data center for the Australia East region can be located in Sydney, Australia; a data center for the Australia Southeast region can be located in Melbourne, Australia; and so on.

[0039] Data centers within a region can be organized into one or more availability domains for high availability and disaster recovery purposes. An availability domain can include one or more data centers within a region. Availability domains within a region are isolated, fault-tolerant, and architected such that data centers in multiple availability domains are unlikely to fail at the same time. For example, availability domains within a region can be architected such that a failure at one availability domain within the region is unlikely to affect the availability of data centers in other availability domains within the same region.

[0040] When a customer or subscriber subscribes or registers to one or more services provided by a CSP, the CSP creates a tenancy for the customer. A tenancy is like an account created for the customer. In certain implementations, a customer’s tenancy exists in a single realm and can access all regions belonging to that realm. Users of the customer can then access the services subscribed to by the customer under this tenancy.

[0041] As indicated above, CSPs build or deploy data centers to provide cloud services to their customers. As the customer base of a CSP grows, the CSP typically builds new data centers in new regions or increases the capacity of already existing data centers to meet the growing needs of customers and better serve customers. Preferably, data centers are built in geographic locations that are very close to the locations of the customers that the data center serves. The geographic proximity between a data center and the customers served by the data center helps to more efficiently use resources and provide faster and more reliable services to customers. As such, CSPs typically build new data centers in new regions in geographic areas that are geographically close to the customers served by the data center. For example, for a growing customer base in Germany, a CSP can build one or more data centers in new regions in Germany.

[0042] Building a data center (or multiple data centers) in a region is also sometimes referred to as building a region. The term "region build" is used to refer to building one or more data centers in a region. Building a data center in a region involves provisioning or creating a set of new resources that are needed or used to provide the set of services that the data center is configured to provide. The end result of the region build process is to create a data center in a region where the data center is able to provide the set of services intended for the data center and includes the set of resources used to provide the set of services.

[0043] Building a new data center in a region is a very complex activity that requires coordination between various service teams. At a high level, this involves performing and coordinating various tasks such as: identifying the set of services to be provided by the data center; identifying the various resources needed to provide the set of services; creating, provisioning, and deploying the identified resources; properly connecting the resources so that they can be used in the intended manner; and so on. Each of these tasks further has sub-tasks that need to be coordinated, which further adds to the complexity. Because of this complexity, currently, building a data center in a region involves several manually initiated or manually controlled tasks that require careful manual coordination. As a result, the task of building a new region (i.e., building one or more data centers in a region) is very time consuming. Building a data center can take time, e.g., months. Moreover, the process is very error prone and sometimes takes several iterations to achieve the desired configuration of the data center, which further adds to the time needed to build the data center. These limitations and issues severely limit the ability of CSPs to grow in a timely manner to respond to the growing needs of customers.

[0044] The orchestration operations have been coordinated and orchestrated by an orchestrator (e.g., a multi- fleet orchestrator, an orchestration service, etc.). In previous implementations, the orchestrator attempts to automatically detect dependencies between operations. The orchestrator maintains various versions of configuration files and / or software artifacts and attempts to intelligently and automatically identify the artifacts and the manner in which the data center build is performed. When the data center is built, the orchestrator utilizes capabilities (e.g., tags that can be turned on or off to indicate the availability of a resource or functionality) to drive the operations. However, both the automatic detection techniques and the use of capabilities have drawbacks.

[0045] Previous implementations of orchestrators lack a precise plan of the work that can be needed (or required) to build a data center prior to the actual build. The orchestrator utilizes service build definitions that are scattered in multiple fleet configuration files and interpreted by the orchestrator at runtime. This causes the orchestrator to perform a non-determined number of releases in a non-determined order, each release publishing a non-determined number of capabilities. To compensate for this behavior of uncertainty, manually curated micro-timelines are generated and used to track the order of operations and work necessary to build the data center. These micro-timelines are neither machine executable nor derived from code. It is not prevented for a service team to change its build automation, which can cause existing micro-timelines to become invalid. Furthermore, when a service's configuration file depends on external data, it is not possible to determine the exact behavior of that service build.

[0046] In previous implementations, tasks are triggered by publishing capabilities. Capability availability does not remain constant throughout a release, and if any optional capabilities are published in the middle of a release, this causes uncertainty in the planned activities. The use of optional capabilities makes it difficult to determine when an expected release publishes a certain capability or if the resource will ever be created. A service team can also introduce changes that create circular dependencies between services that cannot be satisfied, causing build deadlocks or dependencies on capabilities that will never be published. For at least these reasons, it is not possible to determine when a dependent release will be unblocked.

[0047] Heterogeneity in different regions also means that there is no single plan on how a service should be orchestrated. Rather, having a different plan for each region further exacerbates the difficulty of understanding how to build a service because capabilities can depend on or be published in certain types of regions and not others.

[0048] Technical Effects

[0049] Embodiments of the present disclosure relate to the utilization of service plans and manifests (SPAMs) that are individually used as specifications for the bootstrapping process of individual services. Service plans and manifests (SPAMs) provide a complete service build description that specifies the releases that can be necessary (or required) to build a service and the deterministic / explicit order of those releases. A SPAM can include explicit expectations of progress expected for each transition (e.g., each release performs corresponding to a particular phase / execution goal). One or more services (e.g., all services to be bootstrapped within a region) can be associated with a corresponding SPAM. The information provided by these SPAMs can be used to eliminate various errors that can occur in a data center build process by identifying problems early in the build lifecycle (e.g., upon submission of the SPAM) rather than at build time. SPAMs can be combined together by an orchestrator (e.g., a multi-group orchestrator, an orchestration service, etc.) and used to form a directed acyclic graph (DAG) of work (e.g., releases) that identifies the expected order of release execution that can be necessary (or required) to build a data center and the capability dependencies between those releases. The defined graph can be pre-validated to detect anomalies such as cycles at creation time and upon subsequent region updates. The graph can be used to support improved error detection prior to and during a build. The graph generated from the SPAMs can be used to drive the region build operations, or it can be used to validate a different graph (e.g., generated from group configurations as in previous implementations) used to drive the region build. The SPAMs provide a deterministic specification of the build implementation for a given service, which reduces (if not eliminates) the non-deterministic deficiencies present in previous implementations that utilized multiple group configurations to identify the releases that can be necessary (or required) to build a service. This improves the observability and understanding of a region build and reduces the time and complexity of identifying root causes when errors occur during a region build.

[0050] Certain Definitions

[0051] A “region” is a logical abstraction that corresponds to a geographic location. A region can include any suitable number of one or more execution targets.

[0052] A “phase” refers to a group of execution targets that can be executed concurrently.

[0053] An “execution target” refers to a unit of change that executes a release. An execution target can be specific to a region and a tenancy. Execution targets can be aggregated into one or more phases. For certain services, an execution target represents an “instance” of a service. A single service can be bootstrapped into each of one or more execution targets. An execution target can be associated with a set of devices (e.g., a data center).

[0054] “Release” refers to a representation of an intent to orchestrate a particular change to a service (e.g., deploy version 8, “add internal DNS record,” etc.). In some embodiments, a release corresponds to an instance of infrastructure provisioning or application deployment. A release can target one or more stages or execution targets.

[0055] “Orchestration” is intended to refer to the collective task associated with the provisioning and deployment of any suitable number of resources (e.g., infrastructure components, artifacts, etc.) corresponding to a single service.

[0056] “Service” refers to a function provided by a collection of resources. The collection of resources for a service includes any suitable combination of infrastructure, platform, or software (e.g., applications) hosted by a cloud provider that can be configured to provide the function of the service. A service can be provided to users over the Internet.

[0057] “Artifact” refers to code deployed to an infrastructure component (e.g., a physical or virtual host) or a Kubernetes engine cluster, which can include, but is not limited to, software (e.g., applications), configuration information for infrastructure components (e.g., configuration files), etc.

[0058] “Group configuration file” or simply “group configuration” refers to a configuration file that describes a collection of resources (e.g., infrastructure components and artifacts, also referred to as a “group”) associated with a single service. A group configuration can correspond to a single release (e.g., provisioning and / or deployment tasks to be executed as a unit). A group configuration can correspond to an infrastructure release or an application release. Any suitable number of releases and corresponding group configurations can be used to construct a service. A group configuration can include declarative statements specifying one or more aspects corresponding to a desired state of the resources for the service of the release.

[0059] “Group” refers to a collection of resources managed by CIOS or a collection of execution targets that can be deployed as a unit. Groups can exist within organizational units referred to as “Projects.”

[0060] “Service state” refers to a point-in-time snapshot of each resource (e.g., infrastructure resources, artifacts, etc.) associated with a service. The service state indicates a status corresponding to the provisioning and / or deployment tasks associated with the service resources.

[0061] IaaS provisioning (or "provisioning") refers to the acquisition of a computer or virtual host for use, even to the extent of installing required libraries or services on it. The phrase "provisioning a device" refers to evolving a device to a state where an end user can use it for their particular use. A device that has gone through the provisioning process can be referred to as a "provisioned device." Preparing a provisioned device (installing libraries and daemons) can be part of provisioning; this preparation is distinct from deploying a new application or new version of an application onto a prepared device. In most cases, deployment does not include provisioning, and provisioning can need to be performed first. Once prepared, a device can be referred to as an "infrastructure component."

[0062] IaaS deployment (or "deployment") refers to the process of providing and / or installing a new application or new version of an application on a provisioned infrastructure component. Once an infrastructure component has been provisioned (e.g., acquired, assigned, prepared, etc.), additional software can be deployed (e.g., provided to and installed onto the infrastructure component). After provisioning and deployment are complete, an infrastructure component can be referred to as a "resource." Examples of resources can include, but are not limited to, virtual machines, databases, object storage, block storage, load balancers, etc.

[0063] "Virtual bootstrap environment" (ViBE) refers to a virtual cloud network provisioned in an overlay of an existing region (e.g., a "host region"). Once provisioned, a ViBE connects to a new region using a communication channel (e.g., an IPsec tunnel VPN). Certain basic core services (or "seed" services) such as a deployment orchestrator, public key infrastructure (PKI) services, etc. can be provisioned in a ViBE. These services can provide the capabilities needed to onboard hardware, establish a chain of trust to the new region, and deploy the rest of the services in the new region. Utilizing a virtual bootstrap environment can prevent circular dependencies between bootstrap resources by leveraging resources of the host region. Services can be staged and tested in a ViBE before a physical region (e.g., a target region) is available.

[0064] "Cloud infrastructure orchestration service" (CIOS) can refer to a system configured to manage provisioning and deployment operations of any suitable number of services as part of a region build.

[0065] "Host region" refers to a region that hosts a virtual bootstrap environment (ViBE). A host region can be used to bootstrap a ViBE.

[0066] "Target region" refers to a region that is under construction.

[0067] A "capability" identifies a resource used during the construction of a region that signals that another resource, service, or feature is available, or that a certain event has occurred. For example, a capability can be published indicating that a resource is available for authorization / authentication processing (e.g., a subset of functionality to be provided by the resource). As another example, a capability can be published indicating that full functionality of a service is available. Capabilities can be used to identify functionality that a resource or service relies on and / or functionality of a resource or service that is available for use. Capabilities can be associated with an alphanumeric identifier and can be used to indicate that the capability is available or unavailable.

[0068] "Publishing a capability" refers to "publishing" as used in "publisher-subscriber" computing designs, or otherwise providing an indication that a particular capability is available (or not available). Capabilities are "published" (e.g., collected by, provided to, pushed to, pulled by, etc. a capability service) to provide an indication that functionality of a resource / service is available or that an event has occurred. In some embodiments, capabilities can be published / transmitted via events, notifications, data transmissions, function calls, API calls, etc. Events (or other notifications / data transmissions / etc.) indicating the availability of a particular capability can be broadcasted / addressed (e.g., published) to a capability service.

[0069] A "capability service" can be a service configured to monitor and maintain capability data indicating capabilities currently available in a region. A capability service can be provided within a cloud infrastructure orchestration system and can be used to identify which capabilities, services, features are available in a region, or which events have occurred in a region. The described capability service can serve as a central repository / authority for all capabilities published in a region (e.g., during region construction).

[0070] An "orchestrator" is intended to refer to a service or system that initiates tasks involved in bootstrapping one or more services during region construction. A multi-group orchestrator (MFO) (one example of an orchestrator) can be a computing component (e.g., service) configured to coordinate events between components of a CIOS in order to provision and deploy services to a target region (e.g., a new region). An orchestrator can track relevant events for each service in a region construction (e.g., through capability and / or skill indications as described herein) and take action in response to these events (e.g., based on determining that upstream dependencies have been satisfied for a given release / skill, etc.).

[0071] A "real-time region data distributor" (RRDD) can be a service or system configured to manage region data. This region data can be injected into a group configuration to dynamically create an execution target for a new region.

[0072] A“telemetry service” can be a service or system configured to manage / monitor time series data associated with one or more services / resources and trigger (e.g., publish, store, etc.) various alerts and / or corresponding alert states based at least in part on analyzing the time series data.

[0073] A“skill service” (also referred to as“Puffin”) can be a service or system configured to store planned and / or actual dependencies between units of service, resource, or functionality (also referred to as“service functions”). It should be appreciated that a unit of functionality can relate to functionality provided by a computing component other than a service.

[0074] A“skill” can represent a unit of functionality exposed and provided by a service to consumers (e.g., other services). The unit of functionality (also referred to as a“service function”) can include all or a subset of all functionality associated with the service. In some embodiments, a skill can be scoped, where access is controlled based on access and / or authorization policies and / or based on an association with a particular namespace. A skill can be provided multiple versions, where one or more aspects of the skill differ from other versions, where each skill version represents a particular implementation of the skill. Each skill version can be identified using a unique skill identifier. Skills are intended to replace (some or all) capabilities and enable enhanced and more accurate progress tracking of regional builds and improved root cause analysis functionality in the event of errors or unexpected events in a build. In some embodiments, a skill can correspond to one or more previously defined capabilities to provide backward compatibility with previously capability-based regional build implementations. A skill’s health can be monitored, and a skill can be configured to maintain health data. A“skill” can refer collectively to any suitable number of data structures in which data defining a skill can be maintained (e.g., a skill metadata 404 of FIG. 4). Figure 4

[0075] A“fleet” refers to a logical environment (e.g., pre-production, production, etc.) in which a skill can be scoped. For example, a skill associated with a production fleet can be separate from a skill of the same name used by a pre-production fleet. A“project” can similarly be used to scope a skill. In some embodiments, a skill can be scoped / applied to a particular environment based at least in part on any suitable combination of attributes that collectively identify a particular application of the skill, such as skillID, skillversionID, compartmentID, namespaceID, producerServiceID, skillName, skillName, fleet, project, etc.

[0076] ​A "service plan specification" or simply "service plan" refers to a specification of the build implementation for a service. A service plan can include any suitable combination of build milestones, execution units, and group configurations. A service plan details the particular releases that a build service can need (or needs) and the order in which the releases are to be executed for the build service. A service plan can separate inter-service coordination from intra-service coordination. A service plan can specify the expected state of a service at any suitable point of a regional build.

[0077] A "service manifest" or simply "manifest" identifies the group configurations and artifacts that a build service is to use. A service manifest can include a collection of service manifest items, each identifying a particular group configuration or artifact that a build service can need (or needs). In some embodiments, a service manifest item can be associated with a git commit hash for a group and all version declarations for any artifacts needed in an application release for the build of that service.

[0078] A "SPAM" (also referred to as a "service provisioning description") refers to the combination of a service plan and a manifest that collectively provide a deterministic specification of the processing for a build service. In some embodiments, a SPAM details the combination and order of releases that a build service can need (or needs). The manifest of a SPAM can define all resources to be used for the releases, while the service plan specifies the order of release execution based on capability dependencies. A SPAM can be used to track compliance of a regional build. A SPAM details the releases that a build service can be required (or needs) to have, where each release can be associated with pre- and post-conditions. A pre-condition can refer to a capability that can (or in some cases, must) exist such that a release can be created that will cause the post-condition to be met. A post-condition can be a capability that should (or in some cases, must) be published after a release is successful. A SPAM can be created by a service team and exported from YAML files that they write. A SPAM can be divided into multiple discrete sections, including execution units that define transitions between explicitly defined points in a build of a service (referred to as "build milestones"). A service can transition from one build milestone to the next by executing the releases defined by the execution units. An execution unit can specify external dependencies (capabilities) that releases defined within the execution unit can need (or needs). A build milestone can specify capabilities that are published by a service that should (or in some cases, must) be available after the service reaches the milestone. In some embodiments, the capabilities specified by a build milestone include capabilities that are intended for consumption by other services.

[0079] A "SPAM set" refers to a collection of SPAMs that are compatible with each other and / or have been previously associated with each other. SPAM sets can be used to derive version sets that can be used to generate a directed acyclic graph with which to drive the operations of building a data center. In some embodiments, SPAM sets can be associated with a range and / or zone context.

[0080] A "build milestone" refers to an entity defined in a service plan that identifies a synchronization point between the build of a service (e.g., the processes used to build a particular service) and the rest of the data center build. Build milestones can be defined relatively coarsely to limit their number and provide a high-level overview of the processes used to build a service. As a non-limiting example, a set of build milestones for a service can include "missing" (e.g., a default starting milestone), "service feature X available," "service available," and "service build complete."

[0081] An "execution unit" refers to another entity of a service plan. One or more execution units can describe the processes used to transition from one build milestone to the next via a directed acyclic graph of CIOS releases (e.g., infrastructure and / or application releases).

[0082] An "execution target checkpoint" or simply "ET checkpoint" refers to a point in a data center build defined for a given execution target. ET checkpoints can be associated with certain pre-conditions (e.g., required capability dependencies) and post-conditions (capability releases) that should be satisfied at the time the ET checkpoint is reached. In some embodiments, steps identified within an execution unit can reference ET checkpoint transitions, which can be logically mapped to expected CIOS releases (e.g., infrastructure or application releases).

[0083] A "zone prototype" can represent the overall structure of a zone (e.g., an ONSR zone, a single-availability-domain zone, a first zone in a domain) that can be used to influence the installation of a service. In some embodiments, a service plan can reference dimensions of a zone prototype to conditionally alter service plan definitions.

[0084] A "version set" can be used to define all group and artifact versions for all services in a particular zone context (e.g., a given particular zone, such as "regionl," and a particular version set identifier, such as "golden" or "break glass"). A version set can be composed of a number of version set items, each of which can specify a group and artifacts for that group. These entities can identify the existence of SPAMs and SPAM sets. As an example, in some embodiments, a version set can be associated with a corresponding SPAM set. Any suitable version set item can be associated with the SPAM from which it is derived.

[0085] “static group analysis” refers to performing static analysis of code (e.g., using a declarative configuration language to identify data center infrastructure components as objects) to infer capability releases and / or dependencies. In some embodiments, static group analysis can be performed with an infrastructure-as-code software tool (e.g., Terraform®). In some embodiments, this software tool can generate one or more data structures (e.g., directed acyclic graphs) representing these dependencies / releases. Each node in the graph can correspond to a group configuration and / or release, with edges identifying dependencies between capability releases and / or releases.

[0086] In some examples, techniques are described herein for implementing a cloud infrastructure orchestration service (CIOS). As briefly described above, these techniques can be configured to manage the bootstrapping (e.g., provisioning and deploying software) of infrastructure components within a cloud environment (e.g., a region). In some cases, a CIOS can include computing components (e.g., a CIOS hub and CIOS regions) that can be configured to manage the bootstrapping tasks (provisioning and deployment) of a given service, as well as orchestrators (e.g., multi-group orchestrators) configured to initiate / manage region builds (e.g., bootstrapping operations corresponding to multiple services in a region / data center).

[0087] A CIOS enables region / data center builds and global infrastructure provisioning and code deployment with minimal manual runtime effort by service teams (e.g., in some cases, beyond initial approval and / or physical transportation of hardware). High-level responsibilities of a CIOS include, but are not limited to, coordinating region builds in an automated manner with minimal human intervention, providing users with a view of the current state of resources managed by the CIOS (e.g., region, cross-region, global, etc.), and managing bootstrapping operations for bootstrapping resources within a region.

[0088] A CIOS can provide view reconciliation, in which a view of the desired state (e.g., desired configuration) of resources can be reconciled with the current / actual state (e.g., current configuration) of resources. In some cases, view reconciliation can include obtaining state data to identify which resources are actually running and their current configuration and / or state. Reconciliation can be performed at various granularities, such as at a service level.

[0089] The CIOS can perform plan generation, where differences between a desired state and a current state of resources are identified. Part of the plan generation can include identifying operations that need to be performed to cause the resources to change from the current state to the desired state. Once the user is satisfied with the plan, the plan can be marked as approved or rejected. Thus, the user can spend less time thinking about the plan, and because the plan is machine generated, the plan is more accurate. The plan is almost too detailed for human consumption; however, the CIOS can provide this data via a sophisticated user interface (UI).

[0090] In some examples, the CIOS can handle change management execution by automatically performing approved plans. Once an execution plan is created and approved, engineers can no longer need to be involved in change management, unless the CIOS initiates a rollback. The CIOS can handle rollback to a previous service version (e.g., when the CIOS detects a decline in service health while executing) by automatically generating a plan that returns the service to a previous (e.g., pre-release) state.

[0091] The CIOS can measure service health by monitoring alerts and performing integration testing. The CIOS can help teams quickly define rollback behavior in the event of a service decline, which it can automatically execute later. The CIOS can automatically generate and display plans and can track approvals. The CIOS can combine provisioning and deployment functionality in a single system that coordinates these tasks in a regional build. The CIOS can discover dependencies between execution tasks at each level (e.g., resource level, execution target level, phase level, service level, etc.) by performing static analysis (e.g., including parsing and processing content) on one or more configuration files. Using these dependencies, the CIOS can generate various data structures from these dependencies that can be used to drive task execution (e.g., tasks related to provisioning of infrastructure resources and deployment of cross-regional artifacts).

[0092] Today, during large-scale incidents (LSEs) – such as major errors, blockages, or delays during region builds – incident management and region build operators often incur significant overhead and sometimes delays, for example, in gathering information, attributing problems, assessing impacts, and restoring services. This is due to the fact that conventional methods are highly human-dependent and unsystematic. Identifying the contributing factors to an incident can be extremely difficult and time-consuming for operators due to the complexity of various dependencies between services. This leads to delays in remediation and a decline in the ability to assess when an incident will end. Similarly, building regions presents challenges because human involvement may be required to troubleshoot and / or detect faults or prevent situations. Typically, service teams struggle to determine which dependencies exist within their services. Services may have dependencies on other services, and vice versa. Furthermore, prior to actual region builds, service teams lack comprehensive metrics regarding whether their region build design will present serious problems that hinder or delay their service builds, such as circular dependencies.

[0093] Figure 1 This is a block diagram of an environment 100 according to at least one embodiment, in which a Cloud Infrastructure Orchestration System (CIOS) 102 may operate to dynamically orchestrate services in a region / data center. CIOS 102 may include, but is not limited to, the following components: a Real-time Regional Data Distributor (RRDD) 104, an orchestrator 106, a CIOS center 108, a CIOS region 110, a capability service 112, a virtual boot environment 116, a Puffin center 118, a Puffin region 120, and one or more alert services 122. Specific functionality provided by CIOS center 108 and CIOS region 110 is described in more detail in U.S. Application No. 17 / 016,754 entitled “Techniques for Deploying Infrastructure Resources with a Declarative Provisioning Tool,” the entire contents of which are incorporated herein by reference for all purposes. In some embodiments, any suitable combination of components of CIOS 102 may be provided as a service. In some embodiments, a portion of CIOS 102 may be deployed to a region (e.g., a data center represented by host region 103). In some embodiments, CIOS 102 may include any suitable number of cloud services ( Figure 1 (Not shown in the image), these cloud services are described below. Figure 2 and Figure 3 A more detailed discussion was held.

[0094] The real-time regional data distributor (RRDD) 104 can be configured to maintain and provide regional data identifying realms, regions, execution targets, and availability domains. In some cases, the regional data can be in any suitable form (e.g., JSON format, data objects / containers, XML, etc.). The regional data maintained by the RRDD 104 can include any suitable number of subsets of data that can be individually referenced by a corresponding identifier. By way of example, an identifier “all_regions” can be associated with a data structure (e.g., a list, a structure, an object, etc.) that includes metadata for all defined regions. As another example, an identifier such as “realms” can be associated with a data structure that identifies a number of realms and a set of regions corresponding to each realm. In general, the regional data can maintain any suitable attributes of one or more realms, region(s), availability domains (ADs), execution target(s) (ETs), etc., such as identifiers, DNS suffixes, statuses (e.g., statuses of regions), etc. The RRDD 104 can be configured to manage region statuses as part of the regional data. The region statuses can include any suitable information indicating a status of bootstrapping within a region. By way of example, some example region statuses can include “initial,” “build,” “production,” “paused,” or “deprecated.” The “initial” status can indicate that a region has not yet been bootstrapped. The “build” status can indicate that bootstrapping of one or more groups within a region has begun. The “production” status can indicate that bootstrapping has completed and the region is ready for validation. The “paused” status can indicate that the CIOS center 108 or CIOS region 110 has paused internal interaction with the region stack, which can be due to operational issues. The “deprecated” status can indicate that the region has been deprecated and can be unavailable and / or not contacted again.

[0095] The CIOS hub 108 is configured to provide any suitable number of user interfaces through which users (e.g., users 109) can interact with the CIOS 102. For example, users can make changes to region data via user interfaces provided by the CIOS hub 108. The CIOS hub 108 can also provide various interfaces that enable users to: view changes made to group configurations and / or artifacts, generate and view plans, approve / reject plans, view status of plan execution (e.g., corresponding to tasks involving infrastructure provisioning, deployment, region build, and / or any suitable number of resources managed by the CIOS 102). The CIOS hub 108 can implement a control plane configured to manage any suitable number of CIOS region 110 instances. The CIOS hub 108 can provide one or more user interfaces to present region data, enabling users 109 to view and / or change region data. The CIOS hub 108 can be configured to invoke functions of the RRDD 104 via any suitable number of interfaces. Generally, the CIOS hub 108 (also referred to as a “provisioning and deployment manager”) can be configured to manage region data directly or indirectly (e.g., via the RRDD 104). The CIOS hub 108 can be configured to compile group configurations (and / or SPAMs) to inject region data as variables into the group configurations (and / or SPAMs). The CIOS hub 108 can be instructed (e.g., by the orchestrator 106) to perform one or more releases (e.g., infrastructure or application releases) corresponding to group configurations.

[0096] Each instance of CIOS region 110 can correspond to a module configured to perform bootstrapping tasks associated with a single service of a region (e.g., a data center such as host region 103). CIOS region 110 can receive desired state data from CIOS hub 108. In some embodiments, the desired state data can include a group configuration that (e.g., via a declarative statement) declares a desired state of resources associated with the service. CIOS hub 108 can maintain current state data that indicates any suitable aspect of a current state of the resources associated with the service. In some embodiments, CIOS region 110 can identify that a change to one or more resources can be needed (or is needed) by comparing the desired state data and the current state data. For example, CIOS region 110 can determine that one or more infrastructure components need to be provisioned, one or more artifacts need to be deployed, or any suitable change to the resources of the service can be needed (or is needed) to bring the state of the resources in line with the desired state. When CIOS region 110 performs a bootstrapping operation, it can publish data indicating various capabilities of a resource as they become available. A "capability" identifies a unit of functionality associated with the service. The unit can be a portion or all of the functionality that the service will provide. By way of example, a capability can be published indicating that a resource is available for authorization / authentication processing (e.g., a subset of the functionality that the resource will provide). As another example, a capability can be published indicating that the full functionality of the service is available. The capabilities can be used to identify the functionality that the resource or service relies on and / or the functionality of the resource or service that is available for use. In some embodiments, CIOS region 110 can transmit data indicating a state transition of a skill. By way of example, in some embodiments, CIOS region 110 performs a bootstrapping operation that results in a skill being published (e.g., skill metadata including a skill state value indicating that the skill has been installed is transmitted). The skill metadata can be transmitted to a Puffin (e.g., Puffin region 120) and used to update the skill state of the corresponding skill.

[0097] Capability service 112 is configured to maintain capability data indicating 1) which capabilities of various services are currently available, 2) whether any resources / services are waiting for a specific capability, 3) which specific resources and / or services are waiting for a given capability, or any suitable combination of the above. Capability service 112 can provide an interface that can request capability data. Capability service 112 can provide one or more interfaces (e.g., application programming interfaces) that enable it to transfer capability data to orchestrator 106, CIOS region 110 (e.g., each instance of CIOS region 110), Puffin region 120, and / or Puffin center 118. In some embodiments, capability service 112 can store capability data in a data repository accessible to one or more components of CIOS 102. Orchestrator 106, CIOS region 110 (e.g., each instance of CIOS region 110), Puffin region 120 and / or Puffin center 118, and / or any suitable component or module of CIOS region 110 may be configured to request capability data from capability service 112, or otherwise obtain capability data (e.g., from a data repository configured to store capability data generated by capability service 112). While capability service 112 is depicted as a separate component of CIOS 102, it should be recognized that in some embodiments, the functionality provided by capability service 112 may be provided, in whole or in part, as part of a skill service via any suitable combination of Puffin center 118 and Puffin region 120.

[0098] In some embodiments, each region component (such as CIOS region 110, capability service 112, Puffin region 120, and / or virtual boot environment 116) can be one of many region components. Each region component can be specific to a given region (e.g., such as...). Figure 1 (As illustrated in the diagram, host region 103). Therefore, another region may include similar but separate region-specific components. In some embodiments, central components (e.g., orchestrator 106, CIOS center 108, RRDD 104, and Puffin center 118) may include one or more components configured to manage build operations corresponding to one or more regions. For example only, a single orchestrator (orchestrator 106) may be used to manage bootstrapping operations for building any suitable number of data centers, or multiple instances of orchestrator 106 may be used, each instance driving bootstrapping operations for a subset or a single data center of these data centers.

[0099] In some embodiments, orchestrator 106 (examples of which can be multi- group orchestrator, orchestration service, etc.) can be configured to drive region build jobs. In some embodiments, orchestrator 106 can manage information describing which group configuration versions and / or artifact versions are to be used to bootstrap a given service (or unit change to a target region) within a region. In some embodiments, orchestrator 106 can manage any suitable combination of group configurations and / or service plans. In some embodiments, orchestrator 106 can be configured to monitor (or otherwise be notified of) changes to region data managed by live region data distributor 104. In some embodiments, receiving an indication that region data has changed can cause orchestrator 106 to trigger a region build. In some embodiments, orchestrator 106 can collect various group configurations, artifacts, and / or SPAMs to be used for a region build. Some or all of the group configurations and / or SPAMs can be configured to be region-agnostic. That is, the group configurations and / or SPAMs can not explicitly identify to which regions the groups are to be bootstrapped. In some embodiments, orchestrator 106 can trigger a data injection process by which (e.g., by CIOS hub 108) the collected group configurations and / or SPAMs are recompiled. During recompilation, operations can be performed (e.g., by CIOS hub 108) to cause region data maintained by live region data distributor 104 to be injected into the configuration files and / or SPAMs. The group configurations and / or SPAMs can reference region data by way of variables / parameters, without requiring hard-coded identification of the region data. Rather than hard-coding the region data and thus making changes more difficult, this data injection can be used to dynamically modify the group configurations and / or SPAMs at runtime.

[0100] In some embodiments, orchestrator 106 can perform static group analysis in which group configurations and / or service plans are parsed to identify resources, execution targets, execution target checkpoints, phases, and dependencies between groups, and in particular to identify circular dependencies that need to be removed. In some embodiments, static group analysis (SFA) data corresponding to such analysis can be stored (e.g., via DB 312) for later use. In some embodiments, orchestrator 106 can generate any suitable number of data structures based on the identified dependencies. These data structures (e.g., directed acyclic graph(s), linked lists, etc.) can be used by CIOS 102 to drive operations to perform region builds. By way of example, these data structures can collectively define an order in which services are to be bootstrapped within a region. Examples of this will be described below with respect to Figure 3The build dependency graph 338 further discusses examples of such data structures. If circular dependencies (e.g., service A requires service B, and vice versa) exist, and are identified through static group analysis and / or graphs, the orchestrator 106 can be configured to notify any appropriate service team that changes to the corresponding group configuration are needed to correct these circular dependencies. The orchestrator 106 can be configured to traverse one or more data structures to manage the order in which services are directed to regions. The orchestrator 106 can identify (e.g., using data obtained from the capability service 112) the capabilities available within a given region at any given time. The orchestrator 106 can utilize this data to identify when it can direct services, when it can prevent direction, and / or when it can resume direction operations associated with previously prevented services. Based on this traversal, the orchestrator 106 can perform various releases in which instructions are transmitted by the orchestrator 106 to the CIOS hub 108 to perform direction operations corresponding to any appropriate number of group configurations. In some examples, the orchestrator 106 can be configured to identify that one or more group configurations can require multiple releases due to circular dependencies discovered in the graph. Accordingly, the orchestrator 106 can transmit multiple sets of instructions to the CIOS hub 108 for a given group configuration to break the circular dependencies identified in the graph.

[0101] In some embodiments, the orchestrator 106 can utilize one or more service plans and manifests (SPAMs). Service plans and manifests can provide a more deterministic specification of the build description for a service than previously provided by one or more group configurations. While group configurations specify aspects of a single release associated with a single service, service plans can provide a single specification of the order and conditional requirements of all releases that can be needed (or required) to perform a build of a given service. Previous implementations of group configurations include optional dependencies, which allow for a degree of indeterminate behavior regarding the order of operations performed during a region build. Including optional dependencies can require the orchestrator 106 to perform multiple passes of a build dependency graph, resulting in wasted processing. These types of dependencies make it difficult, if not impossible, for the system to track region build progress, identify remaining operations that have not been performed, and / or identify when a build is complete. Service plans and manifests (SPAMs) can be used to eliminate at least some of the drawbacks of previous indeterminate approaches.

[0102] SPAMs (one SPAM corresponds to a service to be bootstrapped in a region) allow service teams to describe corresponding operations that can be needed (or required) to build their service, and can allow a distinction between internal coordination (e.g., of operations within a service) and external coordination (e.g., of operations between components of different services). Multiple visualizations can be provided via one or more user interfaces (e.g., via orchestrator 106 or any suitable component of CIOS 102). One visualization can depict a directed acyclic graph that describes build operations within a given service; while another visualization can depict a directed acyclic graph that describes an order of build operations corresponding to multiple services (e.g., all services of a region / data center). As a specific example, one or more visualizations can present a region-level directed acyclic graph (DAG) that includes only external coordination (e.g., an order of operations corresponding to coordination between services), while omitting operations internal to each service. For example, this DAG can depict nodes corresponding to capabilities (or skills) of a service that other services depend on, while excluding nodes corresponding to capability (or skill) dependencies between service components / functional units of the same service.

[0103] A SPAM can include an external interaction interface that includes a service build definition that includes multiple build milestones. Each build milestone can be associated with a set of capabilities (and / or skills) that the service is expected to release after reaching a given milestone. To transition between build milestones, a SPAM can include an execution unit that encapsulates a directed acyclic graph (DAG) of one or more releases, each release equivalent to an operation previously defined with a single group configuration. Each execution unit can define a set of build-time dependencies that identify one or more capabilities (and / or skills) needed to execute the at least one release of the execution unit.

[0104] A SPAM can include a service build implementation. Execution units of a SPAM can describe one or more releases that can be needed (or required) to build a service, and can define multiple execution units. Each execution unit can be associated with one or more execution target checkpoint transitions, each of which can be used to specify expected capabilities that should be available prior to a release, and capabilities that should be released as a result of executing a release.

[0105] In some embodiments, orchestrator 106 can be configured to aggregate SPAMs corresponding to each service to be deployed in a region to compose a larger directed acyclic graph (e.g., Figure 3The construction dependency graph 338) that can capture all the operations necessary for the build region / data center. The aggregation of SPAM identified from this can be referred to as a "SPAM set." In some embodiments, the orchestrator 106 can utilize the DAG generated from the SPAM set to validate the DAG and / or operations performed using the group configuration, while the DAG generated from the group configuration is used to drive the build operation / release execution. Alternatively, the orchestrator 106 can utilize the DAG generated from the SPAM set to drive the build operation / release execution. The system can utilize the SPAM / SPAM set to generate a deterministic execution plan with which the region build can be executed.

[0106] In some embodiments, the Puffin Center 118 can provide multiple user interfaces with which one or more skills can be defined. Skills can be used in conjunction with previous capabilities or can replace previous capabilities and enable improvements to previous capability-based implementations. In contrast to capabilities, skills can be scoped (e.g., controllable through access and authorization policies), versioned, and attributed to specific services and / or contacts. Skills can be associated with a lifecycle and can be monitored for health, and designed to have higher visibility / accessibility than capabilities. The Puffin Center 118 can provide an authoritative registry for skills. Various user interfaces managed by the Puffin Center 118 can be used to define, maintain, and manage skills provided by each service, as well as their dependencies with other services. The Puffin Center 118 can be used to declare and persist strongly-defined metadata for services in a versioned manner. This metadata can be used to generate blueprints for build-time and runtime dependencies. These blueprints can be used to validate build plans, drive orchestration decisions during region builds, and improve participation time and diagnostic time measures during region builds and / or large-scale events (LSEs).

[0107] The Puffin hub 118 can be configured to serve as a true source of services and can maintain metadata including upstream and downstream dependencies for each service and service team contact information and methods for each service across regions and domains (e.g., collections of regions). Each skill can represent a unit of functionality exposed and provided by a service to consumers (e.g., other services). In some embodiments, a skill can be scoped, where access is controlled based on access and / or authorization policies and / or based on association with a particular namespace. A skill can be associated with multiple versions, where one or more aspects of the skill differ from previous versions, where each skill version represents a particular implementation of the skill. Each skill version can be identified using a unique skill identifier. In some embodiments, the Puffin hub 118 can be configured to generate skills corresponding to previously defined capabilities in order to provide backward compatibility with previous capability-based region building implementations.

[0108] In some embodiments, Puffin can maintain compatibility between skills and capabilities such that any suitable combination of the two can be utilized to define processes by which services are to be built. Based on maintaining mappings between skills and / or capabilities published by a service, Puffin can ensure that skills can be transitioned based on capabilities and / or capabilities can be published as a result of state changes to corresponding skills. In some embodiments, Puffin can generate “shadow skills” (e.g., system-generated skills representing corresponding capabilities) and / or shadow capabilities (e.g., system-generated capabilities published when corresponding skills are transitioned to installed states). These features provided by Puffin enable orchestrators to drive orchestration during region building (e.g., during processes for building data centers) using any suitable combination of skills and / or capabilities.

[0109] In some embodiments, skills can be mapped to one or more capabilities. Puffin region 120 can be configured to publish and / or store skill metadata based on capability data published (or stored) by capability service 112. In some embodiments, Puffin region 120 can publish capability data to capability service 112 and / or store such data based at least in part on publishing a skill or identifying that a skill has transitioned to a particular state or is otherwise associated with a particular state. In some embodiments, some services can utilize group configurations that express progress using capabilities, while other services can utilize service plans and manifests that define deterministic build processes in which progress is expressed using capabilities and / or skills. Using the mapping (or mappings) between skills and capabilities, Puffin region 120 can enable region builds to be performed using any suitable combination of capabilities and / or skills to indicate: 1) a service or resource function is available, 2) a particular event has occurred, 3) a particular fact is true, 4) a condition has been met, or any suitable combination of the above. The mapping(s) enable CIOS 102 to perform region builds / data center builds using any suitable combination of capabilities and / or skills, thereby enabling service teams to transition from a capability-based implementation to a skill-based implementation.

[0110] In some embodiments, any suitable computing component of a Puffin service (e.g., Puffin hub 118 and / or Puffin region 120) can be configured to monitor the health and / or lifecycle of a skill according to a predefined skill lifecycle. Health monitoring can be performed using one or more alerts associated with a given skill. In some embodiments, a telemetry service (e.g., an example of alert service(s) 122) can utilize application programming interfaces provided by a Puffin service (including Puffin hub 118 and / or Puffin region 120) when an alert is triggered. As another example, a Puffin service (e.g., Puffin region 120) can request alert data from alert service(s) 122 and / or a storage location at which alert service(s) 122 store alert data. The Puffin service can present information related to the health of a skill based on alerts corresponding to the obtained alert data and their corresponding association with a given skill via one or more user interfaces.

[0111] In some embodiments, a Puffin service (e.g., Puffin hub 118 and / or Puffin region 120) can expose one or more application programming interfaces (APIs) with which verification operations can be performed. For example, a SPAM describing a build process for one or more services can be provided (e.g., by orchestrator 106) via a given API. The Puffin service (e.g., Puffin hub 118) can perform any suitable operations for verifying that all services and skills identified in the SPAM have been previously registered with the Puffin service, and that the build process defined in the SPAM does not violate previously defined dependencies maintained by the Puffin service. Additionally, or alternatively, the orchestrator 106 can perform any suitable verification checks, such as determining whether each group configuration and / or artifact identified in a manifest for a given service is referenced in the corresponding service plan for that service, and / or determining that the service plan does not reference a group configuration and / or artifact that is not referenced in the manifest. The orchestrator 106 can perform verification operations (e.g., including static analysis of the service plans) to determine that the service plans do not have circular dependencies. If a circular dependency is found in a service plan, the orchestrator 106 can provide a notification and / or restrict use of the service plan and corresponding manifest. In some embodiments, such restrictions can include restricting the addition of the service plan and manifest to a SPAM set (e.g., a set of SPAMs used to perform a regional build). In some embodiments, the orchestrator 106 can perform any suitable verification operations to ensure that SPAMs of a SPAM set and / or SPAMs being considered for addition to an existing SPAM set are mutually compatible. This can include analyzing the SPAM set (either alone or with the SPAM being considered for addition) to ensure that the SPAMs of the SPAM set do not include circular dependencies.

[0112] In some embodiments, a user can request the construction of a new region (e.g., target region 114). This may involve bootstrapping resources corresponding to various services. In some embodiments, target region 114 may be uncommunicative (and / or insecure) when the region construction request is initiated. Instead of delaying bootstrapping until target region 114 becomes available and is configured to perform boot operations, CIOS 102 can use a virtual boot environment (e.g., Virtual Boot Environment (ViBE) 116) to initiate the region construction. ViBE 116 may be an overlay network hosted by host region 103 (a pre-existing region that has a core set of services previously configured and is communicative and secure). Orchestrator 106 can utilize the resources of host region 103 to boots resources into ViBE 116 (often referred to as "building a ViBE"). For example, orchestrator 106 can provide instructions via CIOS hub 108 to boot another instance of CIOS region within ViBE 116 from an instance of CIOS region 110 within a host region (e.g., host region 103). Once the CIOS region within the ViBE is available for processing, services targeting the target region 114 can continue to be bootstrapped within the ViBE 116. When the target region 114 becomes available to perform boot operations, previously booted services within the ViBE 116 can be migrated to the target region 114. Utilizing these techniques, CIOS 102 can significantly increase the speed of region construction by drastically reducing the need for any manual input and / or configuration. In some embodiments, any suitable combination of components depicted as part of CIOS 102 can be used individually as... Figures 13-17 Examples of cloud services (e.g., Figure 13 (1356) and can be configured to be used in any suitable infrastructure mode (such as the combination below) Figures 13-17 The operation is described in the example.

[0113] Figure 2 This is used to illustrate a method for constructing a virtual boot environment (ViBE) according to at least one embodiment 202 ( Figure 1 A block diagram of the environment and method 200 (example of ViBE 116). ViBE 202 represents an existing area (e.g., host area 204). Figure 1 Example of host region 103), and in one embodiment, a virtual cloud network supplied within the coverage of a host region service enclave. ViBE 202 represents a target region (e.g., a region within a construction area, such as...). Figure 1 The environment that can temporarily serve the target area (114) before it becomes available.

[0114] In order to guide the new area (e.g., Figure 1target region 114), a core set of services can be directed. While these core sets of services exist in the host region 204, they do not yet exist in the ViBE (nor in the target region). These basic core services provide the functionality needed to provision the equipment, establish a chain of trust to the new region, and deploy the remaining services into the region. The ViBE 202 can be a tenancy deployed in the host region 204 and used as a virtual region.

[0115] When the target region is available to provide bootstrapping operations, the ViBE 202 can connect to the target region so that the services in the ViBE can interact with the services and / or infrastructure components of the target region. This will enable the deployment of production grade services rather than self-contained seed services as in previous systems and can be connected to the target region over the internet. Conventionally, seed services are deployed as part of a container swarm and are used to bootstrap the dependencies needed to build the region. Using the existing region’s infrastructure / tools, resources can be bootstrapped (e.g., provisioned and deployed) into the ViBE 202 and connected to the service estate of the region (e.g., host region 204) in order to provision (reserve and / or configure) hardware and deploy services until the target region is self-sufficient and can communicate directly. Leveraging the ViBE 202 allows dependencies to be satisfied and the needed services to be provided in order to be able to provision / prepare the infrastructure and deploy software while leveraging the resources of the host region to break the circular dependencies of the core services.

[0116] The orchestrator 206 (examples of the orchestrator 106 Figure 1 The orchestrator 206 can be configured to perform operations to build (e.g., configure) the ViBE 202. The orchestrator 206 can obtain applicable group configurations and / or SPAMs corresponding to various resources to be bootstrapped to a new region (in this case, the ViBE region, ViBE 202). By way of example, the orchestrator 206 can obtain a group configuration (e.g., a “ViBE group configuration”) that identifies aspects of the bootstrap capability services 208 (e.g., examples of the capability services 112) and / or workers 210. In some embodiments, the orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap skill services (e.g., examples of the skill services 110) and / or workers 210. The orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap dependency services (e.g., examples of the dependency services 108) and / or workers 210. The orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap infrastructure services (e.g., examples of the infrastructure services 104) and / or workers 210. The orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap target region services (e.g., examples of the target region services 116) and / or workers 210. The orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap target region infrastructure services (e.g., examples of the target region infrastructure services 114) and / or workers 210. The orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap target region workers 212 and / or workers 210. The orchestrator 206 can additionally obtain a group configuration that identifies any suitable portion of the bootstrap target region 118 and / or workers 210. Figure 1Aspects of the Puffin region 120). In some embodiments, one or more service plans and manifests (SPAMs) can be used to identify these aspects (e.g., specifying operations previously defined in one or more group profiles and / or resources / artifacts that can be required (or needed) for end-to-end onboarding services) for onboarding any suitable combination of the capability services 208, workers 210, and / or Puffin region services 209. As another example, the orchestrator 206 can obtain another group configuration and / or SPAM corresponding to onboarding the domain name service (DNS) 212 to the ViBE 202.

[0117] The method 200 can begin at step 1, where the orchestrator 206 can instruct the CIOS hub 214 (e.g., the CIOS hub 108 and the CIOS hub 214 of FIGS. 1 Figure 1 and Figure 2 ). For example, the orchestrator 206 can transmit a request (e.g., including a ViBE group configuration, which can be a group configuration identified in a service plan) to request onboarding of the capability services 208 and workers 210 (and, in some embodiments, also the Puffin region 209) that do not yet exist in the ViBE 202. In some embodiments, a corresponding SPAM for the capability services 208, workers 210, and / or Puffin region 209 can be utilized instead of or in addition to the ViBE group configuration. In some embodiments, the CIOS hub 214 can have access to all group configurations and / or SPAMs. Thus, in some examples, the orchestrator 206 can transmit an identifier of the ViBE group configuration, and the CIOS hub 214 can independently obtain the ViBE group configuration from storage (e.g., from the database (DB) 308 or the DB 312 of FIG. 1). Figure 3

[0118] At step 2, the CIOS hub 214 can provide the ViBE group configuration to the CIOS region 216 via a corresponding request. At step 3, the CIOS region 216 can resolve the ViBE group configuration to identify and perform specific infrastructure provisioning and deployment operations.

[0119] In some embodiments, the CIOS region 216 can utilize additional corresponding services for provisioning and deployment. For example, at step 4, the CIOS region 216 CIOS region can instruct the deployment orchestrator 218 (e.g., a core service of the host region 204 or other example that writes, builds, and deploys application software) to execute instructions that, in turn, cause the capability services 208, workers 210, and, in some embodiments, also the Puffin region 209 to be onboarded within the ViBE 202.

[0120] ​At step 5, capability data can be transmitted to the capability service 208 (from the CIOS region 216, the deployment orchestrator 218 via the worker 210, or otherwise) indicating that resources corresponding to the ViBE group are available. The capability service 208 can persistently save this data. In some embodiments, the capability service 208 adds this information to a list of ViBE available capabilities that it maintains. For example, the capabilities provided to the capability service 208 at step 5 can indicate that the capability service 208 and the worker 210 (and in some embodiments, also the Puffin region 209) are available for processing. In some embodiments, skill metadata can be transmitted to the Puffin region 209 indicating that any suitable combination of functionality corresponding to the capability service 208, the worker 210, and / or the Puffin region 209 is available.

[0121] At step 6, the orchestrator 206 can identify that the capability service 208, the worker 210, and / or the Puffin region 209 are available based on receiving or obtaining data (identifiers corresponding to capabilities and / or skills) from the capability service 208 and / or the Puffin region 209.

[0122] In some embodiments, the published capabilities can be processed by the Puffin region 209 (e.g., the Puffin region 120 of FIG. 1) prior to being processed by the orchestrator 206. In some embodiments, the Puffin region 209 can be configured to provide forward and backward compatibility between skills and capabilities. For example, in some embodiments, if a capability is published to the Puffin region 209, the Puffin region 209 can query known skills (e.g., via a skill table or other suitable record of registered / previously generated skills) to check whether any skills are associated with the capability. If there are no skills associated with the capability, the Puffin region 209 can be configured to create a skill (referred to as a “shadow skill”) to represent the capability using a skill construct (e.g., including the data structures discussed below in connection with FIG. 3). When the orchestrator 206 publishes a skill (or updates a skill state) during processing of a region build, the Puffin region 209 can receive this data and identify one or more capabilities associated with the corresponding skill(s). The Puffin region 209 can publish any or all capabilities associated with the skill that have not yet been published. In some embodiments, publishing such data can include storing an indication that these capabilities are available. In this way, the Puffin region 209 can support full compatibility between capabilities and skills such that any suitable combination of the two can be leveraged to drive operations performed during a region build. Figure 1 Figure 4

[0123] ​​While some embodiments describe shadow skill generation constructed at build time, it should be recognized that the Puffin service can generate shadow skills at any appropriate time and according to various methods. For example, shadow skills can be generated by the Puffin service (e.g., Figure 1 Puffin center 118 and / or Puffin area 120, and / or Figure 2 Historical capability data (e.g., capability data historically published during one or more previous region builds) can be obtained at any appropriate time (e.g., before a region build is initiated, before deployment within a region, after a region build is completed, etc.) in Puffin regions such as 209. In some embodiments, historical capability data may be (e.g., by...) Figure 1 The capability service instance (112) is stored in a data repository accessible to the Puffin service. The Puffin service can process historical capability data (e.g., one or more files, records, tables, data structures, etc.) to identify one or more capabilities for which a corresponding skill does not currently exist. Identifying a corresponding skill may include matching any suitable portion of a capability's token or label with any suitable attributes and / or portions of attributes associated with the service (e.g., one or more tokens / words of the service name and / or identifier). The Puffin service can generate a shadow skill for each historically published capability that does not match any known skill. As described above, these shadow skills can be configured to represent the corresponding historically published capability and can be used to maintain compatibility between skills and capabilities, as well as compatibility between skill-based service building definitions (e.g., SPAM) and capability-based service building definitions (e.g., groups, SPAM, etc.).

[0124] At step 7, as a result of receiving / acquiring data at step 6, orchestrator 206 may instruct CIOS center 214 to route DNS services (e.g., DNS 212) to ViBE 202. This instruction may identify or include specific group configurations and / or SPAM corresponding to the DNS service.

[0125] At step 8, CIOS center 214 may instruct CIOS zone 216 to deploy DNS 212 to ViBE 202. In some embodiments, DNS group configuration and / or SPAM for DNS 212 may be provided by CIOS center 214.

[0126] In step 9, worker 210 (now deployed in ViBE 202) can be assigned by CIOS zone 216 to the task of deploying DNS 212. The worker can then proceed as described above. Figure 3The described approach executes a declarative infrastructure provider to identify the set of operations required to deploy DNS 212. These operations can be identified, at least in part, based on comparing the corresponding portion of the group configuration (desired state) or SPAM with the current state of the (currently non-existent) resources associated with DNS 212.

[0127] At step 10, deployment orchestrator 218 can instruct worker 210 to deploy DNS 212 according to the operation identified at step 9. As illustrated, worker 210 continues to perform operations to deploy DNS 212 to ViBE 202 at step 11. At step 12, worker 210 can notify capability service 208 (via capability) or Puffin zone 209 (directly or via capability service 208 and using skills) that DNS 212 is available in ViBE 202. Orchestrator 206 can then identify the resources available associated with the ViBE group configuration and DNS group configuration, and can continue to route any appropriate number of additional resources to ViBE.

[0128] After completing steps 1-12, the processing for building ViBE 202 can be considered complete, and ViBE 202 can be considered built and ready for additional boots (e.g., bootstrapping various cloud services, such as...). Figure 13 Cloud service 1356). At any appropriate time during steps 1-12, Puffin region 209 can receive alerts from one or more alert services (e.g., cloud services 1356). Figure 1 The alarm service (one or more) 122 receives and / or obtains alarm data. In some embodiments, the alarm data may be processed by the Puffin area 209 (or the Puffin area 209 may transmit the alarm data or data derived from the alarm data to...). Figure 1(Puffin Center 118). In some embodiments, Puffin Region 209 (and / or Puffin Center 118) may transmit skill health information to Orchestrator 206, indicating the corresponding health status associated with one or more skills. In some embodiments, Puffin Region 209, Puffin Center 118, and / or Orchestrator 206 may be configured to perform operations that may pause (partially or completely) any suitable portion of the operations discussed above in conjunction with Method 200. In some embodiments, this may cause the region state associated with the region where Method 200 is performed to be updated to indicate that the construction of that region is paused. In some embodiments, Puffin Region 209, Puffin Center 118, and / or Orchestrator 206 may be configured to resume the operation of Method 200 (and update the region state accordingly) at least in part based on user input, subsequent alert data indicating updates to the health status of one or more skills, skill health coverage values, etc.

[0129] Figure 3 This is a block diagram illustrating an environment and method 300 for directing services to a target area using ViBE according to at least one embodiment.

[0130] Method 300 may begin at step 1, where user 302 (e.g., a service team member) may communicate with Puffin center 340 (e.g., Figure 1 The Puffin Center 340 can interact with any suitable number of user interfaces managed by the Puffin Center 318. The Puffin Center 340 can be configured to read service and / or skill metadata from a predefined file, or the user 302 can enter service and / or skill metadata at one or more of the provided user interfaces. In some embodiments, the Puffin Center 340 can store all service and skill metadata and serve as its centralized authority. Any suitable user can view the service and / or skill metadata at any suitable time, such as before and / or during the execution of a region build.

[0131] In step 2, user 303 can utilize the CIOS center 304 (respectively...) Figure 1 CIOS Center 108 and Figure 2 (Example of CIOS Center 214) provides any suitable user interface to modify zone data. For example, user 303 can create a new zone to which several services will be routed.

[0132] At step 3, CIOS center 304 can perform an operation to send the changes to RRDD 306 (e.g., Figure 1(Example of RRDD 104). At step 4, RRDD 306 may store the received region data in database 308, which is a data repository configured to store region data, including any suitable identifiers, attributes, statuses, etc., such as region, AD, domain, ET, etc. In some embodiments, updater 307 may be used to store region data in database 308 or in any suitable data repository from which such updates can be accessed (e.g., by a service team). In some embodiments, updater 307 may be configured to notify updates made to database 308 (e.g., via any suitable electronic notification).

[0133] In step 5, the arranger 310 (respectively) Figure 1 The 106 and / or programmable typewriter Figure 2 (Example of orchestrator 206) can detect changes to region data. In some embodiments, orchestrator 310 can be configured to poll RRDD 306 to learn about changes to region data. In some embodiments, RRDD 306 can be configured to publish or otherwise notify orchestrator 310 of region data changes.

[0134] At step 6, detecting a change in region data can trigger orchestrator 310 to obtain a version set (e.g., a version set associated with a specific identifier, such as a "gold version set" identifier) ​​that identifies a specific version of each group configuration and a specific version of each artifact to be used to build the region. This version set can be obtained from DB 312. As group configurations and / or artifacts evolve and change over time, multiple versions of each artifact can be maintained, and certain versions of each artifact can be used for region building. The version set can be persisted in DB 312, allowing orchestrator 310 to identify which versions of group configurations and artifacts to use to build the region (e.g., ViBE region, target region / non-ViBE region, etc.). Group configurations (e.g., group configurations of all versions) and / or artifacts (e.g., artifacts of all versions) can be stored in DB 308, DB 312, or CIOS center 304 and / or any suitable data repository accessible to orchestrator 310.

[0135] In some embodiments, orchestrator 310 may identify any suitable number of SPAMs (collectively, a “SPAM set”) corresponding to the infrastructure to be provisioned and the artifacts to be deployed, as part of a region build. In some embodiments, each SPAM may identify a version corresponding to one or more group configurations and / or one or more artifacts that may be required (or needed) to build a single service. In embodiments where one or more SPAMs are used, the SPAM(s) (or any suitable portions thereof) may be stored within DB 312 and used to identify specific group configuration and / or artifact versions to be used for building a region. In some embodiments, the group configuration and / or artifact versions of the SPAM set may be included in a version set and stored within DB 312. This allows some service teams to leverage the set of group configurations to define the build implementation of their services, while other service teams may choose to leverage SPAMs to define the build implementation of their services.

[0136] In some embodiments, group version sets / version set items can be derived from any suitable number of SPAMs, and orchestrator 310 can be configured to verify the compliance of group behavior (e.g., build / orchestration operations identified in the group configuration) with the processing defined by the corresponding SPAM. Orchestrator 310 can be configured to ingest SPAMs that provide information that may be needed (or required in some cases) for the pre-build work plan and introduce better safeguards than those available in previous embodiments. Any suitable number of SPAMs can be aggregated into a corresponding SPAM set in a manner similar to how groups can be aggregated into version sets. SPAM sets can enforce invariances, i.e., all SPAMs within the set are mutually compatible and combined to form a feasible graph of releases required for the build region. In some embodiments, SPAM sets can be used within a given region context to improve service build progress tracking. SPAM operations can be validated before they are applied and rejected if invalid, unlike unconditionally applied version set item operations. The use of SPAM enables the orchestrator 310 to build deterministic work plans before building areas, prevent updates that may jeopardize or disrupt ongoing or future builds, improve the tracking of service build processes, detect deviations from SPAM specifications in group behavior, and alert operators to deviations and conditions.

[0137] At step 7, the orchestrator 310 may request the CIOS center 304 to recompile each group configuration associated with the version set (and / or identified by SPAMs in the SPAM set) using the current region data. In some embodiments, the request may indicate the version of each group configuration and / or artifact.

[0138] At step 8, CIOS center 304 can (e.g., directly or via real-time region data distributor 306) obtain current region data from DB 308 and retrieve any suitable group configurations and artifacts based on the version requested by orchestrator 310.

[0139] At step 9, CIOS center 304 can recompile the obtained group configurations using the area data obtained at step 8 to inject the current area data into these group configurations. CIOS center 304 can return the compiled group configurations to orchestrator 310. In some embodiments, CIOS center 304 can simply indicate that compilation is complete, and orchestrator 310 can access the recompiled group configurations via RRDD 306.

[0140] In some embodiments, at step 10, orchestrator 310 may perform static group analysis on the recompiled group configuration (and / or SPAM). As part of the static group analysis, orchestrator 310 may parse the group configuration (and / or SPAM) (e.g., using libraries associated with a declarative infrastructure provider (e.g., Terraform®, etc.)) to identify dependencies. Data generated by the static group analysis (e.g., “SFA data,” including the identified dependencies) may be stored for later use. Based on the analysis and the identified dependencies (e.g., SFA data), orchestrator 310 may generate any appropriate number of data structures (e.g., directed acyclic graphs) that identify the order of releases identified in the group configuration (or from any appropriate portion of one or more service plans, such as from the group configuration entity of a service plan). Based on any portion of the group configuration (and / or SPAM, including but not limited to...) Figure 8 The group configuration entity 800 generates and specifies the releases necessary for building services and the order of releases, which can be referred to as a "service DAG". In some embodiments, the orchestrator 310 can generate a directed acyclic graph (referred to as a "build graph") corresponding to each SPAM, where each node represents a build milestone, and edges indicate the execution units and capabilities (and / or skills) that transform services between build milestones. Each execution unit can represent multiple releases that transform services between build milestones when executed. Any suitable number of service DAGs can be combined to form a build dependency graph 338. The build dependency graph 338 can be an acyclic directed graph that identifies the order in which releases are executed to bootstrap one or more services in a new region.

[0141] In some embodiments, the build dependency graph 338 may be a region-level dependency graph, which includes every release that each service to be bootstrapping within a region / datacenter may need (or require). Each node in the build dependency graph 338 may correspond to any suitable portion of the bootstrapping services. A particular bootstrapping order may be identified at least in part based on dependencies. In some embodiments, dependencies may be expressed as attributes of nodes and / or indicated via edges of the graph connecting the nodes. The orchestrator 310 may traverse the build dependency graph 338 (e.g., starting from the starting node) to drive the region build operation. Any suitable portion of the service DAG and / or build dependency graph 338 may be accessible via one or more user interfaces (e.g., by...). Figure 1 The CIOS 102 can be presented through one or more interfaces provided by any suitable component (including the orchestrator 310, CIOS center 304, etc.).

[0142] In some embodiments, orchestrator 310 may utilize a cycle detection algorithm to detect the existence of cycles (e.g., service A depends on service B, and vice versa). Orchestrator 310 may identify orphaned capability dependencies. For example, orchestrator 310 may identify orphaned nodes in the dependency graph 338 that are not connected to any other nodes. Orchestrator 310 may identify erroneously published capabilities (e.g., when a capability is published prematurely, and the corresponding functionality is not yet available). Orchestrator 310 may detect from the graph the existence of one or more instances of the same capability being published. In some embodiments, any suitable number of these errors may be detected, and orchestrator 310 (or another suitable component, such as CIOS center 304) may be configured to notify or otherwise present this information to the user (e.g., via electronic notifications, user interface, etc.). In some embodiments, orchestrator 310 may be configured to force the deletion / recreation of resources to break circular dependencies and may again provide instructions to CIOS center 304 to perform bootstrapping operations for these resources and / or their corresponding group configurations.

[0143] The starting node of the dependency graph 338 can correspond to building ViBE 316 (or the various services within ViBE), and the second node can correspond to the bootstrap DNS. Steps 11-16 can be associated with deploying the resources and / or artifacts identified in the corresponding ViBE group configuration or SPAM (via deployment orchestrator 317). Figure 2 Example of deployment orchestrator 218) to ViBE 316 (e.g., respectively) Figure 1 ViBE 116 and Figure 2 (This corresponds to the example of ViBE 202). That is, Figure 3 Steps 11-16 are generally related to Figure 2Steps 1-6 correspond to those in the previous steps. Once notification of the existence of the ability (or skill) is received (e.g., instructions are given respectively), Figure 2 (If the capability services 208, workers 210 and Puffin regions 209, corresponding to capability services 318, workers 320 and / or Puffin regions 342, have been deployed / available), the orchestrator 310 can then restart traversing the dependency graph 338 to identify the next operation / release to be performed.

[0144] Orchestrator 310 can continue traversing the dependency graph 338 to identify one or more releases corresponding to the deployment of DNS 322. Steps 17-22 can be performed to deploy DNS 322. Figure 2 (Example of DNS 212). These operations can generally be used with... Figure 2 Steps 7-12 correspond to each other.

[0145] At step 22, capabilities (or skills) may be published and / or stored, indicating that DNS 322 is available. In some embodiments, CIOS zone 314 and / or deployment orchestrator 317 may first transmit the availability of capabilities or skills (e.g., to capability service 318 or Puffin zone 342, respectively). If a skill has been published, then Puffin zone 342 may transmit data to capability service 318 to indicate that one or more corresponding capabilities have been published. After detecting the publication of a capability (e.g., data provided via capability service 318, possibly triggered based on skill-related data provided by Puffin zone 342), orchestrator 310 may restart traversing the dependency graph 338. In this traversal, orchestrator 310 may identify any suitable portions of an instance of a CIOS zone (e.g., an example of CIOS zone 314) that will be deployed to ViBE 316. In some embodiments, steps 17-22 may be substantially repeated to deploy CIOS zone (ViBE) 326 ( Figure 1 Instances of CIOS region 110 and CIOS region 314, and worker 328 are deployed to ViBE 316. Capabilities available in CIOS region (ViBE) 326 can be transferred to capability service 318. If a skill is used to indicate that CIOS region (ViBE) 326 is available, then Puffin region 342 can transfer data to capability service 318 indicating that one or more corresponding capabilities are available. The interaction between Puffin region 342 and capability service 318 allows any suitable combination of capabilities and / or skills to express the progress of region construction. In some embodiments, when the build dependency graph 338 identifies changes made through capability releases and dependencies, the progress demonstrated by a skill release can be used to trigger a corresponding capability release so that the skill can trigger the progress of region construction.

[0146] After detecting that CIOS region (ViBE) 326 is available, orchestrator 310 can restart traversing the build dependency graph 338. During this traversal, orchestrator 310 can identify deployment orchestrators (e.g., deployment orchestrator 330 (an example of deployment orchestrator 317)) that will be deployed to ViBE 316. In some embodiments, steps 16-21 can be substantially repeated regarding the deployment of deployment orchestrator 330. Information identifying the capability can be transmitted to the capability service 318 (e.g., by CIOS region 314, worker 320, and / or Puffin region 342) indicating that deployment orchestrator 330 is available.

[0147] After deploying the deployment orchestrator 330, ViBE 316 can be considered available to handle subsequent requests. Upon detecting the availability of deployment orchestrator 330, orchestrator 310 can instruct subsequent bootstrap requests to be routed to ViBE components instead of utilizing host zone components (components of host zone 332). Therefore, orchestrator 310 can continue traversing the build dependency graph 338, instructing release execution to ViBE 316 at each node via CIOS center 304. CIOS center 304 can then transmit release requests to CIOS zone (ViBE) 326 to implement release execution according to the instructions of orchestrator 310.

[0148] At any suitable point during this process, target area 334 may become available. Indication of target area availability can be identified from area data of target area 334 provided by user 303 (e.g., as an update to the area data). The availability of target area 334 may depend on establishing a network connection between target area 334 and an external network (e.g., the Internet). This network connection may be supported by a public network (e.g., the Internet), but may use software security tools (e.g., IPsec) to provide one or more encrypted tunnels (e.g., IPsec tunnels, such as tunnel 336) from ViBE 316 to target area 334. As used herein, “IPSec” refers to a suite of protocols used for authenticating and encrypting network traffic over a network using the Internet Protocol (IP), and may include one or more available implementations of this suite (e.g., Openswan, Libreswan, strongSwan, etc.). The network can connect ViBE 316 to the service territory of target area 334.

[0149] Before establishing an IPsec tunnel, the initial network connection to target area 334 can reside on a connection sufficient to allow bootstrapping networking services (e.g., an out-of-band VPN tunnel) until an IPsec gateway can be deployed on an asset (e.g., a bare metal asset) within target area 334. To bootstrap network resources in the target area, deployment orchestrator 330 can deploy an IPsec gateway on the asset within target area 334. Deployment orchestrator 330 can then deploy a VPN host in target area 334, configured to terminate the IPsec tunnel from ViBE 316. Once services in ViBE 316 (e.g., deployment orchestrator 330, service A, etc.) can establish an IPsec connection with the VPN host in target area 334, bootstrapping operations from ViBE 316 to target area 334 can begin.

[0150] In some embodiments, the bootstrapping operation can begin with a service in ViBE 316, provisioning resources in target region 334 to support managed instances of the core service when deploying it from ViBE 316. For example, a host provisioning service can provision a hypervisor on infrastructure (e.g., bare metal host) in target region 334 to allocate compute resources for VMs. When the host provisioning service completes the allocation of physical resources in target region 334, it can publish information indicating a capability that indicates the physical resources in target region 334 have been allocated. This capability can be (e.g., by worker 328) published to capability service 318 via CIOS region (ViBE) 326.

[0151] After establishing the hardware allocation for target region 334 and publishing it to capability service 318, CIOS region (ViBE) 326 can orchestrate the deployment of core service instances from ViBE 316 to target region 334. This deployment can be similar to the process described above for building ViBE 316, but uses components of ViBE (e.g., CIOS region (ViBE) 326, worker 328, deployment orchestrator 330) instead of components of the host region 332 service territory (e.g., CIOS region 314 and deployment orchestrator 317). The deployment operation can typically correspond to steps 17-22 above.

[0152] When a service is deployed from ViBE 316 to target zone 334, the DNS records associated with that service can correspond to instances of the service in ViBE 316. The DNS records associated with the service can be updated at any appropriate time to complete the deployment to target zone 334. In other words, instances of the service in ViBE 316 can continue to receive traffic (e.g., requests) until the DNS records are updated. A service can be partially deployed to target zone 334 and (e.g., to capability service 318) publish information indicating that the service is partially deployed. For example, a service running in ViBE 316 can be deployed to target zone 334 along with its corresponding compute instances, load balancers, and associated applications and other software, but may require waiting for database data to be migrated to target zone 334 before full deployment. DNS records (e.g., those managed by DNS 322) can still be associated with the service in ViBE 316. Once the service's data migration is complete, the DNS records can be updated to point to the operational service deployed in target zone 334. Then, services deployed in target region 334 can receive service traffic (e.g., requests), while service instances in ViBE 316 can no longer receive service traffic.

[0153] At any appropriate time during method 300, Puffin area 209 can receive alerts from one or more alert services (e.g., one or more alert services 344). Figure 1 Examples of one or more alarm services 122) receive and / or obtain alarm data. In some embodiments, alarm data may be processed by Puffin area 342 (or Puffin area 342 may transmit alarm data or data derived from alarm data to Puffin center 340). In some embodiments, Puffin area 342 and / or Puffin center 340 may transmit skill health information to orchestrator 310, indicating the corresponding health status associated with one or more skills. In some embodiments, Puffin area 342, Puffin center 340 and / or orchestrator 310 may be configured to perform any suitable portion of the operations discussed above in conjunction with method 300. In some embodiments, Puffin area 342, Puffin center 340 and / or orchestrator 310 may be configured to resume and / or perform any suitable portion of the operations of method 300 (e.g., based at least in part on user input, subsequent alarm data indicating an update to the health status associated with one or more skills, based at least in part on skill health coverage values, etc.).

[0154] Figure 4 It is a block diagram depicting a data model 400 representing skill-related metadata according to at least one embodiment.Figure 4 Each data structure depicted may include an ID (e.g., an identifier) ​​that uniquely identifies the data structure. This ID can be used to refer to a specific instance of a particular data structure.

[0155] In some embodiments, service metadata 402 may include any suitable data corresponding to the service. Service metadata 402 may include any suitable attributes of the service and their corresponding values, while skill metadata 404 may similarly include any suitable attributes of the skill and their corresponding values. The association between service metadata 402 and skill metadata 404 can indicate the relationship between the service and the skill (e.g., the service is expected to release the skill during construction or runtime). Figure 4 As illustrated, service metadata 402 can be stored in multiple data structures (e.g., namespace data structure 408 and service data structure 406), but any suitable number or type of data structure can be used. Service metadata can include, but is not limited to, suitable combinations of the following: ID, service name (corresponding to the name of the service), quarantine zone ID (corresponding to the identifier of the quarantine zone to which the service is deployed), product part ID, namespace ID (the identifier of the namespace associated with the service), namespace name (the name associated with the namespace associated with the service), and / or the quarantine zone ID corresponding to the namespace. In some embodiments, service metadata 402 can be organized (read from storage, uploaded to...). Figure 1 (e.g., Puffin Center 118). In some embodiments, service metadata 402 may be obtained by Puffin Center 118 from another system, or typically obtained using processing that does not include user input of the information through any user interface provided by Puffin Center.

[0156] Skill metadata 404 may include any suitable number of data structures (e.g., data structures 410-420). In some embodiments, skill data structure 410 may include attributes and values ​​corresponding to any suitable combination of skill ID, skill name, skill fleet, major version, isDeprecated indicator, one or more capabilities (e.g., a set of capability identifiers), useInstead indicator, quarantine zone ID, producer ID, namespace ID, and recovery ring level. In some embodiments, the values ​​stored in skill data structure 410 for quarantine zone ID, producer ID, and / or namespace ID may be matched with the quarantine zone ID, service name, or namespace name of service metadata 402, respectively. Matching between one or more of these attribute values ​​can be used as an association between skill metadata 404 and service metadata 402 (indicating that the corresponding service is expected to release the skill at a certain point in time).

[0157] Skill version data structure 412 can be associated with skill data structure 410, at least in part, based on the matching of the skill ID value of skill version data structure 412 with the ID value of skill data structure 410. Skill version data structure 412 may include attributes and values ​​corresponding to any suitable combination of the following: (skill version) ID, skill ID (e.g., a unique identifier for a skill), major and / or minor versions that individually or collectively identify a particular implementation of a skill, patch version (e.g., a version identifier for a skill to be used to correct a previous bug), deprecation indicator (indicating whether a skill has been deprecated), health check attributes (one or more instances of alarm data referencing one or more instances of health check data structure 414), installation status (indicating the installation status, such as declared, selected, installing, installed, banned, deactivated, uninstalling), health status (e.g., indicating the health status of a skill, such as unknown, healthy, unhealthy, etc.), and observability attributes. Observability attributes may be used to store any suitable data identifying the operational or data points required to collect telemetry, alarm, and / or log data for a skill version. Skill version data structure 412 can be associated with health check data structure 414, which can be configured to maintain any appropriate number of alert tags associated with that skill. For example, the healthCheck property of skill version data structure 412 can reference any appropriate number of health check data structures, corresponding to one or more instances of health check data structure 414.

[0158] In some embodiments, the health check data structure 414 may include any suitable combination of alarm identifier (alarm ID, a unique identifier indicating an alarm), alarm label name (the name of the alarm), quarantine zone identifier (quarantine zone ID, indicating the quarantine zone to which the alarm is defined), continuation lexicon (lexicon that can be used to obtain the alarm transition history), namespace identifier (namespace ID, indicating the specific namespace to which the alarm is defined), and status value (indicating the health status corresponding to the alarm). Alarm data corresponding to multiple alarms can be maintained in the health check data structure 414. For example, the alarm ID may include a list of multiple alarm IDs corresponding to a list of alarm label names stored in the alarm label name attribute. The quarantine zone ID attribute may also be a list of quarantine zone IDs corresponding to the alarms and labels in the alarm ID and alarm label name ID attributes of the health check data structure 414. In some embodiments, multiple sets of attributes alarmID, alarmLabelName, compartmentID, continuationToken, and status may be stored, each set of attributes corresponding to a single alarm.

[0159] In some embodiments, the health check data structure 414 may be stored with one or more alarm services (e.g., Figure 3 344, one or more alarm services Figure 1 The data corresponds to one or more alarm services 122. For example, the namespace identifier of the health check data structure may store the namespace corresponding to the skill (e.g., corresponding to an instance of skill data structure 410). In some embodiments, the association between a skill and an alarm may be maintained at least in part based on storing the same namespace identifier in the namespace ID attribute of an instance of health check data structure 414 and the namespace ID attribute of an instance of skill data structure 410. In some embodiments, the status attribute may store a value indicating the health and / or alarm status of the skill (e.g., an alarm identified by alarmID, an alarm identified by namespaceID, etc.). In embodiments where status from multiple alarm services is utilized, multiple status attributes may be employed to maintain the status of each corresponding alarm (e.g., one status for alarmID, another status for namespaceID, etc.).

[0160] Skill data structure 410 can be associated with skill metadata data structure 416. Skill metadata data structure 416 can include attributes and values ​​of any suitable combination of (an instance of skill metadata data structure 416) ID, Jira queue, owner contact, organization leader, and phone book ID. Phone book ID can be an identifier corresponding to a separate system configured to store contact data. Skill metadata data structure 416 can be used to store any suitable contact data (e.g., name, email, address, phone number, etc.) associated with the skill and the service associated with that skill (e.g., service team members).

[0161] Skill data structure 410 can be associated with skill consumer data structure 418. Skill consumer data structure 418 may include attributes and values ​​of any suitable combination of (skill consumer's) ID, type, status, consumption region, version requirement, consumed skill ID, and consumed service ID. Skill consumer data structure 418 can be configured to store any suitable information about services and / or skills that depend on the skills defined in skill metadata 404.

[0162] The skill data structure 410 can be associated with the skill group data structure 420. The skill group data structure may include attributes and values ​​such as the (skill group) ID, the skill group name, and any suitable combination of a set of one or more skill IDs associated with that skill group.

[0163] Each of the data structures 406-420 can be stored in one or more data repositories, and the data structure can be identified and obtained, at least in part, based on values ​​stored in another data structure through the associations discussed above (e.g., via lookup and / or query operations). For example, all skills associated with a service can be identified by querying one or more data repositories for all skill data structures associated with producer IDs that match the IDs in the service data structure 406 of the service metadata 402.

[0164] Although Figure 4 The document presents a certain number and specific combinations of data structures, but any suitable number or type of attributes and / or values ​​and / or data structures can be utilized. In some embodiments, Figure 4 Data in any data structure described herein can be separated into multiple data structures, or combined and stored in a manner similar to [the original text]. Figure 4 The data structures depicted are fewer than those in other data structures. The relationships indicated between these data structures can be related to... Figure 4 The relationships shown may be similar, or they may be different. As a non-limiting example, the data depicted using data structures 410-420 may be similarly stored in more or fewer data structures. For example, in some embodiments, the data depicted in data structures 410-420 may be provided in a single data structure.

[0165] Figure 4 Each data structure can be at least partially based on an identifier that references one or more other data structures. Figure 4 This is related to other data structures within the skill data structure 410. For example, an instance of the skill version data structure 412 can be associated with a specific instance of the skill data structure 410, at least in part, based on the match between the value of the skillID attribute of the skill version data structure 412 and the value of the ID attribute of a specific instance of the skill data structure 410. As another example, the skill consumer data structure 418 can be associated with a skill based on referencing the ID of the skill data structure 410 in its consumingSkillID attribute. As yet another example, an instance of the skill group data structure 420 can be associated with one or more instances of the skill data structure 410 based on referencing the IDs of those skills in its skill attribute. As yet another example, the skill version data structure 412 can reference one or more instances of the health check data structure 414, at least in part, based on referencing the IDs of those health check data structures in its healthCheck attribute. An instance of the skill data structure 410 can be associated with a service, at least in part, based on referencing the ID of the service data structure 406 corresponding to a specific service via its producerServiceID attribute.

[0166] Any suitable number of skill metadata 404 instances (corresponding to individual skills) can be associated with a single service metadata 402 instance and can be used to represent the processing of a deployment service, where the order of deployment tasks is represented by instances of skill metadata 404. Each skill corresponding to an instance of service skill metadata 404 can be tracked, updated, or otherwise analyzed to present information about the service's deployment processing, drive service deployment, validate build plans, or... Figure 3 The dependency relationship diagram 338, etc.

[0167] Figure 5 This is a block diagram depicting a data model 500 representing various metadata associated with SPAM (e.g., service plans and inventories) according to at least one embodiment. Each of the data structures 502-508 can include any suitable number of attributes with corresponding values ​​(as shown in the figure). These data structures can be identified in a shared file or any suitable number of files. SPAM can be... Figure 5 Maintained within the data structures described herein or different data structures. Figure 5 The data structure can be combined or separated in any suitable way to maintain the metadata corresponding to SPAM.

[0168] Figure 5 Each of the data structures depicted (e.g., data structures 502-508) may include an ID (e.g., an identifier) ​​with a corresponding value that uniquely identifies the data structure. This ID can be used to refer to a specific instance of a particular data structure. Each of the data structures 502-508 may include an attribute corresponding to “spamName”, which can be used to maintain the association between data structures 502-508 (e.g., indicating that each of data structures 502-508 corresponds to the same SPAM).

[0169] In some embodiments, the SPAM data structure 502 may include any suitable data corresponding to SPAM. For example... Figure 5As depicted, the SPAM data structure 502 includes a phonebookID attribute. The corresponding value of this attribute can identify the service and / or contact identified as the owner of the SPAM. In some embodiments, the value corresponding to the "phonebookID" attribute can be an identifier corresponding to a separate system or database configured to maintain contact information for a service team. The phonebookID value can be used as a lookup value to retrieve corresponding contact information (e.g., email address, service team name, phone number, etc.) for the service that owns the SPAM corresponding to the SPAM data structure 502. The SPAM data structure 502 may include corresponding IDs that identify specific instances of a service plan data structure (e.g., service plan data structure 504) and a service list (e.g., service list 506) via the attributes "servicePlanID" and "serviceManifestID," respectively. Therefore, in some embodiments, the SPAM data structure 502 maintains a mapping between service plans (represented by the service plan data structure 504) and lists (represented by the service list data structure 506).

[0170] Service plan data structure 504 can represent a service plan and entities included within it. Service plan data structure 504 can include a "buildMilestones" entity, which has corresponding values ​​comprising an ordered list of build milestone identifiers (e.g., names, alphanumeric strings, etc.). The "buildMilestones" attribute can identify, include, or otherwise relate to... Figure 6 The construction milestone entity 600 corresponds to this.

[0171] Figure 6 This is a block diagram depicting a build milestone entity 600 of a service plan and inventory according to at least one embodiment. The build milestone entity 600 can identify any suitable number of build milestones corresponding to service builds. Figure 6 As described in the document, the construction milestone entity 600 identifies four construction milestones corresponding to code snippets 602-608.

[0172] Code snippet 602 identifies a build milestone titled "absent," its corresponding description, and a list of capability releases. This list indicates the capabilities (also known as "capability dependencies" or capabilities that this build milestone depends on) that should (in some cases, must) be released before transitioning to this build milestone. In this instance, the list of capability releases is empty, indicating that no capability releases are expected before transitioning to the build milestone "absent."

[0173] Code snippet 604 identifies a build milestone titled "service-partial," its corresponding description, and a list of capabilities to be released. This list indicates the capabilities that should (and in some cases, must) be released before transitioning to this build milestone. In this example, the list of capabilities to be released includes "serviceA_namespace," indicating that the "serviceA_namespace" capability is expected / required to be released before the transition to the build milestone "service-partial" can occur.

[0174] Code snippet 606 identifies a build milestone titled "service-available," its corresponding description, and a list of capabilities to be released. This list indicates the capabilities that should (and in some cases, must) be released before transitioning to this build milestone. In this example, the list of capabilities to be released includes "serviceA_backend," indicating that the "serviceA_backend" capability is expected / required to be released before the transition to the build milestone "service-available" can occur.

[0175] Code snippet 608 identifies a build milestone titled "complete," its corresponding description, and a list of capabilities to be released. This list indicates capabilities that should (and in some cases, must) be released before transitioning to this build milestone. In this instance, the list of capabilities to be released is empty, indicating that no capabilities are expected to be released before the transition to the build milestone "complete" can occur.

[0176] In some embodiments, each build milestone specifies the release of features that a service should have published prior to reaching a particular build milestone. The set of build milestones can indicate a high-level overview of the processes used to build a service and can be used / consumed by external consumers (e.g., other services that depend on the services associated with the build milestones). Build milestones can be defined and used to express partial functionality available for a service. This allows other services to continue building as soon as the functionality that other services depend on becomes available, rather than waiting for the relevant services to become fully available. Build milestones can be used for coordination between services. When building a service does not involve coordination with other services, the service plan for that service may not include build milestones or may include multiple default build milestones (e.g., “absent” and “complete”). In some embodiments, build milestones defined in each service plan for regional builds can be used to generate high-level graphs and / or sequence diagrams. This can provide service teams with a simplified graphical view of complexity, making it easier to understand and synthesize regional build processes. As a non-limiting example, with Figure 3Compared to the build dependency graph 338, which may include dozens of nodes (representing capabilities) for each service, the graph / diagram generated using build milestones for each corresponding service may include a reduced number of nodes (e.g., 3 for each service, 4 for each service, depending on the implementation of the service's build milestones).

[0177] Return to Figure 5 A service plan, represented by service plan data structure 504, may include execution unit entities. As depicted, service plan data structure 504 includes an "ExecutionUnit" property with corresponding values, which comprises a collection of execution units. Each execution unit can be used to describe how the service transitions from one build milestone to the next. Execution units can describe a directed acyclic graph of releases that need to be executed. The definition of this graph centralizes the definitions of how releases (e.g., groups / phases / change types (e.g., Infra or App)) interact. The order of execution units can be driven by the order of build milestones defined in the corresponding build milestone entities.

[0178] Figure 7 An example execution unit entity 700 is depicted according to an example service plan based on at least one embodiment. The execution unit entity 700 can identify any suitable number of execution units (e.g., by corresponding construction milestone entities such as...). Figure 6 Each transition between ordered pairs of construction milestone entities (600) is an execution unit. Figure 7 Two such transitions are described (e.g., the transition between the construction milestone "absent" and "service-partial", and another transition between the construction milestone "service-partial" and "service-available"). Another unit of execution could be defined for the transition between the construction milestones "service-available" and "complete", but for the sake of brevity, Figure 7 Not described in the text. Since construction milestones are defined as having a sequential order, execution units are also defined as having a sequential order due to their association with ordered construction milestones.

[0179] like Figure 7 As depicted, execution unit entity 700 includes execution unit 702 and execution unit 704. Execution unit 702 may correspond to the transition between the build milestones "absent" and "service-partial". Execution unit 704 may correspond to the transition between the build milestones "service-partial" and "service-available". Execution unit entity 700 may include additional execution units, but these are not specified in the text. Figure 7Described in the text.

[0180] Each execution unit may include one or more external capability dependencies. For example, execution unit 702 specifies a set of external capability dependencies, as depicted at 705. These external capability dependencies may include one or more capabilities that should (or in some cases, must) be released before transitioning from one build milestone (e.g., an "absent" build milestone) to another build milestone (e.g., a "service-partial" build milestone). The set of external capability dependencies may be the union of all released external capability requirements defined in the step section of the corresponding execution unit. Figure 5 As described herein, external capability dependencies are indicated to include the capabilities “certificate_service_ready” and “capability_service_ready”. Any suitable number of external capability dependencies can be specified. In some embodiments, the execution steps may not include external capability dependencies.

[0181] Each execution unit may indicate one or more capabilities that are expected to be released upon completion of the execution of the release corresponding to that execution unit. For example, execution unit 702 specifies a set of capabilities to be released at 707, indicating a superset of all capabilities expected to be released once the execution of the release indicated in step section 706 is completed. Capability releases may additionally or alternatively be defined as being associated with a build milestone related to the execution.

[0182] Each execution unit may include any suitable number of steps (or groups of steps) that, when executed, move a service from one build milestone to another. For example, execution unit 702 specifies a step segment 706 including an InfraAppPair step type. An InfraAppPair step specifies one or more infrastructure releases expressed using ET aliases (e.g., ET:Alias ​​flock_DP / phase_DP / alarms_et) and group / phase execution target (ET) transitions (e.g., Transition: absent→DP_complete_alarms_infra). For each ET alias, each release reference may represent an ET checkpoint transition performed with the release. In some embodiments, the orchestrator (e.g., Figure 1 The orchestrator 106 can be configured to enforce infrastructure releases defined in the InfraAppPair step prior to releases defined in the corresponding App segment of the same InfraAppPair step. Figure 7In the examples provided, the releases corresponding to the transitions absent→DP_complete_alarms_infra and absent→DP_initial_region_infra can be performed before the application releases corresponding to absent→DP_complete_alarms_app and absent→DP_initial_region_app, as specified in step section 706. In some embodiments, the orchestrator can create a directed acyclic graph or another suitable ordered execution plan that schedules infrastructure releases before application releases defined in the InfraAppPair step type. In some embodiments, execution objectives of different phases may not be included in the same InfraAppPair step, and / or, if an order is enforced in the corresponding group configuration, ET ordering is required to reflect the order defined in that group configuration.

[0183] Other step types can be utilized. For example, if the region building stages have no dependencies on each other, then a parallel step type can be used, and if their external dependencies are resolved substantially simultaneously, then they can be executed in parallel (e.g., substantially simultaneously, concurrently, and / or overlapping). If two distinctly different stages (e.g., with distinctly different names) have direct dependencies on each other (e.g., their corresponding releases are expected to be executed serially), then a serial step type can be used.

[0184] Once all releases for the corresponding execution unit that caused the transition are complete, all anticipated releases for the build milestone (e.g., the build milestone to which the transition occurred) can be published. For example, the capability "serviceA_namespace" can be published after the transition to the build milestone "service-partial", such as... Figure 6 As indicated in code snippet 604.

[0185] As another example, execution unit 704 includes a step segment 708 that comprises multiple InfraAppPair step types, each including one or more infrastructure releases and application releases. These InfraAppPair steps can be grouped and ordered using serial step types, as indicated at 710.

[0186] Figure 8 An example group configuration entity 800 is depicted according to at least one embodiment of an example service plan. The group configuration entity 800 can be used to specify... Figure 7The generic configuration for release is expressed in the execution unit entity 700. Specifications for all execution units included in the installation process can be indicated here. The group configuration entity 800 can indicate any suitable number of projects (e.g., the project "sample-project" identified at 802) and any suitable number of groups for each project (e.g., the group "service-dp" identified at 804). Any suitable number of stages can be defined for each group within the stage section 806. Similarly, any suitable number of execution targets can be defined for each stage within the execution target section 808. Each execution target section can include any suitable number of execution target (ET) checkpoints defined within the checkpoint section 810. In some embodiments, the service plan including the group configuration entity 800 can be validated to ensure that all references included in the service plan reference valid components (e.g., by...). Figure 1 (Components managed by CIOS 102).

[0187] Checkpoint section 810 can define any suitable number of checkpoints. These checkpoints can be grouped by type (e.g., "Infra" indicates infrastructure change type, "App" indicates application change type), as depicted, or each checkpoint can indicate a corresponding change type. In some embodiments, the order in which checkpoints are provided in checkpoint section 810 can define the order in which checkpoints are executed. In some embodiments, when a checkpoint indicates a corresponding change type, an infrastructure change-related release can precede any application change-related release, even if an application change-related release is listed in checkpoint section 810 before an infrastructure release. Each checkpoint can indicate a corresponding group profile (as depicted at 812), external and / or internal capability dependencies (as depicted at 814), and capability releases for each release (as depicted at 816). Providing these mappings in group configuration entity 800 simplifies Figure 7 The execution unit entity 700. The group configuration entity 800 can be used to map the execution unit steps that reference ET checkpoints of the group configuration entity 800 to CIOS management components that define the release (e.g., group profiles). Providing this mapping in the group configuration entity 800 simplifies... Figure 7 The execution unit entity 700.

[0188] One of the challenges of previous implementations was understanding what progress should occur each time a release is performed for a specific stage and group of changes, especially when that stage has optional capability dependencies. For orchestrators in previous implementations (e.g., Figure 1Each release initiated by the orchestrator 106 is unclear about which optional capability dependencies are required and which capabilities should be released through the release execution. The ET checkpoint defined in the group configuration entity 800 can resolve this ambiguity by explicitly specifying the capability dependencies for each release of the execution target and the expected capability releases.

[0189] Figure 9 It describes a service plan according to at least one embodiment (e.g., service plan 902 (with) Figure 5 The service plan data structure 504 corresponds to the example service plan) and its parts are related to the list (e.g., list 904). Figure 5 The diagram 900 shows the relationship between the service list 506 and the example service list 506.

[0190] Service Plan 902 may include Figure 6 The construction milestone entity 600 Figure 7 Execution unit entity 700 and Figure 8 Any suitable combination of group configuration entities 800. In some embodiments, service plan 902 indicates service build implementations at different granularity levels. The highest granularity level indicates build milestones (e.g., build milestones 906-912, which have corresponding build milestone entities similar to build milestone entity 600). Service plan 902 can be used to specify service build as a series of build milestones that a service undergoes during its build process. Each milestone can represent an interaction that occurs between a given service and other services (e.g., publishing the ability to unblock the build of other services, and / or consuming new capabilities published by another service). These build milestones can be used to understand at a high level how a service is built and the inter-service coordination required by that service, without needing to understand the group of all services and their intra-service coordination.

[0191] Build milestones 906-912 can be individually associated with a set of external capabilities, the transition to which a build milestone depends on that set of external capabilities. These capabilities may include anticipated released capabilities related to external services (e.g., service 914, including other services involved in the region build). As a non-limiting example, build milestone 906 may depend on a capability set 916 (including one or more capabilities) defined in a corresponding execution unit transition specifying the transition to build milestone 906. Build milestones 906-912 may be associated with the release of capabilities required to start / continue the installation of another service. For example, build milestone 908 may be associated with a capability set 918, which includes one or more capabilities expected to be released prior to the transition to build milestone 908. In some embodiments, build milestones may be used to generate high-level sequence diagrams that can be used to identify the progress of the region build.

[0192] Each build milestone can be associated with a corresponding execution unit. For example, build milestone 906 can be associated with execution unit 920 (and...). Figure 7 Each execution unit (including execution unit 920) is associated with an instance of execution unit entity 700. Each execution unit may include any suitable number of releases (such as release 922) and the order in which these versions are executed. Each release may be expressed within the execution unit as an execution target checkpoint transition. The corresponding execution target checkpoint transition may indicate external and / or internal capability dependencies of the transition / release and may provide a mapping to the corresponding group configuration identified in service list 904. For example, release 922 may ultimately be mapped to a specific group configuration using service list item 924. Service list item 924 may be identified by an identifier referenced by execution unit 920 and provided in the execution target checkpoint corresponding to release 922.

[0193] Using the entities of the service plan, one or more acyclic graphs can be generated. As a non-limiting example, a directed acyclic graph defining service construction can be generated. This DAG can be referred to as a "service DAG" and can include any suitable number of nodes representing corresponding releases and the order in which these releases are performed to build the service. Nodes themselves or edges between nodes can be associated with external and / or internal capability dependencies. In some embodiments, a graph, list, sequence graph, or any suitable data structure can be generated for the services and / or any suitable number of services built for a region using construction milestones corresponding to one or more services. This data structure can be referred to as a "milestone plan." As yet another example, a service plan (e.g., as part of a SPAM set including service plans and a list of one or more services corresponding to them) can be used to generate... Figure 3 The dependency graph for building the system is shown in Figure 338. (As above...) Figure 3 As described in [the document], constructing a dependency graph 338 can be used (e.g., by [other entities]). Figure 3 The orchestrator 310 drives the region building operations (to execute, for example, a deterministic order of infrastructure and application releases for the region / data center).

[0194] In some embodiments, service list 904 can be used to specify the group versions and artifact versions that will be used to create releases for the execution targets specified in service plan 902. Service list 904 can be used to verify service plan 902 at least in part based on the fact that each release identified in service plan 902 is included in service list 904. In some embodiments, each service list item (e.g., service list item 924) can be mapped to a version set item, such that the service list can be used to verify the version set used by CIOS 102 to perform region builds. As a non-limiting example, a SPAM set can be constructed that includes all SPAMs corresponding to the services to be bootstrated within a region / datacenter. The list of SPAM sets can be used to verify the version set (if used) to ensure that all group profiles and artifacts referenced in the SPAM set are included in the version set to be used to build the region.

[0195] Figure 10 An example user interface (UI) 1000 is depicted according to at least one embodiment for viewing aspects of services related to region construction. In some embodiments, UI 1000 may be provided by Figure 1 Any suitable component of CIOS 102 (e.g., respectively) Figure 1 CIOS Center 108 Figure 2 CIOS Center 214 and / or Figure 3 CIOS Center 304 Figure 1The editor 106, etc., is hosted. Any suitable data corresponding to the region construct can be presented on the user interface (UI) 1000. In some embodiments, UI 1000 may include UI elements (e.g., UI element 1002, drop-down menu) for selecting a specific region construct. After selecting a specific region construct (e.g., region X) from UI element 1002, data corresponding to that region construct can be presented on UI 1000. In some embodiments, UI 1000 includes a filter element 1004. Filter element 1004 may include any suitable user interface element, such as an edit box, radio button, check box, selection button, etc. Search element 1004 can be used to provide user input that can identify data presented via tabs 1010-1016. As depicted, search element 1004 includes an edit box 1006, a search button 1007, and a quick search area 1008. Edit box 1006 can be used to enter any suitable text defining one or more search parameters. As depicted, user data indicating search parameters for the project name "api-gateway" is shown. Once entered, the user can select search button 1007 to apply the search parameters. The data provided within tabs 1010-1016 can be identified, at least in part, based on data corresponding to project names including "api-gateway". In some embodiments, once the search parameters are used, an interface element for searching can be generated and provided via quick search area 1008. After selecting an interface element, the corresponding search parameters associated with that interface element can be used to identify the data provided via tabs 1010-1016.

[0196] UI 1000 may include region data 1018. Any suitable data associated with a region may be provided as part of region data 1018. For example, region data may include, but is not limited to, the name or other identifier of the selected region (e.g., region X), the status of the region's construction (e.g., none, under construction, in production, paused, deprecated, etc.), a progress indicator indicating the number of capabilities that have been released (e.g., 3350), the total number of capabilities expected to be released (e.g., 3548), and one or more processing indicators (e.g., indicators that the construction is processing events and passes).

[0197] The data presented on UI 1000 can initially correspond to "Group" tabs (e.g., Group tab 1010). In some embodiments, Group tab 1010 may include any suitable information corresponding to any service associated with search parameters provided via edit box 1006. If no search parameters are entered, Group tab 1010 may initially present data corresponding to each group / service associated with the region construction of region X. Figure 10 As depicted, group tab 1010 includes area 1019, which presents entries 1020, 1022, and 1024, each corresponding to an item in the area's structure that includes the string "api-gateway". Initially, entries 1020, 1022, and 1024 can be presented in a collapsed view. Entry 1024 provides an example of a collapsed view.

[0198] Each of entries 1020, 1022, and 1024 may correspond to a different group (e.g., a group / group profile specifying one or more releases that can be executed by the unit, each group associated with the project name "api-gateway"). Each entry may include any suitable group data, such as the group name (e.g., corresponding to a group profile), phonebook data (e.g., identifiers or contact data indicating service owner names and / or contact information, links to systems configured to maintain service owner data and / or contact information, etc.), version set identifiers (e.g., identifying the version set of infrastructure and / or the version set of application releases), the status of one or more infrastructure releases associated with the group, the status of one or more application releases associated with the group, the number of capabilities that the releases of the group commonly depend on, an indication of the number of capabilities expected to be generated / released by executing the releases of the group, and an indication of the number of components (e.g., other services) that depend on the capabilities generated by the group.

[0199] In some embodiments, at least a portion of the data provided via entries 1020, 1022, and 1024 can be maintained by any suitable combination of data structures 502-508, which can be used to represent entities of the service plan (such as...). Figure 6 The construction milestone entity 600 Figure 7 Execution unit entity 700 and Figure 8 The group configuration entity 800) and the corresponding list (e.g., Figure 9(Listing 904). For example, the values ​​of the project name, group name, and phone book ID can be identified by any suitable combination of data structures 502-508. In some embodiments, the number of capabilities that the release corresponding to entry 1020 depends on, and the number of capabilities that the group release will generate / publish, can be determined by any suitable combination of build milestones, execution units, and / or execution target checkpoints identified in the corresponding service plan. In some embodiments, the number of dependent groups (e.g., a set of releases that depend on one or more capabilities generated by executing a release of a given group) can be identified based on the number of SPAMs identified for other services, including dependencies on capabilities expected to be generated / published by the group release corresponding to entry 1022. Selecting the text of the project name, group name, or phone book ID can navigate the user to one or more additional user interfaces, within which details corresponding to the selected item can be presented. For example, selecting the group name “api-gateway” from area 1019 can navigate the user to an interface configured to provide any suitable data associated with the group name “api-gateway”.

[0200] After selecting interface element 1026, area 1019 can be modified to include an expanded view of item 1020. This expanded view of item 1020 can include area 1028, within which additional details related to the group corresponding to item 1020 can be provided. For example, each item can correspond to a group's stage / change type combination. Figure 10 Additional details, as described, may include the phase name, change type, indication of whether the change is automatically approved, status corresponding to the last release, number of open tickets (e.g., a percentage of the total number of tickets / work requests created for a given phase / group combination), indication of the number of capability dependencies for the group, indication of the number of capabilities expected to be generated by executing the release corresponding to the phase / group / change type, and indication of the number of services dependent on the capabilities generated by executing the release for the given group. Selecting interface element 1026 again removes area 1028 and the corresponding data from the display, thus returning entry 1020 to the collapsed view.

[0201] In some embodiments, summary data 1030 may be presented via group tab 1010. Summary area 1030 may present data corresponding to all groups presented via area 1019. Summary data 1030 may include the following indications: the number of distinct phonebook IDs associated with the groups presented within area 1019, the number of unique item names (e.g., one here due to filter parameters provided via text box 1006), the number of groups, group status indicators (e.g., as depicted at 1032, 14 groups successfully executed, 6 groups awaiting capability release, 0 groups indicating warnings, 0 groups indicating errors, and 0 groups awaiting retries), the total number of capabilities jointly consumed by the groups in area 1019, and the number of capabilities jointly generated by the groups in area 1019.

[0202] Selecting the Events tab 1012 navigates the user to a user interface where they can view event data corresponding to the group in region 1019. Events may include the time / timestamp when the event was received, the event name, and the event type. Selecting the Via tab 1014 evokes any suitable data corresponding to one or more passes performed during region construction. Selecting the Graph tab 1016 presents the user with a graph and / or any suitable combination of visualizations corresponding to the build dependency graph 338, service DAG, build milestone graph / chart, or any suitable combination of the above.

[0203] Figure 11 This is a schematic diagram depicting an example user interface 1100 corresponding to a graph of one or more services constructed according to at least one embodiment. In some embodiments, the user interface 1100 can select... Figure 10 The diagram will appear after the Graph tab 1016 or another suitable interface. The diagram presented via the user interface 1100 can be a service DAG, a construction milestone diagram / sequence diagram, or... Figure 3 Example of the build dependency graph 338.

[0204] User interface 1100 may include region 1102, within which any suitable portion of Figure 1106 may be displayed. In the depicted example, Figure 1106 may be... Figure 3 The build dependency diagram 338 corresponds to this. In some embodiments, a portion of the data presented within 1104 may correspond to a specific service or group (e.g., the api_gateway service or group). Figure 1106 may present upstream dependencies that can be identified within region 1104 (e.g., capabilities X, capabilities Y, and any suitable number of capabilities corresponding to user interface element 1107). Figure 10The 27 consumed capabilities indicated in the diagram), and one or more dependent services / capabilities can be depicted using element 1109. In some embodiments, dependency graph 1106 may be part of a larger graph (e.g., constructing dependency graph 338).

[0205] In some embodiments, any suitable portion of Figure 1106 can initially be hidden based on a default or selected viewing extent. For example, user interface element 1109 indicates 10 services / groups depending on the capabilities generated by the api_gateway. In some embodiments, user interface element 1132 can be used to modify the viewing extent of region 1104. For example, the slider of user interface element 1132 can be moved upwards to increase the extent of the viewing area (e.g., zoom out to present a larger portion of Figure 1106), or moved downwards to decrease the extent of the viewing area (e.g., zoom in on Figure 1106). For example, if user interface element 1132 is moved from... Figure 11 If the position depicted in the figure is shifted downwards to zoom in on Figure 1106, then user interface element 1109 can be expanded to present or replace the dependent capabilities corresponding to user interface element 1109 (e.g., 10), and / or user interface element 1130 can be expanded to indicate a list of capabilities generated by the api_gateway. In some embodiments, zooming outwards can reduce the information presented via the presented element, while zooming inwards can present additional information. Furthermore, or alternatively, zooming out can present additional data of Figure 1106, such as additional capabilities depending on at least one of the capabilities that the api_gateway depends on. The slider of user interface element 1132 can be used to expand the view to allow viewing of the entire build dependency graph 338.

[0206] Element 1128 can be presented in region 1104. Element 1128 can be used for multiple upstream and downstream dependencies. The number of upstream / downstream dependencies may differ from the number presented or indicated within region 1104. As depicted, element 1128 indicates that the API gateway has 27 upstream dependencies and 4 downstream dependencies, or in other words, the API gateway depends on 27 capabilities, and four capabilities (or services) consume / depend on capabilities generated by the API gateway.

[0207] User interface 1100 may include various options in area 1102, such as table option 1108, graph option 1110, and dependency drop-down menu 1112. In some embodiments, an option may be selected by default (e.g., graph option 1110). Although not depicted in the figure, selecting table option 1108 may present dependency data corresponding to Figure 1106 in tabular form (e.g., as entries in a table). In some embodiments, Figure 1106 may be generated based on skill metadata (in implementations that utilize skills) and / or one or more SPAMs that define such dependencies. Dependency drop-down menu 812 may be used to select build-time dependencies (dependencies identified from static analysis), runtime dependencies, service DAGs, build milestone graphs, etc. In some embodiments, user interface element 1132 may not be included in area 1104 when service DAGs and / or build milestone graphs are presented via user interface 1100.

[0208] Figure 12 This is a flowchart depicting an example method 1200 for driving data center construction using service plans and inventory according to at least one embodiment. The operation of method 1200 can be performed by... Figure 1 Any suitable combination of the components of CIOS 102 may be executed in any suitable order. For example only, method 1200 may be executed by... Figure 1 The orchestrator 106 and CIOS center 108 execute this. Method 1200 is expected to include a quantity ratio... Figure 12 The more or fewer operations shown.

[0209] Method 1200 may begin at 1202, wherein service plans corresponding to services among multiple services to be booted within the cloud computing environment can be maintained. In some embodiments, the service plans (e.g., by...) Figure 5 Data structures 502 and 504 (representing a service plan 902) specify the processes used to bootstrap services within a cloud computing environment. These bootstrap processes can be associated with ordered sets of building milestones and ordered sets of releasing milestones within the service plan. Figure 6 The 600th instance of a construction milestone provides a sample collection of construction milestones. Figure 7 The execution unit 702 depicts an ordered set of releases. In some embodiments, the service plan includes a list of at least one of a set of configuration files or a set of artifacts to be used to perform processes for bootstrapping the service (e.g., Figure 9 (Related to List 904).

[0210] In some embodiments, each build milestone in the ordered set of build milestones may be associated with a corresponding subset of the released ordered set. The first representation may include multiple nodes and multiple edges. Each of the multiple nodes may correspond to a build milestone in the ordered set of build milestones. Each of the multiple edges may represent a transition between two build milestones in the ordered set of build milestones. In some embodiments, the build milestones in the ordered set of build milestones may be associated with one or more external capability dependencies, which respectively indicate the capabilities expected to be released by the second service during a second processing phase for bootstrapping the second service within a cloud computing environment.

[0211] In some embodiments, the service plan also includes an execution unit that specifies the transition between two build milestones from an ordered set of build milestones (e.g., Figure 7 (Execution unit entity 700). An execution unit can associate transformations with a subset of an ordered set of releases. An execution unit can express one or more releases, at least in part, based on an execution target checkpoint transformation corresponding to the execution target checkpoint definition specified in the service plan. An execution unit is associated with: 1) external capability dependencies, which indicate dependencies on a first capability to be published by another service among a plurality of services; and 2) internal capabilities, which indicate dependencies on a second capability expected to be published by that service.

[0212] At 1204, a first representation can be generated that depicts an ordered set of build milestones associated with the processing used to bootstrap a service within a cloud computing environment. In some embodiments, the first representation may include a first set of nodes and a first set of edges, each node in the first set of nodes representing a corresponding build milestone in the ordered set of build milestones, and each edge in the first set of edges corresponding to at least one release in a subset of releases that, when executed, transition the service from a first build milestone in the ordered set of build milestones to a second build milestone in the ordered set of build milestones. In some embodiments, the first set of nodes in the first representation individually identifies one or more external capability dependencies on one or more capabilities associated with one or more other services.

[0213] At 1206, a second representation can be generated that depicts an ordered set of releases associated with the processing used to bootstrap the service within a cloud computing environment. In some embodiments, the second representation (e.g., a service DAG describing the internal processing of the bootstrap service) may include nodes and edges. Each node may correspond to a release, and each edge may correspond to at least one capability dependency between releases. In some embodiments, each node of the second representation may be associated with one or more internal capabilities that indicate a corresponding dependency on another capability expected to be published by the service.

[0214] At 1208, it can be found in the user interface (e.g., Figure 10 A first representation (e.g., a build graph) is provided on the UI 1000, which depicts an ordered set of build milestones associated with the processes used to bootstrap the service within a cloud computing environment. In some embodiments, the first representation represents the processes used to fully bootstrap the service within a cloud computing environment at a first granularity.

[0215] At 1210, it can be found in the user interface (e.g., Figure 10 A second representation (e.g., a service DAG) is provided on the UI 1100, which depicts an ordered set of releases associated with the processes used to bootstrap the service within a cloud computing environment. In some embodiments, the first representation depicts the processes used to bootstrap the service within a cloud computing environment at a first granularity, while the second representation depicts the processes used to bootstrap the service within a cloud computing environment at a second granularity. In some embodiments, the second granularity may be different from the first granularity.

[0216] In some embodiments, in response to receiving navigation options associated with the first representation, a second representation is provided at the user interface. For example, (e.g., when presenting the first representation (which presents a construction milestone)) (e.g., via...) Figure 11 The user interface element 1132) receives user input submitted at the user interface, indicating the selection of navigation options associated with the first representation, which allows... Figure 11 Region 1104 transitions from presenting a first representation (e.g., a higher-level overview of the processes used to build a service, a construction graph that indicates inter-service interactions / capabilities) to presenting a second representation (e.g., a lower-level view of the processes used to build a service, such as the service DAG of the service that indicates intra-service interactions / capabilities).

[0217] Example cloud service infrastructure architecture

[0218] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0219] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into the VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0220] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0221] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be started on demand), etc.

[0222] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0223] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0224] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provided to define how inbound / outbound traffic to the network and one or more virtual machines (VMs) will be configured. Other infrastructure elements, such as load balancers, databases, etc., may also be provided. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are desired.

[0225] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0226] Figure 13This is a block diagram 1300 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 1302 may be communicatively coupled to a secure host lease 1304, which may include a virtual cloud network (VCN) 1306 and a secure host subnet 1308. In some examples, service operator 1302 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device can be a workstation computer running any of the various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as Google Chrome OS). Alternatively or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without Kinect® gesture input), and / or a personal messaging device capable of communicating over a network that can access VCN 1306 and / or the internet.

[0227] VCN 1306 may include a local peering gateway (LPG) 1310, which may be communicatively coupled to SSH VCN 1312 via LPG 1310 contained in Secure Shell (SSH) VCN 1312. SSH VCN 1312 may include an SSH subnet 1314, and SSH VCN 1312 may be communicatively coupled to control plane VCN 1316 via LPG 1310 contained in control plane VCN 1316. Furthermore, SSH VCN 1312 may be communicatively coupled to data plane VCN 1318 via LPG 1310. Control plane VCN 1316 and data plane VCN 1318 may be contained in a service lease 1319 that may be owned and / or operated by an IaaS provider.

[0228] The control plane VCN 1316 may include a control plane demilitarized zone (DMZ) layer 1320 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 1320 may include one or more load balancer (LB) subnets 1322, a control plane application (app) layer 1324 that may include one or more application subnets 1326, and a control plane data layer 1328 that may include one or more database (DB) subnets 1330 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1322 contained in the control plane DMZ layer 1320 may be communicatively coupled to one or more application subnets 1326 contained in the control plane application layer 1324 and an Internet gateway 1334 that may be contained in the control plane VCN 1316. The application subnets 1326 may be communicatively coupled to one or more DB subnets 1330 contained in the control plane data layer 1328, as well as a service gateway 1336 and a Network Address Translation (NAT) gateway 1338. The control plane VCN 1316 may include the service gateway 1336 and the NAT gateway 1338.

[0229] The control plane VCN 1316 may include a data plane mirror application layer 1340, which may include one or more application subnets 1326. The one or more application subnets 1326 included in the data plane mirror application layer 1340 may include a virtual network interface controller (VNIC) 1342 capable of executing a compute instance 1344. The compute instance 1344 may communicatively couple the one or more application subnets 1326 of the data plane mirror application layer 1340 to the one or more application subnets 1326 that may be included in the data plane application layer 1346.

[0230] Data plane VCN 1318 may include data plane application layer 1346, data plane DMZ layer 1348, and data plane data layer 1350. Data plane DMZ layer 1348 may include one or more LB subnets 1322 communicatively coupled to one or more application subnets 1326 of data plane application layer 1346 and Internet gateway 1334 of data plane VCN 1318. One or more application subnets 1326 may be communicatively coupled to service gateway 1336 and NAT gateway 1338 of data plane VCN 1318. Data plane data layer 1350 may also include one or more DB subnets 1330 communicatively coupled to one or more application subnets 1326 of data plane application layer 1346.

[0231] The Internet gateway 1334 of control plane VCN 1316 and data plane VCN 1318 can be communicatively coupled to metadata management service 1352, which can be communicatively coupled to public Internet 1354. Public Internet 1354 can be communicatively coupled to NAT gateway 1338 of control plane VCN 1316 and data plane VCN 1318. Service gateway 1336 of control plane VCN 1316 and data plane VCN 1318 can be communicatively coupled to cloud service 1356.

[0232] In some examples, the service gateway 1336, either the control plane VCN 1316 or the data plane VCN 1318, can make application programming interface (API) calls to the cloud service 1356 without traversing the public internet 1354. API calls from the service gateway 1336 to the cloud service 1356 can be unidirectional: the service gateway 1336 can make API calls to the cloud service 1356, and the cloud service 1356 can send requested data to the service gateway 1336. However, the cloud service 1356 may not initiate API calls to the service gateway 1336.

[0233] In some examples, secure host lease 1304 can be directly connected to service lease 1319, which would otherwise be isolated. Secure host subnet 1308 can communicate with SSH subnet 1314 via LPG 1310, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 1308 to SSH subnet 1314 allows secure host subnet 1308 to access other entities within service lease 1319.

[0234] Control plane VCN 1316 may allow users of service lease 1319 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 1316 may be deployed or otherwise used in data plane VCN 1318. In some examples, control plane VCN 1316 may be isolated from data plane VCN 1318, and the data plane mirror application layer 1340 of control plane VCN 1316 may communicate with the data plane application layer 1346 of data plane VCN 1318 via VNIC 1342, which may be included in both the data plane mirror application layer 1340 and the data plane application layer 1346.

[0235] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1354, which can transmit requests to the metadata management service 1352. The metadata management service 1352 can transmit requests to the control plane VCN 1316 via internet gateway 1334. Requests can be received by one or more LB subnets 1322 contained in the control plane DMZ layer 1320. The LB subnets 1322 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 1326 contained in the control plane application layer 1324. If the request is validated and requires a call to the public internet 1354, the call to the public internet 1354 can be transmitted to a NAT gateway 1338 that can make calls to the public internet 1354. The request may expect stored metadata to be stored in one or more DB subnets 1330.

[0236] In some examples, the data plane mirroring application layer 1340 can facilitate direct communication between the control plane VCN 1316 and the data plane VCN 1318. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 1318. Through VNIC 1342, the control plane VCN 1316 can communicate directly with the resources contained in the data plane VCN 1318, and thus can perform configuration changes, updates, or other appropriate modifications to these resources.

[0237] In some embodiments, the control plane VCN 1316 and data plane VCN 1318 may be included in service lease 1319. In this case, the system's users or customers may not own or operate the control plane VCN 1316 or data plane VCN 1318. Alternatively, the IaaS provider may own or operate both the control plane VCN 1316 and data plane VCN 1318, and both planes may be included in service lease 1319. This embodiment can enable the isolation of networks that might prevent users or customers from interacting with resources from other users or customers. Furthermore, this embodiment can allow users or customers of the system to privately store databases without relying on the public Internet 1354, which may not have the desired level of threat prevention for storage.

[0238] In other embodiments, one or more LB subnets 1322 included in the control plane VCN 1316 may be configured to receive signals from the service gateway 1336. In this embodiment, the control plane VCN 1316 and the data plane VCN 1318 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1354. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on service lease 1319, which may be isolated from the public internet 1354.

[0239] Figure 14 This is a block diagram 1400 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1402 (e.g., Figure 13 The service provider 1302 can communicatively couple to the secure host lease 1404 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1404 may include a virtual cloud network (VCN) 1406 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1408 (e.g., Figure 13 The secure host subnet 1308). VCN 1406 may include a local peering gateway (LPG) 1410 (e.g., Figure 13 The LPG 1310, which can be communicatively coupled to the SSH VCN 1412 (e.g., LPG 1310 contained in the Secure Shell (SSH) VCN 1412) Figure 13 SSH VCN 1312). SSH VCN 1412 can include SSH subnet 1414 (e.g., Figure 13 SSH subnet 1322), and SSH VCN 1412 can be communicatively coupled to control plane VCN 1416 via LPG 1410 included in control plane VCN 1416 (e.g., Figure 13 Control plane VCN 1316). Control plane VCN 1416 may be included in service lease 1419 (e.g., Figure 13 In the service lease 1319), and the data plane VCN 1418 (e.g., Figure 13 The data plane VCN 1318 may be included in a customer lease 1421 that may be owned or operated by a user or customer of the system.

[0240] Control plane VCN 1416 may include control plane DMZ layer 1420 (e.g., Figure 13 The control plane DMZ layer 1320), which may include one or more LB subnets 1422 (e.g.,Figure 13 (one or more) LB subnets 1322), may include (one or more) application subnets 1426 (e.g., Figure 13 The control plane application layer 1424 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324 may include one or more database (DB) subnets 1430 (e.g., similar to...). Figure 13 The control plane data layer 1428 of (one or more) DB subnets 1330 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1422 contained in the control plane DMZ layer 1420 can be communicatively coupled to one or more application subnets 1426 contained in the control plane application layer 1424 and an Internet gateway 1434 that can be contained in the control plane VCN 1416 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1426 can communicatively couple to DB subnet(s) 1430 contained in control plane data layer 1428 and service gateway 1436 (e.g., Figure 13 Service gateway 1336) and Network Address Translation (NAT) gateway 1438 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0241] The control plane VCN 1416 may include a data plane mirror of the application layer 1440, which may contain one or more application subnets 1426 (e.g., Figure 13 The data plane mirror application layer 1340). One or more application subnets 1426 contained in the data plane mirror application layer 1440 may include computational instances 1444 (e.g., similar to...). Figure 13 The virtual network interface controller (VNIC) 1442 (e.g., VNIC 1342) of the computing instance 1344. The computing instance 1444 may facilitate the mirroring of the application subnet(s) 1426 of the application layer 1440 in the data plane and may be included in the application layer 1446 in the data plane (e.g., Figure 13 Communication between one or more application subnets 1426 in the data plane application layer 1346 via VNIC 1442 contained in the data plane mirror application layer 1440 and VNIC 1442 contained in the data plane application layer 1446.

[0242] The Internet gateway 1434, included in the control plane VCN 1416, can be communicatively coupled to the metadata management service 1452 (e.g.,Figure 13 Metadata management service 1352), which can communicatively couple to the public Internet 1454 (e.g., Figure 13 The public internet 1454 can communicatively couple to a NAT gateway 1438 contained in a control plane VCN 1416. The service gateway 1436 contained in the control plane VCN 1416 can communicatively couple to a cloud service 1456 (e.g., ...). Figure 13 Cloud services (1356).

[0243] In some examples, data plane VCN 1418 may be included in customer lease 1421. In this case, the IaaS provider may provide control plane VCN 1416 for each customer, and the IaaS provider may set up a unique compute instance 1444 for each customer, included in service lease 1419. Each compute instance 1444 may allow communication between control plane VCN 1416 included in service lease 1419 and data plane VCN 1418 included in customer lease 1421. Compute instance 1444 may allow resources provisioned in control plane VCN 1416 included in service lease 1419 to be deployed or otherwise used in data plane VCN 1418 included in customer lease 1421.

[0244] In other examples, an IaaS provider's customer may have a database residing in customer lease 1421. In this example, control plane VCN 1416 may include data plane mirror application layer 1440, which may include one or more application subnets 1426. Data plane mirror application layer 1440 may reside in data plane VCN 1418, but may not reside in data plane VCN 1418. That is, data plane mirror application layer 1440 may have access to customer lease 1421, but may not reside in data plane VCN 1418 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 1440 may be configured to invoke data plane VCN 1418, but may not be configured to invoke any entity contained in control plane VCN 1416. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 1416 in the data plane VCN 1418, and the data plane mirroring application layer 1440 can facilitate the customer's expected deployment or other use of resources.

[0245] In some embodiments, an IaaS provider's customer may apply filters to data plane VCN 1418. In this embodiment, the customer may determine what data plane VCN 1418 can access, and the customer may restrict access from data plane VCN 1418 to the public internet 1454. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1418's access to any external networks or databases. Applying filters and controls to data plane VCN 1418 contained in customer lease 1421 helps isolate data plane VCN 1418 from other customers and the public internet 1454.

[0246] In some embodiments, cloud service 1456 may be invoked by service gateway 1436 to access services that may not exist on public internet 1454, control plane VCN 1416, or data plane VCN 1418. The connection between cloud service 1456 and control plane VCN 1416 or data plane VCN 1418 may not be real-time or continuous. Cloud service 1456 may reside on different networks owned or operated by an IaaS provider. Cloud service 1456 may be configured to receive calls from service gateway 1436 and may be configured not to receive calls from public internet 1454. Some cloud services 1456 may be isolated from other cloud services 1456, and control plane VCN 1416 may be isolated from cloud services 1456 that may not be in the same region as control plane VCN 1416. For example, control plane VCN 1416 may be located in "Region 1," and cloud service "Deployment 13" may be located in both "Region 1" and "Region 2." If the service gateway 1436, contained in the control plane VCN 1416 located in region 1, makes a call to deployment 13, then that call can be transmitted to deployment 13 in region 1. In this example, the control plane VCN 1416 or deployment 13 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 13 in region 2.

[0247] Figure 13 This is a block diagram 1500 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1502 (e.g., Figure 13 The service provider 1302) can communicatively couple to the secure host lease 1504 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1504 may include a virtual cloud network (VCN) 1506 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1508 (e.g., Figure 13 The secure host subnet 1308). VCN 1506 can include LPG 1510 (e.g.,Figure 13 The LPG 1310), which can be communicatively coupled to the SSH VCN 1512 via the LPG 1510 included in the SSH VCN 1512 (e.g., Figure 13 SSH VCN 1312). SSH VCN 1512 can include SSH subnet 1514 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1512 can be communicatively coupled to control plane VCN 1516 via LPG 1510 included in control plane VCN 1516 (e.g., Figure 13 The control plane VCN 1316) and coupled to the data plane VCN 1518 via the LPG 1510 contained in the data plane VCN 1518 (e.g., Figure 13 Data plane 1318). Control plane VCN 1516 and data plane VCN 1518 may be included in service lease 1519 (e.g., Figure 13 In the service rental (1319).

[0248] The control plane VCN 1516 may include a subnet 1522 that may contain one or more load balancer (LB) subnets (e.g., Figure 13 The control plane DMZ layer 1520 of (one or more) LB subnets 1322) (e.g., Figure 13 The control plane DMZ layer 1320 may include one or more application subnets 1526 (e.g., similar to...). Figure 13 The control plane application layer 1524 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324), and may include (one or more) DB subnets 1530, and the control plane data layer 1528 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1522 contained in the control plane DMZ layer 1520 can be communicatively coupled to one or more application subnets 1526 contained in the control plane application layer 1524 and an Internet gateway 1534 that can be contained in the control plane VCN 1516 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1526 can communicatively couple to DB subnet(s) 1530 contained in control plane data layer 1528 and service gateway 1536 (e.g., Figure 13 The service gateway) and Network Address Translation (NAT) gateway 1538 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1516 may include the service gateway 1536 and the NAT gateway 1538.

[0249] Data plane VCN 1518 may include data plane application layer 1546 (e.g., Figure 13 Data plane application layer 1346), data plane DMZ layer 1548 (e.g., Figure 13 Data plane DMZ layer 1348), and data plane data layer 1550 (e.g., Figure 13 The data plane data layer 1350. The data plane DMZ layer 1548 may include one or more trusted application subnets 1560 and one or more untrusted application subnets 1562 that are communicatively coupled to the data plane application layer 1546, and one or more LB subnets 1522 of the Internet gateway 1534 contained in the data plane VCN 1518. The one or more trusted application subnets 1560 may be communicatively coupled to the service gateway 1536, the NAT gateway 1538 contained in the data plane VCN 1518, and one or more DB subnets 1530 contained in the data plane data layer 1550. The one or more untrusted application subnets 1562 may be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518 and the one or more DB subnets 1530 contained in the data plane data layer 1550. The data plane data layer 1550 may include one or more DB subnets 1530 that can be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518.

[0250] One or more untrusted application subnets 1562 may include one or more primary VNICs 1564(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1566(1)-(N). Each tenant VM 1566(1)-(N) may be communicatively coupled to a corresponding application subnet 1567(1)-(N) that may be contained in a corresponding container egress VCN 1568(1)-(N), which may be contained in a corresponding customer lease 1570(1)-(N). A corresponding secondary VNIC 1572(1)-(N) may facilitate communication between one or more untrusted application subnets 1562 contained in data plane VCN 1518 and application subnets contained in container egress VCN 1568(1)-(N). Each container exit VCN 1568(1)-(N) may include a NAT gateway 1538, which can communicatively couple to the public Internet 1554 (e.g., Figure 13 (Public Internet 1354).

[0251] Internet gateway 1534, contained in control plane VCN 1516 and data plane VCN 1518, can be communicatively coupled to metadata management service 1552 (e.g., Figure 13 A metadata management system 1352 is provided, which can communicatively couple to the public internet 1554. The public internet 1554 can communicatively couple to a NAT gateway 1538 contained in a control plane VCN 1516 and a data plane VCN 1518. A service gateway 1536 contained in a control plane VCN 1516 and a data plane VCN 1518 can communicatively couple to a cloud service 1556.

[0252] In some embodiments, the data plane VCN 1518 may be integrated with the customer lease 1570. Such integration may be useful or desired by the IaaS provider's customer in certain situations, such as when support may be expected during code execution. A customer may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer.

[0253] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality to be attached to data plane layer application 1546. The code running this functionality may execute in VMs 1566(1)-(N) and may not be configured to run anywhere else on data plane VCN 1518. Each VM 1566(1)-(N) may be connected to a customer lease 1570. The corresponding container 1571(1)-(N) contained in VMs 1566(1)-(N) may be configured to run the code. In this case, there may be dual isolation (e.g., container 1571(1)-(N) running the code, where container 1571(1)-(N) may be contained in at least one or more untrusted application subnets 1562 containing VMs 1566(1)-(N)), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1571(1)-(N) may be communicatively coupled to customer lease 1570 and may be configured to transmit or receive data from customer lease 1570. Containers 1571(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 1518. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of containers 1571(1)-(N).

[0254] In some embodiments, one or more trusted application subnets 1560 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1560 may be communicatively coupled to one or more database subnets 1530 and configured to perform CRUD operations in one or more database subnets 1530. One or more untrusted application subnets 1562 may be communicatively coupled to one or more database subnets 1530, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1530. Containers 1571(1)-(N) that may be contained in each customer's VM 1566(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1530.

[0255] In other embodiments, the control plane VCN 1516 and the data plane VCN 1518 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1516 and the data plane VCN 1518. However, communication can occur indirectly through at least one method. The LPG 1510 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1516 and the data plane VCN 1518. In another example, either the control plane VCN 1516 or the data plane VCN 1518 may invoke the cloud service 1556 via the service gateway 1536. For example, an invocation of the cloud service 1556 from the control plane VCN 1516 may include a request for a service that can communicate with the data plane VCN 1518.

[0256] Figure 13 This is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1602 (e.g., Figure 13 The service provider 1302) can communicatively couple to the secure host lease 1604 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1604 may include a virtual cloud network (VCN) 1606 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1608 (e.g., Figure 13 The secure host subnet 1308). VCN 1606 can include LPG 1610 (e.g., Figure 13 The LPG 1310), the LPG 1610 can be accessed via SSH VCN 1612 (e.g., LPG 1310), Figure 13The LPG 1610 in SSH VCN 1312 is communicatively coupled to SSH VCN 1612. SSH VCN 1612 may include SSH subnet 1614 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1612 can be communicatively coupled to control plane VCN 1616 via LPG 1610 included in control plane VCN 1616 (e.g., Figure 13 The control plane VCN 1316) and coupled to the data plane VCN 1618 via the LPG 1610 contained in the data plane VCN 1618 (e.g., Figure 13 Data plane 1318). Control plane VCN 1616 and data plane VCN 1618 may be included in service lease 1619 (e.g., Figure 13 In the service rental (1319).

[0257] The control plane VCN 1616 may include one or more LB subnets 1622 (e.g., Figure 13 The control plane DMZ layer 1620 of (one or more) LB subnets 1322) (e.g., Figure 13 The control plane DMZ layer 1320 may include one or more application subnets 1626 (e.g., Figure 13 The control plane application layer 1624 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324 may include (one or more) DB subnets 1630 (e.g., Figure 13 The control plane data layer 1628 of (one or more) DB subnets 1530 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1622 contained in the control plane DMZ layer 1620 can be communicatively coupled to one or more application subnets 1626 contained in the control plane application layer 1624 and an Internet gateway 1634 that can be contained in the control plane VCN 1616 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1626 can communicatively couple to DB subnet(s) 1630 contained in control plane data layer 1628 and service gateway 1636 (e.g., Figure 13 The service gateway) and Network Address Translation (NAT) gateway 1638 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0258] The data plane VCN 1618 may include the data plane application layer 1646 (e.g., Figure 13 Data plane application layer 1346), data plane DMZ layer 1648 (e.g., Figure 13 Data plane DMZ layer 1348), and data plane data layer 1650 (e.g., Figure 13 The data plane data layer 1350). The data plane DMZ layer 1648 may include one or more trusted application subnets 1660 that can be communicatively coupled to the data plane application layer 1646 (e.g., Figure 13 (one or more) trusted application subnets 1560) and (one or more) untrusted application subnets 1662 (e.g., Figure 13 The data plane includes one or more untrusted application subnets 1562 and one or more LB subnets 1622 of an Internet gateway 1634 contained in data plane VCN 1618. One or more trusted application subnets 1660 may communicatively couple to a service gateway 1636 contained in data plane VCN 1618, a NAT gateway 1638 contained in data plane VCN 1618, and one or more DB subnets 1630 contained in data plane data layer 1650. One or more untrusted application subnets 1662 may communicatively couple to a service gateway 1636 contained in data plane VCN 1618 and one or more DB subnets 1630 contained in data plane data layer 1650. Data plane data layer 1650 may include one or more DB subnets 1630 that may communicatively couple to a service gateway 1636 contained in data plane VCN 1618.

[0259] One or more untrusted application subnets 1662 may include a primary VNIC 1664(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1666(1)-(N) residing within one or more untrusted application subnets 1662. Each tenant VM 1666(1)-(N) may run code in a corresponding container 1667(1)-(N) and is communicatively coupled to an application subnet 1626 that may be contained in a data plane application layer 1646 contained in a container egress VCN 1668. A corresponding secondary VNIC 1672(1)-(N) may facilitate communication between one or more untrusted application subnets 1662 contained in a data plane VCN 1618 and the application subnets contained in a container egress VCN 1668. The container egress VCN may include a public internet 1654 (e.g., Figure 13 The public internet (1354) uses NAT gateway 1638.

[0260] Internet gateway 1634, contained in control plane VCN 1616 and data plane VCN 1618, can be communicatively coupled to metadata management service 1652 (e.g., Figure 13 A metadata management system 1352 is provided, which can communicatively couple to the public internet 1654. The public internet 1654 can communicatively couple to a NAT gateway 1638 contained in a control plane VCN 1616 and a data plane VCN 1618. A service gateway 1636 contained in a control plane VCN 1616 and a data plane VCN 1618 can communicatively couple to a cloud service 1656.

[0261] In some examples, Figure 13 The architecture shown in the block diagram 1600 can be considered as... Figure 13 This is an exception to the pattern shown in the architecture of block diagram 1500, and this pattern may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1667(1)-(N) contained in each customer's VM 1666(1)-(N). Container 1667(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 1672(1)-(N) contained in one or more application subnets 1626 of the data plane application layer 1646, which may be contained in a container egress VCN 1668. The auxiliary VNIC 1672(1)-(N) can transmit the call to a NAT gateway 1638, which can then transmit the call to the public internet 1654. In this example, containers 1667(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1616 and from other entities contained in the data plane VCN 1618. Containers 1667(1)-(N) can also be isolated from resources from other clients.

[0262] In other examples, a client can use containers 1667(1)-(N) to invoke cloud service 1656. In this example, the client can run code within containers 1667(1)-(N) requesting a service from cloud service 1656. Container 1667(1)-(N) can then forward the request to a secondary VNIC 1672(1)-(N), which can then forward the request to a NAT gateway, which can forward the request to the public internet 1654. The public internet 1654 can then forward the request via internet gateway 1634 to one or more LB subnets 1622 contained in control plane VCN 1616. In response to determining that the request is valid, one or more LB subnets can forward the request to one or more application subnets 1626, which can then forward the request to cloud service 1656 via service gateway 1636.

[0263] It should be recognized that the IaaS architectures 1300, 1400, 1500, and 1600 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0264] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0265] Figure 13 An example computer system 1700, in which various embodiments can be implemented, is illustrated. System 1700 can be used to implement any of the computer systems described above. As shown, computer system 1700 includes a processing unit 1704 that communicates with a plurality of peripheral subsystems via a bus subsystem 1702. These peripheral subsystems may include a processing acceleration unit 1706, an I / O subsystem 1708, a storage subsystem 1718, and a communication subsystem 1724. Storage subsystem 1718 includes a tangible computer-readable storage medium 1722 and system memory 1710.

[0266] Bus subsystem 1702 provides a mechanism for allowing various components and subsystems of computer system 1700 to communicate with each other as intended. While bus subsystem 1702 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1702 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0267] A processing unit 1704, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1700. One or more processors may be included in the processing unit 1704. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1704 may be implemented as one or more independent processing units 1732 and / or 1734, each including a single-core or multi-core processor. In other embodiments, the processing unit 1704 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0268] In various embodiments, processing unit 1704 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1704 and / or storage subsystem 1718. With appropriate programming, processor(s) 1704 can provide the various functions described above. Computer system 1700 may additionally include processing acceleration unit 1706, which may include digital signal processor (DSP), dedicated processor, etc.

[0269] I / O subsystem 1708 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, keyboards, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye gesture into input in an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0270] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0271] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1700 to a user or other computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0272] Computer system 1700 may include storage subsystem 1718, which provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality described in the embodiments of this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 1704, provide the aforementioned functionality. Storage subsystem 1718 may also provide a repository for storing data used according to this disclosure.

[0273] like Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure 13 Figure As illustrated in the example, storage subsystem 1718 may include various components, including system memory 1710, computer-readable storage medium 1722, and computer-readable storage medium reader 1720. System memory 1710 may store program instructions that can be loaded and executed by processing unit 1704. System memory 1710 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1710, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0274] System memory 1710 may also store operating system 1716. Examples of operating system 1716 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations where computer system 1700 executes one or more virtual machines, the virtual machine and its guest operating system (GOS) may be loaded into system memory 1710 and executed by the cores of one or more processors or processing units 1704.

[0275] System memory 1710 can have different configurations depending on the type of computer system 1700. For example, system memory 1710 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, system memory 1710 may include a basic input / output system (BIOS), which contains basic routines that facilitate the transfer of information between components within the computer system 1700, such as during startup.

[0276] Computer-readable storage medium 1722 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information for use by computer system 1700, including instructions executable by processing unit 1704 of computer system 1700.

[0277] Computer-readable storage medium 1722 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for the storage and / or transmission of information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or other tangible computer-readable media.

[0278] For example, computer-readable storage medium 1722 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, disk drives that read from or write to removable non-volatile magnetic disks, and optical disc drives that read from or write to removable non-volatile optical discs (such as CD ROMs, DVDs, and Blu-ray® discs or other optical media). Computer-readable storage medium 1722 may include, but is not limited to: Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and so on. Computer-readable storage medium 1722 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for computer system 1700.

[0279] Machine-readable instructions executable by one or more processors or cores of the processing unit 1704 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0280] The communication subsystem 1724 provides an interface to other computer systems and networks. The communication subsystem 1724 serves as an interface for receiving data from other systems and sending data from computer system 1700 to other systems. For example, the communication subsystem 1724 enables computer system 1700 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1724 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1724 may provide a wired network connection (e.g., Ethernet).

[0281] In some embodiments, the communication subsystem 1724 may also represent one or more users who can use the computer system 1700 to receive input communications in the form of structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc.

[0282] For example, the communication subsystem 1724 can be configured to receive data feeds 1726 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0283] Furthermore, the communication subsystem 1724 can also be configured to receive data in the form of a continuous data stream, which may include event streams 1728 and / or event updates 1730 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quotation machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0284] The communication subsystem 1724 can also be configured to output structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1700.

[0285] The computer system 1700 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing system.

[0286] Due to the constantly evolving nature of computers and networks, the description of the computer system 1700 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including application applets) or combinations thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0287] The embodiments may be implemented using a computer program product comprising a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure.

[0288] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0289] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0290] Accordingly, the specification and drawings are intended to be illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0291] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as encompassing both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0292] Exclusion language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such exclusion language is generally not intended to, and should not, imply that some embodiments require at least one of X, at least one of Y, or at least one of Z to be present individually.

[0293] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0294] Exemplary embodiments of this disclosure may be described in accordance with the following terms.

[0295] Clause 1. A computer-implemented method is disclosed. This computer-implemented method may include a service plan maintained by a cloud infrastructure orchestration system corresponding to services among a plurality of services to be bootstrapped within a cloud computing environment. In some embodiments, the service plan specifies processes for bootstrapping the services within the cloud computing environment. The processes for bootstrapping the services may be associated within the service plan with an ordered set of build milestones and an ordered set of releases. The computer-implemented method may include generating a first representation by the cloud infrastructure orchestration system depicting the ordered set of build milestones associated with the processes for bootstrapping the services within the cloud computing environment. The computer-implemented method may include generating a second representation by the cloud infrastructure orchestration system depicting the ordered set of releases associated with the processes for bootstrapping the services within the cloud computing environment. The computer-implemented method may include providing, at a user interface, the first representation depicting the ordered set of build milestones associated with the processes for bootstrapping the services within the cloud computing environment. The computer-implemented method may include providing, at a user interface, the second representation depicting the ordered set of releases associated with the processes for bootstrapping the services within the cloud computing environment.

[0296] Clause 2. The computer-implemented method as described in Clause 1, wherein each construction milestone in the ordered set of construction milestones is associated with a corresponding subset of the ordered set of release milestones.

[0297] Clause 3. A computer-implemented method as described in Clause 1 or 2, wherein a first represents a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes corresponds to a construction milestone in an ordered set of construction milestones, and wherein each of the plurality of edges represents a transition between two construction milestones in the ordered set of construction milestones.

[0298] Clause 4. A computer-implemented method as described in any of Clauses 1 to 3, wherein each of the ordered sets of construction milestones is associated with one or more external capability dependencies, the one or more external capability dependencies indicating dependencies on capabilities expected to be published by the second service during a second process for bootstrapping the second service within a cloud computing environment.

[0299] Clause 5. A computer-implemented method as described in any one of Clauses 1 to 4, wherein the service plan further includes an execution unit that specifies a transition between two build milestones in an ordered set of build milestones, the execution unit associating the transition with a subset of the released ordered set.

[0300] Clause 6. A computer-implemented method as described in Clause 5, wherein the execution unit is associated with: 1) an external capability dependency indicating a dependency on a first capability to be published by another service of the plurality of services; and 2) an internal capability indicating a dependency on a second capability expected to be published by the service.

[0301] Clause 7. A computer-implemented method as described in any one of Clauses 1 to 6, wherein the first indicates a first granularity describing the process for booting a service within a cloud computing environment, and wherein the second indicates a second granularity describing the process for booting a service within a cloud computing environment, the second granularity being different from the first granularity.

[0302] Clause 8. A cloud infrastructure orchestration system (“System”) is disclosed. The System may include one or more processors and one or more memories storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations. Operations may include maintaining service plans corresponding to services among a plurality of services to be bootstrapped within a cloud computing environment. In some embodiments, the service plan specifies processes for bootstrapping services within the cloud computing environment. Processes for bootstrapping services may be associated with ordered sets of build milestones and ordered sets of releases within the service plan. Operations may include generating a first representation depicting the ordered sets of build milestones associated with the processes for bootstrapping services within the cloud computing environment. Operations may include generating a second representation depicting the ordered sets of releases associated with the processes for bootstrapping services within the cloud computing environment. Operations may include providing at a user interface the first representation depicting the ordered sets of build milestones associated with the processes for bootstrapping services within the cloud computing environment. Operations may include providing at a user interface the second representation depicting the ordered sets of releases associated with the processes for bootstrapping services within the cloud computing environment.

[0303] Clause 9. A cloud infrastructure orchestration system as described in Clause 8, wherein each construction milestone in an ordered set of construction milestones is associated with a corresponding subset of an ordered set of release milestones.

[0304] Clause 10. A cloud infrastructure orchestration system as described in Clause 8 or 9, wherein a first represents a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes corresponds to a construction milestone in an ordered set of construction milestones, and wherein each of the plurality of edges represents a transition between two construction milestones in the ordered set of construction milestones.

[0305] Clause 11. A cloud infrastructure orchestration system as described in any of Clauses 8 to 10, wherein each of the ordered sets of construction milestones is associated with one or more external capability dependencies, the one or more external capability dependencies indicating dependencies on capabilities expected to be published by the second service during a second processing for bootstrapping the second service within a cloud computing environment.

[0306] Clause 12. A cloud infrastructure orchestration system as described in any of Clauses 8 to 11, wherein the service plan further includes an execution unit for a transition between two build milestones in an ordered set of designated build milestones, the execution unit associating the transition with a subset of the released ordered set.

[0307] Clause 13. A cloud infrastructure orchestration system as described in Clause 12, wherein the execution unit is associated with: 1) external capability dependencies, which indicate dependencies on a first capability to be published by another service among the plurality of services; and 2) internal capabilities, which indicate dependencies on a second capability to be published by the service.

[0308] Clause 14. A cloud infrastructure orchestration system as described in any one of Clauses 8 to 13, wherein "first" indicates a first granularity describing the processing for bootstrapping services within a cloud computing environment, and wherein "second" indicates a second granularity describing the processing for bootstrapping services within a cloud computing environment, the second granularity being different from the first granularity.

[0309] Clause 15. A non-transitory computer-readable medium is disclosed. This non-transitory computer-readable medium may store computer-executable instructions that, when executed by one or more processors of a cloud infrastructure orchestration system, cause the one or more processors of the cloud infrastructure orchestration system to perform operations. Operations may include maintaining a service plan corresponding to a service among a plurality of services to be bootstrapped within a cloud computing environment. In some embodiments, the service plan specifies processing for bootstrapping the services within the cloud computing environment. Processes for bootstrapping the services may be associated within the service plan with an ordered set of build milestones and an ordered set of releases. Operations may include generating a first representation depicting the ordered set of build milestones associated with the processing for bootstrapping the services within the cloud computing environment. Operations may include generating a second representation depicting the ordered set of releases associated with the processing for bootstrapping the services within the cloud computing environment. Operations may include providing at a user interface the first representation depicting the ordered set of build milestones associated with the processing for bootstrapping the services within the cloud computing environment. Operations may include providing at a user interface the second representation depicting the ordered set of releases associated with the processing for bootstrapping the services within the cloud computing environment.

[0310] Clause 16. A non-transitory computer-readable medium as described in Clause 15, wherein each construction milestone in an ordered set of construction milestones is associated with a corresponding subset of an ordered set of release milestones.

[0311] Clause 17. A nontransitory computer-readable medium as described in Clause 15 or 16, wherein the first representation includes a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes corresponds to a construction milestone in an ordered set of construction milestones, and wherein each of the plurality of edges represents a transition between two construction milestones in the ordered set of construction milestones.

[0312] Clause 18. A nontransitory computer-readable medium as described in any of Clauses 15 to 17, wherein each of the ordered sets of construction milestones is associated with one or more external capability dependencies, the one or more external capability dependencies indicating dependencies on capabilities expected to be published by the second service during a second processing for bootstrapping the second service within a cloud computing environment.

[0313] Clause 19. A nontransitory computer-readable medium as described in any one of Clauses 15 to 18, wherein the service plan further includes an execution unit for a transition between two build milestones in an ordered set of specified build milestones, the execution unit associating the transition with a subset of the released ordered set.

[0314] Clause 20. A cloud infrastructure orchestration system as described in Clause 19, wherein an execution unit is associated with: 1) an external capability dependency indicating a dependency on a first capability to be published by another of the plurality of services; and 2) an internal capability indicating a dependency on a second capability to be published by the service.

[0315] All references cited in this article (including publications, patent applications and patents) are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0316] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A computer-implemented method, comprising: A service plan is maintained by the cloud infrastructure orchestration system and corresponds to a service among multiple services to be bootstrapped in the cloud computing environment. The service plan specifies the processing for bootstrapping the services in the cloud computing environment. The processing for bootstrapping the services is associated with an ordered set of building milestones and an ordered set of releasing milestones within the service plan. A first representation is generated by the cloud infrastructure orchestration system, which depicts an ordered set of build milestones associated with the processing used to bootstrap services within a cloud computing environment; A second representation is generated by the cloud infrastructure orchestration system, which depicts an ordered set of releases associated with the processing used to bootstrap services within the cloud computing environment; The cloud infrastructure orchestration system provides a first representation at the user interface of an ordered set of building milestones associated with the processes used to bootstrap services within a cloud computing environment; as well as The cloud infrastructure orchestration system provides a second representation at the user interface that depicts an ordered set of releases associated with the processing used to bootstrap services within a cloud computing environment.

2. The computer-implemented method of claim 1, wherein each construction milestone in the ordered set of construction milestones is associated with a corresponding subset of the ordered set of release milestones.

3. The computer-implemented method of claim 1 or 2, wherein the first represents a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes corresponds to a construction milestone in an ordered set of construction milestones, and wherein each of the plurality of edges represents a transition between two construction milestones in the ordered set of construction milestones.

4. The computer-implemented method of any one of claims 1 to 3, wherein each construction milestone in an ordered set of construction milestones is associated with one or more external capability dependencies, the one or more external capability dependencies indicating dependencies on capabilities expected to be published by the second service during a second process for bootstrapping the second service within a cloud computing environment.

5. The computer-implemented method of any one of claims 1 to 4, wherein the service plan further includes an execution unit that specifies a transition between two construction milestones in an ordered set of construction milestones, the execution unit associating the transition with a subset of the released ordered set.

6. The computer-implemented method of claim 5, wherein the execution unit is associated with: 1) an external capability dependency indicating a dependency on a first capability to be published by another service of the plurality of services; and 2) an internal capability indicating a dependency on a second capability expected to be published by the service.

7. The computer-implemented method of any one of claims 1 to 6, wherein the first indicates a first granularity describing the process for booting services within a cloud computing environment, and wherein the second indicates a second granularity describing the process for booting services within a cloud computing environment, the second granularity being different from the first granularity.

8. A cloud infrastructure orchestration system, comprising: One or more processors; as well as One or more memories store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: Maintain service plans corresponding to services among multiple services to be bootstrapped within the cloud computing environment. The service plans specify the processes used to bootstrap the services within the cloud computing environment. The processes used to bootstrap the services are associated with ordered sets of building milestones and ordered sets of releasing milestones within the service plans. Generate a first representation, which depicts an ordered set of build milestones associated with the processes used to bootstrap services within a cloud computing environment; Generate a second representation that depicts an ordered set of releases associated with the processing used to bootstrap services within a cloud computing environment; Provide a first representation at the user interface of an ordered set of build milestones associated with the processes used to bootstrap services within a cloud computing environment; as well as Provide a second representation at the user interface that depicts an ordered set of releases associated with the processing used to bootstrap services within a cloud computing environment.

9. The cloud infrastructure orchestration system of claim 8, wherein each construction milestone in the ordered set of construction milestones is associated with a corresponding subset of the ordered set of release milestones.

10. The cloud infrastructure orchestration system of claim 8 or 9, wherein the first represents a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes corresponds to a construction milestone in an ordered set of construction milestones, and wherein each of the plurality of edges represents a transition between two construction milestones in the ordered set of construction milestones.

11. The cloud infrastructure orchestration system of any one of claims 8 to 10, wherein each of the ordered set of construction milestones is associated with one or more external capability dependencies, the one or more external capability dependencies indicating dependencies on capabilities expected to be published by the second service during a second processing for bootstrapping the second service within the cloud computing environment.

12. The cloud infrastructure orchestration system of any one of claims 8 to 11, wherein the service plan further includes an execution unit for a transition between two build milestones in an ordered set of build milestones, the execution unit associating the transition with a subset of the released ordered set.

13. The cloud infrastructure orchestration system of claim 12, wherein the execution unit is associated with: 1) external capability dependencies, which indicate dependencies on a first capability to be published by another service among the plurality of services; and 2) internal capabilities, which indicate dependencies on a second capability to be published by the service.

14. The cloud infrastructure orchestration system of any one of claims 8 to 13, wherein the first indicates a first granularity describing the processing for bootstrapping services within a cloud computing environment, and wherein the second indicates a second granularity describing the processing for bootstrapping services within a cloud computing environment, the second granularity being different from the first granularity.

15. A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors of a cloud infrastructure orchestration system, cause the one or more processors of the cloud infrastructure orchestration system to perform the following operations: Maintain service plans corresponding to services among multiple services to be bootstrapped within the cloud computing environment. The service plans specify the processes used to bootstrap the services within the cloud computing environment. The processes used to bootstrap the services are associated with ordered sets of building milestones and ordered sets of releasing milestones within the service plans. Generate a first representation, which depicts an ordered set of build milestones associated with the processes used to bootstrap services within a cloud computing environment; Generate a second representation that depicts an ordered set of releases associated with the processing used to bootstrap services within a cloud computing environment; Provide a first representation at the user interface of an ordered set of build milestones associated with the processes used to bootstrap services within a cloud computing environment; as well as Provide a second representation at the user interface that depicts an ordered set of releases associated with the processing used to bootstrap services within a cloud computing environment.

16. The non-transitory computer-readable medium of claim 15, wherein each construction milestone in the ordered set of construction milestones is associated with a corresponding subset of the ordered set of release milestones.

17. The non-transitory computer-readable medium of claim 15 or 16, wherein the first representation includes a plurality of nodes and a plurality of edges, wherein each of the plurality of nodes corresponds to a construction milestone in an ordered set of construction milestones, and wherein each of the plurality of edges represents a transition between two construction milestones in the ordered set of construction milestones.

18. The non-transitory computer-readable medium of any one of claims 15 to 17, wherein each construction milestone in an ordered set of construction milestones is associated with one or more external capability dependencies, the one or more external capability dependencies indicating dependencies on capabilities expected to be published by the second service during a second processing for bootstrapping the second service within a cloud computing environment.

19. The non-transitory computer-readable medium of any one of claims 15 to 18, wherein the service plan further includes an execution unit for a transition between two construction milestones in an ordered set of construction milestones, the execution unit associating the transition with a subset of the released ordered set.

20. The cloud infrastructure orchestration system of claim 19, wherein the execution unit is associated with: 1) external capability dependencies, which indicate dependencies on a first capability to be published by another service among the plurality of services; and 2) internal capabilities, which indicate dependencies on a second capability to be published by the service.

Images