Re-entry Service Deployment
CIOS automates data center construction by using predefined configuration files and dependency management to streamline the provisioning and deployment process, enhancing efficiency and scalability in cloud infrastructure services.
Patent Information
- Application Number
- JP2024547129
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-01-18
- Filing Date
- 2023-02-02
- Publication Date
- 2026-11-19
- Estimated Expiration
- 2043-02-02
AI Technical Summary
Building data centers in cloud infrastructure services is a complex, time-consuming, and error-prone process that requires significant manual effort, limiting the ability to scale computing resources in response to growing customer needs.
The Cloud Infrastructure Organization Service (CIOS) automates the provisioning and deployment of resources and infrastructure for data centers using predefined configuration files, identifies dependencies, and manages the order of operations to minimize manual adjustments and reduce construction time.
CIOS significantly reduces the time required to build data centers and minimizes manual effort, enabling more efficient and scalable region construction by ensuring tasks are executed only when dependent resources are available, thus reducing the risk of errors.
Smart Images

Figure 0007954213000001 
Figure 0007954213000002 
Figure 0007954213000003
Abstract
Description
TECHNICAL FIELD
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This non-provisional application claims priority to U.S. Provisional Patent Application No. 63 / 308,003 filed on February 8, 2022 entitled "Techniques for Bootstrapping a Region Build", U.S. Provisional Patent Application No. 63 / 312,814 filed on February 22, 2022 entitled "Techniques for Implementing Virtual Data Centers", U.S. Provisional Patent Application No. 63 / 314,927 filed on February 28, 2022 entitled "Reentrant Service Deployments", and U.S. Patent Application No. 18 / 098,617 filed on January 18, 2023 entitled "Reentrant Service Deployments", the disclosures of all of which are incorporated herein by reference in their entireties.
[0002] FIELD OF THE INVENTION The present disclosure relates to techniques for constructing data centers in cloud computing environments. More specifically, techniques for creation, provisioning, and deployment of resources and infrastructure for constructing a data center configured to provide a set of services are described. BACKGROUND ART
[0003] Background Today, cloud infrastructure services build data centers using many individual services (for example, bootstrapping various resources in data centers within a specific geographical area). In some examples, a region is a logical abstraction corresponding to a local geographical area where one or more data centers are located (or are located). Building a data center can involve provisioning and configuring infrastructure resources, as well as deploying code (for example, for various services) to these resources. The actions required to build a data center are sometimes collectively referred to as performing a “region build.” Since a region can contain any number of data centers, a region build can involve actions to build multiple data centers. Traditional tools for building data centers require considerable manual effort. Furthermore, the bootstrap operation of a service may depend on other features and / or services in a region that may not be available. As the number of service teams and regions increases, the tasks performed to organize provisioning and deployment increase significantly. Relying heavily on manual effort for bootstrapping services and / or building data centers is time-consuming, risky, and may not scale well. [Overview of the Initiative]
[0004] overview This disclosure generally relates to the construction of a data center. More specifically, it describes techniques for creating, provisioning, and deploying resources and infrastructure for constructing a data center configured to provide a set of services. Techniques are provided for scheduling and executing multiple releases of the services. For a service in which releases have been executed in the past, a second release may be scheduled for the service in response to optional capability dependencies of the service being issued (available) in the data center. Various embodiments are described herein, including methods, systems, non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors.
[0005] In one embodiment, a first release of the flock is scheduled and executed upon determination that all necessary capability dependencies for the flock for the service have been issued. After the first release is executed, a second release of the flock is scheduled upon determination that new capabilities have been issued in the data center and identification that the newly issued capabilities are optional capability dependencies for the flock for the service. The second release is then executed. The newly issued capabilities identify the functional unit associated with the flock for the service and identify a set of resources associated with the service.
[0006] Required capability dependencies for a service flock are identified as capabilities issued in the data center before the first release of the flock is scheduled and executed. Optional capability dependencies for a service flock are identified as capabilities that do not need to be issued in the data center before the first release of the flock is scheduled and executed.
[0007] In one example, a service flock includes a set of one or more resources for providing the service. This set of resources includes infrastructure components and artifacts associated with the service. In another example, information identifying the service flock is declared in the service flock configuration. The service flock configuration identifies a set of one or more resources associated with the service.
[0008] In one example, a dependency graph is obtained, and the dependency graph is used to identify newly issued capabilities as optional capability dependencies of a flock for a service. In one example, the dependency graph is an acyclic dependency graph.
[0009] In one example, executing a second release involves generating a plan. The plan identifies the current configuration and target configuration of the data center associated with the execution of the second release. The plan is then executed. In one example, executing the plan results in the issuance of additional new capabilities in the data center. In another example, in response to executing the plan, the metadata associated with the service flock configuration is updated.
[0010] In one example, a cloud infrastructure organization system is disclosed. This system determines that a new capability has been issued in the data center, identifies that the newly issued capability is an optional capability dependency of a flock for a service, and that the first release of the flock has been scheduled and executed in the past. The system then schedules a second release of the flock. In one example, the system adds the second release of the flock to a queue and scans the queue to determine if one or more unexecuted releases of the flock exist in the queue. At least one of the unexecuted releases may contain the second release of the flock. In response to this determination, the system then merges one or more unexecuted releases of the flock in the queue to generate a merged release of the flock. The merged release of the flock is then executed. In one example, the queue is a general-purpose scheduling queue shared by multiple services provided by the data center. In another example, the queue is a flock-specific queue for a flock for a service.
[0011] The above will become more apparent in conjunction with other features and embodiments by referring to the following specification, claims, and accompanying drawings.
[0012] To facilitate the identification of discussions of any particular element or behavior, the leading one or more digits of a reference number represent the figure number in which that element was first introduced. [Brief explanation of the drawing]
[0013] [Figure 1] This is a block diagram of an environment in which a Cloud Infrastructure Organization Service (CIOS) can operate to dynamically provide bootstrap services in a region, according to at least one embodiment. [Figure 2] This is a block diagram showing an environment and method for constructing a virtual bootstrap environment (ViBE) according to at least one embodiment. [Figure 3]This block diagram shows an environment and method for bootstrapping a service to a target region using ViBE, according to at least one embodiment. [Figure 4] This is a simplified flowchart illustrating the process performed to schedule a re-entry release according to one embodiment. [Figure 5] This is an illustrative diagram of a scheduling queue implemented by a multi-flock orchestrator (MFO) as shown in Figure 1, according to one embodiment. [Figure 6] This is an illustrative diagram of an integrated release in a scheduling queue implemented by a multi-flock orchestrator (MFO) as shown in Figure 1, according to one embodiment. [Figure 7] This is a simplified flowchart illustrating the process performed to generate and execute an integrated release according to one embodiment. [Figure 8] This is a block diagram showing a pattern for implementing a service-type cloud infrastructure system according to at least one embodiment. [Figure 9] This block diagram shows another pattern for implementing a service-type cloud infrastructure system according to at least one embodiment. [Figure 10] This block diagram shows another pattern for implementing a service-type cloud infrastructure system according to at least one embodiment. [Figure 11] This block diagram shows another pattern for implementing a service-type cloud infrastructure system according to at least one embodiment. [Figure 12] A block diagram showing an exemplary computer system according to at least one embodiment. [Modes for carrying out the invention]
[0014] Detailed explanation In the following description, specific details are provided for illustrative purposes to enable a complete understanding of certain embodiments. However, it will become clear that various embodiments can be realized without these specific details. The drawings and this specification are not intended to be limiting in any way. In this specification, the use of the word “exemplary” means “serving as an example, case, or illustration.” No embodiment or design described as “exemplary” in this specification shall necessarily be construed as being preferable or advantageous to other embodiments or designs.
[0015] In recent years, the adoption of cloud services has been rapidly increasing. Currently, various types of cloud services are offered by different cloud service providers (CSPs). The term "cloud service" is generally used to describe services or functions that are made available on demand (for example, through a subscription model) by a CSP through the use of the CSP's systems and infrastructure (cloud infrastructure). Typically, the servers and systems that make up the CSP's infrastructure and are used to deliver cloud services to customers are separate from the customer's own on-premises servers and systems. Therefore, customers can use cloud services provided by a CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide registered customers with easy, scalable, and on-demand access to applications and computing resources without requiring the customer to invest in procuring the infrastructure used to deliver the service or function. Various different types or models of cloud services may be offered, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). Customers can subscribe to one or more cloud services offered by a CSP. Customers can be any entity, including individuals, organizations, and corporations.
[0016] As described above, a CSP is responsible for providing infrastructure and resources used to provide cloud services to registered customers. The resources provided by the CSP may include both hardware and software resources. These resources may include, for example, compute resources (e.g., virtual machines, containers, applications, processors), memory resources (e.g., databases, data stores), networking resources (e.g., routers, host machines, load balancers), identities, and other resources. In some embodiments, the resources provided by the CSP to provide a set of cloud services CSP (cloud services CSP) are organized as a data center. A data center may be configured to provide a specific set of cloud services. The CSP is responsible for installing the infrastructure and resources used to provide the specific set of cloud services in the data center. The CSP may build one or more data centers.
[0017] The data centers provided by the CSP may be hosted in different regions. A region is a local geographic area that can be identified by a region name. Regions are generally independent of each other and may be separated by long distances, for example, across countries or continents. Regions are grouped as realms. Examples of CSP regions include US West, US East, Australia East, Australia Southeast, and the like.
[0018] A region may include one or more data centers, and these data centers are located within a certain geographical area corresponding to the region. As an example, the data centers in a region may be located in a city within that region. For instance, for a specific CSP, the data centers in the US West region may be located in San Jose, California, the data centers in the US East region may be located in Ashburn, Virginia, the data centers in the Australia East region may be located in Sydney, Australia, the data centers in the Australia Southeast region may be located in Melbourne, Australia, and so on.
[0019] Data centers within a region may be organized as one or more availability domains used for high availability and disaster recovery. An availability domain may include one or more data centers within the region. The availability domains within a region are isolated from each other, fault-tolerant, and designed such that the possibility of simultaneous failures occurring in data centers across multiple availability domains is extremely low. For example, the availability domains within a region may be structured such that a failure in one availability domain within the region is unlikely to affect the availability of data centers in other availability domains within the same region.
[0020] When a customer or subscriber registers or signs up for one or more services provided by a CSP, the CSP creates a tenancy for that customer. A tenancy is similar to an account created for the customer. In some embodiments, a tenancy for a customer resides in a single realm and has access to all regions belonging to that realm. Then, the customer's users can access the services that the customer has registered for under this tenancy.
[0021] As mentioned above, CSPs provide cloud services to their customers by building or deploying data centers. As a CSP's customer base expands, CSPs typically build new data centers in new regions or increase the capacity of existing data centers to meet and better serve their customers' growing demands. It is preferable that data centers be built geographically close to the locations of the customers they serve. Geographic proximity between a data center and the customers it serves helps in more efficient use of resources and in providing customers with faster and more reliable service. Therefore, CSPs typically build new data centers in new regions in geographical areas that are geographically close to the customers they serve. For example, if a CSP's customer base is expanding in Germany, it may build one or more data centers in a new region in Germany.
[0022] The act of building one or more data centers in a region is sometimes referred to as a region build. The term "region build" is used to describe the construction of one or more data centers in a region. Building a data center in a region involves provisioning or creating a set of new resources required or used to provide a set of services that the data center is configured to provide. The final result of the region build process is the creation of a data center in the region that is capable of providing the set of services intended for the data center and contains the set of resources used to provide those services.
[0023] Building a new data center in a region is an extremely complex task requiring extensive coordination between various bootstrap operations. At a high level, this involves performing and coordinating various tasks such as identifying a set of services to be provided by the data center, identifying the various resources required to provide those services, creating, provisioning, and deploying the identified resources, and properly describing the resources to enable their intended use. Each of these tasks has further subtasks that require coordination, further increasing the complexity. Due to this complexity, building a data center in a region currently involves multiple manual initiation or manual control tasks that require careful manual coordination. As a result, the task of building a new region (building one or more data centers in a region) is very time-consuming. Building a data center can take months, for example. Furthermore, the process is highly error-prone and may require multiple iterations before the desired configuration of the data center is achieved, further increasing the time required for data center construction. These constraints and problems severely limit the CSP's ability to scale computing resources in a timely manner in response to growing customer needs.
[0024] Technical effects This disclosure describes techniques for reducing the time and manual effort required to build one or more data centers in a region. The techniques disclosed significantly reduce the time required to build data centers in a region and minimize the manual adjustments needed. Previously, building a data center in a region could take weeks or months, but by using the techniques described herein, new data centers can be built in a region in a relatively much shorter time.
[0025] This specification discloses a Cloud Infrastructure Organization Service (CIOS) configured to bootstrap services to a new data center (e.g., provision and deploy) based on a predefined configuration file that identifies resources (e.g., infrastructure components and software to be deployed) for implementing a given change to a data center. The CIOS can identify dependencies between resources, execution targets, phases, and flocks by parsing and analyzing the configuration file (e.g., flock configuration). The CIOS may generate specific data structures through analysis, and may use these data structures to drive operations and manage the order in which services are bootstrapped to a region. The CIOS may use these data structures to identify when services can be bootstrapped, when bootstrapping is blocked, and / or when bootstrap operations associated with previously blocked services can be resumed. It is convenient for the CIOS to identify circular dependencies within the data structures and to be able to perform actions to eliminate / resolve these circular dependencies prior to task execution. By using these techniques, the CIOS significantly reduces the risk of executing a task before the resources on which the task depends are available.
[0026] By utilizing the technologies disclosed herein, CIOS may perform data center modifications through parallel processing optimizations, while ensuring that tasks do not start until the functionalities on which they depend become available in the region. In this way, CIOS enables more efficient region construction, significantly reducing the time required to build data centers and the wasted computing resources seen in conventional methods.
[0027] A certain definition A "region" is a logical abstraction corresponding to a geographical location. A region may contain one or more execution targets of any suitable type. In some embodiments, the execution targets may also correspond to data centers.
[0028] An "execution target" represents the smallest unit of change required to execute a release. A "release" represents a declaration of intent to organize specific changes to a service (e.g., a version 8 deployment, adding an internal DNS record, etc.). For most services, an execution target represents an "instance" of the service. A single service may be bootstrapped for each of one or more execution targets. An execution target may be associated with a set of devices (e.g., a data center).
[0029] "Bootstrap" is intended to represent the collective task associated with the provisioning and deployment of any suitable number of resources (e.g., infrastructure components, artifacts, etc.) corresponding to a single service.
[0030] A "service" represents a set of resources that provide a function. A set of resources relating to a service includes any suitable combination of infrastructure, platforms, or software (e.g., applications) hosted by a cloud provider that can be configured to provide the functionality of the service. The service may be available to users over the internet.
[0031] An "artifact" refers to code deployed to an infrastructure component or Kubernetes engine cluster, which can include software (e.g., applications), configuration information for infrastructure components (e.g., configuration files), etc.
[0032] A "flock config" refers to a configuration file (or set of configuration files) that describes all the resources (e.g., infrastructure components and artifacts) associated with a single service. A flock config may contain declarations that specify one or more aspects corresponding to the desired state of the service's resources.
[0033] The "service state" represents a snapshot at a specific point in time of all resources associated with a service (e.g., infrastructure resources, artifacts, etc.). The service state indicates the status of the provisioning and / or deployment tasks associated with the service resources.
[0034] IaaS provisioning (or "provisioning") refers to acquiring computers or virtual hosts for use, and even includes installing the necessary libraries or services on them. The expression "provisioning a device" refers to the evolution of a device into a state where it can be used by an end user for a specific purpose. A device that has gone through the provisioning process is sometimes called a "provisioned device." Preparing a provisioned device (installing libraries and daemons) can be part of provisioning, but this is different from deploying a new application or a new version of an application to a prepared device. In most cases, deployment does not include provisioning, and provisioning may need to be performed first. A prepared device is sometimes called an "infrastructure component."
[0035] An IaaS deployment (or "deployment") refers to the process of providing and / or installing a new application or a new version of an application to provisioned infrastructure components. Once infrastructure components are provisioned (e.g., acquired, allocated, prepared, etc.), additional software deployment (e.g., provided to and installed on the infrastructure components) may occur. After provisioning and deployment are complete, infrastructure components may be referred to as "resources." Examples of resources include, but are not limited to, virtual machines, databases, object storage, block storage, and load balancers.
[0036] A "capability" identifies a functional unit associated with a service. This unit can be some or all of the functions that the service provides. For example, a capability may be issued indicating that a resource is available for authorization / authentication processing (e.g., a subset of the functions the resource provides). Another example is a capability being issued indicating that all of the functions of a service are available. Capabilities can be used to identify the functions that a resource or service depends on, and / or the functions of a resource or service that are available.
[0037] A "virtual bootstrap environment" (ViBE) represents a virtual cloud network provisioned over an existing region (e.g., a "host region"). The provisioned ViBE connects to the new region using a communication channel (e.g., IPSec Tunnel VPN). Within the ViBE, essential core services (or "seed" services), such as a deployment orchestrator and public key infrastructure (PKI) services, may be provisioned. These services can provide the capabilities necessary for bringing hardware online, establishing a trust chain to the new region, and deploying other services in the new region. By using a virtual bootstrap environment, circular dependencies between bootstrap resources can be prevented by utilizing the resources of the host region. Services can be staging and tested within the ViBE before the physical region (e.g., the target region) becomes available.
[0038] A "Cloud Infrastructure Orchestration Service" (CIOS) can represent a system configured to manage the provisioning and deployment of any number of services as part of a region build.
[0039] A multi-flock orchestrator (MFO) may be a computing component (e.g., a service) that provisions and deploys services to a target region (e.g., a new region) by coordinating events between CIOS components. The MFO tracks events related to each service in the region build and takes action in response to those events.
[0040] The "host region" refers to the region hosting the virtual bootstrap environment (ViBE). The host region can be used to bootstrap the ViBE.
[0041] The "target region" represents the region being built. "Publishing a capability" refers to the "publishing" or providing an indicator that a particular capability is available (or unavailable), as used in "publisher-subscriber" computing designs. Capabilities provide an indicator that a resource / service's functionality is available through "publishing" (e.g., collection by a capability service, provision to a capability service, push, pull, etc.). In some embodiments, capabilities can be published / transmitted by events, notifications, data transmissions, function calls, API calls, etc. Events (or other notifications / data transmissions, etc.) indicating the availability of a particular capability can be broadcast / addressed (e.g., published) to a capability service.
[0042] A "capabilities service" may be a flock configured to model dependencies between different flocks. A capabilities service may be provided within a cloud infrastructure organization service and may define capabilities, services, and features that are available in a region.
[0043] A "Real-time Regional Data Distributor" (RRDD) may be a service or system configured to manage regional data. This regional data can dynamically create execution targets for new regions by being injected into a flock configuration.
[0044] In some examples, this specification describes techniques for implementing Cloud Infrastructure Orchestration Services (CIOS). Such techniques can be configured to manage the bootstrapping (e.g., software provisioning and deployment) of infrastructure components within a cloud environment (e.g., a region), as briefly described above. In some cases, the CIOS may comprise computing components (e.g., CIOS Central and CIOS Regional, both described in more detail below) that can be configured to manage a multi-flock orchestrator (described in more detail below) which is configured to initiate / manage bootstrap tasks (provisioning and deployment) for a given service, as well as region building (e.g., bootstrap operations for multiple services).
[0045] CIOS enables region building and global infrastructure provisioning and code deployment, minimizing the manual execution effort required by service teams (for example, beyond initial hardware approval and / or physical transport in some cases). CIOS's advanced responsibilities include, but are not limited to, coordinating region building, providing users with a current view of resources managed by CIOS (e.g., intra-regional, inter-regional, global, etc.), and managing the bootstrap operations for bootstrapping resources within a region.
[0046] CIOS may provide view arbitration that can arbitrate a view of a resource's desired state (e.g., desired configuration) with the resource's current / actual state (e.g., current configuration). Depending on the case, view arbitration may include retrieving state data that identifies the resources actually running and their respective current configurations and / or states. Arbitration can be performed at various granularities, such as service levels.
[0047] CIOS can perform plan generation that identifies the differences between the desired and current states of a resource. Part of the plan generation may include identifying the actions required to transition the resource from its current state to the desired state. In some examples, CIOS may present the generated plan to the user for approval. In these examples, CIOS can allow the user to approve or reject the plan based on user input. This reduces the time the user spends inferring the plan, and because the plan is machine-generated, it is more accurate. While most plans are too detailed for human consumption, CIOS can provide this data through an advanced user interface (UI).
[0048] In some cases, CIOS can handle change management execution through the execution of approved plans. Once an execution plan is created and approved, engineers may no longer need to participate in change management unless CIOS initiates a rollback. CIOS can handle rollbacks to previous service versions by generating a plan to revert the service to a previous state (e.g., pre-release) if it detects a decline in service health during execution.
[0049] CIOS can measure service health by monitoring alerts and performing integration tests. CIOS can help teams quickly define rollback behavior in case of service degradation and implement it later. CIOS can generate and display plans and track approvals. CIOS can combine provisioning and deployment capabilities in a single system that coordinates these tasks across regional builds. CIOS also supports the discovery of flocks (e.g., service resources such as flock configurations for any number of services), artifacts, resources, and dependencies. Through static analysis of one or more configuration files (e.g., including content parsing and processing), CIOS can discover dependencies between execution tasks at any level (e.g., resource level, execution target level, phase level, service level, etc.). Using these dependencies, CIOS can generate various data structures from these dependencies that can be used to drive task execution (e.g., tasks related to provisioning infrastructure resources and deploying artifacts across regions).
[0050] Figure 1 is a block diagram of an environment 100 in which a Cloud Infrastructure Organization Service (CIOS) 102 may operate to dynamically provide bootstrap services in a region, according to at least one embodiment. CIOS 102 may comprise, but is not limited to, a Real-Time Regional Data Distributor (RRDD) 104, a Multi-Flock Orchestrator (MFO) 106, a CIOS Central 108, a CIOS Regional 110, and a Capability Service 112. Specific functions of the CIOS Central 108 and CIOS Regional 110 are described in detail in U.S. Patent Application No. 17 / 016,754, “Techniques for Deploying Infrastructure Resources with a Declarative Provisioning Tool,” the full contents of which are incorporated herein by reference. In some embodiments, any preferred combination of the components of CIOS 102 may be provided as a service. In some embodiments, any part of CIOS 102 may be deployed in a region (for example, a data center represented by a host region 103). In some embodiments, CIOS102 may include any number of preferred cloud services (not shown in Figure 1) as will be discussed in more detail below with respect to U.S. Patent Application No. 17 / 016,754 and Figures 2 and 3.
[0051] The Real-Time Regional Data Distributor (RRDD) 104 may be configured to maintain and provide regional data that identifies realms, regions, execution targets, and availability domains. The regional data may, if applicable, be in any preferred form (e.g., JSON format, data objects / containers, XML, etc.). The regional data maintained by RRDD 104 may include any preferred number of data subsets that can be individually referenced by corresponding identifiers. For example, the identifier "all_regions" may be associated with a data structure (e.g., a list, structure, object, etc.) containing metadata for all specified regions. Alternatively, an identifier such as "realm" may be associated with a data structure that identifies metadata for many realms and a set of regions corresponding to each realm. Generally, the regional data may maintain any preferred attributes of one or more realms, regions, availability domains (ADs), execution targets (ETs), etc. (e.g., identifiers, DNS suffixes, status (e.g., region status), etc.). RRDD 104 may be configured to manage region status as part of the regional data. Region status may include any preferred information indicating the bootstrap status within a region. As an example, some exemplary region states may include “Initial,” “Build,” “Generate,” “Paused,” or “Decommissioned.” The “Initial” state may indicate a region that has not yet been bootstrapped. The “Build” state may indicate that bootstrapping has begun for one or more flocks within the region. The “Generate” state may indicate that bootstrapping is complete and the region is ready for validation. The “Paused” state may indicate that CIOS Central 108 or CIOS Regional 110 has temporarily suspended internal interaction with the regional stack due to possible operational issues. The “Decommissioned” state may indicate that a region has been decommissioned and may be unavailable and / or may not be reconnectable.
[0052] CIOS Central 108 is configured to provide any number of user interfaces from which a user (e.g., user 109) can interact with CIOS 102. For example, a user can modify region data through a user interface provided by CIOS Central 108. CIOS Central 108 may also provide a variety of additional interfaces that allow users to check changes made to flock settings and / or artifacts, generate and check plans, approve / reject plans, and check the status of plan execution (e.g., tasks involving infrastructure provisioning, deployment, region building, and / or a desired state of any number of resources managed by CIOS 102). CIOS Central 108 may implement a control plane configured to manage any number of CIOS Regional 110 instances. CIOS Central 108 can provide one or more user interfaces that allow user 109 to check and / or modify region data by presenting it. CIOS Central 108 is configurable to invoke the functions of RRDD 104 through any number of interfaces. Generally, CIOS Central 108 may be configured to manage regional data directly or indirectly (for example, via RRDD 104). CIOS Central 108 may also be configured to inject regional data into flock configurations as variables through flock configuration compilation.
[0053] Each instance of CIOS Regional 110 may correspond to a module configured to perform a bootstrap task associated with a single service in the region. CIOS Regional 110 may receive desired state data from CIOS Central 108. In some embodiments, desired state data may include flock settings that declare (e.g., by declaration statements) the desired state of the resources associated with the service. CIOS Central 108 may maintain current state data indicating an optional preferred mode of the current state of the resources associated with the service. In some embodiments, CIOS Regional 110 may identify one or more resources that require modification by comparing the desired state data with the current state data. For example, CIOS Regional 110 may determine that it is necessary to provision one or more infrastructure components, deploy one or more artifacts, or make an optional preferred modification to the resources of the service to bring the state of those resources to a desired state. When performing a bootstrap operation, CIOS Regional 110 may publish data indicating the various capabilities of the resources that have become available. A "capability" identifies a functional unit associated with a service. This unit can be some or all of the functions that the service provides. For example, a capability may be issued indicating that a resource is available for authorization / authentication processing (e.g., a subset of the functions the resource provides). Another example is a capability being issued indicating that all of the functions of a service are available. Capabilities can be used to identify the functions that a resource or service depends on, and / or the functions of a resource or service that are available.
[0054] The capability service 112 is configured to maintain capability data indicating 1) the capabilities of various services currently available, 2) whether any resource / service is waiting for a particular capability, or 3) which specific resources and / or services, or any preferred combination thereof, are waiting for a given capability. The capability service 112 may also provide an interface from which capability data can be requested. The capability service 112 may also provide one or more interfaces (e.g., an application programming interface) that enable the transmission of capability data to the MFO 106 and / or the CIOS regional 110 (e.g., each instance of the CIOS regional 110). In some embodiments, any preferred component or module of the MFO 106 and / or the CIOS regional 110 may be configured to request capability data from the capability service 112.
[0055] In some embodiments, the multi-flock orchestrator (MFO) 106 may be configured to drive region construction attempts. In some embodiments, the MFO 106 can manage information representing flock / flock configuration versions and / or artifact versions used to bootstrap a given service in a region (or configure a change unit in a target region). In some embodiments, the MFO 106 may be configured to monitor (or be notified of) changes to region data managed by the real-time regional data distributor 104. In some embodiments, region construction may be initiated by the MFO 106 upon receipt of an indicator that region data has been changed. In some embodiments, the MFO 106 may collect various flock configurations and artifacts used for region construction. Some or all of the flock configurations may be configured to be region-independent; that is, the flock configurations may not explicitly identify the region to which the flocks are bootstrapped. In some embodiments, the MFO 106 may initiate a data injection process in which the collected flock configurations are recompiled (for example, by CIOS Central 108). During recompilation, the regional data maintained by the real-time regional data distributor 104 is injected into the configuration file by the execution of an operation (for example, by CIOS Central 108). The flock configuration can reference the regional data through variables / parameters without requiring hardcoded identification of the regional data. The flock configuration can be dynamically modified at runtime through this data injection without the modification becoming more difficult due to hardcoded regional data.
[0056] The multi-flock orchestrator 106 can perform static flock analysis, which identifies dependencies between resources, execution targets, phases, and flocks by analyzing the flock configuration, particularly identifying circular dependencies that need to be eliminated. In some embodiments, the MFO 106 can generate any number of suitable data structures based on the identified dependencies. These data structures (e.g., directed acyclic graphs, linked lists, etc.) may be used by the cloud infrastructure orchestration service 102 to drive operations for performing region construction. As an example, these data structures may collectively define the order in which services are bootstrapped within a region. An example of such a data structure is discussed separately below with respect to the construction dependency graph 338 in Figure 3. If circular dependencies exist (e.g., service A requires service B, and vice versa) and are identified by static flock analysis and / or graphs, the MFO may be configured to notify any suitable service team that changes to the flock configuration are needed to correct these circular dependencies. The MFO 106 can be configured to manage the order in which services are bootstrapped into a region by passing through one or more data structures. MFO106 can identify capabilities available within a given region at any given time (for example, using data obtained from capability service 112). Using this data, MFO106 can identify when a service can be bootstrapped, when a bootstrap is blocked, and / or when a bootstrap operation associated with a previously blocked service can be resumed. Based on this process, MFO106 can execute a variety of releases, in which MFO106 executes bootstrap operations corresponding to any number of flock settings by sending instructions to CIOS Central 108. In some examples, MFO106 may be configured to identify that one or more flock settings may require multiple releases due to circular dependencies observed in the graph.As a result, the MFO106 may resolve cyclic dependencies identified in the graph by sending multiple instruction sets to the CIOS central 108 for a given flock setting.
[0057] In some embodiments, a user may request the creation of a new region (for example, target region 114). This may include bootstrapping resources to support a variety of services. In some embodiments, target region 114 may not be communicable (and / or secure) at the time the region creation request is initiated. Rather than delaying the bootstrap until target region 114 becomes available and configured to perform the bootstrap operation, CIOS 102 may initiate region creation using a virtual bootstrap environment 116. The virtual bootstrap environment (ViBE) 116 may be an overlay network hosted by host region 103 (an existing region with a core set of services previously configured, which is communicable and secure). MFO 106 can use the resources of host region 103 to bootstrap resources to ViBE 116 (commonly referred to as "building the ViBE"). For example, MFO106 can provide a command through CIOS Central 108 to bootstrap another instance of the CIOS regional in ViBE116 to an instance of the CIOS regional 110 in the host region (e.g., host region 103). Once the CIOS regional in ViBE becomes available for processing, the bootstrapping of services to the target region 114 can continue within ViBE116. If the target region 114 is available for the bootstrap operation to be performed, previously bootstrapped services in ViBE116 can move to the target region 114. By utilizing these techniques, CIOS102 can significantly improve the speed of region construction by greatly reducing the need to provide any manual input and / or configuration.
[0058] Figure 2 is a block diagram showing an environment 200 and method for constructing a virtual bootstrap environment (ViBE) 202 (an example of ViBE 116 in Figure 1) according to at least one embodiment. ViBE 202 represents a virtual cloud network provisioned on an overlay of an existing region (for example, host region 103 in Figure 1, and in one embodiment, host region 204 which is a host region service enclave). ViBE 202 represents an environment in which services may be staged for a target region (for example, target region 114 in Figure 1, or a region under construction) before the target region becomes available.
[0059] For bootstrapping a new region (for example, target region 114 in Figure 1), a core set of services may be bootstrapped. These core services reside in host region 204 but not in ViBE (or the target region). These essential core services provide the functionality necessary for device provisioning, establishing a trust chain to the new region, and deploying other services (e.g., flocks) to the region. ViBE202 may be a tenancy deployed in host region 204. This can be considered a virtual region.
[0060] If a target region is available for providing bootstrap operations, ViBE202 can connect to the target region so that services within that ViBE can interact with services and / or infrastructure components in the target region. This enables the deployment of generation-level services instead of self-contained seed services as in traditional systems, and requires connectivity to the target region over the internet. Traditionally, seed services were deployed as part of a container collection and used to bootstrap the dependencies necessary to build the region. By using existing regional infrastructure / tools, resources can connect to a service enclave in the region (e.g., host region 204) for hardware provisioning and service deployment until ViBE202 is bootstrapped (e.g., provisioning and deployment) and the target region is self-sufficient and can communicate directly. Using ViBE202 enables the establishment of dependencies and services necessary for infrastructure provisioning / preparation and software deployment, while resolving circular dependencies of core services by utilizing resources in the host region.
[0061] The multi-flock orchestrator (MFO) 206 may be configured to perform an action to build (for example, configure) ViBE 202. The MFO 206 can obtain applicable flock settings for various resources to be bootstrapped into a new region (in this case, the ViBE region of ViBE 202). For example, the MFO 206 may be configured to obtain a flock setting (for example, "ViBE flock setting") that identifies the aspects of the bootstrap capability service 208 and worker 210. As another example, the MFO 206 may be configured to obtain a different flock setting for the bootstrapping of the Domain Name Service (DNS) 212 to ViBE 202.
[0062] In step 1, MFO206 may instruct CIOS Central 214 (for example, examples of CIOS Central 108 and CIOS Central 214 in Figures 1 and 2, respectively). For example, MFO206 may send a request (for example, including ViBE flock settings) to bootstrap capability services 208 and workers 210 that do not yet exist in ViBE202 at this point. In some embodiments, CIOS Central 214 may have access to all flock settings. Thus, in some examples, MFO206 may send an identifier for the ViBE flock settings rather than the files themselves, and CIOS Central 214 may retrieve it independently from storage (for example, DB308 or flock DB312 in Figure 3).
[0063] In step 2, CIOS Central 214 may provide the ViBE flock configuration to CIOS Regional 216 in response to a corresponding request. In step 3, CIOS Regional 216 may parse the ViBE flock configuration to identify and perform specific infrastructure provisioning and deployment actions.
[0064] In some embodiments, the CIOS regional 216 may utilize additional corresponding services for provisioning and deployment. For example, in step 4, the CIOS regional 216 may instruct the deployment orchestrator 218 (e.g., a core service or other example of description, build, and deployment application software in host region 204) to execute instructions to bootstrap the capability service 208 and worker 210 in ViBE 202.
[0065] In step 5, capabilities may be sent to the capability service 208 (via the CIOS regional 216, deployment orchestrator 218, worker 210, etc.) indicating that resources corresponding to the ViBE flock are available. The capability service 208 may persist this data. In some embodiments, the capability service 208 adds this information to a list that holds capabilities available in ViBE. For example, the capabilities given to the capability service 208 in step 5 may indicate that the capability service 208 and worker 210 are available for processing.
[0066] In step 6, the MFO 206 may, based on the receipt or acquisition of data (an identifier corresponding to a capability) from the capability service 208, identify that the capability indicates that the capability is available to the capability service 208 and the worker 210.
[0067] In step 7, as a result of receiving / retrieving data in step 6, MFO206 may instruct CIOS Central 214 to bootstrap a DNS service (e.g., DNS212) to ViBE202. These instructions may identify or include specific flock settings corresponding to the DNS service.
[0068] In step 8, the CIOS Central 214 may instruct the CIOS Regional 216 to deploy DNS212 to ViBE202. In some embodiments, the CIOS Central 214 provides the DNS flock configuration for DNS212.
[0069] In step 9, CIOS regional 216 may assign worker 210, deployed on ViBE202, to the task of deploying DNS212. The worker may run a declarative infrastructure provisioner as described above in relation to Figure 3 to identify a set of actions that need to be performed for the deployment of DNS212 (for example, by comparing the flock configuration (desired state) to the current state of the (non-existent) resources associated with the flock).
[0070] In step 10, the deployment orchestrator 218 may instruct worker 210 to deploy DNS212 according to the operation identified in step 9. As shown in the figure, worker 210 proceeds in step 11 to perform the operation to deploy DNS212 to ViBE202. In step 12, worker 210 notifies capability service 208 that DNS212 is available in ViBE202. MFO206 may then identify the resources associated with the ViBE flock configuration and DNS flock configuration as available and proceed to bootstrap any suitable number of additional resources to ViBE.
[0071] After completing steps 1 through 12, the process for building ViBE202 is complete, and it can be considered that ViBE202 has been built.
[0072] Figure 3 is a block diagram showing an environment 300 and method for bootstrapping a service to a target region using ViBE, according to at least one embodiment.
[0073] In Step 1, User 302 may modify the region data using any preferred user interface provided by CIOS Central 304 (examples of CIOS Central 108 and CIOS Central 214 in Figures 1 and 2, respectively). For example, User 302 may create a new region on which many services are bootstrapped.
[0074] In step 2, the CIOS central 304 may perform an action to send the changes to the RRDD 306 (an example of the RRDD 104 in Figure 1). In step 3, the RRDD 306 may store the received region data in the database 308, which is a datastore configured to store region data including any preferred identifiers, attributes, status, etc., such as region, AD, realm, ET, etc. In some embodiments, the use of the updater 307 may be used to store the region data in the database 308 or any preferred datastore that can access such update information (for example, by a service team). In some embodiments, the updater 307 may be configured to notify the database 308 of the update information (for example, by any preferred electronic notification).
[0075] In step 4, the MFO310 (examples of MFO106 and 206 in Figures 1 and 2, respectively) may be configured to detect changes in region data. In some embodiments, the MFO310 may be configured to poll the RRDD306 for changes in region data. In some embodiments, the RRDD306 may be configured to issue or notify the MFO310 of changes in region data.
[0076] In step 5, detecting changes in region data may trigger MFO310 to retrieve a version set (e.g., a version set associated with a specific identifier, such as a "golden version set" identifier) that identifies a specific version of each flock (e.g., a service) being bootstrapped to a new region, and a specific version of each artifact corresponding to that flock. The version set may be retrieved from DB312. As flocks evolve and change, the versions of the corresponding settings and artifacts used to build the region may also change. These changes can be persisted in the flock DB312 so that MFO310 can identify the versions of flock settings and artifacts that should be used to build the region (e.g., ViBE region, target region / non-ViBE region, etc.). The flock settings (e.g., all versions of the flock settings) and / or artifacts (e.g., all versions of the artifacts) may be stored in any suitable data store accessible by DB308, DB312, or CIOS Central 304 and / or MFO310.
[0077] In step 6, MFO310 may instruct CIOS Central 304 to recompile each of the flock settings associated with the version set using the current region data. In some embodiments, this request may specify the versions of each flock setting and / or the artifacts corresponding to these flock settings.
[0078] In step 7, CIOS Central 304 may retrieve the current regional data from DB 308 (for example, directly or via the real-time regional data distributor 306) and read out any preferred flock settings and artifacts according to the version requested by MFO 310.
[0079] In step 8, the CIOS central 304 may inject the current region data into the flock configuration by recompiling the flock configuration with the region data obtained in step 7. The CIOS central 304 may return the compiled flock configuration to the MFO 310. In some embodiments, the CIOS central 304 may simply indicate that compilation has been performed, and the MFO 310 may access the recompiled flock configuration via the RRDD 306.
[0080] In step 9, MFO310 may perform a static analysis of the recompiled flock configuration. As part of the static analysis, MFO310 may identify dependencies between flocks by parsing the flock configuration (for example, using a library associated with a declarative infrastructure provisioner, e.g., Terraform). From this analysis and identified dependencies, MFO310 can generate a build dependency graph 338. The build dependency graph 338 may be a directed acyclic graph that identifies the order in which flocks are bootstrapped into new regions (and / or the changes indicated in the flock configuration are applied). Each node in the graph may correspond to a bootstrap of any preferred portion of a particular flock. The specific bootstrap order may be identified at least in part on dependencies. In some embodiments, dependencies may be expressed as attributes of nodes and / or specified by graph edges connecting the nodes. MFO310 may drive the region build operation by traversing the graph (for example, starting from the start node).
[0081] In some embodiments, the MFO310 may utilize a cycle detection algorithm to detect the presence or absence of cycles (for example, service A depends on service B, and vice versa). The MFO310 can identify dependencies of orphaned capabilities. For example, the MFO310 can identify orphaned nodes in the build dependency graph 338 that are not connected to other nodes. The MFO310 may also identify improperly issued capabilities (for example, when a capability is issued prematurely and the corresponding functionality is not yet actually available). The MFO310 can detect from the graph that there are one or more instances issuing the same capability. In some embodiments, any number of these errors may be detected, and the MFO310 (or another preferred component, such as the CIOS Central 304) may be configured to notify or present this information to the user (for example, via electronic notification, user interface, etc.). In some embodiments, the MFO310 may be configured to resolve circular dependencies by forcibly deleting / recreating resources, and may also instruct the CIOS Central 304 to perform bootstrap operations on these resources and / or corresponding flock settings.
[0082] The initiating node may handle the bootstrap of the ViBE flock, and the second node may handle the bootstrap of the DNS. Steps 10-15 correspond to the deployment of the ViBE flock to ViBE316 (for example, examples of ViBE116 and 202 in Figures 1 and 2, respectively) by the deployment orchestrator 317 (which is an example of the deployment orchestrator 218 in Figure 2). That is, steps 10-15 in Figure 3 roughly correspond to steps 1-6 in Figure 2. Once it is notified that a capability exists that corresponds to the deployment of the ViBE flock (for example, indicating that capability services 318 and worker 320 corresponding to capability service 208 and worker 210 in Figure 2 are available), the MFO310 resumes traversing the build dependency graph 338 and identifies the next action to be performed.
[0083] For example, MFO310 may continue traversing the build dependency graph 338 to identify that a DNS flock is to be deployed. Steps 16-21 may be performed to deploy DNS322 (an example of DNS212 in Figure 2). These actions may roughly correspond to steps 7-12 in Figure 2.
[0084] In step 21, a capability indicating that DNS322 is available may be stored. Upon detection of this capability, MFO310 may resume traversing the build dependency graph 338. During this traversal, MFO310 may identify that any preferred portion of an instance of a CIOS regional (for example, an example of CIOS regional 314) is to be deployed to ViBE316. In some embodiments, steps 16-21 may be substantially repeated with respect to the deployment of CIOS regional (ViBE)326 (CIOS regional 314, an instance of CIOS regional 110 in Figure 1) and worker 328 to ViBE316. Alternatively, a capability indicating that CIOS regional (ViBE)326 is available may be sent to the capability service 318.
[0085] Upon detecting that CIOS Regional (ViBE) 326 is available, MFO 310 may resume traversing the build dependency graph 338. During this traversal, MFO 310 may identify that a deployment orchestrator (for example, deployment orchestrator 330, which is an example of deployment orchestrator 317) is to be deployed to ViBE 316. In some embodiments, steps 16-21 may be substantially repeated with respect to the deployment of deployment orchestrator 330. Information identifying the capability indicating that deployment orchestrator 330 is available may also be sent to capability service 318.
[0086] After the deployment orchestrator 330 is deployed, ViBE 316 may be considered available for processing subsequent requests. Upon detecting that the deployment orchestrator 330 is available, MFO 310 may instruct the routing of subsequent bootstrap requests to the ViBE components rather than to utilize the host region components (components of host region 332). This allows MFO 310 to continue traversing the build dependency graph 338 at each node instructing a flock deployment to ViBE 316 via CIOS Central 304. CIOS Central 304 may instruct CIOS Regional (ViBE) 326 to deploy resources according to the flock configuration.
[0087] At some point in this process, target region 334 may become available. An indicator that the target region is available may be identifiable from the region data for target region 334 provided by user 302 (for example, as region data update information). The availability of target region 334 may depend on the establishment of a network connection between target region 334 and an external network (e.g., the internet). The network connection may be supported over a public network (e.g., the internet), but may also be provided by the use of software security tools (e.g., IPSec) to provide one or more encrypted tunnels from ViBE316 to target region 334 (e.g., an IPSec tunnel such as tunnel 336). As used herein, "IPSec" represents a set of protocols for authenticating and encrypting network traffic on a network using Internet Protocol (IP), and may include one or more available embodiments of the set of protocols (e.g., Openswan, Libreswan, strongSwan, etc.). The network may connect ViBE316 to a service enclave in target region 334.
[0088] Prior to the establishment of the IPSec tunnel, the initial network connectivity to target region 334 may be sufficient (e.g., an out-of-band VPN tunnel) to enable the bootstrapping of networking services until the IPSec gateway is deployed to an asset (e.g., a bare-metal asset) within target region 334. To bootstrapping the network resources in target region 334, the deployment orchestrator 330 may deploy an IPSec gateway on an asset within target region 334. The deployment orchestrator 330 may also deploy a VPN host in target region 334, which is configured to terminate the IPSec tunnel from ViBE316. Once services in ViBE316 (e.g., deployment orchestrator 330, service A, etc.) are able to establish IPSec connectivity with the VPN host in target region 334, the bootstrap operation from ViBE316 to target region 334 may begin.
[0089] In some embodiments, the bootstrap operation may be initiated by a service in ViBE316 that supports hosting instances of core services deployed from ViBE316 by provisioning resources in the target region 334. For example, a host provisioning service may allocate computing resources for VMs by provisioning hypervisors to infrastructure (e.g., bare metal hosts) in the target region 334. Once the host provisioning service has completed allocating physical resources in the target region 334, it may issue information indicating a capability that physical resources in the target region 334 have been allocated. This capability may be issued to the capability service 318 via the CIOS regional (ViBE) 326 (e.g., by a worker 328).
[0090] Once the hardware allocation for target region 334 is established and posted to capability service 318, CIOS regional (ViBE) 326 can orchestrate the deployment of core service instances from ViBE 316 to target region 334. This deployment may be similar to the process described above for building ViBE 316, but uses ViBE components (e.g., CIOS regional (ViBE) 326, worker 328, deployment orchestrator 330) instead of components of the service enclave in host region 332. The deployment operation may broadly correspond to steps 16-21 described above.
[0091] When a service is deployed from ViBE316 to target region 334, the DNS records associated with that service may correspond to the instance of the service in ViBE316. The DNS records associated with the service may be updated later to complete the deployment of the service to target region 334. In other words, the instance of the service in ViBE316 may continue to receive traffic (e.g., requests) to the service until the DNS records are updated. The service may be partially deployed to target region 334, and information indicating the capability that the service is partially deployed may be published (e.g., to capability service 318). For example, a service running in ViBE316 may be deployed to target region 334 along with its corresponding compute instance, load balancer, and associated applications and other software, but it may be necessary to wait for the database data to move to target region 334 before the deployment is complete. DNS records (e.g., managed by DNS322) may still be associated with the service in ViBE316. Once the service data migration is complete, the DNS record may be updated to point to the operational service deployed in target region 334. Subsequently, the service deployed in target region 334 will receive traffic for that service (e.g., requests), while the instance of the service in ViBE316 does not need to receive traffic for that service.
[0092] Re-entry Service Deployment This disclosure generally relates to the construction of a data center. More specifically, it describes techniques for creating, provisioning, and deploying resources and infrastructure for constructing a data center configured to provide a set of services. It also provides techniques for releasing services and running multiple releases. For services in which releases have been previously scheduled and run, a second release may be scheduled for the service in response to optional capability dependencies of the service being issued (available) in the data center.
[0093] In one embodiment, services provided in a constructed data center may depend on other resources and services. These dependencies can be required or optional. Resources used by a service are provisioned and deployed in the data center through the scheduling and execution of service releases. A service release may be scheduled and executed even if optional dependencies are not met, provided that all required dependencies for the service are met (or issued or made available in the data center). As a result of the execution of the release, resources used by the service are created and deployed. For a service for which a release has already been scheduled and executed as a result of all required dependencies being met, a second release may be scheduled and executed for that service if any optional dependencies for the service are later met. This second release is referred to as a re-entry deployment or re-entry release. The second release can create additional resources for the service. One or more re-entry releases may be scheduled and executed for a service, each re-entry release and its execution potentially enabling additional capabilities for the service.
[0094] In one embodiment, a capability may identify a functional unit associated with a service and represent a set of resources associated with a service. A capability can represent one or more resources, which may be software or hardware resources, functions, services, etc. Capabilities may be identified by capability labels. Examples of capability labels include "Capability_A", "Capability_B", "Capability_C", etc. For example, a capability labeled Capability_A may correspond to resources R1 and R2, and be indicated as Capability_A={R1,R2}. Similarly, Capability_B={R3}, Capability_C={R4,R5,R6}, etc.
[0095] A capability is considered to be issued (or activated or made available) in a computing environment such as a data center when all resources corresponding to that capability have been created, deployed, and are available for use for their intended purpose within the data center. In the example above, Capability_A is considered to be issued with respect to the data center when resources R1 and R2 corresponding to Capability_A have been created and deployed in the data center and are ready for use for their intended purpose.
[0096] A data center can provide a variety of services. For a given service, a set of one or more resources used or required to provide that service is referred to as a flock for that service. A flock may include one or more resources. A flock typically represents a set of managed resources responsible for providing a service. Information related to a service flock is declared as part of the service flock configuration. Since the service flock configuration is usually stored in file form, it is also referred to as the service flock configuration file. The service flock configuration describes the flocks associated with the service (for example, a set of resources including infrastructure components and artifacts associated with the service). The service flock configuration may include declarations specifying one or more aspects corresponding to a desired state of the resources associated with the service. The flock configuration may identify various settings and configurations associated with the flock resources, as well as other parameters associated with the service. In one embodiment, the flock configuration includes Terraform code, and the flock configuration file is a Terraform file.
[0097] In a typical data center environment configured to provide multiple services, services may have dependencies on other services or resources. For example, service B may require service A to exist before service B can bootstrap (potentially by using service A). Service C may require service B to exist before service C can bootstrap in the data center. Thus, multiple dependencies can exist between services. In one embodiment, these dependencies are clearly expressed by the use of capabilities. For example, a service flock configuration can identify any capabilities on which a service depends. Capabilities on which a service depends are referred to as the service's capability dependencies (also referred to as the service's flock configuration's capability dependencies).
[0098] Flocks associated with a service (also referred to as service flocks) are described in the flock configuration specified for that service. The flock configuration may be in the form of a flock configuration file. A service capability dependency is also referred to as a capability dependency for a flock for that service. A service capability dependency (for example, a service flock capability dependency) may be a required capability dependency or an optional capability dependency. A required capability dependency for a service (or service flock) is one that requires the capability to be issued or enabled in the data center before a release can be scheduled for that service or service flock. An optional capability dependency for a service flock allows the release of that service flock to still be scheduled if the optional capability dependency (optional capability) has not been issued in the data center, provided that the optional required capability dependencies for that flock are met. Therefore, optional capability dependencies do not block the release of a service. If an optional capability dependency becomes available or is issued, another release of the service flock is scheduled and executed. Thus, a service flock may potentially be released and executed multiple times (sometimes referred to as the reentrant nature of service flock releases). Generally, a service release is scheduled and executed when the required capability dependencies for the service have been issued, but one or more optional capability dependencies are not yet available. A release and subsequent execution of a service flock without one or more optional capability dependencies being issued or available would result in an instantiation of a version of the service that possesses only some, not all, of the capabilities. It should be noted that not all service flocks need to have capability dependencies, whether required or optional.
[0099] Service capability dependencies may be explicitly declared in the service flock configuration, or they may be implicitly determined from the service flock configuration (by MFO as described below). If capability dependencies are explicitly declared in the flock configuration, this flock configuration includes code or metadata that explicitly declares the capability dependencies. In other cases, capability dependencies may be inferred from the code in the flock configuration (for example, by MFO106 if MFO106 reads and parses the flock configuration). A service flock configuration may identify zero or more required capabilities for the service flocks specified by that configuration. Therefore, a service does not need to have capability dependencies.
[0100] As a result of the release and execution of a service flock, one or more capabilities may be issued in the data center (though not necessarily). Information identifying any capabilities issued by the execution of a service flock release is also declared in the service flock configuration. In one embodiment (such as the embodiment shown in Figure 1), MFO 106 schedules the release of a service flock. Then, for example, the release is executed by CIOS Central 108 and CIOS Regional 110 in Figure 1, and if the release is successful, zero or more capabilities may be issued in the data center. In this way, the release of a service flock and the corresponding execution result in the issuance of additional capabilities in the data center.
[0101] In one embodiment (such as the embodiment shown in Figure 1), as part of the process of building a data center in a region where the data center will provide a specific set of services, a flock configuration for a specific set of services is provided to MFO106. MFO106 then reads and parses these flock configuration files, and for each flock described by the flock configuration, identifies the flock's capability dependencies (if any) and the capabilities issued (if any) when executing one or more releases of the flock. MFO106 then constructs an acyclic dependency graph to represent the various capability dependencies between the flock configuration and the corresponding flocks. The dependency graph also identifies the dependency order of the capabilities identified by multiple flock configurations. Based on the identified dependencies, MFO106 can generate any number of suitable data structures. These data structures (e.g., directed graphs, directed acyclic graphs, linked lists, etc.) may be used by CIOS102 (e.g., MFO106, CISO Central 108, CIOS Regional 110) to facilitate operations for building one or more data centers in a region (also referred to as region construction execution). The processes performed by MFO106 in analyzing flock settings and generating acyclic dependency graphs may also be referred to as static flock analysis.
[0102] Capability dependencies between different flocks specified in flock configurations for different services can be illustrated by the following examples: A flock configuration for service A (FC_A) specifies a flock (F_A) for service A. FC_A does not identify any capability dependencies of F_A, but it does identify that Capability_A was issued upon the successful release and execution of flock F_A. A flock configuration (FC_B) can specify information associated with a flock (F_B) for service B, indicating that Capability_A is a required capability dependency and that Capability_B was issued upon the successful execution of flock F_B for service B. A flock configuration (FC_C) can specify information for a flock (F_C) for service C, indicating that Capability_B is a required capability dependency and that Capability_C was issued to the data center upon the successful release and execution of flock F_C for service C. Given these flock settings, the dependency graph generated by MFO106 identifies the following: FC_A(F_A)→Capability_A→FC_B(F_B)→Capability_B→FC_C(F_C)→Capability_C If F_A has no capability dependencies, then in the data center, the release and execution of F_A will issue Capability_A, which is the required capability dependency of F_B; the release and execution of F_B will issue Capability_B, which is the required capability dependency of F_C; and the release and execution of F_C will issue Capability_C. Regarding capabilities, the dependency graph is as follows: Capability_B depends on Capability_A, Capability_C depends on Capability_B, and is represented as Capability_A → Capability_B → Capability_C. In this way, the dependency graph generated by MFO106 for the data center identifies dependencies between multiple capabilities that are declared in the flock configuration corresponding to the various services instantiated in the data center, or inferred from the flock configuration.
[0103] In the case of a dependency example (Capability_A → Capability_B → Capability_C), Capability_B is referred to as a direct dependency of Capability_C. Similarly, Capability_A is referred to as a direct dependency of Capability_B. Furthermore, Capability_A is referred to as an indirect dependency of Capability_C. If Capability_A depends on another capability, this other capability also becomes an indirect dependency of Capability_C, and so on. Generally, in a hierarchical dependency graph, for a particular capability X in the graph, the parent capabilities of capability X in the graph are referred to as the direct capability dependencies of capability X. All ancestor capabilities of the parent capability in the graph are referred to as indirect capability dependencies of capability X. Capabilities in a dependency graph can have multiple levels of indirect capability dependencies.
[0104] The MFO then schedules the release of individual service flocks based on the availability of capabilities issued in the data center. In the example above, assuming that all capability dependencies are necessary, with Capability_B depending on Capability_A and Capability_C depending on Capability_B, the MFO will first schedule the release of flock F_A for service A. If the release is successful, Capability_A will be issued in the data center. Then, after determining that the necessary capability dependency for flock F_B for service B (i.e., Capability_A) has been met, the MFO 106 will schedule the release of flock F_B for service B. If the release of service B is successful, Capability_B will be issued in the data center. Then, after MFO106 determines that the necessary capability dependencies (i.e., Capability_B) for flock F_C for service C have been met, it will schedule the release of flock F_C for service C, resulting in the issuance of Capability_C. In this way, as the number of flock releases scheduled and executed increases, the number of capabilities issued in the data center environment also increases, thus triggering additional releases in the data center environment.
[0105] Re-entry releases and deployments occur when a flock for a service has at least one optional capability dependency. Optional capability dependencies may be explicitly declared in the service flock configuration or inferred from the service flock configuration. In some cases, optional dependencies are typically tied to conditional statements (e.g., X if COUNT is greater than 1, Y otherwise). As mentioned above, the MFO determines whether a dependency is required or optional during static flock analysis.
[0106] As described above, if all necessary capability dependencies have been issued, the release of the flock for the service will be scheduled and executed even if one or all of the optional capability dependencies of the service flock have not been issued. For example, in the above example, if Capability_A, a necessary capability dependency for flock F_B for service B, is issued, the release of F_B will be scheduled and executed. Here, we assume that the flock configuration FC_B for service B further specifies that flock F_B has an additional optional capability dependency "Capability_D". Capability_D may be issued in the data center after Capability_A has been issued and the release of F_B has already been scheduled and executed. If Capability_D is then issued, another release (re-entry release) will be scheduled and executed for flock F_B. If only the required capability dependency Capability_A is issued, and not the optional capability dependency Capability_D, the initial release and execution of F_B may result in an instantiation of a version of service B that possesses only some, not all, of the capabilities. If the optional capability dependency Capability_D is satisfied, the re-entry release and execution of the flock F_B may result in an instantiation of a version of service B that possesses all of the capabilities.
[0107] In one embodiment, MFO106 may schedule releases by preparing a "path" of the flock for service for an initial release if required capability dependencies are issued, or for a re-entry release if optional capability dependencies are issued. The release / path may identify a specific execution target. MFO106 then provides the prepared release / path to CIOS Central 108 for execution. CIOS Central 108 then executes the release / path in coordination with CIOS Regional 110. In one embodiment, the execution of the release / path involves preparing a "plan" for the release and subsequently executing the plan. As part of the plan preparation, the current state of the target system (e.g., a target data center) with respect to the service on which the execution will occur is determined, and the final desired state of the target system (as indicated in the release) is determined. A plan is then generated for the transition of the target system from the current state to the final state. This plan is then applied (by using apply commands) or executed. Assuming the execution is successful, zero or more capabilities may be issued in the data center as a result of the execution. These capabilities may satisfy the (required or optional) dependencies of several other flocks.
[0108] The following is an example of a flock configuration section that declares required and optional capabilities.
[0109] ad = module.service_availability_domain.availability_domain pool = module.service_availability_domain.pool_name } resource "odo_deployment" deploy_tenancy_creator { ad = module.service_availability_domain.availability_domain alias = "${local.serviceName}${local.serviceSuffix}" Below { url = local.artifact_versions["tenancy-creator"].uri build_tag = local.artifact_versions["tenancy-creator"].version type = local.artifact_versions["tenancy-creator"].type } node_ids = [ data.odo_nodes.deployment_node.nodes[0].resource_id ] } |resource "capability" "tenancy_creator_initial_tenancies_created" { name = "tenancy_creator_initial_tenancies_created" depends_on = [ odo_deployment.deploy_tenancy_creator ] } # Do this if you have dependencies that are not needed by the bootstrap. # Note: This is a data source, not a resource. data "capability" "get_alarm_available" { # This field is the name of the actual capability it depends on. name = "telemetry_alarm" } / / Resource example resource "telemetry_alarm" heartbeatAlarm { # Here, the "count=" pattern delays the creation of resources until the underlying service becomes available. count = data.capability.get_alarm_available.available ? 1 : 0 compartment_id = var.compartment_id display_name = "Missing Host Heartbeat in Fleet - ${var.fleet_name}" query = "ServiceHostReporter.heartbeat[1m].groupBy(host).absent()" pending_duration = var.pending_duration project = var.project_name fleet = var.fleet_name severity = var.severity.normal is_enabled = true body = < <BODY Figure 4 is a simplified flowchart 400 showing the process performed to schedule a re-entry release according to one embodiment. The process shown in Figure 4 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, or it may be implemented using hardware, or a combination of these. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods described later in Figure 4 are illustrative and not intended to be limiting. Figure 4 shows various processing steps that occur in a particular order or sequence, but not intended to be limiting. In an alternative embodiment, the process may be executed in some different order, or some steps may be executed in parallel. In one embodiment (such as the embodiment shown in Figure 1), the process shown in Figure 4 may be executed by the MFO 106 in coordination with the CIOS Central 108, the CIOS Regional 110, and the Capability Service 112.
[0110] In the embodiment shown in Figure 1, processing may be triggered when MFO 106 determines that a new capability has been issued (e.g., made available) in the data center. For example, in the embodiment shown in Figure 1, capability service 112 is responsible for tracking issued capabilities. When a new capability is issued, such as due to the successful execution of a flock release for various services in the data center, capability service 112 tracks the issued capability and makes the information available to other systems. For example, when a new capability is issued, MFO 106 receives a signal from capability service 112 indicating that a new capability has been issued, and this signal may include information identifying the new capability. In another embodiment, MFO 106 may query capability service 112 and receive a response identifying the newly issued capability (or a list of all issued capabilities) including the new specific capability determined in 402.
[0111] In some other embodiments, capabilities issued with respect to the data center may be tracked by the use of other techniques. For example, a registry or database that identifies issued capabilities may be maintained. MFO106 may query this registry or database to find out which new capabilities have been issued (or receive a signal when new capabilities are added to this database or registry).
[0112] In 404, MFO106 may determine that a newly issued capability identified in 402 is an optional capability dependency of a particular flock for a particular service in which a release of that particular flock was previously scheduled and successfully executed. As a result of scheduling and subsequent execution of past releases, the particular service is already functioning with several capabilities in the data center. In one embodiment, MFO106 uses an acyclic dependency graph generated by MFO106 by performing a static flock analysis (e.g., parsing and reading flock configurations associated with services supported by the data center) to determine that a newly issued capability is an optional capability dependency of a particular service and its associated flock.
[0113] In 406, MFO106 schedules a new “re-entry” release for a particular flock. In one embodiment, as part of the processing of 406, if MFO106 determines that a release is necessary for the progress of the data center construction, the resulting plans, approvals, and applications are captured in an encapsulation model called a path. MFO106 then uses the path to drive the release. In such an embodiment, as part of 406, MFO106 creates a path for the scheduled release.
[0114] In 408, the cooperating CIOS Central 110 and CIOS Regional 110 execute the release scheduled by MFO 106 in 406.
[0115] Assuming that the execution of release 408 is successful, in 410, MFO106 may update the metadata associated with a particular flock to indicate the success of the execution. In one embodiment, the metadata associated with the flock settings of a particular flock may be updated.
[0116] Furthermore, if execution is successful and a new capability (or multiple new capabilities) is issued in 412, this information may be persisted and provided to the system components of interest. For example, capability service 112 may be notified of the newly issued capability. If a registry or database is used to track services issued for a data center, the registry or database may be updated with information indicating the issuance of a new capability.
[0117] A successful release execution may issue zero or more new capabilities to the data center. Therefore, this particular service may already be functioning in the data center with certain capabilities due to one or more successful past release executions, but in this case, the successful execution of the re-entry release in 408 may have added additional capabilities.
[0118] For example, this service may be an identity management service configured to provide authentication and authorization functions in the data center. The flock configuration of the identity management service may identify one or more required and optional capability dependencies of the service. If MFO106 determines that all of the required capability dependencies of the service have been met (e.g., issued in the data center), it may schedule an initial release of the service even if one or more of the optional capability dependencies of the service have not been met at the time of release scheduling (e.g., one or more of the optional capability dependencies have not been issued in the data center). Successful execution of this release by CIOS Central 108 and CIOS Regional 110 results in the identity management service being instantiated in the data center and capable of providing certain authentication and authorization functions. If MFO106 determines at a later point that a new optional capability dependency of the identity management service has been issued, it may schedule a new re-entry release of the service for each of the processes described above as shown in Figure 4. The successful execution of a new release (for example, as described in 408 above) may add additional capabilities to the identity management service. For example, after executing a new re-entry release, the identity management service may be able to perform authorizations using certain technologies (such as OAUTH2.0 (Open Authorization) authorizations) that were not possible after the initial release.
[0119] As described above, multiple releases can be scheduled and executed for a service. The first release can be scheduled by MFO106 after all the necessary capability dependencies for the service, as indicated in the service's flock configuration, have been met and are available or published. This first release and execution may add capabilities to the service. When optional capability dependencies for the service become available in the data center, one or more subsequent releases can be scheduled and executed for the service. As a result of these one or more subsequent releases and executions, additional extended capabilities may be added to the service. Therefore, when optional capability dependencies for the service are published, the flock configuration corresponding to the service is re-accessed, and additional releases are scheduled and executed for the service (i.e., the flock corresponding to the service). This is called the re-entry property of service deployment because the same service (or the same service flock) can be provisioned and deployed multiple times over time.
[0120] As described above, the successful execution of the release in 408 may result in the issuance of zero or more additional capabilities to the data center. In some cases, the new capabilities issued as a result of the execution of 408 may be another optional capability dependency of the same service (i.e., the same service flock). In such a situation, MFO106 may determine that a second optional capability dependency of the service has been satisfied and, in response, schedule yet another re-entry release of the service flock, following the process shown in Figure 4 for the execution of the second re-entry release.
[0121] Release / path integration In some situations, multiple capabilities may be issued within a small time window in which the issued capabilities are optional capability dependencies of a service flock. In this scenario, MFO106 may schedule multiple releases over time for the same service flock and create multiple paths for the multiple releases when optional capability dependencies become available. In one embodiment, the MFO may be configured to implement a scheduling queue for scheduling multiple releases for a service flock in the data center. When a release is scheduled for a particular flock for a particular service, MFO106 creates a path for that flock and adds this path to the scheduling queue. The CIOS central 108 then selects the releases scheduled for execution (e.g., paths corresponding to releases) from the queue, starting with the release at the top of the queue, and executes the releases in coordination with the CIOS regional 110.
[0122] The MFO may be configured to implement various types of scheduling queues for scheduling multiple releases for a service flock. Figure 5 is an exemplary diagram of a scheduling queue implemented by the MFO shown in Figure 1, according to one embodiment. In one embodiment, the MFO may be configured to generate a general-purpose scheduling queue 502 shared by different service flocks in a data center. For example, in the embodiment shown in Figure 5, the general-purpose scheduling queue 502 is shared by service A and service B in a data center and includes releases (SA,R1) 504 and (SA,R2) 508 scheduled for the execution of the flock for service A, and releases (SB,R1) 506 and (SB,R2) 510 scheduled for the execution of the flock for service B.
[0123] In one embodiment, the MFO may be configured to generate flock-specific queues for different service flocks in the data center. For example, as shown in Figure 5, scheduling queue 512 is a flock-specific queue for service A, containing scheduled releases (SA,R1) 504 and (SA,R2) 508 for the execution of flocks for service A. Scheduling queue 514 is a flock-specific queue for service B, containing scheduled releases (SB,R1) 506 and (SB,R2) 510 for the execution of flocks for service B. The scheduling queues 502, 512, and 514 shown in Figure 5 are for illustrative purposes only and are not limiting. In one alternative embodiment, the scheduling queues may be configured to store more or fewer scheduled releases for service flocks in the data center.
[0124] In the queue structure described above (i.e., scheduling queues 502, 512, and 514), for a given queue, there may be multiple unexecuted releases (multiple paths) for a given flock in the queue, depending on the rate at which MFO 106 schedules releases to the queue and the rate at which releases are selected from the queue for execution. In one embodiment, upon determining that multiple unexecuted releases exist in the queue (e.g., a flock-specific queue or a general-purpose scheduling queue), MFO 106 may, if possible, consolidate the multiple releases (paths) for the flock in the queue into a single unified release / path, replacing two or more releases / paths in the queue with the unified release / path for the flock. For example, in the embodiment shown in Figure 5, upon determining that two releases (SA,R1) 504 and (SA,R2) 508 scheduled for the execution of a flock for service A exist in the scheduling queue 502, the MFO 106 may, if possible, merge the two releases (paths) for the flock in queue 502 into a single unified release / path, and replace two or more releases / paths in the queue with the unified release / path. In some cases, it is also possible to merge the unified release / path with another release / path in the queue.
[0125] Figure 6 is an illustrative diagram of a unified release in a scheduling queue implemented by the MFO shown in Figure 1, according to one embodiment. As shown in Figure 6, upon determining that two releases (SA,R1) 504 and (SA,R2) 508 scheduled for execution exist in the queue (e.g., general-purpose scheduling queue 502), the MFO 106 may, if possible, unify the two releases (paths) in the queue into a single unified release / path and replace two or more releases / paths in the queue with the unified release / path 604. Similarly, upon determining that two releases (SB,R1) 506 and (SB,R2) 510 scheduled for execution exist in the general-purpose queue 502 for the flock of service B, the MFO 106 may, if possible, unify the two releases (paths) in the queue into a single unified release / path and replace two or more releases / paths in the queue with the unified release / path 606.
[0126] Integration is performed for efficiency. In many cases, the time required to run an integrated release / path is shorter than the total time required to run the individual integrated releases / paths. Also, in many cases, the compute, memory, and networking resources used to run an integrated release / path are less than the resources required to run the individual integrated releases / paths. This is mainly due to the processes involved in running a release / path. In one embodiment, running a release involves generating a plan from the current configuration to the target configuration expressed in the release, and then running this plan. Creating and running the plan is resource-intensive and time-consuming in terms of compute and memory resources, so it is more efficient to integrate multiple unrunned paths into a single path and then create and apply a single plan. for example, Capability C1 is issued at time T1. At time T2, the MFO creates the first pass / release (P1) of a flock (referred to as flock A) for service A, which is an optional capability dependency of C1, and places it in a queue specific to flock A or a general-purpose queue as described above. Capability C2 is issued at time T3 (close to T1). At time T4, the MFO creates a second pass / release (P2) for flock A, which is another optional capability dependency of C2, and places it in the same queue as flock A. The MFO then determines that two unexecuted paths, P1 and P2, exist in the queue for flock A. The MFO then merges P1 and P2 into a single path P3 and replaces P1 and P2 in the queue with the merged path P3. Path P3 is then provided to the CIOS for planning and execution.
[0127] Figure 7 is a simplified flowchart 700 illustrating the process performed to generate and execute an integrated release according to one embodiment. The process shown in Figure 7 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, or it may be implemented using hardware, or a combination of these. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods described later in Figure 7 are illustrative and not intended to be limiting. Figure 7 shows various processing steps occurring in a particular order or sequence, but not intended to be limiting. In an alternative embodiment, the process may be executed in some different order, or some steps may be executed in parallel. In one embodiment (such as the embodiment shown in Figure 1), the process shown in Figure 7 may be executed by the MFO 106 in coordination with the CIOS Central 108, the CIOS Regional 110, and the Capability Service 112.
[0128] In the embodiment shown in Figure 7, processing may be triggered when MFO106 determines that a new capability has been issued (for example, made available) in the data center. For example, in the embodiment shown in Figure 1, the capability service 112 is responsible for tracking issued capabilities.
[0129] In 704, MFO106 may determine that the newly issued capability identified in 702 is an optional capability dependency of a particular flock for a particular service in which the release of a particular flock was previously scheduled and successfully executed.
[0130] In 706, MFO106 schedules a new “re-entry” release for a particular flock in response to the issuance of an optional capability dependency and adds this new release to the queue. As previously stated, the MFO may add the new release to a general scheduling queue shared by multiple service flocks (e.g., 502) or to a queue specific to a particular flock for a particular service (e.g., 512 or 514).
[0131] In 708, the MFO 106 scans the queue to determine if there are any pending releases in the queue for a particular flock. In one example, at least one of the pending releases includes a new release of a flock scheduled by the MFO in 706.
[0132] In 710, MFO106 generates a flock unrelease by integrating the flock's unexecuted releases (i.e., unexecuted paths).
[0133] In 712, the MFO provides the integrated release to CIOS for the preparation and execution of the plan.
[0134] Exemplary Cloud Service Infrastructure Architecture As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also provide a variety of services associated with these infrastructure components (e.g., billing, monitoring, logging, load balancing, and clustering, etc.). Therefore, since these services can be policy-driven, IaaS users can maintain application availability and performance by implementing policies that promote load balancing.
[0135] In some cases, IaaS customers can access resources and services through a wide area network (WAN), such as the internet, and use the cloud provider's services to install other elements of their application stack. For example, a user can log into the IaaS platform and create virtual machines (VMs), install operating systems (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on those VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.
[0136] In most cases, the cloud computing model requires the participation of a cloud provider. This cloud provider may, but does not have to be, a third-party service specializing in providing IaaS (e.g., granting, leasing, or selling). Alternatively, an entity may choose to deploy a private cloud and become its own provider of infrastructure services.
[0137] In some cases, IaaS deployment is the process of placing a new application or a new version of an application on a prepared application server, etc. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, the customer may be responsible for handling (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be spun up on demand).
[0138] In some cases, IaaS provisioning represents acquiring the computers or virtual hosts to be used, and may even represent installing the necessary libraries or services on them. In most cases, a deployment does not include provisioning, and provisioning may need to be performed first.
[0139] In some cases, IaaS provisioning presents two distinct challenges. Firstly, there is the initial challenge of provisioning the initial set of infrastructure before anything is operational. Secondly, there is the challenge of evolving the existing infrastructure after all provisioning is complete (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be addressed by enabling the declarative definition of infrastructure configuration. In other words, the infrastructure (e.g., the required components and the way those components interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources, and how each resource operates as a whole) can be described declaratively. In some cases, once the topology is defined, a workflow may be generated to create and / or manage the various components described in the configuration files.
[0140] In some examples, infrastructure can consist of many interconnected elements. For instance, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., potential on-demand pools of configurable and / or shared computing resources). In some examples, there may also be one or more inbound / outbound traffic group rules provisioning that define how inbound and / or outbound network traffic is configured, as well as one or more virtual machines (VMs). Other infrastructure elements such as load balancers and databases may also be provisioned. As there is a demand for and / or addition of more infrastructure elements, the infrastructure can gradually evolve.
[0141] In some cases, the adoption of sequential deployment techniques can enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is desirable to be deployed to one or more (but often many) different generation environments (e.g., geographically diverse locations, sometimes even worldwide). However, in some examples, the infrastructure to which the code is deployed must be configured first. In some cases, manual provisioning, the use of provisioning tools for provisioning resources, and / or the use of deployment tools for deploying code after infrastructure provisioning are also possible.
[0142] Figure 8 is a block diagram 800 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 802 may be communicably coupled to a secure host tenancy 804 which may include a virtual cloud network (VCN) 806 and a secure host subnet 808. In some examples, the service operator 802 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistive devices (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) that run software such as Microsoft Windows Mobile® and / or a variety of mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and are capable of using the Internet, email, short message service (SMS), BlackBerry®, or other communication protocols. Alternatively, a general-purpose personal computer can serve as a client computing device, such as a personal computer and / or laptop computer running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. A workstation computer running any of the various commercially available UNIX® or UNIX-like operating systems can also serve as a client computing device, such as, but are not limited to, various GNU / Linux operating systems like Google Chrome OS.Alternatively or in addition, the client computing device may be any other electronic device, such as a thin client computer that can communicate via a network that can access the VCN806 and / or the Internet, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device.
[0143] VCN806 may comprise an LPG810 that can be communicatively coupled to SSH VCN812 via a local peering gateway (LPG)810 included in Secure Shell (SSH) VCN812. SSH VCN812 may comprise an SSH subnet 814, and SSH VCN812 may be communicatively coupled to control plane VCN816 via an LPG810 included in control plane VCN816. Furthermore, SSH VCN812 may be communicatively coupled to data plane VCN818 via an LPG810. Control plane VCN816 and data plane VCN818 may be included in a service tenancy 819 owned and / or operated by an IaaS provider.
[0144] The control plane VCN 816 may comprise a control plane demilitarized zone (DMZ) layer 820 that operates as a perimeter network (for example, part of the corporate network between the corporate intranet and the external network). DMZ-based servers can be limited in responsibility and help deter breaches. The DMZ layer 820 may also comprise a control plane application layer 824 that may include one or more load balancer (LB) subnets 822, an application subnet 826, and a control plane data layer 828 that may include a database (DB) subnet 830 (for example, a front-end DB subnet and / or a back-end DB subnet). The LB subnet 822 included in the control plane DMZ layer 820 may be communicatively coupled to the application subnet 826 included in the control plane application layer 824 and an internet gateway 834 that may be included in the control plane VCN 816, and the application subnet 826 may be communicatively coupled to the DB subnet 830, a service gateway 836, and a network address translation (NAT) gateway 838 included in the control plane data layer 828. The control plane VCN816 may include a service gateway 836 and a NAT gateway 838.
[0145] The control plane VCN 816 may comprise a data plane mirror application layer 840 that may include an application subnet 826. The application subnet 826 included in the data plane mirror application layer 840 may comprise a virtual network interface controller (VNIC) 842 that can run a compute instance 844. The compute instance 844 may be communicatively coupled to the application subnet 826 of the data plane mirror application layer 840 and to the application subnet 826 that may be included in the data plane application layer 846.
[0146] The data plane VCN818 may comprise a data plane application layer 846, a data plane DMZ layer 848, and a data plane data layer 850. The data plane DMZ layer 848 may comprise an LB subnet 822 that can be communicatively coupled to the application subnet 826 of the data plane application layer 846 and the internet gateway 834 of the data plane VCN818. The application subnet 826 may be communicatively coupled to the service gateway 836 and the NAT gateway 838 of the data plane VCN818. The data plane data layer 850 may comprise a DB subnet 830 that can be communicatively coupled to the application subnet 826 of the data plane application layer 846.
[0147] The Internet gateway 834 of the control plane VCN816 and data plane VCN818 can be communicatively coupled to a metadata management service 852, which can be communicatively coupled to the public internet 854. The public internet 854 can be communicatively coupled to the NAT gateway 838 of the control plane VCN816 and data plane VCN818. The service gateway 836 of the control plane VCN816 and data plane VCN818 can be communicatively coupled to a cloud service 856.
[0148] In some cases, a service gateway 836 of the control plane VCN816 or data plane VCN818 can make application programming interface (API) calls to a cloud service 856 without going through the public internet 854. API calls from the service gateway 836 to the cloud service 856 can be unidirectional. The service gateway 836 can make API calls to the cloud service 856, and the cloud service 856 can send the requested data to the service gateway 836. However, the cloud service 856 does not have to initiate an API call to the service gateway 836.
[0149] In some examples, secure host tenancy 804 may be directly connected to service tenancy 819, or otherwise isolated. Secure host subnet 808 may communicate with SSH subnet 814 via LPG 810, which may enable bidirectional communication through a system that is otherwise isolated. By connecting secure host subnet 808 to SSH subnet 814, secure host subnet 808 becomes able to access other entities within service tenancy 819.
[0150] The control plane VCN816 may enable users of the service tenancy 819 to configure or provision desired resources. Desired resources provisioned in the control plane VCN816 may be deployed or used in the data plane VCN818. In some examples, the control plane VCN816 can be isolated from the data plane VCN818, and the data plane mirror application layer 840 of the control plane VCN816 can communicate with the data plane application layer 846 of the data plane VCN818 via a VNIC 842 which may be included in the data plane mirror application layer 840 and the data plane application layer 846.
[0151] In some examples, a user or customer of the system can make a request (for example, perform create, read, update, or delete (CRUD) operations) through the public internet 854, and the public internet 854 can send the request to the metadata management service 852. The metadata management service 852 can send the request to the control plane VCN 816 through the internet gateway 834. The request may be received by the LB subnet 822 included in the control plane DMZ layer 820. The LB subnet 822 may determine that the request is valid, and in response to this determination, the LB subnet 822 may send the request to the application subnet 826 included in the control plane application layer 824. If the request is validated and a call to the public internet 854 is required, the call to the public internet 854 may be sent to a NAT gateway 838 that can make a call to the public internet 854. Metadata that is deemed desirable to be stored by the request may be stored in the DB subnet 830.
[0152] In some cases, the data plane mirror application layer 840 may facilitate direct communication between the control plane VCN816 and the data plane VCN818. For example, it may be desirable to apply configuration changes, updates, or other preferred modifications to resources contained in the data plane VCN818. The control plane VCN816 can perform configuration changes, updates, or other preferred modifications to resources by communicating directly with the resources contained in the data plane VCN818 via VNIC842.
[0153] In some embodiments, the control plane VCN816 and the data plane VCN818 may be included in the service tenancy 819. In this case, the system user, i.e., the customer, does not have to own either the control plane VCN816 or the data plane VCN818, or does not have either of them running. Alternatively, the IaaS provider may own both the control plane VCN816 and the data plane VCN818, or have both running, or both may be included in the service tenancy 819. This embodiment may enable network isolation that can prevent interaction between the user, i.e., the customer, and other users, i.e., other customers' resources. Furthermore, this embodiment may enable private storage of databases by the system user, i.e., the customer, without having to rely on the public internet 854, which may not have the desired level of threat prevention for storage.
[0154] In another embodiment, the LB subnet 822 included in the control plane VCN 816 can be configured to receive signals from the service gateway 836. In this embodiment, the control plane VCN 816 and the data plane VCN 818 may be configured to be invoked by the IaaS provider's customer without calling the public internet 854. The IaaS provider's customer is likely to prefer this embodiment because the database used by the customer may be stored in a service tenancy 819 that is controlled by the IaaS provider and can be isolated from the public internet 854.
[0155] Figure 9 is a block diagram 900 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 902 (for example, service operator 802 in Figure 8) may be communicatively coupled to a secure host tenancy 904 (for example, secure host tenancy 804 in Figure 8), which may include a virtual cloud network (VCN) 906 (for example, VCN806 in Figure 8) and a secure host subnet 908 (for example, secure host subnet 808 in Figure 8). The VCN 906 may comprise an LPG 910, which may be communicatively coupled to an SSH VCN 912 (for example, SSH VCN812 in Figure 8) via a local peering gateway (LPG) 810 (for example, LPG810 in Figure 8). SSH VCN912 may comprise an SSH subnet 914 (for example, SSH subnet 814 in Figure 8), and SSH VCN912 may be communicably coupled to control plane VCN916 (for example, control plane VCN816 in Figure 8) via an LPG 910 included in control plane VCN916. Control plane VCN916 may be included in service tenancy 919 (for example, service tenancy 819 in Figure 8), and data plane VCN918 (for example, data plane VCN818 in Figure 8) may be included in customer tenancy 921, which may be owned or operated by the system's users, i.e., customers.
[0156] The control plane VCN916 may comprise a control plane DMZ layer 920 (for example, the control plane DMZ layer 820 in Figure 8) which may include an LB subnet 922 (for example, the LB subnet 822 in Figure 8), a control plane application layer 924 (for example, the control plane application layer 824 in Figure 8) which may include an application subnet 926 (for example, the application subnet 826 in Figure 8), and a control plane data layer 928 (for example, the control plane data layer 828 in Figure 8) which may include a DB subnet 930 (similar to the database (DB) subnet 830 in Figure 8). The LB subnet 922 included in the control plane DMZ layer 920 is communicatively coupled to the application subnet 926 included in the control plane application layer 924 and the Internet gateway 934 (for example, Internet gateway 834 in Figure 8) which may be included in the control plane VCN 916. The application subnet 926 may be communicatively coupled to the DB subnet 930 included in the control plane data layer 928, the service gateway 936 (for example, service gateway 836 in Figure 8), and the Network Address Translation (NAT) gateway 938 (for example, NAT gateway 838 in Figure 8). The control plane VCN 916 may comprise the service gateway 936 and the NAT gateway 938.
[0157] The control plane VCN916 may comprise a data plane mirror application layer 940 (for example, the data plane mirror application layer 840 in Figure 8) which may include an application subnet 926. The application subnet 926 included in the data plane mirror application layer 940 may comprise a virtual network interface controller (VNIC) 942 (for example, VNIC842) which can run a compute instance 944 (for example, similar to compute instance 844 in Figure 8). The compute instance 944 may facilitate communication between the application subnet 926 of the data plane mirror application layer 940 and the application subnet 926 that may be included in the data plane application layer 946 via the VNIC942 included in the data plane mirror application layer 940 and the VNIC942 included in the data plane application layer 946 (for example, the data plane application layer 846 in Figure 8).
[0158] The Internet gateway 934 included in the control plane VCN916 may be communicably coupled to a metadata management service 952 (e.g., metadata management service 852 in Figure 8), which may be communicably coupled to the public internet 954 (e.g., public internet 854 in Figure 8). The public internet 954 may be communicably coupled to a NAT gateway 938 included in the control plane VCN916. The service gateway 936 included in the control plane VCN916 may be communicably coupled to a cloud service 956 (e.g., cloud service 856 in Figure 8).
[0159] In some examples, the data plane VCN918 may be included in a customer tenancy 921. In this case, the IaaS provider may provide a control plane VCN916 for each customer, or the IaaS provider may configure a unique compute instance 944 included in a service tenancy 919 for each customer. Each compute instance 944 may enable communication between the control plane VCN916 included in the service tenancy 919 and the data plane VCN918 included in the customer tenancy 921. The compute instance 944 may enable the deployment or use of resources provisioned in the control plane VCN916 included in the service tenancy 919 in the data plane VCN918 included in the customer tenancy 921.
[0160] In another example, the IaaS provider's customer may have a database located in customer tenancy 921. In this example, the control plane VCN916 may comprise a data plane mirror application layer 940, which may include application subnet 926. The data plane mirror application layer 940 may reside in data plane VCN918, but does not have to. That is, the data plane mirror application layer 940 may be accessible to customer tenancy 921, but does not have to reside in data plane VCN918, nor is it owned or operated by the IaaS provider's customer. The data plane mirror application layer 940 may be configured to make calls to data plane VCN918, but does not have to be configured to make calls to any entity included in control plane VCN916. The customer is expected to want to deploy or use resources in the data plane VCN918 that are provisioned in the control plane VCN916, and the data plane mirror application layer 940 can facilitate the deployment or other use of the resources desired by the customer.
[0161] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN918. In this embodiment, the customer can determine what the data plane VCN918 can access, and may also restrict access from the data plane VCN918 to the public internet 954. The IaaS provider may not be able to apply filters or control the data plane VCN918's access to any external network or database. The application of filters and controls by the customer to the data plane VCN918 included in the customer tenancy 921 may help isolate the data plane VCN918 from other customers and the public internet 954.
[0162] In some embodiments, the cloud service 956 becomes accessible by a call from the service gateway 936 to a service that could not exist on the public internet 954, the control plane VCN916, or the data plane VCN918. The connection between the cloud service 956 and the control plane VCN916 or data plane VCN918 does not have to be live or continuous. The cloud service 956 may reside on different networks owned or operated by the IaaS provider. The cloud service 956 may be configured to receive calls from the service gateway 936 and may be configured not to receive calls from the public internet 954. Some cloud services 956 may be isolated from other cloud services 956, and the control plane VCN916 may be isolated from cloud services 956 that could not be in the same region as the control plane VCN916. For example, the control plane VCN916 may be located in "Region 1", and the cloud service "Deployment 8" may be located in Region 1 and "Region 2". If a call to deployment 8 is made by a service gateway 936 included in the control plane VCN916 located in region 1, the call may be sent to deployment 8 in region 1. In this example, the control plane VCN916 or deployment 8 in region 1 does not have to be communicatively coupled to or in communication with deployment 8 in region 2.
[0163] Figure 10 is a block diagram 1000 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1002 (e.g., service operator 802 in Figure 8) may be communicatively coupled to a secure host tenancy 1004 (e.g., secure host tenancy 804 in Figure 8), which may include a virtual cloud network (VCN) 1006 (e.g., VCN806 in Figure 8) and a secure host subnet 1008 (e.g., secure host subnet 808 in Figure 8). VCN 1006 may comprise an LPG 1010 (e.g., LPG810 in Figure 8), which may be communicatively coupled to SSH VCN 1012 (e.g., SSH VCN812 in Figure 8). SSH VCN1012 may comprise an SSH subnet 1014 (for example, SSH subnet 814 in Figure 8), and SSH VCN1012 may be communicatively coupled to control plane VCN1016 (for example, control plane VCN816 in Figure 8) via an LPG 1010 included in control plane VCN1016, and may also be communicatively coupled to data plane VCN1018 (for example, data plane VCN818 in Figure 8) via an LPG 1010 included in data plane VCN1018. Control plane VCN1016 and data plane VCN1018 may be included in service tenancy 1019 (for example, service tenancy 819 in Figure 8).
[0164] The control plane VCN1016 may comprise a control plane DMZ layer 1020 (e.g., control plane DMZ layer 820 in Figure 8) which may include a load balancer (LB) subnet 1022 (e.g., LB subnet 822 in Figure 8), a control plane application layer 1024 (e.g., control plane application layer 824 in Figure 8) which may include an application subnet 1026 (e.g., similar to application subnet 826 in Figure 8), and a control plane data layer 1028 (e.g., control plane data layer 828 in Figure 8) which may include a DB subnet 1030. The LB subnet 1022 included in the control plane DMZ layer 1020 may be communicatively coupled to the application subnet 1026 included in the control plane application layer 1024 and to an internet gateway 1034 (for example, internet gateway 834 in Figure 8) which may be included in the control plane VCN 1016. The application subnet 1026 may be communicatively coupled to the DB subnet 1030 included in the control plane data layer 1028, to a service gateway 1036 (for example, the service gateway in Figure 8), and to a network address translation (NAT) gateway 1038 (for example, the NAT gateway 838 in Figure 8). The control plane VCN 1016 may comprise the service gateway 1036 and the NAT gateway 1038.
[0165] The data plane VCN 1018 may comprise a data plane application layer 1046 (for example, the data plane application layer 846 in Figure 8), a data plane DMZ layer 1048 (for example, the data plane DMZ layer 848 in Figure 8), and a data plane data layer 1050 (for example, the data plane data layer 850 in Figure 8). The data plane DMZ layer 1048 may comprise an LB subnet 1022 that can be communicatively coupled to the trusted application subnet 1060 and the untrusted application subnet 1062 of the data plane application layer 1046, as well as the internet gateway 1034 included in the data plane VCN 1018. The trusted application subnet 1060 may be communicatively coupled to the service gateway 1036 included in the data plane VCN 1018, the NAT gateway 1038 included in the data plane VCN 1018, and the DB subnet 1030 included in the data plane data layer 1050. The non-trusted application subnet 1062 can be communicatively coupled to the service gateway 1036 included in the data plane VCN 1018 and the DB subnet 1030 included in the data plane data layer 1050. The data plane data layer 1050 may comprise the DB subnet 1030, which can be communicatively coupled to the service gateway 1036 included in the data plane VCN 1018.
[0166] The untrusted application subnet 1062 may comprise one or more primary VNICs 1064(1) to 1064(N) that can be communicatively coupled to tenant virtual machines (VMs) 1066(1) to 1066(N). Each tenant VM 1066(1) to 1066(N) may be communicatively coupled to each application subnet 1067(1) to 1067(N) that can be contained within each container output VCN 1068(1) to 1068(N) that can be contained within each customer tenancy 1070(1) to 1070(N). Each secondary VNIC 1072(1) to 1072(N) can facilitate communication between the untrusted application subnet 1062 contained within the data plane VCN 1018 and the application subnet contained within the container output VCN 1068(1) to 1068(N). Each container output VCN 1068(1) to 1068(N) may be equipped with a NAT gateway 1038 that can be communicably connected to the public internet 1054 (for example, the public internet 854 in Figure 8).
[0167] The Internet gateway 1034 included in the control plane VCN1016 and data plane VCN1018 can be communicatively coupled to a metadata management service 1052 (for example, the metadata management system 852 in Figure 8), which can be communicatively coupled to the public internet 1054. The public internet 1054 can be communicatively coupled to a NAT gateway 1038 included in the control plane VCN1016 and data plane VCN1018. The service gateway 1036 included in the control plane VCN1016 and data plane VCN1018 can be communicatively coupled to a cloud service 1056.
[0168] In some embodiments, the data plane VCN 1018 may be integrated with a customer tenancy 1070. This integration may be useful or desirable for the IaaS provider's customers, for example, if they may want support for executing code. The customer may provide code to be executed, which may be destructive, communicate with other customer resources, or have undesirable effects. In response, the IaaS provider may determine whether to execute the code provided to the IaaS provider by the customer.
[0169] In some examples, an IaaS provider's customer may grant temporary network access to the IaaS provider and request functionality to be granted to the data plane application layer 1046. The code that performs this functionality may run in VMs 1066(1) to 1066(N), and may not be configured to run elsewhere on the data plane VCN 1018. Each VM 1066(1) to 1066(N) may be connected to a single customer tenancy 1070. Each container 1071(1) to 1071(N) contained within VMs 1066(1) to 1066(N) may be configured to execute code. In this case, a double isolation may exist (containers 1071(1)-1071(N) executing the code may be contained within VMs 1066(1)-1066(N) that are at least in the non-trusted app subnet 1062), which may help prevent damage to the IaaS provider's network or a different customer's network by erroneous or undesirable code. Containers 1071(1)-1071(N) may be communicatively coupled to customer tenancy 1070 and may be configured to send or receive data to or from customer tenancy 1070. Containers 1071(1)-1071(N) do not have to be configured to send or receive data to or from any other entity in the data plane VCN 1018. Upon completion of code execution, the IaaS provider may disable or discard containers 1071(1)-1071(N).
[0170] In some embodiments, the trusted application subnet 1060 may be configured to execute code owned or operated by the IaaS provider. In this embodiment, the trusted application subnet 1060 may be communicatively coupled to the DB subnet 1030 and may be configured to perform CRUD operations in the DB subnet 1030. The non-trusted application subnet 1062 may be communicatively coupled to the DB subnet 1030, but in this embodiment, may be configured to perform read operations in the DB subnet 1030. Containers 1071(1) to 1071(N) contained in each customer's VMs 1066(1) to 1066(N) and capable of executing code from the customer may not be communicatively coupled to the DB subnet 1030.
[0171] In other embodiments, the control plane VCN1016 and the data plane VCN1018 do not have to be directly communicatively coupled. In this embodiment, direct communication between the control plane VCN1016 and the data plane VCN1018 is not required; however, communication can be performed indirectly by at least one method. An LPG1010 that can facilitate communication between the control plane VCN1016 and the data plane VCN1018 may be established by the IaaS provider. In another example, the control plane VCN1016 or the data plane VCN1018 can make a call to the cloud service 1056 via the service gateway 1036. For example, a call from the control plane VCN1016 to the cloud service 1056 may include a request for a service that can communicate with the data plane VCN1018.
[0172] Figure 11 is a block diagram 1100 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1102 (e.g., service operator 802 in Figure 8) may be communicatively coupled to a secure host tenancy 1104 (e.g., secure host tenancy 804 in Figure 8), which may include a virtual cloud network (VCN) 1106 (e.g., VCN806 in Figure 8) and a secure host subnet 1108 (e.g., secure host subnet 808 in Figure 8). VCN 1106 may comprise an LPG 1110 (e.g., LPG810 in Figure 8), which is contained in SSH VCN 1112 (e.g., SSH VCN812 in Figure 8). SSH VCN1112 may comprise an SSH subnet 1114 (for example, SSH subnet 814 in Figure 8), and SSH VCN1112 may be communicatively coupled to control plane VCN1116 (for example, control plane VCN816 in Figure 8) via an LPG 1110 included in control plane VCN1116, and may also be communicatively coupled to data plane VCN1118 (for example, data plane VCN818 in Figure 8) via an LPG 1110 included in data plane VCN1118. Control plane VCN1116 and data plane VCN1118 may be included in service tenancy 1119 (for example, service tenancy 819 in Figure 8).
[0173] The control plane VCN1116 may comprise a control plane DMZ layer 1120 (for example, the control plane DMZ layer 820 in Figure 8) which may include an LB subnet 1122 (for example, the LB subnet 822 in Figure 8), a control plane application layer 1124 (for example, the control plane application layer 824 in Figure 8) which may include an application subnet 1126 (for example, the application subnet 826 in Figure 8), and a control plane data layer 1128 (for example, the control plane data layer 828 in Figure 8) which may include a DB subnet 1130 (for example, the DB subnet 1030 in Figure 10). The LB subnet 1122 included in the control plane DMZ layer 1120 is communicatively coupled to the application subnet 1126 included in the control plane application layer 1124 and the Internet gateway 1134 (for example, Internet gateway 834 in Figure 8) which may be included in the control plane VCN 1116. The application subnet 1126 may be communicatively coupled to the DB subnet 1130 included in the control plane data layer 1128, the service gateway 1136 (for example, the service gateway in Figure 8), and the Network Address Translation (NAT) gateway 1138 (for example, the NAT gateway 838 in Figure 8). The control plane VCN 1116 may comprise the service gateway 1136 and the NAT gateway 1138.
[0174] The data plane VCN 1118 may comprise a data plane application layer 1146 (for example, the data plane application layer 846 in Figure 8), a data plane DMZ layer 1148 (for example, the data plane DMZ layer 848 in Figure 8), and a data plane data layer 1150 (for example, the data plane data layer 850 in Figure 8). The data plane DMZ layer 1148 may comprise a trusted application subnet 1160 (for example, the trusted application subnet 1060 in Figure 10) and an untrusted application subnet 1162 (for example, the untrusted application subnet 1062 in Figure 10) of the data plane application layer 1146, as well as an LB subnet 1122 that can be communicatively coupled to an internet gateway 1134 included in the data plane VCN 1118. Trusted application subnet 1160 may be communicatively coupled to service gateway 1136 included in data plane VCN 1118, NAT gateway 1138 included in data plane VCN 1118, and DB subnet 1130 included in data plane data layer 1150. Non-trusted application subnet 1162 may be communicatively coupled to service gateway 1136 included in data plane VCN 1118 and DB subnet 1130 included in data plane data layer 1150. Data plane data layer 1150 may comprise DB subnet 1130 which can be communicatively coupled to service gateway 1136 included in data plane VCN 1118.
[0175] The untrusted application subnet 1162 may comprise primary VNICs 1164(1) to 1164(N) that can be communicatively coupled to tenant virtual machines (VMs) 1166(1) to 1166(N) residing within the untrusted application subnet 1162. Each tenant VM 1166(1) to 1166(N) is capable of executing code in each container 1167(1) to 1167(N) and can be communicatively coupled to an application subnet 1126 that may be included in the data plane application layer 1146 that may be included in the container output VCN 1168. The secondary VNICs 1172(1) to 1172(N) can facilitate communication between the untrusted application subnet 1162 included in the data plane VCN 1118 and the application subnet included in the container output VCN 1168. The container output VCN may include a NAT gateway 1138 that can be communicably connected to the public internet 1154 (for example, the public internet 854 in Figure 8).
[0176] The Internet gateway 1134 included in the control plane VCN1116 and data plane VCN1118 can be communicatively coupled to a metadata management service 1152 (for example, the metadata management system 852 in Figure 8), which can be communicatively coupled to the public internet 1154. The public internet 1154 can be communicatively coupled to a NAT gateway 1138 included in the control plane VCN1116 and data plane VCN1118. The service gateway 1136 included in the control plane VCN1116 and data plane VCN1118 can be communicatively coupled to a cloud service 1156.
[0177] In some examples, the pattern shown by the architecture in block diagram 1100 of Figure 11 is considered an exception to the pattern shown by the architecture in block diagram 1000 of Figure 10, and is considered desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customer (for example, in an unconnected region). Each customer has containers 1167(1) to 1167(N) contained within VMs 1166(1) to 1166(N), each of which is accessible to the customer in real time. Each container 1167(1) to 1167(N) may be configured to make calls to each of the secondary VNICs 1172(1) to 1172(N) contained within the application subnet 1126 of the data plane application layer 1146, which may be contained within the container output VCN 1168. Secondary VNICs 1172(1) to 1172(N) can send calls to the NAT gateway 1138, which may then send the calls to the public internet 1154. In this example, the customer-accessible containers 1167(1) to 1167(N) can be isolated from the control plane VCN 1116, as well as from other entities included in the data plane VCN 1118. Furthermore, the containers 1167(1) to 1167(N) can also be isolated from other customer resources.
[0178] In another example, a customer can use containers 1167(1) to 1167(N) to invoke cloud service 1156. In this example, a customer can execute code in containers 1167(1) to 1167(N) to request services from cloud service 1156. Containers 1167(1) to 1167(N) can send this request to secondary VNICs 1172(1) to 1172(N), which can send the request to a NAT gateway, which can send the request to the public internet 1154. The public internet 1154 can send the request to LB subnet 1122, which is included in control plane VCN 1116, via internet gateway 1134. In response to the determination that the request is valid, the LB subnet can send the request to the application subnet 1126, and the application subnet 1126 can send the request to the cloud service 1156 via the service gateway 1136.
[0179] Naturally, the IaaS architectures 800, 900, 1000, and 1100 shown in the drawings may have components other than those shown. Furthermore, the embodiments shown in the drawings are only a few examples of cloud infrastructure systems that may encompass one embodiment of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown, may combine two or more components, or may have different configurations or arrangements of components.
[0180] In one embodiment, the IaaS system described herein may include the provision of a set of applications, middleware, and database services delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. Oracle Cloud Infrastructure (OCI), offered by the assignee, is an example of such an IaaS system.
[0181] Figure 12 shows an exemplary computer system 1200 in which various embodiments can be realized. System 1200 may be used to realize any of the computer systems described above. As shown in the drawing, computer system 1200 comprises a processing unit 1204 that communicates with many peripheral subsystems via a bus subsystem 1202. Peripheral subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 comprises a tangible computer-readable storage medium 1222 and a system memory 1210.
[0182] The bus subsystem 1202 provides a mechanism for various components and subsystems of the computer system 1200 to communicate with each other as intended. Although the bus subsystem 1202 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1202 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, and local bus, using any of a variety of bus architectures. For example, such architectures may include industry standard architecture (ISA) buses, microchannel architecture (MCA) buses, extended ISA (EISA) buses, video electronics standards (VESA) local buses, and peripheral interconnect (PCI) buses, which can be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard.
[0183] A processing unit 1204, which can be implemented as one or more integrated circuits (for example, conventional microprocessors or microcontrollers), controls the operation of the computer system 1200. The processing unit 1204 may include one or more processors. These processors may include single-core or multi-core processors. In one embodiment, the processing unit 1204 may be implemented as one or more independent processing units 1232 and / or 1234, each containing a single-core or multi-core processor. In another embodiment, the processing unit 1204 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.
[0184] In various embodiments, the processing unit 1204 can execute a variety of programs in response to program code and maintain multiple concurrently running programs or processes. At any given time, some or all of the program code to be executed may reside in the processor 1204 and / or the storage subsystem 1218. With suitable programming, the processor 1204 can provide the various functions described above. The computer system 1200 may also include a processing acceleration unit 1206 which may include a digital signal processor (DSP), a dedicated processor, and / or similar.
[0185] The I / O subsystem 1208 may include user interface input devices and user interface output devices. User interface input devices may include pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include motion detection and / or gesture recognition devices such as Microsoft Kinect® motion sensors that enable user control and interaction with input devices such as Microsoft Xbox® 360 game controllers through a natural user interface using gestures and voice commands. User interface input devices may also include eye gesture recognition devices such as Google Glass® blink detectors that detect the user's eye activity (e.g., blinking during shooting and / or menu selection) and convert eye gestures into input to an input device (e.g., Google Glass®). Furthermore, the user interface input device may include a voice recognition detection device that enables user interaction with a voice recognition system (e.g., Siri® Navigator) via voice commands.
[0186] Furthermore, user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as auditory / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. Furthermore, user interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.
[0187] User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may also include flat panel devices such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. Generally, the use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1200 to a user or another computer. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.
[0188] The computer system 1200 may include a storage subsystem 1218 containing software elements, which are shown here as currently located in the system memory 1210. The system memory 1210 may store program instructions that can be loaded and executed by the processing unit 1204, as well as data generated during the execution of these programs.
[0189] Depending on the configuration and type of the computer system 1200, the system memory 1210 may be volatile (e.g., random access memory (RAM)) and / or non-volatile (e.g., read-only memory (ROM), flash memory). RAM typically contains data and / or program modules that are immediately accessible to the processing unit 1204, as well as data and / or program modules that are currently being operated and executed by the processing unit 1204. In some embodiments, the system memory 1210 may contain several different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some embodiments, ROM may typically store a basic input / output system (BIOS) containing basic routines that help transmit information between elements within the computer system 1200, such as at startup. Also, as a non-limiting example, the system memory 1210 may include application programs 1212, program data 1214, and an operating system 1216, which may include client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc. As an example, Operating System 1216 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, a variety of commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.
[0190] Furthermore, the storage subsystem 1218 may provide a tangible computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some embodiments. The storage subsystem 1218 may also store software (programs, code modules, instructions) that, when executed by the processor, provides the functionality described above. These software modules or instructions may be executed by the processing unit 1204. The storage subsystem 1218 may also provide a repository for storing data used in accordance with this disclosure.
[0191] Furthermore, the storage subsystem 1200 may include a computer-readable storage medium reader 1220 which can be further connected to the computer-readable storage medium 1222. Together, optionally, the computer-readable storage medium 1222, in combination with the system memory 1210, may comprehensively represent storage media for temporarily and / or permanently containing, storing, transmitting, and retrieving computer-readable information, as well as remote, local, fixed, and / or removable storage devices.
[0192] Furthermore, the computer-readable storage medium 1222 containing code or a portion of code may include, but is not limited to, any suitable medium known or used in the art (including storage and communication media), including volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or other tangible computer-readable media. This may also include non-tangible computer-readable media such as any other medium that can be used for transmitting data signals, data transmission, or desired information and is accessible by the computer system 1200.
[0193] As an example, the computer-readable storage medium 1222 may include a hard disk drive that reads and writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads and writes to a removable non-volatile magnetic disk, and an optical disk drive that reads and writes to a removable non-volatile optical disk such as a CD-ROM, DVD, Blu-ray® disc, or other optical medium. The computer-readable storage medium 1222 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Furthermore, the computer-readable storage medium 1222 may include SSDs based on volatile memory such as flash memory-based SSDs, enterprise flash drives, semiconductor drives (SSDs) based on non-volatile memory such as semiconductor ROM, semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data to the computer system 1200.
[0194] The communication subsystem 1224 provides interfaces to other computer systems and networks. The communication subsystem 1224 serves as an interface for sending and receiving data between the computer system 1200 and other systems. For example, the communication subsystem 1224 may enable the computer system 1200 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1224 may comprise radio voice and / or data networks (using, for example, cellular technology, 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (using advanced data network technologies such as the IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof), a Global Positioning System (GPS) receiver component, and / or radio frequency (RF) transceiver components for accessing other components. In some embodiments, the communication subsystem 1224 may provide wired network connectivity (e.g., Ethernet) as an addition to or alternative to the radio interface.
[0195] In addition, in some embodiments, the communication subsystem 1224 is capable of receiving input communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., on behalf of one or more users who may be using the computer system 1200.
[0196] For example, the communication subsystem 1224 may be configured to receive data feeds 1226 in real time from users of other communication services, such as social networks and / or web feeds like Twitter® feeds, Facebook® updates, RSS (Rich Site Summary) feeds, and / or real-time updates from one or more third-party sources.
[0197] Furthermore, the communication subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, which may be continuous or virtually infinite with no explicit end. Examples of applications that generate continuous data include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.
[0198] Furthermore, the communication subsystem 1224 may be configured to output structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that can communicate with one or more streaming data source computers connected to the computer system 1200.
[0199] The computer system 1200 can be one of a variety of types, including portable handheld devices (such as iPhone® mobile phones, iPad® computing tablets, and PDAs), wearable devices (such as Google Glass® head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing systems.
[0200] Due to the constantly changing nature of computers and networks, the description of the illustrated computer system 1200 is intended only as an example. Many other configurations are possible, whether with more or fewer components than the illustrated system. For example, customized hardware may be used, and / or specific elements in hardware, firmware, software (including applets), or combinations may be implemented. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosures and teachings contained herein, those skilled in the art will recognize other methods and / or techniques for realizing various embodiments.
[0201] While specific embodiments have been described above, various modifications, changes, alternative configurations, and equivalents are also included in the scope of this disclosure. The embodiments are not limited to operation within a particular data processing environment, but can freely operate within multiple data processing environments. Furthermore, while the embodiments have been described using a specific set of transactions and steps, it will be clear to those skilled in the art that the scope of this disclosure is not limited to the set of transactions and steps described. Various features and aspects of the embodiments described above may be used individually or in combination.
[0202] Furthermore, while embodiments have been described using specific combinations of hardware and software, it will be recognized that other combinations of hardware and software are also included in the scope of this disclosure. Embodiments may be implemented in hardware only, in software only, or by using a combination of these. The various processes described herein may be implemented on the same processor or on any combination of different processors. Thus, where a component or module is described as being configured to perform a certain operation, such configuration may be achieved, for example, by designing an electronic circuit to perform this operation, by programming a programmable electronic circuit (such as a microprocessor) to perform this operation, or by any combination thereof. Processes may communicate using a variety of techniques, including but not limited to prior art for inter-process communication. Also, different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.
[0203] Embodiments may be realized by using a computer program product that, when executed by a processor, includes a computer program / instruction that causes the processor to execute any of the methods described herein.
[0204] Accordingly, this specification and the drawings are intended to be illustrative and not limiting. However, it is clear that additions, differences, deletions, and other modifications and changes may be made without departing from the broader idea and scope set forth in the claims. For this reason, specific embodiments of this disclosure have been described, but these are not intended to be limiting. Various modifications and equivalents are included in the following claims.
[0205] In the context describing embodiments of the disclosure (in particular, in the context of the following claims), the terms “a,” “an,” and “the,” and similar reference subjects, shall be interpreted to encompass both singular and plural forms unless otherwise indicated herein or there is a clear contextual inconsistency. The terms “comprising,” “having,” “including,” and “containing” shall be interpreted as open-ended terms (i.e., “including, but not limited to,”) unless otherwise indicated herein. The term “connected” shall be interpreted as including, attaching, or integrally combining, in whole or in part, even if something is intervening. Unless otherwise indicated herein, descriptions of ranges of values are intended merely as a concise way of referring individually to each distinct value contained within that range, and each distinct value is incorporated herein as if it were individually described herein. Unless otherwise indicated herein or there is a clear contextual inconsistency, all methods described herein may be performed in any preferred order. The use of any examples or illustrative expressions (e.g., "such as") described herein is intended solely to facilitate the understanding of the embodiments and, unless otherwise claimed, does not limit the scope of this disclosure. Nothing described herein should be construed as indicating that any non-claimed element is essential to the implementation of this disclosure.
[0206] Unless otherwise specified, disjunctive expressions such as the phrase "at least one of X, Y, or Z" are intended to be understood in contexts in which they are commonly used to indicate that an item, term, etc., can be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive expressions are not intended, nor should they, to imply, that a particular embodiment requires the presence of at least one of X, at least one of Y, or at least one of Z.
[0207] This specification describes preferred embodiments of the Disclosure, including the best known modes for the execution of the Disclosure. Those skilled in the art will be able to see, by reading the above description, variations of these preferred embodiments. Those skilled in the art may adopt such variations as appropriate, and the Disclosure may be executed in a manner different from the specific description herein. Therefore, to the extent permitted by applicable law, this Disclosure includes all modifications and equivalents of the subject matter described in the claims appended herein. Furthermore, unless otherwise indicated herein, any combination of all possible modifications of the elements described above is included in this Disclosure.
[0208] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as they are incorporated herein by reference, with each reference being individually and specifically indicated.
[0209] While the above specification describes aspects of the disclosure with reference to specific embodiments, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein, without departing from the broader concept and scope of this specification. Accordingly, this specification and the drawings should be considered illustrative and not limiting.
Claims
1. A method performed by a computer system, The aforementioned computer system determines that a new capability has been issued in the data center, The computer system identifies that the newly issued capability is an optional capability dependency of a service flock. Includes, The first release of the aforementioned flock has been scheduled and executed in the past, and the flock includes a set of one or more resources for providing the service. The method performed by the aforementioned computer system is: The computer system, in response to the identification, schedules a second release of the flock. The computer system performs the second release of the flock. Methods performed by computer systems, including, further.
2. The method performed by the computer system according to claim 1, wherein the first release is scheduled and executed by the computer system when all the necessary capability dependencies of the flock have been issued.
3. The method executed by the computer system according to claim 2, wherein the necessary capability dependencies of the flock are identified by the computer system as capabilities issued in the data center before the first release of the flock is scheduled and executed by the computer system.
4. The method performed by the computer system according to claim 1, wherein the optional capability dependency of the flock is identified by the computer system as a capability that does not need to be issued in the data center before the first release of the flock is scheduled and executed by the computer system.
5. The method performed by the computer system according to claim 1, wherein the set of one or more resources includes infrastructure components and artifacts associated with the service.
6. A method performed by a computer system according to claim 1, wherein information identifying the flock for the service is declared in the flock setting for the service, and the flock setting for the service identifies one or more of the set of resources associated with the service.
7. A method performed by the computer system according to claim 1, wherein the newly issued capability identifies a functional unit associated with the flock of the service and identifies one or more of the set of resources associated with the service.
8. The method performed by the aforementioned computer system is: The aforementioned computer system obtains a dependency graph, The computer system uses the dependency graph to identify that the newly issued capability is an optional capability dependency of the flock for the service. It further includes, The method performed by the computer system according to claim 1, wherein the dependency graph is an acyclic dependency graph.
9. Performing the second release mentioned above means The computer system includes generating a plan, The computer system identifies the current configuration of the data center and the target configuration of the data center, the plan being associated with the execution of the second release. To execute the aforementioned plan A method performed by the computer system according to claim 1, further comprising:
10. A method performed by the computer system according to claim 9, wherein the execution of the plan results in the issuance of additional new capabilities in the data center.
11. The method performed by the computer system according to claim 9, further comprising updating metadata associated with the flock settings of the flock for the service in response to the computer system performing the plan.
12. It is a cloud infrastructure organization system, Memory and One or more processors configured to perform processing A cloud infrastructure organization system comprising, wherein the processing is performed by a computer system as described in any one of claims 1 to 11.
13. A program that causes one or more processors to perform a method performed by a computer system as described in any one of claims 1 to 11.
Citation Information
Patent Citations
Method and Apparatus for On-Demand Synthesis and Decomposition of Service Infrastructure
JP2009540469A
Management of delivery of code and dependent data using application container
JP2020074139A
Techniques for managing dependencies of an orchestration service
US20210224076A1
Techniques for utilizing directed acyclic graphs for deployment instructions
WO2021150435A1
