Methods, systems, media, and apparatuses for deploying infrastructure resources using a declarative provisioning tool

Abstract

This describes a technique for implementing an infrastructure orchestration service. The scheduler can receive a configuration file for a first deployment of resources at an execution target, which may include a first publication identifier. Resources can be deployed at the execution target according to the file. The current state of the resources can be stored. The scheduler can receive a second version of the file for a new deployment at the execution target, which may include a second publication identifier. At least one worker node can execute a plugin to compare the first identifier with the second identifier. If the first identifier differs from the second identifier, the plugin can compare the current state of the resource with the expected state based on the second identifier. If the expected state differs from the current state, the resource is deployed at the execution target according to the second identifier.

Description

[0001] Cross-reference to related applications

[0002] This application claims the rights and priorities of U.S. Application No. 62 / 963,335, filed January 20, 2020, entitled “TECHNIQUES FOR DEPLOYING INFRASTRUCTURE RESOURCES WITH A DECLARATIVE PROVISIONING TOOL,” filed September 10, 2020, the contents of which are incorporated herein by reference in their entirety for all purposes. Background Technology

[0003] Today, cloud infrastructure services utilize numerous separate services to provision and deploy code and configuration across many regions of the cloud infrastructure service (separately). These tools require significant manual work to use, especially considering that provisioning cloud infrastructure resources is typically declarative and deploying them is often imperative. Additionally, cloud infrastructure services will need to continue growing as the number of service teams and regions increases. Some cloud infrastructure service deployment strategies, which involve spending per region, may not scale well. Summary of the Invention

[0004] This describes a technique for implementing an infrastructure orchestration service. In some examples, the method may include a scheduler that can receive a configuration file for a first deployment of a resource at an execution target, the configuration file including a first publication identifier. The resource can be deployed at the execution target according to the file. The current state of the resource can be stored. The scheduler may receive a second version of the file for a new deployment at the execution target, which may include a second publication identifier. At least one worker node can execute a plugin to compare the first identifier with the second identifier. If the first identifier differs from the second identifier, the plugin can compare the current state of the resource with the expected state according to the second identifier. If the expected state differs from the current state, the resource is deployed at the execution target according to the second identifier.

[0005] In other examples, the system may include at least one processor and at least one memory that can store computer-executable instructions, configuring the processor to perform operations when executed by the processor. Operations may include a scheduler receiving a configuration file for a first deployment of a resource at an execution target, the configuration file including a first publication identifier. The resource may be deployed at the execution target according to the configuration file, and the current state of the resource may be stored. The scheduler may receive a second version of the configuration file for a new deployment at the execution target, which may include a second publication identifier. At least one worker node may execute a plugin to compare the first publication identifier with the second publication identifier. If the first publication identifier differs from the second publication identifier, the plugin may be used to compare the current state of the resource with a desired state of the resource according to a second configuration file. If the desired state differs from the current state, the resource is deployed at the execution target according to the second version of the configuration file.

[0006] In another example, a computer-readable storage medium may store computer-executable instructions that, when executed by at least one processor, can cause the processor to perform operations. Operations may include a scheduler receiving a configuration file for a first deployment of a resource at an execution target, the configuration file including a first publication identifier. The resource may be deployed at the execution target according to the configuration file, and the current state of the resource may be stored. The scheduler may receive a second version of the configuration file for the new deployment at the execution target, which may include a second publication identifier. At least one worker node may execute a plugin to compare the first publication identifier with the second publication identifier. If the first publication identifier differs from the second publication identifier, the plugin may be used to compare the current state of the resource with a desired state of the resource according to a second configuration file. If the desired state differs from the current state, the resource may be deployed at the execution target according to the second version of the configuration file.

[0007] In another example, the apparatus may include components for performing steps of any method described herein. Attached Figure Description

[0008] To facilitate the identification of any particular element or action in discussion, one or more of the most significant digits in the figure references refer to the figure number in which the element was first introduced.

[0009] Figure 1 It is a block diagram used to illustrate an architecture of at least some elements for implementing a cloud infrastructure orchestration service according to at least one embodiment.

[0010] Figure 2It is a block diagram used to illustrate an architecture of at least some elements for implementing a cloud infrastructure orchestration service according to at least one embodiment.

[0011] Figure 3 It is a flowchart used to illustrate an example group (flock) according to at least one embodiment.

[0012] Figure 4 This is a flowchart used to illustrate an example group according to at least one embodiment.

[0013] Figure 5 It is a block diagram used to illustrate an architecture of at least some elements for implementing a cloud infrastructure orchestration service according to at least one embodiment.

[0014] Figure 6 It is a block diagram used to illustrate an architecture of at least some elements for implementing a cloud infrastructure orchestration service according to at least one embodiment.

[0015] Figure 7 It is a diagram illustrating code for implementing at least some elements of a cloud infrastructure orchestration service according to at least one embodiment.

[0016] Figure 8 It is a diagram illustrating code for implementing at least some elements of a cloud infrastructure orchestration service according to at least one embodiment.

[0017] Figure 9 It is a flowchart illustrating an example process for implementing at least some elements of a cloud infrastructure orchestration service according to at least one embodiment.

[0018] Figure 10 It is a block diagram used to illustrate an architecture of at least some elements for implementing a cloud infrastructure orchestration service according to at least one embodiment.

[0019] Figure 11 It is a flowchart illustrating an example process for implementing at least some elements of a cloud infrastructure orchestration service according to at least one embodiment.

[0020] Figure 12 It is a block diagram used to illustrate a distributed system according to at least one embodiment.

[0021] Figure 13 It is a block diagram illustrating one or more components of a system environment according to at least one embodiment, wherein services provided by one or more components of the system of the embodiment can be provided as cloud services through the system environment.

[0022] Figure 14 This is a block diagram illustrating an example computer system in which various embodiments of the present disclosure may be implemented. Detailed Implementation

[0023] In some examples, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In some examples, IaaS is one of the three main categories (or subcategories) of cloud computing services. Most consider the other main categories to be Software as a Service (SaaS) and Platform as a Service (PaaS), and sometimes SaaS may be considered a broader category that includes both PaaS and IaaS, and some even consider IaaS to be a subcategory of PaaS.

[0024] In the IaaS model, cloud computing providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.).

[0025] In some cases, IaaS providers can also offer a variety of services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering). Therefore, because these services may be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.

[0026] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the rest of the application stack. For example, a user can log in to an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software onto the VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0027] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities can also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0028] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, the customer may be responsible for (e.g., on self-service virtual machines that can be spun up on demand) the processing (OS), middleware, and / or application deployment.

[0029] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, and even installing the necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0030] In some cases, there are two distinct challenges with IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is operational. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.) once everything is provisioned. In some cases, these challenges can be addressed by enabling the configuration of the infrastructure to be declaratively defined. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more profiles. Therefore, the overall topology of the infrastructure (e.g., what resources depend on which and how they work together) can be declaratively described. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the profiles can be generated. In contrast to declarative instructions (e.g., including information about what to do, which may only include the expected output), imperative instructions include information about how to do what needs to be done (e.g., including information about the steps, components, and state required to accomplish the task).

[0031] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potentially on-demand, configurable, and / or shared pools of compute resources), also known as the core network. In some examples, there may also be one or more security group rules, provided to define how the network's security will be configured, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc., may also be provided. As more infrastructure elements are expected and / or added, the infrastructure can evolve gradually.

[0032] As mentioned above, one way to supply infrastructure is to describe it declaratively. Therefore, a configuration file can be a declarative document that simply describes each of the aforementioned infrastructure components and how they interact. The configuration file can describe the resources and associated fields required to create an element, and then other elements referencing the previously described elements can be described. In some examples, the supply tool can then generate workflows for creating and managing the elements described in the configuration file.

[0033] In some cases, the workflow of a provisioning tool can be configured to execute various commands. One executable function is view reconciliation, where the provisioning tool compares a view of the current infrastructure (e.g., the expected state of the infrastructure) with how the infrastructure is actually operating. In some cases, performing view reconciliation may include querying various resource providers or infrastructure resources to identify which resources are actually running. Another function that a provisioning tool can perform is plan generation, where the provisioning tool compares the actually running infrastructure components with what the provisioning tool hopes the state will look like (e.g., the desired configuration). In other words, plan generation can determine what changes need to be made to bring resources to the latest expected state. In some cases, a third function is execution (e.g., application) functionality, where the provisioning tool can execute plans generated by the plan generation function.

[0034] Typically, provisioning tools can be configured to obtain a configuration file, parse the declarative information contained therein, and programmatically / automatically determine the order in which resources need to be provisioned to execute the plan. For example, if a VPC needs to be started before security group rules and VMs are started, the provisioning tool will be able to make that decision and start them in that order without user intervention and / or without requiring that information to be included in the configuration file.

[0035] In some cases, continuous deployment techniques can be used to deploy infrastructure code across various virtual computing environments. Additionally, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that is expected to be deployed to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, provisioning tools can be used to provision resources, and / or deployment tools can be used to deploy the code once the infrastructure is provisioned.

[0036] As mentioned above, there are typically two different tools for controlling infrastructure resources, one for handling the provisioning of infrastructure resources and the other for deploying code, with orchestration between these two tools being performed manually. However, at a certain scale, manual implementation always leads to deviations. Therefore, automated tools for provisioning and deploying virtual infrastructure enable more efficient and reliable technologies for realizing virtual cloud environments.

[0037] In some examples, problems can arise when users manually make changes to code between the provisioning and deployment phases, especially when two tools are used. As described in this article, techniques that use a single tool for both provisioning and deployment can mitigate this by automating the process, eliminating the need for manual code changes. There may be situations where a slight change to how a user codes something can create significant problems during deployment. In some examples, when an operator first performs an action in a new area (e.g., a typo in the code), the object coded with the typo may remain unchanged forever. If the application is deployed with that typo, and the application is insensitive to the typo (e.g., it still works), it is possible that at some point in the future, additional code changes might become sensitive to the typo and crash the entire system. Therefore, the techniques presented in this article can remove the gap between provisioning and deployment, which can often lead to problems.

[0038] Typically, deployment modeling is declarative, allowing configuration files to be used to declare infrastructure resources. For example, create, read, update, delete (CRUD) instructions are often used to generate deployment files using general representational state transition (REST) ​​concepts (e.g., REST application programming interfaces (APIs)). However, deployment itself typically does not follow the same concept. Additionally, while infrastructure provisioning tools tend to be truly powerful and / or expressive, deployment tools tend to be more restrictive about the operations they can perform (e.g., they are imperative rather than declarative). In some examples, imperative tools receive imperative instructions that do not necessarily describe the workflow, state, etc., of components. For example, a central infrastructure service might be imperative, meaning it receives imperative instructions (e.g., expected outputs) rather than declarative instructions (e.g., descriptions of the various steps required to perform a task). However, in some examples, the central infrastructure service described in this paper can be declarative. Therefore, there has long been a need for a tool capable of handling both functional requirements within a cloud environment (e.g., provisioning and deployment of infrastructure elements).

[0039] In some examples, this document describes technologies used to implement Cloud Infrastructure Orchestration Services (CIOS). As briefly described above, such technologies can be configured to manage the provisioning and deployment of infrastructure assets within a cloud environment. In some cases, CIOS may include two types of services: central and regional components (e.g., CIOS central and CIOS regions). The following terms will be used throughout the document:

[0040] • Infrastructure components – Long-lived infrastructure components that support running code.

[0041] Examples: Deploying applications, load balancers, Domain Name System (DNS) entries, object buckets, etc.

[0042] • Artifacts – Code that is being deployed to a deployment application or Kubernetes engine cluster, or configuration information that is being applied to infrastructure components (hereinafter,

[0043] These may be read-only resources ("config").

[0044] • Deployment tasks – Short-lived tasks typically associated with deploying or testing code. Additionally, deployment tasks are modeled as having a lifespan that does not exceed the length of the resource that created them.

[0045] Example: "deploy $artifact to $environment"

[0046] "watch$alarm for 10minutes", "execute$testSuite" or

[0047] "wait for $manualApproval"

[0048] For example, CIOS can model deployment orchestrator deployments as creating resources that are transformed into an available state when they are completed.

[0049] Because CIOS maintains the state of its associated declarative providers, CIOS can control the lifecycle of these short-lived resources, as it is associated with publication.

[0050] • Resources – Resources that can be CRUD

[0051] o CIOS models each of the constructs listed above as a resource. This modeling is discussed in detail in the next section.

[0052] • Groups – CIOS encapsulates the control plane and all its components. They exist primarily to model ownership of infrastructure components and point to those components.

[0053] ●Group config—Describes the collection of all infrastructure components, artifacts, and deployment tasks associated with a single service.

[0054] Each group has only one group config. The group config check-in source is controlled.

[0055] The group configuration is declarative. They expect CIOS to provide the domain, region, ad, and artifact version as input.

[0056] oGroups are fine-grained—groups consist of individual services and supporting infrastructure.

[0057] • Status — A snapshot of the status of each resource in the group at a specific point in time.

[0058] ●Publish—A tuple of a specific version of the group config and the specific version of each artifact it references.

[0059] o views a post as describing a state that may not yet exist.

[0060] • Release Plan – The set of steps CIOS will take to transition all regions from their current state to the state described in the release.

[0061] The release plan has a limited number of steps and clearly defined start and end times.

[0062] • Application – This is a noun. A single attempt to execute a release plan. An action that changes the current state of a group.

[0063] CIOS can be described as an orchestration layer that applies configurations to downstream systems (e.g., worldwide). It is designed to allow worldwide infrastructure provisioning and code deployment without manual work from service teams (e.g., beyond initial approval in some cases). The advanced responsibilities of CIOS include, but are not limited to:

[0064] • Provides the team with a view of the current status of resources managed by CIOS, including any ongoing change activities.

[0065] • Help the team plan and release new changes.

[0066] • Coordinate activities across various downstream systems within the region to execute approved release plans without human intervention.

[0067] • Coordinate activities across regions / sectors to implement approved release plans worldwide.

[0068] In some examples, CIOS handles onboarding by enabling teams to provide configuration information to CIOS via check-in code. Additionally, CIOS can automate much more, making this a more substantial practice compared to previous implementations. In some cases, CIOS handles pre-deployment by providing teams with the ability to automatically deploy and test code. In others, CIOS handles change management (CM) policy writing by enabling the automatic generation of plans to roll out new artifacts (e.g., worldwide) as teams build them. It does this by checking the current state of each region and the current CIOS config (which itself can be an artifact). Additionally, teams can review these plans and iterate on them by changing the CIOS config and requesting CIOS to re-plan. Once the team is satisfied with the plan, they can create a "release" referencing the plan. The plan can then be marked as approved or rejected. While teams can still write CMs, they are merely pointers to CIOS plans. Therefore, teams can spend less time reasoning about plans. Plans are more accurate because they are machine-generated. The plan is far too detailed for human processing; however, it can be displayed through a sophisticated user interface (UI).

[0069] In some examples, CIOS can handle the execution of Change Management (CM) by automating the execution of deployment plans. Once a release plan is created and approved, engineers will no longer be involved in CM unless CIOS initiates a rollback. In some cases, this may require teams to automate tasks that are currently performed manually. In some examples, when CIOS detects a decline in service health during execution, it can handle rollback change management (CM) by automatically generating a plan to return the group to its original (e.g., pre-release) state. In some examples, CIOS can handle urgent / tactical deployment changes by receiving release plans that cover a subset of regions and / or resources managed by CIOS and then executing those plans.

[0070] Additionally, CIOS supports defining the primitives necessary for fully automated, worldwide deployments. For example, CIOS can measure service health by monitoring alerts and executing integration tests. In the event of service degradation, CIOS can help teams quickly define rollback behavior, which can then be executed automatically. CIOS can automatically generate and display release plans and track approvals. In some cases, the language teams use to describe desired deployment behavior can be declarative. Within a system, CIOS can combine code deployment and infrastructure config (e.g., provisioning) capabilities. CIOS also supports flexible ordering of components across regions and across regions. Teams can express ordering via check-in configs. Teams can programmatically invoke CIOS's planning and release APIs.

[0071] Figure 1 An architecture 100 is depicted, illustrating the technologies used to at least implement a CIOS center 102. In some examples, the CIOS center 102 may be a service that processes operations at the "group" level. The CIOS center 102 has several responsibilities, including but not limited to:

[0072] • Serves as an authentication gateway for group metadata changes and publishing operations.

[0073] • Store group metadata for groups and provide an authoritative mapping to deployment artifacts and CIOS repositories.

[0074] • Global release with cross-phase and goal coordination.

[0075] • Synchronize to implement strategies such as "no more than one ongoing group post at a time".

[0076] ● Detect changes to group configurations and artifacts, and trigger the generation of releases for such changes.

[0077] In some examples, the Source Code Version Control Management Service (SCVMS) 104 can be configured to store authority group configurations, and the Artifact Notification Service (ANS) 106 can be subscribed to by the CIOS Center 102, allowing the CIOS Center 102 to be notified of new artifact builds. The CIOS Center 102 can then map incoming changes to the affected groups and initiate release plans where desired. Additionally, in some examples, the Artifact Push Service (APS) can be invoked by the CIOS Center 102 prior to release to the target region to ensure that any artifacts required for successful release before the target region is present.

[0078] In some examples, a customer (e.g., an engineer) 108 can invoke CIOS Central 102 to CRUD groups and / or publish, and view the status of ongoing CIOS activities. Group Management Service 110 may include one or more APIs to manipulate groups, View / Schedule / Approve Service 112 may include a CRUD API to create and approve plans, and view a central copy of the status of all CIOS-managed resources, Change Monitoring Service 114 can monitor SCVMS 104 for changes to group configs and can receive notifications from ANS 106 regarding changes to other artifacts, and Status Ingestor Service 116 can create copies of region statuses in CIOS Central Database (DB) 118, allowing View / Schedule / Approve 112 to expose them. In some examples, CIOS Central DB 118 may be a DB for groups, plans, and statuses. Group information can be authoritative; everything else can be stale copies of data from CIOS Region 120.

[0079] In some examples, Engineer 108 can (e.g., via Ingress Agent Cluster 122) make API calls to Group Management Service 110 to create a list of groups. The protocol for making such API calls could be Secure Hypertext Transfer Protocol (HTTPS), etc. The relevant Access Control Lists (ACLs) used for this operation could include a Local Area Network (LAN) 124 or other private connection. For example, CIOS can manage / control alternative network connections (e.g., dedicated, leased, and / or private connections) that use the public internet to connect a customer's local data center or network to CIOS. Additionally, authentication and authorization (e.g., for Engineer 108) can be performed by a reservation system portal (e.g., a reservation service) that allows users to manage machine infrastructure. In some cases, CIOS Center 102 can use Java Database Connectivity (JDBC), etc., to store group metadata, schedules, and status in a central DB 118. In some examples, ANS 106 can be configured to notify Change Monitoring Service 114 when a new artifact has been published. ANS 106 can use HTTPS, and both authentication and authorization can be handled by mutual transport layer security services. Additionally, in some cases, Change Monitoring Service 114 can poll SCVMS104 for group configuration changes. This polling can be performed using Secure Shell (SSH) or other protocols. Authentication for Change Monitoring Service 114 can be handled by the CIOS system account, and authorization can be handled by SCVMS104.

[0080] In some examples, Engineer 108 can use the View / Schedule / Approve service 112 to perform one or more of the following operations: Engineer 108 can schedule and / or approve plans by calling CIOS Center 102 to generate and approve plans. Engineer 108 can view the status of ongoing CIOS activities worldwide by calling CIOS Center 102. Additionally, Engineer 108 can view a copy of the status of CIOS-managed resources worldwide through CIOS Center 102. These API calls (etc.) can be performed via HTTPS or similar protocols. Additionally, the relevant ACLs can be controlled by LAN 124, and both authentication and authorization can be handled by the scheduling service. In some examples, the View / Schedule / Approve service 112 can request a plan and (e.g., using HTTPS, etc.) push plan approval to all areas of CIOS Zone 120. The relevant ACLs can be controlled using a security list managed by Wide Area Network (WAN) Gateway 126. Authentication can be handled through mutual transport layer security, and authorization can be handled through various identity policies. Furthermore, the State Ingestor Service 116 can monitor the CIOS Zone 120 to obtain job status or status changes, allowing the CIOS to provide a central view of themselves upon request (e.g., also using HTTPS). ACLS for this purpose can also be handled by the WAN Gateway 126, and both authentication and authorization can be handled by mutual transport layer security services.

[0081] Figure 2 An architecture 200 is depicted illustrating the technologies used to at least implement CIOS Zone 202. In some examples, CIOS Zone 202 is where most of the work on declarative provisioning and planning, as well as the approved release of applications, can occur. In some cases, each instance of CIOS Zone 202 can have a zone frontend service that can handle operations at the "execution target" level. It can be configured to perform the following:

[0082] • Process all CIOS authentications from incoming operations originating from CIOS Center 102.

[0083] • Implement a rule that only one "execution" (plan / import resources / apply plan) can be in progress at a time for a given execution goal.

[0084] • Manages a binary artifact store for declarative infrastructure provisioning artifacts used as inputs and outputs during declarative infrastructure provisioning execution. Examples of inputs are declarative infrastructure provisioning configuration files and input state files. Typical output is a final state file.

[0085] • For any given execution, request work from the CIO executor and poll for the results from the CIO executor.

[0086] In some cases, the CIOS frontend can rely on the CIOS executor 206 (also referred to as the "scheduler" in this document), which handles the actual execution. In some examples, the CIOS executor runs at the "execution" level, and it can:

[0087] • Track the pool of available worker nodes

[0088] • Query incoming job requests and assign them to eligible workers when available.

[0089] • Track worker status and perform updates for reporting to clients.

[0090] • Dead nodes can be detected via a lease protocol, and tasks assigned to dead nodes can be made to fail, depending on the task status.

[0091] • Provides facilities for canceling / terminating / pausing / resuming execution, and can map those executions to facilities to pass cancellation / termination / resumption information to worker nodes.

[0092] In some cases, a CIOS executor can rely on CIOS workers, which can assign tasks for execution to workers and provide facilities for workers to update job progress. Worker services operate at the "task" granularity. Each worker is an agent that executes the tasks assigned to it and reports task status and output. Each worker can:

[0093] • Poll the executor worker API for each assigned work item and take action to match the assignment state with its local state:

[0094] Start a container to poll for task items that do not exist locally.

[0095] ο Terminate containers that are running locally and do not have corresponding assigned task items.

[0096] • Report work status

[0097] • Causes the inputs and outputs of the job container to occur

[0098] • Launch and monitor the declarative infrastructure provisioning container for the actual work of releasing against the execution objectives.

[0099] CIOS workers can rely on CIOS executors to poll for work from worker endpoints of the CIOS executors and report results to them. For all coordination, workers can rely on the executor. Additionally, CIOS workers can also rely on CIOS Zone 202, where worker services read input from one or more APIs associated with the zone frontend service and write output to it. Examples of input are configuration and startup state files and import mappings. Examples of output are declarative provisioning procedures, output declarative provisioning state files, and import result states.

[0100] In some examples, CIOS Zone 202 can be a zone service used to manage zone instances / deployments of CIOS. CIOS Zone 202 is responsible for authoritatively storing and managing plans and states related to a specific zone. Zone DB 204 can be a CIOSDB used for the state and plans within a specific zone. This is Figure 1 The central DB 208 is an authoritative copy of a subset of the CIOS regions. Scheduler 206 can be responsible for managing worker cluster capacity, assigning tasks to workers, and keeping track of task status. In some cases, Task DB 208 is another CIOS DB used for task status. The data in this DB is primarily for operational purposes. Additionally, workers 210 can be a cluster of Java Virtual Machines (JVMs) that manage declarative provisioning images. These receive instructions from scheduler 206 and transmit results to both scheduler 206 and CIOS regions 202. CIOS container 212 can run declarative provisioning actions in its own private docker 214 container. This container does not need to contain secrets. Additionally, in some examples, signing broker 216 can be configured to prevent secret leakage via declarative provisioning tools to avoid including secrets in the declarative provisioning image. Instead, CIOS can perform request signing or initiate mutual transport layer security (mTLS) services within the broker. This also makes it easier to use FIPS-compliant cryptographic libraries.

[0101] In some examples, CIOS center 102 can invoke CIOS zone 202 to create schedules, push approvals, monitor job status (service principal), and retrieve declarative provider status (service principal). Ingress proxy 218 can be configured as an ACL, and various identity policies can be used for both authentication and authorization. Alternatively, in some examples, ingress proxy 218 can be replaced by a load balancer configured to balance the load of incoming requests, schedules, etc. In some cases, CIOS zone 202 can run a declarative provider by requesting scheduler 206 to run the declarative provider. Worker 210 can query scheduler 206 what it should run and can report the status to scheduler 206 upon completion. In some cases, mTLS can handle both authentication and authorization for CIOS zone 202 and worker 210. Additionally, when worker 210 needs to run a declarative provider, it does so within a Docker container by interacting with local Docker 214. Authentication at this stage can be handled by local Unix sockets. The Docker protocol can be used for this final step; however, HTTPS can be used for the preceding steps.

[0102] In some examples, CIOS container 212 enables declarative providers to interact (via API) with signing agent 216, which perceives itself as invoking various CIOS services. Signing agent 216 listens on a temporary port known only to the declarative provider for each invocation instance. Signing agent 216 can initiate request signing or mTLS and can pass the declarative provider's invocation to other CIOS services within the service enclave. In some cases, signing agent 216 may also communicate with one or more public CIOS services 220. For example, signing agent 216 will use internal endpoints of public services where possible. For services without internal endpoints, it must use egress agent 222 to reach external endpoints. This use of signing agent 216 may not be for cross-region communication; for example, the egress agent whitelist in each region may only be for public IP ranges within that region. In some examples, worker 210 can then persist the state and logs from declarative providers in CIOS region 202, allowing them to leak to CIOS hub 102.

[0103] Using CIOS, there are several phases that a representative customer experiences: Login, Pre-release, Worldwide Release, and Tactical Release. For the Pre-release phase, the following is an example of what happens between when a new artifact is being built and released, and Release 1 (e.g., R1). This should replace some or most of the current change management process. As the relevant artifact is built, CIOS can automatically generate a release using "Latest Version of Everything in the Group." A release is a specific version of a group configuration with specific inputs such as artifact version, domain, region, and ad. A release contains a roll-forward plan for each region and metadata describing the region ordering. Each region plan is a set of actions that a declarative provider will take to implement the group configuration in that region. Teams with a pre-release environment can use CIOS to automatically release and test software in that environment. Teams can configure CIOS to automatically test rollback plans. Teams will be able to inspect and approve releases via the CIOS UI. Teams can approve some, but not all, of the region plans within a release. If "Latest Version of Everything" does not produce a suitable plan, the team can request CIOS to generate a plan for a carefully selected artifact version.

[0104] For the global release phase, here's an example of how a team executes a tomorrow's version of today's "normal CM". Once the release is approved, CIOS pushes each approved regional plan to the corresponding region. CIOS acts independently within each region to apply the approved plan. CIOS will only execute the set of actions explicitly described in that regional plan. It will fail, not "think independently". The CIOS UI shows the team the execution progress. The CIOS UI prompts the team when manual approval is required. If execution fails due to CIOS or downstream service outages, CIOS can notify the team and prompt them for the following steps (e.g., abort, retry). CIOS does perform retries, but some downstream system outages will exceed its retrying intentions. If execution fails due to a decline in service health or a failed test, CIOS will assist the team in rolling the group back to its starting state. CIOS will notify (e.g., page) the team when it initiates an automatic rollback. The team must approve the rollback plan, and then CIOS will execute it.

[0105] For the tactical release phase, here's an example of how a team can execute a tomorrow's version of an "urgent CM." When generating a plan, the team might ask CIOS to target the plan to specific resources in several ways: by topology (e.g., domain, region, AD, etc.), by resource type (e.g., "metrics config only" or "deploy orchestration service deployment only"), or a combination of these (e.g., in a separate manner). The team approves tactical releases just like worldwide releases. CIOS orchestrates them in a similar way. If the team needs to deploy a tactical release while an active worldwide release exists, CIOS will stop executing the worldwide release in the target region and then begin executing the tactical release.

[0106] In some examples, the state of a declarative provider (e.g., traditionally a file) is the authoritative record of the resource group managed by the declarative provider. It contains a mapping between the logical identifier of each resource from the configuration file and the actual identifier of the resource. While the declarative provider is creating a resource, certain kinds of failures can prevent the actual identifier from being recorded in the state. When this happens, the actual identifier is lost for the declarative provider. These can be referred to as "orphaned resources."

[0107] For most resources, orphanage represents waste—a declarative provider starts (for example) an instance it forgot, but instead starts another instance on its next run. For resources with unique constraints or client-provided identifiers, orphanage prevents the declarative provider from making progress. For example, if a declarative provider creates user "nglass" and fails to orphanize it, then the next run of the declarative provider will attempt to create "nglass" and fail because a user with that username already exists. In some cases, orphanage is simply a problem when adding new resources to the state. In some cases, the declarative provider's refresh behavior can naturally recover from failures to record updates and deletions.

[0108] CIOS needs to be robust in the event of downstream service outages or CIOS itself failures. Because CIOS can utilize declarative providers to apply changes, robustness must exist around running and maintaining declarative providers' state. Declarative providers perform "small-scale" retries—sufficient to avoid outages lasting a few minutes. For example, a cloud provider might retry for up to 30 minutes. A downstream system outage lasting longer than 30 minutes will cause the declarative provider to fail. When a declarative provider fails, it logs all changes it successfully made in its state and then exits. To retry, CIOS must re-execute the declarative provider. Re-executing the declarative provider also allows CIOS to retry in the event of a CIOS failure. In some cases, CIOS can cycle through the following operations:

[0109] • Refresh — A declarative provider calls the GET API to retrieve a new snapshot of each resource described in its state.

[0110] • Planning – Taking into account the most recently refreshed current state, the declarative provider generates a plan (a set of specific API calls) that will achieve the desired state.

[0111] • Application – A set of steps in a declarative provider execution plan.

[0112] When executing a declarative provider, CIOS can always run all three steps. The refresh operation helps recover from any updates or deletions that were never recorded. CIOS checks the results of the planned operations and compares them to the approved release plan. If the newly generated plan contains operations not in the approved release plan, CIOS may fail and can notify the service team.

[0113] Figure 3 A directed acyclic graph (DAG) 300 is depicted for the example group 302. For a single group config in CIOS, the process of code / config progressing from check-in to production can be described from the initial test deployment to the final production (prod) deployment. Internally, CIOS references each element in the process, executing an objective (ET)—this is an internal API specification and does not leak out into the group config. CIOS executes ETs based on the DAG 300 defined in the group config. Each ET (e.g., ET-1, ET-2, ET-3, ET-4, ET-5, ET-6, and ET-7) is roughly a copy of the service described in the group config.

[0114] Figure 4 The diagram illustrates DAG 400 for example group 402. In the group configuration, CIOS is very strict about how teams represent this process—they must model it using cloud infrastructure leases and regions. Teams should not use domains to model processes. CIOS allows teams to use multiple leases within a domain and multiple regions within a lease. However, CIOS does not allow teams to use the same region twice within a lease (although they can use the same region twice within a domain—in different leases). DAG 400 illustrates the representation of processes using leases and regions. Figure 3This is a version of DAG 300. This example is for coverage services where the pre-production ET is located in the production area. Service enclave services will have unstable and stable leases in Release 1. In DAG 400, IAD is the area airport code for Dulles Airport in Washington, D.C., YYZ is the area airport code for Toronto, Ontario, PHX, LHR, and FRA are the area airport codes for Phoenix, London, and Frankfurt, respectively, and LUF and LFI are used for two different air force bases.

[0115] In one embodiment, the CIOS and / or other techniques described herein are improvements on each of Terraform (a declarative provisioning tool), Tanden (a code generation tool), and Oracle Deployment Orchestrator (ODO). Additionally, in some examples, the CIOS and / or other techniques described herein may be implemented using at least a portion of the Terraform, Tanden, and ODO tools.

[0116] This describes a technique for implementing an infrastructure orchestration service. In some examples, one method may include a scheduler that can receive a configuration file for a first deployment of a resource at an execution target, the configuration file including a first publication identifier. The resource can be deployed at the execution target according to the file. The current state of the resource can be stored. The scheduler may receive a second version of the file for a new deployment at the execution target, which may include a second publication identifier. At least one worker node can execute a plugin to compare the first identifier with the second identifier. If the first identifier differs from the second identifier, the plugin can compare the current state of the resource with a desired state based on the second identifier. If the desired state differs from the current state, the resource is deployed at the execution target according to the second identifier.

[0117] In other examples, the system may include at least one processor and at least one memory that can store computer-executable instructions, configuring the processor to perform operations when these instructions are executed by the processor. Operations may include a scheduler receiving a configuration file for a first deployment of resources at an execution target, the configuration file including a first publication identifier. Resources may be deployed at the execution target according to the configuration file, and the current state of the resources may be stored. The scheduler may receive a second version of the configuration file for a new deployment at the execution target, which may include a second publication identifier. At least one worker node may execute a plugin to compare the first publication identifier with the second publication identifier. If the first publication identifier differs from the second publication identifier, the plugin may be used to compare the current state of the resources with a desired state of the resources according to a second configuration file. If the desired state differs from the current state, the resources may be deployed at the execution target according to the second version of the configuration file.

[0118] In another example, a computer-readable storage medium may store computer-executable instructions that, when executed by at least one processor, can cause the processor to perform operations. Operations may include a scheduler receiving a configuration file for a first deployment of a resource at an execution target, the configuration file including a first publication identifier. The resource may be deployed at the execution target according to the configuration file, and the current state of the resource may be stored. The scheduler may receive a second version of the configuration file for the new deployment at the execution target, which may include a second publication identifier. At least one worker node may execute a plugin to compare the first publication identifier with the second publication identifier. If the first publication identifier differs from the second publication identifier, the plugin may be used to compare the current state of the resource with the desired state of the resource according to a second configuration file. If the desired state differs from the current state, the resource may be deployed at the execution target according to the second version of the configuration file.

[0119] Figure 5 A combined block diagram 500 depicts an architecture of at least some elements for implementing a cloud infrastructure orchestration service according to at least one embodiment. Block diagram 500 illustrates a CIOS hub 502 (e.g., Figure 1 CIOS center 502) and CIOS region 504. CIOS center 502 may include control plane 506 (e.g., CIOS center 102) and CIOS region 504. Figure 1 Control plane 110), View / Schedule / Approve service 508 (e.g., Figure 1 View / plan / approval service 112), change management service 510 (e.g., Figure 1 Change management service 114) and status ingestor service 512 (e.g., Figure 1 Status capture service 116).

[0120] User 514 (for example, Figure 1 Engineer 108 can create a release that may include at least one deployment, at least one test, either the two or any other combination of suitable declarative infrastructure provisioning tool operations, and user 514 can do so via LAN gateway 516 (e.g., Figure 1The LAN 124 sends the release to the CIOS center 502. The CIOS center 502 can post-process the release. The control plane 506, which may be a group management component, can generate a random release ID and assign this randomly generated release ID to the release. The control plane 506 can augment the release using at least one code file generated in the post-processing. These code files may include a provider file, which may include at least one declarative infrastructure provisioning tool plugin called a provider, and a local file. The local file may include the operation of generating a random release ID, and the random release ID generated in the local file is injected into the provider file. The result of the post-processing by the control plane 506 can be a release (an augmented release) with the original code generated by the user and the augmented code generated through post-processing.

[0121] The service 508 can be viewed / planned / approved via WAN gateway 518 (e.g., Figure 1 The WAN gateway 126 sends the extended advertisement to CIOS zone 504. The extended advertisement can also be sent by CIOS zone 520 (e.g., ...). Figure 2 CIOS region 502) receives data from a computing device that generates status reports and keeps track of tasks entering CIOS region 504. CIOS region 520 can receive data from an extended distribution to scheduler 522 (e.g., ...). Figure 2 The scheduler 206 sends tasks, and the scheduler can be a worker 524 (e.g., ...). Figure 2 The computing device for worker 210 (typically assigned to workers with the least workload). In some embodiments, CIOS region 520 and scheduler 522 may be on the same computing device. Worker 524 may be a computing device capable of performing tasks and plugins assigned by scheduler 522, and worker 524 may be part of a worker cluster that may include many workers 524. Worker 524 may be connected to CIOS container 526 (e.g., ...). Figure 2 Interacting with CIOS container 212), this container can exist in docker 528 (e.g., Figure 2 Within Docker container 514, CIOS container 526 can examine the differences between the expected state of the execution target and the actual state of the execution target related to the task assigned to worker 524. If CIOS container 526 identifies a difference, worker 524 can execute the task; if CIOS container 526 does not identify a difference, worker 524 may not execute the task. By executing the task, API calls to cloud services can be made, and these API calls can be transmitted via a signature proxy 532 (e.g., ...). Figure 2(Signature proxy 216). Signature proxy 532 can be a general HTTP proxy, and signature proxy 532 can control outgoing network traffic in CIOS zone 504. Specifically, signature proxy 532 can cut off all outgoing network traffic under certain circumstances, which will... Figure 10 and Figure 11 Further discussion is needed.

[0122] In some embodiments, a release expected to be executed at the execution target may include the same resource (release resource) as the resource currently at the execution target (execution target resource). In this case, execution of the release may still be expected, but existing declarative infrastructure provisioning operations may not execute the release resource because the release resource is the same as the execution target resource. However, the techniques described herein can be used to force the release resource to be executed at the execution target by enforcing a diff (e.g., a comparison function that identifies differences in a configuration file). Thus, one technical improvement achieved by the techniques described herein is the ability to force declarative infrastructure provisioning to process releases, which are essentially redeployments of resources that already exist and / or have been deployed, even if the declarative infrastructure provisioning is not designed to handle the automatic redeployment of resources bound to a release.

[0123] Enforcing differences can involve indicating a difference between the current state and the desired state of a declarative infrastructure provisioning tool. In some embodiments, enforcing differences can be achieved at least in part by generating a random release ID. User 514 may expect the published resource to be executed at the execution target regardless of the current state of the resource at the execution target, even if the execution target resource is the same as the published resource. In this case, user 514 can choose an option to generate a random release ID in post-processing performed in control plane 506. However, in some cases, control plane 506 may always generate a random release ID. User 514 can then choose whether to use this by writing more configuration (e.g., overriding the default "use release ID to enforce differences" behavior). In some embodiments, once generated, the random release ID can be permanently bound to the release. The release can be sent to CIOS region 504, and the tasks for the release can be assigned to worker 524 by scheduler 522. Worker 524 can communicate with CIOS container 526, which can run declarative infrastructure provisioning tool operations. CIOS container 526 can check the difference between the current state of the resource and the desired state of the resource. Within this check, CIOS container 526 can determine whether the release includes a randomly generated release ID, which can be considered as enforcing a difference. If the randomly generated release ID is unique and detected by CIOS container 526, then CIOS container 526 can determine that there is indeed a difference between the current state and the desired state, even if the resource to be deployed is the same as the resource that has already been deployed. In response to identifying this difference, the newly released resource can be deployed.

[0124] Figure 6 CIOS region 602 is depicted according to at least one embodiment for implementing cloud infrastructure orchestration services (e.g., Figure 5 The block diagram of the CIOS region 504 architecture is shown in Figure 600. Similar to... Figure 5 In part, CIOS region 602 may include CIOS region 604 (e.g., Figure 5 CIOS region 520) and scheduler 606 (e.g., Figure 5 The scheduler 606 and the CIOS region 604 can both be computing devices. In some embodiments, the CIOS region 604 and the scheduler 606 can be on the same computing device. The scheduler 606 can be communicatively coupled to a task DB 608 that helps keep track of assigned tasks, and to a worker 610, which can be a computing device that performs the tasks. The worker 610 can be part of a worker cluster that may include many workers 610. The worker 610 can be communicatively coupled to a CIOS container 612 (e.g., Figure 5 The CIOS container (528) can be used in Docker (e.g., 614). Figure 5 A computing device that runs declarative infrastructure provisioning tools within a Docker container (530). Worker 610 can communicate with and send publications to CIOS container 612, and CIOS container 612 can determine if there is a discrepancy between the current state and the desired state. If a discrepancy exists, worker 610 can execute the assigned task, and based on executing the assigned task, worker 610 can execute an outgoing API call, which can be made via a signed proxy 616 (e.g., ...). Figure 5 (Signature proxy 532). In some cases, signature proxy 616 can allow outgoing API calls or cut off all outgoing network traffic, which will... Figure 10 and Figure 11 This will be discussed further. While Task DB 608 is shown as the database for Scheduler 606, it can also (or alternatively) be configured as the database for all CIOS zones 604. Database 608 can be an authority on changes occurring within a zone. An example of what "authority" means: regarding what has happened to a specific execution target within CIOS zone 602, the CIOS center (e.g., Figure 5 The 502 error may be outdated, which could lead to CIOS zone 604 requiring changes to be made when it is unsafe to do so. In this example, CIOS zone database 6008 can determine whether it is safe to do so (e.g., whether there is / does not exist any change activity that has occurred against the target). In this example, CIOS zone 604 will check database 608 and return an HTTP 409 (e.g., conflict) to CIOS zone 502.

[0125] In some embodiments, the user (e.g., Figure 5User 514 may wish to retry the deployment, testing, or any other suitable execution of a related resource. Some non-limiting examples of reasons for retrying resource execution could be random failures of previous executions, vulnerabilities in the code of previous executions, etc. In addition to determining whether a unique release ID is bound to a release, CIOS container 612 can also check the execution ID and the state of the associated resource. Each time the execution of a resource is attempted, scheduler 610 can generate a unique execution ID that can be bound to the execution of the resource. In a non-limiting example where a user sends a retry command for a release, scheduler 610 can generate an execution ID and can assign the task associated with the retry command to worker 612. Worker 612 can send the release ID, execution ID, and resource state to CIOS container 612. If the release ID differs from the current state, CIOS container 612 can identify the difference and worker 610 can execute the task. If the release ID of the desired state is the same as the release ID of the current state, CIOS container 612 can check the execution ID and release state. If CIOS container 612 determines that the currently executing execution ID is different from the previously executed execution ID, and if CIOS container 612 determines that the previously executed release status was "failed" or any suitable similar status, then worker 610 may attempt to execute the resource's task again. If CIOS container 612 determines that the execution ID is the same as the previous execution or the release status is not "failed," then worker 610 may not execute the task.

[0126] Figure 7 Code snippet 700 for implementing at least some elements of a cloud infrastructure orchestration service according to at least one embodiment is shown. Snippet 700 depicts code for defining resources that can be deployed at an execution target. Snippet 700 may include resource type 702, resource name 704, release ID 706, and resource status 708. Resource type 702 may include deployment, testing, or any other suitable resource type for the associated resource. Resource name 704 is depicted as “executor_ad1” in snippet 700, but resource name 704 can be any name suitable for describing the associated resource. Release ID 706 is depicted as “7e29d6aa-7dc6-4268-9f90-e57caf76e714” in snippet 700, but release ID 706 can be any randomly generated string used to identify a unique release. Resource status 708 is depicted as “successful” in snippet 700, but resource status 708 can be any suitable status used to describe the associated resource (e.g., “successful”, “failed”, “approved”, “requires approval”, etc.).

[0127] In some embodiments, fragment 700 can be used as an address or identifier for a resource. Worker nodes (e.g., Figure 6 Worker 610 can receive tasks for executing resources. Worker nodes can send resources to CIOS containers (e.g., ...). Figure 6 The CIOS container (612) can at least partially use fragment 700 to determine whether a resource is unique. The CIOS container can compare the resource type 702, resource name 704, and publication ID 706 (expected state) from the resource with the same item (current state) currently at the execution target. If a difference exists between the expected state and the current state, the CIOS container can identify the difference, and the worker node can execute the resource at the execution target.

[0128] In some embodiments, fragment 700 may include another line of code, referred to as the execution ID, for identifying a unique execution attempt. The execution ID may be a string of characters, which may be generated by a scheduler node (e.g., Figure 6 The scheduler node (606) generates random execution IDs. For each execution attempt of a resource, the scheduler node can generate a random execution ID and send the resource with the execution ID to the worker node. The worker node can send the resource with the execution ID to the CIOS container, which can determine if there is a difference between the current state and the resource with the execution ID (desired state). If the execution ID of the desired state is different from the current state and the resource state 708 of the current state is "failed," the CIOS container can determine that there is a difference between the current state and the desired state. Based on the difference identified by the CIOS container, the worker node can execute the resource at the execution target. This operation can be similar to the retry operation discussed earlier.

[0129] Figure 8 Code snippets 800 and 802 are shown for implementing at least some elements of a cloud infrastructure orchestration service according to at least one embodiment. Snippets 800 and 802 depict the user (e.g., Figure 5 Post-processing of posts created by user 514 can be handled by group management components (e.g., Figure 5 The code generated by the control plane 506. Fragment 800 shows an example of a provider file that may include a provider, which may be a declarative infrastructure provisioning tool plugin. The provider file may include a release ID label 804 and an execution ID label 806. The provider file may also include any other suitable providers or plugins for the relevant release. The release ID label 804 may be a release ID (e.g., Figure 7The release ID (706) is the operation injected into the resource, and as depicted in fragment 800, the release ID annotation 804 is "local.release.id", but can be any suitable annotation used to inject the release ID into the resource. The execution ID annotation 806 can be the operation injected into the resource, and as depicted in fragment 800, the execution ID annotation 806 is "local.execution.id", but can be any suitable annotation used to inject the execution ID into the resource.

[0130] Fragment 802 illustrates an example of a local file that can be generated by the group management component during post-processing of a resource. The local file may include a release ID 808 and execution target information 810. As depicted in fragment 802, the release ID 808 is “39ed7b9c-5519-4069-896d-8e17ed4fc29e”, but the release ID 808 can be any string suitable for uniquely identifying a release. The execution target information 810 may include various information about the execution target intended to execute the release on. As depicted in fragment 802, the execution target information 810 includes an execution target name 812, but the execution target information can be any suitable information associated with the execution target. As depicted in fragment 802, the execution target name 812 is “ap-melbourne-1”, but the execution target name 812 can be any suitable name of the execution target intended to deploy the resource thereto.

[0131] The local file for fragment 802 can include both a release ID (808) and an execution ID. This may occur at a CIOS center (e.g., Figure 1 During post-processing of a release at CIOS center 102, the group management component can generate a release ID 808, which can be included in a local file. An execution ID can be generated by a scheduler node, which can then update the local file using the execution ID. Both the release ID 808 and the execution ID can be injected into the provider file of fragment 800 by utilizing release ID label 804 and execution ID label 806.

[0132] Figure 9An example flowchart illustrating a process 900 for implementing a CIOS according to certain embodiments of the present disclosure is shown. This process is illustrated as a logic flowchart, where each operation can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation can represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0133] Additionally, process 900 can be executed under the control of one or more computing devices or computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that executes jointly on one or more processors, via hardware, or a combination thereof. As described above, the code can be stored on a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. In some embodiments, process 900 can be executed in parallel by multiple processors. The computer-readable storage medium can be non-transitory.

[0134] Figure 9 An example flowchart illustrating a process 900 for implementing CIOS to deploy resources at an execution target, according to at least one embodiment, is depicted. Process 900 may begin at block 902, where the scheduler node (e.g., Figure 2 The scheduler 206) receives a configuration file for the first deployment of resources at the execution target, which may include a first release identifier (hereinafter referred to as the first release ID) (e.g., Figure 8 The release ID is 808. The scheduler node can send configuration files to worker nodes (e.g., ...). Figure 2 Worker 210), the worker node can be part of a worker cluster that includes many worker nodes.

[0135] In box 904, a first deployment of resources according to the configuration file is performed at the execution target; in some embodiments, this is performed by worker nodes. In some examples, the execution target may be empty or otherwise have no resources deployed to it. Worker nodes may send the configuration file to a CIOS container (e.g., Figure 2The CIOS container 212 is used to determine whether there is a difference between the resources currently at the execution target (current state) and the resources in the configuration file (desired state). Since the execution target may not have any resources in box 904, a difference may exist and worker nodes may perform the first deployment of resources at the execution target.

[0136] In box 906, the current state of the resource at the execution target is stored. In some examples, the current state of the resource at the execution target can be stored once the worker node has performed the first deployment of the resource at the execution target. The current state can be used, at least in part, in subsequent comparisons to determine whether to deploy a subsequent release at the execution target.

[0137] In box 908, the scheduler node receives a second profile for a second deployment of resources at the execution target. This second profile may include a second release identifier, hereinafter referred to as the second release ID. The scheduler node may send the second profile along with the task to the worker node to perform the second deployment of the resources at the execution target.

[0138] In box 910, the worker node executes a plugin to compare the first release ID with the second release ID. The worker node can then send a second profile to the CIOS container. In some examples, the CIOS container can determine whether there is a difference between the current state, as stored in box 906 of process 900, and the expected state, as defined in the second profile, by comparing the first release ID with the second release ID.

[0139] In box 912, if the CIOS container determines there is no difference between the first release ID and the second release ID, the worker node takes no action, and the second deployment may not occur. In box 914, if the CIOS container determines there is a difference between the first release ID and the second release ID, the CIOS container can compare the current state of the resource at the execution target with the expected state of the resource defined in the second configuration file. In some examples, if the CIOS container determines there is a difference between the first release ID and the second release ID at box 910, the CIOS container can skip directly to box 916.

[0140] In box 916, if the CIOS container determines that the current state of a resource differs from the expected state of the resource as defined in the second configuration file, the resource can be deployed at the execution target according to the second configuration file. Worker nodes can complete the tasks assigned to them by the scheduler node by performing a second deployment of the resource at the execution target. In some examples, the CIOS container may determine that the first release ID and the second release ID are the same, which may indicate that the user wants to retry the deployment. In this case, the CIOS container can compare the first execution ID of the first configuration file and the second execution ID of the second configuration file. If the CIOS container determines that the first execution ID and the second execution ID are different, the CIOS container can check the release status of the first configuration file. If the release status of the first configuration file is "failed," or indicates that the first deployment of the resource may not have had any other similar status that has been successful, then the worker node can perform a second deployment of the resource as defined in the second configuration file.

[0141] Figure 10 A simplified architecture 1000 for illustrating techniques to prevent concurrent execution of declarative infrastructure provisioning tools, according to at least one embodiment, is depicted. The simplified architecture 1000 may include a CIOS region 1002 (e.g., similar to...). Figure 2 CIOS region 202), scheduler 1004 (e.g., similar to Figure 2 Scheduler 206), worker 1006 (e.g., similar to...) Figure 2 Workers 210), CIOS container 1008 (e.g., similar to...) Figure 2 CIOS container 212), Docker 1010 (e.g., similar to...) Figure 2 docker214) and signing agent 1012 (e.g., similar to ...) Figure 2 (Signature agent 216). Scheduler 1004 can be responsible for managing the capacity of workers 1006 in the worker cluster, assigning tasks to workers 1006, and keeping track of task status. The worker cluster can be a JVM cluster that manages declarative provisioning images. The worker cluster receives instructions from scheduler 1004 and transmits the results to both scheduler 1004 and CIOS region 1002. CIOS container 1008 can run declarative provisioning actions in its own private Docker container 1010.

[0142] The signature agent 1012 can control the network connectivity of CIOS region 1002, which can specify whether any API calls are made to cloud service 1014. The signature agent 1012 can test the lease 1016 between scheduler 1004 and each worker 1006 from the worker cluster. In an example embodiment, the lease 1016 is a heartbeat notification, where each worker 1006 can continuously send notifications to scheduler 1004 that it is running and performing assigned tasks. The frequency of the heartbeat notifications can be the same across all workers 1006 in the worker cluster, can be selected by scheduler 1004, or can be configured by the system's user / developer. In some examples, the frequency can be two seconds, three seconds, four seconds, five seconds, or any other suitable amount of time for monitoring the progress of worker 1006. If the lease 1016 is valid, meaning that scheduler 1004 has received a heartbeat notification from worker 1006, the signature agent 1012 can execute a call 1018 to cloud service 1014. If lease 1016 is invalid, meaning scheduler 1004 has not received a heartbeat notification, signature agent 1012 may not execute call 1018 to the cloud service. Instead, signature agent 1012 may cut off all outgoing network traffic and not generate API calls to prevent concurrent execution of tasks assigned to the worker cluster. In the example scenario where signature agent 1012 cuts off all outgoing network traffic, worker 1006, CIOS container 1008, or any other suitable component may continue to run the declarative provider, but may not make progress due to the lack of API calls to cloud service 1014.

[0143] In the example embodiment, each worker 1006 in the worker cluster is assigned at most one unique task by the scheduler 1004, and each worker sends a heartbeat notification to the scheduler 1004, thereby letting the scheduler 1004 know that the lease 1016 is valid. A worker 1006 that successfully sends a heartbeat notification can be considered a healthy worker, and a worker 1006 that fails to send a heartbeat notification can be considered an unhealthy worker. Reasons for a worker 1006 not sending a heartbeat notification include worker host failure, worker process failure, failure of the relationship between worker 1006 and scheduler 1004 (e.g., load balancer, network connectivity, etc.), or any other suitable reason for not sending a heartbeat notification. If there are unhealthy workers in the worker cluster, there may be a risk of concurrent execution of tasks. If tasks are executed concurrently, there may be a risk of infrastructure damage. When the signature agent 1012 does not receive a heartbeat notification and identifies an unhealthy worker, it can mitigate or eliminate this risk by cutting off all outgoing network traffic, thereby preventing any API calls.

[0144] Figure 11 Example flowcharts illustrating a process 1100 for implementing a CIOS according to certain embodiments of this disclosure are shown. This process is illustrated as a logic flowchart, where each operation can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation can represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0145] Additionally, process 1100 can be executed under the control of one or more computing devices or computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that executes jointly on one or more processors, via hardware, or a combination thereof. As described above, the code can be stored on a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. In some embodiments, process 1100 can be executed in parallel by multiple processors. The computer-readable storage medium can be non-transitory.

[0146] Figure 11 An example flowchart illustrating process 1100 for implementing CIOS to prevent concurrent execution of declarative infrastructure providers, according to at least one embodiment, is depicted. Process 1100 may begin at block 1102, where worker nodes (e.g., Figure 10 Worker 1006 receives tasks for deploying infrastructure resources. Tasks can be assigned by a scheduler node (e.g., ...). Figure 10 The scheduler (1004) assigns tasks to worker nodes, and at any given time a worker node can only be assigned one unique task. However, in some examples, worker nodes can handle many tasks at once.

[0147] In box 1104, the worker node provides a heartbeat notification to the scheduler node. The heartbeat notification can be a notification that the worker node is running and performing a task assigned to it by the scheduler node. The delivery of heartbeat notifications from the worker node to the scheduler node can be referred to as a lease (e.g., Figure 10 (Rental fee 1016).

[0148] In box 1106, the signature proxy (e.g., Figure 10The signature agent (1012) tracks heartbeat notifications sent by worker nodes to the scheduler node. Each worker node in the worker cluster can send a heartbeat notification to the scheduler node, and the signature agent can track each of these heartbeat notifications in an attempt to identify any worker nodes that may be unhealthy.

[0149] In box 1108, the signature agent receives from the worker node a request corresponding to the execution of the task assigned to the worker node. The request may involve a request to a cloud service (e.g., Figure 10 Make API calls to the cloud service 1002.

[0150] In box 1110, the signature agent determines whether the worker node's lease is valid, for example, as requested in box 1108. The signature agent can determine whether the heartbeat notification from the worker node was correctly sent to the scheduler node. If the heartbeat notification was not correctly sent from the worker node to the scheduler node, the signature agent can consider the worker node unhealthy.

[0151] In box 1112, the signing agent determines the lease is valid, and the signing agent makes calls to the cloud service on behalf of the worker node. If the worker node successfully sends a heartbeat notification to the scheduler node, the signing agent can determine that the worker node is healthy. The signing agent can then perform API calls to the cloud service on behalf of the healthy worker node.

[0152] In block 1114, based on the signing agent's determination that the lease is invalid, the signing agent does not make any calls to the cloud service and instead blocks worker nodes from accessing the cloud service. If a worker node fails to successfully send a heartbeat notification to the scheduler node, the signing agent can determine that the worker node is unhealthy. An unhealthy worker node can execute tasks that are already being executed by different healthy worker nodes. Executing tasks concurrently in this way may risk damaging the infrastructure. To mitigate or eliminate this risk, the signing agent can cut off all outgoing network traffic and prevent worker nodes from accessing the cloud service, which prevents the concurrent execution of tasks assigned to worker nodes.

[0153] Figures 12-13 The illustrations depict aspects of an example environment for implementing aspects of this disclosure, according to various embodiments. Figure 12A simplified diagram of a distributed system 1200 for implementing embodiments of the present disclosure is depicted. In the illustrated embodiment, the distributed system 1200 includes one or more client computing devices 1202, 1204, 1206, and 1208 configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., via one or more networks 1210. A server 1212 may be communicatively coupled to remote client computing devices 1202, 1204, 1206, and 1208 via network 1210.

[0154] In various embodiments, server 1212 may be adapted to run one or more services or software applications, such as services and applications providing identity management services. In some embodiments, server 1212 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided to users of client computing devices 1202, 1204, 1206, and / or 1208 as web-based services, cloud services, or under a Software as a Service (SaaS) model. Users operating client computing devices 1202, 1204, 1206, and / or 1208 may then interact with server 1212 using one or more client applications to utilize the services provided by these components.

[0155] exist Figure 12 In the configuration depicted, software components 1218, 1220, and 1222 of system 1200 are shown as being implemented on server 1212. In other embodiments, one or more components of system 1200 and / or the services provided by these components may also be implemented by one or more of client computing devices 1202, 1204, 1206, and / or 1208. A user operating a client computing device can then utilize one or more client applications to use the services provided by these components. These components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that various different system configurations, different from those of distributed system 1200, are possible. Therefore, Figure 12 The embodiments shown are an example of a distributed system for implementing the system of the embodiments and are not intended to be limiting.

[0156] Client computing devices 1202, 1204, 1206, and / or 1208 may include various types of computing systems. For example, client devices may include systems running software such as Microsoft Windows. Software like this, and / or portable handheld devices with multiple mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, etc. (e.g., Cellular phone Computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google) (Head-mounted display). The device can support a variety of applications, such as various internet-related applications, email, short message service (SMS) applications, and can use various other communication protocols. Client computing devices can also include general-purpose personal computers, including, for example, those running various versions of Microsoft... Apple Personal computers and / or laptops running Linux operating systems. Client computing devices can be running various commercially viable operating systems. Workstation computers running any of the following operating systems: or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as, for example, Google Chrome OS). Client computing devices may also include electronic devices capable of communicating via one or more networks, such as thin client computers, internet-enabled gaming systems (e.g., with or without...). A gesture input device (Microsoft Xbox game console) and / or a personal messaging device.

[0157] although Figure 12 The distributed system 1200 is shown as having four client computing devices, but any number of client computing devices can be supported. Other devices (such as devices with sensors) can interact with server 1212.

[0158] The network(s) 1210 in the distributed system 1200 can be any type of network familiar to those skilled in the art that can support data communication using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk, etc. For example only, the network(s) 1210 can be a local area network (LAN), an Ethernet-based network, a token ring-based network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, or a wireless network (e.g., in the IEEE 1002.11 protocol suite). (and / or any other wireless protocol under which a network operates), and / or any combination of these and / or other networks.

[0159] Server 1212 may consist of one or more general-purpose computers, special-purpose server computers (for example, including PC (personal computer) servers), Server 1212 may consist of a server, a mid-range server, a mainframe computer, a rack server, etc., a server farm, a server cluster, or any other suitable arrangement and / or combination. Server 1212 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization. One or more flexible logical storage device pools may be virtualized to maintain the server's virtual storage devices. The virtual network may be controlled by server 1212 using software-defined networking. In various embodiments, server 1212 may be adapted to run one or more services or software applications described in the foregoing disclosure. For example, server 1212 may correspond to a server according to embodiments of this disclosure for performing the processes described above.

[0160] Server 1212 can run operating systems, including any of those discussed above, as well as any commercially available server operating system. Server 1212 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, etc. Servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from Oracle, Microsoft, Sybase, IBM, etc.

[0161] In some implementations, server 1212 may include one or more applications to analyze and integrate data feeds and / or event updates received from users of client computing devices 1202, 1204, 1206, and 1208. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 1212 may receive real-time updates from one or more third-party information sources and continuous data streams, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 1212 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 1202, 1204, 1206, and 1208.

[0162] The distributed system 1200 may also include one or more databases 1214 and 1216. These databases may provide a mechanism for storing information such as user identity information and other information used in embodiments of this disclosure. Databases 1214 and 1216 may reside in a variety of locations. For example, one or more of databases 1214 and 1216 may reside on a non-transitory storage medium local to server 1212 (and / or within server 1212). Alternatively, databases 1214 and 1216 may be located remotely from server 1212 and communicate with server 1212 via a network-based connection or a dedicated connection. In one set of embodiments, databases 1214 and 1216 may reside in a storage area network (SAN). Similarly, any necessary files for performing functions belonging to server 1212 may be stored locally on server 1212 and / or remotely, as appropriate. In one set of embodiments, databases 1214 and 1216 may include relational databases, such as those provided by Oracle, adapted to store, update, and retrieve data in response to commands in SQL format.

[0163] Figure 13 An example computer system 1300 that can be used to implement embodiments of the present disclosure is illustrated. In some embodiments, computer system 1300 can be used to implement any of the various servers and computer systems described above. Figure 13 As shown, the computer system 1300 includes various subsystems, including a processing subsystem 1304 that communicates with multiple peripheral subsystems via a bus subsystem 1302. These peripheral subsystems may include a processing acceleration unit 1306, an I / O subsystem 1308, a storage subsystem 1318, and a communication subsystem 1324. The storage subsystem 1318 may include a tangible computer-readable storage medium 1322 and system memory 1310.

[0164] Bus subsystem 1302 provides a mechanism for enabling various components and subsystems of computer system 1300 to communicate with each other as intended. Although bus subsystem 1302 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1302 may be any of several types of bus structures using any of a variety of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses that may be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0165] The processing subsystem 1304 controls the operation of the computer system 1300 and may include one or more processing units 1332, 1334, etc. A processing unit may include one or more processors, including single-core or multi-core processors, one or more cores of a processor, or a combination thereof. In some embodiments, the processing subsystem 1304 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some embodiments, some or all of the processing units of the processing subsystem 1304 may be implemented using custom circuitry such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0166] In some embodiments, the processing unit in processing subsystem 1304 may execute instructions stored in system memory 1310 or computer-readable storage medium 1322. In various embodiments, the processing unit may execute multiple program or code instructions and may maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside on system memory 1310 and / or computer-readable storage medium 1310, including potentially on one or more storage devices. With appropriate programming, processing subsystem 1304 can provide the various functions described above for dynamically modifying documents (e.g., web pages) in response to usage patterns.

[0167] In some embodiments, the processing acceleration unit 1306 may be provided for performing custom processing or for offloading some processing performed by the processing subsystem 1304 in order to accelerate the overall processing performed by the computer system 1300.

[0168] I / O subsystem 1308 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 1300. Generally, the term "input device" is used to encompass all possible types of devices and mechanisms for inputting information into computer system 1300. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens incorporated into a display, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as those enabling users to control the input device (Microsoft...). The 360 ​​Game Controller (a device that provides an interface for receiving input using gestures and verbal commands) and interacting with input devices is a Microsoft product. Motion sensors. User interface input devices may also include eye gesture recognition devices, such as those from Google. A blink detector detects eye movements from the user (e.g., blinks when taking a photo and / or making menu selections) and translates these eye gestures into the input device (e.g., Google). Input from a voice recognition system (e.g., voice commands). Additionally, the user interface input device may include features that enable the user to interact with a voice recognition system via voice commands. A voice recognition sensing device that interacts with a navigator.

[0169] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers, and graphics tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0170] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tube (CRT), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is used to encompass all possible types of devices and mechanisms for outputting information from computer system 1300 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0171] Storage subsystem 1318 provides a repository or data repository for storing information used by computer system 1300. Storage subsystem 1318 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the functionality described above, when executed by processing subsystem 1304, can be stored in storage subsystem 1318. This software can be executed by one or more processing units of processing subsystem 1304. Storage subsystem 1318 may also provide a repository for storing data used according to this disclosure.

[0172] The storage subsystem 1318 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. For example... Figure 13 As shown, the storage subsystem 1318 includes system memory 1310 and computer-readable storage medium 1322. System memory 1310 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS) containing basic routines that facilitate the transfer of information between elements within computer system 1300, such as during startup, may be stored in ROM. RAM may contain data and / or program modules currently operated and executed by processing subsystem 1304. In some implementations, system memory 1310 may include various types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0173] For example, rather than limiting, such as Figure 13 As shown, system memory 1310 can store applications 1312, program data 1314, and operating system 1316, which may include client applications, web browsers, middleware applications, relational database management systems (RDBMS), etc. For example, operating system 1316 may include various versions of Microsoft... Apple and / or Linux operating system, multiple commercially available options Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS (and / or mobile operating systems, such as iOS) and / or mobile operating systems Phone OS 10OS and OS operating system.

[0174] Computer-readable storage medium 1322 may store programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the functionality described above to the processor, when executed by processing subsystem 1304, may be stored in storage subsystem 1318. For example, computer-readable storage medium 1322 may include non-volatile memory such as hard disk drives, disk drives, CD-ROMs, DVDs, Blu-ray discs, etc. Optical disc drives, such as disks or other optical media. Computer-readable storage media.

[0175] 1322 may include, but is not limited to, Drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital video tapes, etc. Computer-readable storage medium 1322 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash-based SSDs, enterprise flash drives, solid-state ROMs, etc.), SSDs based on volatile memory (such as SSDs based on solid-state RAM, dynamic RAM, static RAM, DRAM), magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash-based SSDs. Computer-readable medium 1322 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1300.

[0176] In some embodiments, the storage subsystem 1300 may further include a computer-readable storage medium reader 1320 that may be further connected to the computer-readable storage medium 1322. The computer-readable storage medium 1322, together with and optionally combined with the system memory 1310, may collectively represent remote, local, fixed and / or removable storage devices and storage media for storing computer-readable information.

[0177] In some embodiments, computer system 1300 may support the execution of one or more virtual machines. Computer system 1300 may execute programs such as hypervisors to facilitate the configuration and management of the virtual machines. Each virtual machine may be allocated memory resources, computing resources (e.g., processor, cores), I / O resources, and networking resources. Each virtual machine may run its own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1300. Therefore, multiple operating systems may potentially run concurrently by computer system 1300. Each virtual machine typically runs independently of other virtual machines.

[0178] The communication subsystem 1324 provides an interface to other computer systems and networks. The communication subsystem 1324 serves as an interface for receiving data from and sending data to other systems from the computer system 1300. For example, the communication subsystem 1324 enables the computer system 1300 to establish a communication channel via the Internet to one or more client devices for receiving and sending information to the client devices. Additionally, the communication subsystem 1324 can be used to transmit notifications of successful login or password re-entry from the privileged account manager to the requesting user.

[0179] The communication subsystem 1324 may support both wired and / or wireless communication protocols. For example, in some embodiments, the communication subsystem 1324 may include radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and / or other components for accessing wireless voice and / or data networks, such as using cellular telephone technology, advanced data network technologies like 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 902.11 series standards, or other mobile communication technologies, or any combination thereof). In some embodiments, in addition to or as an alternative to a wireless interface, the communication subsystem 1324 may provide a wired network connection (e.g., Ethernet).

[0180] The communication subsystem 1324 can receive and transmit data in various forms. For example, in some embodiments, the communication subsystem 1324 can receive input communications in the form of structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, etc. For example, the communication subsystem 1324 can be configured to receive (or transmit) data feeds 1326 in real time from users of social media networks and / or other communication services, such as... feed, Updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0181] In some embodiments, the communication subsystem 1324 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1328 of real-time events and / or event updates 1330, and may be continuous or unbounded in nature without a definite end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0182] The communication subsystem 1324 can also be configured to output structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, etc., to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1300.

[0183] Computer system 1300 can be one of various types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), and wearable devices (e.g., Google). Head-mounted displays, personal computers, workstations, mainframes, kiosks, server racks, or any other data processing systems.

[0184] Due to the constantly changing nature of computers and networks, Figure 13 The description of the computer system 1300 depicted is intended only as a concrete example. It has more than Figure 13 Many other configurations of the system depicted, with more or fewer components, are possible. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0185] Some of the systems depicted in the diagram can be provided in various configurations. In some embodiments, the system can be configured as a distributed system, wherein one or more components of the system are distributed across one or more networks in one or more cloud infrastructure systems.

[0186] A cloud infrastructure system is a collection of one or more server computing devices, network devices, and / or storage devices. These resources can be partitioned and allocated to their customers by cloud service providers in several ways. For example, cloud service providers (such as Oracle Corporation of Redwood Coast, California) can provide various types of cloud services, including but not limited to one or more services offered under the Software as a Service (SaaS) category, services offered under the Platform as a Service (PaaS) category, services offered under the Infrastructure as a Service (IaaS) category, or other categories of services that include hybrid services. Examples of SaaS services include, but are not limited to, the ability to build and deliver a suite of on-demand applications (such as Oracle Fusion applications). SaaS services enable customers to leverage applications that run on a cloud infrastructure system without requiring customers to purchase software for the applications. Examples of PaaS services include, but are not limited to, services that enable organizations (such as Oracle) to consolidate existing applications on a shared, common architecture, and the ability to build new applications that utilize shared services provided by the platform (such as Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), and other services). IaaS services can facilitate customer management and control of underlying computing resources, such as storage, networking, and other basic computing resources, when leveraging services provided by SaaS and PaaS platforms.

[0187] Figure 14This is a simplified block diagram of one or more components of a system environment 1400 according to an embodiment of the present disclosure, through which services provided by one or more components of the embodiment system can be provided as cloud services. In the illustrated embodiment, system environment 1400 includes one or more client computing devices 1404, 1406, and 1408, which users can use to interact with a cloud infrastructure system 1402 providing cloud services. The client computing devices can be configured to operate client applications, such as web browsers, proprietary client applications (e.g., OracleForms), or other applications, which users of the client computing devices can use to interact with the cloud infrastructure system 1402 to use the services provided by the cloud infrastructure system 1402.

[0188] It should be understood that the cloud infrastructure system 1402 depicted in the figures may have components other than those depicted. Furthermore, the embodiments shown in the figures are merely one example of a cloud infrastructure system that can be incorporated into embodiments of this disclosure. In some other embodiments, the cloud infrastructure system 1402 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations or arrangements.

[0189] Client computing devices 1404, 1406, and 1408 can be devices similar to those described above for 1202, 1204, 1206, and 1208.

[0190] Although the exemplary system environment 1400 is shown to have three client computing devices, any number of client computing devices can be supported. Other devices (such as devices with sensors) can interact with the cloud infrastructure system 1402.

[0191] One or more networks 1410 can facilitate communication and data exchange between clients 1404, 1406, and 1408 and cloud infrastructure system 1402. Each network can be any type of network familiar to those skilled in the art that supports data communication using any of a variety of commercially available protocols, including those described above for one or more networks 1210.

[0192] The cloud infrastructure system 1402 may include one or more computers and / or servers, which may include those described above for server 1212.

[0193] In some embodiments, the services provided by a cloud infrastructure system may include a wide range of services offered on demand to users of the cloud infrastructure system, such as online data storage and backup solutions, web-based email services, hosted office suites and document collaboration services, database processing, managed technical support services, and so on. The services provided by the cloud infrastructure system can be dynamically scaled to meet the needs of its users. A specific instantiation of a service provided by the cloud infrastructure system is referred to herein as a “service instance.” Generally, any service available to users from a cloud service provider’s system via a communication network (such as the Internet) is referred to as a “cloud service.” In a public cloud environment, the servers and systems that make up the cloud service provider’s system differ from the on-premises servers and systems of a customer’s own. For example, the cloud service provider’s system may host applications, and users may subscribe to and use these applications on demand via a communication network such as the Internet.

[0194] In some examples, services within a computer network cloud infrastructure may include protected computer network access to storage devices, hosted databases, hosted web servers, software applications, or other services provided to users by a cloud provider or as otherwise known in the art. For example, services may include password-protected access to remote storage devices in the cloud via the Internet. As another example, services may include hosted relational databases based on web services and scripting language middleware engines for web developers. As yet another example, services may include access to email software applications hosted on a cloud provider's website.

[0195] In some embodiments, cloud infrastructure system 1402 may include a suite of application, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such a cloud infrastructure system is the Oracle public cloud provided by this assignee.

[0196] In various embodiments, cloud infrastructure system 1402 may be adapted to automatically provision, manage, and track customer subscriptions to services provided by cloud infrastructure system 1402. Cloud infrastructure system 1402 may provide cloud services via different deployment models. For example, services may be provided under a public cloud model, in which cloud infrastructure system 1402 is owned by an organization selling cloud services (e.g., owned by Oracle) and the services are available to the general public or businesses in different industries. As another example, services may be provided under a private cloud model, in which cloud infrastructure system 1402 is operated for a single organization and can provide services to one or more entities within that organization. Cloud services may also be provided under a community cloud model, in which cloud infrastructure system 1402 and the services provided by cloud infrastructure system 1402 are shared by several organizations in a relevant community. Cloud services may also be provided under a hybrid cloud model, which is a combination of two or more different models.

[0197] In some embodiments, the services provided by the cloud infrastructure system 1402 may include one or more services offered under the Software as a Service (SaaS) category, the Platform as a Service (PaaS) category, the Infrastructure as a Service (IaaS) category, or other categories of services that include hybrid services. Customers may subscribe to one or more services provided by the cloud infrastructure system 1402 via a subscription order. The cloud infrastructure system 1402 then performs processing to provide the services in the customer's subscription order.

[0198] In some embodiments, the services provided by the cloud infrastructure system 1402 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, application services may be provided by the cloud infrastructure system via a SaaS platform. The SaaS platform may be configured to provide cloud services belonging to the SaaS category. For example, a SaaS platform may provide the ability to build and deliver a suite of on-demand applications on an integrated development and deployment platform. The SaaS platform may manage and control the underlying software and infrastructure used to provide SaaS services. By utilizing the services provided by the SaaS platform, customers can leverage applications running on the cloud infrastructure system. Customers can obtain application services without having to purchase separate licenses and support. A variety of different SaaS services may be provided. Examples include, but are not limited to, services providing solutions for sales performance management, enterprise integration, and business agility for large organizations.

[0199] In some embodiments, platform services may be provided by a cloud infrastructure system via a PaaS platform. The PaaS platform can be configured to provide cloud services that fall under the PaaS category. Examples of platform services may include, but are not limited to, services that enable organizations (such as Oracle) to integrate existing applications on a shared, public architecture, and the ability to build new applications that leverage the shared services provided by the platform. The PaaS platform can manage and control the underlying software and infrastructure used to provide PaaS services. Customers can obtain PaaS services provided by the cloud infrastructure system without having to purchase separate licenses and support. Examples of platform services include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), and others.

[0200] By leveraging the services provided by a PaaS platform, customers can employ programming languages ​​and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, the platform services provided by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion Middleware Service), and Java cloud services. In one embodiment, the database cloud service may support a shared services deployment model, which enables organizations to aggregate database resources and offer Database as a Service to customers in the form of a database cloud. Within the cloud infrastructure system, the middleware cloud service can provide customers with a platform to develop and deploy various business applications, and the Java cloud service can provide customers with a platform to deploy Java applications.

[0201] Various infrastructure services can be provided by IaaS platforms within cloud infrastructure systems. Infrastructure services facilitate the management and control of underlying computing resources (such as storage, networking, and other basic computing resources) for customers utilizing services provided by SaaS and PaaS platforms.

[0202] In some embodiments, the cloud infrastructure system 1402 may also include infrastructure resources 1430 for providing resources for offering various services to customers of the cloud infrastructure system. In one embodiment, infrastructure resources 1430 may include a pre-integrated and optimized combination of hardware (such as servers, storage devices, and networking resources) to perform services provided by PaaS platforms and SaaS platforms.

[0203] In some embodiments, resources in cloud infrastructure system 1402 can be shared by multiple users and dynamically reallocated on demand. Additionally, resources can be allocated to users in different time zones. For example, cloud infrastructure system 1430 can enable a first group of users in a first time zone to utilize the resources of the cloud infrastructure system for a specified number of hours, and then enable the same resources to be reallocated to another group of users located in a different time zone, thereby maximizing resource utilization.

[0204] In some embodiments, multiple internal shared services 1432 may be provided, shared by different components or modules of the cloud infrastructure system 1402 and services provided by the cloud infrastructure system 1402. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, cloud-enabled services, email services, notification services, file transfer services, etc.

[0205] In some embodiments, cloud infrastructure system 1402 can provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) within the cloud infrastructure system. In one embodiment, cloud management functionality may include the ability to provision, manage, and track customer subscriptions received by cloud infrastructure system 1402.

[0206] In one embodiment, as depicted in the figure, cloud management functionality may be provided by one or more modules, such as order management module 1420, order orchestration module 1422, order provisioning module 1424, order management and monitoring module 1426, and identity management module 1428. These modules may include or be provided using one or more computers and / or servers, which may be general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0207] In exemplary operation 1434, a customer using a client device such as client device 1404, 1406, or 1408 can interact with cloud infrastructure system 1402 by requesting one or more services provided by cloud infrastructure system 1402 and placing an order to subscribe to one or more services provided by cloud infrastructure system 1402. In some embodiments, the customer can access cloud user interfaces (UIs), cloud UI 1412, cloud UI 1414, and / or cloud UI 1416, and place subscription orders via these UIs. Order information received by cloud infrastructure system 1402 in response to a customer placing an order may include information identifying the customer and the one or more services provided by cloud infrastructure system 1402 that the customer intends to subscribe to.

[0208] After a customer has placed an order, the order information is received via cloud UI 1412, 1414 and / or 1416.

[0209] At operation 1436, the order is stored in order database 1418. Order database 1418 can be one of several databases operated by cloud infrastructure system 1418 and in conjunction with other system elements.

[0210] At operation 1438, the order information is forwarded to the order management module 1420. In some cases, the order management module 1420 can be configured to perform order-related billing and accounting functions, such as verifying the order and, after verification, reserving the order.

[0211] At operation 1440, order information is transmitted to order orchestration module 1422. Order orchestration module 1422 can use the order information to orchestrate the supply of services and resources for orders placed with customers. In some cases, order orchestration module 1422 can orchestrate the supply of resources to support subscribed services using the services of order supply module 1424.

[0212] In some embodiments, the order orchestration module 1422 enables the management of business processes associated with each order and applies business logic to determine whether an order should be provisioned. At operation 1442, upon receiving a new subscription order, the order orchestration module 1422 sends a request to the order provisioning module 1424 to allocate resources and configure those resources required to satisfy the subscription order. The order provisioning module 1424 enables the allocation of resources for services subscribed by customers. The order provisioning module 1424 provides a level of abstraction between the cloud services provided by the cloud infrastructure system 1400 and the physical implementation layer used to provision resources to provide the requested services. The order orchestration module 1422 can therefore be isolated from implementation details such as whether services and resources are actually provisioned on demand or pre-provisioned and allocated / assigned only upon request.

[0213] At operation 1444, once services and resources are provisioned, notifications of the provided services can be sent by the order provisioning module 1424 of the cloud infrastructure system 1402 to customers on client devices 1404, 1406, and / or 1408. At operation 1446, customer subscription orders can be managed and tracked by the order management and monitoring module 1426. In some cases, the order management and monitoring module 1426 can be configured to collect service usage statistics from subscription orders, such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime.

[0214] In some embodiments, the cloud infrastructure system 1400 may include an identity management module 1428. The identity management module 1428 may be configured to provide identity services, such as access management and authorization services within the cloud infrastructure system 1400. In some embodiments, the identity management module 1428 may control information about customers who wish to utilize the services provided by the cloud infrastructure system 1402. Such information may include information authenticating the identities of such customers and information describing which actions those customers are authorized to perform relative to various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). The identity management module 1428 may also include management of descriptive information about each customer and about how and by whom that descriptive information can be accessed and modified.

[0215] Although specific embodiments of this disclosure have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments of this disclosure are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Additionally, although embodiments of this disclosure have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0216] Furthermore, while embodiments of this disclosure have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments of this disclosure may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same or different processors in any combination. Thus, where a component or module is described as being configured to perform certain operations, such a configuration may be implemented, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations, or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0217] Therefore, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, although specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims. Modifications include any related combination of the disclosed features.

Claims

1. A method for deploying infrastructure resources using declarative provisioning tools, the method comprising: The scheduler receives a configuration file from the group manager of the central infrastructure service for the first deployment of resources at the execution target. The configuration file includes (i) a declarative infrastructure provisioning tool plugin generated by the group manager and (ii) a first release identifier ID inserted into the configuration file by the group manager. The resource is deployed to the execution target by at least one of multiple worker nodes according to the configuration file; Store the current state of the resource; The scheduler receives a second version of the configuration file from the group manager for new deployment at the execution target, the second version of the configuration file including a second release ID inserted by the group manager into the second version of the configuration file; The declarative infrastructure provisioning tool plugin is executed through at least one of the plurality of worker nodes to compare the first release ID with the second release ID; Based on the difference between the first release ID and the second release ID, the current state of the resource is compared with the expected state of the resource according to the second version of the configuration file via the Cloud Infrastructure Orchestration Service CIOS container; as well as Since the current state differs from the desired state, the resource is deployed at the execution target according to the second version of the configuration file by assigning the task to at least one of the multiple worker nodes.

2. The method of claim 1, wherein the scheduler is implemented by a regional infrastructure service.

3. The method of claim 2, wherein the regional infrastructure service implements the declarative infrastructure provisioning tool.

4. The method of claim 2 or claim 3, wherein the scheduler communicates with the plurality of worker nodes.

5. The method of claim 1, wherein the central infrastructure service is mandatory.

6. The method of claim 1, wherein at least one of the first release ID or the second release ID is randomly generated by the group manager of the central infrastructure service.

7. The method of claim 1, wherein the configuration file further includes a first execution ID, wherein a second version of the configuration file further includes a second execution ID, and the method further includes: Based on the fact that the current state is the same as the expected state, the first execution ID and the second execution ID are compared through the CIOS container; as well as Since the first execution ID is different from the second execution ID, the resource is deployed at the execution target according to the second version of the configuration file.

8. A system for deploying infrastructure resources using declarative provisioning tools, the system comprising: One or more processors; as well as One or more memories storing computer-executable instructions that, when executed by the one or more processors, configure the one or more processors to: The scheduler receives a configuration file from the group manager of the central infrastructure service for the first deployment of resources at the execution target. The configuration file includes (i) a declarative infrastructure provisioning tool plugin generated by the group manager and (ii) a first release identifier ID inserted into the configuration file by the group manager. The resource is deployed to the execution target by at least one of multiple worker nodes according to the configuration file; Store the current state of the resource; The scheduler receives a second version of the configuration file from the group manager for new deployment at the execution target, the second version of the configuration file including a second release ID inserted by the group manager into the second version of the configuration file; The declarative infrastructure provisioning tool plugin is executed through at least one of the plurality of worker nodes to compare the first release ID with the second release ID; Based on the difference between the first release ID and the second release ID, the current state of the resource is compared with the expected state of the resource according to the second version of the configuration file via the Cloud Infrastructure Orchestration Service CIOS container; as well as Since the current state differs from the expected state, the resources are deployed at the execution target using the declarative infrastructure provisioning tool according to the second version of the configuration file by assigning the task to at least one of the multiple worker nodes.

9. The system of claim 8, wherein the scheduler is implemented by a regional infrastructure service.

10. The system of claim 9, wherein the scheduler communicates with the plurality of worker nodes.

11. The system of claim 8, wherein at least one of the first release ID or the second release ID is randomly generated by the group manager of the central infrastructure service.

12. The system of claim 8, wherein the configuration file further includes a first execution ID, wherein a second version of the configuration file further includes a second execution ID, and wherein the one or more memories store additional computer-executable instructions that, when executed by the one or more processors, configure the one or more processors to further: Based on the fact that the current state is the same as the expected state, the first execution ID and the second execution ID are compared through the CIOS container; and Since the first execution ID is different from the second execution ID, the resource is deployed at the execution target according to the second version of the configuration file.

13. A non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations including: The scheduler receives a configuration file from the group manager of the central infrastructure service for the first deployment of resources at the execution target. The configuration file includes (i) a declarative infrastructure provisioning tool plugin generated by the group manager and (ii) a first release identifier ID inserted into the configuration file by the group manager. The resource is deployed to the execution target by at least one of multiple worker nodes according to the configuration file; Store the current state of the resource; The scheduler receives a second version of the configuration file from the group manager for new deployment at the execution target, the second version of the configuration file including a second release ID inserted by the group manager into the second version of the configuration file; The declarative infrastructure provisioning tool plugin is executed through at least one of the plurality of worker nodes to compare the first release ID with the second release ID; Based on the difference between the first release ID and the second release ID, the current state of the resource is compared with the expected state of the resource according to the second version of the configuration file via the Cloud Infrastructure Orchestration Service CIOS container; as well as Since the current state differs from the desired state, the resource is deployed at the execution target according to the second version of the configuration file by assigning the task to at least one of the multiple worker nodes.

14. The non-transitory computer-readable storage medium of claim 13, wherein the scheduler communicates with the plurality of worker nodes.

15. The non-transitory computer-readable storage medium of claim 13, wherein at least one of the first release ID or the second release ID is randomly generated by the group manager of the central infrastructure service.

16. The non-transitory computer-readable storage medium of claim 13, wherein the configuration file further includes a first execution ID, wherein a second version of the configuration file further includes a second execution ID, and the non-transitory computer-readable storage medium stores additional computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: Based on the fact that the current state is the same as the expected state, the first execution ID and the second execution ID are compared through the CIOS container; and Since the first execution ID is different from the second execution ID, the resource is deployed at the execution target according to the second version of the configuration file.

17. An apparatus for deploying infrastructure resources using declarative provisioning tools, the apparatus comprising components for performing the method according to any one of claims 1-7.

Images