Configuration freeze and change management for services deployed via continuous delivery on a data center configured on a cloud platform

Abstract

A computing system, e.g., a multi-tenant system, deploys software artifacts to a datacenter created in a cloud platform using a cloud platform infrastructure language that is cloud platform agnostic. The system allows a user to specify a system configuration freeze for a time interval for a service running in a datacenter configured on the cloud platform. During the system freeze, changes to the system are prohibited or put on hold. The system generates a pipeline for modifying the system configuration of the service. The pipeline includes a change stage for making changes associated with the service and a pre-change stage for acquiring a lock. When the system receives a request to perform a system freeze of a datacenter entity, the system acquires a lock for the service of the datacenter entity. As a result, execution of a pipeline configured to change the system configuration of the service is put on hold.

Description

[Technical field]

[0001] The present disclosure relates generally to managing software releases in a cloud computing platform and enforcing a system configuration freeze for services deployed via continuous delivery on a data center configured on a cloud computing platform, as well as managing changes in software artifacts deployed on a data center configured on a cloud computing platform. [Background technology]

[0002] Organizations are increasingly turning to cloud platforms (or cloud computing platforms) such as AWS (AMAZON® WEB SERVICES), GOOGLE® CLOUD PLATFORM, MICROSOFT® AZURE, etc. for their infrastructure needs. Cloud platforms provide servers, storage, databases, networking, software, etc. to organizations over the Internet. Organizations are shifting their services to cloud platforms that offer scalability and elasticity of computing resources.

[0003] An organization maintains a cloud infrastructure on a cloud platform using a continuous delivery platform that can manage and deploy applications on the cloud platform. Such a continuous delivery platform allows an organization to simplify the software deployment process and manage applications, firewalls, clusters, servers, load balancers, and other computing infrastructure on the cloud platform. The continuous delivery platform makes it easier for developers to make changes to software artifacts that affect services running on the system and to deploy the updated software artifacts. However, there are periods during which any changes to services running on the system are undesirable. The organization providing the system may prefer to impose a system freeze (also called a feature freeze or computing moratorium) for such periods, for example, when the system is expecting particularly high traffic. Modifying software artifacts associated with a service may reduce the stability of the system, since the organization may face unforeseeable situations, such as software defects or software bugs, as a result of the changes.

[0004] A large-scale system, such as a multi-tenant system, may manage services for multiple organizations that represent tenants of the multi-tenant system and may interact with multiple cloud platforms. A multi-tenant system may need to maintain thousands of tenants across multiple cloud platforms. Furthermore, the software, languages, and features supported by each cloud platform may differ. As a result, tracking changes made to services deployed on a data center configured on a cloud platform is a tedious and error-prone process. As a result, even if there is a computing moratorium on a service deployed in an online system, there is a high possibility that a developer may change software artifacts associated with the service, thereby reducing the stability of the system.

[0005] Furthermore, several users and teams of users may be involved in software releases associated with a service. As a result, tracking changes made to services deployed on a data center configured on a cloud platform is complicated because frequent changes may be made to source code and system configurations. Tracking changes to services deployed on a large system is cumbersome because related information is either difficult to access or unavailable. This results in high maintenance costs for multi-tenant systems to support and track software releases and changes to software releases on a data center configured on a cloud platform. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram of a system environment illustrating a multi-tenant system that configures a data center on a cloud platform, according to one embodiment. [Figure 2A] 2 is a block diagram illustrating a system architecture of a deployment module 210 according to one embodiment. [Figure 2B] 1 illustrates an overall process for deploying software artifacts to a data center, according to one embodiment. [Diagram 3] FIG. 2 is a block diagram illustrating the architecture of a software release management module according to one embodiment. [Figure 4] 1 illustrates an example of a data center declarative specification according to one embodiment. [Diagram 5] 1 illustrates an exemplary data center created on a cloud platform based on a declarative specification, according to one embodiment. [Figure 6] FIG. 2 is a block diagram illustrating the creation of a data center on a cloud platform based on a declarative specification, according to one embodiment. [Figure 7] 1 illustrates an overall process for generating a pipeline for deployment of software artifacts on a data center configured on a cloud platform, according to one embodiment. [Figure 8] 1 illustrates an exemplary master pipeline, according to one embodiment. [Figure 9] 1 illustrates the overall process performed by stages for an environment of a master pipeline on a cloud platform, according to one embodiment. [Figure 10] 1 illustrates an exemplary master pipeline, according to one embodiment. [Figure 11] 1 illustrates a system architecture of a system configuration freeze module, according to one embodiment. [Figure 12] 1 illustrates an exemplary pipeline for forcing a system configuration freeze, according to one embodiment. [Figure 13] 1 illustrates a process for making changes to system configurations of services deployed on a data center in a cloud platform, according to one embodiment. [Figure 14] 1 illustrates an overall process for performing a system configuration freeze of data center entities of a data center configured on a cloud platform, according to one embodiment. [Figure 15] 1 illustrates a system architecture for a change processing module, according to one embodiment. [Figure 16] 1 illustrates an exemplary master pipeline for managing changes, according to one embodiment. [Figure 17] 1 illustrates an overall process for change management of services deployed on a data center configured on a cloud platform, according to one embodiment. [Figure 18] 1 illustrates a process performed by a change management stage of a master pipeline, according to one embodiment. [Figure 19] 1 illustrates a process for managing a queue that collects event information related to changes, according to one embodiment. [Figure 20] 2 is a block diagram illustrating a functional diagram of an exemplary computer system for use in the environment of FIG. 1, according to one embodiment.

[0007] The drawings depict various embodiments for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the embodiments described herein.

[0008] The drawings use like reference numbers to identify like elements. A letter following a reference number, such as "115a," indicates that the text specifically refers to the element with that particular reference number. A reference number in text without a following letter, such as "115," refers to any or all of the elements in the figure with that reference number. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A cloud platform provides computing resources, such as storage, computing resources, and applications, to computing systems on an on-demand basis over a public network, such as the Internet. Cloud platforms enable enterprises to minimize up-front costs for setting up computing infrastructure and also enable enterprises to launch and run applications more quickly with less maintenance overhead. Cloud platforms also enable enterprises to adjust computing resources to rapidly fluctuating and unpredictable demand. Enterprises can create data centers using cloud platforms for use by the enterprise's users. However, implementing a data center on each cloud platform requires expertise in the technology of the cloud platform.

[0010] An embodiment creates data centers in a cloud platform using a cloud platform infrastructure language that is cloud platform independent. The system receives a cloud platform independent declarative specification of the data center. The declarative specification describes the structure of the data center and may not provide instructions specifying how to create the data center. The cloud platform independent declarative specification is configured to generate the data center on any of a plurality of cloud platforms and is specified using the cloud platform infrastructure language. The system receives information identifying a target cloud platform for creating the data center and compiles the cloud platform independent declarative specification to generate a cloud platform specific data center representation. The system transmits the cloud platform specific data center representation and a set of instructions for execution in the target cloud platform. The target cloud platform executes the instructions to configure the data center using the platform specific data center representation. The system provides a user with access to computing resources of the data center configured by the cloud platform.

[0011] The system performs operations related to a software release on a data center configured on a cloud platform, such as deploying the software release, provisioning resources, performing a rollback of the software release, etc. The system accesses a data center configured on a target cloud platform. The data center is generated based on a cloud platform independent declarative specification that includes a hierarchy of data center entities. Each data center entity includes one or more of: (1) a service, or (2) one or more other data center entities. The system generates a cloud platform independent master pipeline that includes: (1) a sequence of stages for deployment of software artifacts, such as a development stage, a test stage, and a production stage, and (2) criteria for promoting the software artifacts from one stage in the sequence of stages to a subsequent stage. The system compiles the cloud platform independent master pipeline to generate a cloud platform dependent detailed pipeline for the target cloud platform with instructions for performing operations related to the services according to a layout of the data center defined by the declarative specification. The system executes a cloud platform dependent detailed pipeline on a target cloud platform to deploy, for example, a software release onto data center entities of a data center.

[0012] The system allows users to specify a system freeze (also called a computing moratorium) for a certain time interval for services running in a data center configured on a cloud platform. During a system freeze, changes to the system are prohibited or put on hold. Changes that may be prevented during a system freeze include upgrades to software artifacts, changes to the configuration of resources used by the system (e.g., infrastructure resources such as load balancers, databases, servers, firewalls, network resources, etc.), data center configuration changes, modifications to libraries and other system artifacts used by the system, modifications to applications, etc. While online systems may use continuous delivery platforms to allow developers to modify source code used in software artifacts, the embodiments disclosed herein allow organizations to put on hold any such changes for a specific time interval.

[0013] As an example, an organization that provides services through an online system may determine that during periods of high traffic, services running on the online system should not be impeded. Thus, embodiments allow the system to be frozen so that the system continues to operate and no changes can be made to the system. This reduces the likelihood of experiencing system downtime.

[0014] As another example, a system may have some problems that require a team to analyze the system to determine the cause of the problem. However, during the period when the system is being analyzed to determine the cause of a particular problem, a system administrator may prefer to freeze the services of the system so that no changes can be made to the software artifacts while the system is being monitored and debugged. This helps isolate problems that fixes to the system while it is being monitored and debugged make it difficult to determine whether the problem was caused by the original software of the service or by some changes made to the software while the problem was being debugged.

[0015] As another example, there may be multiple subsystems running within a system. The disclosed embodiments allow the subsystems to be modified in a mutually exclusive manner such that only one subsystem (or a small set of subsystems) is modified at a time. A particular subsystem may be modified at a time by deploying a software artifact for that subsystem. A system administrator may prefer to impose a moratorium on the remaining subsystems such that only one subsystem is modified at a time. Once a subsystem is deployed and tested to ensure that it is running reliably before allowing other subsystems to proceed with making changes to their respective software artifacts, the moratorium is lifted. The embodiments allow such a moratorium to be imposed on a system, thereby selectively preventing modifications to software artifacts deployed within the system.

[0016] The cloud platform is also referred to herein as the substrate. The system may represent, but is not limited to, a multi-tenant system and may be any online system or any computing system that has network access to the cloud platform.

[0017] Overall System Environment 1 is a block diagram of a system environment illustrating a multi-tenant system constituting a data center on a cloud platform, according to one embodiment. System environment 100 includes multi-tenant system 110, one or more cloud platforms 120, and one or more client devices 105. In other embodiments, system environment 100 may include more or fewer components.

[0018] The multi-tenant system 110 stores information for one or more tenants 115. Each tenant may be associated with a business that represents a customer of the multi-tenant system 110. Each tenant may have multiple users that interact with the multi-tenant system via client devices 105.

[0019] A cloud platform may also be referred to as a cloud computing platform or a public cloud environment. A tenant may use a cloud platform infrastructure language to provide a declarative specification of a data center to be created on a target cloud platform 120 and may use the data center to perform operations, such as provisioning resources, performing software releases, etc. A tenant 115 may create one or more data centers on the cloud platform 120. A data center represents a set of computing resources including servers, applications, storage, memory, etc. that may be used by a user, e.g., a user associated with the tenant. Each tenant may provide different functionality to the tenant's users. Thus, each tenant may run different services on the data center configured for the tenant. A multi-tenant system may implement different mechanisms for software release and deployment for each tenant. A tenant may further obtain or develop a version of software that includes instructions for various services running in the data center. An embodiment allows a tenant to deploy specific versions of software releases for different services running on different computing resources of the data center.

[0020] The computing resources of a data center are secure and cannot be accessed by users who are not authorized to access them. For example, a data center 125a created for a user of tenant 115a cannot be accessed by a user of tenant 115b unless access is explicitly granted. Similarly, a data center 125b created for a user of tenant 115b cannot be accessed by a user of tenant 115a unless access is explicitly granted. Furthermore, services provided by a data center can be accessed by a computing system outside of the data center only if access is granted to the computing system according to the declarative specification of the data center.

[0021] In a multi-tenant system 110, data for multiple tenants may be stored in the same physical database. However, the database is configured such that one tenant's data is kept logically separate from other tenants' data, such that one tenant does not have access to another tenant's data unless such data is explicitly shared. It is transparent to the tenants that their data may be stored in tables that are shared with other customers' data. A database table may store rows for multiple tenants. Thus, in a multi-tenant system, various elements of the system's hardware and software may be shared by one or more tenants. For example, the multi-tenant system 110 may run an application server that processes requests for several tenants simultaneously. However, the multi-tenant system enforces tenant-level data isolation to ensure that one tenant's jobs do not access other tenants' data.

[0022] Examples of cloud platforms include AWS (AMAZON WEB SERVICES), GOOGLE CLOUD PLATFORM, or MICROSOFT AZURE. The cloud platform 120 provides computing infrastructure services that can be used on-demand by the tenants 115 or by any computing system outside the cloud platform 120. Examples of computing infrastructure services provided by the cloud platform include servers, storage, databases, networking, security, load balancing, software, analytics, intelligence, and other infrastructure service capabilities. These infrastructure services can be used by the tenants 115 to build, deploy, and manage applications in a scalable and secure manner.

[0023] The multi-tenant system 110 may include a tenant data store that stores data for various tenants of the multi-tenant store. The tenant data store may store data for different tenants in separate physical structures, such as separate database tables or separate databases. Alternatively, the tenant data store may store data for multiple tenants in a shared structure. For example, user accounts for all tenants may share the same database table. However, the multi-tenant system stores additional information to logically separate data for different tenants.

[0024] Each component shown in FIG. 1 represents one or more computing devices, such as Microsoft TM Windows TM Compatible Operating Systems (OS): Apple TMThe computing devices may be conventional computer systems running OS X and / or Linux distributions. The computing devices may also be client devices having computing capabilities such as personal digital assistants (PDAs), mobile phones, video game systems, etc. Each computing device stores software modules that store instructions.

[0025] Interaction between the various components of system environment 100 is typically performed over a network, not shown in Figure 1. In one embodiment, the network uses standard communication technologies and / or protocols. In another embodiment, the entities may use custom and / or proprietary data communication technologies instead of or in addition to those described above.

[0026] Although the techniques disclosed herein are described in the context of a multi-tenant system, these techniques may be implemented using other systems that may not be multi-tenant systems. For example, an online system used by a single organization or enterprise may use the techniques disclosed herein to create one or more data centers on one or more cloud platforms 120.

[0027] System Architecture The multi-tenant system 110 includes a deployment module for deploying software artifacts on a cloud platform. The deployment module can perform various operations associated with a software release, such as provisioning resources on the cloud platform, deploying the software release, performing a rollback of the software artifact installed on a data center entity, etc. Figure 2 is a block diagram illustrating a system architecture of a deployment module 210, according to one embodiment. The deployment module 210 includes a data center creation module 220 and a software release management module 230. Other embodiments can have different and / or other components than those described herein, and functionality can be distributed between the components in a different manner.

[0028] The data center creation module 220 includes instructions for creating a data center on the cloud platform. The software release management module 230 includes instructions for deploying software releases or software artifacts for various services or applications running on the data center created by the data center creation module 220.

[0029] The data center generation module 220 receives a cloud platform independent declarative specification of a data center from a user, e.g., a tenant user. The cloud platform independent declarative specification of a data center specifies various entities of the data center. In one embodiment, the cloud platform independent declarative specification of a data center includes a hierarchical organization of data center entities, where each data center entity may include one or more services, one or more other data center entities, or a combination of both. FIG. 4 describes the various types of data center entities in further detail. The data center generation module 220 receives the platform independent declarative specification and a target cloud platform as input and generates a cloud platform specific metadata representation for the target cloud platform. The data center generation module 220 deploys the generated cloud platform specific metadata representation on the target cloud platform to create a data center on the target cloud platform according to the declarative specification.

[0030] Software release management module 230 receives as inputs (1) artifact version map 225 and (2) master pipeline 235. Artifact version map 225 identifies specific versions of software releases or deployment artifacts that are targeted for deployment on specific data center entities. Artifact version map 225 maps data center entities to software release versions that are targeted for deployment on the data center entities. Master pipeline 235 contains instructions for operations related to software releases on a data center, such as deploying a service, destroying a service, provisioning resources for a service, destroying resources for a service, etc.

[0031] The master pipeline 235 may include instructions for performing operations related to software releases for different environments, such as a development environment, a test environment, a canary environment, and a production environment, and instructions for determining when a software release is promoted from one environment to another. For example, if the deployment of a software release in a development environment executes a number of test cases that exceeds a threshold, the software release is promoted to a test environment for further testing, e.g., system-level and integration testing. If the software release in the test environment passes a threshold of test coverage, the software release is promoted to a canary environment where the software release is provided to a small subset of users on a trial basis. If the software release in the canary environment runs without errors for a threshold time, the software release is promoted to a production environment where the software release is provided to all users.

[0032] The software release management module 230 compiles the input artifact version map 225 and master pipeline 235 to generate a cloud platform specific detailed pipeline 255 that is sent to the target cloud platform. The cloud platform specific detailed pipeline 255 includes instructions for deploying the appropriate version of the software release or deployment artifact on the data center entity as specified in the artifact version map 225. The software release management module 230 can receive a modification to one of the inputs. For example, a user may modify the input artifact version map 225 and provide the same master pipeline 235. Thus, the same master pipeline is used, but a different software release is deployed on the data center entity. The software release management module 230 recompiles the input to generate a new cloud platform specific detailed pipeline 255 that deploys the version of the software release according to the new artifact version map 225.

[0033] The artifact version map may also be referred to as a deployment manifest, a version manifest, a software release map, or a software artifact version map. The master pipeline may also be referred to as a master deployment pipeline or a master orchestration pipeline.

[0034] FIG. 2B illustrates an overall process for deploying software artifacts to a data center, according to one embodiment. FIG. 2B illustrates a layout of a data center 265 including various data center entities. As shown in FIG. 2B, an artifact version map 225 identifies different versions of software targeted for release on different data center entities 275 of the data center 265. A master pipeline represents the flow of deployment artifacts through various environments of the data center. The software release management module 230 combines the information in the master pipeline 235 with the artifact version map 225 to determine a cloud platform specific detailed pipeline 255 that maps appropriate versions of software artifacts on the data center entities according to the artifact version map 225.

[0035] 3 is a block diagram illustrating the architecture of software release management module 230, according to one embodiment. Software release management module 230 includes an analysis module 310, a pipeline generator module 320, an artifact version map store 330, a pipeline store 340, a system configuration freeze module 350, a change processing module 355, and a pipeline execution engine 360. Other embodiments may include more, fewer, or different modules than those shown in FIG. 3 herein.

[0036] 3 may operate in a distributed manner on different systems, for example, the pipeline generator module 320 may execute on a computing system of a multi-tenant system, while the pipeline execution engine 360 ​​may execute on a computing system of a cloud platform in which data center entities and services are deployed.

[0037] The analysis module 310 analyzes various types of user inputs, including declarative specifications of data centers, artifact version maps 225, and master pipelines 235. The analysis module 310 generates data structures and metadata representations of the processed inputs and provides the generated data structures and metadata representations to other modules of the software release management module 230 for further processing.

[0038] The metadata store 340 stores various transformed metadata representations of the data center generated by the software release management module 230. The transformed metadata representations may be used to perform rollbacks to previous versions in case of encountering problems with the current version of the data center. The transformed metadata representations may also be used for validation, auditing, control, etc. at various stages of the transformation process.

[0039] The pipeline generator module 320 processes the master pipeline along with the artifact version map received as input to generate a detailed pipeline for a target cloud platform. The pipeline includes stages that include instructions for provisioning a service or deploying an application to deploy software release versions for various services on the cloud platform according to the artifact version map.

[0040] The artifact version map store 330 stores artifact version maps received from users, and the pipeline store 340 stores the master pipelines and the pipelines generated by the pipeline generator module 320 .

[0041] The system configuration freeze module 350 receives and processes requests to perform a system configuration freeze of services running on data center entities of a data center configured on the cloud platform. The received request specifies one or more services or data center entities configured on the cloud platform and a time interval during which a system configuration freeze is requested to be performed for the services or data center entities. Further details of the system configuration freeze module 350 are provided in connection with FIG. 11. Various processes performed by the system configuration freeze module 350 are described herein.

[0042] The change processing module 355 identifies requests for changes made to services installed in the data center and tracks information describing the changes. The change processing module 350 interacts with the change management system and records details of the execution of pipelines that implement service changes. The recorded details can be used for auditing purposes, for example, to determine why a particular change in the configuration of a service in the data center was made.

[0043] Pipeline execution engine 360 ​​executes the detailed pipeline generated by pipeline generator module 320. In one embodiment, pipeline execution engine 360 ​​is a system such as SPINNAKER that executes pipelines for software release / deployment. Pipeline execution engine 360 ​​parses the pipeline and executes each stage of the pipeline on a target cloud computing platform. Pipeline execution engine 360 ​​can run on one or more computing systems of the cloud platform.

[0044] Cloud platform-based data center creation 4 illustrates an example of a declarative specification of a data center, according to one embodiment. The declarative specification 410 includes multiple data center entities. A data center entity is an instance of a data center entity type, and multiple instances of each data center entity type can exist. Examples of data center entities include data centers, service groups, services, teams, environments, and schemas.

[0045] The declarative specification 410 includes definitions of various types of datacenter entities, including service groups, services, teams, environments, and schemas. The declarative specification includes one or more instances of a datacenter. Below are descriptions of various types of datacenter entities and examples thereof. These examples are illustrative and show some of the attributes of the datacenter entities. Other embodiments may include different attributes, and attributes with the same functionality may be given different names than shown herein. In one embodiment, the declarative specification is specified using hierarchical objects, e.g., Javascript object notation (JSON), that conform to a predefined schema.

[0046] A service group 520 represents a set of capabilities and features and services provided by one or more computing systems that can be independently constructed and delivered, according to one embodiment. A service group may also be referred to as a logical service group, a functional unit, or a bounded context. A service group 520 may also be considered as a set of services for a set of cohesive technical use case functionality provided by one or more computing systems. A service group 520 enforces security boundaries. A service group 520 defines a scope for modifications. Thus, any modifications to entities such as capabilities, features, or services provided by one or more computing systems in a service group 520 may be propagated to entities in the service group as needed or appropriate, but not to entities that exist outside the bounded definition of the service group 520. A data center may include multiple service groups 520. A service group definition specifies attributes including a name, a description, an identifier, a schema version, and a set of service instances. An example of a service group is a blockchain service group that includes a set of services used to provide blockchain functionality. Similarly, a security service group provides security features. A user interface service group provides functionality for a particular user interface feature. The Shared Documents service group provides the ability to share documents across users. There may be several other service groups as well.

[0047] Service groups support reusability of specifications so that tenants or users interested in developing a data center have a library of service groups they can easily use. Boundaries around the services in a service group are based on security and network concerns, among others. Service groups are associated with protocols for performing interactions with the service group. In one embodiment, a service group provides a collection of APIs (Application Programming Interfaces) and services that implement those APIs. Additionally, service groups are substrate agnostic. Service groups provide a blast radius scope for services within the service group, such that any failure of a service within the service group has impact limited to the services within the service group and minimal impact outside the service group.

[0048] The following is an example of a service group specification: The service group specifies various attributes that represent metadata of the service group and includes the set of services within the service group. There may be other types of metadata specified for a service group that are not shown here. [Table 1] JPEG0007673906000002.jpg234169

[0049] As shown in the above example, a service group can specify a set of clusters. A cluster represents a set of computing nodes, e.g., a set of servers, a set of virtual machines, or a set of containers (such as KUBERNETES containers). A physical server can run multiple containers, where each container has its own share of filesystem, CPU, memory, process space, etc.

[0050] A service group specifies a set of services. A service group can specify a cluster for a service, such that a data center deployed on a cloud platform runs a cluster of computing nodes and maps the service to the cluster based on the specified mapping, if included in the declarative specification. For example, in the service group example shown above, service instance serviceinstance0002 is specified to run on cluster instance cluster1.

[0051] A service group can specify security groups, each of which specifies a set of services that are allowed to interact with each other. Services outside the security group are required to pass additional authentication to communicate with services in the security group. Alternatively, services in the security group use one protocol to interact with each other, and services outside the security group use a different protocol that requires enhanced authentication to interact with services in the security group. Thus, a security group specifies policies that determine how services can interact with each other. A security policy can specify one or more environments to which the security policy is applicable. For example, a security policy policy1 can apply to a particular environment env1 (e.g., production environment), and another security policy policy2 can apply to another environment env2 (e.g., development environment). A security policy may be specified for a service group type or for a particular service type.

[0052] In one embodiment, a security policy specifies expressions for filtering service groups based on various attributes, such that the security policy is applicable to a filtered set of service groups. For example, a security policy may specify a list of IP (Internet Protocol) addresses that are whitelisted for a set of service groups identified by the filtered set, such that those computing systems are allowed access to the service group or a particular set of services within the service group.

[0053] In one embodiment, a security policy may specify a set of source services and a set of destination services for a service group. A source service for a particular service specifies the services outside of the security group that are allowed to connect with this particular service. A destination service for a particular service specifies the services outside of the security group that this particular service needs to connect with. During provisioning and deployment, the data center creation module generates instructions for the cloud platform that implement the particular network policy using the cloud platform's specific features and network capabilities, such that the network policy implements the security policy specified in the declarative specification.

[0054] A datacenter entity called a cell represents a set of services that interact with each other in a vertical fashion and can be scaled with further instances or copies of the cell, i.e. copies of the set of services. Creating multiple instances of a cell allows the system to scale a set of services that interact with each other. A datacenter instance can contain one or more cells. Each cell can contain one or more services. A datacenter can contain service groups or instances of cells.

[0055] A service definition specifies metadata about the type of service, e.g., database service, load balancer service, etc. The metadata describes various attributes of the service, including the name of the service, a description of the service, the location of documentation for the service, any subservices associated with the service, the owner of the service, the team associated with the service, build dependencies of the service, which specify other services this service depended on when built, start dependencies of the service, which specify other services that should be running when this particular service is started, authorized clients, DNS (Domain Name Server) names associated with the service, service status, support level of the service, etc. The service definition specifies a listen port attribute that specifies the ports on which the service can listen for different communication protocols, e.g., a service can listen on port p1 for UDP protocol and port p2 for TCP protocol. Other services in the data center can interact with the service through the ports specified by the service.

[0056] A service definition specifies a destination endpoint, e.g., an attribute outbound-access that specifies that the service requires access to a specified external URL (Uniform Resource Locator). During deployment, the data center generation module ensures that the cloud platform implements the access policy such that an instance of this service type is provided with the required access to the external URL.

[0057] An outbound access specification can identify one or more environment types for a service to which the outbound access is applicable. For example, outbound access for a set of endpoints S1 can apply to a particular environment env1 (e.g., a production environment), and outbound access for a set of endpoints S2 can apply to another environment env2 (e.g., a development environment).

[0058] Below is an example of a service definition. [Table 2] JPEG0007673906000004.jpg51170

[0059] The team definition 450 includes team member names and other attributes of the team, such as name, email, communication channel, etc. The following is an example of a team definition: A service can be associated with one or more teams responsible for modifications made to the service. Thus, any modifications made to the service are approved by the team. A service may be associated with a team responsible for maintaining the service after it is deployed to the cloud platform. A team may be associated with a service group and correspondingly, all services in that service group. For example, the team approves any changes to the service group, e.g., the services that are part of the service group. A team may be associated with a data center and therefore, all service groups in the data center. Team association specified at the data center level provides a default team for all service groups in the data center, which in turn provides a default team for all services in the service group.

[0060] According to one embodiment, a team association specified at the function level overrides a team association provided at the datacenter level. Similarly, a team association specified at the service level overrides defaults that may have been provided by team associations specified at the service group level or datacenter level. A team can determine how certain actions are taken for a datacenter entity associated with the team. The team association further determines the number of accounts on the cloud platform that are created for generating the final metadata representation of the datacenter by the compiler to the cloud platform and for provisioning and deploying the datacenter on the cloud platform. The datacenter creation module 210 creates one or more user accounts in the cloud platform and provides access to the user accounts to the team members. Thus, the team members are permitted to perform certain actions associated with the datacenter entity associated with the team, such as making or approving structural changes to the datacenter entity, including debugging and testing issues that may be identified for the datacenter entity, or maintenance of the datacenter entity as it is deployed.

[0061] Conventional techniques associate the same team with a datacenter throughout the design process, resulting in organizational structure having influence on the design of the datacenter or service group. Embodiments decouple team definition from the configuration that defines the datacenter entity, thereby reducing the influence of the team on the design and architecture of the datacenter entity. [Table 3]

[0062] Environment definition 460 specifies the type of system environment represented by the data center, for example, development environment, staging environment, test environment, or production environment. Schema definition 470 specifies a schema that specifies the syntax of certain data center entity definitions. Schema definition 470 is used to validate various data center entity definitions. The data center generation module determines the security policies of the data center in the cloud platform specific metadata representation based on the environment. For example, a particular set of security policies may be applicable to environment env1, and a different set of security policies may be applicable to environment env2. For example, a security policy may provide more restricted access in a production environment compared to a development environment. A security policy may specify the length of time that a security token is allowed to exist for a particular purpose. For example, a long access token (e.g., a week-long access token) may be allowed in a development environment, while an access token with a shorter life span (e.g., a few hours) may be used in a production environment. An access token may allow a user or service to access a particular cloud platform resource.

[0063] A datacenter definition 420 specifies attributes and components of a datacenter instance. A declarative specification can specify multiple datacenter instances. A datacenter definition 420 specifies attributes including the datacenter's name, description, type of environment, set of service groups, team, domain name servers, etc. A datacenter definition can specify a schema definition, and any metadata representation generated from the datacenter definition is validated against the specified schema definition. A datacenter contains a set of core services and capabilities that enable other services to function within the datacenter. An instance of a datacenter can be deployed within a particular cloud platform and associated with a particular environment type, e.g., development, testing, staging, production, etc.

[0064] The following is one definition of a datacenter instance: A datacenter instance definition includes a list of service groups contained in the datacenter instance, as well as other attributes including the datacenter's environment, a datacenter identifier, a name, a region representing a geographic region, one or more teams associated with the datacenter, and a schema version. [Table 4]

[0065] FIG. 5 illustrates some exemplary data centers created on a cloud platform based on a declarative specification, according to one embodiment. The data centers 510 may be created based on the declarative specification processed by the data center generation module 210. As shown in FIG. 5, multiple data centers may be configured in the cloud platform 120. Each data center 510 may correspond to a tenant 115 of the multi-tenant system 110. A tenant 115 may create one or more data centers 510. Alternatively, the data centers 510 may be created by any computing system. Each data center includes one or more service groups. For example, data center 510a includes service groups 520a and 520b, and data center 510b includes service group 520c. A data center may include multiple instances of a particular type of service group. Each service group includes a set of services. For example, service group 520a includes services 530a and 530b, service group 520b includes services 530a, 530b, and 530c, and service group 520c includes services 530e, 530f, and 530g. A service group may include multiple instances of services of the same service type.

[0066] The data center generation module 220 creates a data center on a cloud platform based on a declarative specification using the following steps: The data center generation module 210 receives a cloud platform independent declarative specification of a data center. The cloud platform independent declarative specification may be for a tenant of a multi-tenant system or for any other computing system, e.g., an online system. The cloud platform independent declarative specification is specified using a cloud platform infrastructure language. The cloud platform independent declarative specification of the data center is configured to generate a data center in any of a plurality of cloud platforms.

[0067] The data center creation module 210 receives information identifying a target cloud platform for creating a data center based on a cloud platform independent declarative specification. The target cloud platform can be any of a number of cloud platforms, for example, AWS, AZURE, GCP, etc. The data center creation module 210 further receives information for connecting to the target cloud platform, for example, credentials for creating a connection with the target cloud platform. The cloud platform may also be referred to as a cloud computing platform.

[0068] The data center generation module 210 compiles the cloud platform independent declarative specification to generate a cloud platform specific data center representation for creating a data center on a target cloud computing platform. For example, the cloud platform specific data center representation can point to user accounts, network addresses, etc. that are specific to the target cloud computing platform.

[0069] The data center creation module 210 transmits the platform-specific data center representation along with instructions for deploying the data center on the target cloud computing platform. The target cloud computing platform executes the instructions for configuring computing resources of the target cloud computing platform to generate a data center according to the platform-specific data center representation. The data center creation module 210 provides users with access to the computing resources of the data center configured by the cloud computing platform. For example, if a data center is created for a tenant of a multi-tenant system, users associated with the tenant are provided with access to the data center.

[0070] 6 is a block diagram illustrating the generation of a data center on a cloud platform based on a declarative specification, according to one embodiment. The data center generation module 210 receives as input a cloud platform independent declarative specification 610. The cloud platform independent declarative specification 610 may be a version of a declarative specification that has been incrementally modified by a user. The data center generation module 210 processes a particular version of the cloud platform independent declarative specification 610. Because the cloud platform independent declarative specification 610 is not specified for any particular target cloud platform, the data center generation module 210 can configure a data center on any target cloud platform based on the cloud platform independent declarative specification 610.

[0071] The data center generation module 210 processes the cloud platform agnostic declarative specification 610 to generate a cloud platform agnostic detailed metadata representation 620 for the data center. The cloud platform agnostic detailed metadata representation 620 defines details of each instance of the data center entities specified in the cloud platform agnostic declarative specification 610. The data center generation module 210 creates unique identifiers for the data center entity instances, e.g., service instances.

[0072] In one embodiment, the cloud platform agnostic detailed metadata representation 620 includes an array of instances of a data center entity type, e.g., an array of service group instances of a particular service group type. Each service group instance includes an array of service instances. A service instance may further include details of a team of users that are authorized to perform specific actions associated with the service instance. The team details are used during provisioning and deployment by the data center creation module 210, for example, to create user accounts for the service instance and enable members of the team to access the user accounts.

[0073] The cloud platform independent detailed metadata representation 620 includes attributes of each instance of the data center entity. Thus, the description of each instance of the data center entity is expanded to include all details. As a result, the cloud platform independent detailed metadata representation 620 of the data center can be significantly larger than the cloud platform independent declarative specification 610. For example, the cloud platform independent declarative specification 610 can be thousands of lines of specification, while the cloud platform independent detailed data center representation 620 can be millions of lines of generated code. As a result, the data center generation module 210 keeps the cloud platform independent detailed metadata representation 620 as immutable, i.e., once the representation is finalized, no modifications are performed on the representation. For example, if any updates, deletions, or additions of the data center entities need to be performed, they are performed on the cloud platform independent declarative specification 610.

[0074] The data center generation module 210 receives a target cloud platform where the data center is expected to be provisioned and deployed, and generates a cloud platform specific detailed metadata representation 630 of the data center. For example, the data center generation module 210 interacts with the target cloud platform to generate specific entities (or resources), such as user accounts, networking resources such as virtual private clouds (VPCs) and subnets on the VPCs, various connections between entities in the cloud platform, etc. The data center generation module 210 receives resource identifiers of resources to be created in the target cloud platform, such as user account names, VPC IDs, etc., and incorporates them into the cloud platform agnostic detailed metadata representation 620 to obtain a cloud platform specific metadata representation 630 of the data center. In one embodiment, the data center generation module 210 creates one unique user account on the cloud platform for each team for a given combination of service group and service. The user account is used by the team to perform interactions with that particular service of that service group, such as to debug, receive alerts, etc.

[0075] The target cloud platform may perform several steps to process the cloud platform specific detailed metadata representation 630. For example, the cloud platform independent declarative specification may specify allowed interactions between services. These allowed interactions are specified in the cloud platform specific detailed metadata representation 630 and implemented as network policies of the cloud platform. The cloud platform may further create security groups to implement network strategies to implement the data center according to the declarative specification.

[0076] The cloud platform independent declarative specification specifies dependencies between services, e.g., start dependencies for each service that list all services that should be running when a particular service is started. The data center generation module 220 generates a cloud platform specific detailed metadata representation of the data center that includes information describing these dependencies, such that the instructions for deploying the services ensure that the cloud platform starts the services in the order specified by the dependencies, such that, for each service, when this service is started, the services that need to be started before this service are running. Thus, the dependencies between services represent a dependency graph, and the cloud platform starts the execution of the services in an order determined based on the dependency graph, such that if service A depends on service B, service B is started before service A is started.

[0077] The data center generation module 220 creates trust relationships between user accounts that allow services to access other services over a secure communication channel. These trust relationships are generated using substrate-specific instructions based on the declarative specification, e.g., generated based on outbound access attributes specified for the services. The data center generation module 220 sends instructions to the cloud platform to create network policies based on cloud platform specific mechanisms that control interactions and access across service groups and services, as specified by configuration of the declarative specification, e.g., outbound access, security groups, security policies, etc.

[0078] The data center generation module 210 deploys the cloud platform specific metadata representation 630 on the particular target cloud platform for which the representation was generated. The data center generation module 210 can use the generated metadata representation to perform various validations, including policy validation, format validation, etc.

[0079] The cloud platform agnostic declarative specification 610 may be referred to as a declared data center representation, the cloud platform agnostic detailed metadata representation 620 may be referred to as a derived metadata representation of the data center, and the cloud platform specific metadata representation 630 may be referred to as a hydrated metadata representation of the data center.

[0080] Overall process for deployment of software artifacts in a data center 7 illustrates an overall process for generating a pipeline for deployment of software artifacts on a data center configured on a cloud platform, according to one embodiment. A data center generation module generates one or more data centers on a target cloud platform (710). Each data center is generated from a cloud platform-independent declarative specification and has a hierarchy of data center entities.

[0081] The software release management module 230 generates (720) a cloud platform agnostic master pipeline. In one embodiment, the cloud platform agnostic master pipeline includes stages corresponding to environments of a data center, e.g., development, test, canary, and production environments. The master pipeline configures a sequence of incremental and / or conditional deployments across various environments, such as development, test, staging, or production. The master pipeline may be triggered by delivery of an image of the software artifact and includes stages or instructions for deploying the build in an environment of type development. The built software artifact is conditionally promoted to one or more test environments, followed by one or more canary environments, before finally being deployed to the production environment. The master pipeline may be customized by a user, e.g., a service owner, to represent a particular orchestration across environments. The master pipeline may be customized to capture a particular promotion criteria for moving from one stage to the next. For example, different tenants of a multi-tenant system may customize the master pipeline in different ways. In one embodiment, the master pipeline by default uses the latest version of software for software artifacts for services, and builds and deploys this version across various environments. Users can use artifact version maps to ensure that specific versions of software artifacts are deployed on specific data center entities.

[0082] In one embodiment, each service deployed in a datacenter has a cloud platform independent master pipeline generated from datacenter entities as defined by the declarative specification of the datacenter, e.g., a master pipeline for the datacenter instance, a master pipeline for the service group, a master pipeline for the cell, a master pipeline for the service, etc. The master pipeline can be triggered by the delivery of an image of the software artifact. The master pipeline can implement continuous deployment controlled by the service owner. The master pipeline may implement on-demand deployment owned by the datacenter instance owner or owned by the release owner.

[0083] Certain portions of the master pipeline may be customized by users, e.g., by tenants of a multi-tenant system deploying services on a data center. For example, a promotion decision pipeline may be customized by a tenant to determine which test cases are executed and what the thresholds are. The software release management module 230 receives 730 customizations to the logic for promoting software artifacts from one stage to another of the cloud platform agnostic master pipeline.

[0084] The software release management module 230 compiles the cloud platform agnostic master pipeline to generate cloud platform specific detailed deployment pipelines that are specific to the hierarchy of data center entities for each data center as specified by the cloud platform agnostic declarative specification for the data center (740).

[0085] The software release management module 230 further receives (750) code for releasing one or more features of the service deployed on the data center. The software release management module 230 executes a cloud platform specific detailed deployment pipeline to deploy (760) software artifacts based on the received code.

[0086] FIG. 8 illustrates an exemplary master pipeline 800, according to one embodiment. The master pipeline represents a sequence of stages that represent progressive conditional deployment across various data center environments. FIG. 8 illustrates stages for different environments in a data center, including development, test, canary, and production environments. Each stage further represents the pipeline that is executed for that stage. Thus, the master pipeline 800 includes a development environment pipeline 810, which feeds into a test environment pipeline 820, which feeds into a canary environment pipeline 830, which feeds into a production environment pipeline 840.

[0087] The pipelines at each stage are hierarchical pipelines that contain lower level pipelines. For example, development environment pipeline 810 includes a development master pipeline that feeds data center pipelines D11, D12, ... depending on the number of data centers specified as having development environments in the data center declarative specification.

[0088] The test environment pipeline 820 includes a test master pipeline that feeds data center pipelines D21, D22, . . . according to the number of data centers specified as having test environments in the data center declarative specification.

[0089] The canary environment pipeline 820 includes a canary master pipeline that feeds data center pipelines D31, D32, ... according to the number of data centers specified as having a canary environment in the data center declarative specification.

[0090] The production environment pipeline 820 includes a production master pipeline that feeds data center pipelines D21, D22, ... according to the number of data centers specified as having test environments in the data center declarative specification.

[0091] Each environment pipeline 810, 820, 830 includes a promotion decision pipeline 815a, 815b, 815c, respectively. The output of the data center pipelines of the environment pipelines is collected by the promotion decision pipeline 815, which determines whether the software artifact is ready for promotion to the next stage. The promotion decision pipeline 815 can determine whether the software artifact for the service is promoted to the next stage based on the test case results obtained by the data center. For example, if more than a threshold number of test cases pass, the promotion decision pipeline 815 promotes the software artifact to the next stage. The last environment stage, e.g., the production environment pipeline, may not have a promotion decision pipeline because there is no subsequent stage to which the software artifact needs to be promoted. As shown in FIG. 8, the development environment pipeline promotion decision pipeline 815a determines whether to promote a software artifact from a development stage to a testing stage, the test environment pipeline promotion decision pipeline 815b determines whether to promote a software artifact from a testing stage to a canary stage, and the canary environment pipeline promotion decision pipeline 815c determines whether to promote a software artifact from the canary stage to a production stage.

[0092] The master pipeline includes multiple pipelines, such as a provisioning pipeline for provisioning resources of a target cloud platform and a deployment pipeline for deploying software artifacts on data center entities. Each pipeline includes a sequence of stages, and each stage represents one or more actions that need to be performed by the target cloud platform for data center provisioning and deployment. The data center generation module 210 generates detailed pipelines for deploying versions of software artifacts on data center entities.

[0093] In one embodiment, the pipeline generator module 320 generates detailed pipelines using pipeline templates that contain variables. The pipeline templates are converted into pipelines by providing specific values ​​for the variables in the pipeline. The process of generating a pipeline from a template is called hydration of the pipeline template. The pipeline templates contain template expressions that are used as placeholders for the actual values ​​used in deployment. For example, the template expressions may be replaced by target-specific parameter values ​​or expressions. Multiple pipeline instances can be generated by hydrating the pipeline template for different targets. Template variables represent parameters that can be replaced with specific values ​​for a given target to generate a pipeline instance specific to that target. For example, the template variable "account_id" may be replaced with the actual value of account_id, e.g., "12345", during hydration.

[0094] In one embodiment, the pipeline generator module 320 generates pipelines in a hierarchical manner based on a hierarchy of data center entities for a data center. For example, a data center includes different types of data center entities including data centers, service groups, services, etc. A data center entity may include one or more child data center entities. For example, a data center includes one or more service groups as child data center entities. A service group includes one or more services as child data center entities. Thus, the data center generation module 210 starts with a data center entity at one level of the hierarchy and generates pipelines for the data center entities below that level. For example, the pipeline generator module 320 starts at the data center level and generates pipelines for the service groups in the data center. For each service group, the pipeline generator module 320 generates pipelines for the services in the service group.

[0095] The process of executing a pipeline according to one embodiment is as follows: The software release deployment module 230 receives a request to deploy a software artifact on a set of data center entities in a target cloud platform. The software release deployment module 230 executes a master pipeline for one or more data centers. The software release deployment module 230 executes an aggregation pipeline for each service group in each data center. The aggregation pipeline includes pipelines for the services in the service groups. For each service in each service group, the pipeline is executed by executing all stages of the pipeline. Execution of the provisioning pipeline results in the provisioning of resources for the service, and the deployment pipeline results in the deployment of the service in the target cloud platform.

[0096] Figure 9 illustrates the overall process performed by stages for an environment of a master pipeline on a cloud platform, according to one embodiment. Steps 910, 920, 930, 940, and 950 may be performed by each environment pipeline 810, 820, 830. The production environment pipeline 3 may only perform steps 910 and 920. The steps illustrated in Figure 9 may be performed for one service or for multiple services specified using a manifest file.

[0097] The environment pipeline for environment E includes instructions for deploying 910 the software to a set of datacenter entities, e.g., a set of datacenter entities designated as having environment E. In one embodiment, the software artifacts are generated by compiling source code for the services. The source code may be obtained from version control software. The set of datacenter entities may include datacenter instances, service groups, cells, services, or any combination thereof.

[0098] The environment pipeline for environment E further includes instructions for executing (920) tests to test the software artifacts deployed on the set of data center entities. The environment pipeline for environment E further includes instructions for evaluating (930) the test results against the promotion criteria, e.g., using the promotion decision pipeline 815. If the promotion criteria are not met, steps 910, 920, 930, and 940 may be repeated using a revised software artifact, e.g., a software artifact generated from source code that includes fixes for specific defects identified during testing (920). The environment pipeline for environment E further includes instructions for proceeding to the next stage (950) if the promotion criteria are met.

[0099] In one embodiment, the master pipeline includes a hierarchy of pipelines. The hierarchy includes multiple levels, and pipelines at a particular level include the pipelines at the next lower level as child pipelines. For example, at the highest level of the hierarchy, the master pipeline includes a release master pipeline that deploys a set of services related to a product. The next level of the hierarchy includes a service master pipeline that represents all deployments of a particular service across various environments. The next level of the hierarchy can include a service group master pipeline followed by a service master pipeline.

[0100] FIG. 10 illustrates an exemplary master pipeline, according to one embodiment. The master pipeline is a hierarchical pipeline, in which each stage of the pipeline can include a pipeline with detailed instructions for executing the stage. The master pipeline hierarchy can mirror the data center hierarchy. For example, the top level of the master pipeline represents a sequence of stages for different environments. Each environment can include one or more pipelines for a data center instance or other type of data center entity. The data center instance pipeline 1010 can include a service group pipeline 1020. Each service group pipeline 1020 can include one or more service pipelines 1030. The data center instance pipeline 1010 can include a cell pipeline 1025, each cell pipeline 1025 includes one or more service pipelines 1030. The service pipelines 1030 can include stages, each stage representing a pipeline that represents instructions for deploying a service for a particular environment. The lowest level pipeline in the hierarchy, or leaf level pipeline, is called a unit pipeline and can include detailed service-specific instructions for performing operations related to the service. For example, the deployment of a service may include a pre-deployment step, a deployment step, a post-deployment step, and a post-deployment test and validation step.A pipeline that is not a leaf-level pipeline and has one or more child pipelines is an aggregate pipeline that orchestrates the execution of the child pipelines.

[0101] The master pipeline may be driven by a pull request that causes the software version control system to receive a request to consider changes committed to an external repository for inclusion in the project's main repository. Thus, the master pipeline is automatically triggered when a pull request is received to deploy the software artifact based on the latest software version for which the pull request was received. The master pipeline performs continuous delivery of the software artifact based on the pull request. The master pipeline may be driven on an on-demand basis, for example, by invoking a request using an application programming interface (API) of the deployment module 210. On-demand deployment based on the master pipeline may be requested for any set of services specified using the API and for any version of a given service. The master pipeline may be invoked to request a rollback from a current version to a previous version or a rollforward from a currently deployed version to a more recent version.

[0102] In one embodiment, the deployment module 210 creates a service master pipeline for each service. These pipelines are triggered when a pull request is received for the software repository. The deployment module 210 receives pipeline templates from a user for a particular service. These pipeline templates include detailed instructions for testing, validating, building, etc. for a particular service. The data center creation module 220 receives a cloud platform independent declarative specification for one or more data centers. The data center creation module 220 creates (or configures) the data centers according to the received cloud platform independent declarative specification. The deployment module 210 receives a promotion decision 815 pipeline. The promotion decision 815 pipeline is integrated into an overall master pipeline.

[0103] The pipeline generator creates all the pipelines for each datacenter from the templates and combines them through the master pipeline in a hierarchical manner, for example as shown in Figure 10. In one embodiment, the pipeline generator generates service pipelines for individual services, the pipeline generator generates cell master pipelines to call the service pipelines, the pipeline generator generates service group master pipelines to call the cell master pipelines, the pipeline generator generates datacenter instance master pipelines to call the service group pipelines, and the pipeline generator generates service master pipelines to call the datacenter instance master pipelines.

[0104] Below is a snippet of the master pipeline showing the various stages. Each stage can specify attributes including the stage name, the type of pipeline, the stage type (e.g. master deploy pipeline or promote pipeline), the previous stage, etc. [Table 5] JPEG0007673906000008.jpg136170

[0105] As shown in the inspector master pipeline, the first stage is the artifact version map. The next stage is the master deployment pipeline for deployment to the development environment. The next stage is the promote pipeline to determine if the software artifact can be promoted to the next stage. The next stage is the master deployment pipeline for deployment to the testing environment. The next stage is the promote pipeline to determine if the software artifact can be promoted to the next stage which is the staging environment.

[0106] Software Artifact Version Map In one embodiment, the deployment module 210 receives an artifact version map that associates various software artifacts and their versions with data center entities. The artifact version map provides a declarative specification of the specific versions of software artifacts that need to be deployed for a service at different data center entities. Each data center entity may be uniquely identified based on its location in the data center hierarchy specified by the declarative specification of the data center. For example, for a service, a software library may serve as a software artifact. A software artifact may have multiple versions, e.g., V1, V2, V3, etc. The artifact version map may specify that version V1 needs to be deployed to data center entities C1 and C2, and version V2 needs to be deployed to data center entities C3 and C4. The deployment module 210 generates master pipelines and instructions that ensure that the appropriate software artifact versions are deployed to the data center entities as specified in the artifact version map.

[0107] In one embodiment, the artifact version map is specified as a Javascript object notation (JSON) file, a YAML file, or a file using any other syntax for representing nested objects. The artifact version map is associated with various data center entities distributed across a hierarchy of data centers. <service> : <version>It can contain a set of key pairs. The artifact version map key pair acts as a whitelist for the corresponding pipelines. If a key for a service is not included in the artifact version map, all pipelines for that service will be excluded during the pipeline execution. Different artifact version maps may be applied to the same master pipeline, resulting in different services being included / excluded during the master pipeline execution.

[0108] Below is an example artifact version map. The artifact version map specifies the environment type using the attribute "env_types". In the example below, an environment type development is specified. An environment type can include one or more data center instances, which can include one or more service groups, which can include one or more services. In the example below, the software artifact name is specified as library1 and the version is specified as version1 and is associated with the service instance instance001. However, the software artifact name and version may be associated with any level of the data center entity in the hierarchy. For example, when a software artifact name and version is specified or of a service group, the software artifact name and version are applicable to all services in the service group unless the software artifact name and version are overridden with a different value of the software artifact name and version specified for a particular service instance in the service group. Similarly, a software artifact name and version can be specified for a data center instance and is applicable to all service groups or cells in the data center instance unless an override value is specified for the service group. [Table 6] JPEG0007673906000010.jpg54169

[0109] In one embodiment, the artifact version map specifies a datacenter entity using a full path of the datacenter entity, e.g., "stagger_group1 / datacenter1 / service_group2 / service1". In one embodiment, the artifact version map specifies a set of datacenter entities using regular expressions within the full path of the datacenter entity. For example, a full path that includes service_group[?] includes service_group1, service_group2, service_group3, etc.

[0110] Below is an example of an artifact version map that specifies a regular expression to define a set of services. The environment types are specified as dev and test, the datacenter entity in the full path including datacenter instance and service group is specified as a wildcard, and the service instance is specified as "service*". Therefore, for all datacenter instances for dev and test environments, for all service groups, for service names matching service*, version V1 of application app1 will be deployed. [Table 7]

[0111] In some embodiments, an artifact version map can specify parameters that are used by the pipeline such that the specified parameters are applicable to the stagger group for which the parameters are specified.

[0112] The following process is used for deployment of software artifacts on a data center configured on a cloud platform, according to one embodiment: A data center generation module generates one or more data centers on a target cloud platform, each data center being generated from a cloud platform independent declarative specification and having a hierarchy of data center entities.

[0113] The software release management module 230 receives as input an artifact version map that maps data center entities to versions of software artifacts. The software release management module 230 further receives as input a cloud platform agnostic master pipeline.

[0114] The software release management module 230 compiles the cloud platform independent master pipeline in conjunction with the artifact version map to generate a cloud platform specific detailed pipeline. In one embodiment, the generated cloud platform specific detailed pipeline includes an artifact version map filter before a particular stage to determine whether the particular stage should be enabled or disabled according to the artifact version map.

[0115] The software release management module 230 further receives code for releasing one or more features of the service deployed on the data center. For example, the code may represent source code obtained from a version control management system that stores a source code repository to which changes are submitted by developers. The software release management module 230 executes a cloud platform-specific deployment pipeline to deploy software artifacts based on the received code.

[0116] The artifact version map and master pipeline can be used to orchestrate various types of operations related to continuous delivery of software artifacts in a cloud-based data center. The artifact version map and master pipeline can be configured to perform aggregate retry operations for a service or service group or any data center entity. The artifact version map contains the configuration of retry operations for the data center entity, including the retry strategy, the threshold number of retries to perform in case of failure to execute a stage of the pipeline, whether confirmation from a user is required before retrying or the retry is performed automatically, etc. For example, the retry strategy may be a fixed backoff strategy that pauses execution for a fixed period of time before retrying. Other retry strategies may be configured using the artifact version map and master pipeline. In one embodiment, the pipeline generator introduces an invoke retrier stage in the aggregate pipeline to trigger the retry strategy if a previous pipeline stage fails. The retry strategy and configuration parameters specified for a data center entity apply to all data center entities and services within the data center entity unless the values ​​are overridden for nested data center entities.

[0117] System Configuration Freeze for Services In one embodiment, the system configuration freeze module 350 performs a system configuration freeze for services deployed on the target cloud platform. The system configuration freeze may be performed in response to a request received from a user, e.g., a system administrator. The request may identify a datacenter entity and request a system configuration freeze for all services running on the datacenter entity. The system configuration freeze ensures that no system configuration changes are made to the services of the datacenter entity and no modifications are made to software artifacts associated with the services running on the datacenter entity.

[0118] In one embodiment, the system receives a request to set a system configuration freeze for a datacenter entity, e.g., a service group, a cell, a datacenter, etc. The system determines the services that are within the datacenter entity, e.g., based on a declarative specification of the datacenter or based on metadata describing the topology hierarchy of the datacenter. For example, the system performs a hierarchical traversal of all datacenter entities below this datacenter entity to identify all services within the datacenter entity. The system performs a system configuration freeze for all services identified within the datacenter entity D1, unless the system receives a request to create an exception (i.e., an override) for service S1 or for a smaller datacenter entity D2 such that the system does not perform a system configuration freeze for a particular service S1 or for a smaller datacenter entity D2 within the identified datacenter entity D1.

[0119] A system configuration freeze may be performed in response to determining that a workload greater than a threshold amount is expected during a time interval. For example, if a tenant of a multi-tenant system expects to process a large number of requests during a time interval, such as a holiday, the tenant may request that services running on the production system be frozen during the time interval to avoid potential interruptions in service.

[0120] In one embodiment, the system can receive a modified declarative specification along with a request to perform a system configuration freeze on a data center entity. For example, a data center may be modified to add a data center entity D2 in the hierarchy of data center entity D1 or to add a service s1 in the hierarchy of data center entity D1. The system creates the requested data center entity D2 or service S1 and enforces a system configuration freeze on the created service or data center entity for a specified time interval.

[0121] A system configuration freeze may be performed in response to determining that a problem associated with a datacenter entity has been diagnosed during a time interval. For example, a tenant of a multi-tenant system may identify a problem associated with a datacenter entity or service, such as a performance problem that causes the service to perform poorly. A system administrator for the tenant may execute diagnostic processes and tools to identify the root cause of the problem. While the problem is being diagnosed, the tenant may prefer to freeze the system configuration of the datacenter entity or service to minimize changes to the system so that the issue can be accurately diagnosed.

[0122] A system configuration freeze may be performed in response to determining that a particular system configuration change is being performed during a time interval. The system configuration freeze prevents any other system configuration changes during this time interval. Thus, the system can ensure that changes to the system configuration that may interfere with each other are performed in a mutually exclusive manner, such that only one change is being performed at a time and other changes are blocked for that period of time.

[0123] Although the examples described herein show processes in the context of a multi-tenant system, the disclosed techniques may be applied to any other system, for example, a system dedicated to a single organization.

[0124] 11 illustrates a system architecture of a system configuration freeze module, according to one embodiment. The system configuration freeze module 350 includes a freeze request processing module 1110, a lock manager 1120, and a system configuration freeze metadata store 1130. Other embodiments may include more or fewer components than those illustrated herein in FIG.

[0125] The freeze request processing module 1110 receives system configuration freeze requests and processes them. The system configuration freeze request may be received from a client device of a user, for example, a system administrator. The system configuration freeze request specifies a time interval during which the system configuration freeze needs to be forced. The system configuration freeze request may identify one or more services that need to be frozen for the specified time interval. The system configuration freeze request may specify a data center entity of the data center for performing the system configuration freeze. Thus, the request specifies that all services running in the data center entity need to be frozen for the specified time interval, such that no modifications are performed on software artifacts associated with these services, and no modifications are performed on any type of configuration of these services. The freeze request processing module 1110 identifies a list of services that need to be frozen for the specified time interval. For example, if the system configuration freeze request specifies a data center entity, the freeze request processing module 1110 identifies all services that are configured to run in this data center entity.

[0126] The system configuration freeze metadata store 1130 stores metadata related to performing a service configuration freeze, including metadata describing a request for a system configuration freeze. The metadata describing the request includes a request identifier, a period for which the system freeze request needs to be enforced, and the datacenter entity for which the system configuration freeze needs to be performed. The time interval can be specified using a start time and an end time, or by using a start time and a length of the time interval. The system configuration freeze metadata store 1130 further stores a mapping from a service to a lock that can be acquired either by the system configuration freeze module 350 to enforce a freeze on the service, or by a pipeline to make modifications to the service configuration. The metadata describing the lock includes an estimate of the length of the period for which the lock is requested.

[0127] The lock manager 1120 receives a request to acquire a lock and acquires the requested lock. The lock manager receives a request to acquire a lock from the freeze request processing module 1110 or from the pipeline execution module. In one embodiment, the lock manager is a distributed lock service that may run on a system distinct from the computing system that executes the freeze request processing module 1110. The lock manager 1120 further receives a request to release a previously acquired lock and releases the lock. Acquiring a lock associated with a service for a particular period of time ensures that pipelines configured to modify the service cannot proceed during that period. Releasing the lock after that time interval allows any pipelines attempting to modify the service's configuration to proceed, thereby ending the system configuration freeze for the service.

[0128] The pipeline generator module 320 generates a pipeline for making changes to the system configuration of the data center entities. The generated pipeline is configured to allow the system configuration freeze module 350 to freeze the configuration of any service.

[0129] FIG. 12 illustrates an example pipeline for forcing a system configuration freeze, according to one embodiment. FIG. 12 illustrates some of the stages of a pipeline generated to modify a service configuration for a service, e.g., to modify software artifacts associated with the service. The generated pipeline may include other stages not shown in FIG. 12. The pipeline illustrated in FIG. 12 may be a portion of the service pipeline 1030 illustrated in FIG. 10.

[0130] The generated pipeline 1200 thus includes a pre-change stage 1210, a change stage 1220, and a post-change stage 1230. The change stage 1220 includes various types of system configuration changes that may be implemented by the software release management module 230. These system configuration changes include, but are not limited to, deploying a new service in the data center entity, destroying a service in the data center entity, provisioning a resource in the data center entity, destroying a resource in the data center entity, performing any utility operation, or performing a rollback of a service deployment by reverting to a previous version of the service or software artifact. The system configuration freeze module 350 is configured to freeze all these types of changes to the data center entity for a certain time interval.

[0131] The pre-change stage 1210 includes instructions to acquire locks associated with the services. The post-change stage 1230 includes instructions to release locks acquired in the pre-change stage 1210. In one embodiment, information identifying locks associated with each service is stored in the system configuration freeze metadata store 1130. Thus, the pipeline generator module 320 can access locks associated with the pipeline, generate instructions to acquire the locks, and include the generated instructions in the pre-change pipeline. Alternatively, the pipeline generator module 320 may access the system configuration freeze metadata store 1130 to acquire identifiers of locks associated with the services, and then generate instructions to acquire the locks. The instructions to acquire locks in the pre-change stage 1210 halt the pipeline execution if the lock cannot be acquired, for example, if the lock has been previously acquired by another entity and has not yet been released. For example, the system configuration freeze module 350 acquires locks for all services that need to be frozen for a certain time interval, thereby halting the execution of any pipeline that attempts to change the system configuration of the services during this time interval. In one embodiment, the instructions in the pre-alteration stage 1210 cause execution of the pipeline 1200 to fail if a lock associated with a service cannot be acquired. In one embodiment, the instructions in the pre-alteration stage 1210 cause execution of the pipeline 1200 to fail if a lock associated with a service cannot be acquired within a threshold period of time. Thus, execution of the pipeline times out and fails after a period of time. If execution of the pipeline fails, subsequent stages, such as the alteration stage 1220 and the post-alteration stage 1230, are not executed.

[0132] Handling a System Configuration Freeze FIG. 13 illustrates a process for making changes to the system configuration of a service deployed on a data center in a cloud platform, according to one embodiment. The data center generation module 220 generates 1310 one or more data centers based on a cloud platform-independent declarative specification, for example, as shown in the process illustrated in FIG. 6 and FIG. 7. Each data center has a set of services deployed within the data center. The software release management module 230 receives 1320 an artifact version map that includes information describing any changes that need to be made to the services installed and running in the data center. For example, the changes can include adding a new service to the data center, removing a service currently deployed in the data center, changing the configuration of a service, deploying a new version of a software artifact for the service, etc.

[0133] The software release management module 230 generates (1330) a cloud platform independent master pipeline that includes a change management stage, such as the master pipeline shown in Figure 12. The software release management module 230 compiles (1340) the cloud platform independent master pipeline to generate a cloud platform specific detailed pipeline that includes instructions for deploying services according to the artifact version map on data centers deployed on the target cloud platform.

[0134] The software release management module 230 receives instructions to modify a system configuration of a service deployed on a data center entity (1350). The software release management module 230 executes a cloud platform specific detailed pipeline to make appropriate modifications to the system configuration of the service according to the artifact version map (1360). The cloud platform specific detailed pipeline includes a change stage including instructions to make changes associated with one or more software artifacts for the data center entity, a pre-change stage including instructions to acquire a lock, and a post-change stage to release the lock after the changes are made. The pipeline execution is interrupted if the pre-change stage fails to acquire the lock.

[0135] 14 illustrates an overall process for performing a system configuration freeze of a data center entity of a data center configured on a cloud platform, according to one embodiment. The system configuration freeze module 350 receives 1410 a request to perform a system configuration freeze for the data center entity over a time interval. The system configuration freeze module 350 identifies all services configured for execution within the data center entity. The system configuration freeze module 350 may identify the services based on a declarative specification of the data center to which the data center entity belongs. The request to perform a system configuration freeze specifies the time interval for which the system configuration freeze is requested to be enforced.

[0136] The system configuration freeze module 350 acquires (1430) locks associated with the services of the data center entity for the time interval. In one embodiment, the system configuration freeze module 350 accesses the system configuration freeze metadata store 1130 to determine an identifier of a lock corresponding to each identified service (1420) and acquires the lock corresponding to the lock identifier when the system configuration freeze time interval begins.

[0137] After the start of the time interval and before the end of the time interval, the software release management module 230 receives (1440) a request to modify the system configuration of a service running on the data center during the time interval. The software release management module 230 executes (1450) a pipeline for deploying software artifacts associated with the service running on the data center entity. The execution of the pipeline results in the execution of a pre-change stage of the pipeline, thereby pausing the pipeline execution until a lock is acquired. After the time interval is completed, the system configuration freeze module 350 releases (1460) the lock acquired for the service running on the data center entity. As a result, the pre-change stage of the pipeline can acquire the lock and complete the pipeline execution (1470), thereby allowing the requested changes to the service to be performed after the time interval. According to some embodiments, the post-change stage releases the lock acquired for the service after the change stage is completed and the requested modifications to the system configuration are made. This process ensures that any pipeline attempting to modify a service running on the data center entity is prevented from making the modifications during the time interval.

[0138] According to some embodiments, the pre-alteration stage includes instructions to determine whether the pre-alteration stage of the pipeline fails to acquire the lock for a period of time that exceeds a threshold. If the pre-alteration stage of the pipeline fails to acquire the lock for a period of time that exceeds the threshold, the execution of the pipeline has failed. Thus, the user is provided with an indication that the pipeline execution has failed and that the user needs to submit subsequent instructions to make system configuration changes to the service.

[0139] In one embodiment, when a system configuration freeze of a data center entity is requested, if a pipeline that is modifying the system configuration of a service of the data center entity is currently running, the system postpones the system configuration freeze, thus delaying the interval at which the system configuration freeze occurs until the current pipeline execution is completed. However, the system prevents any new pipeline execution that may change the system configuration for the service of the data center entity from beginning until the system configuration freeze is completed. This occurs because the currently running pipeline has a lock associated with the service, which causes the system configuration freeze request to be pending when the system configuration freeze module 350 attempts to acquire the lock for the service of the data center entity. A system administrator can manually abort the currently running pipeline to allow the system configuration freeze to proceed.

[0140] In one embodiment, if a failure occurs during execution of a pipeline when a system configuration freeze is forced on a data center entity, the pipeline execution may cause locks to be acquired but not released. Lock manager 1120 performs a lock garbage collection process that checks for any locks that remain unreleased after the system configuration freeze time interval. If lock manager 1120 identifies locks associated with services of the data center entity after the time interval, lock manager 1120 releases those locks.

[0141] In some embodiments, locks are associated with a duration that represents an estimate of the length of time the lock is expected to be acquired. The system can use the duration of the lock as an estimate of how long the pipeline execution will take. The system can prioritize certain pipeline executions based on their expected duration, e.g., shorter pipelines may be prioritized above pipelines that take significantly longer. However, if a pipeline execution takes longer than the duration specified for the lock, the system simply records the discrepancy. The system maintains a list of services whose specified lock duration is significantly shorter than the actual duration the lock was acquired. The system prioritizes these pipelines from other pipelines that have the exact duration specified for the lock. In one embodiment, the system automatically determines the lock duration based on the pipeline's past execution time.

[0142] In some embodiments, the system configuration freeze is associated with a priority. The service modification requests are further associated with a priority scale. If during a system configuration freeze interval, the software release management module 230 receives a service modification request such that the priority of the service modification request is higher than the priority of the system configuration freeze, the data center entity's service modification request is allowed to proceed during that time interval despite the system configuration freeze. The system provides this mechanism, so that if there are some urgent fixes required for issues that may occur during a system configuration freeze, such as a moratorium, the fixes are allowed to proceed.

[0143] Change Processing Module System Architecture 15 illustrates a system architecture of a change processing module 355 according to one embodiment. The change processing module 355 includes a change decision module 1510, a change management client 1520, an event queue manager 1530, an event listener, and an event queue store 1550. Other embodiments may include more or fewer components than those illustrated herein in FIG.

[0144] The change determination module 1510 identifies changes to be performed on services installed on one or more data centers. In one embodiment, the change determination module 1510 receives a set of instructions that identify changes to be performed compared to a current configuration of services on the data centers. For example, the changes may specify new services to be added, existing services to be removed, service configurations that need to be modified, etc.

[0145] In one embodiment, the change determination module 1510 stores various versions of the artifact version map. The software release management module 230 stores a version V1 of the artifact version map that corresponds to the current configuration of the data center on the target cloud computer. The software release management module 230 receives a version V2 of the artifact version map that needs to be implemented on the target cloud computer. The version V2 of the artifact version map has differences compared to the version V1. The change determination module 1510 compares the two versions of the artifact version map to determine the differences between them. Thus, the change determination module 1510 compares the new version of the received artifact version map with the previous version of the currently deployed artifact version map to identify the changes that are requested in the new version of the artifact version map. The changes can include new services that need to be installed on a particular data center entity, versions of existing services that need to be changed (e.g., upgraded) on a particular data center entity, services that need to be removed from a particular data center entity, etc. The pipeline generator generates a master pipeline to implement the requested changes to the service configuration.

[0146] The change management client 1520 interacts with a change management system to store information describing changes to be made. A change management system may be any system that stores records describing changes and manages change-related tasks such as sending alerts or messages related to the changes, receiving approvals for particular changes, etc. The change management client 1520 interacts with the change management system by calling the change management system's API. The change management system may also be an external system that provides change management as a service.

[0147] The event queue manager 1530 creates and manages queues that store events related to deployment tasks. The event queue manager 1530 can create queues for various teams that need to be notified of specific events. For example, a team may be responsible for a data center entity defined in the declarative specification of the data center. The change processing module identifies specific events, such as failure or success events for services installed in the data center entity, and notifies the team. The event listener 1540 listens for events in the pipeline execution engine 360. The events describe detailed actions performed for each pipeline, such as whether a particular stage executed successfully or failed to execute, or whether a particular stage generated a message. The event listener 1540 categorizes the events and associates them with specific data center entities. Thus, the event listener 1540 provides information describing the event to the event queue manager 1530 for storage in the event queue. The event queue store 1550 stores different queues, for example queues for one or more teams defined in the declarative specification of the data center.

[0148] FIG. 16 illustrates an exemplary master pipeline for managing changes, according to one embodiment. The master pipeline includes stages similar to those illustrated in FIG. 8 corresponding to different system environments where software artifacts are promoted for deployment to a data center. Thus, the master pipeline 1600 includes stages for different environments of a data center including a development environment, a test environment, a canary environment, and a production environment. The master pipeline further includes a change management stage. Thus, the master pipeline 1600 includes a development environment stage 810, which feeds a test environment stage 820, which feeds a change management stage 1610, which feeds a canary environment pipeline 830, which feeds a production environment pipeline 840, which feeds a change closure stage 1620. Each stage represents a pipeline that is executed for that stage.

[0149] The master pipeline represents a set of pipelines that form a hierarchy, e.g., a pipeline that corresponds to each data center entity in the hierarchy of data center entities formed by the data center as specified by the declarative specification used to generate the data center. The change management stage of each pipeline performs actions related to managing the changes represented by the master pipeline. For example, the change management stage includes instructions for performing interactions with a change management system.

[0150] In one embodiment, the change management stage includes instructions for creating a change case in the change management system that represents a set of changes to be performed by the master pipeline. A change case in the change management system represents information that describes a set of changes to be performed on services in a data center. The set of changes may be stored as one or more records in a database of the change management system.

[0151] In one embodiment, a configuration file, e.g., an artifact version map, includes information describing an existing change case to be used to store information describing a set of changes to be performed on services in a data center. For example, a user, such as a system administrator, can create a change case and provide it in a configuration file for use by the change management stage.

[0152] The change management stages of the various pipelines represented by the master pipeline provide the status of the deployment of the various services to the change management system for storage in association with the change case. For example, a pipeline for a particular service provides the status of the deployment of that service to the change management system. The status can indicate whether the deployment was successful, whether the deployment failed, whether the deployment generated errors or warnings, etc.

[0153] The change closure stage closes the change case if all services specified by the artifact version map are successfully deployed. Closing a change case prevents further modifications from being made to the change case, e.g., no further information can be stored in the change case. However, a user can review the information stored in the change case, e.g., to audit specific changes made to services.

[0154] Change Management Process FIG. 17 illustrates an overall process of change management for services deployed on a datacenter configured on a cloud platform, according to one embodiment. The datacenter generation module 220 generates 1710 one or more datacenters based on a cloud platform-independent declarative specification, for example, as shown in the process illustrated in FIG. 6 and FIG. 7. Each datacenter has a set of services deployed within the datacenter. The software release management module 230 receives 1720 an artifact version map that includes information describing any changes that need to be made to the services installed and running in the datacenter. For example, the changes can include adding a new service to the datacenter, removing a service currently deployed in the datacenter, changing the configuration of a service, deploying a new version of a software artifact for the service, etc.

[0155] The software release management module 230 generates 1730 a cloud platform independent master pipeline that includes a change management stage, for example, the master pipeline shown in FIG. 16. The cloud platform independent master pipeline may further include a change close stage at the end of the pipeline. The software release management module 230 compiles 1740 the cloud platform independent master pipeline to generate a cloud platform specific detailed pipeline that includes instructions for deploying the services according to the artifact version map on the data center deployed on the target cloud platform.

[0156] The software release management module 230 receives (1750) source code for building software artifacts and compiling them for deployment on the target cloud platform. The software release management module 230 executes (1760) cloud platform specific detailed pipelines to deploy the appropriate versions of the software artifacts according to the artifact version map. The execution of the cloud platform specific detailed pipelines results in the execution of a change management stage for each service being deployed according to the artifact version map. The execution of the change management stage of the master pipeline can result in the creation of a change case in the change management system. The execution of the change management stage of the pipeline for the individual service sends a status of the execution of the pipeline for storage in association with the change case in the change management system. The execution of the change management stage of the pipeline for the individual service can cause the pipeline to wait for approval before promoting the software artifacts processed by the pipeline to a subsequent stage, for example, from a test stage to a canary stage or from a canary stage to a production stage. The execution of the change closure stage of the master pipeline closes the change case such that further modifications cannot be recorded to the change case.

[0157] 18 illustrates a process performed by the change management stage of a master pipeline, according to one embodiment. The software release management module 230 creates 1810 a change case in the change management system. The cloud platform agnostic master pipeline includes a set of pipelines including at least a pipeline for each service being deployed or modified in the data center. Each pipeline from the set includes a change management stage.

[0158] The software release management module 230 repeats steps 1820, 1830, and 1840 for each pipeline associated with a service in the data center. The software release management module 230 executes (1820) the pipeline associated with the service. The software release management module 230 sends (1830) a status of the execution of the pipeline to the change management system. The software release management module 230 receives approval to proceed to the next stage. Approval may be received from a user or team associated with the service. If the software release management module 230 receives an indication that promotion to the next stage is not approved, the software release management module 230 executes steps to repeat some of the steps of the pipeline, for example, after receiving revised source code or modified software artifacts. For example, approval may be denied if the number of test cases passed is less than a threshold. Thus, the source code is revised and the steps are repeated until at least a threshold number of test cases have passed and approval is received. Once all pipelines in the set of pipelines have been approved and the set of pipelines has been successfully executed, the software release management module 230 executes a change closure stage that closes the change case.

[0159] In one embodiment, software release management module 230 monitors events logged by the pipeline execution engine that indicate the status of each action performed by the pipeline execution engine. Software release management module 230 analyzes each event within the context of the data center to determine whether the event is significant enough to be logged in a change case, and then sends information describing the event to a queue for reporting to a team associated with the service.

[0160] FIG. 19 illustrates a process for managing queues that collect event information related to changes in service configurations, according to one embodiment. An event listener 1940 of the change processing module 355 listens 1910 for events from the pipeline execution engine. For each event, the change processing module 355 performs the following steps 1920, 1930, 1940, and 1940: The change processing module 355 determines the service for which the event was generated. Thus, the change processing module 355 evaluates the event to determine whether the event is significant enough to be recorded by the change management system and reported to a team associated with the service. If the change processing module 355 determines that the event is significant enough to be recorded and reported, the change processing module 355 identifies 1930 a queue for the team associated with the service. The change processing module 355 sends 1940 the event information to the identified queue. The event is reported to the team associated with the service via the queue. The change processing module 355 sends 1950 the event-based status update to the change management system for storage in association with the change case.

[0161] Computer Architecture 20 is a high-level block diagram illustrating a functional view of an exemplary computer system for use as one of the entities depicted in the environment 100 of FIG. 1, according to one embodiment. At least one processor 2002 is shown coupled to a chipset 2004. Further coupled to the chipset 2004 are memory 2006, storage 2008, keyboard 2010, graphics adapter 2012, pointing device 2014, and network adapter 2016. A display 2018 is coupled to the graphics adapter 2012. In one embodiment, the functionality of the chipset 2004 is provided by a memory controller hub 2020 and an I / O controller hub 2022. In another embodiment, the memory 2006 is coupled directly to the processor 2002 instead of the chipset 2004.

[0162] The storage device 2008 is a non-transitory computer-readable storage medium such as a hard drive, a compact disc read only memory (CD-ROM), a DVD, or a solid-state memory device. The memory 2006 holds instructions and data used by the processor 2002. The pointing device 2014 may be a mouse, a trackball, or other type of pointing device and is used in combination with the keyboard 2010 to input data into the computer system 2000. The graphics adapter 2012 displays images and other information on the display 2018. The network adapter 2016 couples the computer system 2000 to a network.

[0163] As is known in the art, computer 2000 may have different and / or other components than those depicted in Figure 20. Additionally, computer 2000 may lack certain illustrated components. For example, a computer system 2000 operating as a multi-tenant system 110 may lack keyboard 2010 and pointing device 2014. Additionally, storage device 2008 may be local and / or remote from computer 2000 (e.g., embodied within a storage area network (SAN)).

[0164] The computer 2000 is adapted to execute computer modules for providing the functions described herein. As used herein, the term "module" refers to computer program instructions and other logic for providing a specified function. A module may be implemented in hardware, firmware, and / or software. A module may include one or more processes and / or may be provided by only a portion of a process. A module is typically stored in the storage device 2008, loaded into the memory 2006, and executed by the processor 2002.

[0165] The type of computer system 2000 used by an entity in the system environment can vary depending on the embodiment and the processing power used by the entity. For example, a client device may be a mobile phone with limited processing power, a small display 2018, and no pointing device 2014. In contrast, a multi-tenant system or cloud platform may include multiple blade servers that work together to provide the functionality described herein.

[0166] Further considerations The particular naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not required or important, and mechanisms for implementing the described embodiments may have different names, formats, or protocols. Furthermore, the system may be implemented through a combination of hardware and software as described, or entirely in hardware elements. Furthermore, the particular division of functionality between various system components described herein is merely exemplary and not required. Functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0167] Some portions of the above description are presented in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. It will be understood that these operations, while described in functional or logical terms, are implemented by computer programs. Further, it has proven convenient at times to refer to arrangements of these operations as modules or by functional names, without loss of generality.

[0168] As is evident from the above discussion, unless specifically indicated otherwise, throughout the description, discussions utilizing terms such as "processing" or "calculating" or "computing" or "determining" or "displaying" are understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the memory or registers of the computer system, or other such information storage, transmission, or display device.

[0169] Certain embodiments described herein include process steps and instructions that are described in the form of an algorithm. It should be noted that the process steps and instructions of the embodiments may be embodied in software, firmware, or hardware, and when embodied in software, may be downloaded to reside on and be operated from different platforms used by the real-time network operating system.

[0170] The described embodiments also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes or may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer-readable medium accessible by the computer. Such a computer program may be stored on a non-transitory computer-readable storage medium, such as, but not limited to, a floppy disk, an optical disk, a CD-ROM, any type of disk including a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus. Furthermore, the computers referred to herein may include a single processor or may be architectures employing multiple processor designs for increased computing power.

[0171] The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the required method steps. The required structure for a variety of these systems, along with equivalent variations, will be apparent to those skilled in the art. Moreover, the present embodiments are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the embodiments described herein.

[0172] Embodiments are well suited to a wide range of computer network systems across numerous topologies, including configuration and management of large networks including storage devices and computers communicatively coupled to disparate computers and storage devices through networks such as the Internet.

[0173] Finally, it should be noted that the language used herein has been chosen primarily for ease of reading and instructional purposes, and may not have been chosen to delineate or limit the inventive subject matter. Accordingly, the disclosure of the embodiments is intended to be illustrative and not limiting.< / version> < / service>

Claims

1. 1. A computer-implemented method for forcing a system configuration freeze for a service deployed on a cloud platform, comprising: accessing a datacenter configured on a target cloud platform, the datacenter running a set of services, the datacenter including a hierarchy of datacenter entities, each datacenter entity including (1) one or more services or (2) one or more other datacenter entities; generating a pipeline for deploying services on data center entities, said pipeline comprising: a change stage including instructions for making changes associated with one or more software artifacts for the data center entity; a pre-alteration stage including an instruction to acquire a lock, pipeline execution being suspended in response to the pre-alteration stage failing to acquire the lock; and receiving a request to perform a system configuration freeze for the data center entity for a time interval; acquiring one or more locks associated with services of said data center entity over said time interval; executing the pipeline to deploy software artifacts associated with services on the data center entity during the time interval; executing the pre-altered stage of the pipeline and suspending the pipeline execution until the lock is acquired; A method comprising:

2. acquiring the lock by the pipeline after the time interval; executing the modification stage to make the modifications associated with one or more software artifacts for the data center entity; The method of claim 1 further comprising:

3. determining that the pre-alteration stage of the pipeline has failed to acquire the lock for a period of time exceeding a threshold; in response to determining that the pre-alteration stage of the pipeline has failed to acquire the lock for a period of time exceeding a threshold, failing execution of the pipeline; The method of claim 1 or 2, further comprising:

4. The pipeline further includes a post-modification stage including instructions for releasing the lock associated with the data center entity, and executing the pipeline includes:

4. The method of claim 1, further comprising: releasing the lock after completing execution of the modification stage to make the modifications associated with the one or more software artifacts for the data center entity.

5. The system configuration freeze is associated with a priority, the method comprising: receiving a request for modification of a service of the data center entity during the time interval of the system configuration freeze, the request having a higher priority than the priority of the system configuration freeze; allowing the modification of the service to continue during the time interval despite the system configuration freeze; The method of any one of claims 1 to 4, further comprising:

6. The method of any preceding claim, wherein the system configuration freeze is performed in response to determining that a workload greater than a threshold amount is expected during the time interval.

7. 7. The method of claim 1, wherein the system configuration freeze is performed in response to determining that a problem associated with the data center entity has been diagnosed during the time interval.

8. 8. The method of claim 1, wherein the system configuration freeze is performed in response to determining that a particular system configuration change is being performed during the time interval, and wherein the system configuration freeze prevents any other system configuration changes during the time interval.

9. After said time interval, identifying one or more locks that have failed to be released by any pipeline; Releasing the identified lock; and The method according to any one of claims 1 to 8, further comprising carrying out the steps comprising:

10. receiving a request for updates to software artifacts for the data center entity in response to receiving a request for a system configuration freeze; in response to determining that an expected time to execute the request to update the software artifact is less than a threshold, delaying execution of the request for system configuration freeze until the execution of the request for update to the software artifact of the data center entity is completed; The method of any one of claims 1 to 9, further comprising:

11. The lock is associated with a length of time interval during which the lock is requested, the method comprising: monitoring whether the lock is released before a time interval of a specified length; sending a warning message in response to determining that the lock will not be released before the specified length of time interval; The method of any one of claims 1 to 10, further comprising:

12. The lock is associated with a length of time interval during which the lock is requested, the method comprising:

12. The method of claim 11, further comprising determining the length of the time interval for acquiring a lock based on past executions of a pipeline.

13. 13. The method of claim 1, wherein the data center configured on the cloud platform is associated with a tenant of a multi-tenant system, and the request for system configuration freeze is requested from a client device associated with the tenant.

14. receiving a cloud platform independent declarative specification of a data center for a tenant of a multi-tenant system, the data center representing a set of computing resources used by a set of users associated with the tenant, the cloud platform independent declarative specification configured to generate the data center on any of a plurality of cloud platforms and specified using a cloud platform infrastructure language; The method of any one of claims 1 to 13, further comprising:

15. compiling the cloud platform independent declarative specification to generate a cloud platform specific data center representation for creating the data center on the target cloud platform; transmitting the cloud platform specific data center representation and a set of instructions for execution at the target cloud platform, the target cloud platform executing the instructions to configure the data center using the cloud platform specific data center representation; The method of claim 14 further comprising:

16. The system configuration freeze is associated with a priority, the method comprising: receiving a task for execution, the task having a higher priority than the priority of the system configuration freeze, the higher priority task resulting in changes to software artifacts of the data center entity during the time interval; allowing the high priority task to execute during the time interval despite the system configuration freeze; and The method according to any one of claims 1 to 15, comprising:

17. A non-transitory computer readable storage medium for storing instructions which, when executed by a computer processor, cause the computer processor to perform the method of any one of claims 1 to 16.

18. 1. A computer system comprising: A computer processor; a non-transitory computer readable storage medium for storing instructions, the instructions, when executed by the computer processor, causing the computer processor to perform the method of any one of claims 1 to 16; and 2. A computer system comprising:

19. 1. A computer-implemented method for managing changes to services running in a data center configured on a cloud platform, comprising: accessing a datacenter configured on a target cloud platform, the datacenter running a set of services, the datacenter including a hierarchy of datacenter entities, each datacenter entity including (1) one or more services or (2) one or more other datacenter entities; receiving a modification to the set of services running on the data center; generating a master pipeline for deploying the service on a target cloud platform, the master pipeline comprising: a plurality of stages including instructions for deploying a service, each stage corresponding to a system environment, at least some of the system environments belonging to a list including a development environment, a test environment, and a production environment; a change management stage including instructions for interacting with a change management system; and compiling the master pipeline to generate a set of pipelines corresponding to the set of services, each generated pipeline including a change management stage; executing each of the set of pipelines, where execution of the change management stages of the pipelines provides a status of deployment of one or more services to the change management system; A method comprising:

20. receiving an audit request for a particular change in a particular service deployed at the data center; in response to receiving the audit request, identifying a status of the deployment of the particular service from the change management system; providing the identified status of deployment of the particular service in response to the audit request; 20. The method of claim 19, further comprising:

21. 21. The method of claim 19 or 20, wherein the change management stage of the master pipeline occurs before the stage for a production environment.

22. The method of any one of claims 19 to 21, wherein the set of pipelines forms a hierarchy corresponding to a hierarchy of the data center entities in the data center.

23. The method of any one of claims 19 to 22, wherein the master pipeline promotes software artifacts for a service through a set of system environments.

24. 24. The method of claim 23, wherein the stage of the master pipeline includes instructions for executing a set of test cases to determine whether to promote the software artifact from a current stage to a next stage.

25. 25. The method of claim 19, wherein the cloud platform independent master deployment pipeline comprises a hierarchy of pipelines having a hierarchical structure corresponding to a hierarchy of the data center entities specified by a cloud platform independent declarative specification.

26. 26. The method of claim 25, wherein the hierarchy of pipelines includes a data center instance pipeline that includes one or more service group pipelines, the service group pipelines including one or more service pipelines.

27. 26. The method of claim 25, wherein the generated pipeline change management stage includes instructions to wait for approval to proceed to the next stage.

28. receiving a cloud platform independent declarative specification; compiling the cloud platform independent declarative specification to generate a cloud platform specific representation of a data center; The method of any one of claims 19 to 27, further comprising:

29. Compiling a cloud platform independent declarative specification is generating a first version of a cloud platform independent detailed metadata representation of the data center from the original declarative specification; generating a second version of the cloud platform independent detailed metadata representation of the data center from the modified declarative specification; The method according to any one of claims 19 to 28, comprising:

30. generating a platform-specific detailed metadata representation for the target cloud platform based on the first version of the cloud platform independent detailed metadata representation; deploying the data center on the target cloud platform based on the platform specific detailed metadata representation; 30. The method of claim 29, further comprising:

31. The method of any one of claims 19 to 30, wherein the cloud platform independent declarative specification includes a definition of one or more data center instances, each data center instance including one or more service groups, each service group including a set of services.

32. The method of any one of claims 19 to 31, wherein the modifications to the set of services running on the data center are specified via an artifact version map.

33. determining the modifications to the set of services running on the data center by comparing the artifact version map to a previously received artifact version map; 33. The method of claim 32, further comprising:

34. A non-transitory computer readable storage medium for storing instructions which, when executed by a computer processor, cause the computer processor to perform the method of any one of claims 19 to 33.

35. 1. A computer system comprising: A computer processor; a non-transitory computer readable storage medium for storing instructions which, when executed by the computer processor, cause the computer processor to perform the method of any one of claims 19 to 33.

Images