Automatic rules for controlled distribution of program instructions

Abstract

Conventionally, engineers (or developers) of a software provider can implement or maintain aspects of an application service by executing jobs or joblets on a variety of ranges of computing resources in a cloud computing environment. However, in some cases, executing certain types of engineer-initiated jobs can negatively impact customer experience and / or satisfaction with the application service. Based on dynamic risk analysis of each job, automated policies for distributing engineer-initiated jobs increase validation testing, range control, and deployment timing. A multi-faceted approach for distributing engineer-initiated jobs ensures adequate regression testing (e.g., via ring validation and cool-down periods), and facilitates controlled distribution (e.g., based on throttling, distribution timing, and deployment sequence position). Automated overrides ensure that critical customer outages can be quickly and effectively addressed by bypassing at least some of the rules. Thus, when distributing engineer-initiated jobs for an application service, customer experience is improved and service outages and customer dissatisfaction are minimized.

Description

Background Technology

[0001] The use of cloud computing environments for software products and services has become popular for both consumers and businesses. Consumers and businesses, as customers of software providers, can use the computing resources of a cloud computing environment as tenants. Customers' end users can use the productivity tools (e.g., features) of application services to upload files, share files, and perform other operations, such as collaborating on tasks. In some aspects, customers can request customization or updates to the features of application services. A cloud computing environment involves a distributed set of resources, including storage, processing units, and data storage devices interconnected via secure network communications. Distributed computing resources can be integrated both physically and virtually. The virtual integration of servers enables the dynamic allocation of resources to tenants in a cloud computing environment. In this way, service providers enable tenants to seamlessly and dynamically scale up or down their usage by continuously evaluating and reallocating resources among the servers hosting the tenants. However, in some cases, service providers may perform certain types of engineer-initiated jobs (e.g., engineering jobs) that negatively impact customer experience and / or satisfaction with application services.

[0002] For example, engineers (or developers) of a software provider can implement or maintain aspects of application services by executing jobs or joblets (hereinafter referred to as "engineering jobs") on compute resources across various scopes in the cloud. As used herein, "scope" can refer to any logical partition of compute resources within a cloud computing environment (e.g., cluster, server, database, VLAN) or any object type associated with an application service (e.g., application endpoint). Scope policies can be used to define common scopes (e.g., virtual machines, clusters, databases, etc.) or scopes based on common policies shared between objects (e.g., application endpoints and devices). Engineering jobs are executed based on program instructions (e.g., scripts) generated by engineers (e.g., standby engineers, OCEs) and / or developers. Engineering jobs can refer to maintenance (e.g., updating cloud infrastructure or application services), code repair (e.g., correcting code), code updates (e.g., updating existing features), configuration (e.g., configuration settings or properties), customization (e.g., enhancements requested by customers), etc.

[0003] Regarding these and other general considerations, the aspects disclosed herein have been addressed. Similarly, while relatively specific problems may be discussed, it should be understood that the examples described are not intended to address specific problems identified in the context of this disclosure or elsewhere. Summary of the Invention

[0004] According to this disclosure, the above and other issues are addressed by an automation strategy that increases validation testing, scope control, and deployment timing based at least in part on dynamic risk analysis of each engineering task to distribute program instructions associated with the engineering task. The "tenant" or customer of the application service may be hosted by the software provider on a physical or virtual server in a cloud computing environment. "Features" may be associated with operations performed by the application service, such as uploading files, creating lists, sharing files, verifying requests, creating sections of a portal page, etc. "Engineering tasks" are coded by program instructions (or scripts) and executed by the software provider to maintain, update, repair, configure, or customize the application service. As used herein, the terms "distribution," "release," or "deployment" for program instructions (e.g., code and / or scripts) refer to the execution of program instructions for at least a portion of a scope.

[0005] As mentioned above, engineering operations can be implemented based on scope-specific program instructions (e.g., scripts). However, problems can arise when engineering operations are deployed without adequate testing, risk assessment, and / or scope control. For example, without sufficient testing, errors (e.g., vulnerabilities) in associated program instructions can negatively impact the customer experience for tenants both within and outside the scope. Depending on the type of engineering operation, the deployment of scripts without risk assessment and scope control can lead to service disruptions for customers and / or failures for software providers by overloading computing resources within the cloud computing environment. Therefore, insufficient internal controls over engineering operations can result in the over-distribution and overly rapid dissemination of program instructions.

[0006] Specifically, to address the problems identified above, this system and method create and apply automated policies based on dynamic analysis of the system and pending engineering tasks. For example, scope attributes can be defined. Scope attributes can include service scope levels (e.g., cluster level, database level, network level, tenant level, etc.) and update characteristics (e.g., read-write or read-only, such as Set / Get commands). Risk levels (e.g., low, medium, high) can also be defined for different combinations of scope attributes. For example, a read-only update characteristic for a database-level scope might be associated with low risk, a read-write update characteristic for a tenant-level scope might be associated with medium risk, and a read-write update characteristic for a multi-cluster level scope might be associated with high risk. In summary, when an engineer selects a “new” scope (e.g., a previously undefined scope), the risk may default to “high.” It should be recognized that different combinations of scope attributes can be associated with different risk levels.

[0007] Based on risk levels, rules (or policies) can be created to automate internal controls for engineering operations. For example, rules can define one or more criteria for implementing engineering operations associated with different risk levels. Criteria can include ring-level validation, throttle, cool-down periods, distribution timing, and deployment sequence location. "Ring-level validation" refers to the requirement to distribute an engineering operation to an internal ring (or "Dogfood ring") before distributing it to an external ring (e.g., the "production ring"). For example, an engineering operation can be distributed to at least some internal users associated with the software provider (e.g., the Dogfood ring) before being distributed to external users associated with the customer (e.g., one or more production rings). Multiple production rings can exist in all aspects, and the remaining criteria can be applied within each production ring. As used herein, "throttle" refers to a subset (or portion) of the range of program instructions that can be executed at each stage (or iteration) of distribution (e.g., the percentage of VMs at the virtual machine (VM) level, the percentage of cross-regional farms at the farm level, the percentage of databases at the database level, etc.). In all aspects, cost-cutting for high-risk operations may be more significant than cost-cutting for low-risk operations (e.g., slower distribution). A “cooling-off period” (or “baking time”) refers to the time interval (e.g., hours, days, weeks, etc.) between phases (or iterations) of distributing an engineering task. In all aspects, during the cooling-off period, feedback (e.g., telemetry data) can be collected and evaluated to ensure the engineering task performs as intended before the next phase of distribution. In all aspects, the cooling-off period for high-risk operations may be longer than the cooling-off period for low-risk operations.

[0008] Additionally, standards can be defined for timing distribution. "Peak" times refer to a portion of the day prioritizing customer demand for cloud resources (e.g., 6:00 AM to 6:00 PM local time), while "off-peak" times refer to a portion of the day for system maintenance, deployment of new or updated features, internal software provider activities, etc. (e.g., 6:00 PM to 6:00 AM local time). A "deployment sequence" refers to the sequence in which engineering jobs (or batches of engineering jobs) are distributed, and a "deployment sequence position" refers to the relative priority (or order) of one engineering job (or batch of engineering jobs) compared to another (e.g., a higher-priority engineering job is distributed before a lower-priority engineering job).

[0009] In some aspects, automated coverage workflows can be implemented. That is, to ensure engineers are capable of quickly resolving critical customer issues, specific engineering tasks (e.g., code fixes) can be exempt from applying rules, or can override rules. For example, automated coverage workflows can be associated with additional criteria, such as distribution by engineers with specific roles (e.g., "OCE" or standby engineers), customer issues associated with a threshold severity level (e.g., SEV0), or affected customers associated with special states (e.g., watchlist, strategic tenants). In this case, automated approval for the coverage rule may be issued when the additional criteria are met. Alternatively, in special or extenuating circumstances, manual approval may be required to cover the rule.

[0010] It should be recognized that the multi-layered approach used for distributing engineering jobs ensures additional validation testing (e.g., via loop validation and cool-down periods) and facilitates metered (or controlled) distribution (e.g., based on throttling, distribution timing, and deployment sequence location). In this way, additional validation testing improves customer experience and product satisfaction; while metered distribution prevents engineering jobs from being distributed too widely and too quickly, thereby preventing customer service disruptions and / or software provider failures due to system resource overload. Automated coverage workflows ensure that critical customer failures can be resolved quickly and effectively by bypassing at least some of the rules. Therefore, the systems and methods described herein address the service disruptions and customer dissatisfaction associated with the distribution of engineering jobs for application services delivered via cloud computing environments.

[0011] This summary is provided to present the selection of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description below, and will be apparent in part from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0012] The following figures illustrate a non-limiting and non-exhaustive example.

[0013] Figure 1 An overview of an exemplary system for distributing program instructions according to various aspects of this disclosure is illustrated.

[0014] Figure 2 An exemplary method of distributing program instructions according to various aspects of this disclosure is illustrated.

[0015] Figure 3 An exemplary method for distributing program instructions at the outer ring level using rule-based applications according to various aspects of this disclosure is illustrated.

[0016] Figure 4 An example of a user script table according to various aspects of this disclosure is illustrated.

[0017] Figure 5 An example of a rule table for distributing user scripts according to various aspects of this disclosure is illustrated.

[0018] Figure 6 An example of a distribution sequence list according to various aspects of this disclosure is illustrated.

[0019] Figure 7 It is a block diagram illustrating exemplary physical components of a computing device that can utilize various aspects of the present disclosure.

[0020] Figure 8A It is a simplified diagram of a mobile computing device that can be used to implement various aspects of the present disclosure.

[0021] Figure 8B This is another simplified block diagram of a mobile computing device that can be used to implement various aspects of the present disclosure. Detailed Implementation

[0022] Various aspects of this disclosure are described more fully below with reference to the accompanying drawings, which form part of this document and illustrate specific exemplary aspects. However, different aspects of this disclosure may be implemented in many different ways and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that this disclosure will be thorough and complete and will fully communicate the scope of these aspects to those skilled in the art. The aspects may be practiced as methods, systems, or devices. Thus, the aspects may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Therefore, the detailed description below should not be construed as limiting.

[0023] This disclosure relates to systems and methods for distributing program instructions to tenants in a cloud computing environment. A cloud computing environment is a distributed system in which application functions, storage, data storage and retrieval, and various processing functions can operate remotely to each other via a network (such as the Internet or a secure intranet). The "tenants" or customers of the application services can be hosted by the software provider on physical or virtual servers within the distributed system. In fact, cloud-based application services are already widely used and relied upon as business productivity tools. In this way, business end-users associated with enterprise customers can collaborate online (e.g., storing / sharing documents, accessing internal sites, lists / libraries) from anywhere in the world. End-users can access the services using various devices (e.g., mobile phones, laptops, and tablets). The application services are available 24 / 7, 365 days a year, with minimal downtime for maintenance.

[0024] Typically, application service customers or tenants have begun to expect frequent updates that provide access to the latest new features and rapid distribution of necessary fixes. Unlike standalone applications where program instructions reside on the end user's local computing device, application services can be continuously serviced by distributing program instructions (e.g., code or scripts) used for engineering jobs to tenants hosted on servers in the cloud. For example, these servers could collectively serve application services to millions of users. However, problems can arise when the distribution of program instructions burdens the cloud infrastructure and / or when the distributed code contains errors. For example, engineering jobs may affect the functionality of resources across the distribution scope, such as virtual or physical servers occupied by tenants, server farms, regions, one or more network certificates in the cloud, etc. For example, in some cases, the distribution of engineering jobs occurs before an appropriate level of regression testing is completed. In other aspects, performing engineering jobs across a scope can overload the system through, for example, excessive resource consumption, CPU utilization, and / or memory usage. Therefore, increased testing, risk analysis, and scope control are required for distributing engineering jobs.

[0025] Engineering jobs encompass a set of scripts executed by software provider engineers or developers to maintain application services. These tasks include updating compute resource characteristics, updating network address characteristics and certificates, resolving customer issues, upgrading system infrastructure, recovering deleted customer data, and making new configurations effective. For example, an engineering job might be directed to updating network address configurations (e.g., adding IPv6 address patterns), topology network data routing configurations, server access control configurations, and virtual machine configurations within physical servers. Some configurations may be specific to one or more tenants, while others may be for multiple tenants across a region or an entire region. However, in the past, engineering jobs have been deployed without adequate testing, risk assessment, and / or scope control, potentially leading to service interruptions and / or poor customer experience. To address these issues, this application describes a multi-layered approach for distributing engineering jobs. For example, the deployment of engineering jobs is controlled by defining the scope of compute resources affected by the deployment, assessing the risks associated with deploying engineering jobs based on the defined scope and update characteristics (e.g., read-only or read-write), and generating a set of rules to control the deployment of engineering jobs.

[0026] For example, rules might involve defining criteria for deploying engineering jobs, such as ring-level validation, throttling, cooldown periods, execution timing, and deployment sequence locations. "Ring-level validation" refers to the requirement to distribute engineering jobs to internal rings (or "Dogfood rings") before distributing them to external rings (e.g., "production rings"). For example, engineering jobs might be distributed to at least some internal users associated with the software provider (e.g., the dogfood rings) before being distributed to external users associated with the customer (e.g., production rings). Multiple production rings can exist in each aspect, and the remaining criteria can be applied within each production ring. In this way, the distribution of program instructions can be implemented according to the hierarchy of regression testing. The hierarchy can be represented by multiple concentric distribution "rings": for example, the "dogfood ring" for internal testing is the innermost ring, while the "production ring" for external (e.g., customer) regression testing is the outermost ring. Within each distribution ring, there can be a group of users who receive evaluations of the application service after the engineering task is distributed, and telemetry data can be collected and evaluated based on user activity within the ring to determine whether the engineering task was performed properly.

[0027] As detailed above, the standard may also involve throttling to restrict the deployment of engineering jobs to a defined portion of the range within each iteration, and establishing cool-down periods between iterations. Additionally, system resources can be evaluated to determine the distribution order based on the throttling. For example, if the throttling instructs that engineering jobs should be performed on no more than 10% of the cluster-level range of clusters in each iteration, then various clusters at the cluster level range can be evaluated. Clusters exhibiting high user activity can be considered "hot," while clusters with low user activity can be considered "cold." To prevent potential service disruptions, engineering jobs can be distributed to cold clusters before warm or hot clusters. That is, the first iteration might distribute engineering jobs to the 10% of clusters considered "cold," while the final iteration might distribute engineering jobs to the 10% of clusters considered "hot," with each intermediate iteration distributing to 10% of clusters in a continuum from cold to warm to hot.

[0028] In various aspects, after each deployment iteration, for example, telemetry data regarding changes in the execution of engineering jobs and / or computational resource consumption can be evaluated during a cooldown period before moving to the next iteration. Therefore, combined ring validation can distribute engineering jobs in each ring based on throttling with a cooldown period between each distribution. For example, based on throttling to 10 tenants per distribution or 5% of tenants per distribution, engineering jobs can be deployed across a series of distributions. After each distribution, the execution of engineering jobs can be evaluated to ensure code stability and user satisfaction during the cooldown period. For example, the cooldown period could be 6 hours to 24 hours. Once a series of distributions in the previous ring is complete, distribution can proceed to the next ring.

[0029] Standards used for distributing engineering jobs can also involve defining the timing of distribution. For example, based on job type (e.g., maintenance vs. code fix), engineering jobs can be distributed during different time periods of the day (e.g., maintenance jobs can be performed during off-peak hours, while urgent code fix jobs can be performed during peak hours to prevent further customer downtime). Peak and off-peak times can be defined based on the tenant's local time associated with the defined scope for the engineering job. In some cases, peak / off-peak times can be scheduled (e.g., peak hours from 6:00 AM to 6:00 PM and off-peak hours from 6:00 PM to 6:00 AM). In other cases, peak / off-peak times may be tenant-specific. For example, some tenants may experience peak usage of application services during weekday business hours from 9:00 AM to 5:00 PM local time; while other tenants may experience peak hours on weekends from noon to midnight local time.

[0030] Furthermore, engineering jobs (or batches of engineering jobs) can be ordered within a "deployment sequence" based on their relative priority (e.g., higher-priority engineering jobs are distributed before lower-priority ones). Typically, engineering jobs are executed in the order they are received (e.g., the order in which engineers choose to execute them). Therefore, based solely on the timing chosen by the engineer, high-priority engineering jobs (e.g., code fixes to resolve customer issues) often execute after low-priority engineering jobs (e.g., general system maintenance). According to this system, the priority levels of multiple engineering jobs selected for execution can be determined. Higher-priority engineering jobs (or batches of engineering jobs) can then be distributed before lower-priority ones (or batches of engineering jobs). When ordering engineering jobs within a distribution sequence, the distribution of individual engineering jobs (or batches of engineering jobs with the same priority) can be based on the loop validation, throttling, cooling, and peak / off-peak criteria described above for a particular engineering job(s).

[0031] In some aspects, automated coverage workflows can be implemented. That is, to ensure engineers are capable of quickly resolving critical customer issues, specific engineering tasks (e.g., code fixes) may be exempt from applying rules, or they may override rules. For example, automated coverage workflows can be associated with additional criteria, such as distribution by engineers with specific roles (e.g., "OCE" or standby engineers), customer issues associated with a threshold severity level (e.g., SEV0), or affected customers associated with special states (e.g., watchlist, strategic tenants). In this case, automated approval for the coverage rule may be issued when one or more additional criteria are met. Alternatively, in some cases, manual approval for the coverage rule may be available.

[0032] It should be recognized that the multi-layered approach used for distributing engineering jobs ensures additional validation testing (e.g., via loop validation and cool-down periods) and facilitates metered or controlled distribution (e.g., based on throttling, distribution timing, and deployment sequence location). In this way, additional validation testing improves customer experience and product satisfaction; while metered distribution prevents engineering jobs from being distributed too broadly or too quickly, thereby preventing customer service disruptions and / or software provider failures due to system resource overload. Automated coverage ensures that critical customer failures can be resolved quickly and effectively by bypassing at least some of the rules described above. Therefore, the systems and methods described herein address the issues of service disruptions and customer dissatisfaction associated with the distribution of engineering jobs for application services delivered via cloud computing environments.

[0033] Figure 1The illustration shows an overview of an exemplary system for distributing program instructions according to various aspects of this disclosure. System 100 may represent a system for distributing program instructions to one or more tenants.

[0034] Developer 102 develops program instructions for engineering jobs executed across a scope to maintain, update, repair, configure, or customize application services hosted by the software provider in cloud computing environment 140. In various ways, developer 102 provides program instructions (e.g., engineering script 103) to code storage 124 associated with distribution coordinator 106. For example, developer 102 may select program instructions for distribution (or execution) across a scope. In some cases, developer 102 may update program instructions after distribution based on telemetry data received from regression testing by telemetry data receiver 112. The telemetry data depicts how end-user 104 interacts with the application service after the engineering job is distributed.

[0035] The customer's end user 104 uses computing devices to access application services. For example, end user 104 may access application services via a cloud computing environment 140. The cloud computing environment 140 includes at least one zone (or region) 150. Zone 150 may contain one or more server clusters 152A-152B. Server clusters 152A-152B may contain virtual servers 160A-160B and physical servers 154A-154D. Server clusters 152A-152B may also include certificate servers 156A-156B. For example, certificate servers 156A-156B store and maintain certificates used for authentication to access network and computing resources (such as databases and servers). Server clusters 152A-152B in zone 150 provide application services to end user 104, such as file sharing, workflow services, and collaboration tools. In all respects, millions of end users 104 may use the various application services available through the cloud computing environment 140.

[0036] The cloud computing environment 140 may include one or more resources, including one or more servers (e.g., including processing units and storage) and one or more databases communicatively coupled via a network. The cloud computing environment may be organized into one or more server clusters 152A-152B containing one or more virtual servers 160A-160B and / or one or more physical servers 154A-154C. In various aspects, server clusters 152A-152B may form zone 150. One or more virtual servers 160A-160B and one or more physical servers 154A-154C organized into server clusters 152A-152B in zone 150 collectively provide application services to end user 104. In various aspects, virtual servers 160A-160B partitioned on one or more physical servers 154A-154C provide dedicated server functionality to one or more clients (e.g., tenants). For example, tenant 1162A can occupy a virtual server 160A on physical server 154D, and tenant 2162B can occupy another virtual server 160B.

[0037] In various aspects, the memory in the corresponding virtual servers 160A-160B and physical servers 154C-154D can store program instructions for execution by a central processing unit (CPU) not shown in the figure. The servers implement application services by executing these program instructions. In some aspects, engineering jobs encoded by engineering script 103 can be executed for a scope to maintain, update, repair, configure, or customize application services. As used herein, "scope" can refer to a tenant, tenant group, server farm cluster, server farm group, user group, or any other logical division of the cloud computing environment 140.

[0038] Distribution coordinator 106 receives the set of program instructions for distribution and then schedules and implements the distribution. In some aspects, developer 102 registers code associated with an engineering job for storage in code storage device 124. In other aspects, developer 102 requests code distribution (e.g., deployment) for a range of program instructions. Distribution coordinator 106 can assign code IDs to the set of program instructions. Engineering jobs can be used to maintain, update, enhance, or correct (fix) problems with application services.

[0039] The attribute determiner 108 can determine the attributes (or parameters) of the program instruction set selected for execution against a scope. As described above, a scope can include any logical partition of the cloud computing environment 140, including virtual servers, physical servers, server farms, zones (or regions), certificate servers, networks, databases, etc. In various respects, the scope can be selected by the developer 102 and indicates the partition of the cloud computing environment 140 on which the program instruction set will be executed. For example, for an engineering job that updates the attributes of a virtual server, the corresponding scope could be the virtual server. Conversely, an engineering job that requests code fixes or enhancements for a tenant might have a corresponding scope of the physical servers hosting the tenant. Therefore, the attributes associated with a scope can be scope level (e.g., farm level, database level, network level, tenant level, etc.). The program instruction set can also be evaluated by the attribute determiner 108. For example, the attributes associated with a program instruction can be update characteristics regarding how the program instruction is executed on the scope (e.g., read-write or read-only, such as Set / Get commands). Additionally, the attribute determiner 108 can determine the job type of the engineering job associated with the program instruction set. For example, engineering jobs can be maintenance, update, configuration, customization, code repair, etc. For instance, job types associated with programming instruction sets can be received from developer 102.

[0040] Risk level determiner 110 can determine the risk level associated with the execution program instruction set for a scope based on attributes determined by attribute determiner 108. For example, the risk level value can be high, medium, or low. As described above, the scope can include any logical division of the cloud computing environment 140, and scope level attributes can include virtual server level, physical server level, server cluster level, zone level, certificate server level, network level, database level, etc. Additionally, update characteristic attributes associated with the program instruction set can include, for example, read-only or read-write. Risk levels (e.g., low, medium, high) can also be defined for different combinations of attributes. For example, read-only update characteristics for a database-level scope might be associated with low risk, read-write update characteristics for a tenant-level scope might be associated with medium risk, and read-write update characteristics for a multi-cluster level scope might be associated with high risk. In summary, if an engineer selects a “new” scope (e.g., a previously undefined scope), the risk may default to “high.” It should be recognized that different combinations of attributes may be associated with different risk levels.

[0041] Rule determiner 112 can determine one or more rules for controlling the distribution of program instruction sets based on risk levels. In various aspects, rules can be determined from a list of predefined rules in a rule table. For example, rules may include throttling of distribution, cool-down periods, and timing of program instruction set execution. "Throttling" refers to a subset (or portion) of the range of program instructions that can be executed at each stage (or iteration) of the distribution (e.g., the percentage of VMs at the virtual machine (VM) level, the percentage of cross-regional clusters at the cluster level, the percentage of databases at the database level, etc.). In various aspects, throttling rules for high-risk operations may be more stringent than throttling rules for low-risk operations (e.g., slower distribution). "Cool-down period" refers to the time period between stages (or iterations) of distributing program instruction sets (e.g., hours, days, weeks, etc.). In various aspects, during the cool-down period, feedback (e.g., telemetry data) can be collected and evaluated to ensure that the program instruction set is properly executed before the next stage of distribution. In various aspects, cool-down rules for high-risk operations may be longer than cool-down rules for low-risk operations. Therefore, for example, a cooling-off rule could be 6 hours for low-risk operations, while for high-risk operations it could be 24 hours. In this way, based on throttling and cooling-off rules, the distribution of program instruction sets can be controlled and verified to prevent poor customer experience. Additionally, it can prevent errors from internal activities and / or developers.

[0042] Alternatively or additionally, rule determiner 112 may determine one or more rules associated with the timing of executing program instruction sets. For example, program instructions may be distributed at different times of the day based on risk level (e.g., low, medium, high) and / or job type (e.g., maintenance vs. code repair). "Peak" time refers to a portion of the day prioritizing customer demand for cloud resources (e.g., 6:00 AM to 6:00 PM local time), while "off-peak" time refers to a portion of the day for system maintenance or internal software provider activities (e.g., 6:00 PM to 6:00 AM local time). Therefore, timing rules may indicate whether program instruction sets should be executed during peak or off-peak periods. For example, timing rules for maintenance jobs may be executed during off-peak periods, while timing rules for code repair jobs may be executed during peak periods to prevent further customer downtime. In either case, service interruptions and / or customer dissatisfaction can be minimized. Alternatively or additionally, system resources may be assessed to determine the order (or timing) of program instruction set distribution based on throttling. For example, if throttling instructs that program instructions should be executed on no more than 10% of the cluster-level range of clusters in each iteration, then various clusters at the cluster level range can be evaluated. Clusters exhibiting high user activity can be considered "hot," while clusters with low user activity can be considered "cold." To prevent potential service interruptions, cold / hot rules can instruct program instruction sets to be distributed to cold clusters before warm or hot clusters. That is, based on cold / hot rules, the first iteration can distribute program instruction sets to the 10% of clusters considered "cold," while the final iteration can distribute program instruction sets to the 10% of clusters considered "hot," with each intermediate iteration distributing to 10% of clusters in a continuum from cold to warm to hot. In this way, rules can be applied dynamically based on the state of system resources at the time of distribution, thereby avoiding customer service interruptions or failures.

[0043] Ring-level validator 114 can receive instructions to begin distributing program instructions associated with an engineering task at a specific ring level. Compared to a conventional system, the deployment of the set of program instructions associated with an engineering task may undergo validation and testing at multiple ring levels. "Ring-level validation" refers to the requirement to distribute an engineering task to an internal ring (or "dogfood ring") before distributing it to an external ring (e.g., a "production ring"). For example, an engineering task may be distributed to at least some internal users associated with the software provider (e.g., the dogfood ring) before being distributed to external users associated with a customer (e.g., one or more production rings). Multiple production rings may exist in all aspects, and additional rules may be applied within each production ring. In this way, the distribution of program instructions can be implemented according to a hierarchy of regression testing. This hierarchy can be represented by multiple concentric distribution "rings": for example, the "dogfood ring" for internal testing is the innermost ring, and the "production ring" for external (e.g., customer) regression testing is the outermost ring. Within each distribution ring, a group of users can receive evaluations of the application service after the engineering task is distributed. Telemetry data can be collected and evaluated based on user activity within the ring to determine whether the engineering task was performed appropriately. Verification can be received from managers overseeing quality control for the application service. Ring-level verification may differ between ring levels. For example, verification at the "dogfood" level might involve developers and managers; while verification at the production level might involve quality control managers and customer relationship managers who maintain communication with affected tenants. In this way, additional verification of the program instruction set can prevent poor customer experience and potential service interruptions. Furthermore, the extra layer of verification protects against internal activity and / or developer errors.

[0044] Code distributor 116 can deploy program instructions based on ring-level verification and the application of one or more rules. For example, at the first ring level, program instructions can be distributed to internal users associated with the software provider. After verification at the first ring level, program instructions can be distributed to external users associated with customers in the second ring level. In some aspects, external users in the second ring level can be specifically selected by the software provider to provide feedback on the execution of program instructions and / or engineering tasks. In other aspects, program instructions are distributed in the second ring level (and any subsequent ring levels) based on throttling rules, cooling rules, and (one or more) timing rules. That is, code distributor 116 executes the rules determined by rule determiner 112. In some aspects, such controlled distribution can be based on a deployment sequence model. The deployment sequence model can control the sequence of distributing different sets of program instructions based on the relative priority of associated engineering tasks (e.g., distributing higher-priority engineering tasks before lower-priority ones). Conventionally, engineering tasks are executed in the order they are received (e.g., the order in which engineers choose to execute engineering tasks). Therefore, high-priority engineering jobs (e.g., code fixes to resolve customer issues) are often executed after low-priority engineering jobs (e.g., general system maintenance). According to this system, code dispatcher 116 can determine the priority levels of multiple engineering jobs associated with selected program instructions for execution. Higher-priority engineering jobs (or batches of engineering jobs) can then be dispatched before lower-priority engineering jobs (or batches of engineering jobs). While sorting engineering jobs within the dispatch sequence, code dispatcher 116 can then dispatch individual program instruction sets (or batches of program instructions of the same priority) based on the ring-level verification, throttling, cooling, and timing rules for a specific program instruction set described above. In some aspects, ring-level verifier 114 verifies each ring level after code dispatch at that ring level before dispatching to the next ring level.

[0045] Coverage executor 120 can identify and execute coverage workflows under specific circumstances. That is, to ensure engineers are capable of quickly resolving critical customer issues, specific engineering tasks (e.g., code fixes) can be exempt from rules or can “cover” the rules described above. For example, a coverage workflow can be associated with coverage criteria, such as distribution by an engineer with a specific role (e.g., “OCE” or a standby engineer), a customer issue associated with a threshold severity level (e.g., SEV0), or an affected customer associated with a special status (e.g., watchlist, strategic tenant). In this case, coverage determiner 118 can issue automatic approval for the coverage rule when one or more coverage criteria are met. Alternatively, in some cases, manual approval for the coverage rule can be issued by the on-duty manager or other authorized individual. Once approved, coverage workflow executor 120 can execute the coverage workflow, allowing the set of program instructions to be executed against a defined scope selected by the developer (e.g., developer 102). Coverage workflows are provided as a safety valve to bypass rules when engineering tasks (e.g., code fixes) resolve urgent customer issues. In some cases, coverage workflows can bypass ring-level validation and rule application. In other cases, coverage workflows can bypass the application of rules within the current ring of ring-level validation. In some aspects, such as to prevent internal activity, overflow workflows may require additional approvals (e.g., automatic or manual as described above). It should be recognized that coverage workflows allow standby engineers to quickly resolve urgent customer issues by bypassing rules, thereby minimizing time to resolution (TTR) and improving customer satisfaction. Simultaneously, meeting coverage criteria can prevent potential internal activity.

[0046] Telemetry data processor 122 can receive and process status information from servers within a distributed range (or a portion thereof). This status information describes how the client's end user 104 interacts with the application service after the distribution of the program instruction set. Alternatively or additionally, the telemetry data can provide information about resource usage within the distributed range, such as CPU utilization, network speed, or bandwidth. For example, the telemetry data may include logs of end-user access to the application service (e.g., click data, number of requests made, response times, etc.) and / or error reports identifying errors or anomalies that occur during the execution of program instructions. In various aspects, developer 102 can receive telemetry data for analyzing the execution or performance of program instructions. In some cases, developer 102 can update the program instruction set to correct any identified problems before further distribution.

[0047] When code distributor 116 completes the distribution of program instructions to the first verification ring level, ring level verifier 114 verifies the next verification ring level for which the program instruction set distributed by code distributor 116 will be used. In some cases, if the developer updates the program instruction set based on telemetry data, code distributor 116 may distribute the updated program instruction set to the next verification ring level. Alternatively, code distributor 116 may distribute the updated program instruction set to the first verification ring level.

[0048] Code storage device 124 stores a set of program instructions. Code storage device 124 can store the set of program instructions by distributing identifiers to appropriate sets. In some aspects, code storage device 124 includes a queue of program instructions for distribution. Developer 102 can add the program instruction set to the queue when selecting a program instruction set for distribution or after modifying the program instruction set (e.g., based on analysis of telemetry data).

[0049] The attribute / rule database 126 stores information about various attributes concerning the scope of risks involved in determining the distribution of program instructions, as well as a list of rules for controlling the distribution of program instructions based on the determined risks. These attributes may include, but are not limited to, scope levels (e.g., cluster level, database level, network level, tenant level, etc.), update characteristics regarding how program instructions are executed at the scope (e.g., read-write or read-only, such as Set / Get commands), and job types associated with the program instruction set. For example, an engineering job may be a maintenance type, update type, configuration type, customization type, code repair type, etc. For example, job types associated with the programming instruction set may be received from developer 102. The rule list may include a set of rules indexed by rule identifiers. The rule list may include, but is not limited to, throttling rules, cooling-off rules, and / or timing rules (or multiple rules). The attribute / rule database 126 may also include a list of coverage criteria for executing coverage workflows, including specific roles for qualified engineers (e.g., "OCE" or standby engineers), threshold severity levels for customer faults (e.g., SEV0), and specific states for affected customers (e.g., watchlist, strategic tenant), etc.

[0050] Tenant database 128 stores information about tenants (or customers). This tenant information may include, but is not limited to: tenant identifier, tenant name, and identifiers of the computing resources hosting the tenant (e.g., virtual servers, physical servers, server farms, and regions). Tenant database 128 may also include statistics on the end-user usage levels of the application services for each tenant.

[0051] It should be recognized that, regarding Figure 1The various methods, devices, applications, features, etc., described herein are not intended to limit System 100 to being performed by the specific applications and features described. Therefore, additional controller configurations may be used to practice the methods and systems herein, and / or may exclude the described features and applications without departing from the methods and systems disclosed herein.

[0052] Figure 2 An exemplary method of distributing program instructions according to various aspects of this disclosure is illustrated.

[0053] exist Figure 2 The diagram illustrates the general sequence of operations for method 200. Typically, method 200 begins with start operation 201 and ends with end operation 228. Method 200 may include more or fewer steps, or may differ from the steps outlined in the diagram. Figure 2 The steps shown are arranged in order. Method 200 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 200 can be executed by gates or circuits associated with a processor, ASIC, FPGA, SOC, or other hardware device. References will be incorporated herein by reference. Figure 1 , 3 Method 200 is explained by the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, and methods described in 4, 5, 6, 7, and 8A-B.

[0054] At operation selection 202, you can select program instructions (e.g., code or scripts) for distribution to a scope. For example, a developer can select program instructions for distribution. Program instructions can perform engineering tasks to maintain application services, such as updating computing resource characteristics, updating network address characteristics and certificates, resolving customer issues, and upgrading system infrastructure. A scope can include any logical partition of a cloud computing environment, including virtual servers, physical servers, server clusters, zones (or regions), certificate servers, networks, databases, etc.

[0055] At operation 204, it can be determined whether the program instructions encode an engineering task for resolving a customer fault associated with a threshold severity level (e.g., SEV0). For example, an SEV0 fault could refer to a critical customer fault. If the engineering task resolves the critical customer fault, the operation proceeds to attribute determination operation 206. If the engineering task does not resolve the critical customer fault, the operation continues to application operation 218.

[0056] At attribute determination operation 206, attributes of program instructions and scopes can be determined. For example, attributes associated with a scope can be scope level (e.g., cluster level, database level, network level, tenant level, etc.). Attributes associated with program instructions can be update characteristics regarding how the program instructions are executed against the scope (e.g., read-write or read-only, such as Set / Get commands). Additionally, the job type of the engineering job associated with the program instruction can be determined. For example, an engineering job can be maintenance type, update type, configuration type, customization type, code fix type, etc.

[0057] At risk determination operation 208, the risk level associated with executing program instructions for a specific scope can be determined based on the aforementioned attributes. This risk level may correspond to the likelihood of adversely affecting the customer by distributing program instructions. For example, read-only update characteristics at the database level might be associated with low risk, read-write update characteristics at the tenant level might be associated with medium risk, and read-write update characteristics at the multi-site cluster level might be associated with high risk. In summary, if an engineer selects a “new” scope (e.g., a scope that was not previously defined), the risk may default to “high.” It should be recognized that different combinations of attributes may be associated with different risk levels.

[0058] At rule determination operation 210, one or more rules for distributing program instructions can be determined based on risk level. In some aspects, rules can be identified from a predefined list of rules (e.g., stored in attribute / rule database 126). In other aspects, one or more rules can be dynamically generated based on risk level. Rules can include ring verification, throttling, cooldown periods, and / or distribution timing. For example, rules requiring ring-level verification of program instructions can be generated. "Ring-level verification" refers to distributing program instructions to an inner ring (or "dogfood ring") before distributing them to an outer ring (e.g., a "production ring"). In various aspects, multiple production rings can exist, and additional rules can be applied within each production ring. Throttling refers to a subset (or portion) of the range of operations that program instructions can perform at each stage (or iteration) of distribution (e.g., the percentage of VMs at the virtual machine (VM) level, the percentage of cross-regional farms at the farm level, the percentage of databases at the database level, etc.). In various aspects, throttling for high-risk operations may be higher than throttling for low-risk operations (e.g., slower distribution). A "cooling-off period" refers to the time interval (e.g., hours, days, weeks, etc.) between phases (or iterations) of an engineering task distribution. During the cooling-off period, feedback (e.g., telemetry data) can be collected and evaluated to ensure the engineering task performs as intended before the next phase of distribution. Cooling-off periods for high-risk tasks may be longer than those for low-risk tasks.

[0059] Additionally, rules for the timing (or ordering) distribution of program instructions can be determined. For example, based on risk level (e.g., low, medium, high) and / or job type (e.g., maintenance vs. code fix), program instructions may be distributed during different time periods of the day. For instance, timing rules may specify whether program instructions should be executed during peak or off-peak hours (e.g., maintenance jobs may be executed during off-peak hours, while urgent code fix jobs may be executed during peak hours to prevent further customer downtime). Alternatively or additionally, system resources can be assessed to determine the distribution order (or timing) of program instruction sets based on throttling. For example, if throttling specifies that program instructions should be executed for no more than 10% of the field-level range of fields in each iteration, various field-level ranges can be assessed. Fields exhibiting high user activity can be considered "hot," while fields with low user activity can be considered "cold." To prevent potential service interruptions, cold / hot rules may specify that program instructions should be distributed to cold fields before warm or hot fields. In this way, rules can be applied dynamically based on the state of system resources at the time of distribution, thereby avoiding customer service interruptions or failures.

[0060] At internal distribution operation 212, program instructions can enter the internal ring for ring-level verification. In various respects, ring-level verification refers to the requirement to distribute engineering work to the internal ring (or "Dogfood ring") before distributing it to the external ring (e.g., the "production ring"). For example, program instructions can be distributed to at least some internal users associated with the software provider in the first-level ring before being distributed to external users associated with the customer (e.g., in the second-level and subsequent-level rings).

[0061] At external distribution operation 214, program instructions can enter the outer ring for verification and be distributed according to the determined rules. In some aspects, external users within the second ring level can be specifically selected by the software provider to provide feedback on the execution of program instructions and / or engineering tasks. In other aspects, program instructions are distributed in the second ring level (and subsequent ring levels) based on rules generated at rule determination operation 210, including throttling rules, cooling-off rules, and timing rules. For example, throttling rules define a subset (or portion) of the range to which program instructions can be executed at each stage (or iteration) of distribution (e.g., percentage of VMs at the virtual machine (VM) level, percentage of cross-regional clusters at the cluster level, percentage of databases at the database level, etc.). Cooling-off rules define the time period (e.g., hours, days, weeks, etc.) between stages (or iterations) of distributing the set of program instructions. Thus, within the second ring level, program instructions can be distributed to a first subset of the range and then distributed to a second subset of the range after the cooling-off period. Additionally, program instructions can be distributed based on timing rules (e.g., during peak or off-peak times) and sequencing rules (e.g., distributing to the colder parts of the range before the warmer or hotter parts, and based on the priority of associated engineering tasks in the distribution sequence). In various aspects, telemetry data can be received and processed (e.g., by telemetry data processor 122) during cooling-off periods between iterations to ensure that program instructions are executed properly and / or engineering tasks are executed as expected before distributing program instructions in the next iteration. In some cases, if an anomaly is identified during the cooling-off period, the distribution of program instructions can be stopped, and the program instructions can be updated accordingly (e.g., by an engineer or developer). In other cases, external distribution operation 214 may include reproducing the distribution of program instructions to portions of the range based on throttling rules. The reproduction rate can be determined based on cooling rules.

[0062] At determination operation 216, it can be determined whether the distribution of program instructions has been completed. For example, it can be determined whether the program instructions have been executed relative to the scope selected by the engineer. That is, whether the program instructions distributed to each of the multiple parts of the scope in multiple iterations have completed the distribution of program instructions for the entire scope. If the distribution is complete, the method can proceed to end operation 228. If the distribution is not complete, the method can proceed to external distribution operation 214 or selection operation 203. For example, if the distribution is not complete and the program instructions are updated (e.g., due to an anomaly identified during the cooldown period after the iteration), the method can proceed to selection operation 203 (refer to "2"). Thereafter, at selection operation 203, the updated program instructions can be selected for distribution. Alternatively, if the distribution is not complete and the program instructions are not updated, the method can proceed to external distribution operation 214 (refer to "1"). In this case, the next iteration of distribution based on throttling, cooldown, and timing rules can be initiated.

[0063] At application operation 218, coverage criteria can be applied to determine whether an engineering job should be automatically approved for immediate distribution. For example, if a procedure instruction code is determined at determination operation 204 for an engineering job addressing a critical customer fault (e.g., SEV 0), criteria for covering (or bypassing) the rules can be evaluated at application operation 218. For example, at application operation 218, the role of the engineer selecting the procedure instruction for distribution and / or the status of the customer associated with the critical customer fault can be evaluated. It should be recognized that additional or alternative coverage criteria can be evaluated at application operation 218.

[0064] At determination operation 220, it can be determined whether at least one (or a threshold number) of the coverage criteria has been met. For example, if the engineer is a standby engineer (“OCE”), at least one coverage criterion can be met. If the customer associated with a critical customer failure is associated with a special status (e.g., watchlist, strategic tenant), and / or the customer has a threshold number of active external users (e.g., 1000), at least one coverage criterion can be met. In some cases, once at least one of the coverage criteria is met, the method can proceed to automatic coverage operation 224. In other cases, a threshold number (or all) of the coverage criteria must be met to proceed to automatic coverage operation 224. If at least one (or a threshold number) of the coverage criteria is not met, the method can proceed to manual determination operation 222.

[0065] At manual determination operation 222, manual approval to bypass the rule can be obtained when at least one (or a threshold number) of the coverage criteria fails to be met. For example, manual approval can be obtained from a duty manager or other authorized personnel. If manual approval is obtained, the method can proceed to manual coverage operation 226. If manual approval is not obtained, the method can proceed to attribute determination operation 206, where the distribution of program instructions is based on the application of the rule.

[0066] At automatic coverage operation 224, once automatically approved, program instructions can be distributed substantially immediately across the scope to resolve critical customer issues. That is, program instructions can bypass the application of rules (e.g., ring-level verification, throttling, cooling, and timing) during distribution, enabling engineers (e.g., OCEs) to quickly resolve critical customer issues. In this way, the time to resolve customer problems (TTR) is minimized, thereby improving customer satisfaction. Simultaneously, internal activities involving code distribution that bypass rules can be prevented based on compliance with coverage criteria. After distributing program instructions substantially immediately across the scope to resolve critical customer issues, the method can proceed to termination operation 228.

[0067] At manual coverage operation 224, following manual approval, program instructions can be distributed substantially immediately across the scope to resolve critical customer issues. That is, program instructions can bypass the application of rules (e.g., ring-level verification, throttling, cooling, and timing) during distribution, enabling engineers (e.g., OCEs) to quickly resolve critical customer issues. In this way, the time to resolve customer problems (TTR) is minimized, thereby improving customer satisfaction. Simultaneously, internal activities involving code distribution that bypass rules can be prevented based on received manual approval. After distributing program instructions substantially immediately across the scope to resolve critical customer issues, the method proceeds to termination operation 228.

[0068] In each respect, the external distribution operation 214 constitutes an iteration of the external deployment of program instructions into at least one external verification ring (as indicated by "A"). The iteration of deployment within the ring includes one or more distributions of program instructions based on the throttling, cooling, and timing rules described above. Figure 3 A more detailed sequence of operations for external distribution operation 214 is described.

[0069] It should be appreciated that the purpose of describing operations 202-228 is to illustrate the methods and systems of the present invention, and is not intended to limit this disclosure to a specific sequence of steps. For example, the steps may be performed in a different order, additional steps may be performed, and the disclosed steps may be excluded without departing from the present invention.

[0070] Figure 3The illustration depicts an exemplary method for distributing program instructions in an outer ring level using a rule-based application according to various aspects of this disclosure. The general sequence of operations in method 300 is as follows: Figure 3 As shown in the figure. Typically, method 300 is illustrated for... Figure 2 The operation sequence of external dispatch operation 214. Method 300 may include more or fewer steps, or may differ from... Figure 3 The steps shown are arranged in order. Method 300 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 300 can be executed by gates or circuits associated with a processor, ASIC, FPGA, SOC, or other hardware device. References will then be made in conjunction with... Figure 1 , 2 Method 300 is explained by the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, and methods described in 4, 5, 6, 7, and 8A-B.

[0071] At external distribution operation 304, during the verification of distribution within the inner ring (e.g., the dogfood ring), program instructions may be distributed to the outer ring. For example, program instructions may be distributed to resources hosted by external users associated with customers in the outer ring (e.g., the production ring).

[0072] At rule distribution operation 306, program instructions can be distributed in the outer ring based on rules (e.g., rules generated at rule determination operation 210), including throttling rules, cooling rules, and timing rules(s). For example, throttling rules define the subset (or portion) of the scope to which program instructions can be executed in each stage (or iteration) of distribution (e.g., percentage of VMs in a VM-level scope, percentage of cross-regional clusters in a cluster-level scope, percentage of databases in a database-level scope, etc.). Cooling rules define the time periods (e.g., hours, days, weeks, etc.) between stages (or iterations) of distributing the set of program instructions. For example, at rule distribution operation 306, program instructions can be distributed to a first subset of the scope (and then to a second and subsequent subsets of the scope after one or more cooling periods). Additionally, program instructions can be distributed based on timing rules (e.g., during peak or off-peak times) and sequencing rules (e.g., distributing to a cold subset of the scope before a warm or hot subset, and in a distribution sequence position based on the priority of the associated engineering job).

[0073] At telemetry data reception operation 308, the telemetry data may be received and processed (e.g., by telemetry data processor 122) during the cooldown period between distribution iterations to ensure that program instructions are executed properly and / or engineering operations are performed as expected before the program instructions are distributed in the next iteration. For example, the telemetry data may include logs related to how end users access application services (e.g., click data, input queries and output results, success reports or error reports, etc.). In various aspects, the telemetry data may be stored in a telemetry database (e.g., telemetry database 130) and may be updated or used in conjunction with dynamic attributes stored in server usage database 126.

[0074] At anomaly determination operation 310, it can be determined whether one or more anomalies occurred during the execution of program instructions during the distribution iteration. An anomaly can refer to any error or exception that occurs during the execution of program instructions, or any error or exception associated with the execution of an associated engineering task. For example, telemetry data can be used to log errors or exceptions based on end-user interactions with the application service (e.g., click data, user requests, response times, etc.) after the execution of program instructions. If one or more anomalies are detected, the distribution of program instructions can be stopped, and the method can proceed to update operation 314. When one or more anomalies are detected, a notification can be sent to the OCE and / or the developers of the program instructions. If no one or more anomalies are detected, the method can proceed to cooldown determination operation 312.

[0075] At the cooling-off determination operation 312, it can be determined whether the cooling-off period has elapsed. The cooling-off period is the time interval (e.g., hours, days, weeks, etc.) required between stages (or iterations) of distributing the program instruction set. For example, the cooling-off period might be 6 hours for low-risk engineering tasks and 24 hours for high-risk engineering tasks. If the time since the program instruction distribution at the rule distribution operation 306 is greater than or equal to the cooling-off period, the method can proceed to the range determination operation 316. If the time since the program instruction distribution at the rule distribution operation 306 is less than the cooling-off period, the method can return to the telemetry data receiving operation 308.

[0076] At update operation 314, program instructions can be updated (e.g., by an engineer or developer) to resolve exceptions. For example, when one or more exceptions are detected during the execution of program instructions, a notification can be sent to the OCE and / or the developer of the program instructions. Updating the program instructions may involve evaluating script errors (e.g., debugging), revising the script to improve the performance of engineering tasks, etc. When updating the program instructions, the method can proceed to... Figure 2 Determine the attribute operation 206.

[0077] At scope determination operation 316, it can be determined whether program instructions have been executed for the scope selected by the engineer. That is, whether program instructions distributed to each subset of the scope in multiple iterations have resulted in execution of program instructions for the entire scope. If distribution for the entire scope has been completed, the method can proceed to... Figure 2 The termination operation is 228. If distribution for the entire range has not been completed, the method may return to rule distribution operation 306, where program instructions will be distributed in the next iteration based on the rules. In some cases, returning to rule distribution operation 306 may further involve entering the next outer validation loop.

[0078] It should be understood that the purpose of describing operations 302-316 is to illustrate the methods and systems of the present invention, and is not intended to limit this disclosure to a specific sequence of steps. For example, the steps may be performed in a different order, additional steps may be performed, and the disclosed steps may be excluded without departing from the present invention.

[0079] Figure 4 An example of a user script table according to various aspects of this disclosure is illustrated.

[0080] For example, Figure 4 The diagram illustrates an exemplary data structure for various parameters (or attributes) associated with program instructions (i.e., code or scripts) related to engineering operations. Attributes may be stored in an attribute / rule database (e.g., attribute / rule database 126). Each row of code list 400 (e.g., rows 1-5) corresponds to a program instruction set. Table entries are sorted based on the code identifier of the program instruction set. Code ID 402 indicates the code ID for each program instruction set (e.g., code IDs S001-S005). Code type 404 indicates the type of code, such as maintenance, configuration, customization, code repair, or code update. Risk level 406 specifies the risk level for deploying each program instruction set. Based on the likelihood of adverse effects on one or more customers during the execution of program instructions, the risk level can include low, medium, and high. Scope 408 specifies the logical partitioning of computing resources within the cloud computing environment where the program instructions are executed. Scope 408 may include, but is not limited to, "Virtual Server (VM)," "Physical Server," "Farmhouse," "Zone," "Network," "Certificate Server," "Database," and "Unknown." As shown in risk level column 406, the risk associated with the unknown range is high.

[0081] Update type 410 indicates the update functionality (or update feature) of a program instruction. In some cases, a program instruction may only include read instructions. In other cases, a program instruction may include read and write instructions. In each aspect, the update feature of the program instruction set affects the risk level. For program instructions that contain read instructions but not write instructions, the overall risk level of adverse impact on customers when executing the program instructions is lower because the program instructions do not modify data or code on cloud computing resources. Alternatively, program instructions that contain write instructions may modify at least some data or code on cloud computing resources, which could adversely affect customers using the same resources (e.g., within the deployment scope).

[0082] Tenant 412 indicates whether the deployment of a program instruction is for a single tenant or multiple tenants in the cloud. In each case, the program instruction may be related to tenant-specific customizations of an application service. In this case, the execution of the program instruction may adversely affect a single tenant (e.g., in an anomalous situation). Alternatively, program instructions maintained for a server farm cluster may affect multiple tenants hosted on that server farm cluster. In this case, any anomalies associated with the execution of the program instruction may adversely affect multiple tenants. The distribution of program instructions that may affect a single tenant may be associated with a lower level of risk compared to the distribution of program instructions that may affect multiple tenants.

[0083] As an example of a program instruction, the program instruction with code ID 402 "S005" represents a script used for engineering operations to modify (read and write) various attributes in databases and certificates for multiple tenants. The value of code type 404 is "maintenance", the risk level 406 is "high", the scope 408 is "database, certificate", the update type 410 is "read / write", and the tenant 412 is "multiple".

[0084] It should be recognized that, regarding Figure 4 The various methods, components, properties, etc., described herein are not intended to limit the system and methods to the specific properties described. Therefore, additional topology configurations can be used to practice the methods and systems described herein, and / or some of the described properties can be excluded without departing from the methods and systems disclosed herein.

[0085] Figure 5 An example of a data structure for a rule table in an exemplary system according to various aspects of this disclosure is illustrated.

[0086] Each row of Table 500 (e.g., rows 1-3) corresponds to a rule. Rule Table 500 includes Rule ID 502, Code ID 504, Code Type 506, Scope 508, Update Type 510, Throttling 512, Cooling 514, and Timing 516. Rule ID 502 indicates the identifier (e.g., R001) of the rule selected for application when a programming instruction set (e.g., S001) is distributed. Code ID 504 indicates the code ID used for each program instruction set (e.g., S001-S003). Code Type 506 indicates the type of job implemented by each programming instruction set, such as maintenance, problem fixing, configuration, or update. Scope 508 specifies the logical partitioning of computing resources within the cloud computing environment that executes the program instruction set, such as "Virtual Server (VM)," "Physical Server," "Farmhouse," "Zone," "Network," "Certificate Server," "Database," and "Unknown." Update Type 510 indicates the update functionality (or update feature) of the program instruction (e.g., "Read-only" or "Read-write"). In each aspect, code type 506, scope 508, and update type 510 identify the attributes of the program instruction. That is, rule R001 applies to cases where the code type of the program instruction is "problem fix", the scope is "server cluster", and the update characteristic is "read / write".

[0087] Throttling 512 indicates a subset (or portion) of the range of program instructions that can be executed in each phase (or iteration) of the distribution (e.g., 10% of the farms, 10% of the virtual servers, or 10% of the databases in each distribution). Cooling 514 indicates the period of time to wait before starting the next round of distribution (e.g., 6 hours or 24 hours). Timing 516 indicates the local time of day used to execute program instructions (e.g., "arbitrary time" and "off-peak"). In this respect, "arbitrary time" allows program instructions to execute during peak or off-peak times.

[0088] For example, a rule with rule ID 502 "R003" represents an exemplary rule applied to the distribution of a set of program instructions for a maintenance job, used to retrieve (e.g., read) attributes in a database and certificate server. In this case, code ID 504 is "S003", code type 506 is "maintenance", scope 508 is "database", update type 510 is "read", throttling 512 is "20% of database", cooling 514 is "6 hours", and timing 516 is "any time". Figure 5 In the example rules shown, rule R001 imposes relatively loose control over the deployment of program instructions. For example, the rule has relatively low throttling (e.g., high distribution) and a short cooldown period (e.g., 6 hours).

[0089] It should be recognized that, regarding Figure 5 The various methods, components, properties, etc., described are not intended to limit the system and methods to the specific properties described. Therefore, additional topology configurations can be used to practice the methods and systems described herein, and / or some of the properties described can be excluded without departing from the methods and systems disclosed herein.

[0090] Figure 6 An example of a distribution sequence list according to various aspects of this disclosure is illustrated.

[0091] An exemplary scheduling table 600 includes a distribution ID 602, a code ID 604, a rule ID 606, a ring level 608, a distribution sequence position 610, and a current state 612. Each row of the table may correspond to an iteration of the distribution of a program instruction set.

[0092] Distribution ID 602 ​​indicates the identifier of the iteration of distribution (e.g., "D001"). Code ID 604 indicates the identifier of the program instruction set being distributed (e.g., "S003"). For example, the program instruction set may be stored in code storage device 124. Rule ID 606 indicates the identifier of at least one rule to be applied during distribution (e.g., throttling, cooling, timing, etc.). Ring level 608 indicates the ring level of testing (e.g., "Dogfood" indicates the internal test ring, "Production 1" indicates the first production level distributed to the tenant). Distribution sequence position 610 indicates the position within the distribution sequence used to distribute program instructions S003 and S001 respectively (e.g., "second" or "first"). Current state 612 indicates the current state of the corresponding distribution. The current state 612 of distribution D001 is "baking 60% complete", while the current state 612 of distribution D002 is "activity 75% complete". The current state 612 indicates that the distribution process is in a "baking state" (e.g., distribution is paused during the cooling period) or an "active state" (e.g., distribution is active), and indicates the percentage of distribution completed (e.g., percentage of completion relative to the range).

[0093] It should be recognized that, regarding Figure 6 The various methods, components, properties, etc., described are not intended to limit the system and methods to the specific properties described. Therefore, additional topology configurations can be used to practice the methods and systems described herein, and / or some properties can be excluded without departing from the methods and systems disclosed herein.

[0094] Figure 7This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700 that can be used to practice the aspects of this disclosure. The computing device components described below may be adapted to the computing device described above. In a basic configuration, the computing device 700 may include at least one processing unit 702 and system memory 704. Depending on the configuration and type of the computing device, the system memory 704 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 704 may include an operating system 705 and one or more program tools 706 adapted to execute the aspects disclosed herein. For example, the operating system 705 may be adapted to control the operation of the computing device 700. Furthermore, the aspects of this disclosure may be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration in Figure 7 The components within the dashed line 708 are shown. The computing device 700 may have additional features or functions. For example, the computing device 700 may also include additional data storage devices (removable and / or non-removable), such as, for example, a disk, optical disk, or magnetic tape. Such additional storage devices... Figure 7 The image is shown using a removable storage device 709 and a non-removable storage device 710.

[0095] As mentioned above, many program tools and data files can be stored in system memory 704. When executed on processing unit 702, program tool 706 (e.g., code distribution application 720) can perform processes including, but not limited to, those described herein. Code distribution application 720 includes a risk level determiner 722, a rule generator 724, a ring validator 726, a coverage executor 728, a distributor 730, and a telemetry data processor 732, as described above. Figure 1 More detailed descriptions are available. Other program tools that may be used under various aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, presentation applications, drawing or computer-aided applications, etc.

[0096] Furthermore, aspects of this disclosure can be implemented in circuits including discrete electronic components, in packages or integrated electronic chips containing logic gates, in circuits utilizing microprocessors, or on a single chip containing electronic components or a microprocessor. For example, aspects of this disclosure can be implemented via a system-on-a-chip (SOC), wherein... Figure 7Each or more of the components illustrated herein may be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operated via the SOC, the capabilities described herein regarding the client switching protocol can be operated via application-specific logic integrated with other components of the computing device 700 onto a single integrated circuit (chip). Other technologies capable of performing logical operations may also be used to practice aspects of this disclosure, such as, for example, AND, OR, and NOT, including but not limited to: mechanical, optical, fluid, and quantum technologies. Furthermore, aspects of this disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0097] The computing device 700 may also have one or more input devices 712, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. It may also include one or more output devices 714, such as a monitor, speaker, printer, etc. The above devices are examples, and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of suitable communication connections 716 include, but are not limited to: radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0098] As used herein, the term "computer-readable medium" can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (e.g., computer-readable instructions, data structures, or program tools). System memory 704, removable storage device 709, and non-removable storage device 710 are examples of computer storage media (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other material that can be used to store information and is accessible by computing device 700. Any such computer storage medium may be part of computing device 700. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0099] Communication media can be embodied in computer-readable instructions, data structures, program tools, or other data in modulated data signals, such as carrier waves or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" can describe a signal that sets or alters one or more characteristics in a way that encodes information in the signal. By way of example, and not limitation, communication media can include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0100] Figure 8A and 8B The illustration depicts a computing device or mobile computing device 800, such as a mobile phone, smartphone, wearable computer (e.g., smartwatch), tablet computer, laptop computer, etc., which can be used to practice various aspects of this disclosure. In some aspects, the user (e.g., Figure 1 The client used by the end user (104) can be a mobile computing device. (See reference) Figure 8A The illustration depicts one aspect of a mobile computing device 800 used to implement these aspects. In a basic configuration, the mobile computing device 800 is a handheld computer with both input and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow the user to input information into the mobile computing device 800. The display 805 of the mobile computing device 800 can also be used as an input device (e.g., a touchscreen display). When included, an optional side input element 815 allows for further user input. The side input element 815 can be a rotary switch, a button, or any other type of manual input element. In an alternative aspect, the mobile computing device 800 can incorporate more or fewer input elements. For example, the display 805 may not be a touchscreen in some aspects. In yet another alternative aspect, the mobile computing device 800 is a portable telephone system, such as a cellular phone. The mobile computing device 800 may also include an optional keypad 835. The optional keypad 835 can be a physical keypad or a “soft” keypad generated on the touchscreen display. In various aspects, output elements include a display 805 for displaying a graphical user interface (GUI), a visual indicator 820 (e.g., a light-emitting diode), and / or an audio transducer 825 (e.g., a speaker). In some aspects, the mobile computing device 800 incorporates a vibration transducer for providing haptic feedback to the user. In yet another aspect, the mobile computing device 800 incorporates input and / or output ports, such as audio input and / or output ports for sending signals to or receiving signals from external devices, e.g., audio input (e.g., a microphone jack), audio output (e.g., a headphone jack), and video output (e.g., an HDMI port).

[0101] Figure 8BThis indicates computing devices, servers (e.g., Figure 1 This is a block diagram of an architecture for one aspect of a system (e.g., architecture) such as physical servers 154A-154D or virtual servers 160A-160B), or mobile computing devices. That is, computing device 800 can be incorporated into system 802 to implement certain aspects. System 802 can be implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, system 802 is integrated as a computing device, such as integrating a digital assistant (PDA) and a wireless phone.

[0102] One or more applications 866 may be loaded into memory 862 and run on or associated with operating system 864. Examples of applications include telephone dialers, email programs, PIM (Personal Information Management) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, etc. System 802 also includes a non-volatile storage area 868 within memory 862. The non-volatile storage area 868 can be used to store persistent information that should not be lost in the event of a power outage of system 802. Applications 866 may use and store information in the non-volatile storage area 868, such as email or other messages used by email applications. A synchronization application (not shown) also resides on system 802 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 868 synchronized with the corresponding information stored on the host computer. It should be appreciated that other applications may be loaded into memory 862 and run on the mobile computing device 800 described herein.

[0103] System 802 has a power supply 870, which can be implemented as one or more batteries. The power supply 870 may also include an external power source, such as an AC adapter or a power docking station for replenishing or recharging the batteries.

[0104] System 802 may also include a wireless interface layer 872 that performs functions for transmitting and receiving radio frequency communications. The wireless interface layer 872 facilitates wireless connectivity between system 802 and the "external world" via a communications operator or service provider. Transmissions to and from the wireless interface layer 872 are controlled by the operating system 864. In other words, communications received by the wireless interface layer 872 can be propagated to application 866 via the operating system 864, and vice versa.

[0105] A visual indicator 820 (e.g., an LED) can be used to provide visual notifications, and / or an audio interface 874 can be used to generate audible notifications via an audio transducer 825. In the illustrated configuration, the visual indicator 820 is a light-emitting diode (LED), and the audio transducer 825 is a speaker. These devices can be directly coupled to a power supply 870 so that when activated, they remain on for a duration specified by the notification mechanism, even if the processor 860 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 874 is used to provide and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 825, the audio interface 874 can also be coupled to a microphone to receive audible input, such as to facilitate telephone conversations. According to various aspects of this disclosure, the microphone can also be used as an audio sensor to facilitate control of notifications, as described below. System 802 may also include a video interface 876, which enables the operation of the onboard camera 830 to record still images, video streams, etc.

[0106] The mobile computing device 800 implementing system 802 may have additional features or functions. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 8B It is shown in the non-volatile memory region 868.

[0107] As described above, data / information generated or captured by mobile computing device 800 and stored via system 802 can be stored locally on mobile computing device 800, or the data can be stored on any number of storage media, which can be accessed by the device via wireless interface layer 872 or via a wired connection between mobile computing device 800 and a separate computing device associated with mobile computing device 800 (e.g., a server computer in a distributed computing network (e.g., the Internet)). It should be appreciated that such data / information can be accessed via mobile computing device 800 via wireless interface layer 872 or via a distributed computing network. Similarly, according to known data / information transmission and storage units, including email and collaborative data / information sharing systems, such data / information can be easily transferred between computing devices for storage and use.

[0108] The descriptions and illustrations of one or more aspects provided in this application are not intended to limit or constrain the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey the property and enable others to make and use the best mode of the claimed disclosure. The claimed disclosure should not be construed as limited to any aspect or detail provided, for example, in this application. Various features (structural and methodological) are intended to be selectively included or omitted, whether shown and described in combination or separately, to produce embodiments with a particular set of features. Having provided a description and illustration of this application, those skilled in the art can contemplate variations, modifications, and alternatives falling within the spirit of the broader aspects of the general inventive concept embodied in this application, without departing from the broader scope of the claimed disclosure.

[0109] In various aspects, a computer-implemented method for distributing program instructions is provided. The method includes: receiving selection of program instructions for distribution, wherein the program instructions, when executed, perform an engineering operation for a scope within a cloud computing environment. Additionally, the method includes: determining whether the engineering operation resolves a critical customer failure; and in response to determining that the engineering operation does not resolve a critical customer failure, determining one or more attributes associated with at least one of the program instructions or the scope. Based on the determined attributes, the method includes determining a risk level for executing the program instructions for the scope; and based on the risk level, the method includes determining one or more rules for managing the distribution of the program instructions. Furthermore, based on the one or more rules, the method includes distributing the program instructions to a subset of the scope.

[0110] In another aspect, a system is provided. The system includes: a processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the system to perform operations. The operations include: receiving selection of program instructions for distribution, wherein the program instructions, when executed, perform engineering operations for a scope of a cloud computing environment. Additionally, the operations include: determining one or more attributes associated with at least one of the program instructions or the scope; and, based on the determined attributes, determining a risk level for executing the program instructions for the scope. Based on the risk level, the operations include determining one or more rules for managing the distribution of the program instructions, wherein the one or more rules include at least throttling rules and cooling rules. Based on the throttling rules, the operations further include distributing the program instructions to a first subset of the scope; and, based on the cooling rules, waiting for a period of time. Based on the throttling rules, the operations include distributing the program instructions to a second subset of the scope.

[0111] In another aspect, a computer storage medium is provided. The computer storage medium stores computer-executable instructions that, when executed by a processor, cause a computer system to perform operations. The operations include: receiving selection of program instructions for distribution, wherein the program instructions, when executed, perform engineering operations for a scope of a cloud computing environment. The operations further include: determining one or more attributes associated with at least one of the program instructions or the scope; and, based on the determined attributes, determining a risk level for executing the program instructions for the scope. Based on the risk level, the operations include determining one or more rules for managing the distribution of the program instructions; and, based on the one or more rules, distributing the program instructions to a subset of the scope. During a time period specified by the one or more rules, the operations include receiving telemetry data regarding the distribution of the program instructions to the subset of the scope; and, based on the telemetry data, updating the program instructions.

[0112] Any one of the above-mentioned aspects can be combined with any other aspect of the above-mentioned aspects. Any one of the one or more aspects described herein.

Claims

1. A computer-implemented method for distributing program instructions, the method comprising: Receive selection of program instructions for distribution, wherein the program instructions, when executed, perform engineering operations for a scope of the cloud computing environment; Determine whether the engineering work resolved the critical customer's problem; In response to determining that the engineering operation does not resolve a critical customer failure, determine one or more attributes associated with at least one of the program instructions or the scope; Based on the identified attributes, determine the risk level of executing the program instructions for the defined scope; Based on the risk level, determine one or more rules for managing the distribution of the program instructions; and Based on one or more of the rules, the program instructions are distributed to a subset of the range.

2. The computer-implemented method according to claim 1, wherein, The scope includes the logical partitioning of resources within the cloud computing environment.

3. The computer-implemented method according to claim 2, wherein, The subset of the range includes a percentage of the range.

4. The computer-implemented method according to claim 1 further includes: In response to determining that the engineering work resolves a critical customer failure, one or more coverage criteria are evaluated; In response to determining that at least one coverage criterion is met, the one or more rules are automatically covered; as well as Distribute the program instructions for the scope without applying the one or more rules.

5. The computer-implemented method according to claim 1, further comprising: In response to determining that the engineering work resolves a critical customer failure, one or more coverage criteria are evaluated; In response to determining that the coverage criteria are not met, a request is made for manual approval of the one or more rules covering the coverage; and In response to receiving the human approval, the program instructions are distributed for the scope without applying the one or more rules.

6. The computer-implemented method according to claim 1, wherein, The one or more rules include one or more of the following: ring-level verification rules, throttling rules, cooling rules, or timing rules.

7. The computer-implemented method according to claim 6, wherein, The ring-level verification rule stipulates that the program instructions are distributed to the internal verification ring before being distributed to the external verification ring.

8. The computer-implemented method according to claim 6, wherein, The throttling rule specifies the subset of the range.

9. The computer-implemented method according to claim 6, wherein, The cooling rule specifies the time period between the sequential distribution of the program instructions.

10. The computer-implemented method according to claim 9, further comprising: Receive telemetry data regarding the distribution of programming instructions to the subset of the range during the said time period.

11. A system comprising: At least one processor; as well as At least one memory storing computer-executable instructions that, when executed by the at least one processor, cause the system to: Receive selection of program instructions for distribution, wherein the program instructions, when executed, perform engineering operations for a scope of the cloud computing environment; Determine one or more attributes associated with at least one of the program instructions or the scope; Based on the identified attributes, determine the risk level of executing the program instructions for the defined scope; Based on the risk level, one or more rules are determined for managing the distribution of the program instructions, wherein the one or more rules include at least throttling rules and cooling rules; Based on the throttling rules, the program instructions are distributed to a first subset of the range; Based on the aforementioned cooling rules, wait for a certain period of time; and Based on the throttling rules, the program instructions are distributed to a second subset of the range.

12. The system according to claim 11, wherein, The scope refers to the object types associated with the application service.

13. The system according to claim 12, wherein, The scope is defined based on a common strategy shared among objects.

14. The system of claim 11, wherein the computer-executable instructions, when executed, further cause the system to: Receive telemetry data regarding the distribution of programming instructions to the first subset of the range during the said time period.

15. A computer storage medium storing computer-executable instructions, which, when executed by a processor, cause a computer system to: Receive selection of program instructions for distribution, wherein, When executed, the program instructions perform engineering operations within the scope of the cloud computing environment. Determine one or more attributes associated with at least one of the program instructions or the scope; Based on the identified attributes, determine the risk level of executing the program instructions for the defined scope; Based on the risk level, determine one or more rules for managing the distribution of the program instructions; Based on one or more of the rules, the program instructions are distributed to a subset of the range; During the time period specified by the one or more rules, receive telemetry data regarding the distribution of the program instructions to the subset of the range; as well as The program instructions are updated based on the telemetry data.

Images