Selectively moderate the implementation of configuration changes across the enterprise

Abstract

Techniques are described herein that enable selective throttling of implementation of configuration changes in an enterprise. For example, incremental phases of implementation on respective subsets of clients are selectively throttled during respective time periods by performing operations on each subset during respective phases prior to respective next phases of implementation of the configuration change. The operations include gathering information about the phase from the clients in the subset; generating a score based at least in part on the information that indicates the extent to which metrics associated with the implementation were satisfied during the phase; comparing the score to a reference score for the subset to determine whether the score is within a tolerance associated with the reference score; and selectively throttling the next phase of implementation based at least in part on whether the score is within the tolerance.

Description

Technical Field

[0001] Embodiments of the present invention relate generally to the field of computers, and more particularly to systems, methods, and computer program products for selectively throttling implementation of configuration changes in an enterprise. Background Art

[0002] Information technology (IT) administrators of an enterprise often manage configuration changes (e.g., software deployments) to the enterprise all at once and do not notice failures or regressions associated with the configuration changes until end users report such failures or regressions to the administrators. Managing configuration changes in this manner is relatively inefficient for the administrators and can be costly to the enterprise to which the enterprise belongs due to lost productivity associated with failures or regressions.

[0003] When managing configuration changes, administrators often consider the following goals: keep downtime low; keep costs low; and improve the user experience for enterprise end users. However, after the implementation of the configuration change is complete, without A / B testing the configuration change, administrators may not be sure whether the actions taken to manage the configuration change were effective. Because the hierarchy of each enterprise may be completely different from that of other enterprises, it is difficult to know which actions to take to minimize costs while maximizing management efficiency. This ambiguity can lead to inefficient management of configuration changes.

[0004] Traditionally, if a regression occurs with respect to the implementation of a configuration change, the end user has a regressive experience. In many cases, performing the wrong configuration change provides less desirable results than doing nothing. Summary of the invention

[0005] Various methods are described herein, particularly for selectively moderating the implementation of configuration changes in an enterprise. For example, client devices in an enterprise may be divided into groups. The client devices in each group may be selected based on any of a variety of factors (e.g., to provide a statistically representative sample of the client devices as a whole in the enterprise). Configuration changes may be implemented on one group of client devices at a time. In each group, configuration changes may be implemented in stages on a corresponding subset of the client devices in the group. Information about this stage of implementation may be used to determine (e.g., automatically determine) whether the next stage of implementation is to be moderated. For example, if this stage of implementation has a negative impact on the user experience of end users associated with the client devices in the subset, the next stage of implementation may be moderated (e.g., automatically correcting a problem that causes a degradation in the user experience or enabling an administrator to manually perform actions to attempt to correct the problem).

[0006] Systems management generally refers to the enterprise-wide management of distributed systems (e.g., computer systems). Some example tasks that may be performed via systems management include, but are not limited to, anti-manipulation management, anti-virus and anti-malware management, security management, storage management, capacity monitoring, server availability monitoring and measurement, monitoring of user activity, network capacity and utilization monitoring, hardware inventory, and software inventory and installation. Systems management often includes various functional components, including, but not limited to, data center infrastructure management, help desk management, network management, security information and event management, and configuration management. Configuration management typically systematically handles changes in a system to maintain the integrity of the system. Such changes may be implemented for beneficial purposes, including, but not limited to: modifying the capabilities of a system; improving the performance, reliability, and / or maintainability of a system; extending the life of a system; reducing the cost, risk, and / or liability of a system; and correcting (multiple) defects of a system.

[0007] In an example method, incremental phases of implementation of a configuration change on a respective subset of a plurality of client devices in an enterprise are selectively throttled (e.g., automatically selectively throttled) during respective consecutive time periods by performing operations on each subset during respective phases prior to respective next phases of implementation of the configuration change. The operations include gathering information about the phase of implementation from the client devices in the subset. The operations also include generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation was satisfied during the phase. The operations also include comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score. The operations also include selectively throttling the next phase of implementation based at least in part on whether the score is within a specified tolerance associated with the reference score.

[0008] Summary of the invention is provided to introduce a series of concepts that will be further described in the detailed description below in a simplified form. Summary of the invention is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In addition, it should be noted that the present invention is not limited to the specific embodiments described in the detailed description and / or other parts of this document. These embodiments are presented in this article only for illustrative purposes. Based on the teachings contained herein, other embodiments will be apparent to those skilled in the art (multiple). BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the invention and, together with the description, further serve to explain the principles involved and to enable one skilled in the relevant art(s) to make and use the disclosed technology.

[0010] Figure 1is a block diagram of an example selective moderation system according to an embodiment.

[0011] Figure 2 Depicted is a flow diagram of an example method for selectively throttling an incremental phase of implementation of a configuration change on respective subsets of a plurality of client devices in an enterprise, according to an embodiment.

[0012] Figure 3 is a block diagram of an exemplary computing system according to an embodiment.

[0013] Figure 4 is a diagram illustrating an example selection of a subset of client devices for implementing corresponding incremental phases of a configuration change, according to an embodiment.

[0014] Figure 5 and Figure 6 is a diagram illustrating execution of an example automatic regression analysis technique with respect to a subset of client devices included in a common group and corresponding to respective stages of implementing a configuration change, according to an embodiment.

[0015] Figure 7 Example pseudo code for calculating a score for one or more devices according to an embodiment is shown.

[0016] Figure 8 An example computer is depicted in which embodiments may be implemented.

[0017] Features and advantages of the disclosed technology will become more apparent from the detailed description set forth below when taken in conjunction with the accompanying drawings, in which like reference numerals identify corresponding elements throughout. In the drawings, like reference numerals generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference numeral. DETAILED DESCRIPTION

[0018] 1. Introduction

[0019] The following detailed description refers to the accompanying drawings which illustrate exemplary embodiments of the present invention. However, the scope of the present invention is not limited to these embodiments, but is defined by the appended claims. Therefore, embodiments other than the embodiments shown in the accompanying drawings, such as modified versions of the illustrated embodiments, may still be covered by the present invention.

[0020] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, whether or not explicitly described, a person skilled in the relevant art will know that such feature, structure, or characteristic may be implemented in conjunction with other embodiments.

[0021] Descriptors such as "first," "second," "third," etc., are used to reference some elements discussed herein. Such descriptors are used to facilitate discussion of example embodiments and do not indicate a required order of the referenced elements unless an affirmative statement is made herein that such an order is required.

[0022] II. Example Embodiments

[0023] The example embodiments described herein can selectively moderate the implementation of configuration changes in an enterprise. For example, client devices in an enterprise can be divided into groups. The client devices in each group can be selected based on any of a variety of factors (e.g., to provide a statistically representative sample of the client devices in the enterprise as a whole). Configuration changes can be implemented on one group of client devices at a time. In each group, configuration changes can be implemented in stages on corresponding subsets of client devices in the group. Information about this stage of implementation can be used to determine whether the next stage of implementation is to be moderated. For example, if this stage of implementation has a negative impact on the user experience of end users associated with the client devices in the subset, the next stage of implementation may be restricted (e.g., automatically correcting problems that cause a degraded user experience or enabling administrators to manually perform operations to attempt to correct the problem).

[0024] Systems management generally refers to the enterprise-wide management of distributed systems (e.g., computer systems). Some example tasks that may be performed via systems management include, but are not limited to, anti-manipulation management, anti-virus and anti-malware management, security management, storage management, capacity monitoring, server availability monitoring and measurement, monitoring of user activity, network capacity and utilization monitoring, hardware inventory, and software inventory and installation. Systems management generally includes various functional components, including, but not limited to, data center infrastructure management, help desk management, network management, security information and event management, and configuration management. Configuration management generally systematically handles changes in a system to maintain the integrity of the system. Such changes may be implemented for beneficial purposes, including, but not limited to: modifying the capabilities of a system; improving the performance, reliability, and / or maintainability of a system; extending the life of a system; reducing the cost, risk, and / or liability of a system; and correcting (multiple) defects of a system.

[0025] The example techniques described herein have a variety of benefits compared to conventional techniques for implementing configuration changes in an enterprise. For example, the example techniques may be able to selectively throttle implementation to control the rate at which the implementation permeates through a group of client devices in an enterprise. The example techniques may be able to predict the impact (e.g., negative impact) that the implementation of a configuration change will have on (multiple) scores before the configuration change is implemented. The example techniques may determine the source of problems with the implementation of the configuration change, predict the performance degradation of client devices in the enterprise caused by the implementation, and solutions to address the problem. The example techniques may determine whether to implement the configuration change and / or notify the enterprise's information technology (IT) administrator of possible problems and / or solutions thereto based on the prediction.

[0026] Example techniques can reduce the amount of time and / or resources (e.g., processor cycles, memory, network bandwidth) consumed to implement configuration changes in an enterprise. For example, by employing the techniques described herein, the amount of time consumed to reach a compliance threshold can be reduced. A compliance threshold can be defined as a specified proportion of client devices in an enterprise on which a configuration change will be implemented within a specified time period.

[0027] Example techniques can improve the efficiency of IT administrators of enterprises. For example, example techniques can increase the amount of data-driven information provided to IT administrators about the implementation of configuration changes, which can increase the confidence of IT administrators in the implementation results. For example, reactive feedback can be provided to IT administrators to remind IT administrators of statistically significant reductions in the scores of client devices on which the implementation of configuration changes occurs. By increasing the amount of data-driven information provided to IT administrators and / or performing actions (e.g., in real time with the progress of implementation) to solve problems associated with implementation, example techniques can reduce the number of steps performed by IT administrators, the amount of work spent by IT administrators, and / or the amount of time spent by IT administrators to supervise the implementation of configuration changes in enterprises. Example techniques can reduce the costs associated with implementing configuration changes in enterprises. For example, by increasing the amount of data-driven information provided to IT administrators and / or performing actions to solve problems associated with implementation, IT administrators do not have to spend time reviewing the progress of implementation to determine potential sources of problems and / or potential actions performed to solve problems. Reducing the time spent by IT administrators to check the progress of implementation leads to reduced costs associated with IT administrators.

[0028] Example techniques can improve the efficiency of end users in an enterprise. Example techniques can mitigate (e.g., eliminate) productivity losses associated with implementation of configuration changes for end users. By mitigating productivity losses for end users, example techniques can reduce the cost of implementing configuration changes.

[0029] Example techniques may improve the efficiency of computing systems used by IT administrators to perform system management operations. For example, example techniques may detect failures or regressions during implementation of configuration changes more quickly than conventional techniques. According to this example, failures or regressions may be detected automatically (e.g., without input from an IT administrator). Example techniques may automatically resolve detected problems, automatically notify an IT administrator of the problem and / or a solution to the problem, etc. Example techniques may be able to resolve such problems in real time as implementation of the configuration change progresses from one stage of implementation to the next.

[0030] Regression analysis can be used to operationalize the implementation of configuration changes in the enterprise and determine statistically significant reductions in scores for client devices with and without the changes and take action or provide insight so that IT administrators can take the most appropriate action moving forward.

[0031] Rather than waiting for the entire implementation of a configuration change (e.g., software deployment) to complete to detect regressions, the example technology can provide reactive feedback to IT administrators to alert them of statistically significant reductions in scores for devices with or without changes. By creating groups for the implementation of configuration changes using automatic deployment rules (ADRs), the example technology can drive the implementation among a subset of client devices and monitor the impact of the implementation before the problem escalates. The example technology can determine what the best client devices are for the implementation and create subsets of various sizes to which the implementation is pushed.

[0032] To promote the effectiveness of the example techniques, it may be desirable to have good metrics about what is regressive and to have a good signal-to-noise ratio. The example techniques may use various regression metrics that are verified when the client device receives the implementation of its configuration change, and based on the response from the client device, determine efficient and effective actions to move forward. After the configuration change is implemented on each client device, various metrics are measured and they can be compiled together to form a score. If the deployment statistically significantly reduces the score of the client device, then the automatic implementation may be stopped. By linking these metrics with metrics analyzed by a third party (e.g., related to a snapshot of the client device), the example technology can pull metrics directly from the client device in real time to monitor the impact of the implementation on the client device. After stopping, the IT administrator (if the IT administrator chooses to do so) can resume the implementation of the configuration change-although analysis is provided, or troubleshooting information can be provided, which allows the IT administrator to make changes to the environment to improve the implementation of subsequent subsets of computing devices.

[0033] Because the model expands and learns from its client device selections and considers the impact that implementation of configuration changes will have on client devices in the enterprise, the example techniques can make predictions before implementing configuration changes to determine whether an administrator should move forward.

[0034] The example technology provides a variety of automated insights and benefits to help mitigate issues that arise during the implementation of a configuration change. When an administrator sets out to implement a configuration change, ADR is used to create statistically optimal groups of client machines to attempt implementation of the configuration change. Using these groups of client devices, the example technology can then drive a subset of client devices in each group through a phased implementation, closely monitoring the impact of the implementation at each phase. Using analytics and regression analysis via proprietary scores, the example technology can determine (a) whether implementation of the configuration change should be advanced in increasing phases and deployed to more client devices, or (b) whether to stop implementation and notify the administrator of the statistically optimal action to move forward. After taking action on the feedback provided to implement, the example technology can continue with a new phase and increase the score of the affected devices, learn from the response to the previous action, and leverage those learnings in future implementations. By providing a relatively high signal-to-noise ratio by learning from past actions, the example technology can provide a more improved user experience than traditional techniques to meet needs and issues that arise during implementation while minimizing end-user impact.

[0035] Examples of data inputs for measuring regressions include, but are not limited to, startup time, application crashes, central processing unit (CPU) usage, random access memory (RAM) usage, disk usage, network usage, error messages, third-party data (e.g., Desktop Analytics TM Data), computer hardware data, and software data. Desktop Analytics TM The software was developed and published by Microsoft Corporation.

[0036] The score can be a weighted average of regression measurements, third-party metrics, and environmental scaling to indicate how the implementation of a configuration change affects client devices. By leveraging the impact of configuration change implementations on past client devices, the model can provide predictions of implementation for other client devices in other subsets and future client devices of similar types. By categorizing the implementation of the configuration change, the areas affected, and the types of client devices, a score can be created to indicate whether an IT administrator should move forward with the implementation. By tracking which environmental, software, and / or hardware factors influence the implementation of a configuration change to result in a net positive score, the example technique can provide insight into actions that are relatively likely to improve the implementation (e.g., by reducing the number and / or severity of issues resulting therefrom) and make changes to the environment when necessary to have a statistically optimal impact on the environment.

[0037] To successfully perform regression analysis, a cohort of client devices is identified that can facilitate an accurate determination of whether the implementation of a configuration change results in a net positive or negative score. To mitigate the resulting noise, metrics are read, including inputs to measure regressions. Previous implementation successes, error messages, client health, device versions, and metrics gathered from third parties can be used to select client devices for the cohort. The cohort may evolve over time by learning from past implementations and subsequent subsets, and the analysis can rotate client devices in the cohort for future implementations. Each stage of the implementation may affect different client devices, with the goal of providing a robust test environment that reflects all client devices in the environment. To achieve this goal, a wide variety of client device types that accurately reflect the makeup of the environment are considered so that the cohort is an accurate representation of the environment, which can provide an accurate feedback loop as the implementation progresses through the various stages.

[0038] Figure 1 1 is a block diagram of an example selective moderation system 100 according to an embodiment. In general, the selective moderation system 100 operates to perform system configuration operations with respect to user devices (aka client devices) 102A-102M. Figure 1 As shown in , the selective moderation system 100 includes an enterprise 108 and (multiple) system management servers 110. The enterprise 108 includes user devices 102A-102M, a network 104, and a management system 106. Communications between the user devices 102A-102M, the management system 106, and (multiple) system management servers 110 are performed over the network 104 using well-known network communication protocols. The network 104 can be a wide area network (e.g., the Internet), a local area network (LAN), another type of network, or a combination thereof.

[0039] The user devices 102A-102M are processing systems capable of communicating with the management system 106. An example of a processing system is a system including at least one processor capable of manipulating data according to an instruction set. For example, the processing system may be a computer, a personal digital assistant, or the like. The user devices 102A-102M are configured to provide a request to a server (not shown) for requesting information stored on the server (or otherwise accessible via the server). For example, a user may initiate a request for executing a computer program (e.g., an application) using a client (e.g., a web browser or other type of client) deployed on a user device owned by the user or otherwise accessible. According to some example embodiments, the user devices 102A-102M are capable of accessing a domain (e.g., a website) hosted by a server so that the user devices 102A-102M may access information available via the domain. For example, such a domain may include a web page, which may be provided as a hypertext markup language (HTML) document and objects (e.g., files) linked therein.

[0040] Each of the user devices 102A-102M may include any client-enabled system or device, including but not limited to a desktop computer, a laptop computer, a tablet computer, a wearable computer such as a smart watch or head-mounted computer, a personal digital assistant, a cellular phone, an Internet of Things (IoT) device, etc. It will be appreciated that any one or more of the user devices 102A-102M may communicate with any one or more of the servers.

[0041] The user devices 102A-102M are also configured to provide information to the management system 106, either automatically or in response to queries from the management system 106. For example, such information may relate to hardware and / or software configurations of the user devices 102A-102M (e.g., updates thereof), workloads of the user devices 102A-102M, resource utilization on the user devices, etc.

[0042] The management system 106 is a processing system capable of communicating with the user devices 102A-102M. The management system 106 is configured to perform operations that facilitate system management of the enterprise 108 in response to instructions received from an IT administrator of the enterprise 108. For example, the management system 106 may provide commands to the system management server(s) 110 indicating actions to be performed by the system management server(s) 110 for performing system management of the enterprise 108. In another example, the management system 106 may provide a request for information to the user devices 102A-102M. The IT administrator may make decisions regarding system management of the user devices 102A-102M and / or the network 104 based at least in part on the information received from the user devices 102A-102M. It will be appreciated that at least some (e.g., all) of the information may be collected by the system management server(s) 110 instead of by the management system 106.

[0043] The system management server(s) 110 are processing systems capable of communicating with the enterprise 108. For example, the system management server(s) 110 may be capable of communicating with the management system 106 and / or the user devices 102A-102M in the enterprise 108. The system management server(s) 110 include selective moderation logic 112. The selective moderation logic 112 is configured to provide information to the management server 106 and / or the user devices 102A-102M and / or collect information from the management server 106 and / or the user devices 102A-102M for performing system management of the enterprise 108. For example, the selective moderation logic 112 may push information to the user devices 102A-102M or provide information in response to a request received from the user devices 102A-102M. The request may be generated by the user or without user intervention. For example, the policy applied to the user device may be completed without an explicit user request. According to this example, the policy is applied in the background even if the user is not logged into the user device. Also according to this example, the user device (e.g., the agent thereon) can poll the server for policies based on a schedule (e.g., once an hour) or based on an event (e.g., device wakeup, user unlocking, etc.). Also according to this example, the server can push policies to the user device (e.g., the agent thereon) via an open HTTP endpoint.

[0044] For non-limiting illustrative purposes, the system management server(s) 110 are shown as being external to the enterprise 108. For example, a system management service provider may own or control the system management server(s) 110 and host system management services that are provided by the selective moderation logic 112 to the entity that owns and / or controls the management system 106. According to this example, the entity may subscribe to the system management service. It will be appreciated that the system management server(s) 110 may be included in the enterprise 108. For example, the system management server(s) 110 may be located in the environment of the entity that owns and / or controls the management system 106.

[0045] According to example embodiments described herein, the selective moderation logic 112 selectively moderates incremental phases of implementation of a configuration change on a respective subset of user devices 102A-102M during respective consecutive time periods by performing operations on each subset during respective phases prior to respective next phases of implementation of the configuration change. The operations include gathering information about the phase of implementation from user devices in the subset. The operations also include generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation was satisfied during the phase. The operations also include comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score. The operations also include selectively moderate the next phase of implementation based at least in part on whether the score is within a specified tolerance associated with the reference score.

[0046] It will be appreciated that at least some of the above functions may be implemented by the management system 106, rather than by the selective moderation logic 112. For example, if the administrator does not want to provide information to the system management server(s) 110 and instead wishes to receive information only from the system management server(s) 110, the management system 106 may gather information about the phase of implementation from the user devices in the subset. The management system 106 may generate a score based at least in part on the information. The management system 106 may compare the score to a reference score for the subset to determine whether the score is within a specified tolerance, or the management system 106 may provide the score to the system management server(s) 110 for comparison with the reference score for the subset. If the management system 106 compares the score to the reference score for the subset, the management system 106 may provide a tolerance indicator to the system management server(s) 110, indicating whether the score is within a specified tolerance.

[0047] The selective moderation logic 112 can be implemented in various ways to provide recommendations for actions in the enterprise, including in hardware, software, firmware, or any combination thereof. For example, at least a portion of the selective moderation logic 112 can be implemented as a computer program code configured to be executed in one or more processors. In another example, at least a portion of the selective moderation logic 112 can be implemented as hardware logic / circuitry. For example, at least a portion of the selective moderation logic 112 can be implemented in a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a complex programmable logic device (CPLD), etc. Each SoC may include an integrated circuit chip that includes one or more of the following: a processor (e.g., a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or other circuits and / or embedded firmware for performing its functions.

[0048] For illustrative purposes and not intended to be limiting, the selective throttling logic 112 is shown as being incorporated in the system management server(s) 110. It will be appreciated that the selective throttling logic 112 (or any portion thereof) may be incorporated in any one or more of the user devices 102A-102M and / or the management system 106. For example, the client-side aspects of the selective throttling logic 112 may be incorporated in any one or more of the user devices 102A-102M and / or the management system 106, and the server-side aspects of the selective throttling logic 112 may be incorporated in the system management server(s) 110.

[0049] Figure 2 A flowchart 200 is depicted of an example method for selectively throttling incremental phases of implementation of configuration changes on respective subsets of a plurality of client devices (e.g., end-user devices) in an enterprise, according to an embodiment. Examples of implementation of configuration changes include, but are not limited to, deployment of software and enabling of disk encryption. For example, flowchart 200 may be performed by system management server(s) 110, management system 106, or a combination thereof, such as Figure 1 For illustrative purposes, flow chart 200 is about Figure 3 , which may be an example implementation of system management server(s) 110, management system 106, or a combination thereof. Figure 3As shown in , computing system 300 includes selective throttling logic 302. Selective throttling logic 302 includes client selection logic 304, scoring logic 306, comparison logic 308, stage throttling logic 310, and question logic 312. Based on the discussion regarding flowchart 200, additional structural and operational embodiments will be apparent to those skilled in the relevant art(s).

[0050] like Figure 2 As shown in FIG. 2 , the method of flowchart 200 begins at step 202. In step 202, a specified phase of implementation is performed. In an example implementation, phase throttling logic 310 performs a specified phase of implementation.

[0051] At step 204, information about the specified phase of implementation is collected from the client devices in the subset. Examples of information include, but are not limited to:

[0052] -System boot time (including core boot time);

[0053] - Login time;

[0054] - Application launch time (e.g., the time from when the application is launched to when the user is able to interact with the application);

[0055] - the total duration and / or number of times the processor utilization exceeded the threshold;

[0056] - the total duration and / or number of times the memory usage exceeds the threshold;

[0057] - the total duration and / or number of memory I / O operations exceeding the threshold;

[0058] -The total duration and / or number of times network I / O operations exceed the threshold;

[0059] - Storage utilization;

[0060] - User device utilization (e.g., the number of hours a user's device is turned on and the user interacts with the device);

[0061] -Battery utilization and battery warning history;

[0062] - Health indicators: frequency and number of unexpected reboots, expected battery life, expected hard drive life, frequency and number of driver crashes, frequency and number of app crashes.

[0063] Any of the example values ​​mentioned above may be averages and / or may be calculated by stage (i.e., per stage). For example, any of the example values ​​mentioned above may be calculated based on (e.g., only) measurements performed during a specified stage of implementation. Any part of the information may be anonymous or non-anonymous. Anonymous information (i.e., anonymous information) cannot identify the client device from which the anonymous information was collected. Therefore, the anonymous information may not be traced back to the client device from which the anonymous information was collected. Non-anonymous information (i.e., non-anonymous information) identifies the client device from which the non-anonymous information was collected. Therefore, the non-anonymous information may be able to be traced back to the client device from which the non-anonymous information was collected. In an example implementation, the scoring logic 306 collects information 318, which includes information about the specified stage of implementation.

[0064] To measure the boot time of a client device, boot tracking can be enabled and events can be logged as soon as the operating system (OS) of the client device can do so. Logs are recorded for each stage, and these logs can be used to measure the boot performance of the client device from power on to the appearance of the login screen. Core boot time is measured as a relatively consistent performance indicator. Core boot time includes booting the operating system and loading services. In addition, the time between login and execution of the first application to be executed on the client device is measured and collected. Parse the event log to determine the boot time specific to the hardware of the client device, and therefore does not include delays introduced by user interaction. In general, the core boot time is not affected by other device customizations, such as installed applications and enterprise IT policy enforcement. However, later boot stages are often affected by device customization. Extracting policies (for example, when displaying a user consent screen before allowing the user to access the client device) may involve network delays and are therefore generally not a good measure of the inherent performance of the client device. In addition, operating system and application updates may drive latency to vary greatly, and therefore may not be an accurate indicator of device characterization. Therefore, by performing log parsing, some factors outside of hardware control can be ignored from consideration when generating scores as described below with reference to step 206. Login time (ie, the time from when the user enters the password until the user can launch the application) is also collected.

[0065] The application is also detected to determine the startup time. The startup time of the application is the time from when the user selects the icon that initiates the execution of the application (for example, by tapping or clicking with a pointing device) to when the application is ready for use. In some examples, this is effectively done by detecting that a new application window has been opened and then determining how long ago the process that caused the new application window started. Since users usually only notice the foreground application that interacts with the user, in some examples, the application startup time is not measured for some or all background processes. Information related to the battery of the client device can indicate (for example, specify) the number of times a low battery warning is displayed (battery warning history), which indicates whether the battery capacity is sufficient for the user. Information related to the battery can indicate how long the user can use the client device between charges. Information about CPU and memory peaks exceeding thresholds is also collected because users may notice such events. In some examples, thresholds of 75% or 80% are used for processor utilization and memory utilization. In some examples, storage I / O performance and network I / O performance (such as latency) are collected. This data collected is indicative of specific hardware performance, but when combined or compared with enterprise data, the collected data can be used to determine whether the performance is acceptable, or whether improvements are recommended and / or required.

[0066] At step 206, a score (also referred to as a user score or user experience score) is generated based at least in part on the information, the score indicating the extent to which the metrics associated with the implementation were satisfied during the specified phase. In an example implementation, scoring logic 306 generates score 326 based at least in part on information 318, indicating the extent to which the metrics associated with the implementation were satisfied during the specified phase.

[0067] In an example embodiment, a score may be generated by performing a weighted calculation (e.g., a weighted combination) on information about a specified phase of implementation. For example, the score may be a weighted average of measurements associated with the corresponding metric. A machine learning (ML) algorithm may be performed to determine weights to be applied to portions of information associated with the corresponding metric based on actual feedback (e.g., key inputs and feedback) in the training data, such as by matching whether a set of scores matches reported human perceptions of whether certain computing devices are operating at acceptable performance levels. In some examples, rule sets are used instead of ML.

[0068] A metric is a threshold value related to the performance of a client device. Examples of metrics include, but are not limited to, maximum boot time; maximum login time; maximum application launch time; maximum duration that processor or memory utilization exceeds a threshold; maximum number of times that processor or memory utilization exceeds a threshold; maximum duration that memory or network I / O operations exceed a threshold; maximum number of times that memory or network I / O operations exceed a threshold; maximum storage utilization; maximum user device utilization; maximum battery utilization; maximum number of battery warnings provided by a client device; maximum frequency or number of unexpected restarts; minimum expected battery life or hard drive life; maximum frequency or number of driver crashes; and maximum frequency or number of application crashes. In some examples, the thresholds for processor utilization, memory utilization, storage device utilization, user device utilization, and / or battery utilization may be 70%, 75%, 80%, or 85%.

[0069] At step 208, the score is compared to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score. For example, the specified tolerance can be a range between a first value and a second value. According to this example, the first value is equal to the reference score minus the specified value, and the second value is equal to the reference score plus the specified value.

[0070] In an example implementation, the comparison logic 308 compares the score 326 to the reference score 320 to determine whether the score 326 is within a specified tolerance associated with the reference score 326. The score 320 being within the specified tolerance indicates that the next stage of implementation is not throttled. The score not being within the specified tolerance indicates that the next stage of implementation will be restricted. The comparison logic 308 may generate a tolerance indicator 328 to indicate whether the score 326 is within a specified tolerance associated with the reference score 320. For example, the tolerance indicator 328 may indicate the extent to which the score 326 is greater than the reference score 320, the extent to which the score 326 is less than the reference score 320, the extent to which the score 326 is within the specified tolerance, the extent to which the score 326 exceeds the specified tolerance, etc.

[0071] At step 210, the next stage of implementation is selectively throttled based at least in part on whether the score is within a specified tolerance associated with the reference score. In an example implementation, the stage throttling logic 310 selectively throttles the next stage of implementation based at least in part on whether the score 326 is within a specified tolerance associated with the reference score 320. For example, the stage throttling logic 310 may selectively throttle the next stage of implementation in response to (based at least in part on) receiving the tolerance indicator 328. For example, the stage throttling logic 310 may selectively throttle the next stage of implementation by performing a selective throttling action 336. The selective throttling action 336 may be a throttled implementation or an unthrottled implementation. According to the example, the stage throttling logic 310 may perform the selective throttling action 336 by throttling the next stage of implementation based at least in part on the tolerance indicator 328 indicating that the score 326 is not within a specified tolerance associated with the reference score 320. For example, the stage throttling logic 310 may throttle the next stage of implementation by delaying the initiation of the next stage or a portion thereof. According to this example, throttling the next phase can reduce the rate at which implementation of the configuration change is performed. Also according to this example, phase throttling logic 310 can perform selective throttling action 336 by not throttling the next phase of implementation based at least in part on tolerance indicator 328 indicating that score 326 is within a specified tolerance associated with reference score 320.

[0072] At step 212, the "next stage" is identified as the "designated stage." In an example implementation, scoring logic 306 identifies the "next stage" as the "designated stage" so that steps 204, 206, 208, and 210 may be performed with respect to the next stage of implementation.

[0073] At step 214, a determination is made as to whether additional phase(s) are to be selectively throttled. If additional phase(s) are to be selectively throttled, the process returns to step 204. Otherwise, flowchart 200 ends. In an example implementation, scoring logic 306 determines whether additional phase(s) are to be selectively throttled. For example, scoring logic 306 may make a determination based at least in part on information 318. According to the example, information 318 may indicate which client devices have been implemented with respect to the configuration change. Also according to the example, if information 318 indicates that the configuration change has not been implemented on all of the plurality of client devices, scoring logic 306 may determine that at least one additional phase is to be selectively throttled. Also according to the example, if information 318 indicates that the configuration change has been implemented on all of the plurality of client devices, scoring logic 306 may determine that no additional phase is to be selectively throttled.

[0074] In an example embodiment, the score indicates a user experience of a first end user resulting from implementing a configuration change on a client device in the subset. According to this embodiment, the first end user is associated with a client device in the subset. According to this embodiment, the reference score indicates a user experience of a second end user. According to this embodiment, the second end user is associated with a client device on which implementation of the configuration change has not yet occurred. For example, the reference score may indicate a user experience of the first end user before the configuration change was implemented on the client devices in the subset.

[0075] In another example embodiment, at step 210, the next phase of implementation is throttled based at least in part on the score not being within a specified tolerance associated with the reference score.

[0076] In yet another example embodiment, the score indicates the performance of the client devices in the subset. For example, the performance of the client devices in the subset may include the amount or proportion of resources (e.g., processor cycles or processing power, memory, network bandwidth) consumed by the client devices in the subset. According to this embodiment, the reference score indicates the performance of the client devices on which the implementation of the configuration change has not yet occurred. For example, the performance of the client devices on which the implementation of the configuration change has not yet occurred may include the amount or proportion of resources consumed by those client devices.

[0077] In yet another example embodiment, the score indicates the security posture (also referred to as the security state) of the client devices in the subset. The security posture indication of the client devices includes the security level of the hardware in the client devices, the software deployed on the client devices, the network to which the client devices are connected, and / or the information stored on the client devices. The security posture of the client devices may indicate the ability of the client devices to resist network security threats. For example, the above security posture may indicate the extent to which the client devices can predict, prevent and / or respond to network threats. The security posture of the client devices may indicate the extent to which the client devices pose a risk to the network security of the enterprise. According to this embodiment, the reference score indicates the security posture of the client devices on which the implementation of the configuration change has not yet occurred.

[0078] In some example embodiments, one or more steps 202, 204, 206, 208, 210, 212, and / or 214 of flowchart 200 may not be performed. Furthermore, steps in addition to or in place of steps 202, 204, 206, 208, 210, 212, and / or 214 may be performed. For example, in an example embodiment, the method of flowchart 200 further includes automatically selecting a group of client devices in the enterprise on which a corresponding incremental portion of the enterprise-wide implementation of a configuration change is to be implemented based at least in part on each group being a statistically representative sample of the entirety of client devices in the enterprise. It will be appreciated that in addition to or as a proxy for each group being a statistically representative sample of the entirety of client devices in the enterprise, a group may be automatically selected based on (a plurality of) other factors. In an example implementation, client selection logic 304 automatically selects a group of client devices. For example, the client selection logic 304 may analyze client device information 314, which includes information about client devices in an enterprise to determine which combinations of client devices constitute a statistically representative sample of the entirety of client devices in the enterprise. The client selection logic 304 may automatically select a group based on (e.g., at least in part based on) the analysis of the client device information 314. According to this embodiment, the difference between the attributes of each statistically representative sample and the attributes of the entirety of client devices in the enterprise is within a specified statistical tolerance. Also according to this embodiment, the first group includes a plurality of client devices. Also according to this embodiment, the first portion of the enterprise-wide implementation of the configuration change includes an incremental phase of implementation of the configuration change on respective subsets of the plurality of client devices.

[0079] In another example embodiment, the method of flowchart 200 also includes automatically selecting a group of client devices in an enterprise on which a corresponding incremental portion of an enterprise-wide implementation of a configuration change is to be implemented, based at least in part on the client devices in each group having a common attribute. For example, the client devices in each group may be located in the same geographic area (e.g., the same city, state, or building); from the same manufacturer; have the same model; have the same software configuration; have the same hardware configuration, and so on. In an example implementation, the client selection logic 304 automatically selects a group of client devices. For example, the client selection logic 304 may analyze the client device information 314, which includes information about the client devices in the enterprise to determine which combinations of client devices have at least one common attribute. The client selection logic 304 may automatically select a group based on the analysis of the client device information 314. According to this embodiment, the first group includes a plurality of client devices. Also according to this embodiment, the first portion of the enterprise-wide implementation of the configuration change includes an incremental phase of the implementation of the configuration change on a corresponding subset of the plurality of client devices.

[0080] In yet another example embodiment, the method of flowchart 200 also includes analyzing historical data to determine the amount of time that an end user has historically authorized the implementation of a configuration change on its client device in response to receiving a request to authorize implementation. In an example implementation, the client selection logic 304 analyzes historical data 316 to determine the amount of time that an end user has historically authorized the implementation of a configuration change on its client device. For example, the client selection logic 304 can monitor the time period between the time instance when each request is provided to each client device and the time instance when the corresponding client device authorizes the implementation requested. The client selection logic 304 can store a log of these time periods in the historical data 316. According to this embodiment, the method of flowchart 200 also includes: automatically selecting a client device group in an enterprise, where a corresponding incremental portion of the enterprise-wide implementation of the configuration change is to be implemented on the client device group, based at least in part on the amount of time that an end user has historically authorized the implementation of a configuration change on its client device in response to receiving a request to authorize implementation. In one aspect of this embodiment, client devices of end users that have historically taken a relatively small amount of time to authorize an implementation may be automatically selected to be included in initial group(s), where incremental portions of the enterprise-wide implementation corresponding to the initial group(s) will be executed prior to other incremental portions of the enterprise-wide implementation.

[0081] For illustrative purposes and not intended to be limiting, client device groups are described above as being automatically selected. It will be appreciated that an administrator of an enterprise can manually select one or more (e.g., all) client device groups in the enterprise on which respective incremental portions of an enterprise-wide implementation are to be implemented.

[0082] In yet another example embodiment, the method of flowchart 200 also includes determining that an external factor that is not a result of implementing a configuration change negatively affects the score. In an example implementation, scoring logic 306 determines that the external factor negatively affects the score. For example, scoring logic 306 may perform a statistical analysis on a variety of factors, including the external factor, to determine the extent to which each factor contributes to the score. Scoring logic 306 may determine that the external factor negatively affects the score based on the statistical analysis. According to this embodiment, comparing the score to a reference score at step 208 includes determining that the score is not within a specified tolerance associated with the reference score. Also according to this embodiment, selectively throttling the next stage of implementation at step 210 includes not throttling the next stage of implementation based at least in part on the negative impact of the external factor on the score, even if the score is not within a specified tolerance associated with the reference score.

[0083] In one aspect of this embodiment, determining that the external factor negatively impacted the score includes determining the extent to which the external factor negatively impacted the score. According to this aspect, determining that the score is not within a specified tolerance associated with the reference score includes determining the extent to which the score exceeds the specified tolerance. Also according to this embodiment, the next stage of unmoderated implementation includes: based at least in part on the negative impact of the external factor on the score being greater than or equal to the extent to which the score exceeds the specified tolerance.

[0084] In another example embodiment, for each client device in the enterprise, a probability that the client device will be attacked by malware is determined. For example, the client selection logic 304 may determine, for each client device in the enterprise, the probability that the client device will be attacked by malware. According to this example, the client selection logic 304 may determine the probability of each client device by analyzing client device information 314, which may indicate the vulnerability of the client device to malware attacks, security software deployed on the client device, etc. The client selection logic 304 may generate an attack indicator 332 to indicate the probability that each device in the enterprise will be attacked by malware. Also according to this embodiment, selectively throttling the next stage of implementation at step 210 includes selectively throttling the next stage of implementation based at least in part on the probability that the corresponding client device in the subset will be attacked by malware. For example, a relatively high probability that the client device will be attacked by malware is conducive to throttling the next stage of implementation; however, a relatively low probability that the client device will be attacked by malware is not conducive to throttling the next stage of implementation. In another example, the implementation of the configuration change may be security software (e.g., developed and released by Microsoft Corporation). ) deployment or update. It will be appreciated that the next phase of implementation may be selectively throttled based on any combination of factors (ie, not just those described).

[0085] In yet another example embodiment, comparing the score to the reference score at step 208 includes determining that the score is not within a specified tolerance associated with the reference score. According to this embodiment, the method of flowchart 200 also includes identifying the source of the problem that caused the score to be not within the specified tolerance associated with the reference score. In an example implementation, scoring logic 306 identifies the source of the problem. For example, user logic 306 can identify the source of the problem by analyzing information 318. According to this implementation, scoring logic 306 can generate problem information 322 to indicate the source of the problem. According to this embodiment, the method of flowchart 200 also includes solving the problem in response to identifying the source of the problem. In an example implementation, problem logic 312 solves the problem. For example, problem logic 312 can solve the problem in response to receiving problem information 322 (e.g., based at least in part on problem information 322 indicating the source of the problem. In another example, problem logic 312 can solve the problem by modifying the configuration change so that the problem no longer occurs. According to this embodiment, selectively throttling the next stage of implementation at step 210 includes executing the next stage of implementation in response to the problem being solved.

[0086] In one aspect of this embodiment, the method of flowchart 200 also includes providing troubleshooting information to an administrator of the enterprise. The troubleshooting information indicates (multiple) steps to be performed to solve the problem. According to this aspect, solving the problem includes solving the problem by performing (multiple) steps based at least in part on instructions received from the administrator. The instructions trigger the execution of the (multiple) steps.

[0087] In yet another example embodiment, comparing the score to the reference score at step 208 includes determining that the score is not within a specified tolerance associated with the reference score. According to this embodiment, the method of flowchart 200 also includes notifying an administrator of the enterprise: based at least in part on the score not being within the specified tolerance associated with the reference score, the next stage of implementation of the configuration change may negatively impact the user experience of the users of the corresponding subset of computing devices. In an example implementation, stage moderation logic 310 notifies an administrator of the enterprise: based at least in part on the tolerance indicator 328 indicating that the score is not within the specified tolerance associated with the reference score 320, the next stage of implementation of the configuration change may negatively impact the user experience of the users of the corresponding subset of computing devices. It should be noted that despite being notified that the next stage of implementation of the configuration change may negatively impact the user experience of the users of the corresponding subset of computing devices, the administrator may still initiate execution of the configuration change.

[0088] It will be appreciated that computing system 300 may not include one or more of selective throttling logic 302, client selection logic 304, scoring logic 306, comparison logic 308, stage throttling logic 310, and / or question logic 312. Furthermore, computing system 300 may include components in addition to or in place of selective throttling logic 302, client selection logic 304, scoring logic 306, comparison logic 308, stage throttling logic 310, and / or question logic 312.

[0089] Figure 44 is a block diagram 400 illustrating an example selection of a subset of client devices for respective incremental stages of implementation of a configuration change according to an embodiment. When an administrator starts a software deployment, as illustrated by arrow 408, selective moderation logic 412 automatically determines a client device group 402 to which the implementation of the configuration change can be pushed in various stages. The selective moderation logic 412 determines a subset of client devices 410 for the respective stages of implementation. The subsets may be created by considering predictive user scores associated with the client devices 402 where applicable. For example, if regression occurs based on the initial implementation, the selective moderation logic 412 may repeatedly select a subset of client devices in the group in future iterations. The subset may include the most statistically ideal client devices and configurations for implementation. For example, the selective moderation logic 412 may analyze regression analysis metrics and historical trends 404 from client devices 402 and / or third-party data 406 to identify the most ideal client devices and configurations for implementation. If there are foreseeable problems, the selective moderation logic 412 may provide an indication of one or more subsets to the administrator.

[0090] Figure 5 and Figure 6 is a diagram illustrating execution of an example automatic regression analysis technique with respect to a subset (ring) of client devices included in a common group and corresponding to respective stages of implementing a configuration change, according to an embodiment. Figure 5 If a net negative score is created from the regression analysis, rather than proceeding with the deployment, the source of the problem can be detected, the predicted degradation can be determined, and the source and predicted degradation can be reported to the administrator so that the administrator can take steps to resolve the problem. Alternatively, the automated system can improve the implementation of the configuration change (e.g., by reducing the number and / or severity of issues associated with the implementation), iterate based on the results of the first implementation, and try the implemented modifications with a new subset of client devices. This may result in a statistically optimal experience for end users and administrators.

[0091] refer to Figure 6, a phase of implementation of the configuration change can be performed, and it can be determined whether the phase has experienced a regression in the scores of the affected devices. For example, the user scores can be calculated 604 before the updated software is deployed, the updated software can be deployed 606, and the user scores can be recalculated 608 after the deployment of the software update is completed to determine whether the phase has experienced a regression in the user scores. If there is no regression, the implant can be advanced to the next subset of client devices. Otherwise, moderation of the implementation may occur so that the implementation can be modified, and the implementation can be unmodified with the optimized changes to obtain a positive score. For example, when the software update is deployed 610 to the client device 602, the user scores associated with the corresponding client device 602 can be collected to determine 614 the aggregated weighted average user score among all target client devices. For example, the user scores associated with the corresponding client device 602 can be weighted and averaged to determine the aggregated weighted average user score. If no regression occurs and a positive user score is returned, the deployment can be continued 616 for the next client device ring. If regression occurs, the deployment can be stopped 618, and the deployment can be modified or the administrator can be notified if applicable.

[0092] Figure 7 An example pseudocode 700 for calculating (e.g., determining) a score for one or more devices (e.g., one or more user devices) according to an embodiment is shown. For example, the score may represent the user experience on a particular device. In some examples, the score ranges from 0 to 100, where 100 is the best experience and 0 is the worst experience. For example, if the boot time is higher than the average level (e.g., longer than the average boot time on a particular device), the user may feel that the experience is poor, and a boot time that is faster than the average level will provide the user with a better experience. Pseudocode 700 may rely on standardized metric data (e.g., when the collected information is converted to metric data, the "better" value is converted to a lower metric number). In some examples, the metric data includes instance data for a particular device (e.g., the core boot time is 5 seconds); a global mean (e.g., the average core boot time is 10 seconds); and a global standard deviation (e.g., the standard deviation of the boot time of all devices is 2.1 seconds). The weight applied to the metric data indicates the importance of the metric to the user's perception of the device experience. In some example calculations, the weights sum to 1.0. For example, for three metrics, the weights may be startup time = 0.1, application startup time = 0.4, and battery life = 0.5.

[0093] The specific calculation is shown in pseudo code 700 and repeated below. ue_score is the user score or user experience score.

[0094] ue_score = 0;

[0095] For each metric

[0096] {

[0097] z_score = min(5, max(-5, (instance - mean) / standard deviation));

[0098] metric_value = (-1*<z_score> +5 / 10);

[0099] ue_score+=metric_value*metric_weight;

[0100] }

[0101] Next metric;

[0102] return ue_score;

[0103] Some examples use alternative calculations of the score. For example, instead of a linear correlation between the z-score and the user experience (ue_score), an exponential relationship can be used (e.g., two times slower will make the score four times worse). In another example, an expectation of time (boot time or application startup time) can be used instead of a calculated mean. Using an expectation of time will lower the score of a computing device that takes longer to perform a task than the user expects, even if the specific computing device performs faster than the average. Other variations are also possible.

[0104] Although the operations of some disclosed methods are described in a specific sequence order for convenience of presentation, it should be understood that unless the specific language set forth herein requires a specific sequence, this description includes rearrangement. For example, the operations described in sequence may be rearranged or performed simultaneously in some cases. In addition, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods can be used in conjunction with other methods.

[0105] Any one or more of selective moderation logic 112, selective moderation logic 302, client selection logic 304, scoring logic 306, comparison logic 308, stage moderation logic 310, question logic 312, selective moderation logic 412, and / or flowchart 200 may be implemented in hardware, software, firmware, or any combination thereof.

[0106] For example, any one or more of the selective moderation logic 112, the selective moderation logic 302, the client selection logic 304, the scoring logic 306, the comparison logic 308, the stage moderation logic 310, the question logic 312, the selective moderation logic 412, and / or the flowchart 200 may be implemented at least in part as computer program code configured to be executed in one or more processors.

[0107] In another example, any one or more of the selective moderation logic 112, the selective moderation logic 302, the client selection logic 304, the scoring logic 306, the comparison logic 308, the stage moderation logic 310, the question logic 312, the selective moderation logic 412, and / or the flowchart 200 may be implemented, for example, at least in part, as hardware logic / circuitry. Such hardware logic / circuitry may include one or more hardware logic components. Examples of hardware logic components include, but are not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SoCs), complex programmable logic devices (CPLDs), and the like. For example, a SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or other circuits and / or embedded firmware for performing its functions.

[0108] III. Further Discussion of Some Example Embodiments

[0109] The example system includes a memory and one or more processors coupled to the memory. The one or more processors are configured to selectively throttle incremental phases of implementation of configuration changes on corresponding subsets of multiple client devices in an enterprise during corresponding continuous time periods by performing operations on each subset during corresponding phases before the corresponding next phase of implementation of the configuration changes. The operations include collecting information about the phase of implementation from the client devices in the subset. The operations also include generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation is satisfied during the phase. The operations also include comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score. The operations also include selectively throttling the next phase of implementation based at least in part on whether the score is within a specified tolerance associated with the reference score.

[0110] In a first aspect of an example system, one or more processors are configured to perform operations further comprising: automatically selecting a plurality of groups of client devices in an enterprise on which to implement a plurality of corresponding incremental portions of an enterprise-wide implementation of a configuration change based at least in part on each group being a statistically representative sample of the entirety of the client devices in the enterprise. A difference between an attribute of each statistically representative sample and an attribute of the entirety of the client devices in the enterprise is within a specified statistical tolerance. According to the first aspect, a first group of the plurality of groups includes a plurality of client devices. Also according to the first aspect, a first portion of the enterprise-wide implementation of the configuration change includes incremental phases of implementation of the configuration change on corresponding subsets of the plurality of client devices.

[0111] In a second aspect of the example system, one or more processors are configured to perform operations further comprising: automatically selecting a plurality of groups of client devices in an enterprise on which to implement a plurality of corresponding incremental portions of an enterprise-wide implementation of a configuration change based at least in part on client devices in each group having a common attribute. According to the second aspect, a first group of the plurality of groups includes a plurality of client devices. Also according to the second aspect, a first portion of the enterprise-wide implementation of the configuration change includes incremental phases of implementation of the configuration change on corresponding subsets of the plurality of client devices. The second aspect of the example system can be implemented in conjunction with the first aspect of the example system, but the example embodiments are not limited in this respect.

[0112] In a third aspect of the example system, one or more processors are configured to perform operations further including: analyzing historical data to determine the amount of time that end users have historically taken to authorize implementation of configuration changes on their client devices in response to receiving requests to authorize implementation. According to the third aspect, one or more processors are configured to perform operations further including: automatically selecting multiple client device groups in an enterprise on which multiple corresponding incremental portions of an enterprise-wide implementation of the configuration change are to be implemented based at least in part on the amount of time that end users have historically taken to authorize implementation of configuration changes on their client devices in response to receiving requests to authorize implementation. The third aspect of the example system can be implemented in conjunction with the first and / or second aspects of the example system, but the example embodiments are not limited in this respect.

[0113] In a fourth aspect of the example system, the score indicates a user experience of a first plurality of end users resulting from implementation of the configuration change on client devices in the subset. According to the fourth aspect, the first plurality of end users are associated with client devices in the subset. Further according to the fourth aspect, the reference score indicates a user experience of a second plurality of end users. Further according to the third aspect, the second plurality of end users are associated with client devices on which implementation of the configuration change has not yet occurred. The fourth aspect of the example system can be implemented in combination with the first, second, and / or third aspects of the example system, but the example embodiments are not limited in this respect.

[0114] In an implementation of the fourth aspect of the example system, the reference score indicates a user experience of the first plurality of end users before the configuration change is implemented on the client devices in the subset.

[0115] In a fifth aspect of the example system, the score indicates performance of the client devices in the subset. According to the fifth aspect, the reference score indicates performance of the client devices on which implementation of the configuration change has not yet occurred. The fifth aspect of the example system can be implemented in conjunction with the first, second, third and / or fourth aspects of the example system, but the example embodiments are not limited in this respect.

[0116] In a sixth aspect of the example system, the score indicates a security posture of the client devices in the subset. According to the sixth aspect, the reference score indicates a security posture of the client devices on which implementation of the configuration change has not yet occurred. The sixth aspect of the example system can be implemented in conjunction with the first, second, third, fourth and / or fifth aspects of the example system, although the example embodiments are not limited in this respect.

[0117] In a seventh aspect of the example system, the one or more processors are configured to perform operations including: throttling a next stage of implementation based at least in part on the score not being within a specified tolerance associated with the reference score. The seventh aspect of the example system may be implemented in conjunction with the first, second, third, fourth, fifth, and / or sixth aspects of the example system, although the example embodiments are not limited in this respect.

[0118] In an eighth aspect of the example system, one or more processors are configured to perform operations including determining that an external factor that is not a result of implementing a configuration change has a negative impact on the score. According to the eighth aspect, one or more processors are configured to perform operations including determining that the score is not within a specified tolerance associated with a reference score. Further according to the eighth aspect, one or more processors are configured to perform operations including not throttling the next stage of implementation based at least in part on the negative impact of the external factor on the score even if the score is not within a specified tolerance associated with the reference score. The eighth aspect of the example system can be implemented in combination with the first, second, third, fourth, fifth, sixth and / or seventh aspects of the example system, but the example embodiments are not limited in this respect.

[0119] In a ninth aspect of the example system, one or more processors are configured to perform operations including: selectively throttling incremental phases of deploying software on respective subsets of client devices. The ninth aspect of the example system may be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, and / or eighth aspects of the example system, although example embodiments are not limited in this respect.

[0120] In a tenth aspect of the example system, one or more processors are configured to perform operations including: selectively throttling enabling incremental phases of disk encryption on respective subsets of client devices. The tenth aspect of the example system may be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, and / or ninth aspects of the example system, although the example embodiments are not limited in this respect.

[0121] In an eleventh aspect of the example system, one or more processors are configured to perform operations including: for each client device in the enterprise, determine a likelihood that the client device will be attacked by malware. According to the eleventh aspect, the one or more processors are configured to perform operations including: further selectively throttle a next stage of implementation based at least in part on the likelihood that the corresponding client devices in the subset will be attacked by malware. The eleventh aspect of the example system can be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, and / or tenth aspects of the example system, although the example embodiments are not limited in this respect.

[0122] In a twelfth aspect of the example system, one or more processors are configured to perform operations including: determining that a score is not within a specified tolerance associated with a reference score; identifying a source of a problem that caused the score to be not within a specified tolerance associated with the reference score; resolving the problem in response to identifying the source of the problem; and performing a next stage of implementation in response to the problem being resolved. The twelfth aspect of the example system can be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, and / or eleventh aspects of the example system, but the example embodiments are not limited in this respect.

[0123] In a thirteenth aspect of the example system, one or more processors are configured to perform operations further including: determining that the score is not within a specified tolerance associated with the reference score. According to the thirteenth aspect, the one or more processors are configured to perform operations further including: notifying an administrator of the enterprise that a next phase of implementation of the configuration change may negatively impact the user experience of users of a corresponding subset of computing devices based at least in part on the score not being within a specified tolerance associated with the reference score. The thirteenth aspect of the example system can be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh, and / or twelfth aspect of the example system, but the example embodiments are not limited in this respect.

[0124] In an example method, incremental phases of implementation of a configuration change on a corresponding subset of a plurality of client devices in an enterprise are selectively throttled during corresponding consecutive time periods by performing an operation on each subset during a corresponding phase prior to a corresponding next phase of implementation of the configuration change. The operation includes gathering information about the phase of implementation from the client devices in the subset. The operation also includes generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation was satisfied during the phase. The operation also includes comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score. The operation also includes selectively throttling the next phase of implementation based at least in part on whether the score is within a specified tolerance associated with the reference score.

[0125] In a first aspect of the example method, the example method further comprises: automatically selecting a plurality of groups of client devices in an enterprise on which to implement a plurality of corresponding incremental portions of an enterprise-wide implementation of a configuration change based at least in part on each group being a statistically representative sample of the client devices in the enterprise as a whole. A difference between an attribute of each statistically representative sample and an attribute of the client devices in the enterprise as a whole is within a specified statistical tolerance. According to the first aspect, a first group of the plurality of groups comprises a plurality of client devices. Also according to the first aspect, a first portion of the enterprise-wide implementation of the configuration change comprises an incremental phase of implementation of the configuration change on corresponding subsets of the plurality of client devices.

[0126] In a second aspect of the example method, the example method further comprises automatically selecting a plurality of groups of client devices in an enterprise on which to implement a plurality of corresponding incremental portions of an enterprise-wide implementation of a configuration change based at least in part on client devices in each group having common attributes. According to the second aspect, a first group of the plurality of groups comprises a plurality of client devices. Also according to the second aspect, a first portion of the enterprise-wide implementation of the configuration change comprises incremental phases of implementation of the configuration change on corresponding subsets of the plurality of client devices. The second aspect of the example method may be implemented in conjunction with the first aspect of the example method, but the example embodiments are not limited in this respect.

[0127] In a third aspect of the example method, the example method further includes: analyzing historical data to determine the amount of time it has historically taken an end user to authorize implementation of a configuration change on their client device in response to receiving a request to authorize implementation. According to the third aspect, the example method further includes: automatically selecting a plurality of client device groups in an enterprise on which to implement a plurality of corresponding incremental portions of an enterprise-wide implementation of a configuration change based at least in part on the amount of time it has historically taken an end user to authorize implementation of a configuration change on their client device in response to receiving a request to authorize implementation. The third aspect of the example method may be implemented in conjunction with the first and / or second aspect of the example method, but the example embodiments are not limited in this respect.

[0128] In a fourth aspect of the example method, the score indicates a user experience of a first plurality of end users resulting from implementation of the configuration change on client devices in the subset. According to the fourth aspect, the first plurality of end users are associated with client devices in the subset. Also according to the fourth aspect, the reference score indicates a user experience of a second plurality of end users. Also according to the fourth aspect, the second plurality of end users are associated with client devices on which implementation of the configuration change has not yet occurred. The fourth aspect of the example method can be implemented in conjunction with the first, second, and / or third aspects of the example method, but the example embodiments are not limited in this respect.

[0129] In an implementation of the fourth aspect of the example method, the reference score indicates a user experience of the first plurality of end users before the configuration change is implemented on the client devices in the subset.

[0130] In a fifth aspect of the example method, the score indicates performance of the client devices in the subset. According to the fifth aspect, the reference score indicates performance of the client devices on which implementation of the configuration change has not yet occurred. The fifth aspect of the example method can be implemented in conjunction with the first, second, third and / or fourth aspects of the example method, but the example embodiments are not limited in this respect.

[0131] In a sixth aspect of the example method, the score indicates a security posture of the client devices in the subset. According to the sixth aspect, the reference score indicates a security posture of the client devices on which implementation of the configuration change has not yet occurred. The sixth aspect of the example method can be implemented in conjunction with the first, second, third, fourth and / or fifth aspects of the example method, but the example embodiments are not limited in this respect.

[0132] In a seventh aspect of the example method, selectively throttling incremental phases of implementation of the configuration change on the respective subset of client devices comprises throttling a next phase of implementation based at least in part on the score not being within a specified tolerance associated with the reference score. The seventh aspect of the example method may be implemented in conjunction with the first, second, third, fourth, fifth, and / or sixth aspect of the example method, although the example embodiments are not limited in this respect.

[0133] In an eighth aspect of the example method, the example method further includes determining that an external factor that is not a result of implementing a configuration change has a negative impact on the score. According to the eighth aspect, comparing the score to a reference score for the subset includes determining that the score is not within a specified tolerance associated with the reference score. Also according to the eighth aspect, selectively throttling a next stage of implementation includes not throttling a next stage of implementation based at least in part on the negative impact of the external factor on the score even if the score is not within a specified tolerance associated with the reference score. The eighth aspect of the example method can be implemented in combination with the first, second, third, fourth, fifth, sixth and / or seventh aspects of the example method, but the example embodiments are not limited in this respect.

[0134] In a ninth aspect of the example method, selectively throttling incremental phases of implementation of configuration changes on the respective subset of client devices comprises: selectively throttling incremental phases of deploying software on the respective subset of client devices. The ninth aspect of the example method may be implemented in combination with the first, second, third, fourth, fifth, sixth, seventh, and / or eighth aspects of the example method, but the example embodiments are not limited in this respect.

[0135] In a tenth aspect of the example method, selectively throttling incremental phases of implementation of the configuration change on the respective subset of client devices comprises: selectively throttling incremental phases of enabling disk encryption on the respective subset of client devices. The tenth aspect of the example method may be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, and / or ninth aspect of the example method, although the example embodiments are not limited in this respect.

[0136] In an eleventh aspect of the example method, the example method further comprises, for each client device in the enterprise, determining a likelihood that the client device will be attacked by malware. According to the eleventh aspect, selectively throttling the next stage of implementation comprises selectively throttling the next stage of implementation based additionally at least in part on the likelihood that the corresponding client devices in the subset will be attacked by malware. The eleventh aspect of the example method may be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, and / or tenth aspect of the example method, although the example embodiments are not limited in this respect.

[0137] In a twelfth aspect of the example method, comparing the score to a reference score for the subset includes: determining that the score is not within a specified tolerance associated with the reference score. According to the twelfth aspect, the example method also includes: identifying a source of a problem that caused the score to be not within a specified tolerance associated with the reference score; and resolving the problem in response to identifying the source of the problem. Also according to the twelfth aspect, selectively throttling a next stage of implementation includes: executing the next stage of implementation in response to the problem being resolved. The twelfth aspect of the example method can be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth and / or eleventh aspects of the example method, but the example embodiments are not limited in this respect.

[0138] In a thirteenth aspect of the example method, comparing the score to a reference score for the subset includes: determining that the score is not within a specified tolerance associated with the reference score. According to the thirteenth aspect, the example method also includes: notifying an administrator of the enterprise: based at least in part on the score not being within a specified tolerance associated with the reference score, a next phase of implementation of the configuration change may negatively impact user experience for users of the corresponding subset of computing devices. The thirteenth aspect of the example method can be implemented in conjunction with the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh and / or twelfth aspect of the example method, but the example embodiments are not limited in this respect.

[0139] An example computer program product includes a computer-readable storage medium having recorded thereon instructions for enabling a processor-based system to perform operations. The operations include selectively throttling incremental phases of implementation of a configuration change on a respective subset of a plurality of client devices in an enterprise during respective consecutive time periods by performing actions on each subset during respective phases prior to respective next phases of implementation of the configuration change. The actions include collecting information about the phase of implementation from the client devices in the subset; generating a score based at least in part on the information, the score indicating the extent to which metrics associated with the implementation were satisfied during the phase; comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score; and selectively throttling the next phase of implementation based at least in part on whether the score is within a specified tolerance associated with the reference score.

[0140] 4. Example Computer System

[0141] Figure 8 An example computer 800 is depicted in which embodiments may be implemented. Figure 1 Any one or more of the user devices 102A-102M, the management system 106, and / or any one or more of the system management servers 110 shown in FIG. 1 ; and / or Figure 3 The computing system 300 shown in the figure can be implemented using a computer 800, including one or more features and / or alternative features of the computer 800. For example, the computer 800 can be a general-purpose computing device in the form of a conventional personal computer, a mobile computer, or a workstation, or the computer 800 can be a special-purpose computing device. The description of the computer 800 provided herein is provided for illustrative purposes and is not intended to be limiting. As known to those skilled in the relevant art(s), the embodiments can be implemented in other types of computer systems.

[0142] like Figure 8 As shown in FIG. 8 , computer 800 includes a processing unit 802, a system memory 804, and a bus 806 that couples various system components including the system memory 804 to the processing unit 802. Bus 806 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. System memory 804 includes read-only memory (ROM) 808 and random access memory (RAM) 810. Basic input / output system 812 (BIOS) is stored in ROM 808.

[0143] The computer 800 also has one or more of the following drives: a hard disk drive 814 for reading from and writing to a hard disk, a disk drive 816 for reading from or writing to a removable disk 818, and an optical disk drive 820 for reading from or writing to a removable optical disk 822 (such as a CDROM, DVDROM, or other optical media). The hard disk drive 814, the disk drive 816, and the optical disk drive 820 are connected to the bus 806 via a hard disk drive interface 824, a disk drive interface 826, and an optical drive interface 828, respectively. The drives and their associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer. Although a hard disk, a removable disk, and a removable optical disk are described, other types of computer-readable storage media may also be used to store data, such as flash memory cards, digital video disks, random access memory (RAM), read-only memory (ROM), etc.

[0144] A plurality of program modules may be stored on a hard disk, a magnetic disk, an optical disk, a ROM, or a RAM. These programs include an operating system 830, one or more application programs 832, other program modules 834, and program data 836. Application programs 832 or program modules 834 may include, for example, any one or more of the selective moderation logic 112, the selective moderation logic 302, the client selection logic 304, the scoring logic 306, the comparison logic 308, the stage moderation logic 310, the question logic 312, and / or the flowchart 200 (including any steps of the flowchart 200), as described herein.

[0145] A user may enter commands and information into the computer 800 through input devices such as a keyboard 838 and a pointing device 840. Other input devices (not shown) may include a microphone, a joystick, a game pad, a satellite dish, a scanner, a touch screen, a camera, an accelerometer, a gyroscope, etc. These and other input devices are often connected to the processing unit 802 through a serial port interface 842 coupled to the bus 806, but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB).

[0146] A display device 844 (eg, a monitor) is also connected to bus 806 via an interface, such as a video adapter 846. In addition to display device 844, computer 800 may also include other peripheral output devices (not shown), such as speakers and printers.

[0147] The computer 800 is connected to a network 848 (eg, the Internet) through a network interface or adapter 850, a modem 852, or other means for establishing communications over the network. The modem 852 may be internal or external and is connected to the bus 806 via the serial port interface 842.

[0148] As used herein, the terms "computer program medium" and "computer readable storage medium" are generally used to refer to media (e.g., non-transitory media) such as a hard disk associated with hard disk drive 814, removable disk 818, removable optical disk 822, and other media such as flash memory cards, digital video disks, random access memory (RAM), read-only memory (ROM), etc. Computer-readable storage media are not signals such as carrier signals or propagating signals. For example, a computer-readable storage medium may not include a signal. Therefore, a computer-readable storage medium itself does not constitute a signal. Computer-readable storage media are distinct from and do not overlap with communication media (excluding communication media). Communication media embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave. The term "modulated data signal" means such a signal that has one or more of its characteristics set or changed in such a way that information is encoded in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared and other wireless media, as well as wired media. Example embodiments also relate to such communication media.

[0149] As noted above, computer programs and modules (including application programs 832 and other program modules 834) may be stored on a hard disk, disk, optical disk, ROM, or RAM. Such computer programs may also be received via the network interface 850 or the serial port interface 842. Such computer programs, when executed or loaded by an application, enable the computer 800 to implement the features of the embodiments discussed herein. Thus, such computer programs represent controllers of the computer 800.

[0150] Example embodiments also relate to a computer program product comprising software (e.g., computer-readable instructions) stored on any computer-usable medium. Such software, when executed in one or more data processing devices, causes (multiple) data processing devices to operate as described herein. Embodiments may employ any computer-usable or computer-readable medium known now or in the future. Examples of computer-readable media include, but are not limited to, storage devices such as RAM, hard drives, floppy disks, CD ROMs, DVD ROMs, zip disks, magnetic tapes, magnetic storage devices, optical storage devices, MEMS-based storage devices, nanotechnology-based storage devices, and the like.

[0151] It will be appreciated that the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.

[0152] V. Conclusion

[0153] Although the subject matter has been described in language specific to structural features and / or actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to fall within the scope of the claims.

Claims

1. A system comprising: Memory; and One or more processors coupled to the memory, the one or more processors configured to selectively throttle incremental phases of implementation of a configuration change on respective subsets of the plurality of client devices during respective consecutive time periods by performing operations on each subset during respective first phases prior to respective next phases of implementation of the configuration change, the operations comprising: collecting information about the first phase of the implementation from the client devices in the subset; generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation was satisfied during the first phase; comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score; and Selectively throttling the next phase of the implementation based at least in part on whether the score is within the specified tolerance associated with the reference score includes selectively delaying implementation of at least a portion of the configuration change implemented at the next phase.

2. The system of claim 1 , wherein the one or more processors are configured to perform operations further comprising: automatically selecting a plurality of groups of client devices in the enterprise on which to implement a plurality of respective incremental portions of the enterprise-wide implementation of the configuration change based at least in part on each group being a statistically representative sample of the population of client devices in the enterprise, wherein a difference between an attribute of each statistically representative sample and an attribute of the population of client devices in the enterprise is within a specified statistical tolerance; wherein a first group of the plurality of groups includes the plurality of client devices; as well as Wherein a first portion of the enterprise-wide implementation of the configuration change comprises: the incremental phase of the implementation of the configuration change on the respective subset of the plurality of client devices.

3. The system of claim 1 , wherein the one or more processors are configured to perform operations further comprising: automatically selecting a plurality of groups of client devices in an enterprise on which to implement a plurality of respective incremental portions of an enterprise-wide implementation of the configuration change based at least in part on the client devices in each group having a common attribute; wherein a first group of the plurality of groups includes the plurality of client devices; as well as Wherein a first portion of the enterprise-wide implementation of the configuration change comprises: the incremental phase of the implementation of the configuration change on the respective subset of the plurality of client devices.

4. The system of claim 1 , wherein the one or more processors are configured to perform operations further comprising: analyzing historical data to determine an amount of time that end users have historically taken to authorize implementation of configuration changes on their client devices in response to receiving a request to authorize said implementation; and Automatically select multiple client device groups in an enterprise on which multiple corresponding incremental portions of an enterprise-wide implementation of the configuration change are to be implemented based at least in part on the amount of time it has historically taken end users to authorize implementation of the configuration change on their client devices in response to receiving a request to authorize the implementation.

5. The system of claim 1 , wherein the scores indicate a user experience startup time for a first plurality of end users, the user experience resulting from implementing the configuration change on the client devices in the subset; wherein said first plurality of end users are associated with said client devices in said subset; wherein the reference score indicates a user experience startup time for a second plurality of end users; as well as Wherein said second plurality of end-users are associated with said client devices on which said implementation of said configuration change has not yet occurred. 6 . The system of claim 5 , wherein the reference score indicates a user experience startup time for the first plurality of end users before the configuration change is implemented on the client devices in the subset.

7. The system of claim 1, wherein the score indicates a first security posture of the client devices in the subset, the first security posture indicating at least one of: the ability of the client devices in the subset to resist network security threats, or the extent to which the client devices in the subset pose a risk to the network security of the enterprise; and wherein the reference score indicates a second security posture of the client device on which the implementation of the configuration change has not yet occurred, the first security posture indicating at least one of: the ability of the client device to defend against network security threats on which implementation of the configuration change has not yet occurred, or The extent to which the client device on which implementation of the configuration change has not yet occurred poses a risk to the enterprise's network security.

8. The system of claim 1, wherein the one or more processors are configured to perform operations comprising: The next phase of the implementation is throttled based at least in part on the score not being within the specified tolerance associated with the reference score.

9. The system of claim 1, wherein the one or more processors are configured to perform operations comprising: determining that factors that were not a result of implementing said configuration change negatively impacted said score; determining that the score is not within the specified tolerance associated with the reference score; and The next phase of the implementing is not throttled based at least in part on negatively affecting the score even if the score is not within the specified tolerance associated with the reference score.

10. The system of claim 1, wherein the one or more processors are configured to perform operations comprising: For each client device, determining a likelihood that the client device will be attacked by malware; and The next phase of the implementing is also selectively throttled based at least in part on the likelihood that the respective client devices in the subset will be attacked by malware.

11. A method comprising: Incremental phases of implementation of a configuration change on respective subsets of the plurality of client devices are selectively throttled during respective consecutive time periods by performing operations on each subset during respective first phases prior to respective next phases of implementation of the configuration change, the operations comprising: collecting information about the first phase of the implementation from the client devices in the subset; generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation was satisfied during the first phase; comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score; and Selectively throttling the next phase of the implementation based at least in part on whether the score is within the specified tolerance associated with the reference score includes selectively delaying implementation of at least a portion of the configuration change implemented at the next phase.

12. The method according to claim 11, further comprising: automatically selecting a plurality of groups of client devices in the enterprise on which to implement a plurality of respective incremental portions of an enterprise-wide implementation of the configuration change based at least in part on each group being a statistically representative sample of the population of client devices in the enterprise, wherein a difference between an attribute of each statistically representative sample and an attribute of the population of client devices in the enterprise is within a specified statistical tolerance; wherein a first group of the plurality of groups includes the plurality of client devices; as well as Wherein a first portion of the enterprise-wide implementation of the configuration change comprises: the incremental phase of the implementation of the configuration change on the respective subset of the plurality of client devices.

13. The method according to claim 11, further comprising: automatically selecting a plurality of groups of client devices in an enterprise on which to implement a plurality of respective incremental portions of an enterprise-wide implementation of the configuration change based at least in part on the client devices in each group having a common attribute; wherein a first group of the plurality of groups includes the plurality of client devices; as well as Wherein a first portion of the enterprise-wide implementation of the configuration change comprises: the incremental phase of the implementation of the configuration change on the respective subset of the plurality of client devices.

14. The method of claim 11, wherein the scores indicate performance of the client devices in the subset; and Wherein the reference score indicates performance of a client device on which the implementation of the configuration change has not yet occurred.

15. The method of claim 11, wherein selectively throttling the incremental phase of the implementation of the configuration change on the respective subset of the client devices comprises: The next phase of the implementation is throttled based in part on the score not being within the specified tolerance associated with the reference score.

16. The method of claim 11, wherein selectively throttling the incremental phase of the implementation of the configuration change on the respective subset of the client devices comprises: Incremental phases of deployment of software on the respective subset of the client devices are selectively throttled.

17. The method of claim 11, wherein selectively throttling the incremental phase of the implementation of the configuration change on the respective subset of the client devices comprises: Enabling incremental phases of disk encryption on the corresponding subset of the client devices is selectively throttled.

18. The method of claim 11, wherein comparing the score to the reference score for the subset comprises: determining that the score is not within the specified tolerance associated with the reference score; The method further comprises: identifying a source of a problem causing the score to be outside the specified tolerance associated with the reference score; solving the problem in response to identifying the source of the problem; Wherein the next stage of selectively moderating the implementation comprises: The next phase of the implementation is performed in response to the problem being resolved.

19. The method of claim 11, wherein comparing the score to the reference score for the subset comprises: determining that the score is not within the specified tolerance associated with the reference score; The method further comprises: Based at least in part on the score not being within the specified tolerance associated with the reference score, notifying an administrator of the enterprise that the next phase of the implementation of the configuration change may negatively impact the experience of users of the respective subset of computing devices.

20. A computer program product comprising a computer readable storage medium, the computer program product being not a signal, the computer readable storage medium having recorded thereon instructions for enabling a processor-based system to perform operations comprising: Incremental phases of implementation of a configuration change on respective subsets of the plurality of client devices in throttling during respective consecutive time periods by performing actions on each subset during respective first phases prior to respective next phases of implementation of the configuration change, the actions comprising: collecting information about the first phase of the implementation from the client devices in the subset; generating a score based at least in part on the information, the score indicating the extent to which a metric associated with the implementation was satisfied during the first phase; comparing the score to a reference score for the subset to determine whether the score is within a specified tolerance associated with the reference score; and Based at least in part on whether the score is within the specified tolerance associated with the reference score, the next phase of the implementation is selectively throttled by selectively delaying initiation of at least a portion of the configuration change implemented at the next phase.

Images