Recommended changes for administrators in your enterprise

Abstract

Described herein is a technique capable of providing recommendations for administrator changes in an enterprise (i.e., administrator change recommendations). A type of intended administrator change that an administrator is about to perform on the enterprise is determined. The type is cross-referenced with information indicating administrator changes performed by (one or more) administrators in (one or more) environments of (one or more) enterprises and values ​​of metrics generated thereby to identify a subset of the information corresponding to the type. A causal relationship is inferred between (one or more) administrator changes performed subsequent to the type of administrator change and an increase in (one or more) values ​​of (one or more) metrics indicated by the information in the subset. Based at least in part on the causal relationship between the recommended administrator change and an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics, a recommendation is made that the recommended administrator change be performed by the administrator.

Description

Background Art

[0001] Organizations often spend a lot of money on enterprise management tools to maintain the health and security of devices in their enterprise. While organizations may have employees adopt enterprise management tools to oversee devices in the company's enterprise, this may not be the employee's primary responsibility or even their top priority. As a result, due to lack of training, information and / or time, information technology (IT) administrators may not be able to manage their devices in an efficient manner. This inability does not allow the company to fully utilize enterprise tools and can lead to performance gaps and security risks, ultimately affecting the company's daily functions.

[0002] Administrators can be provided with training, customer support channels, and customer open source solutions to familiarize themselves with and use specific enterprise management tools. While these platforms can be helpful, customers may not be able to access some information due to lack of resources. Likewise, inexperienced administrators may place too much trust in these channels, which may result in introducing new aspects to their device administrator techniques that may not be necessary in their specific situation. Summary of the invention

[0003] Various methods for providing administrator change recommendations (i.e., administrator change recommendations) in an enterprise are described herein. An administrator change is a change to an enterprise configuration. For example, an administrator change recommendation can be provided via an enterprise management tool, which is used by an administrator to perform system management of an enterprise. System management generally refers to an enterprise-wide administrator of a distributed system (e.g., a computer system). Some example tasks that can be performed via system management include, but are not limited to: anti-manipulation management, anti-virus and anti-malware management, security management, storage management, capacity monitoring, server availability monitoring and measurement, user activity monitoring, network capacity and utilization monitoring, hardware inventory, and software inventory and installation. System management generally includes a variety of functional components, including, but not limited to, data center infrastructure management, help desk management, network management, security information and event management, and configuration management. Configuration management generally systematically handles changes in the system to maintain the integrity of the system. Such changes can be implemented for beneficial purposes, including, but not limited to, the ability to modify the system; improving the performance, reliability, and / or maintainability of the system; extending the life of the system; reducing the cost, risk, and / or liability of the system; and correcting (one or more) defects of the system.

[0004] In a first example method, information is collected that indicates administrator changes made by an administrator in an environment of an enterprise and further indicates values ​​of metrics caused by the administrator changes. The information is analyzed and classified into categories based at least in part on (one or more) characteristics of the environment in which the administrator changes were made. A type of expected administrator change that the administrator is about to perform on the enterprise is determined based at least in part on an operation performed by the administrator using an enterprise management tool. The type of expected administrator change is cross-referenced with the information to identify a subset of information corresponding to the type. By analyzing the subset, a causal relationship between (one or more) administrator changes made after an administrator change of the type indicated by the information in the subset and an increase in (one or more) values ​​of (one or more) corresponding metrics indicated by the information in the subset is inferred based at least in part on cross-referencing the type of expected administrator change with the information in the subset. Based at least in part on the one or more categories into which at least one of the subsets is classified and further based at least in part on the causal relationship between the recommended administrator change and an increase in at least one of (one or more) values ​​of at least one of (one or more) corresponding metrics, it is recommended that the recommended administrator change be performed by the administrator, the information in at least one of the subsets indicating that the recommended administrator change has been made after an administrator change of the type.

[0005] In a second example method, a type of expected administrator change that an administrator is about to perform on an enterprise is determined based at least in part on an operation performed by the administrator using an enterprise management tool. The type of expected administrator change is cross-referenced with information indicating an administrator change made by an administrator in a corresponding environment of the corresponding enterprise and further indicating a value of a metric produced by the administrator change to identify a subset of information corresponding to the type. By analyzing the subset, a causal relationship between (one or more) administrator changes made after an administrator change of the type indicated by the information in the subset and an increase in (one or more) values ​​of (one or more) corresponding metrics indicated by the information in the subset is inferred based at least in part on cross-referencing the type of expected administrator change and the information in the subset. Based at least in part on the one or more categories into which at least one of the subsets is classified and further at least in part on the causal relationship between the recommended administrator change and an increase in at least one of (one or more) values ​​of at least one of (one or more) corresponding metrics, it is recommended that the recommended administrator change be performed by the administrator, the information in at least one of the subsets indicating that the recommended administrator change has been made after the type of administrator change. Based at least in part on (one or more) characteristics of the environment in which the administrator change is made, the information is classified into categories including one or more categories.

[0006] The present invention is provided to introduce some concepts that will be further described in the following specific embodiments in a simplified form. The present invention is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. In addition, it should be noted that the present invention is not limited to the specific embodiments described in the specific embodiments and / or other parts of this article. These embodiments provided herein are for illustrative purposes only. Based on the teachings contained herein, other embodiments will be apparent to those skilled in the art (one or more). BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, further serve to explain the principles involved and to enable one or more persons skilled in the relevant art(s) to make and use the disclosed technology.

[0008] Figure 1 is a block diagram of an example administrator change recommendation system according to an embodiment.

[0009] Figure 2 According to the embodiment Figure 1 A block diagram of an example implementation of an administrator change recommendation system is shown in FIG.

[0010] Figures 3 to 5 Depicted is a flow diagram of an example method for providing administrator change recommendations in accordance with an embodiment.

[0011] Figure 6 is a system diagram of an exemplary computing system according to an embodiment.

[0012] Figure 7 Example pseudo code for calculating user scores and user experience scores according to an embodiment is shown.

[0013] Figure 8 An example computer is depicted in which embodiments may be implemented.

[0014] Features and advantages of the disclosed technology will become more apparent from the detailed description set forth below when taken in conjunction with the accompanying drawings, in which like reference numerals identify corresponding elements throughout. In the drawings, like reference numerals generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference numeral. DETAILED DESCRIPTION

[0015] I. Introduction

[0016] The following detailed description refers to the accompanying drawings showing exemplary embodiments of the present invention. However, the scope of the present invention is not limited to these embodiments, but is defined by the appended claims. Therefore, embodiments other than the embodiments shown in the drawings, such as modified versions of the embodiments shown, may still be covered by the present invention.

[0017] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment does not necessarily include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of (one or more) persons skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0018] Descriptors such as "first," "second," "third," etc. are used to reference some elements discussed herein. Such descriptors are used to facilitate discussion of example embodiments and do not indicate a required order of the referenced elements unless such order is expressly stated herein.

[0019] II. Example Embodiments

[0020] The example embodiments described herein can provide recommendations for administrator changes in an enterprise (i.e., administrator change recommendations). Administrator changes are changes to an enterprise configuration. For example, administrator change recommendations can be provided via an enterprise management tool, which is used by an administrator to perform system management of an enterprise. System management generally refers to enterprise-wide administrators of distributed systems (e.g., computer systems). Some example tasks that can be performed via system management include, but are not limited to, anti-manipulation management, anti-virus and anti-malware management, security management, storage management, capacity monitoring, server availability monitoring and measurement, user activity monitoring, network capacity and utilization monitoring, hardware inventory, and software inventory and installation. System management generally includes a variety of functional components, including, but not limited to, data center infrastructure management, help desk management, network management, security information and event management, and configuration management. Configuration management generally systematically handles changes in the system to maintain the integrity of the system. Such changes can be implemented for beneficial purposes, including, but not limited to, the ability to modify the system; improving the performance, reliability, and / or maintainability of the system; extending the life of the system; reducing the cost, risk, and / or liability of the system; and correcting (one or more) defects of the system.

[0021] Compared to conventional system management techniques, the example techniques described herein have multiple benefits. For example, the example techniques may be able to recommend administrator changes to information technology (IT) administrators to facilitate system management of the enterprise managed by the administrator. The example techniques can reduce the time and / or amount of resources (e.g., processor cycles, memory, network bandwidth) consumed by the management system. The example techniques can reduce the costs associated with the management system. For example, by recommending administrator changes to the administrator, the administrator does not need to spend time looking at resources to determine which administrator changes to perform and / or which administrator changes not to perform. Reducing the time spent by the administrator to determine administrator changes results in a reduction in costs associated with the administrator. The example techniques can improve the efficiency of the computing system that the administrator uses to perform system management operations.

[0022] Example techniques can improve the efficiency of IT administrators. For example, by recommending (one or more) administrator changes to an IT administrator, example techniques can reduce the number of steps performed by the IT administrator, the amount of work expended by the IT administrator, and / or the amount of time spent by the IT administrator overseeing system administration for an enterprise.

[0023] Example techniques may improve efficiency for end users. For example, by recommending administrator change(s) to an IT administrator, example techniques may enable the IT administrator to optimize the operation of an end user's device and / or optimize the end user's user experience based on currently available resources and configurations. It should be noted that the administrator change(s) may be recommended directly to the end user (e.g., not through an IT administrator), which may enable the end user to optimize the operation of the end user's device and / or optimize the end user's user experience.

[0024] Figure 1 1 is a block diagram of an example administrator change recommendation system 100 according to an embodiment. Generally speaking, the administrator change recommendation system 100 operates to perform system configuration operations with respect to the enterprises 114A-114M. Figure 1 As shown in , the administrator change recommendation system 100 includes (one or more) system management servers 102 and enterprises 114A-114M. Communication between (one or more) system management servers 102 and enterprises 114A-114M is performed through a network 108 using well-known network communication protocols. The network 108 can be a wide area network (e.g., the Internet), a local area network (LAN), another type of network, or a combination thereof.

[0025] Each of the enterprises 114A-114M includes an administrator system 104, a network 110, and a user device 106. For example, the first enterprise 114A includes a first administrator system 104A, a first network 110A, and a first user device 106A. The second enterprise 114B includes a second administrator system 104B, a second network 110B, and a second user device 106B. The Mth administrator system 104M includes an Mth administrator system 104M, an Mth network 110M, and an Mth user device 106M. In each of the enterprises 114A-114M, communications between the administrator system 104 and the user device 106 are performed over the network 110 using well-known network communication protocols. The network 110 may be a wide area network (e.g., the Internet), a local area network (LAN), another type of network, or a combination thereof.

[0026] In each of the enterprises 114A-114M, the user device 106 is a processing system capable of communicating with the administrator system 104. An example of a processing system is a system including at least one processor capable of manipulating data according to a set of instructions. For example, the processing system may be a computer, a personal digital assistant, or the like. The user device 106 is configured to provide a request to a server (not shown) to request information stored on the server (or otherwise accessed via the server). For example, a user may initiate a request to execute a computer program (e.g., an application) using a client (e.g., a web browser, a web crawler, or other type of client) deployed on a user device, which is owned or otherwise accessible by the user. According to some example embodiments, the user device 106 is capable of accessing a domain (e.g., a website) hosted by a server, so that the user device 106 can access information available via the domain. Such a domain may include a web page, for example, a web page may be provided as a hypertext markup language (HTML) document and objects (e.g., files) linked therein.

[0027] Each of the user devices 106 may include any client-enabled system or device, including but not limited to a desktop computer, a laptop computer, a tablet computer, a wearable computer such as a smart watch or a head-mounted computer, a personal digital assistant, a cellular phone, an Internet of Things (IoT) device, etc. It will be appreciated that any one or more of the user devices 106 may communicate with any one or more of the servers.

[0028] The user device 106 is also configured to provide information to the administrator system 104, automatically or in response to a query from the administrator system 104. For example, such information may relate to the hardware and / or software configuration of the user device 106 (e.g., updates thereof), the workload of the user device 106, resource utilization on the user device, etc.

[0029] The administrator system 104 is a processing system capable of communicating with the user device 106. The administrator system 104 is configured to perform operations in response to instructions received from an IT administrator of the enterprise 114 to facilitate system management of the enterprise 114. For example, the administrator system 104 may provide commands to the system management server(s) 102 indicating administrator changes to be executed by the system management server(s) 102 to perform system management of the enterprise 114. In another example, the administrator system 104 may provide a request for information to the user device 106. The IT administrator may make decisions regarding system management of the user device 106 and / or the network 110 based, at least in part, on the information received from the user device 106. It will be appreciated that at least some (e.g., all) of the information may be collected by the system management server(s) 102 instead of by the administrator system 104.

[0030] The system management server(s) 102 are processing systems capable of communicating with the enterprises 114A-114M. For example, the system management server(s) 102 may be capable of communicating with the administrator system 104 and / or the user device 106 in each enterprise 114. The system management server(s) 102 include administrator change recommendation logic 112. For the purpose of performing system management of the enterprise 114, the administrator change recommendation logic 112 is configured to provide information to and / or collect information from the administrator system 104 and / or the user device 106 in each enterprise 114. For example, the administrator change recommendation logic 112 can push information to the user device 106, or provide information in response to a request received from the user device 106. The request can be user-generated or generated without user participation. For example, the policy applied to the user device is completed without an explicit user request. According to this example, the policy is applied in the background even if no user is logged into the user device. Further in accordance with this example, the user device (e.g., an agent thereon) can poll the server for policies on a schedule (e.g., once an hour) or on an event (e.g., device wakeup, user unlocking, etc.). Further in accordance with this example, the server can push policies to the user device (e.g., an agent thereon) via an open HTTP endpoint.

[0031] According to the example embodiments described herein, the administrator change recommendation logic 112 collects information indicating administrator changes made by administrators in the environment of the enterprise 114A-114M and further indicating the values ​​of the metrics generated by the administrator changes. The administrator change recommendation logic 112 analyzes the information and classifies it into categories based at least in part on the characteristics of the environment in which the administrator changes were made. The administrator change recommendation logic 112 determines the type of expected administrator changes that the administrator is about to perform on the enterprise 114 based at least in part on the operations performed by the administrator using the enterprise management tool. For example, the enterprise management tool can be accessed by the administrator via a client deployed on the administrator system 104. The administrator change recommendation logic 112 cross-references the type of expected administrator changes and the information to identify a subset of information corresponding to the type. The administrator change recommendation logic 112 infers a causal relationship between the administrator change(s) made after the administrator change of the type indicated by the information in the subset and the increase in the value(s) of the corresponding metric(s) indicated by the information in the subset by analyzing the subset, at least in part based on cross-referencing the type of expected administrator changes and the information in the subset. The administrator change recommendation logic 112 recommends a recommended administrator change to be performed by an administrator based at least in part on one or more categories into which at least one of the subsets is classified, and further based at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics, information in at least one of the subsets indicating that the recommended administrator change has been performed after an administrator change of the type has been performed.

[0032] It will be appreciated that at least some of the above functions may be implemented by the administrator system 104, rather than by the administrator change recommendation logic 112. For example, if the administrator does not want to provide information to the system management server(s) 102, but rather wishes to receive information only from the system management server(s) 102, the administrator system 104 may determine the type of expected administrator changes that the administrator is about to perform on the enterprise 114 based, at least in part, on the operations performed by the administrator using the enterprise management tool. The administrator system 104 may cross-reference the type of expected administrator changes with information indicating administrator changes made by the administrator in the respective environments of the respective enterprises 114A-114M and further indicating the values ​​of the metrics produced by the administrator changes to identify a subset of information corresponding to the type. The administrator system 104 may infer, by analyzing the subset, a causal relationship between the administrator change(s) made after the administrator change of the type indicated by the information in the subset and the increase in the value(s) of the corresponding metric(s) indicated by the information in the subset, based, at least in part, on cross-referencing the type of expected administrator changes with the information in the subset. Administrator system 104 may recommend a recommended administrator change to be performed by an administrator based at least in part on one or more categories into which at least one of the subsets is classified and further based at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics, information in at least one of the subsets indicating that the recommended administrator change has been made after an administrator change of the type has been made. According to this example, information about administrator changes may have been classified into categories, including one or more categories, by administrator change recommendation logic 112 based at least in part on characteristics of the environment in which the administrator change was made. Further according to this example, administrator system 104 may receive information from administrator change recommendation logic 112 in response to the information classified by administrator change recommendation logic 112.

[0033] The administrator change recommendation logic 112 can be implemented in various ways to provide recommendations for administrator changes in the enterprise, including being implemented in hardware, software, firmware, or any combination thereof. For example, at least a portion of the administrator change recommendation logic 112 can be implemented as a computer program code configured to be executed in one or more processors. In another example, at least a portion of the administrator change recommendation logic 112 can be implemented as hardware logic / circuitry. For example, at least a portion of the administrator change recommendation logic 112 can be implemented in a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a complex programmable logic device (CPLD), etc. Each SoC may include an integrated circuit chip that includes one or more processors (e.g., a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or other circuits and / or embedded firmware to perform its functions.

[0034] For purposes of illustration, the administrator change recommendation logic 112 is shown as being incorporated into the system management server(s) 102 and is not intended to be limiting. It will be appreciated that the administrator change recommendation logic 112 (or any portion(s) thereof) may be incorporated into the administrator system 104 and / or any one or more user devices 106. For example, the client side aspects of the administrator change recommendation logic 112 may be incorporated into the administrator system 104 and / or one or more user devices 106, and the server side aspects of the administrator change recommendation logic 112 may be incorporated into the system management server(s) 102.

[0035] Figure 2 is a block diagram of an administrator change recommendation system 200 according to an embodiment, wherein the administrator change recommendation system is Figure 1 An example implementation of the administrator change recommendation system 100 is shown in FIG. Figure 2 As shown in , the administrator change recommendation system 200 includes (one or more) system management servers 202, clients 204A-204D and client machines 206A-206D. (One or more) system management servers 202 can communicate with clients 204A-204D and client machines 206A-206D, as described above with respect to Figure 1 The system management server(s) 202 includes administrator change recommendation logic 212, which may include the above-mentioned Figure 1 The administrator can change any functionality described by the recommendation logic 112.

[0036] For example, the administrator change recommendation logic 212 can provide administrator change recommendations in an enterprise management tool. For example, the administrator change recommendation logic 212 can use the enterprise management tool to synchronize the collection of telemetry data from the client 204A-204D at regular time intervals. The information received from each of the clients 204A-204D can include the device type in the enterprise, the application used, the deployment details, the update schedule, the crash details, the popular task sequence, and any other information using the tool. Therefore, a large amount of data can be collected from the client 204A-204D. In an exemplary embodiment, the administrator change recommendation logic 212 provides a query to the client 204A-204D, querying whether the user of the client 204A-204D wishes to opt-in to provide such telemetry or its specific aspects (e.g., driver updates) to the administrator change recommendation logic 212. According to this embodiment, if the user of the client (e.g., any client 204A-204D) chooses to provide telemetry or its specific aspects, the administrator change recommendation logic 212 uses the enterprise management tool to collect telemetry or its specific aspects from the client. In further accordance with this embodiment, if a user of a client does not elect to provide telemetry or particular aspects thereof, then the administrator change recommendation logic 212 does not collect telemetry or particular aspects thereof from the client.

[0037] The administrator change recommendation logic 212 includes a clustering algorithm 216 and a recommendation checker 218. The administrator change recommendation logic 212 can use a clustering algorithm 216 to analyze and classify data according to different customer environments and preferences. Different parameters can include the size of the enterprise (i.e., the number of customers) and parameters specific to the situation being evaluated. Each category in the obtained (environment-oriented) category can include multiple subcategories to cover specific scenarios that administrators may face. Then, when the customer uses the enterprise management tool, the tool can evaluate the type of administrator changes that the user wants to perform and cross-reference the type with information from the overall pool that has been analyzed and classified to provide the user with recommended administrator changes that may eventually be followed. If the customer decides to use the administrator change, the tool can automatically fill in the fields of the recommended administrator change (e.g., the next best administrator change) and confirm with the user before executing the recommended administrator change (e.g., asking for the user's approval).

[0038] An example data flow for providing administrator change recommendations will now be described. Clients 204A-204D may send data to (one or more) system management servers 202. (One or more) system management servers 202 may use clustering algorithms 216 to analyze data received from clients 204A-204D. After the clustering algorithm generates administrator change recommendations based on data analysis, the administrator change recommendations may pass through a recommendation checker 218 so that the administrator change recommendations are manually verified before being sent to clients 204A-204D. Once the administrator change recommendations pass the recommendation checker 218, the administrator change recommendation logic 212 sends the administrator change recommendations to the clients that will be improved by the recommended administrator changes. For example, by improving the performance (e.g., efficiency, speed) of client machines associated with the clients or by improving the statistical likelihood of such performance improvements as a result of implementing the recommended administrator changes, the clients may be improved by the recommended administrator changes.

[0039] For example, let's consider a scenario involving software updates, particularly operating system (OS) updates. In order to create a sufficient software update information base, a variety of telemetry data can be collected from each customer. Existing telemetry channels can be used to collect more data, such as the last update time, the last update software, boot time, central processing unit (CPU) utilization, battery utilization, application crash times, and network connectivity. The OS update summary algorithm can analyze the data to determine whether the update is successful. The last update software and the last update time specify how long ago and what updates the customer deployed to its client machines 206A-206D. Similarly, the update time can be used to narrow down the CPU and battery utilization of a specific update. Whether the update is successful can be determined by looking at the application crashes and the time of the crashes. If there are more Windows event log crashes after the update than before the update, this situation may be a symptom of the update. It will be recognized that any suitable characteristics can be used to measure the success of the update. Given this information for each client of a customer, the administrator change recommendation logic 212 can consider the average boot time, CPU usage, battery usage, and application crashes after the update of the client machine from this customer to indicate the success of the update for this customer. These averages can be provided to rows of a summary data table along with customer-specific characteristics (such as the number of clients). The same summary can be run on all other customers sending telemetry data, and their entries can be put into a data table.

[0040] For this scenario, let's assume that a new OS update came out 6 months ago and customer A may be considering deploying this update to its client machine 206A. From the data collection, we can see that other customers have downloaded the new operating system update. In order to determine the effectiveness of the new OS update, the administrator change recommendation logic 212 can consider all customers with the new update along with other parameters. For example, the parameters can include a user score for each customer. The user score can be used as a metric to indicate whether the corresponding customer is considered healthy. The administrator change recommendation logic 212 can then run a clustering algorithm 216 on the user scores and other parameters of all customers who have downloaded this update so that the OS update can be evaluated.

[0041] Clustering algorithm 216 may perform the following steps:

[0042] 1. Get the user score and the value of each parameter from each customer (in this case, number of clients, CPU usage, battery, app crashes, startup time, etc.).

[0043] 2. Simulate all user scores in the graph and randomly select N means on the graph. N is a positive integer (e.g., 1, 2, 3, 4, 5, etc.).

[0044] 3. For each customer, calculate the vector distance from its "point" to all found means.

[0045] 4. Classify customers using the closest mean.

[0046] 5. Repeat steps 3-4 for each customer.

[0047] 6. Recalculate the mean by taking the mean of the points for each group of classified customers.

[0048] 7. Repeat steps 3 through 6 until the mean changes slightly. For example, steps 3-6 may be repeated until the mean changes by less than a threshold amount.

[0049] The clustering algorithm 216 may indicate that customers are commonly affected by OS updates and that the updates may be categorized into 5 different categories. Based on this information, the clustering algorithm 216 may determine the closest mean for the parameter for customer A. If the average user score is higher than the average user score for customer A, the clustering algorithm 216 may provide an update recommendation for customer A.

[0050] Before sending this administrator change recommendation to the client, the administrator change recommendation may go through a level of manual verification via a recommendation checker 218. The recommendation checker 218 may ensure that the administrator change recommendation is valid and check the work of the clustering algorithm 216 to ensure that the administrator change recommendation makes sense. After passing this step, the administrator change recommendation logic 212 may send a notification of the administrator change recommendation to client A, and client A may choose whether to deploy the update. It will be appreciated that over time, the functionality of the recommendation checker 218 may be automated by utilizing a machine learning algorithm that observes the behavior of the recommendation checker 218 and is ultimately able to perform the work of the recommendation checker 218 independently.

[0051] Figures 3 to 5 Flowcharts 300, 400, and 500 are depicted for example methods for providing administrator change recommendations according to an embodiment. Flowcharts 300, 400, and 500 may be provided by, for example, Figure 1 For the purpose of illustration, reference is made to the system management server 102, one of the administrator systems 104A-104M, or a combination thereof. Figure 6 The flowcharts 300, 400, and 500 are described with reference to the computing system 600 shown in FIG. 6 , which may be an example implementation of the system management server (s) 102, one of the administrator systems 104A-104M, or a combination thereof. Figure 6 As shown in , computing system 600 includes administrator change recommendation logic 602. Administrator change recommendation logic 602 includes analysis logic 604, determination logic 606, cross-reference logic 608, recommendation logic 610, table logic 612, query logic 614, administrator change logic 616, and discount logic 618. Further structural and operational embodiments will be apparent to those skilled in the relevant art(s) based on the discussion regarding flowcharts 300, 400, and 500.

[0052] like Figure 3 As shown in , the method of flowchart 300 begins at step 302. In step 302, information is collected. The information indicates administrator changes made by administrators in the environment of the enterprise and further indicates the value of the metric resulting from the administrator change. In an example implementation, analysis logic 604 collects information 620, which indicates administrator changes made by administrators in the environment of the enterprise and further indicates the value of the metric resulting from the administrator change.

[0053] At step 304, the information is analyzed and classified into categories based at least in part on the characteristic(s) of the environment in which the administrator change was made. In an example implementation, the analysis logic 604 analyzes the information 620 and classifies the information into categories. The information 620 may indicate the characteristic(s) of the environment in which the administrator change was made. Thus, the analysis logic 604 may extract or derive the characteristic(s) of the environment in which the administrator change was made from the information 620.

[0054] At step 306, a type of expected administrator change that the administrator is about to perform on the enterprise is determined based, at least in part, on an operation performed by the administrator using the enterprise management tool. In an example implementation, determination logic 606 determines the type of expected administrator change that the administrator is about to perform on the enterprise. For example, determination logic 606 may receive an operation indicator 626 that indicates the type of expected administrator change that the administrator is about to perform. According to this example, determination logic 606 may determine the type of expected administrator change that the administrator is about to perform in response to receiving operation indicator 626 (e.g., based, at least in part, on the type indicated by operation indicator 626). Determination logic 606 may generate a type indicator 640 to indicate the type of administrator change.

[0055] At step 308, the type of expected administrator change is cross-referenced with the information to identify a subset of information to which the type corresponds. In an example implementation, cross-reference logic 608 cross-references the type of expected administrator change with information 620 to identify a subset of information to which the type corresponds. Cross-reference logic 608 may generate a subset indicator 642 to indicate the subset of information to which the type corresponds.

[0056] At step 310, a causal relationship is inferred between (one or more) administrator changes made after the administrator change of the type and an increase in (one or more) values ​​of (one or more) corresponding metrics indicated by the information in the subset. The information in the subset indicates (one or more) administrator changes made after the administrator change of the type. The causal relationship is inferred by analyzing the subset. The causal relationship is inferred based at least in part on cross-referencing the type of expected administrator changes and the information in the subset. In an example implementation, the recommendation logic 610 infers a causal relationship between (one or more) administrator changes made after the administrator change of the type (as indicated by the information 620 in the subset) and an increase in (one or more) values ​​of (one or more) corresponding metrics indicated by the information 620 in the subset by analyzing the subset based at least in part on cross-referencing the type of expected administrator changes with the information 620 in the subset. The recommendation logic 610 can extract or derive the (one or more) administrator changes made after the administrator change of the type from the information 620 in the subset.

[0057] At step 312, based at least in part on the one or more categories that at least one of the subsets is classified into and further based at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics, a recommended administrator change is recommended to be performed by the administrator, the information in at least one of the subsets indicating that the recommended administrator change has been performed after the type of administrator change. In an example implementation, the recommendation logic 610 provides an administrator change recommendation 650, the administrator change recommendation recommending the recommended administrator change to be performed by the administrator based at least in part on the one or more categories that at least one of the subsets is classified into and further based at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics. According to this implementation, the information 620 in at least one of the subsets indicates that the recommended administrator change was performed after the type of administrator change. The recommendation logic 610 may provide the administrator change recommendation 650 in response to receiving the category information 648 and the subset indicator 642. According to this example, the recommendation logic 610 may cross-reference the category indicated by the category information 648 and the subset indicated by the subset indicator 642 to determine one or more categories into which the at least one subset is classified.

[0058] In an example embodiment, analyzing and classifying the information at step 304 includes classifying the information in each category into a plurality of subcategories corresponding to corresponding tasks performed by administrators in the environment. According to this embodiment, recommending recommended administrator changes at step 310 includes recommending recommended administrator changes to be performed by the administrator based further at least in part on the subcategories into which the subsets are classified.

[0059] In some example embodiments, one or more steps 302, 304, 306, 308, 310, and / or 312 of flowchart 300 may not be performed. Furthermore, steps in addition to or in place of steps 302, 304, 306, 308, 310, and / or 312 may be performed. For example, in an example embodiment, the method of flowchart 300 further includes, for each administrator change indicated by the information, determining an amount by which a measured value of an attribute associated with an end user and / or device for which the administrator change is directed changes in response to the administrator change. For example, determination logic 606 may determine an amount by which a measured value of an attribute changes in response to the administrator change. According to this embodiment, recommending the recommended administrator change at step 312 includes further recommending a recommended administrator change to be performed by the administrator based at least in part on an amount by which a measured value of an attribute associated with each administrator change associated with the information in the subset changes.

[0060] In one aspect of this embodiment, collecting information at step 302 includes collecting data about (one or more) regional events and / or (one or more) global events. During the time period during which the information is collected, each regional event affects the behavior of enterprise end users within a limited geographic area. During the time period during which the information is collected, each global event affects the behavior of enterprise end users throughout the world. According to this aspect, collecting information at step 302 also includes determining the extent to which (one or more) regional events and / or (one or more) global events change the amount of attribute measurement for each administrator change. For example, the information may include first information and second information. The first information may indicate a first amount of measurement change for each administrator change in the absence of (e.g., before) (one or more) regional events and (one or more) global events. The second information may indicate a second amount of measurement change for each administrator change in response to (one or more) regional events and / or (one or more) global events that are occurring. According to this example, determining the extent at step 302 may include comparing the first amount and the second amount (e.g., subtracting the first amount from the second amount) to determine the extent. Further in accordance with this aspect, recommending the recommended administrator changes in step 312 includes further recommending the recommended administrator changes to be performed by the administrator based at least in part on the extent of the amount of change in the measured values ​​of the attributes of each administrator change associated with information within the (one or more) regional events and / or (one or more) global event change subsets.

[0061] In another aspect of this embodiment, the method of flowchart 300 includes analyzing historical data about administrator changes made by an administrator to determine that a subset of parameters collected about administrator changes may contribute to a change in a corresponding measurement value of an attribute. For example, the analysis may reveal that each parameter in the subset has caused a change in a corresponding measurement value of the attribute in the past. According to this example, the analysis may reveal that each parameter not included in the subset has not caused a change in a corresponding measurement value of the attribute in the past. In another example, the analysis may reveal that a proportion of instances of each parameter in the subset has caused a change in a corresponding measurement value of the attribute in the past that is greater than or equal to a threshold proportion. According to this example, the analysis may reveal that a proportion of instances of each parameter not included in the subset is less than a threshold proportion, and the subset has caused a change in a corresponding measurement value of the attribute in the past. In an example implementation, analysis logic 604 analyzes historical data to determine that a subset of parameters may contribute to a change in a corresponding measurement value of the attribute. For example, the historical data may be included in information 620. According to this aspect, collecting information in step 302 includes limiting the collected information to a subset of parameters.

[0062] In another example embodiment, the method of flowchart 300 also includes, for each administrator change indicated by the information, determining, in response to the administrator change, an amount of a user experience score associated with the end user and / or device to which the administrator change is directed. The user experience score indicates the quality of the user experience associated with the end user and / or device. For example, the quality of the user experience may be based at least in part on the amount of time the end user waits for a process operating on the device to perform an operation, the frequency of crashes of the device (or an application running on the device), the number of errors encountered by the user regarding the application running on the device, etc. In an example, the information may include first information and second information. For each administrator change, the first information may indicate a first value of the corresponding user experience score without the administrator change. For each administrator change, the second information may indicate a second value of the corresponding user experience score in response to the administrator change. According to this example, for each administrator change, determining the amount by which the corresponding user experience score is changed in response to the administrator change may include comparing the first value and the second value (e.g., subtracting the first value from the second value) to determine the amount. For example, determination logic 606 may determine the amount by which the user experience score is changed in response to the administrator change. According to this embodiment, recommending recommended administrator changes at step 312 includes recommending recommended administrator changes to be performed by the administrator based further at least in part on the amount of user experience score for each administrator change associated with the change based on information in the subset.

[0063] In one aspect of this embodiment, collecting information at step 302 includes collecting data about (one or more) regional events and / or (one or more) global events. During the time period during which the information was collected, each regional event affected the behavior of enterprise end users within a limited geographic area. During the time period during which the information was collected, each global event affected the behavior of end users of enterprises throughout the world. According to this aspect, collecting information at step 302 also includes determining the extent to which (one or more) regional events and / or (one or more) global events changed the user experience score for each administrator change. Further according to this aspect, recommending recommended administrator changes at step 312 includes further recommending recommended administrator changes to be performed by the administrator based at least in part on the extent to which the user experience score of each administrator change is associated with information in at least a subset of (one or more) regional events and / or (one or more) global events.

[0064] In another aspect of this embodiment, the method of flowchart 300 includes analyzing historical data about administrator changes made by an administrator to determine that a subset of parameters collected about administrator changes may contribute to a change in a corresponding user experience score. For example, the analysis may reveal that each parameter in the subset has caused a change in the corresponding user experience score in the past. According to this example, the analysis may reveal that each parameter not included in the subset has not caused a change in the corresponding user experience score in the past. In another example, the analysis may reveal that a proportion of instances of each parameter in the subset that caused a change in the corresponding user experience score in the past is greater than or equal to a threshold proportion. According to this example, the analysis may reveal that a proportion of instances of each parameter not included in the subset is less than a threshold proportion, and the subset has caused a change in the corresponding user experience score in the past. In an example implementation, analysis logic 604 analyzes historical data to determine that a subset of parameters collected about administrator changes may contribute to a change in the corresponding user experience score. According to this aspect, collecting information in step 302 includes limiting the collected information to a subset of parameters.

[0065] In yet another example embodiment, the method of flowchart 300 further includes, for each administrator change indicated by the information, determining an amount by which the end-user productivity of an end-user that is the target of the administrator change changes in response to the administrator change. In an example, the information may include first information and second information. The first information may indicate, for each administrator change, a first amount of the end-user productivity of the end-user before the administrator change. The second information may indicate, for each administrator change, a second amount of the end-user productivity of the end-user in response to the administrator change. According to this example, for each administrator change, determining an amount by which the end-user productivity of the end-user changes in response to the administrator change may include comparing a first value and a second value (e.g., subtracting the first value from the second value) to determine the amount. For example, determination logic 606 may determine an amount by which the end-user productivity of an end-user that is the target of the administrator change changes in response to the administrator change. According to this embodiment, recommending the recommended administrator change at step 312 includes recommending a recommended administrator change to be performed by the administrator based further, at least in part, on the amount of change in the end-user productivity of each administrator change associated with the information in the subset.

[0066] In one aspect of this embodiment, collecting information at step 302 includes collecting data about (one or more) regional events and / or (one or more) global events. During the time period during which the information is collected, each regional event affects the behavior of enterprise end users within a limited geographic area. During the time period during which the information is collected, each global event affects the behavior of enterprise end users throughout the world. According to this aspect, collecting information at step 302 also includes determining the extent to which (one or more) regional events and / or (one or more) global events change the productivity of end users for each administrator change. For example, the information may include first information and second information. The first information may indicate a first amount of end-user productivity changed by each administrator in the absence of (e.g., before) (one or more) regional events and (one or more) global events. The second information may indicate a second amount of end-user productivity changed by each administrator in response to the ongoing (one or more) regional events and / or (one or more) global events. According to this example, determining the extent to which (one or more) regional events and / or (one or more) global events change the productivity of end users for each administrator change may include comparing the first amount and the second amount (e.g., subtracting the first amount from the second amount) to determine the extent. Further in accordance with this aspect, recommending recommended administrator changes in step 312 includes further recommending recommended administrator changes to be performed by the administrator based at least in part on the extent to which each administrator change associated with the information in the subset changes end-user productivity.

[0067] In another aspect of this embodiment, the method of flowchart 300 includes analyzing historical data about administrator changes made by administrators to determine that a subset of parameters collected about administrator changes may contribute to changes in corresponding productivity. For example, the analysis may reveal that each parameter in the subset has caused a change in corresponding productivity in the past. According to this example, the analysis may reveal that each parameter not included in the subset has not caused a change in corresponding productivity in the past. In another example, the analysis may reveal that a proportion of instances in which each parameter in the subset has caused a change in corresponding productivity in the past is greater than or equal to a threshold proportion. According to this example, the analysis may reveal that a proportion of instances of each parameter not included in the subset is less than a threshold proportion, and the subset has caused changes to application productivity in the past. In an example implementation, analysis logic 604 analyzes historical data about administrator changes made by administrators to determine that a subset of parameters collected about administrator changes may contribute to changes in corresponding productivity. According to this aspect, collecting information in step 302 includes limiting the collected information to a subset of parameters.

[0068] In yet another example embodiment, the method of flowchart 300 also includes, for each administrator change indicated by the information, determining, in response to the administrator change, the extent to which the security of the device targeted by the administrator change is changed. For example, the information may include first information and second information. For each administrator change, the first information may indicate a first security amount experienced by the device in the absence of the administrator change (e.g., before the administrator change). The second information may indicate, for each administrator change, a second security amount experienced by the device in response to the occurrence of the administrator change. According to this example, determining the extent to which the security of the device is changed in response to the administrator change may include comparing the first amount and the second amount (e.g., subtracting the first amount from the second amount) to determine the extent. For example, determination logic 606 may determine, in response to the administrator change, the extent to which the security of the device targeted by the administrator change is changed. According to this embodiment, recommending the recommended administrator change in step 312 includes further recommending the recommended administrator change performed by the administrator based at least in part on the extent to which the security of each administrator change associated with the information in the subset is changed.

[0069] In one aspect of this embodiment, collecting information at step 302 includes collecting data about (one or more) regional events and / or (one or more) global events. During the time period during which the information was collected, each regional event affected the behavior of enterprise end users within a limited geographic area. During the time period during which the information was collected, each global event affected the behavior of enterprise end users throughout the world. According to this aspect, collecting information at step 302 also includes determining the degree of security of each administrator change associated with the (one or more) regional events and / or (one or more) global events. Further according to this aspect, recommending recommended administrator changes at step 312 includes recommending recommended administrator changes to be performed by the administrator based further at least in part on the degree of security of each administrator change associated with information within the (one or more) regional events and / or (one or more) global event change subsets.

[0070] In another aspect of this embodiment, the method of flowchart 300 includes analyzing historical data about administrator changes made by an administrator to determine that a subset of parameters collected about administrator changes may contribute to changes in device security. For example, the analysis may reveal that each parameter in the subset has caused a change in device security in the past. According to this example, the analysis may reveal that each parameter not included in the subset has not caused a change in device security in the past. In another example, the analysis may reveal that a proportion of instances in which each parameter in the subset caused a change in device security in the past is greater than or equal to a threshold proportion. According to this example, the analysis may reveal that a proportion of instances of each parameter not included in the subset is less than a threshold proportion, and the subset has caused a change in device security in the past. In an example implementation, analysis logic 604 analyzes historical data about administrator changes made by an administrator to determine that a subset of parameters collected about administrator changes may contribute to changes in device security. According to this aspect, collecting information in step 302 includes limiting the collected information to a subset of parameters.

[0071] In another example embodiment, the method of flowchart 300 also includes determining that the person has approved the recommendation of the recommended administrator change to be performed by the administrator. For example, determination logic 606 can determine that the person has approved the recommendation of the recommended administrator change. According to this example, determination logic 606 can provide approval query 636 to query logic 614. Approval query 636 can request an indication of whether the person has approved the recommendation of the recommended administrator change. Query logic 614 can generate query 654 in response to receiving approval query 636. Query 654 can request the person to confirm whether the person has approved the recommendation of the recommended administrator change. Query logic 614 can receive confirmation 656, thereby confirming that the recommendation of the recommended administrator change has been approved by the person. Query logic 614 can generate approval indicator 638 based at least in part on receiving confirmation 656 to indicate that the recommendation of the recommended administrator change has been approved by the person. Determination logic 606 can determine that the person has approved the recommendation of the recommended administrator change based at least in part on approval indicator 638 indicating the same. According to this embodiment, recommending the recommended administrator change at step 312 includes recommending the recommended administrator change to be performed by the administrator based further at least in part on determining that the human has approved the recommendation of the recommended administrator change to be performed by the administrator.

[0072] In yet another example embodiment, the method of flowchart 300 further includes collecting historical rating data indicating a change in the rating of the application from a rating system external to the enterprise, the change resulting from an update to the application. For example, analysis logic 604 may collect the historical rating data. According to this example, information 620 may include the historical rating data. According to this embodiment, recommending the recommended administrator change at step 312 includes further recommending a recommended administrator change performed by the administrator based at least in part on the historical rating data. For example, the historical rating data may be included in historical information 630.

[0073] In yet another example embodiment, the method of flowchart 300 also includes collecting feedback data indicating ratings of previously recommended administrator changes. Each rating is received from an administrator who received a recommendation to perform a corresponding previously recommended administrator change. Each rating indicates a quality of a user experience resulting from the performance of the corresponding previously recommended administrator change in response to receiving the recommendation. For example, analysis logic 604 may collect feedback data 632, which may indicate ratings of previously recommended administrator changes. According to this example, information 620 may include feedback data 632. Analysis logic 604 may provide feedback data 632 to recommendation logic 610, although example embodiments are not limited in this respect. According to this embodiment, recommending the recommended administrator change in step 312 includes further recommending the recommended administrator change to be performed by the administrator based at least in part on the ratings of the previously recommended administrator changes.

[0074] In another example embodiment, the method of flowchart 300 also includes collecting feedback data indicating whether a previously recommended administrator change was performed by an administrator in response to receiving a recommendation to perform the previously recommended administrator change. For example, analysis logic 604 may collect feedback data 632, which may indicate whether the administrator performed the previously recommended administrator change. According to this example, information 620 may include feedback data 632. Analysis logic 604 may provide feedback data 632 to recommendation logic 610, although example embodiments are not limited in this respect. According to this embodiment, recommending the recommended administrator change at step 312 includes further recommending the recommended administrator change to be performed by the administrator based at least in part on whether the previously recommended administrator change was performed by the administrator in response to receiving the recommendation.

[0075] In yet another example embodiment, the method of flowchart 300 also includes determining the proportion of administrators who have performed the recommended administrator change among administrators who have received recommendations to perform the recommended administrator change. For example, if ten administrators receive recommendations to perform the recommended administrator change, and two of the ten administrators perform the recommended administrator change, the proportion will be 2 / 10=1 / 5. The above numbers are provided for non-limiting, illustrative purposes. For example, determination logic 606 can determine the proportion of administrators. According to this example, determination logic 606 can monitor whether the administrator performs the recommended administrator change to determine the proportion. According to this example, historical information 630 can indicate the proportion of administrators. According to this embodiment, the method of flowchart 300 also includes notifying the administrator of the proportion of administrators who have performed the recommended administrator change. For example, recommendation logic 610 can provide notification 652 to notify the administrator of the proportion.

[0076] In one aspect of this embodiment, the method of flowchart 300 also includes determining whether a proportion of administrators who have performed the recommended administrator changes is greater than or equal to a threshold. For example, determination logic 606 may determine whether a proportion of administrators who have performed the recommended administrator changes is greater than or equal to a threshold. According to this aspect, notifying the administrator of the proportion of administrators who have performed the recommended administrator changes includes notifying the administrator of the proportion of administrators who have performed the recommended administrator changes based at least in part on a determination that the proportion is greater than or equal to the threshold.

[0077] In yet another example embodiment, the method of flowchart 300 further includes analyzing, using machine learning (ML) techniques, the recommendations of the recommended administrator changes made to the administrator and the expected administrator changes and the categories of information on which the recommendations of the recommended administrator changes are based to determine the recommended administrator changes to recommend. For example, analysis logic 604 may use machine learning techniques to analyze historical information 630, which may indicate recommended and expected administrator changes and the categories of information, to determine the recommended administrator changes to recommend.

[0078] In some example embodiments, the analysis logic 604 uses a neural network to perform machine learning. Examples of neural networks include, but are not limited to, feedforward neural networks and long short-term memory (LSTM) neural networks. A feedforward neural network is an artificial neural network in which the connections between units in the neural network do not form loops. A feedforward neural network allows data to flow forward (e.g., from an input node to an output node), but a feedforward neural network does not allow data to flow backward (e.g., from an output node to an input node). In an example embodiment, the analysis logic 604 uses a feedforward neural network to train a machine learning model for determining ML-based confidence. Such ML-based confidence can be used to determine the likelihood of an event occurring.

[0079] The LSTM neural network is a recursive neural network with memory and allows data to flow forward and backward in the neural network. The LSTM neural network is capable of remembering values ​​for short or long time periods. Therefore, the LSTM neural network can prevent the stored values ​​from being diluted iteratively over time. In one example, the LSTM neural network may be able to store information 620 over time. For example, the LSTM neural network can generate a recommendation model by utilizing such information. In another example, the LSTM neural network is capable of remembering the relationship between administrator changes, metrics (e.g., before and / or after the administrator changes occurred), information categories, characteristics of the environment in which the administrator changes were made, types of administrator changes, recommended administrator changes, and ML-based confidence derived therefrom.

[0080] The analysis logic 604 may include training logic and reasoning logic. The training logic is configured to train a machine learning algorithm, and the reasoning logic uses the algorithm to determine (e.g., infer) ML-based confidence. For example, the training logic may provide sample administrator changes, sample metrics, sample information categories, sample characteristics of the environment in which the administrator changes are made, the type of sample administrator changes, sample recommended administrator changes, and sample confidence as inputs to the algorithm to train the algorithm. The sample data may be labeled. The machine learning algorithm may be configured to derive a relationship between administrator changes, metrics, information categories, characteristics of the environment in which the administrator changes are made, the type of administrator changes, recommended administrator changes, and the resulting ML-based confidence. The reasoning logic is configured to utilize the machine learning algorithm trained by the training logic to determine the ML-based confidence when the administrator changes, metrics, information categories, characteristics of the environment in which the administrator changes are made, the type of administrator changes, and the recommended administrator changes are provided as inputs to the algorithm.

[0081] In another example embodiment, the method of flowchart 300 also includes collecting user information about user changes made on corresponding client devices of corresponding end users in the enterprise based at least in part on instructions from the corresponding end users. Each instruction indicates that a corresponding change is to be made on the corresponding client device. For example, analysis logic 604 can collect user information 624 about user changes. According to this embodiment, the method of flowchart 300 also includes analyzing user information and classifying it into categories based at least in part on (one or more) characteristics of the environment in which the user changes are made. For example, analysis logic 604 can analyze user information 624 and classify it into categories. Further according to this embodiment, cross-referencing the type of expected administrator changes with the information in step 308 includes cross-referencing the type of expected administrator changes with the information and the user information to identify a subset corresponding to this type.

[0082] In yet another example embodiment, the method of flowchart 300 further includes Figure 4 One or more of the steps shown in the flowchart 400. Figure 4 As shown in , the method of flowchart 400 begins at step 402. In step 402, it is determined that an administrator initiates execution of the recommended administrator change. For example, determination logic 606 can determine that the administrator initiates execution of the recommended administrator change. According to this example, determination logic 606 can make a determination in response to initiation indicator 628 (e.g., based at least in part on initiation indicator 628 indicating that the administrator initiated execution of the recommended administrator change). Further according to this example, determination logic 606 can generate fill instructions 634 in response to determining that the administrator initiated execution of the recommended administrator change.

[0083] At step 404, based at least in part on a determination that the administrator initiated execution of the recommended administrator change, fields defining corresponding attributes of the recommended administrator change are populated in the electronic form. For example, the form logic 612 may populate the fields in the electronic form. According to this example, the form logic 612 may populate the fields in response to receiving the populate instruction 634. Further according to this example, the form logic 612 may generate a completion indicator 644 in response to the fields being populated. The completion indicator 644 indicates that the populating of the fields in the electronic form has been completed.

[0084] At step 406, a query is provided to the administrator. The query requests confirmation that the recommended administrator changes will be performed according to the properties defined by the corresponding fields. For example, the query logic 614 can provide the query 654 to the administrator. According to this example, the query 654 can request confirmation that the recommended administrator changes will be performed according to the properties defined by the corresponding fields. Further according to this example, the query logic 614 can provide the query 654 to the administrator in response to receiving the completion indicator 644 (e.g., based at least in part on the completion indicator indicating that the field population in the electronic form has been completed).

[0085] At step 408, based at least in part on the confirmation received from the administrator, the recommended administrator change is performed according to the attributes defined by the corresponding fields. For example, the administrator change logic 616 can perform the administrator change 658 according to the attributes based at least in part on receiving the confirmation. According to this example, the query logic 614 can receive a confirmation 656 from the administrator confirming that the recommended administrator change will be performed according to the attributes defined by the corresponding fields. For example, the query logic 614 can receive the confirmation 656 in response to the query 654. Further according to this example, the query logic 614 can generate the administrator change instruction 646 in response to receiving the confirmation 656. For example, receiving the confirmation 656 can trigger the generation of the administrator change instruction 646. Further according to this example, the administrator change logic 616 can perform the administrator change 658 in response to receiving the administrator change instruction 646 (e.g., based at least in part on receiving the administrator change instruction 646).

[0086] like Figure 5 As shown in , the method of flowchart 500 begins at step 502. In step 502, a type of expected administrator change that an administrator is about to perform on an enterprise is determined based at least in part on an operation performed by the administrator using an enterprise management tool. In an example implementation, determination logic 606 determines the type of expected administrator change based at least in part on the operation performed by the administrator. For example, determination logic 606 may determine the type of expected administrator change in response to receiving an operation indicator 626. According to this example, operation indicator 626 may indicate the type of expected administrator change.

[0087] At step 504, the type of expected administrator change is cross-referenced with information indicating administrator changes made by administrators in the corresponding environment of the corresponding enterprise and further indicating values ​​of metrics produced by the administrator changes to identify a subset of information to which the type corresponds. In an example implementation, cross-reference logic 608 cross-references the type of expected administrator change with information 620 indicating administrator changes made by administrators in the corresponding environment of the corresponding enterprise and further indicating values ​​of metrics produced by the administrator changes to identify a subset of information 620 to which the type corresponds. Cross-reference logic 608 may generate a subset indicator 642 to indicate the subset of information to which the type corresponds.

[0088] At step 506, a causal relationship is inferred between the administrator change(s) made after the administrator change of the type and the increase in the value(s) of the corresponding metric(s) indicated by the information in the subset. The information in the subset indicates the administrator change(s) made after the administrator change of the type. The causal relationship is inferred by analyzing the subset. The causal relationship is inferred based at least in part on cross-referencing the type of expected administrator change and the information in the subset. In an example implementation, the recommendation logic 610 infers the causal relationship between the administrator change(s) made after the administrator change of the type (as indicated by the information 620 in the subset) and the increase in the value(s) of the corresponding metric(s) indicated by the information 620 in the subset by analyzing the subset based at least in part on cross-referencing the information 620 in the subset. The recommendation logic 610 can extract or derive the administrator change(s) made after the administrator change of the type from the information 620 in the subset.

[0089] At step 508, based at least in part on the one or more categories that at least one of the subsets is classified into and further based at least in part on an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics, a recommended administrator change is recommended to be performed by the administrator, the information in at least one of the subsets indicating that the recommended administrator change has been made after the type of administrator change. Based at least in part on the (one or more) characteristics of the environment in which the administrator change is made, the information is classified into categories, including one or more categories. In an example implementation, the recommendation logic 610 provides an administrator change recommendation 650, the administrator change recommendation based at least in part on the one or more categories that at least one of the subsets is classified into and further based at least in part on an increase in at least one of the (one or more) values ​​of at least one of the (one or more) corresponding metrics, the recommended administrator change is recommended to be performed by the administrator. The information 620 in at least one of the subsets indicates that the recommended administrator change was made after the type of administrator change. The information 620 may indicate the (one or more) characteristics of the environment in which the administrator change is made. The analysis logic 604 may extract or derive the (one or more) characteristics of the environment in which the administrator change is made from the information 620. The analysis logic 604 may classify the information 620 into categories based at least in part on the features. The analysis logic 604 may generate category information 648 to indicate (e.g., specify) the categories. The recommendation logic 610 may provide administrator change recommendations 650 in response to receiving the category information 648 and the subset indicator 642. For example, the recommendation logic 610 may cross-reference the categories indicated by the category information 648 and the subset indicated by the subset indicator 642 to determine one or more categories into which the at least one subset is classified.

[0090] In some example embodiments, one or more steps 502, 504, 506, and / or 508 of flowchart 500 may not be performed. In addition, steps in addition to or in place of steps 502, 504, 506, and / or 508 may be performed. For example, in an example embodiment, recommending the recommended administrator change at step 508 includes recommending the recommended administrator change via an administrator change recommendation service, which is configured to provide administrator change recommendations to the administrator. According to this embodiment, the method of flowchart 500 also includes providing the administrator with a cost discount for using the administrator change recommendation service. The discount depends on whether the administrator allows the information collected about the administrator changes made by the administrator to facilitate the information collected about the administrator changes made by the administrator in the corresponding environment of the corresponding enterprise. For example, discount logic 618 can provide a discount offer 660 to the administrator. The discount offer 660 provides a cost discount for using the administrator change recommendation service, with the proviso that the administrator allows the information collected about the administrator changes made by the administrator to facilitate the information 620 about the administrator changes made by the administrator.

[0091] Although the operations of some disclosed methods are described in a particular order for ease of presentation, it should be understood that this description includes rearrangement unless the specific language set forth herein requires a particular order. For example, the operations described in sequence may be rearranged or performed simultaneously in some cases. In addition, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods can be used in conjunction with other methods.

[0092] The user scores and user experience scores described herein may be determined based on the techniques described for availability scores in U.S. patent application Ser. No. 16 / 389,909, filed on April 19, 2019 (hereinafter referred to as the "'909 application"). For example, the user scores and user experience scores may be determined by performing a weighted calculation on collected information (e.g., information received from a user device). It will be appreciated that the techniques described in the '909 application may be extended to consider any one or more of the factors mentioned herein, in addition to or in lieu of any one or more of the factors mentioned in the '909 application, for determining the user scores and user experience scores. For example, the '909 application mentions the following example factors:

[0093] -System boot time by stage (including core boot time);

[0094] - Phased login time;

[0095] - Application launch time (e.g., the time from when the application is launched to when the user is able to interact with the application);

[0096] - The total duration and number of times the processor utilization exceeded the threshold;

[0097] - The total duration and number of times the memory usage exceeds the threshold;

[0098] - The total duration and number of times that memory I / O operations exceed the threshold;

[0099] -The total duration and number of times network I / O operations exceed the threshold;

[0100] - Storage utilization;

[0101] - User device utilization (e.g., the number of hours the user's device was turned on and the user was interacting with the device);

[0102] -Battery utilization;

[0103] - Health indicators: frequency and number of unexpected reboots, expected battery life, expected hard drive life, frequency and number of driver and / or app crashes.

[0104] Figure 7 An example pseudo code 700 for calculating (e.g., determining) a user score and a user experience score according to an embodiment is shown. For example, a user score or a user experience score may represent a user experience on a particular device. In some examples, the user score and the user experience score range from 0 to 100, where 100 is the best experience and 0 is the worst experience. For example, if the boot time is higher than the average (e.g., longer than the average boot time on a particular device), the user may feel that the experience is poor, and a boot time higher than the average will provide a better experience for the user. Pseudo code 700 may rely on standardized metric data (e.g., when the collected information is converted into metric data, a "better" value is converted into a lower metric number). In some examples, the metric data includes example data for a particular device (e.g., the core boot time is 5 seconds); a global mean (e.g., the average core boot time is 10 seconds); and a global standard deviation (e.g., the standard deviation of the boot time of all devices is 2.1 seconds). The weight applied to the metric data indicates the importance of the metric to the user's perception of the computing device experience. In some example calculations, the weights sum to 1.0. For example, for three metrics, the weights may be startup time = 0.1, application startup time = 0.4, and battery life = 0.5.

[0105] The specific calculation is shown in pseudo code 700 and repeated below. ue_score is the user score or user experience score.

[0106] ue_score = 0;

[0107] for each metric

[0108] {

[0109] z_score=min(5,max(-5,(instance-mean) / standard deviation));

[0110] metric_value=(-1*<z_score> +5 / 10);

[0111] ue_score+=metric_value*metric_weight;

[0112] }

[0113] next metric;

[0114] return ue_score;

[0115] Some examples use alternative calculations of user scores and user experience scores. For example, instead of a linear correlation between the z-score and the user experience (ue_score), an exponential relationship can be used (e.g., being two times slower will make the usability score four times worse). In another example, time expectations (boot time or application startup time) can be used instead of calculated means. Using time expectations will reduce the user score and user experience score of computing devices that take longer to perform a task than the user expects, even if the particular computing device performs at a faster-than-average speed. Other variations are also possible.

[0116] Any one or more of the administrator change recommendation logic 112, the administrator change recommendation logic 602, the analysis logic 604, the determination logic 606, the cross-reference logic 608, the recommendation logic 610, the table logic 612, the query logic 614, the administrator change logic 616, the discount logic 618, the flowchart 300, the flowchart 400 and / or the flowchart 500 may be implemented in hardware, software, firmware, or any combination thereof.

[0117] For example, any one or more of the administrator change recommendation logic 112, the administrator change recommendation logic 602, the analysis logic 604, the determination logic 606, the cross-reference logic 608, the recommendation logic 610, the table logic 612, the query logic 614, the administrator change logic 616, the discount logic 618, the flowchart 300, the flowchart 400 and / or the flowchart 500 may be implemented at least in part as computer program code configured to be executed in one or more processors.

[0118] In another example, any one or more of the administrator change recommendation logic 112, the administrator change recommendation logic 602, the analysis logic 604, the determination logic 606, the cross-reference logic 608, the recommendation logic 610, the table logic 612, the query logic 614, the administrator change logic 616, the discount logic 618, the flowchart 300, the flowchart 400, and / or the flowchart 500 may be implemented at least in part as hardware logic / circuitry. Such hardware logic / circuitry may include one or more hardware logic components. Examples of hardware logic components include, but are not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SoCs), complex programmable logic devices (CPLDs), and the like. For example, a SoC may include an integrated circuit chip that includes one or more processors (e.g., microcontrollers, microprocessors, digital signal processors (DSPs), etc.), memory, one or more communication interfaces, and / or other circuits and / or embedded firmware to perform its functions.

[0119] III. Further Discussion of Some Example Embodiments

[0120] An example system for providing administrator change recommendations includes a memory and one or more processors, the one or more processors being coupled to the memory. The one or more processors are configured to collect information indicating administrator changes made by an administrator in an enterprise's environment and further indicating the value of a metric generated by the administrator changes. The one or more processors are also configured to analyze the information and classify the information into multiple categories based at least in part on one or more features of the environment in which the administrator changes were made. The one or more processors are also configured to determine the type of expected administrator changes that the administrator is about to perform on the enterprise based at least in part on the operations performed by the administrator using an enterprise management tool. The one or more processors are also configured to cross-reference the type of expected administrator changes with the information to identify multiple subsets of information corresponding to the type. The one or more processors are also configured to analyze the subset based at least in part on the type of expected administrator changes and the information in the subset, inferring a causal relationship between one or more administrator changes made after the type of administrator changes and an increase in one or more values ​​of one or more corresponding metrics indicated by the information in the subset. The one or more processors are also configured to recommend a recommended administrator change to be performed by an administrator based at least in part on the one or more categories into which at least one of the subsets is classified and further based at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the one or more values ​​of at least one of the one or more corresponding metrics, the information in at least one of the subsets indicating that the recommended administrator change has been made after an administrator change of the type.

[0121] In a first aspect of an example system, one or more processors are configured to, for each administrator change indicated by information, determine, in response to the administrator change, an amount by which a measured value of a property associated with at least one of a plurality of end users or a plurality of devices for which the administrator change was made. According to the first aspect, the one or more processors are configured to further recommend a recommended administrator change to be performed by the administrator based at least in part on the amount by which the measured value of the property for each administrator change associated with the information in the subset is changed.

[0122] In a first implementation of the first aspect of the example system, one or more processors are configured to collect data about at least one of one or more regional events or one or more global events. During the time period for which the information is collected, each regional event affects the behavior of enterprise end users within a limited geographic area. During the time period for which the information is collected, each global event affects the behavior of enterprise end users globally. According to the first implementation, the one or more processors are configured to determine the extent to which at least one of the one or more regional events or the one or more global events changes the measured value of a property for each administrator to change. Further according to the first implementation, the one or more processors are configured to recommend a recommended administrator change to be performed by the administrator based at least in part on the extent to which at least one of the one or more regional events or the one or more global events changes the measured value of the property for each administrator to change that is involved in the information in the subset.

[0123] In a second implementation of the first aspect of the example system, the one or more processors are configured to analyze historical data about administrator changes made by an administrator to determine that a subset of the plurality of parameters collected about the administrator changes may contribute to a change in a corresponding measured value of the attribute. According to the second implementation, the one or more processors are configured to limit the collected information to the subset of the plurality of parameters.

[0124] In a third implementation of the first aspect of the example system, the amount of change in the measured value of an attribute associated with at least one of the multiple end users or the multiple devices targeted by the administrator change includes at least one of the following: the amount of change in a user experience score associated with at least one of the multiple end users or the multiple devices targeted by the administrator change, wherein the user experience score indicates a quality of user experience associated with at least one of the multiple end users or the multiple devices; the amount of change in end-user productivity of the multiple end users that are the target of the administrator change; or the extent of the security change of the multiple devices targeted by the administrator change.

[0125] In a second aspect of the example system, the one or more processors are configured to classify the information in each category into a plurality of subcategories corresponding to a plurality of corresponding tasks performed by an administrator in the environment. According to the second aspect, the one or more processors are configured to recommend a recommended administrator change to be performed by the administrator based further at least in part on the subcategories into which the subsets are classified. The second aspect of the example system can be implemented in conjunction with the first aspect of the example system, although the example embodiments are not limited in this respect.

[0126] In a third aspect of the example system, the one or more processors are configured to determine whether the person has approved the recommendation of the recommended administrator change to be performed by the administrator. According to the third aspect, the one or more processors are configured to recommend the recommended administrator change to be performed by the administrator based further at least in part on the determination that the person has approved the recommendation of the recommended administrator change to be performed by the administrator. The third aspect of the example system may be implemented in conjunction with the first aspect and / or the second aspect of the example system, although the example embodiments are not limited in this respect.

[0127] In a fourth aspect of the example system, one or more processors are configured to collect historical rating data indicating changes in ratings of applications from a rating system external to the enterprise, the changes resulting from updates to the applications. According to the fourth aspect, the one or more processors are configured to recommend recommended administrator changes to be performed by the administrator based further at least in part on the historical rating data. The fourth aspect of the example system may be implemented in conjunction with the first aspect, the second aspect, and / or the third aspect of the example system, although the example embodiments are not limited in this respect.

[0128] In a fifth aspect of the example system, one or more processors are configured to collect feedback data indicating ratings of previously recommended administrator changes. Each rating is received from an administrator who received a recommendation to perform a corresponding previously recommended administrator change. Each rating indicates a quality of a user experience resulting from the performance of the corresponding previously recommended administrator change in response to receiving the recommendation. According to the fifth aspect, the one or more processors are configured to recommend a recommended administrator change to be performed by the administrator based further at least in part on the ratings of the previously recommended administrator changes. The fifth aspect of the example system can be implemented in combination with the first aspect, the second aspect, the third aspect, and / or the fourth aspect of the example system, although the example embodiments are not limited in this respect.

[0129] In a sixth aspect of the example system, the one or more processors are configured to collect feedback data indicating whether a previously recommended administrator change was performed by the administrator in response to receiving a recommendation to perform the previously recommended administrator change. According to the sixth aspect, the one or more processors are configured to recommend a recommended administrator change to be performed by the administrator based further at least in part on whether the previously recommended administrator change was performed by the administrator in response to receiving the recommendation. The sixth aspect of the example system may be implemented in conjunction with the first, second, third, fourth, and / or fifth aspects of the example system, although the example embodiments are not limited in this respect.

[0130] In a seventh aspect of the example system, the one or more processors are configured to determine a proportion of administrators who have performed the recommended administrator change among administrators who have received recommendations to perform the recommended administrator change. According to the seventh aspect, the one or more processors are configured to notify the administrator of the proportion of administrators who have performed the recommended administrator change. The seventh aspect of the example system can be implemented in conjunction with the first, second, third, fourth, fifth, and / or sixth aspects of the example system, although the example embodiments are not limited in this respect.

[0131] In an implementation of the seventh aspect of the example system, the one or more processors are configured to determine whether a proportion of administrators that have performed the recommended administrator change is greater than or equal to a threshold. According to this implementation, the one or more processors are configured to notify the administrator of the proportion of administrators that have performed the recommended administrator change based at least in part on a determination that the proportion is greater than or equal to the threshold.

[0132] In an eighth aspect of the example system, the one or more processors are configured to use machine learning techniques to analyze the recommendations of the recommended administrator changes made to the administrator and the expected administrator changes and information categories on which the recommendations of the recommended administrator changes are based to determine the recommended administrator changes to recommend. The eighth aspect of the example system can be implemented in conjunction with the first, second, third, fourth, fifth, sixth, and / or seventh aspects of the example system, although the example embodiments are not limited in this respect.

[0133] In a ninth aspect of the example system, one or more processors are configured to collect user information about user changes made on corresponding client devices of corresponding terminal users in an enterprise based at least in part on instructions from corresponding terminal users, each instruction indicating that a corresponding change is to be made on the corresponding client device. According to the ninth aspect, the one or more processors are configured to analyze user information and classify user information into multiple categories based at least in part on one or more characteristics of the environment in which the user changes are made. Further according to the ninth aspect, the one or more processors are configured to cross-reference the type of expected administrator changes with the information and the user information to identify multiple subsets corresponding to the type. The ninth aspect of the example system can be implemented in combination with the first aspect, the second aspect, the third aspect, the fourth aspect, the fifth aspect, the sixth aspect, the seventh aspect and / or the eighth aspect of the example system, although the example embodiments are not limited in this respect.

[0134] In an example method, a type of expected administrator change to be performed by an administrator to an enterprise is determined based at least in part on an operation performed by the administrator using an enterprise management tool. The type of expected administrator change is cross-referenced with information indicating an administrator change made by an administrator in a corresponding environment of the corresponding enterprise and further indicating a value of a metric produced by the administrator change to identify a plurality of subsets of information corresponding to the type. By analyzing the subset based at least in part on cross-referencing the type of expected administrator change with the information in the subset, a causal relationship between one or more administrator changes made after an administrator change of the type indicated by the information in the subset and an increase in one or more values ​​of one or more corresponding metrics indicated by the information in the subset is inferred. Based at least in part on one or more categories into which at least one of the subsets is classified and further at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the one or more values ​​of at least one of the one or more corresponding metrics, a recommended administrator change is recommended to be performed by the administrator, the information in at least one of the subsets indicating that the recommended administrator change has been made after the type of administrator change. The information is classified into a plurality of categories, the plurality of categories including one or more categories based at least in part on one or more characteristics of the environment in which the administrator change was made.

[0135] In a first aspect of the example method, recommending the recommended administrator change includes recommending the recommended administrator change via an administrator change recommendation service. According to the first aspect, the example method also includes providing the administrator with a discount on the cost of using the administrator change recommendation service. The discount depends on whether the administrator allows the information collected about the administrator changes made by the administrator to facilitate the information collected about the administrator changes made by the administrator in the corresponding environment of the corresponding enterprise.

[0136] In a second aspect of the example method, the example method further includes determining that an administrator initiated execution of the recommended administrator change. According to the second aspect, the example method further includes populating a plurality of fields in an electronic form based at least in part on the determination that the administrator initiated execution of the recommended administrator change, the plurality of fields defining a plurality of respective attributes of the recommended administrator change. The second aspect of the example method may be implemented in conjunction with the first aspect of the example method, although the example embodiments are not limited in this respect.

[0137] In an implementation of the second aspect of the example method, the example method further includes providing a query to the administrator. The query requests confirmation that the recommended administrator change is to be performed according to the attributes defined by the corresponding field. According to this implementation, the example method further includes performing the recommended administrator change according to the attributes defined by the corresponding field based at least in part on the confirmation received from the administrator.

[0138] In a third aspect of the example method, the example method further includes determining that the person has approved the recommendation of the recommended administrator change to be performed by the administrator. According to the third aspect, recommending the recommended administrator change includes recommending the recommended administrator change to be performed by the administrator based further at least in part on the determination that the person has approved the recommendation of the recommended administrator change to be performed by the administrator. The third aspect of the example method may be implemented in conjunction with the first aspect and / or the second aspect of the example method, although the example embodiments are not limited in this respect.

[0139] An example computer program product includes a computer-readable storage medium having recorded thereon instructions for enabling a processor-based system to perform operations to provide administrator change recommendations. The operations include determining a type of expected administrator change that an administrator is about to perform on an enterprise based at least in part on operations performed by the administrator using an enterprise management tool. The operations also include cross-referencing the type of expected administrator change with information indicating administrator changes made by the administrator in a corresponding environment of the corresponding enterprise and further indicating values ​​of metrics generated by the administrator changes to identify multiple subsets of information corresponding to the type. The operations also include inferring a causal relationship between one or more administrator changes made after an administrator change of the type indicated by the information in the subset and an increase in one or more values ​​of one or more corresponding metrics indicated by the information in the subset by analyzing the subset based at least in part on cross-referencing the type of expected administrator change with the information in the subset. The operations also include recommending a recommended administrator change to be performed by the administrator based at least in part on one or more categories into which at least one of the subsets is classified and further based at least in part on a causal relationship between the recommended administrator change and an increase in at least one of the one or more values ​​of at least one of the one or more corresponding metrics, the information in at least one of the subsets indicating that the recommended administrator change has been made after an administrator change of the type. The information is categorized into a plurality of categories including one or more categories based at least in part on one or more characteristics of an environment in which the administrator change was made.

[0140] IV. Example Computer System

[0141] Figure 8 An example computer 800 is depicted in which various embodiments may be implemented. Figure 1 Any one or more of the user devices 106A-106M shown in, any one or more of the administrator system(s) 104A-104M, and / or any one or more of the system management servers 102; Figure 2Any one or more of the system management server(s) 202, any one or more of the clients 204A-204D, and / or any one or more of the client machines 206A-206D shown in FIG. 1 ; and / or Figure 6 The computing system 600 shown in the figure can be implemented using a computer 800 that includes one or more features of computer 800 and / or alternative features. Computer 800 can be a general-purpose computing device in the form of, for example, a conventional personal computer, a mobile computer, or a workstation, or computer 800 can be a special-purpose computing device. The description of computer 800 provided herein is provided for the purpose of illustration and not for limitation. As known to those skilled in the relevant art, the embodiments can be implemented in other types of computer systems.

[0142] like Figure 8 As shown in , computer 800 includes a processing unit 802, a system memory 804, and a bus 806, which couples various system components including the system memory 804 to the processing unit 802. Bus 806 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. System memory 804 includes read-only memory (ROM) 808 and random access memory (RAM) 810. Basic input / output system 812 (BIOS) is stored in ROM 808.

[0143] The computer 800 also has one or more of the following drives: a hard disk drive 814 for reading and writing to a hard disk; a magnetic disk drive 816 for reading and writing to a removable magnetic disk 818; and an optical disk drive 820 for reading and writing to a removable optical disk 822 such as a CD ROM, DVD ROM or other optical media. The hard disk drive 814, magnetic disk drive 816 and optical disk drive 820 are connected to the bus 806 via a hard disk drive interface 824, a magnetic disk drive interface 826 and an optical disk drive interface 828, respectively. The drive and its associated computer-readable storage medium provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computer. Although a hard disk, a removable magnetic disk and a removable optical disk are described, other types of computer-readable storage media may be used to store data, such as a flash memory card, a digital video disk, a random access memory (RAM), a read-only memory (ROM), etc.

[0144] Many program modules can be stored on a hard disk, disk, optical disk, ROM or RAM. These programs include an operating system 830, one or more application programs 832, other program modules 834 and program data 836. Applications 832 or program modules 834 may include, for example, computer program logic for implementing administrator change recommendation logic 112, administrator change recommendation logic 602, analysis logic 604, determination logic 606, cross-reference logic 608, recommendation logic 610, table logic 612, query logic 614, administrator change logic 616, discount logic 618, flowchart 300 (including any steps of flowchart 300), flowchart 400 (including any steps of flowchart 400) and / or flowchart 500 (including any steps of flowchart 500), as described herein.

[0145] A user may enter commands and information into the computer 800 through input devices such as a keyboard 838 and a pointing device 840. Other input devices (not shown) may include a microphone, a joystick, a game pad, a satellite dish, a scanner, a touch screen, a camera, an accelerometer, a gyroscope, etc. These and other input devices are often coupled to the processing unit 802 through a serial port interface 842 that is coupled to the bus 806, but may be coupled through other interfaces, such as a parallel port, a game port, or a universal serial bus (USB).

[0146] A display device 844 (eg, monitor) is also connected to bus 806 via an interface, such as a video adapter 846. In addition to display device 844, computer 800 may include other peripheral output devices (not shown), such as speakers and printers.

[0147] The computer 800 is connected to a network 848 (eg, the Internet) through a network interface 850 or adapter, a modem 852, or other device for establishing communications over the network. The modem 852 may be internal or external and is connected to the bus 806 via the serial port interface 842.

[0148] As used herein, the terms "computer program medium" and "computer-readable storage medium" are used to refer generally to media (e.g., non-transitory media) such as a hard disk associated with hard disk drive 814, removable disk 818, removable optical disk 822, and other media such as flash memory cards, digital video disks, random access memory (RAM), read-only memory (ROM), etc. Such computer-readable storage media are distinguished from and do not overlap with (do not include) communication media. Communication media contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave. The term "modulated data signal" refers to a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared, and other wireless media, as well as wired media. Example embodiments are also directed to such communication media.

[0149] As described above, computer programs and modules (including application programs 832 and other program modules 834) can be stored on a hard disk, disk, optical disk, ROM or RAM. Such computer programs can also be received via the network interface 850 or the serial port interface 842. When executed or loaded by the application, such computer programs enable the computer 800 to implement the features of the embodiments discussed herein. Therefore, such computer programs represent the controller of the computer 800.

[0150] Example embodiments are also directed to a computer program product comprising software (e.g., computer-readable instructions) stored on any computer-usable medium. When executed in one or more data processing devices, such software causes (one or more) data processing devices to operate as described herein. Embodiments may employ any computer-usable or computer-readable medium known now or in the future. Examples of computer-readable media include, but are not limited to, storage devices such as RAM, hard disk drives, floppy disks, CD ROMs, DVD ROMs, zip disks, magnetic tapes, magnetic storage devices, optical storage devices, MEMS-based storage devices, nanotechnology-based storage devices, and the like.

[0151] It will be appreciated that the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.

[0152] V. Conclusion

[0153] Although the subject matter has been described in language specific to structural features and / or actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to be within the scope of the claims.

Claims

1. A system for providing administrator change recommendations, the system comprising: Memory; as well as a processing system coupled to the memory, the processing system being configured to: analyzing historical data regarding historian changes made in a specified environment of a specified enterprise to determine that a subset of a plurality of parameters collected regarding the historian changes can contribute to a change in a corresponding measured value of a property associated with at least one of a plurality of end users or a plurality of devices that are targets of the historian changes; collecting information by limiting the collected information to the subset of the plurality of parameters, the information indicating administrator changes made in a specified environment of a specified enterprise and further indicating values ​​of metrics resulting from the administrator changes; analyzing the information and classifying the information into a plurality of categories based at least in part on at least one characteristic of the specified environment in which the administrator change was made; determining a type of intended administrator change to be performed on the first enterprise based at least in part on the enterprise management tool operation; cross-referencing the types of the expected administrator changes with the information to identify a plurality of subsets of the information to which the types correspond; inferring a causal relationship between at least one administrator change to a second configuration of the second enterprise made subsequent to an administrator change of the type indicated by the information in the subset and an increase in a value of a metric indicated by the information in the subset by analyzing the subset based at least in part on cross-referencing the type of the expected administrator change with the information in the subset; as well as Based at least in part on information from at least one of the subsets indicating that a specified administrator change to the second configuration of the second enterprise was made after the type of administrator change was made to the first configuration of the second enterprise, and further based at least in part on a causal relationship between the specified administrator change and an increase in the value of the metric, and further based at least in part on at least one category into which the at least one of the subsets is classified, recommending the specified administrator change to be performed on the configuration of the first enterprise as a result of determining that the expected administrator change is to be performed on the first enterprise.

2. The system of claim 1, wherein the processing system is configured to: for each administrator change indicated by the information, determining an amount by which a measured value of a property associated with at least one of a plurality of end users or a plurality of devices that is a target of the administrator change is changed in response to the administrator change; and The specified administrator change to be performed is further recommended based at least in part on the amount by which the measured value of the attribute for each administrator change is changed associated with the information in the subset.

3. The system of claim 2, wherein the amount by which the measured value of the attribute associated with at least one of the plurality of end users or the plurality of devices that is a target of the administrator's change comprises: An amount by which a user experience score associated with at least one of the plurality of end users or the plurality of devices that is a target of the administrator change changes, the user experience score indicating a quality of the user experience associated with the plurality of end users or the at least one of the plurality of devices.

4. The system of claim 2, wherein the amount by which the measured value of the attribute associated with at least one of the plurality of end users or the plurality of devices that is a target of the administrator's change comprises: An amount of change in end-user productivity of the plurality of end-users that are the target of the administrator change.

5. The system of claim 2, wherein the processing system is configured to: collecting said information including data about at least one of a regional event or a global event, The regional event affects the behavior of end users of enterprises within a limited geographic area during the time period during which the information is collected, said global event impacting the behavior of end users of enterprises around the world during said time period in which said information is collected; determining an extent of an amount of change in a measured value of a property changed by each administrator for said at least one of said regional event or said global event; as well as Recommending the designated administrator change to be performed on the configuration of the first enterprise as a result of determining that the intended administrator change is to be performed on the first enterprise based at least in part on information from at least one of the subsets indicating that a designated administrator change to the second configuration of the second enterprise was performed after the type of administrator change was performed on the first configuration of the second enterprise, and further based at least in part on a causal relationship between the designated administrator change and an increase in the value of the metric, and further based at least in part on at least one category into which the at least one of the subsets is classified, and further based at least in part on the extent to which the measured value of the attribute for each administrator change associated with the information in the subset is changed by the at least one event from the regional event or the global event.

6. The system of claim 1, wherein the processing system is configured to: categorizing the information in each category into a plurality of subcategories corresponding to a plurality of respective tasks performed in the specified environment; and Based further at least in part on the subcategory into which the subset is categorized, the designated administrator change to be performed is recommended.

7. The system of claim 1, wherein the processing system is configured to: determining whether a human has approved a recommendation for the designated administrator change to be performed to change the configuration of the first enterprise; and The designated administrator change to be performed is recommended based further at least in part on determining that the person has approved the recommendation of the designated administrator change to be performed.

8. The system of claim 1, wherein the processing system is configured to: collecting feedback data indicating ratings of previously recommended administrator changes, each rating indicating a quality of a user's experience resulting from performing a respective previously recommended administrator change in response to a recommendation to perform the respective previously recommended administrator change; and The specified administrator change to be performed is recommended further based at least in part on the rating of the previously recommended administrator change.

9. The system of claim 1, wherein the processing system is configured to: collecting feedback data indicating whether a previously recommended administrator change was performed in response to receiving a recommendation to perform the previously recommended administrator change; and The specified administrator change to be performed is recommended further based at least in part on whether the previously recommended administrator change was performed in response to receiving the recommendation.

10. The system of claim 1, wherein the processing system is configured to: determining a proportion of administrators who have received a recommendation to perform the designated administrator change and administrators who have performed the designated administrator change; and Notification of the proportion of the administrators who have performed the designated administrator change is provided.

11. The system of claim 10, wherein the processing system is configured to: determining whether the proportion of the administrators who have performed the designated administrator change is greater than or equal to a threshold; and Based at least in part on a determination that the proportion is greater than or equal to the threshold, providing the notification of the proportion of the administrators that have performed the designated administrator change.

12. The system of claim 1, wherein the processing system is configured to: Recommendations for designated administrator changes, as well as anticipated administrator changes, and categories of information upon which the recommendations for the designated administrator changes are based are analyzed using machine learning techniques to determine the designated administrator changes to recommend.

13. The system of claim 1, wherein the processing system is configured to: collecting user information regarding user changes made on respective client devices of respective end users in the enterprise based at least in part on instructions from the respective end users, each instruction indicating that a respective change is to be made on the respective client device; as well as analyzing the user information and classifying the user information into the plurality of categories based at least in part on at least one characteristic of the environment in which the user change was made; as well as The type of the expected administrator change is cross-referenced with the information and the user information to identify the plurality of subsets to which the type corresponds.

14. The system of claim 1, wherein the processing system is configured to: Perform the specified administrator changes as described.

15. A method for providing administrator change recommendations, the method comprising: analyzing historical data regarding historian changes made in a specified environment of a specified enterprise to determine that a subset of a plurality of parameters collected regarding the historian changes can contribute to a change in a corresponding measured value of a property associated with at least one of a plurality of end users or a plurality of devices that are targets of the historian changes; collecting information by limiting the collected information to the subset of the plurality of parameters, the information indicating administrator changes made in a specified environment of a specified enterprise and further indicating values ​​of metrics resulting from the administrator changes; determining a type of intended administrator change to be performed on the first enterprise based at least in part on the enterprise management tool operation; cross-referencing the types of the expected administrator changes with information indicating the administrator changes made in respective specified environments of respective specified enterprises and further indicating values ​​of metrics produced by the administrator changes to identify a plurality of subsets of the information to which the types correspond; inferring a causal relationship between at least one administrator change to a second configuration of the second enterprise made subsequent to an administrator change of the type indicated by the information in the subset and an increase in a value of a metric indicated by the information in the subset by analyzing the subset based at least in part on cross-referencing the type of the expected administrator change with the information in the subset; as well as Based at least in part on information from at least one of the subsets indicating that a specified administrator change to the second configuration of the second enterprise was made after the type of administrator change was made to the first configuration of the second enterprise, and further based at least in part on a causal relationship between the specified administrator change and an increase in the value of the metric, and further based at least in part on at least one category into which the at least one subset of the subsets is classified, recommending the specified administrator change to be performed to change the configuration of the first enterprise as a result of determining that the expected administrator change is to be performed on the first enterprise, wherein the information is classified into multiple categories including the at least one category based at least in part on at least one characteristic of the specified environment in which the administrator change was made.

16. The method of claim 15, wherein recommending the designated administrator change comprises: recommending the designated administrator change via an administrator change recommendation service; and The method further comprises: A cost discount for using the administrator change recommendation service is provided, the discount being contingent upon allowing information about administrator changes made to the first enterprise to be collected to facilitate information about the administrator changes made in the specified environment of the specified enterprise.

17. The method according to claim 15, further comprising: Determining that execution of the designated administrator change is initiated; as well as Based at least in part on a determination that execution of the designated administrator change is initiated, a plurality of fields are populated in a spreadsheet, the plurality of fields defining a plurality of respective properties of the designated administrator change.

18. The method according to claim 17, further comprising: providing a query requesting confirmation that the specified administrator changes are to be performed according to the attributes defined by the corresponding fields; as well as Based at least in part on receiving the confirmation, the designated administrator change is performed according to the attributes defined by the corresponding fields.

19. The method according to claim 15, further comprising: determining that the person has approved the recommendation for said designated administrator change to be implemented; Recommended changes for designated administrators include: The designated administrator change to be performed is recommended based further at least in part on determining that the person has approved the recommendation for the designated administrator change to be performed.

20. The method of claim 15, further comprising: collecting said information including data regarding at least one of a regional event that impacted behavior of end-users of businesses within a limited geographic region during a time period during which said information was collected or a global event that impacted behavior of end-users of businesses throughout the world during said time period during which said information was collected; as well as determining a degree of said amount of change of said measured value of said attribute for each administrator change of said at least one of said regional event or said global event; The designated administrator change to be performed is recommended further based at least in part on the extent to which the measured value of the attribute for each administrator change is changed associated with the at least one of the regional event or the global event.

21. A computer program product comprising a non-transitory computer-readable storage medium having recorded thereon instructions for enabling a processor-based system to perform operations to provide administrator change recommendations, the operations comprising: analyzing historical data regarding historian changes made in a specified environment of a specified enterprise to determine that a subset of a plurality of parameters collected regarding the historian changes can contribute to a change in a corresponding measured value of a property associated with at least one of a plurality of end users or a plurality of devices that are targets of the historian changes; collecting information by limiting the collected information to the subset of the plurality of parameters, the information indicating administrator changes made in a specified environment of a specified enterprise and further indicating values ​​of metrics resulting from the administrator changes; determining a type of intended administrator change to be performed on the first enterprise based at least in part on the enterprise management tool operation; cross-referencing the types of the expected administrator changes with information indicating the administrator changes made in respective specified environments of respective specified enterprises and further indicating values ​​of metrics produced by the administrator changes to identify a plurality of subsets of the information to which the types correspond; inferring a causal relationship between at least one administrator change to a second configuration of the second enterprise made subsequent to an administrator change of the type indicated by the information in the subset and an increase in a value of a metric indicated by the information in the subset by analyzing the subset based at least in part on cross-referencing the type of the expected administrator change with the information in the subset; as well as Based at least in part on information from at least one of the subsets indicating that a specified administrator change to the second configuration of the second enterprise was made after the type of administrator change was made to the first configuration of the second enterprise, and further based at least in part on a causal relationship between the specified administrator change and an increase in the value of the metric, and further based at least in part on at least one category into which the at least one subset of the subsets is classified, recommending the specified administrator change to be performed to change the configuration of the first enterprise as a result of determining that the expected administrator change is to be performed on the first enterprise, wherein the information is classified into multiple categories including the at least one category based at least in part on at least one characteristic of the specified environment in which the administrator change was made.

Images