Dynamically monitor cloud-based application services

Abstract

In a non-limiting example of the present disclosure, systems, methods, and devices for monitoring telemetry data from a cloud-based application service are provided. Telemetry data for a plurality of operations of the cloud-based application service may be analyzed, wherein the analysis includes comparing a first time series with a second time series, and wherein data from the second time series is associated with an operation that was performed earlier in time than an execution of the operation associated with the first time series. Based on the analysis, one or more operational changes in the cloud-based application service may be identified, and at least one telemetry monitor may be dynamically configured based on the identified one or more operational changes.

Description

Background Art

[0001] As computing moves increasingly to the cloud, the systems that support the large number of users and cloud-based applications that they use are constantly being modified. The infrastructure of cloud-based systems requires constant monitoring to maintain, let alone update and add additional features and functionality. As new software builds are constantly added to the infrastructure of cloud-based systems through the development cycle and eventually to production environments, modifying existing monitors and creating new monitors to keep up with the ever-changing builds becomes increasingly resource-intensive and difficult.

[0002] It is against this general technical environment that aspects of the technology disclosed herein are considered.In addition, although a general environment has been discussed, it should be understood that the examples described herein should not be limited to the general environment identified in the background. Summary of the invention

[0003] This summary is provided to introduce in simplified form a selection of concepts that will be further described in the detailed description section below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to serve as an aid in determining the scope of the claimed subject matter. Other aspects, features, and / or advantages of the examples will be set forth in part in the subsequent description and in part will be apparent from the description or may be learned by practice of the disclosure.

[0004] The non-limiting examples of the present disclosure describe systems, methods, and devices for automatically identifying software modifications in cloud-based application services and dynamically configuring operation monitors in those environments. Based on the analysis of time series information from a telemetry service associated with or integrated in a cloud-based application service, monitors can be automatically generated, modified, and / or deleted. The telemetry service can receive operation data, including operation logs, from operations performed by users of the cloud-based application service. It can be determined that the monitor should be generated, modified, and / or deleted by automatically comparing the time series data before the software update with the time series data after the software update. When a new monitor is dynamically generated and / or modified, the dynamic monitor engine can determine the appropriate monitoring technology, baseline operation range, and / or threshold for marking the operation for further review for each corresponding operation type. The dynamic monitor engine can apply one or more machine learning models when making these determinations and setting these ranges and thresholds. The dynamic monitor engine can apply these models in the context of the processing resources available to the cloud-based application service, thereby allocating operation analysis bandwidth to each monitor according to the resources available in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting and non-exhaustive examples are described with reference to the following figures:

[0006] Figure 1is a schematic diagram illustrating an example distributed computing environment for dynamically configuring monitors for cloud-based application services.

[0007] Figure 2 A basic flow chart for dynamically configuring a monitor for a cloud-based application service is shown.

[0008] Figure 3 A graphical display of operational data for a cloud-based application service is shown, wherein dynamically configured monitors are applied to quality of service metrics.

[0009] Figure 4 A graphical display of operational data for a cloud-based application service is shown, with a dynamically configured monitor applied to a total unexpected failure metric.

[0010] Figure 5 An exemplary method for dynamically configuring a monitor for a cloud-based application service is provided.

[0011] Figure 6 and Figure 7 is a simplified diagram of a mobile computing device that can be used to practice aspects of the present disclosure.

[0012] Figure 8 is a block diagram illustrating example physical components of a computing device that can be used to practice aspects of the present disclosure.

[0013] Fig. 9 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0014] Various embodiments will be described in detail with reference to the accompanying drawings, wherein like reference numerals denote like parts and components throughout the several views. Reference to various embodiments does not limit the scope of the appended claims. In addition, any examples set forth in this specification are not intended to limit and merely set forth some of the many possible embodiments of the appended claims.

[0015] The various embodiments and examples described above are provided by way of illustration only and should not be construed as limiting the appended claims. Those skilled in the art will readily appreciate that various modifications and changes may be made without following the example embodiments and applications shown and described herein and without departing from the true spirit and scope of the claims.

[0016] Examples of the present disclosure provide systems, methods, and devices for dynamically creating monitors for software implemented in a cloud-based application service. In an example, the monitor may provide a mechanism for identifying code regressions and / or other service function losses (e.g., server problems, network problems, etc.) in the implemented software. For example, code regressions may be included in new software packages, updates, and / or patches. In some examples, code regressions may be associated with one or more cloud-based applications, such as cloud-based document processing applications, spreadsheet applications, calendar applications, presentation applications, storage applications, video applications, real-time electronic messaging applications, voice messaging applications, video communication applications, and / or email applications.

[0017] In some examples, the monitor may analyze signals associated with operational failures with respect to one or more cloud-based applications. For example, when an operation for a cloud-based application fails, and / or an operation for a cloud-based application causes the application to crash or fail when it is executed, a signal indicating the presence of an operational event or operational failure may be reported to the monitor and / or a telemetry service associated with the monitor. The monitor may receive and analyze data associated with the execution of one or more operations of an application hosted by a cloud-based application service. For example, the monitor may analyze data associated with a "save" operation, a "send" operation, a "new document" operation, a "copy" operation, a "paste" operation, and any other operation that may be performed by a cloud-based application.

[0018] The telemetry service can receive and store information associated with operations that have been performed by applications hosted by the cloud-based service. The telemetry service can store information about each performed operation, including: the time each operation was performed; an identification code for each operation performed; the duration that each operation took to complete or time out; whether each operation was successful or unsuccessful; an indication of whether a monitor is receiving data associated with each operation; the name of each monitor associated with each operation; the server or server cluster that performed each operation, etc. Thus, the telemetry service can maintain a continuous time series of operation data, which includes numerical values ​​for successfully performed operations and numerical values ​​for unsuccessfully performed operations. The monitor can use this information to flag operations that may be associated with code regressions and / or other issues such as network and / or hardware issues.

[0019] According to an example, when a software update is pushed to a cloud-based application service (e.g., one or more applications hosted by the service are modified), a telemetry analysis and comparison engine associated with the cloud-based application service can compare one or more operational data time series within a duration just before the software update with a time series corresponding to the time after the software update has been implemented to the service. The telemetry analysis and comparison engine can check the operation logs of each time series and compare them with each other to determine which operations exist in the time series before the software update and which operations exist in the time series after the software update. In this way, it can be determined which operations in the time series after the software update have been added, modified and / or deleted compared to the time series before the software update. For any operation that has been added via a software update, a dynamic monitor engine can generate a new monitor for marking potential problems associated with those newly added operations. For any operation that has been modified via a software update, a dynamic monitor engine can modify an existing monitor for marking potential problems associated with those modified operations. For any operation that has been deleted via a software update, a dynamic monitor engine can delete each corresponding monitor that exists in the time series before the software update.

[0020] When dynamically generating new monitors and modifying existing monitors, the dynamic monitor engine can analyze the time series operation data associated with the operation for which the monitor is being generated and modified. For example, when a new monitor for a new operation is being generated, the dynamic monitor engine can analyze the operation data of the operation log including the new operation within a duration. That is, the dynamic monitor can analyze at least one time series of the operation. The dynamic monitor can use one or more machine learning models to determine the baseline that can be used by the monitor. The baseline can be related to the number of operations that are successfully or unsuccessfully executed over time, the percentage of operations that are successfully executed over time, and / or the duration (i.e., delay) spent to complete each operation within a set amount of time. The dynamic monitor engine can use time series data to identify the standardized range of one or more metrics in those metrics. For example, the dynamic monitor engine can determine that a baseline of 0-1000 unexpected failures per five minutes should be established for the first operation; a baseline for completing operations between 50 milliseconds and 150 milliseconds should be established; and / or a baseline of 90-100% successfully executed operation requests should be established for the operation. These are just examples, and it should be understood that various machine learning models applied to different data sets can dynamically establish different standardized baselines. In addition, each baseline established can be dynamic or static. That is, for a dynamic baseline, the baseline may change based on various factors, including the time of day, the day of the week, the month, and other contextual data. Alternatively, a static baseline can remain unchanged regardless of the contextual data associated with it.

[0021] In some examples, when generating and / or modifying the monitor, the dynamic monitor engine can determine a threshold outside the baseline to mark the operation for further analysis and / or review. In other examples, the threshold can be set manually. For example, a threshold can be set for the number of operation failures received every X minutes exceeding the baseline number of operation failures received every X minutes. In another example, a threshold can be set for the lower limit of the baseline percentage of the operation that the operation must be lower than for the monitor to mark the operation for the success percentage of the operation every X minutes. In yet another example, a threshold can be set for a latency metric so that a specific duration exceeding the upper baseline latency must be met in order for the monitor to mark the operation of the operation. In an additional example, unless the time series for the operation exceeds two or more thresholds (e.g., two or more thresholds in the latency threshold, the unsuccessful operation threshold, and / or the success percentage threshold), the operation may not be marked.

[0022] The systems, methods, and devices described herein provide technical advantages for detecting code regressions and problems that affect functionality in cloud-based application services. The dynamic and automated nature of the described mechanisms reduces the memory and processing costs (CPU cycles) associated with accurately detecting code regressions in software builds for large cloud-based application services. In addition, the time and human resources required to create, modify, and / or delete monitors for such services are significantly reduced. The described mechanisms provide a way for cloud-based services to automatically generate, modify, and delete operational monitors on the fly, so that when new software updates are pushed out to those services, operations can be immediately monitored for code regressions without taking up the time and resources of human developers. Even if there is a strong developer pool available, it would be difficult or even impossible to keep the current state of monitors for the constantly changing software in large cloud-based application services without the currently described mechanisms. The dynamic nature of the described mechanism also provides a way for each monitor to take into account the processing resources available to the system when determining how many time series should be analyzed for each operation, how long each analyzed time series should be, how many operations should and can be monitored based on the constraints of the system, and what baselines and thresholds should be applied to mark each operation for further review (e.g., if fewer resources are available, the baseline and / or thresholds may be more generous).

[0023] Figure 11 is a schematic diagram showing an example distributed computing environment 100 for dynamically configuring a monitor for a cloud-based application service. The distributed computing environment 100 includes a service modification subenvironment 101, a network and processing subenvironment 136, and a dynamic monitor subenvironment 130. The network and processing subenvironment 136 includes a network 140, via which any computing device in the distributed computing environment 100 can communicate with each other; a server computing device 138; and a time series data / event log data storage device 142.

[0024] A monitoring service that monitors operational data from one or more cloud-based applications may reside on one or more computing devices in the network and processing subenvironment 136, and the monitoring service may receive operational data from a telemetry service. In some examples, the monitoring service and the telemetry service may be different services. In other examples, the monitoring service and the telemetry service may be the same service. The telemetry service may receive operational data (e.g., operation success counts, operation failure counts, operation delay data, etc.), which may be used by one or more monitors of the monitoring service. For example, one or more cloud-based applications may report operational errors to the telemetry service, and the monitor of the monitoring service may determine whether the degradation of the service quality associated with one or more cloud-based applications corresponds to a code regression in a new or modified operation included in a new software build, a server problem, and / or a network problem. In some examples, when operational successes and errors occur, they may be automatically reported to the telemetry database by the cloud-based application. In other examples, there may be an opt-in system so that, at least in a production environment, a user must opt-in to allow operational data to be automatically reported to the telemetry database.

[0025] Service modification subenvironment 101 includes application development subenvironment 128 and application subenvironment 102. There are three development teams in application development environment 128 (although there could be more or fewer). Team A is responsible for creating, modifying and / or maintaining software and software updates for application A 104. Team B is responsible for creating, modifying and / or maintaining software and software updates for application B 112. Team C is responsible for creating, modifying and / or maintaining software and software updates for application C 120. In this example, each development team has pushed out software updates for their respective applications, which are shown by software updates 127, which are pushed to those respective applications via cloud-based application services running in network and processing subenvironment 136.

[0026] In the example shown, a previous build of application A 104 prior to implementing software update 127 includes operation 1 106, operation 2 108, and operation 3 110. Operation 3 110 is shown with a dashed line around it to illustrate that it has been removed from application A 104 via software update 127. A previous build of application B 112 prior to implementing software update 127 includes operation 4 114 and operation 5 116. Operation 6 118 has been added to application B 112 via implementing software update 127. Application C 120 prior to implementing software update 127 includes operation 7 122, operation 8 124, and operation 9 126. Operation 9 126 is shown with a dashed line around it to illustrate that it has been modified from a previous version via implementing software update 127.

[0027] A telemetry service running in the network and processing sub-environment may store time series data and event logs of operations associated with the cloud-based application service in a time series data / event log database 142. For example, the telemetry service may receive operational data for multiple operations associated with one or more cloud-based applications and store the information in addition to the time at which the information was received and / or the time at which each corresponding operational event occurred. One or more monitors associated with the cloud-based application service may analyze the operational time series data from the telemetry service to determine whether a code regression exists in the software of one or more applications hosted by the cloud-based application service.

[0028] In dynamic monitor sub-environment 130 , old monitor 132 corresponds to each monitor that was monitoring telemetry data for the cloud-based application service prior to implementation of software update 127 . That is, monitor M1 106* monitors telemetry data of operation 1 106 of application A 104; monitor M2 108* monitors telemetry data of operation 2 108 of application A 104; monitor M3 110* monitors telemetry data of operation 3 110 of application A 104; monitor M4 114* monitors telemetry data of operation 4 114 of application B 112; monitor M5 116* monitors telemetry data of operation 5 116 of application B 112; monitor M7 122* monitors telemetry data of operation 7 122 of application C 120; monitor M8 122* monitors telemetry data of operation 8 124 of application C; and monitor M9 126* monitors telemetry data of operation 9 126 of application C.

[0029] A telemetry analysis and comparison engine associated with the telemetry service and / or monitoring service can analyze time series data and event logs from operational events that occurred in a time series before implementing the software update 127 and compare that data with time series data and event logs from operational events that occurred after implementing the software update 127. The time series data and / or event logs can include the name of each operation performed in connection with the cloud-based application service, an indication of whether each operational event was successful or unsuccessful, the duration that each operation took to complete and / or time out, a designation of whether a monitor exists for monitoring each operation, and / or a designation of a specific monitor that is monitoring each operation if such a monitor exists. In comparing the time series data and event logs from the pre-software update time series and the post-software update time series, the telemetry analysis and comparison engine can determine which operations have been deleted, added, and / or modified by implementing the software update 127. In addition, the telemetry analysis and comparison engine can determine based on the time series comparison which operations in the time series after the software update have existing monitors, which operations in the time series after the software update need to be updated based on the operations modified by implementing the software update 127, which operations after the software update require new monitors to be created for them because they have been added by implementing the software update 127, and / or whether monitors need to be deleted and which monitors to delete because the operations have been deleted by implementing the software update 127.

[0030] Once it is determined which operations have been modified, deleted, and / or added via software update 127, the dynamic monitor engine may modify one or more existing monitors, delete one or more existing monitors, and / or generate one or more new monitors. That is, for each operation deleted via software update 127 for which a monitor exists, the dynamic monitor engine may delete the monitor; for each new operation added via software update 127, the dynamic monitor engine may generate a new monitor; and for each operation modified via software update 127 for which a monitor exists, the dynamic monitor engine may modify the corresponding monitor.

[0031] When modifying and / or generating a new monitor, the dynamic monitor engine can apply one or more machine learning models (e.g., Holt Winters, principal component analysis, etc.) to the telemetry data set to determine one or more baselines of operation success or failure levels and / or counts, and / or baselines for operation execution delays. For example, one or more machine learning models can be applied to one or more software post-update time series of operation data and determine the baselines into which the success, error, and / or delay data points from the time series fall. In some examples, the baseline can be dynamic because it changes based on one or more factors (e.g., time of day, date, month, context, etc.). In other examples, the baseline can be static because it does not change regardless of the factors associated with it. For example, a dynamic baseline can be generated for an operation failure count and / or an operation failure percentage, where the number of operation failures and / or the percentage of operation failures increase during various times of the day and / or night. Alternatively, a static baseline can be generated for an operation failure count and / or an operation failure percentage, where the number and / or percentage of operation failures remain relatively static over the course of a day.

[0032] Once a baseline is determined for an operation, a threshold outside of the baseline can be identified to mark time series data that exceeds the threshold. For example, a monitor of an operation can identify a threshold number of operation failures outside of the baseline number of failures, which the time series must exceed within a threshold duration (i.e., a "time window") in order for the monitor to mark the time series as potentially related to code regression or otherwise having potential problems. In another example, a monitor of an operation can identify a threshold percentage of operation failures outside of the baseline failure percentage, which the time series must exceed within the threshold duration in order for the monitor to mark the time series as potentially related to code regression or otherwise having potential problems. In another example, a monitor of an operation can identify a threshold duration outside of the baseline delay duration, which the operation of the time series must exceed in order for the monitor to mark the time series as potentially related to code regression or otherwise having potential problems. The threshold can be automatically determined based on one or more machine learning models, and / or the threshold can be manually selected. As with the baseline, the threshold can be dynamic or static. In some examples, the threshold can be set to zero (i.e., any reading above or below the baseline can result in an operation mark).

[0033] This article is about Figure 3 and Figure 4 Additional information on baseline and threshold generation and their application is provided.

[0034] In the dynamic monitor sub-environment 130, new monitors 134 correspond to monitors that exist for the merged operations of software update 127. Specifically, while monitors M1 106*, M2 108*, M4 114*, M5 116*, M7 122*, and M8 124* remain after software update 127 because their corresponding operations have not been modified or deleted, monitor M3 110* has been deleted (as indicated by the dashed line surrounding it) because its corresponding operation (operation 3 110) has been deleted via software update 127, monitor M9 126* has been modified (as indicated by the dotted line surrounding it) because its corresponding operation (operation 9 126) has been dynamically modified via software update 127, and monitor M6 118* has been dynamically generated by the dynamic monitor engine because its corresponding operation (operation 6 118) has been added to application B 112 via software update 127.

[0035] Figure 2 A basic flow chart 200 for dynamically configuring a monitor for a cloud-based application service is shown. The flow chart 200 includes an application development sub-environment 228 and a software update 227. The application sub-environment 228 includes three development teams (Team A, Team B, and Team C), which are all software development teams that create and update software for cloud-based application services. In this example, the team has launched a new software update (Software Update 227) that is being applied to the cloud-based application service. The software update has been applied to one or more cloud-based applications, and time series data of the applications before and after the software update has been stored in the time series data / event log database 202.

[0036] The log data mining unit 204 includes analysis and comparison of operation logs from one or more pre-software update time series and one or more post-software update time series. The comparison engine can identify which operations and corresponding monitors exist in the cloud-based application service and / or telemetry service before the software update, and compare those operations and monitors with the operations and monitors that exist in the cloud-based application service and / or telemetry service after the software update.

[0037] The results of the log data mining are shown by comparison result unit 205. Comparison result unit 205 includes old operations 206 (i.e., operations that existed before the software update and / or still exist in the previous software build after the software update), new operations 208 (i.e., operations that have been added via software update 227), and modified operations 210 (i.e., operations that existed before the software update and have been modified via the incorporation of software update 227).

[0038] New monitor generation unit 212 illustrates generation of new monitors that are dynamically created via application of machine learning models to time series data of operations that are new to the cloud-based application service based on implementation of software update 227. New monitor generation unit 212 also illustrates deletion of existing monitors that are no longer useful due to deletion of operations from the cloud-based application service based on their removal via software update 227. Additionally, new monitor generation unit 212 illustrates modification of existing monitors that are modified to accurately flag code regressions based on corresponding operations that have been modified via software update 227.

[0039] Figure 3 A graphical display 302 of operational data for a cloud-based application service is shown, wherein dynamically configured monitors are applied to quality of service metrics. Data corresponding to the graphical display 302 may be associated with a telemetry service that receives and stores operational event data and operational event logs for a cloud-based application service. The telemetry service may maintain a log of each operation that has been initiated by a user of a cloud-based application hosted on the service. These logs may include an identification code for each operation initiated, whether each initiated operation was successfully or unsuccessfully executed, the duration that each operation took to complete or time out, an identification code for a server or server cluster that executed each request associated with each operation, an indication of whether a monitor is associated with each initiated operation, and / or an identification code for each monitor associated with each initiated operation.

[0040] The telemetry service may take the raw operational event data that it collects / receives and generate one or more graphs of that data, such as the graph shown on graphical display 302 . Figure 3 The figure in is a service quality graph (for cloud-based application services) that shows the percentage (on the Y axis) of initiated operations of a specific operation type that have been successfully executed over a duration (on the X axis). In this example, the dynamic monitor engine has automatically generated a new monitor for the specific operation represented in the figure. In doing so, the dynamic monitor engine has determined and set a baseline percentage 304 corresponding to the range of successful operations for the specific operation that the specific operation typically falls into. The range shown by the diagonal rectangle from 85% to 100% of the baseline percentage 304 is shown as being static over the duration included in the figure. However, it should be understood that the baseline can be dynamic or static. For example, the baseline shown can be associated with a daytime standardized range for a specific operation, and the baseline can vary for a nighttime standardized range for a specific operation.

[0041] In this example, threshold 308 has been identified and set through a machine learning mechanism or manual interaction with the telemetry service. Threshold 308 corresponds to a percentage outside of the baseline to which the quality of service must fall in order for the telemetry service to flag a particular operation as a potential issue (e.g., the particular operation may be associated with a code regression). In this example, threshold 308 has been set to 77.5% (i.e., 7.5% below the lower limit of baseline percentage 304). Therefore, in this example, the monitor for this particular operation will flag portion 306 of the graph that is below threshold 308. Although the particular operation has been flagged due to the identification of portion 306 of the graph that is below threshold 308, time series data and / or time series data surrounding the time series data may be provided for analysis of the flagged issue.

[0042] Figure 4 A graphical display 402 of operational data for a cloud-based application service is shown, wherein dynamically configured monitors are applied to an overall unexpected failure metric. Data corresponding to the graphical display 402 may be associated with a telemetry service that receives and stores operational event data and operational event logs for a cloud-based application service. The telemetry service may maintain a log of each operation that has been initiated by a user of a cloud-based application hosted on the service. These logs may include an identification code for each operation initiated, whether each initiated operation was successfully or unsuccessfully executed, the duration that each operation took to complete or time out, an identification code for a server or server cluster that executed each request associated with each operation, an indication of whether a monitor is associated with each initiated operation, and / or an identification code for each monitor associated with each initiated operation.

[0043] The telemetry service may take the raw operational event data that it collects / receives and generate one or more graphs of that data, such as the graph shown on graphical display 402 . Figure 4The figure in is an unexpected failure graph (for cloud-based application services) that shows the number of initiated operations (on the Y axis) of a specific operation type that has caused an unexpected failure within a duration (on the X axis). In this example, the dynamic monitor engine has automatically generated a new monitor for the specific operation represented in the figure. In doing so, the dynamic monitor engine has determined and set a baseline number range 404 for unexpected failures for the specific operation. The distribution of such numbers corresponds to the unexpected failure range for the specific operation that the specific operation usually falls into. The range shown by the diagonal rectangle on the graphical display 402 has been set to 0 unexpected failures for the specific operation to 1750 unexpected failures for the specific operation. The range can be based on applying one or more machine learning models to time series data for the specific operation and has been identified. In this example, the range is a static range; however, it should be understood that the baseline can be dynamic or static. For example, the baseline shown may relate to a daytime normalized range for a particular operation, and the baseline may vary for a nighttime normalized range for a particular operation (e.g., more or fewer users may use that particular operation during certain hours compared to certain other hours).

[0044] In this example, threshold 408 has been identified and set by applying a machine learning model to one or more time series for a particular operation or by manual interaction with a telemetry service. Threshold 408 corresponds to the minimum number of overall unexpected failures above the baseline that needs to be reached in the time window of the time series in order for the telemetry service to mark the particular operation as a potential problem (e.g., the particular operation may be related to a code regression). In this example, threshold 408 has been set to 3000 unexpected failures (i.e., more than the baseline upper limit of 1250 unexpected errors). Therefore, in this example, the monitor for the particular operation will mark the portion 406 of the graph that is above threshold 408. Although the particular operation has been marked due to the identification of portion 406 of the graph that is above threshold 408, time series data and / or time series data surrounding the time series data can be provided for analysis of the marked problem.

[0045] Figure 5 An exemplary method 500 for dynamically configuring a monitor for a cloud-based application service begins at a start operation and flow moves to operation 502 .

[0046] At operation 502, telemetry data for a plurality of operations for a cloud-based application service is analyzed. The analysis may include comparing a first time series with a second time series, wherein data from the second time series is associated with an operation that was performed prior in time to the execution of the operation associated with the first time series. Each analyzed time series may include operational event data from operations associated with the cloud-based application service. The operational event data may include: an identification code / name of each operation initiated, executed, and / or completed; an occurrence time at which each operation was initiated, executed, and / or completed; an indication of whether there is a monitor for each operation initiated, executed, and / or completed; an identification code of each monitor that exists for each operation initiated, executed, and / or completed; a duration for each operation to complete and / or time out; and / or an identification code of a server or server group that attempted to perform each operation. In some examples, a comparison of the two time series may be automatically initiated based on an indication that a software update has been provided to the cloud-based application service. The second time series may correspond to a duration before the software update was incorporated into the cloud-based application service, and the first time series may correspond to a duration after the software update was incorporated into the cloud-based application service.

[0047] Flow continues from operation 502 to operation 504, where one or more operational changes in the cloud-based application service are identified. The identification is based on the time series comparison performed at operation 502. The identification is performed by determining whether operations have been added, deleted, and / or modified based on a software update to the cloud-based application service. For example, when a software update is incorporated into the cloud-based application service, one or more operations included in one or more applications hosted by the cloud-based application service prior to the software update may be modified, added, or deleted based on the incorporation of the software update.

[0048] The process continues from operation 504 to operation 506, where at least one telemetry monitor is dynamically configured based on one or more operational changes identified at operation 502. As described herein, dynamic configuration can be performed by a dynamic monitor engine and applying one or more machine learning models to time series data. Dynamic configuration can include selecting appropriate monitoring techniques to detect failure modes for each new operation associated with one or more operational changes. Dynamic configuration can additionally or alternatively include automatically defining a baseline failure rate for each new operation associated with one or more operational changes, and / or defining a threshold failure rate based on the baseline failure rate for each new operation associated with one or more operational changes. In some examples, dynamically configuring at least one telemetry monitor can include automatically defining a time series window. The time series window can include operations performed in a duration, and each telemetry monitor for each new operation associated with one or more operational changes will monitor the duration. The dynamic monitor engine can configure the monitor to have static and / or dynamic baselines and / or thresholds.

[0049] In some examples, the number of one or more new telemetry monitors to be generated can be automatically determined based on the available bandwidth for monitoring operations in the cloud-based application service. In additional examples, the number of time series to be monitored and / or the duration of each time series can be automatically determined based on the available bandwidth for monitoring operations in the cloud-based application service. For example, the processing resources available to the telemetry service may dictate that only a set number of time series can be analyzed within a set duration. Thus, when monitors are dynamically created, the number of time series they process within that time range can be automatically calculated and built into each corresponding monitor.

[0050] Flow moves from operation 506 to an end operation and method 500 ends.

[0051] Figure 6 and Figure 7 A mobile computing device 600 is shown, such as a mobile phone, a smartphone, a wearable computer, a tablet computer, an e-reader, a laptop computer, an AR-compatible computing device, or a VR computing device, which can be used to practice embodiments of the present disclosure. Figure 6, shows one aspect of a mobile computing device 600 for implementing these aspects. In a basic configuration, the mobile computing device 600 is a handheld computer having both an input unit and an output unit. The mobile computing device 600 typically includes a display 605 and one or more input buttons 610 that allow a user to input information into the mobile computing device 600. The display 605 of the mobile computing device 600 can also be used as an input device (e.g., a touch screen display). If included, an optional side input unit 615 allows further user input. The side input unit 615 can be a rotary switch, a button, or any other type of manual input unit. In alternative aspects, the mobile computing device 600 can incorporate more or fewer input units. For example, in some embodiments, the display 605 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 600 is a portable telephone system, such as a cellular phone. The mobile computing device 600 can also include an optional keypad 635. The optional keypad 635 can be a physical keypad or a "soft" keypad generated on the touch screen display. In various embodiments, the output unit includes a display 605 for showing a graphical user interface (GUI), a visual indicator 620 (e.g., a light emitting diode), and / or an audio transducer 625 (e.g., a speaker). In some aspects, the mobile computing device 600 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 600 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0052] Figure 7 700 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, a mobile computing device 700 may incorporate a system (e.g., architecture) 702 to implement some aspects. In one embodiment, the system 702 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 702 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0053] One or more application programs 766 may be loaded into the memory 762 and run on or in association with the operating system 864. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 702 also includes a non-volatile storage area 768 within the memory 762. The non-volatile storage area 768 may be used to store persistent information that should not be lost when the system 702 is powered off. The application program 766 may use and store information in the non-volatile storage area 768, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 702 and is programmed to interact with a corresponding synchronization application residing on the host to keep the information stored in the non-volatile storage area 768 synchronized with the corresponding information stored at the host. It should be understood that other applications may be loaded into the memory 762 and run on the mobile computing device 700, including instructions for providing and operating a digital assistant computing platform.

[0054] System 702 has a power source 770, which can be implemented as one or more batteries. Power source 770 can also include an external power source, such as an AC adapter or a powered docking cradle to replenish or recharge the batteries.

[0055] The system 702 may also include a wireless interface layer 772 that performs the functions of sending and receiving radio frequency communications. The wireless interface layer 772 facilitates wireless connections between the system 702 and the "outside world" via a communications carrier or service provider. Transmissions to and from the wireless interface layer 772 are performed under the control of the operating system 764. In other words, communications received by the wireless interface layer 772 may be propagated to the application 766 via the operating system 764, and vice versa.

[0056] The visual indicator 620 can be used to provide visual notifications, and / or the audio interface 774 can be used to generate auditory notifications via the audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to the power supply 770 so that when activated, they will remain on for the duration specified by the notification mechanism even if the processor 760 and other components may be turned off to save battery power. The LED can be programmed to remain lit indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 774 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 774 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to an embodiment of the present disclosure, as will be described below, the microphone can also be used as an audio sensor to facilitate the control of notifications. The system 702 can also include a video interface 776, which enables the operation of the onboard camera 630 to record still images, video streams, etc.

[0057] The mobile computing device 700 implementing the system 702 may have additional features or functionality. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data or other data. Figure 7 This is shown by the non-volatile storage area 768.

[0058] Data / information generated or captured by the mobile computing device 700 and stored via the system 702 may be stored locally on the mobile computing device 700, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the wireless interface layer 772 or via a wireless connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 700 via the wireless interface layer 772 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transmission and storage means, including electronic mail and collaborative data / information sharing systems.

[0059] Figure 8is a block diagram illustrating the physical components (e.g., hardware) of a computing device 800 that can be used to practice aspects of the present disclosure. The computing device components described below may have computer executable instructions for dynamically configuring an operation monitor for a cloud-based application service. In a basic configuration, the computing device 800 may include at least one processing component 802 and a system memory 804. Depending on the configuration and type of the computing device, the system memory 804 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 804 may include an operating system 805 suitable for running one or more operation monitoring programs. For example, the operating system 805 may be suitable for controlling the operation of the computing device 800. In addition, embodiments of the present disclosure may be implemented in conjunction with a graphics library, other operating systems, or any other application, and are not limited to any particular application or system. This basic configuration is Figure 8 808. The computing device 800 may have additional features or functionality. For example, the computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data or other data. Figure 8 It is illustrated by a removable storage device 809 and a non-removable storage device 810.

[0060] As described above, a plurality of program modules and data files may be stored in the system memory 804. When executed on the processing component 802, the program module 806 (e.g., the real-time code defect telemetry application 820) may perform processing including, but not limited to, aspects as described herein. According to an example, the telemetry analysis and comparison engine 811 may identify one or more pre-software update time series and compare these time series with one or more post-software update time series to determine whether a new operation monitor should be generated, whether an existing operation monitor should be modified, and / or whether an existing operation monitor should be deleted. The dynamic monitor engine 813 may perform one or more operations associated with generating new monitors and modifying existing monitors for operations in a cloud-based application service. The log analysis engine 815 may perform one or more operations associated with analyzing operation logs from one or more time series and applying the information to a machine learning model to create and / or modify monitors and / or determine which operational changes have been made via implementing a software update to a cloud-based application service. The monitor allocation engine 817 can perform one or more operations associated with determining system resources available for cloud-based application services and / or telemetry services and setting operational monitoring criteria for each monitor in the system (e.g., how many time series are monitored for each operation, how long the time series are, etc.).

[0061] In addition, embodiments of the present disclosure may be practiced on a circuit including discrete electronic units, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing an electronic unit or a microprocessor. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 8 Each or many components shown in can be integrated into a single integrated circuit. Such a SOC device may include one or more processing components, graphics components, communication components, system virtualization components, and various application functions, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the ability of the client switching protocol can be operated via dedicated logic integrated with other components of the computing device 800 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be practiced in a general-purpose computer or in any other circuit or system.

[0062] The computing device 800 may also have one or more input devices 812, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output devices 814 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples, and other devices may be used. The computing device 800 may include one or more communication connections 816 that allow communication with other computing devices 850. Examples of suitable communication connections 816 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuits; universal serial bus (USB), parallel and / or serial ports.

[0063] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures or program modules. System memory 804, removable storage device 809 and non-removable storage device 810 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 800. Any such computer storage media may be part of computing device 800. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0064] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0065] Fig. 9 One aspect of the architecture of a system for processing data received from a remote source at a computing system, such as a personal / general purpose computer 904, a tablet computing device 906, or a mobile computing device 908, as described above, is shown. The content displayed at the server device 902 can be stored in different communication channels or other storage types. For example, a directory service 922, a web portal 924, a mailbox service 926, an instant messaging repository 928, or a social networking site 930 can be used to store various documents. Program modules 806 can be employed by a client that communicates with the server device 902, and / or program modules 806 can be employed by the server device 902. The server device 902 can provide data to and from client computing devices, such as personal / general purpose computers 904, tablet computing devices 906, and / or mobile computing devices 908 (e.g., smart phones), over a network 915. For example, the above description of Figures 6 to 8 The depicted computer system may be embodied in a personal / general purpose computer 904, a tablet computing device 906, and / or a mobile computing device 908 (e.g., a smartphone). In addition to receiving graphics data that may be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these embodiments of the computing device may obtain content from a repository 916.

[0066] For example, aspects of the present disclosure are described above with reference to the block diagrams and / or operational descriptions of the methods, systems, and computer program products according to aspects of the present disclosure. The functions / actions noted in the boxes may not occur in the order shown in any flow chart. For example, depending on the functions / actions involved, two boxes shown in succession may actually be performed substantially simultaneously, or the boxes may sometimes be performed in reverse order.

[0067] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the best mode of the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether it is shown and described in combination or separately, various features (in structure and method) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. Description and illustration of the present disclosure have been provided, and those skilled in the art can envision changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, and these changes, modifications and alternative aspects do not deviate from the broader scope of the disclosure claimed.

[0068] The various embodiments described above are provided by way of illustration only and should not be construed as limiting the appended claims. Those skilled in the art will readily recognize various modifications and changes that may be made without following the example embodiments and applications shown and described herein and without departing from the true spirit and scope of the appended claims.

Claims

1. A method for monitoring telemetry data from a cloud-based application service, the method comprising: Analyzing telemetry data for a plurality of operations performed by an application running on the cloud-based application service, wherein the analyzing includes comparing: a first time series comprising data from a run of the application prior to an application update, and a second time series comprising data from a run of the application after the application was updated; identifying, based on the analyzing, a first operational change in the cloud-based application service, the first operational change comprising adding a new operation to the application; Based on the first operation change, dynamically configure the telemetry monitor, wherein the configuration includes: establishing a baseline via applying a first machine learning model to the second time series, the baseline comprising a normalized error margin during a threshold duration for the new operation, and establishing an error threshold for the duration being outside of the baseline via applying a second machine learning model to the second time series; analyzing a third time series comprising data from execution of the application after the application is updated; determining that a value of the error for the duration in the third time series exceeds the error threshold; and A portion of the third time series corresponding to the duration for which the value of the error is determined to exceed the error threshold is marked.

2. The method of claim 1, wherein: Dynamically configuring the telemetry monitor includes selecting an appropriate monitoring technique to detect a failure mode for the first operational change.

3. The method of claim 1, wherein: The baseline established is a static baseline.

4. The method of claim 1, wherein: The baseline established is a dynamic baseline.

5. The method of claim 1, wherein: The identified first operational change comprises a new operation included in a software update, and wherein dynamically configuring the telemetry monitors comprises generating one or more new telemetry monitors for monitoring data associated with the new operation.

6. The method of claim 1, wherein: There are a number of new operational changes identified based on the analysis, and there are a number of new telemetry monitors generated to monitor data associated with the new operations, the number being determined based on available bandwidth for monitoring operations in the cloud-based application service.

7. The method of claim 1, further comprising: A second operational change in the cloud-based application service is identified, wherein the second operational change includes an operation that has been removed from an application hosted by the cloud-based application service.

8. The method of claim 1, further comprising: A second operational change in the cloud-based application service is identified, wherein the second operational change includes an operation that has been modified from an application hosted by the cloud-based application service.

9. A system for monitoring telemetry data from a cloud-based application service, comprising: a memory for storing executable program code; as well as one or more processors functionally coupled to the memory, the one or more processors responsive to computer executable instructions contained in the program code and configured to: Analyzing telemetry data for a plurality of operations performed by an application running on the cloud-based application service, wherein the analyzing includes comparing: a first time series comprising data from a run of the application prior to an application update, and a second time series comprising data from a run of the application after the application was updated; identifying, based on the analyzing, a first operational change in the cloud-based application service, the first operational change comprising adding a new operation to the application; Based on the first operation change, dynamically configure the telemetry monitor, wherein the configuration includes: establishing a baseline via applying a first machine learning model to the second time series, the baseline comprising a normalized error margin during a threshold duration for the new operation, and establishing an error threshold for the duration being outside of the baseline via applying a second machine learning model to the second time series; analyzing a third time series comprising data from execution of the application after the application is updated; determining that a value of the error for the duration in the third time series exceeds the error threshold; and A portion of the third time series corresponding to the duration for which the value of the error is determined to exceed the error threshold is marked.

10. The system of claim 9, wherein: In automatically generating the telemetry monitor, the one or more processors are further responsive to the computer executable instructions contained in the program code and are configured to: Select appropriate monitoring techniques to detect failure modes for the operation in question.

11. A computer readable storage device comprising executable instructions that, when executed by one or more processors, facilitate monitoring telemetry data from a cloud-based application service, the computer readable storage device comprising instructions executable by the one or more processors to: analyzing telemetry data for a plurality of operations performed by an application running on the cloud-based application service, wherein: The analysis involved comparing: a first time series comprising data from a run of the application prior to an application update, and a second time series comprising data from a run of the application after the application was updated; identifying, based on the analyzing, a first operational change in the cloud-based application service, the first operational change comprising adding a new operation to the application; Based on the first operation change, dynamically configure the telemetry monitor, wherein the configuration includes: establishing a baseline via applying a first machine learning model to the second time series, the baseline comprising a normalized error margin during a threshold duration for the new operation, and establishing an error threshold for the duration being outside of the baseline via applying a second machine learning model to the second time series; analyzing a third time series comprising data from execution of the application after the application is updated; determining that a value of the error for the duration in the third time series exceeds the error threshold; and A portion of the third time series corresponding to the duration for which the value of the error is determined to exceed the error threshold is marked.

Images