Automatically detecting code regressions from time series data

Abstract

In a non-limiting example of the present disclosure, systems, methods, and apparatus for detecting and classifying service issues associated with cloud-based services are provided. Operational event data for a plurality of operations associated with a cloud-based application service may be monitored. A statistically based unsupervised machine learning model may be applied to the operational event data. A subset of the operational event data may be marked as potentially associated with a code regression, wherein the subset includes a time series of the operational event data. A neural network may be applied to the time series of the operational event data, and if the neural network classifies the time series as associated with a positive code regression category, the time series of the operational event data may be marked for subsequent tracking.

Description

Technical Field

[0001] The present invention relates to automatically detecting code regressions. Background Art

[0002] As computing increasingly moves to the cloud, the systems that support the vast number of users and cloud-based applications they employ are constantly being modified. The infrastructure of cloud-based systems requires constant monitoring to maintain, let alone update and add additional features and functionality. As new software versions are added to the infrastructure of cloud-based systems, it can be difficult to determine whether operational failures are related to code defects, network issues, and / or hardware problems within the service. Furthermore, even if engineering teams know the general causes of service issues, these teams often struggle to identify the urgency of these issues.

[0003] It is against this general technical environment that the aspects of the technology disclosed herein are considered.In addition, although a general environment has been discussed, it should be understood that the examples described herein should not be limited to the general environment identified in the background. Summary of the Invention

[0004] This summary is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description section below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to serve as an aid in determining the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description that follows and in part will be obvious from the description or may be learned by practice of the disclosure.

[0005] Non-limiting examples of the present disclosure describe systems, methods, and devices for identifying and classifying code regressions, network issues, and / or hardware issues from telemetry data associated with cloud-based application services. The described mechanism includes a three-tiered approach that first acquires a large amount of time-series telemetry data and applies an unsupervised machine learning model to the data to preliminarily identify a subset of the data as potentially associated with code regressions, network issues, and / or hardware issues. The second tier takes the subset and applies a neural network to it to further reduce the size of the data potentially associated with code defects, network issues, and / or hardware issues. The neural network can be applied to the raw data and / or a graphical image matrix representing the raw data. The neural network can be used to classify the data into one or more additional categories. The third tier applies one or more language processing models to the operational data logs associated with the smaller subset identified and classified by the convolutional neural network. The language processing model can further classify the dataset into categories such as severity of the issue type, type of operational error, location of the issue, and / or identification of the server or server cluster corresponding to the issue. The fully classified data can then be sent to a human reviewer for one or more follow-up actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Non-limiting and non-exhaustive examples are described with reference to the following figures:

[0007] Figure 1 is a diagram illustrating an example distributed computing environment for detecting and classifying service issues associated with cloud-based application services.

[0008] Figure 2 An exemplary graph including operational error data from a cloud-based application service is illustrated.

[0009] Figure 3 Illustrate the use of a convolutional neural network to process operational error data to identify and classify service issues associated with cloud-based application services.

[0010] Figure 4 Illustrated is a graphical image representation of a time series that can be fed into a neural network for classification of potential operational errors.

[0011] Figure 5A An exemplary method for detecting and classifying service issues associated with a cloud-based application service is provided.

[0012] Figure 5B is another exemplary method for detecting and classifying service issues associated with a cloud-based application service.

[0013] Figure 6 and Figure 7 is a simplified diagram of a mobile computing device that can be used to practice aspects of the present disclosure.

[0014] Figure 8 is a block diagram illustrating example physical components of a computing device that can be used to practice aspects of the present disclosure.

[0015] Figure 9 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0016] Various embodiments will be described in detail with reference to the accompanying drawings, wherein like reference numerals represent like parts and components throughout the several views. Reference to various embodiments does not limit the scope of the appended claims. Furthermore, any examples set forth in this specification are not intended to be limiting and merely illustrate some of the many possible embodiments of the appended claims.

[0017] The various embodiments and examples described above are provided by way of illustration only and should not be construed as limiting the appended claims. Those skilled in the art will readily recognize various modifications and changes that can be made without following the example embodiments and applications shown and described herein and without departing from the true spirit and scope of the claims.

[0018] Examples of the present disclosure provide systems, methods, and devices for detecting code regressions, network issues, and hardware issues associated with cloud-based application services. For example, a code regression may be included in a new software package, update, and / or patch. In some examples, a code regression may be associated with one or more cloud-based applications, such as a cloud-based document processing application, a spreadsheet application, a calendar application, a presentation application, a storage application, a video application, an instant electronic messaging application, a voice messaging application, a video communication application, and / or an email application. In some examples, a system monitor may monitor for signals associated with operational failures associated with one or more cloud-based applications. For example, when an operation of a cloud-based application fails and / or an operation of a cloud-based application causes the application to crash or fail during execution, a signal indicating the presence of an operational event or operational failure may be reported to the system monitor. The system monitor may monitor one or more operations associated with the cloud-based application. For example, the system monitor may monitor for "save" operations, "send" operations, "new document" operations, "copy" operations, "paste" operations, and any other operations that may be performed by a cloud-based application.

[0019] A telemetry service associated with a cloud-based application service can receive operational data associated with each operation initiated and / or performed by a user accessing an application hosted by the cloud-based application service. The information collected by the telemetry service can include: timestamp information corresponding to each operation initiated by the cloud-based application service, the amount of time those operations took to complete and / or timed out, the identification of those operations, what the result of each initiated operation was (e.g., unexpected failure, success, expected failure), and / or the server or server cluster that performed each operation. The systems, methods, and devices described herein can apply a three-tiered approach to identify whether service issues associated with a cloud-based application service are related to code regressions, network issues, and / or hardware issues. These mechanisms also allow IT support and developers of the system to identify which operations and / or code blocks are responsible for service issues and the severity of those issues.

[0020] The first layer of operations takes one or more time series of operational event data and applies an unsupervised statistical machine learning model to it. The unsupervised machine learning model can identify service degradations that are potentially related to one or more of code regressions, network issues, and / or hardware issues. The machine learning model can identify such issues by identifying data points that fall outside a standardized baseline for a specific operation (a baseline of unexpected errors, expected errors, and a ratio of unexpected errors to successful operations). Although the unsupervised machine learning models that can be applied to the first layer are generally computationally inexpensive, and they are able to identify a small subset of operations and / or time series that may be related to potential service issues from a larger dataset, they may still have a relatively large false positive rate associated with them.

[0021] The second layer of operations may take a subset of operational event data and / or time series data that has been identified as potentially related to code regression, network issues, and / or hardware issues and feed that data into a neural network that has been manually trained on similar datasets to classify those datasets. Categories may include a positive identification that the dataset is related to code regression, network issues, and / or hardware issues; a negative indication that the dataset is related to code regression, network issues, and / or hardware issues; a percentage likelihood classification that the dataset is related to code regression, network issues, and / or hardware issues; and / or the severity of the problem corresponding to the dataset if it is identified as related to code regression, network issues, and / or hardware issues. The neural network may be applied to one or more graphical image matrix representations of the raw data and / or the dataset. In some examples, the neural network may include a convolutional neural network.

[0022] The third layer of operations takes a subset of operational event data and / or time series data that has been classified by the neural network as being related to code regressions, network issues, and / or hardware issues, and parses the operational logs for each of these operations in the dataset to further classify the dataset based on one or more of the following: severity, type of operation, geographic location of origin, and / or server or server cluster associated with the issue. The third layer can also output a human-readable summary of the error that occurred. In some examples, the candidate items can then be passed to a processing queue that determines whether there are any duplicates from previous days, and those non-duplicates can be sorted by severity and / or urgency. One or more engineering teams can then review the candidate items in the queue and upgrade any of these items as needed.

[0023] The systems, methods, and devices described herein provide technical advantages for identifying the causes of service issues in cloud-based application services and categorizing these causes based on their severity and / or urgency so that they can be appropriately addressed by engineering teams. The mechanism described herein leverages a three-tiered approach that applies three different types of modeling techniques to first identify a subset of potentially problematic operations within the service using a computationally inexpensive statistical machine learning model with an associated high false positive identification rate. The size of this subset is then reduced by applying a computationally more expensive neural network to the subset. Finally, one or more language processing models are applied to the data, whose size has been further reduced by the classification performed by the neural network. By further applying the neural network to the first subset of data, the processing and manual review costs associated with traditional approaches to addressing this problem, which simply apply inexpensive but ineffective modeling techniques to the full operational event dataset, are significantly reduced. Because the size of the data requiring urgent processing by the language processing model is reduced, parsing the smaller subset of operational logs categorized by the neural network also reduces processing and memory costs. Furthermore, applying the language processing model to the event logs provides savings in processing and manual review because the engineering team does not have to manually review every event log in the smaller subset categorized by the neural network. Instead, these teams can focus their time, energy, and computing resources on issues that affect the service in the most urgent way.

[0024] Figure 1 1 is a schematic diagram illustrating an example distributed computing environment 100 for detecting and classifying service issues associated with cloud-based application services. Distributed computing environment 100 includes an application user subenvironment 102, operations 106, a process model subenvironment 117, and an operation tag subenvironment 122. Distributed computing environment 100 also includes a network and processing subenvironment 108, which includes a network 112, via which any computing device in distributed computing environment 100 can communicate with each other; a server computing device 110; and a time series data / event log database 114.

[0025] A monitoring service that monitors operational data from one or more cloud-based applications can reside on one or more computing devices in the network and processing subenvironment 108, and the monitoring service can receive operational data from a telemetry service. In some examples, the monitoring service and the telemetry service can be different services. In other examples, the monitoring service and the telemetry service can be the same service. The telemetry service can receive operational data (e.g., operation success counts, operation failure counts, operation latency data, etc.), which can be used by one or more monitors in the monitoring service. For example, one or more cloud-based applications can report operational errors to the telemetry service, and the monitors in the monitoring service can determine whether a degradation in service quality associated with the one or more cloud-based applications corresponds to a code regression in new or modified operations included in a new software version, a server issue, and / or a network issue. In some examples, when operational successes and errors occur, they can be automatically reported to a telemetry database by the cloud-based applications. In other examples, there can be an opt-in system such that, at least in a production environment, users must opt-in to allow automatic reporting of operational data to the telemetry service / database.

[0026] The monitoring service and / or telemetry service can store time series data associated with application operations in the time series / data event log database 114. The time series information can include timestamp information corresponding to each operation initiated by the cloud-based application service, the amount of time those operations took to complete and / or timed out, the identities of those operations, what the outcome of each initiated operation was (e.g., unexpected failure, success, expected failure), and / or the server or server cluster that performed each operation.

[0027] Multiple cloud-based application users in application user sub-environment 102 can access and perform various actions in applications hosted on the cloud-based application service. There may be hundreds, thousands, or even millions of cloud-based application users simultaneously accessing the cloud-based application service and performing operations on the applications hosted therein. As each of these actions is performed, a corresponding operation request is sent to the cloud-based application service in network and processing sub-environment 108. This is illustrated by operation 106, which shows multiple operation requests being sent from computing devices in application user sub-environment 102 to the cloud-based application service in network and processing sub-environment 108. As described above, as each of these operations is performed by the cloud-based application service, information corresponding to the execution is stored in time series data / event log database 114. Time series data / event log database 114 can be processed by the telemetry service to identify code regressions, hardware, and / or network issues that affect the user experience in the cloud-based application service. In some examples, the processing performed by the telemetry service can be performed by one or more server computing devices, such as server computing device 110. In other examples, processing of operational / telemetry data may be performed on local computing devices associated with development, engineering, and / or review teams, such as computing devices in operational marking sub-environment 122 .

[0028] The processing of operational data stored in the time series data / event log database 114 is illustrated by the processing model sub-environment 117. In an example, the time series data of all operations or a subset of operations executed by the cloud-based application service can be provided to a statistically based unsupervised machine learning model to identify whether there is a potential code regression associated with one or more of these operations. This is illustrated by the unsupervised machine learning sub-environment 116. For example, the machine learning models applied to the time series data may include a Gaussian model, a bloxplot method model, a one-class support vector machine model, and / or a density-based application space clustering with a noise model. These models can identify whether one or more operational events in the time series fall within the normalized pattern of the system or whether they are outliers that may require further review. In some examples, the machine learning model can be applied to one or more of the following: the number of successes in the time series ("S1"); the number of unexpected failures in the time series ("S2"); and / or the expected failures in the time series ("S3"). In other examples, the machine learning model can be applied to a function of one or more of those data points (e.g., "S4," where S4 = f(S1, S2, S3)).

[0029] While these types of unsupervised machine learning models are traditionally computationally cheap and fast, they do not provide a high degree of accuracy in their classifications. This can be problematic for identifying potential issues in large cloud-based application services, as the machine learning model may flag a large number of operations as related to code regressions, ultimately leading to false positives. For example, such a model might flag over 2,000 operational events per day as related to code regressions, each of which potentially requires further review. Therefore, even for resource-rich cloud-based services, the subsequent resources required to address these false positives can be overwhelming.

[0030] According to an example, instead of sending all operational events labeled by the machine learning model at the unsupervised machine learning sub-environment 116 directly to a human reviewer, the telemetry system may pass those labeled operational events and / or time series data associated with those labeled operational events to a neural network, as shown in the neural network sub-environment 118. In an example, the neural network may include a convolutional neural network.

[0031] The information passed to the neural network can include time series data associated with the marked operational events, including each marked data point and / or time series information for the corresponding operations before and / or after each marked data point. For example, if the unsupervised machine learning model marks one or more operations performed at time X as potentially related to a code regression, data related to the execution of these one or more operations can be passed to the neural network alone or together with data associated with operations performed from a specified duration before time X to time X and / or from time X to a specified duration after time X. In some examples, the data can be processed by the neural network in its original relational or hierarchical form. In other examples, the data can be processed by the neural network into a graphical image representation of the raw data (i.e., based on computer vision). In some examples, the neural network can be applied to one or more of the following: the number of successes in the time series ("SI"); the number of unexpected failures in the time series ("S2"); and / or the expected failures in the time series ("S3"). In other examples, the neural network can be applied to a function or an image matrix representation of the function of one or more of those data points (e.g., "S4", where S4=f(S1, S2, S3)).

[0032] The following combination Figure 3 and Figure 4The and corresponding descriptions provide additional details regarding the processing of operational data by the neural network.,Although more expensive from a processing perspective, the neural,network can provide up to 90% accuracy for correctly classifying operational,events and / or time series data points as related to code regressions,,network issues, and / or hardware issues, with this accuracy,increasing with each additional cycle of supervised training.

[0033] Once the smaller subset of labeled operations and / or time series data points has been identified / labeled by the neural network in the neural network sub-environment 118, the smaller subset of data can be passed to one or more language processing models in the language processing sub-environment 120 for processing to determine the cause of each labeled issue (e.g., code regression, network error, hardware issue, etc.), as well as the severity and / or urgency of the issue. For example, the one or more language processing models can be one or more of: a recurrent neural network; a convolutional neural network; a bag-of-words model; a rule-based model; a random forest model; and / or a word segmentation model. For example, the operation log associated with each operation in the subset that has been labeled by the neural network in the neural network sub-environment 118 is further processed by the one or more language processing models in the language processing sub-environment 120.

[0034] The operation log may include information about each marked operation, including: the identity of each marked operation, the result of each marked operation (e.g., success, unexpected failure, expected failure), the time each marked operation was initiated and / or executed, the duration each marked operation took to execute successfully and / or time out, and / or the identity of the server or server cluster that executed each marked operation. The language processing model may classify each marked operation based on its operation type, execution location, error code, latency, and / or severity. The severity classification may be related to how severe the problem is based on its impact on cloud-based application users in the application user subenvironment 102 (e.g., how many users were affected; how long the users were affected; whether the problem crashed the application and / or the entire device; etc.). Although the language processing model and parsing of the operation log are expensive from a processing perspective, much less logs are processed at the third layer due to the pre-processing performed by the unsupervised machine learning processing in the unsupervised machine learning subenvironment 116 and the neural network processing in the neural network subenvironment 118.

[0035] Once the issues have been categorized in the language processing sub-environment 120, these issues and their categorization can be sent for human review in the operational tagging sub-environment 122. Human reviewers and engineers can quickly and easily parse the issues based on their severity based on the categorization completed in the language processing sub-environment 120, and can resolve these issues based on the relevant error type that each issue has been categorized into.

[0036] Figure 2 An exemplary chart 200 including operational error data from a cloud-based application service is illustrated. Chart 202 is a quality of service graph of a time series of operational event data, which represents the percentage of successful operations (in the Y axis) of a particular operation type performed over a period of time (in the X axis). Chart 206 is a total unexpected failure graph of a time series of operational event data, which represents the total number of unexpected operational failures of a particular operation type (in the Y axis) performed over a period of time (in the X axis). Chart 210 is the total number of executed operations (in the Y axis) performed over a period of time (in the X axis), whether successful or unsuccessful. The time series represented by each of charts 202, 206, and 210 is the same time series. That is, the time represented by each X axis of these charts is the same specific time and duration, and each of these charts represents operational data for the same specific operation.

[0037] In this example, the unsupervised machine learning model applied to the time series may have marked a portion of each graph as potentially related to code regression, network errors, and / or hardware issues. This is illustrated by the bold boxes (i.e., boxes 204, 208, and 212). As shown by the data represented in box 204 of the quality of service graph 202, the percentage of successfully executed operations for a particular operation decreases in a downward manner from approximately 100% to approximately 75%. As shown by the data represented in box 208 of the total unexpected failures graph 206, the number of unexpected failures for the same time period increases in an upward manner from approximately 1,000 unexpected errors per unit time to approximately 6,000 unexpected errors per unit time. As shown by the data represented in box 212 of the total count graph 210, the number of operations executed in the same time period remains relatively constant, at approximately 50,000 operations per unit time. One or more unsupervised machine learning models applied to the time series can identify patterns indicating that the operations executed in the time periods defined by boxes 204, 208, and 212 are potentially related to code regression, network issues, and / or hardware issues. The numbers and values ​​shown in each of charts 202 , 206 , and 210 are exemplary, and it should be understood that other numbers, values, percentages, and graphical patterns may be identified as being associated with potential problems by applying one or more unsupervised machine learning models.

[0038] Figure 3Illustrated is the use of a convolutional neural network 310 to process operational error data to identify code regressions associated with cloud-based application services. Figure 2 The boxes and corresponding graphics data are again represented by a portion of the graphics subenvironment 302, which includes a box 304 (corresponding to Figure 2 204 in, representing the quality of the service graph data that has been labeled by one or more unsupervised machine learning models), box 306 (corresponding to Figure 2 ), representing unexpected failure graph data that has been labeled by one or more unsupervised machine learning models, and box 208 (corresponding to Figure 2 Box 212 in the figure represents the total executed operation graph data that has been labeled by one or more unsupervised machine learning models).

[0039] Data corresponding to the time represented by each of the blocks 304, 306, and 308 for the specific operations represented by each of those graphs can be provided to a convolutional neural network 310 that has been trained to identify code regressions, network errors, and / or hardware issues associated with operations in a cloud-based application service. In some examples, only the time windows represented by each of the blocks 304, 306, and 308 can be provided to the neural network 310. That is, in some examples, only the specific time windows and operational data from those time windows that have been labeled by the machine learning model can be provided to the neural network. In other examples, operational data for those time windows and for a specified duration before and / or after each of those time windows can be provided to the neural network 310 for classification. The time series provided to the neural network 310 can therefore represent seconds, minutes, hours, days, weeks, or months of operational data for the specific operations.

[0040] In this example, neural network 310 has been trained to provide one of two outputs for each input time series: "Problem" 312 (i.e., the data represented by the time series is associated with a code regression, network problem, and / or hardware problem), or "No Problem" 314 (i.e., no code regression, network problem, and / or hardware problem is present in the time series data). However, neural network 310 can be trained to classify time series operational data into additional categories, including: a specific type of misclassification (e.g., code regression, network problem, hardware problem), the percentage likelihood that the time series operational data is associated with a specific type of problem, the severity of the labeled problem, etc.

[0041] Figure 4A graphical image representation of a time series 400 is illustrated, which can be provided to a neural network for classification of potential operational errors. In this example, each block in the time series 400 represents one hour of operational data for a particular operation. Based on the data represented in each corresponding row, the shading of each block represents the number of successes, unexpected errors, or expected errors. The darker the shading of the block, the more operations that belong to the corresponding category (success, unexpected error, expected error). In this example, each block in the top row of blocks represents the number of operations of a particular operation type that were successfully performed within the one-hour time series represented by each corresponding block; each block in the middle row of blocks represents the number of unexpected errors of a particular operation type received within the one-hour time series represented by each corresponding block; and each block in the bottom row of blocks represents the number of expected errors of a particular operation type received within the one-hour time series represented by each corresponding block.

[0042] In this example, neural network 402 has been trained to classify a three-by-twelve graphical representation of data, as shown in time series 400. However, it should be understood that neural network 402 can additionally or alternatively be trained to classify graphical representations of time series data containing more or fewer units (e.g., ten-by-ten, twelve-by-twelve, etc.). Similarly, while neural network 402 is shown as having been trained to classify data based on a combined graphical representation of three value types (i.e., success, unexpected error, expected error), neural network 402 can additionally or alternatively be trained to classify data based on a single value type and / or any combination of these value types. Furthermore, while neural network 402 is shown in this example as having been trained to classify time series data into either a "problem" 404 category or a "no problem" 406 category, it should be understood that neural network 402 can be trained to classify time series data into one or more alternative or additional categories (e.g., severity of the problem, type of problem, etc.). Furthermore, while the graphical representation of data is shown as being fed to the neural network 402 as blocks with shading representing various values, other graphical representations of the same data may be generated and fed to the neural network 402 for classification depending on what data type and / or image type it has been trained on.

[0043] Figure 5A

[0066] An exemplary method 500A for detecting and classifying service issues associated with a cloud-based application service begins at a start operation and flow moves to operation 502A.

[0044] At operation 502, operational event data for a plurality of operations associated with a cloud-based application service is monitored. According to an example, the plurality of operations may each have the same operation type (e.g., a save document operation type; a send message operation type; an open new document operation type, etc.). In other examples, the plurality of operations may have different operation types. The operational data may be monitored by a monitoring service and / or a telemetry service. The operational data may include: an indication of the operation type of each operation being performed; an indication of the time each operation was performed and / or initiated; an indication of the time it took each operation to complete or time out; an identification of the server or server cluster that performed each operation; and / or an approximate location from which each operation request originated.

[0045] The process continues from operation 502A to operation 504A, in which a statistically based unsupervised machine learning model is applied to the operational event data. In an example, the statistically based unsupervised machine learning model can be applied to a specified time series of the monitored operational data. In an example, the time series can include operations of a specific type that have been performed or initiated within a specified duration. The specified duration can be seconds, minutes, hours, days, weeks, or months. According to some examples, the statistically based unsupervised machine learning model can be applied only to a subset of operations of a specific operational type that are performed within a specified duration (e.g., based on location, based on the server performing the operation, etc.), rather than applying the statistically based unsupervised machine learning model to every one of those operations. For example, the unsupervised machine learning model applied to time series data can include a Gaussian model, a bloxplot method model, a class of support vector machine models, and / or a density-based application spatial clustering with a noise model. These models can identify whether one or more operational events in the time series belong to the normalized pattern of the system, or whether they are outliers that may require further review.

[0046] From operation 504A, flow continues to operation 506A, where, based on application of the unsupervised machine learning model, a subset of the operational event data is marked as potentially associated with a code regression, a network issue, and / or a hardware issue. In some examples, the subset of the operational event data can include all or a portion of the time series of the operational event data to which the statistical-based unsupervised machine learning model was applied at operation 504.

[0047] Flow continues from operation 506A to operation 508A, where a neural network is applied to the time series of operational event data. In an example, the neural network may be a convolutional neural network applied to a graphical image representation of the time series of operational event data. The neural network may have been manually trained to identify code regressions, network issues, and / or hardware issues based on manually classified time series operational event data.

[0048] Flow continues from operation 508A to operation 510A, where the time series of operational event data is marked for subsequent tracking if the neural network classifies the time series as associated with a positive code regression category. That is, if the neural network identifies all or a portion of the operational event data to which it is applied as corresponding to a code regression, the operational event data is marked for subsequent tracking. In some examples, the neural network may have one or more additional categories into which it can classify the time series data. For example, the neural network may classify the time series / operational event data into categories based on the type of problem associated with it (e.g., code regression, network error, hardware problem), the severity of the problem (e.g., based on how many users are affected, based on the type of operation that caused the problem, etc.), and / or based on the likelihood that the problem is related to one or more problems (e.g., the likelihood that the problem is related to a code regression is greater than 70%, the likelihood that the problem is related to a network error is greater than 90%, etc.).

[0049] Flow moves from operation 510A to an end operation and method 500A ends.

[0050] Figure 5B

[0066] Another exemplary method 500B for detecting and classifying service issues associated with a cloud-based application service begins at start operation 502B.

[0051] At operation 502B, a time series data set corresponding to a plurality of operational events of a cloud-based application service is received. The operational events may be associated with the same type of operation being performed over a period of seconds, minutes, hours, days, and / or weeks. In other examples, the operational events may be associated with different types of operations being performed over a period of seconds, minutes, hours, days, and / or weeks.

[0052] From operation 502B, the process continues to operation 504B, in which an image matrix of the data set is generated. The image matrix may include a graphic classification of each operation event into one of a plurality of result types, including: an expected error result type; an unexpected error result type; and a successful result type. Each cell of the matrix may correspond to a time unit of the time series. For example, a group of cells in the matrix corresponding to expected error result types may illustrate the number of expected errors associated with executing a particular operation type per unit time in the matrix. Another group of cells in the matrix corresponding to unexpected error result types may illustrate the number of unexpected errors associated with executing a particular operation type per unit time in the matrix. Another group of cells in the matrix may correspond to the number of operations of a particular operation type that were successfully executed per unit time in the matrix. The number of operations that resulted in expected results, unexpected results, and / or successful executions may be graphically illustrated in the image matrix in various ways, including shading, coloring, graphically represented by value elements, and the like.

[0053] From operation 504B, the flow continues to operation 506B, where a convolutional neural network is applied to the image matrix of the dataset. The convolutional neural network may have been trained to classify the image matrix into one or more categories. The categories may include: yes problem type (i.e., the image matrix / time series is associated with a code regression, a network problem, and / or a hardware problem), no problem type (i.e., the image matrix / time series is not associated with a code regression, a network problem, and / or a hardware problem), a problem type classification (i.e., the image matrix / time series is classified as a specific problem type that caused an operational error), and / or a problem classification type severity.

[0054] From operation 506B, flow continues to operation 508B, where the time series dataset is classified by the convolutional neural network as corresponding to a code regression or not corresponding to a code regression. The time series dataset may also be classified into any other categories discussed above into which a neural network may be trained to classify the dataset.

[0055] Flow moves from operation 508B to an end operation and method 500B ends.

[0056] Figure 6 and Figure 7 A mobile computing device 600 is shown in which embodiments of the present disclosure may be practiced, such as a mobile phone, smartphone, wearable computer, tablet computer, e-reader, laptop computer, AR-compatible computing device, or VR computing device. Figure 6, illustrates one aspect of a mobile computing device 600 for implementing these aspects. In a basic configuration, the mobile computing device 600 is a handheld computer having both input and output elements. The mobile computing device 600 typically includes a display 605 and one or more input buttons 610 that allow a user to enter information into the mobile computing device 600. The display 605 of the mobile computing device 600 may also function as an input device (e.g., a touch screen display). If included, an optional side input element 615 allows further user input. The side input element 615 may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 600 may incorporate more or fewer input elements. For example, in some embodiments, the display 605 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 600 is a portable telephone system, such as a cellular phone. The mobile computing device 600 may also include an optional keypad 635. The optional keypad 635 may be a physical keypad or a "soft" keypad generated on the touch screen display. In various embodiments, the output elements include a display 605 for displaying a graphical user interface (GUI), a visual indicator 620 (e.g., a light emitting diode), and / or an audio transducer 625 (e.g., a speaker). In some aspects, the mobile computing device 600 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 600 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0057] Figure 7 7 is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, mobile computing device 700 can implement some aspects in conjunction with system (e.g., architecture) 702. In one embodiment, system 702 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, games, and media client / player). In some aspects, system 702 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0058] One or more application programs 766 can be loaded into memory 762 and run on or in association with operating system 864. Examples of applications include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. System 702 also includes a non-volatile storage area 768 within memory 762. Non-volatile storage area 768 can be used to store persistent information that should not be lost when system 702 loses power. Applications 766 can use and store information in non-volatile storage area 768, such as email or other messages used by email applications. A synchronization application (not shown) also resides on system 702 and is programmed to interact with a corresponding synchronization application residing on a host computer to synchronize information stored in non-volatile storage area 768 with corresponding information stored on the host computer. It should be understood that other applications can be loaded into memory 762 and run on mobile computing device 700, including instructions for providing and operating a digital assistant computing platform.

[0059] System 702 has a power supply 770, which can be implemented as one or more batteries. Power supply 770 can also include an external power source, such as an AC adapter or a powered docking station to replenish or recharge the batteries.

[0060] System 702 may also include a radio interface layer 772 that performs the functions of sending and receiving radio frequency communications. Radio interface layer 772 facilitates wireless connectivity between system 702 and the "outside world" via a communications carrier or service provider. Transmissions to and from radio interface layer 772 are controlled by operating system 764. In other words, communications received by radio interface layer 772 can be passed to application programs 766 via operating system 764, and vice versa.

[0061] The visual indicator 620 can be used to provide visual notifications, and / or the audio interface 774 can be used to generate auditory notifications via the audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to the power supply 770 so that when activated, they remain on for the duration specified by the notification mechanism, even if the processor 760 and other components may be turned off to conserve battery power. The LED can be programmed to remain illuminated indefinitely until the user takes action to indicate the device's power status. The audio interface 774 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 774 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to embodiments of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of notifications, as described below. The system 702 may also include a video interface 776 that enables the onboard camera 630 to operate to record still images, video streams, and the like.

[0062] The mobile computing device 700 implementing the system 702 may have additional features or functionality. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 7 denoted by non-volatile storage area 768.

[0063] The data / information generated or captured by the mobile computing device 700 and stored via the system 702 may be stored locally on the mobile computing device 700, as described above, or the data may be stored on any number of storage media that can be accessed by the device via the radio interface layer 772 or via a wireless connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via the mobile computing device 700 via the radio interface layer 772 or via a distributed computing network. Similarly, such data / information can be readily transferred between computing devices for storage and use according to well-known data / information transmission and storage means, including electronic mail and collaborative data / information sharing systems.

[0064] Figure 8is a block diagram illustrating the physical components (e.g., hardware) of a computing device 800 that can be used to practice aspects of the present disclosure. The computing device components described below may have computer executable instructions for detecting code regressions, network issues, and / or hardware issues associated with cloud-based application services. In a basic configuration, the computing device 800 may include at least one processing unit 802 and system memory 804. Depending on the configuration and type of the computing device, the system memory 804 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 804 may include an operating system 805 suitable for running one or more code regression system monitoring programs. For example, the operating system 805 may be suitable for controlling the operation of the computing device 800. Furthermore, embodiments of the present disclosure may be implemented in conjunction with a graphics library, other operating systems, or any other application and is not limited to any particular application or system. This basic configuration is Figure 8 808. The computing device 800 may have additional features or functionality. For example, the computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 8 809 and non-removable storage device 810.

[0065] As described above, multiple program modules and data files can be stored in the system memory 804. When executed on the processing unit 802, the processing that the program module 806 (e.g., the real-time code regression telemetry application 820) can perform includes, but is not limited to, the aspects described herein. According to an example, the operation failure monitoring engine 811 can perform one or more operations associated with monitoring telemetry data for multiple operations performed by an application hosted by a cloud-based application service. The statistical machine learning model engine 813 can perform one or more operations associated with applying one or more machine learning models to operational event time series data to determine whether one or more data points in those time series are related to code regression, network problems, and / or hardware problems in the service, as well as the labeling of these data points. The neural network engine 815 can perform one or more operations on the time series operational event data that has been labeled by the statistical machine learning model engine 813, including applying a convolutional neural network to the data. The convolutional neural network may have been trained to classify the time series operational event data into one or more categories, such as problem type, problem severity, etc. The language processing classification model 817 can obtain time series data that has been positively classified as being related to code regression, network issues, and / or hardware issues, and apply the language processing model to these operation logs to further classify the data into categories, such as which operation is responsible for the problem, which server or server group is responsible for the problem, the severity of the problem's impact on the system, and so on.

[0066] Furthermore, embodiments of the present disclosure may be implemented on a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC) wherein Figure 8 Each or many of the components shown in can be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated by an SOC, the functions described herein regarding the ability of the client to switch protocols can be operated by dedicated logic integrated with other components of the computing device 800 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be implemented within a general-purpose computer or in any other circuit or system.

[0067] The computing device 800 may also have one or more input devices 812, such as a keyboard, a mouse, a pen, an audio or voice input device, a touch or slide input device, etc. Output devices 814 may also be included, such as a display, speakers, a printer, etc. The aforementioned devices are examples, and other devices may be used. The computing device 800 may include one or more communication connections 816 that allow communication with other computing devices 850. Examples of suitable communication connections 816 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuits; universal serial bus (USB), parallel and / or serial ports.

[0068] As used herein, the term computer-readable media may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 804, removable storage device 809, and non-removable storage device 810 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other article that can be used to store information and can be accessed by computing device 800. Any such computer storage media may be part of computing device 800. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0069] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0070] Figure 9One aspect of the architecture of a system for processing data received from a remote source at a computing system, such as a personal / general purpose computer 904, a tablet computing device 906, or a mobile computing device 908, as described above, is illustrated. The content displayed at the server device 902 can be stored in different communication channels or other storage types. For example, a directory service 922, a web portal 924, a mailbox service 926, an instant messaging repository 928, or a social networking site 930 can be used to store various documents. Program modules 806 can be employed by clients that communicate with the server device 902, and / or program modules 806 can be employed by the server device 902. The server device 902 can provide data to and from client computing devices, such as personal / general purpose computers 904, tablet computing devices 906, and / or mobile computing devices 908 (e.g., smart phones), over a network 915. For example, the above description of Figure 6-8 The depicted computer system may be embodied in a personal / general purpose computer 904, a tablet computing device 906, and / or a mobile computing device 908 (e.g., a smartphone). In addition to receiving graphics data that may be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these embodiments of the computing device may obtain content from a repository 916.

[0071] For example, aspects of the present disclosure are described above with reference to the block diagrams and / or operational descriptions of methods, systems, and computer program products according to aspects of the present disclosure. The functions / actions noted in the blocks may not occur in the order shown in any flowchart. For example, depending on the functions / actions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0072] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the best mode of the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. No matter whether shown and described in combination or separately, various features (structures and methods) are intended to be selectively included or omitted to produce an embodiment with a specific feature group. Description and explanation of the disclosure have been provided, and those skilled in the art can envision changes, modifications and alternative aspects within the spirit of the broader aspects of the overall inventive concept embodied in this application, which changes, modifications and alternative aspects do not deviate from the broader scope of the disclosure claimed.

[0073] The various embodiments described above are provided by way of illustration only and should not be construed as limiting the appended claims. Those skilled in the art will readily recognize various modifications and changes that can be made without following the example embodiments and applications shown and described herein and without departing from the true spirit and scope of the appended claims.

Claims

1. A method for detecting code regression associated with a cloud-based application service, the method comprising: monitoring operational event data of a plurality of operations associated with the cloud-based application service to generate a time series of the operational event data; applying a statistically based unsupervised machine learning model to the time series of operational event data, wherein the statistically based unsupervised machine learning model is trained to identify data points that fall outside a normalized pattern of related operations in the plurality of operations; labeling the identified data points to generate a subset of the time series of the operational event data; providing the subset of the time series of the operational event data to a neural network trained to classify the subset of the time series of the operational event data; and Based on the classification, the time series of the operational event data is marked for subsequent tracking.

2. The method according to claim 1, wherein The neural network is a convolutional neural network, and the convolutional neural network is applied to the graphical image representation of the subset of the time series of the operational event data.

3. The method according to claim 1, wherein The neural network has been further trained to identify code regressions based on manually classified time series of the operational event data.

4. The method according to claim 2, wherein: Applying the convolutional neural network to the graphical image representation of the subset of the time series of the operational event data includes classifying the subset of the time series of the operational event data into one of a plurality of categories comprising: a positive code regression category; and a negative code regression category.

5. The method of claim 1 , further comprising: If the neural network classifies the subset of the time series as a negative code regression category, then the subset of the time series of the operational event data is ignored.

6. The method of claim 1, wherein: The statistically based unsupervised machine learning model is one of: a Gaussian model; a box plot method model; a one-class support vector machine model; and a density-based application space clustering with a noise model.

7. The method of claim 1 , further comprising: identifying a plurality of error logs corresponding to operational events in the time series of the operational event data; applying a language processing model to each of the error logs; as well as Each operational event in the time series of operational event data is classified into a severity category based on application of the language processing model.

8. The method of claim 7, wherein: The classification is based on one or more of: a type of operational event; a number of users affected by the code regression; and a number of sites currently still experiencing errors associated with the code regression.

9. The method of claim 7, wherein: The language processing model is one of the following: a recurrent neural network; a convolutional neural network; a bag-of-words model; a rule-based model; a random forest model; and a word segmentation model.

10. A system for identifying and classifying service issues associated with a cloud-based application service, comprising: a memory for storing executable program code; as well as one or more processors functionally coupled to the memory, the one or more processors responsive to computer-executable instructions contained in the program code and configured to: receiving a time series data set corresponding to a plurality of operational events of the cloud-based application service; applying a statistically based unsupervised machine learning model to the time series dataset, wherein the statistically based unsupervised machine learning model is trained to identify data points that fall outside a normalized pattern of related operations among the plurality of operations; labeling the identified data points to generate a subset of the time series dataset; generating an image matrix of the subset of the time series dataset, wherein the image matrix includes a graphical classification for each of the operational events; applying a convolutional neural network to the image matrix of the subset of the time series dataset; and The subset of the time series dataset is classified as corresponding to a code regression or not corresponding to a code regression by the convolutional neural network.

11. The system of claim 10, wherein: The convolutional neural network has been trained to identify code regressions based on a time series of manually classified operational event data.

12. A computer-readable storage device comprising executable instructions that, when executed by one or more processors, facilitate detecting code regressions associated with a cloud-based application service, the computer-readable storage device comprising instructions executable by the one or more processors to: monitoring operational event data of a plurality of operations associated with the cloud-based application service to generate a time series of the operational event data; applying a statistics-based unsupervised machine learning model to the time series of operational event data, wherein: The statistically based unsupervised machine learning model is trained to identify data points that fall outside of a standardized pattern of related operations among the plurality of operations; labeling the identified data points to generate a subset of the time series of the operational event data; providing the subset of the time series of the operational event data to a neural network trained to classify the subset of the time series of the operational event data; and Based on the classification, the time series of the operational event data is marked for subsequent tracking.

13. The computer-readable storage device of claim 12, wherein: The neural network is a convolutional neural network.

14. The computer-readable storage device of claim 12, wherein: The neural network has been further trained to identify code regressions based on a time series of manually classified operational event data.

15. The computer-readable storage device of claim 12, wherein: The neural network is a convolutional neural network, and the convolutional neural network is applied to a graphical image representation of the subset of the time series of the operational event data, and when applying the convolutional neural network to the graphical image representation, the instructions are further executable by the one or more processors to: The subset of the time series of operational event data is classified into one of a plurality of categories comprising: a positive code regression category; and a negative code regression category.

Images