Systems and methods for anomaly detection

Abstract

A method for identifying a contributing cause of an anomaly, comprising: receiving a time series data set representing a metric that varies over time, wherein the time series data has at least two dimensions; for each of two or more timestamps in the time series data set, generating first and second charts that represent: (i) the metric at that timestamp, (ii) at least two dimensions at that timestamp, and (iii) the association between the metrics at that timestamp; analyzing the first and second charts associated with each timestamp to identify a specific timestamp that includes an anomaly; and, analyzing the first and second charts associated with the identified specific timestamp to identify the nodes that contributed to the anomaly.

Description

Background Art

[0001] Sometimes, a system such as a computer system experiences anomalies such as spikes in memory usage or software crashes. This can help determine the cause of such anomalies, such as a new software update that unexpectedly causes an increase in memory utilization. However, it is difficult to determine the cause of anomalies in complex systems. Summary of the Invention

[0002] One implementation of the present disclosure relates to a method for identifying a contributing cause of an anomaly, including: receiving a time series data set representing measured values that vary over time, where the time series data has at least two dimensions; for each of two or more timestamps in the time series data set, generating first and second charts that represent: (i) the measured value at that timestamp, (ii) at least two dimensions at that timestamp, and (iii) the association between the measured values at that timestamp, where the measured value of the first chart includes the measured value corresponding to that timestamp, and where the measured value of the second chart is based on an expected value generated from the received time series data set; analyzing the first and second charts associated with each timestamp to identify a specific timestamp that includes an anomaly; and analyzing the first and second charts associated with the identified specific timestamp to identify the nodes that contribute to causing the anomaly.

[0003] In some implementations, the first chart includes a plurality of nodes having measured values associated with the measured value at that timestamp, each node corresponding to one or more of at least two dimensions, and where the first chart further includes edges connecting two nodes among the plurality of nodes, the edges representing the association between the two nodes; and where the second chart includes a plurality of nodes having expected values associated with the measured value at that timestamp, each node corresponding to one or more of at least two dimensions, and where the second chart further includes edges connecting two nodes among the plurality of nodes, the edges representing the association between the two nodes. In some implementations, analyzing the first and second charts associated with each timestamp includes: generating first and second feature vectors associated with the first and second charts of a specific timestamp, and comparing the angle between the first and second feature vectors.

[0004] In some implementations, analyzing the first and second graphs associated with an identified particular timestamp includes: performing an operation using a first eigenvector associated with the first graph and a second eigenvector associated with the second graph, where the result includes the importance of each of the multiple nodes of the first graph in contributing to the anomaly. In some implementations, the operation is vector subtraction, and where the result is a vector. In some implementations, analyzing the first and second graphs associated with an identified particular timestamp includes: measuring the entropy of the first and second graphs. In some implementations, generating the first and second graphs includes: generating a plurality of first nodes; iteratively generating parent nodes, where each parent node is connected to a child node by an edge; and, iteratively assigning weights to the edges connecting each parent node and each child node. In some implementations, at least two dimensions describe the characteristics of an entity that measures values in a time series dataset. In some implementations, an expected value is generated based on a Bayesian analysis of the time series dataset. In some implementations, the method further includes: determining, by one or more processing circuits, a root cause of the anomaly based on the identified nodes, where the root cause is associated with the dimensions of the identified nodes. In some implementations, the method further includes determining, by one or more processing circuits, a solution to the anomaly based on the root cause.

[0005] Another implementation of the present disclosure relates to a system for identifying contributing causes of an anomaly, including one or more processing circuits, the one or more processing circuits including one or more processors and one or more memories, the one or more memories having instructions stored thereon that, when executed by the one or more processors, cause the one or more processing circuits to: receive a time series dataset representing metric values that vary over time, where the time series data has at least two dimensions; for each of two or more timestamps in the time series dataset, generate a first and second graph, the first and second graphs representing: (i) the metric values at that timestamp, (ii) at least two dimensions at that timestamp, and (iii) the associations between the metric values at that timestamp, where the metric values of the first graph include the metric values corresponding to that timestamp, and where the metric values of the second graph are based on an expected value generated from the received time series dataset; analyze the first and second graphs associated with each timestamp to identify a particular timestamp that includes an anomaly; and, analyze the first and second graphs associated with the identified particular timestamp to identify the nodes that contribute to causing the anomaly.

[0006] In some implementations, the first graph includes a plurality of nodes, each node corresponding to one or more of at least two dimensions, and wherein the first graph further includes edges connecting two of the plurality of nodes, the edges representing an association between the two nodes; and wherein the second graph includes a plurality of nodes having expected values associated with the metric values at that timestamp, each node corresponding to one or more of at least two dimensions, and wherein the second graph further includes edges connecting two of the plurality of nodes, the edges representing an association between the two nodes. In some implementations, analyzing the first and second graphs associated with each timestamp includes: generating first and second feature vectors associated with the first and second graphs of a particular timestamp, and comparing the angles between the first and second feature vectors.

[0007] In some implementations, analyzing the first and second graphs associated with an identified particular timestamp includes: performing an operation using a first feature vector associated with the first graph and a second feature vector associated with the second graph, wherein the result includes the importance of each of the plurality of nodes of the first graph in contributing to the anomaly. In some implementations, the operation is vector subtraction, and wherein the result is a vector. In some implementations, analyzing the first and second graphs associated with an identified particular timestamp includes: measuring the entropy of the first and second graphs. In some implementations, generating the first and second graphs includes: generating a plurality of first nodes; iteratively generating parent nodes, wherein each parent node is connected to a child node by an edge; and, iteratively assigning weights to the edges connecting each parent node and each child node. In some implementations, at least two dimensions describe the characteristics of an entity that generates the metric values in the time series dataset. In some implementations, the expected values are generated based on a Bayesian analysis of the time series dataset.

[0008] Another implementation of the present disclosure relates to one or more non-transitory computer-readable storage media having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive a time series data set representing a time-varying metric value, where the time series data has at least two dimensions, and where at least two dimensions describe characteristics of an entity that generates the value of the metric in the time series data set; for each of two or more timestamps in the time series data set, generate first and second charts that represent: (i) the metric value at that timestamp, (ii) at least two dimensions at that timestamp, and (iii) the association between the metric values at that timestamp, where the metric value of the first chart includes the metric value corresponding to that timestamp, and where the metric value of the second chart is based on an expected value generated from the received time series data set; analyze the first and second charts associated with each timestamp to identify specific timestamps that include anomalies; and, analyze the first and second charts associated with the identified specific timestamps by generating first and second feature vectors associated with the first and second charts of the specific timestamp and comparing the angles between the first and second feature vectors to identify nodes that contribute to the anomalies that result.

[0009] In some implementations, the expected value is generated based on a Bayesian analysis of the time series data set. In some implementations, the one or more processors are further configured to determine a root cause of the anomaly based on the identified nodes, where the root cause is associated with the dimensions of the identified nodes. In some implementations, the one or more processors are further configured to determine a solution to the anomaly based on the root cause.

[0010] Aspects and implementations may be combined as appropriate. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram illustrating a data analysis system for anomaly detection according to an illustrative implementation.

[0012] Figure 2 is a flowchart illustrating a method of using a Figure 1 system to identify one or more anomaly sources according to one implementation.

[0013] Figure 3 is a flowchart illustrating a method of using a Figure 1 system to generate charts for anomaly detection according to an illustrative implementation.

[0014] Figures 4A to 4C is a diagram illustrating a method of using a Figure 1 system to identify one or more anomaly sources according to an illustrative implementation.

[0015] Figure 5 is a block diagram of a computing system according to an illustrative implementation. Detailed implementation

[0016] The following is a more detailed description of various concepts related to methods, apparatuses, and systems for detecting anomalies and determining the main contributors to the anomalies, and their implementations. The various concepts introduced above and discussed in more detail below can be implemented in any of a variety of ways, as the concepts described are not limited to any particular implementation.

[0017] Typically, the root cause of a problem needs to be identified. For example, a new software update may be pushed to multiple devices, and it may be necessary to determine why the new software update caused some of the devices to crash. This is commonly referred to as root cause analysis.

[0018] Root cause analysis can be performed by comparing changes in metrics such as memory utilization, processor cycles, or failure rates over time. For example, many remote devices in a network may start to malfunction, and root cause analysis may determine that a new software version is the cause of the device failures based on identifying an increase in the number of device failures consistent with the deployment of the new software version. This disclosure relates to systems, methods, and apparatuses for identifying anomalous events and determining the main contributors to the identified anomalous events.

[0019] A solution includes a data analysis system that receives time series data, generates a diagram representing a hierarchy of the time series data for each timestamp, and compares the diagrams to identify anomalies in the data and determine one or more nodes contributing to the anomalies. In various implementations, the diagram is a data object composed of a set of vertices and a set of edges. The set of vertices is a finite non-empty set. The set of edges may be empty, but otherwise its elements are binary subsets of the set of vertices. In some implementations, the elements of the set of edges include edge weights. In various implementations, a set is a collection of distinct objects, none of which is the set itself. In various implementations, an anomaly is a deviation from an expected characteristic (e.g., value, structure, etc.). For example, anomalies may include peaks in processor clock speed, rapid increases in heat associated with a component, peaks in component current consumption, and / or sudden decreases in content interaction. In various implementations, the data analysis system generates two diagrams for each timestamp of the data, a first diagram based on observed values of a metric, and a second diagram based on expected values of the metric. In various implementations, the diagram includes nodes (e.g., set of vertices) corresponding to dimensions of the data and edges (e.g., set of edges) corresponding to connections between the dimensions. For example, the diagram may include a first node corresponding to a region such as the United States, a second node corresponding to a software version, and a third node corresponding to both the region and the software version, where the third node is linked to the first and second nodes, thereby indicating that the software version is deployed in the region (or alternatively the region deploys the software version). In various implementations, the data analysis system generates the second diagram based on a Bayesian analysis of the data. For example, the data analysis system may generate the second diagram by analyzing the data to determine the expected values of metrics associated with specific dimensions at each point in time (e.g., for each timestamp). In various implementations, the data analysis system may identify a set of two diagrams with anomalies (e.g., one based on observed metrics and one based on expected values of the metrics) and may compare the two diagrams to identify the nodes contributing to the anomalies. In various implementations, the data analysis system may determine the cause of the anomalies based on the identified nodes (e.g., dimensions). As a simple example, the data analysis system may receive a set of time series data corresponding to metrics of computer memory usage over a two-week period. The data may include several dimensions such as computer type, software version, region, etc. The data analysis system may generate an observed diagram (e.g., an object composed of a set of vertices and a set of edges, etc.) and an expected diagram (e.g., an object composed of a set of vertices and a set of edges, etc.) for each timestamp of the received data. The data analysis system may compare pairs of diagrams to identify timestamps including anomalies such as peaks in memory utilization, and may compare the observed diagram and the expected diagram corresponding to the identified timestamps to identify the nodes contributing to the anomalies, such as nodes corresponding to a specific software version in a specific region.

[0020] Now refer toFigure 1 , shows a data analysis system 100 for identifying anomalous events and determining the primary contributors to the identified anomalous events according to an illustrative implementation. In various implementations, the data analysis system 100 receives data from external systems and generates a set of charts based on observed metrics and expected metrics. The data analysis system 100 can analyze the set of charts to identify the set of charts associated with an anomaly. The data analysis system 100 can analyze the set of charts to identify one or more primary contributors (e.g., causes, factors, etc.) associated with the anomaly. In some implementations, the data analysis system 100 is a distributed processing system, such as a distributed processing cluster, distributed servers, and / or a cloud computing system.

[0021] In various implementations, the data analysis system 100 receives data (e.g., time series data, etc.) from external systems. For example, the data analysis system 100 can communicate via a network such as the Internet, local area network, wide area network, metropolitan area network, or other regional network, intranet, satellite network, combinations thereof, or any other type of electronic communication network to receive data from external systems. The data analysis system 100 can be a server, distributed processing cluster, cloud processing system, or any other computing device. The data analysis system 100 can include or execute at least one computer program or at least one script. In some implementations, the data analysis system 100 includes a combination of software and hardware, such as one or more processors configured to execute one or more scripts. External systems can include other computing systems, such as analysis systems, server management systems, distributed processing systems, etc.

[0022] The data analysis system 100 may include a processing circuit 110 having a processor 120 and a memory 130. The memory 130 may have instructions stored thereon that, when executed by the processor 120, cause the processing circuit 110 to perform the various operations described herein. The operations described herein may be implemented using software, hardware, or a combination thereof. The processor 120 may include a microprocessor, an ASIC, an FPGA, etc., or a combination thereof. In many implementations, the processor 120 may be a multi-core processor or a processor array. The memory 130 may include, but is not limited to, an electronic, optical, magnetic, or any other storage device capable of providing program instructions to the processor 120. The memory 130 may include a floppy disk, a CD-ROM, a DVD, a magnetic disk, a memory chip, a ROM, a RAM, an EEPROM, an EPROM, a flash memory, an optical medium, or any other suitable memory from which the processor 120 may read instructions. These instructions may include code from any suitable computer programming language, such as, but not limited to, C, C++, C#, Java, JavaScript, Perl, HTML, XML, Python, and Visual Basic. The memory 130 may include an observed graph builder circuit 132, an expected graph builder circuit 134, an analysis circuit 136, and a reporting circuit 138.

[0023] The observed graph builder circuit 132 may receive data and generate a data structure based on the received data. In various implementations, the observed graph builder circuit 132 receives data from an external system. In various implementations, the data includes time series data. In various implementations, the observed graph builder circuit 132 generates a data object (e.g., a graph) consisting of a vertex set and an edge set, where each element in the vertex set corresponds to a data slice having one or more dimensions. For example, a 3-dimensional slice may be an element of the vertex set having observed metric values associated with a specific country, a specific software version, and a specific content item. In various implementations, the graph generated by the observed graph builder circuit 132 corresponds to the observed metric values included in the received data. The generation of the data structure is discussed below with reference to Figure 3 A simple example is as follows.

[0024] The Observation Chart Builder Circuit 132 can receive data from an external system. For each time point, in addition to the metric, the data can be multi-series data that also has one or more dimensions. For example, a single multi-series data element might be: "time_value=2010-10-1 10:00:00.000-01:00,country_id=2116,serving_restriction=SERVING_RESTRICTION_NO_RESTRICTION,creative_id=1238484849,interactions=12", where "interactions" is the metric. The Observation Chart Builder Circuit 132 can partition the multi-series data by timestamp. For example, the Observation Chart Builder Circuit 132 can partition the multi-series data into multiple data structures, each data structure containing data corresponding to a specific timestamp (e.g., "2010-10-1 10:00:00.000-01:00", as shown above, etc.). The Observation Chart Builder Circuit 132 can generate nodes for each multi-series data element (assuming no duplicates in the multi-series data) and / or one or more edges connecting node pairs (e.g., vertices in a vertex set, etc.). For example, a node can be a tuple including one or more dimensions and a metric value, and an edge can be a tuple including a first node, a second node, and an edge weight. In various implementations, the metric value corresponds to the observed metric value included in the received data. The edge weight can correspond to the impact of the second node on the metric value of the first node. For example, the first node can have a metric value of 100 and can be linked via an edge to 3 other nodes: the second node has a metric value of 30, the third node has a metric value of 30, and the fourth node has a metric value of 40, where the edge weight of the edge linking the first node and the fourth node is 40. In some implementations, the edge weights are normalized. For example, in the previous example, the edge weight of the edge linking the first node and the fourth node can be 0.4. In various implementations, the chart structure generated by the Observation Chart Builder Circuit 132 is hierarchical. For example, a first node with dimensions "country_id" and "software_version" might be hierarchically lower than a second node with a single dimension "country_id". Continuing the previous example, the metric value of the second node can correspond to the sum of the metric values of all other nodes with the dimension "country_id". In various implementations, the edges are directed. For example, an edge can connect a parent node to a child node. In various implementations, the chart is a directed acyclic graph (DAG) or other flow graph.

[0025] The expected graph builder circuit 134 can receive data and generate a data structure based on the received data. In various implementations, the expected graph builder circuit 134 receives data from an external system. In various implementations, the data includes time series data. In various implementations, the expected graph builder circuit 134 generates a data object (e.g., a graph) consisting of a set of vertices and a set of edges, where each element in the set of vertices corresponds to a data slice having one or more dimensions. For example, a 3D slice can be an element of the set of vertices having expected metric values associated with a particular country, a particular software version, and a particular content item. In various implementations, the graph generated by the expected graph builder circuit 134 corresponds to the expected metric values generated using the received data. In various implementations, the expected graph builder circuit 134 generates a graph similar to the observed graph builder circuit 132, except that the expected metric values (e.g., the metric values generated using statistical analysis, etc.) corresponding to each node (e.g., slice, etc.) are used instead of the observed metric values themselves. A simple example is as follows.

[0026] The expected graph builder circuit 134 may receive data from an external system. In various implementations, this data is the same data as that received by the observed graph builder circuit 132. Additionally or alternatively, the expected graph builder circuit 134 may receive additional data. For example, in addition to the data received by the observed graph builder circuit 132, the expected graph builder circuit 134 may also receive historical data associated with a metric. In addition to the metric, for each time point, the data may be multi-series data with one or more dimensions. For example, a single multi-series data element may be: "time_value=2010-10-1 00:00:00.000-01:00,country_id=2116,serving_restriction=SERVING_RESTRICTION_NO_RESTRICTION,creative_id=1238484849,interactions=12", where "interactions" is the metric. In some implementations, the expected graph builder circuit 134 partitions the multi-series data by timestamp. In various implementations, the expected graph builder circuit 134 analyzes the received data to generate an expected metric value corresponding to each slice. For example, the expected graph builder circuit 134 may implement a Bayesian time series algorithm that analyzes the received data and / or historical data to generate a metric value associated with the finest-grained nodes (e.g., nodes at the bottom of the hierarchy, nodes with the largest number of dimensions, etc.). In some implementations, the expected graph builder circuit 134 generates a probability distribution associated with the metric value at each slice. In various implementations, the expected graph builder circuit 134 uses Bayesian statistics to generate the expected metric value. However, many other statistical methods and / or algorithms, such as machine learning (e.g., neural networks, etc.), may be used. The expected graph builder circuit 134 may generate nodes (e.g., vertices in a vertex set) for each slice and / or one or more edges of a connected node pair. For example, a node may be a tuple including one or more dimensions and a metric value, and an edge may be a tuple including a first node, a second node, and an edge weight. In various implementations, the metric value is the expected metric value generated by the expected graph builder circuit 134. The edge weight may correspond to the influence of the second node on the metric value of the first node. In some implementations, the edge weight is normalized. In various implementations, the graph structure generated by the expected graph builder circuit 134 is hierarchical. In various implementations, the expected graph builder circuit 134 generates the graph bottom-up. For example, the expected graph builder circuit 134 may generate an expected metric value for the finest-grained nodes and may propagate parent nodes and edges based on the expected metric values of the child nodes. In various implementations, the edges are directed.In various implementations, the graph is a directed acyclic graph (DAG) or other flow graph. In various implementations, it is expected that the graph builder circuit 134 generates a graph for each timestamp in a multi-series of data. In various implementations, each graph generated by the expected graph builder circuit 134 is paired with a graph generated by the observed graph builder circuit 132 corresponding to the same timestamp.

[0027] The analysis circuit 136 can analyze the graphs generated by the observed graph builder circuit 132 and the expected graph builder circuit 134 to identify abnormal events and determine the main contributors to the identified abnormal events. In various implementations, the analysis circuit 136 analyzes a set of graphs associated with each timestamp to identify the set of graphs corresponding to timestamps including anomalies. Additionally or alternatively, the analysis circuit 136 can analyze the set of graphs to identify one or more nodes. In various implementations, one or more nodes correspond to the main contributors to the identified anomalies. For example, the analysis circuit 136 can receive a set of graphs associated with memory utilization across multiple dimensions and can identify an anomaly corresponding to a peak in memory utilization associated with a device in a specific region and can identify a specific node contributing to the anomaly (e.g., a slice corresponding to a specific dimension, etc.). In various implementations, the analysis circuit 136 generates a measurement of node activity associated with the graph. For example, the analysis circuit 136 can generate one or more eigenvectors based on the adjacency matrix of the graph. In some implementations, the analysis circuit 136 generates one or more PageRank vectors. For example, the analysis circuit 136 can generate a PageRank vector for each graph and can compare the PageRank vectors to identify anomalies and / or nodes. As an additional example, the analysis circuit 136 can generate a first PageRank vector for a first set of graphs and can generate a second PageRank vector for a second set of graphs and can generate the cosine similarity between the first and second PageRank vectors to identify an observed graph with an anomaly (e.g., an observed graph that is significantly different from the expected graph, a graph that may have an abnormal node, etc.). As yet another example, the analysis circuit 136 can generate a first PageRank vector for a first graph (e.g., an observed graph) and a second PageRank vector for a second graph (e.g., an expected graph), and can compare the first and second PageRank vectors component by component to identify a specific node. The analysis of the graph will be discussed in detail below with reference to Figures 4B to 4C The analysis of the graph will be discussed in detail below with reference to

[0028] The reporting circuit 138 can generate and send reports. In various implementations, the reporting circuit 138 sends reports to an external system. The report may include one or more identified anomalies and / or one or more primary contributors to the identified anomalies. In various implementations, the primary contributor includes a dimension associated with the node. For example, the reporting circuit 138 can send a report indicating that an anomaly has been detected and that the cause of the anomaly is associated with a specific software version in a specific area. In various implementations, the reporting circuit 138 identifies the root cause. For example, the reporting circuit 138 can receive information from the analysis circuit 136 and identify the root cause based on the received information. In some implementations, the reporting circuit 138 generates statistical information. For example, the reporting circuit 138 can perform a statistical analysis on the information received from the analysis circuit 136 and generate aggregated statistics. Additionally or alternatively, the reporting circuit 138 can identify specific components and / or entities associated with the anomaly. For example, the reporting circuit 138 can identify a specific software version that caused a peak in memory utilization on a user device.

[0029] In various implementations, the reporting circuit 138 sends data periodically. For example, the reporting circuit 138 can send data at a predetermined time. As another example, the reporting circuit 138 can send data at regular intervals (e.g., every ten minutes, every ten hours, etc.). In some implementations, the reporting circuit 138 sends data dynamically. For example, the reporting circuit 138 can send data in response to receiving information from the analysis circuit 136.

[0030] Now refer to Figure 2 , a method 200 for identifying anomaly events and determining primary contributors to the identified anomaly events is shown according to an illustrative implementation. In various implementations, the data analysis system 100 executes the method 200. For example, an external system can send data to the data analysis system 100 and the data analysis system 100 can execute the method 200 to identify one or more primary contributors associated with the anomaly and can send the primary contributor to the external system.

[0031] In step 202, the data analysis system 100 can receive time series data. The time series data can be associated with a metric over time. For example, the time series data can include memory utilization values associated with a distributed processing system during operation. In various implementations, the time series data is one-dimensional or multi-dimensional. For example, the time series data can include regions, languages, task groups, software versions, identifiers, etc. The time series data can be received from various sources. For example, the data analysis system 100 can receive time series data from an optimization engine, a mobile application management platform, a web server, an alert system, an analysis system, an advertising service system, and / or any other system.

[0032] In step 204, the data analysis system 100 may partition the time series data by timestamps. In various implementations, step 204 includes generating separate data objects corresponding to each timestamp in the time series data. For example, the data analysis system 100 may generate a table for each timestamp, where each table includes time series values associated with a particular timestamp. In step 206, the data analysis system 100 may generate a graph from the time series data. In various implementations, the data analysis system 100 generates a graph from the data objects separated by timestamps. For example, the data analysis system 100 may generate a pair of graphs for each table corresponding to a particular timestamp. In various implementations, the data analysis system 100 generates two graphs for each timestamp. For example, the data analysis system 100 may generate a first graph based on the observed metric values corresponding to a particular timestamp and may generate a second graph based on the expected metric values generated based on the observed metric values at a particular timestamp. In some implementations, the data analysis system 100 generates an expected graph for each timestamp based on historical data. The generation of the graph will be discussed in detail with reference to the following Figure 3 The generation of the graph will be discussed in detail.

[0033] In step 208, the data analysis system 100 compares the generated graphs to determine a set of graphs. For example, the data analysis system 100 may analyze the set of graphs generated in step 206 to identify a pair of graphs with anomalies. An anomaly may correspond to the difference between the observed metric values and the expected metric values. In various implementations, step 208 includes generating a PageRank vector for each graph. For example, the data analysis system 100 may generate a PageRank vector for each graph and may compare the cosine similarity between the PageRank vectors of the observed graph and the expected graph to identify the set of graphs with anomalies. The PageRank vector may identify the importance level of a particular node in the graph based on the number and quality of the connections of the nodes. In some implementations, the PageRank vector is the solution of a linear system where, is a column stochastic matrix, 0 ≤ a < 1, and is a probability vector. In some implementations, the data analysis system 100 may compare the similarity of the graphs. For example, the data analysis system 100 may implement a function where, G t is the first graph, G t+h is the second graph, activity t is the measurement of the activity associated with the first graph, activity t+his a measure of the activity associated with the second graph, and θ is the angle between the vectors representing the first graph and the second graph. In some implementations, activity corresponds to the PageRank of the graph.

[0034] In step 210, the data analysis system 100 can identify one or more sources of anomalies based on the set of graphs. For example, the data analysis system 100 can analyze the set of graphs determined in step 208 to identify one or more nodes that contribute to the identified anomalies. In various implementations, the identified nodes correspond to the nodes that are the main contributors (e.g., are the main factors, significantly contribute to bring about, etc.) to the identified anomalies. For example, the identified anomaly can be a peak (e.g., a metric) in the memory allocation associated with a mobile application, and the data analysis system 100 can identify the nodes (e.g., libraries, etc.) corresponding to a specific version of the dependencies used by the mobile application. In some implementations, the data analysis system 100 can identify the root cause associated with the anomaly. For example, the data analysis system 100 can determine which dimensions are associated with the nodes (e.g., based on slicing, etc.) and can determine which dimension caused the anomaly. In various implementations, step 210 includes comparing PageRank vectors. For example, the data analysis system 100 can calculate the per-component absolute difference between the observed graph and the expected graph to identify the nodes with the maximum value (e.g., the largest difference, etc.).

[0035] In step 212, the data analysis system 100 can send the results. In various implementations, the data analysis system 100 can send the results to an external system. For example, the data analysis system 100 can send the results to an external analysis system. In various implementations, the results include one or more root causes of the identified anomalies. Additionally or alternatively, the results can include additional information associated with the anomaly (e.g., what devices are affected, information describing the anomaly, the time when the anomaly occurred, context information associated with the anomaly, etc.). In some implementations, step 212 remedies the cause of the anomaly. For example, if a new software update causes the device to crash, the data analysis system 100 can identify the software version that caused the crash and can push a patch (e.g., software restoration, etc.) to correct the error. In various implementations, step 212 includes determining a solution for the anomaly and / or resolving the anomaly. For example, step 212 can include deploying additional computing resources to an area that is experiencing an unusually slow response time (e.g., due to a computing bottleneck, etc.).

[0036] Now refer to Figure 3, which more particularly shows step 206 of generating a chart from time series data according to an illustrative implementation. In various implementations, the data analysis system 100 performs step 206 twice, once to generate an observed chart and once to generate an expected chart. Additionally or alternatively, the data analysis system 100 may perform step 206 multiple times to generate multiple charts. Briefly, step 206 includes generating a chart bottom-up (e.g., first generating the most specific nodes and building up to the most general nodes, etc.). In various implementations, the data analysis system 100 generates and normalizes the edges between nodes when constructing subsequent nodes. In various implementations, the data analysis system 100 generates a chart based on observed metrics (e.g., time series data received during step 202, etc.). Additionally or alternatively, the data analysis system 100 may generate expected metrics (e.g., based on time series data received from historical data, etc. during step 202) and may generate a chart based on the generated expected metrics. For example, the data analysis system 100 may receive time series data corresponding to the observed metrics and may perform Bayesian time series analysis to calculate the expected metrics, and may use the expected metrics to generate an expected chart.

[0037] In step 300, the data analysis system 100 may generate leaf nodes and parent nodes. In various implementations, generating leaf nodes includes generating one or more elements for a vertex set. For example, the data analysis system 100 may generate elements of a vertex set for a node corresponding to having a metric "interactions = 12" and dimensions "time_value = 2010-10-11 00:00:00.000-01:00, country_id = 2116, serving_restriction = SERVING_RESTRICTION_NO_RESTRICTION, creative_id = 1238484849". In various implementations, leaf nodes correspond to the lowest-level nodes. For example, a leaf node may represent a slice with the maximum number of dimensions. In various implementations, a parent node represents the parent of a leaf node. To continue the previous example, the parent node may be a node having a metric "interactions = 22" and dimensions "time_value = 2010-10-11 00:00:00.000-01:00, country_id = 2116, serving_restriction = SERVING_RESTRICTION_NO_RESTRICTION". In various implementations, step 300 includes generating an edge between the leaf node and the parent node. For example, the data analysis system 100 may generate elements of an edge set corresponding to an edge having a value "value = 12" and connections "vertex_1 = 383, vertex_2 = 288". In some implementations, the edge set uses identifiers to identify elements in the vertex set. In various implementations, the value associated with an edge corresponds to the contribution of the child node to the metric value of the parent node. For example, the edge value (e.g., edge weight, etc.) may be calculated as W i = weight(V i ) = ∑m slice , where W i is the edge value of the edge, V i is the metric value associated with the parent node, and m slice is the metric value associated with the child node. In some implementations, an adjacency matrix is used to represent the edges. In various implementations, the data analysis system 100 generates elements of the vertex set and the edge set based on observed metric values (e.g., time series data received in step 202, etc.). Additionally or alternatively, the data analysis system 100 may generate elements of the vertex set and the edge set based on expected metric values (e.g., metric values generated based on a Bayesian analysis of the time series data received in step 202, etc.). In some implementations, a Laplacian matrix is used to represent the graph.

[0038] In step 302, the data analysis system 100 may normalize the connections between the generated nodes. For example, after performing step 300, the generated edges may have different values that are not comparable (e.g., based on different metric values, etc.), and the data analysis system 100 may normalize the edge weights to be within a range (e.g., zero to one, etc.). In various implementations, the data analysis system 100 implements a function where a i，j is the weight of the edge connecting the first node i and the second node j, W j is the metric value associated with the first node, and W i is the metric value associated with the second node. For example, the data analysis system 100 may normalize the edge weights associated with the edges between the leaf nodes and the parent nodes generated in step 300 to be within the range [0, 1].

[0039] In step 304, the data analysis system 100 may generate parent nodes. For example, the data analysis system 100 may generate a parent node corresponding to whichever is the highest-level node currently present on the generated graph. As an additional example, in the first execution of step 304, the data analysis system 100 may generate a third-level node (e.g., the parent's parent of the leaf node, etc.), and in the second execution of step 304, the data analysis system 100 may generate a fourth-level node (e.g., the parent's parent's parent of the leaf node, etc.). In various implementations, step 304 includes identifying slices from the received time series data that are at a higher level of abstraction (e.g., higher level, etc.) than the nodes currently included in the graph, and generating nodes corresponding to the identified slices. In some implementations, the generated nodes are not connected (e.g., via an edge) to other nodes. In various implementations, step 304 includes generating an edge connecting the generated parent node and the corresponding child node. In various implementations, steps 302 and 304 are repeated until all nodes have been generated (e.g., until all time series data has been included in the generated graph, etc.). In various implementations, the top node in the generated graph is the root node. In various implementations, steps 300-304 are used to generate an observed graph (e.g., representing a portion of the time series data corresponding to a single timestamp). Additionally or alternatively, steps 300-304 may be used to generate an expected graph (e.g., representing expected metric values generated based on the time series data and / or other data, etc.). In various implementations, the data analysis system 100 executes steps 300-304 for each timestamp in the time series data.

[0040] Now refer to Figures 4A to 4C, shows process 400 for generating a chart, identifying a chart with an anomaly, and determining a primary contributor to the identified anomaly according to an illustrative implementation. In various implementations, data analysis system 100 executes process 400. Process 400 may begin at step 410 when time series data 402 is received from an external system. In various implementations, data analysis system 100 receives time series data 402 from an external system such as an analytics system. Time series data 402 may include metrics such as memory usage and multiple dimensions such as device, software version, mobile device, etc. In various implementations, time series data 402 is hierarchical data. In some implementations, time series data 402 is granular data from which hierarchies and / or aggregations are calculated. For example, data analysis system 100 may receive time series data including a 5-dimensional slice "time_value = 2010-10-11 00:00:00.000 - 01:00, country_id = 2116, serving_restriction = SERVING_RESTRICTION_NO_RESTRICTION, creative_id = 1238484849, memory_usage = 10", and may aggregate across all values of "creative_id" to generate a 4-dimensional slice that is the parent of the 5-dimensional slice. In various implementations, the parent node has a metric value that is the sum of the metric values associated with the corresponding child nodes. In various implementations, data analysis system 100 may calculate aggregations at each possible level. Continuing with the previous example, data analysis system 100 may generate parent nodes corresponding to aggregations across "country_id" (1-dimensional slice), "country_id" and "serving_restriction" (2-dimensional slice), "serving_restriction" and "creative_id" (another 2-dimensional slice), "country_id", "serving_restriction" and "creative_id" (3-dimensional slice), etc.

[0041] In various implementations, step 410 includes separating time series data 402 by timestamp. In various implementations, data analysis system 100 separates time series data 402 by timestamp to generate one or more data structures 404. For example, data analysis system 100 may generate multiple tables from time series data 402, each table including time series data 402 associated with a specific timestamp.

[0042] At step 420, the data analysis system 100 may generate one or more chart sets 412 based on the data structure 404. In various implementations, each chart set 412 includes two charts, a first chart 406 and a second chart 408. Additionally or alternatively, each chart set 412 may include a different number of charts. The first chart 406 may be associated with an observed metric and the second chart 408 may be associated with an expected metric. In various implementations, step 420 includes generating the expected metric. For example, the data analysis system 100 may calculate the probability distribution of the metric associated with each slice based on the time series data 402 and may determine the expected metric based on the probability distribution. In various implementations, the data analysis system 100 performs a Bayesian analysis of the time series data 402 and / or other data (e.g., historical data, etc.). For example, the data analysis system 100 may perform a Bayesian analysis of the time series data 402 to generate an expected metric associated with a slice, such as "time_value = 2019-10-13 00:00:00:000-07:00, country_id = 2152, serving_restriction = SERVING_RESTRICTION_NO_RESTRICTION, creative_id = 12347805, memory_usage = 10, metric_mean = 11, upper_bound = 13.393993, lower_bound = 2.33993390". In various implementations, the expected metric associated with a slice includes an upper bound and a lower bound. In various implementations, the data analysis system 100 generates the first chart 406 based on the data structure 404. For example, the data analysis system 100 may generate the first chart 406 to represent the metric associated with each slice included in the time series data 402 and / or each slice calculated at a higher level (e.g., aggregation of lower level dimensions, etc.). In various implementations, the data analysis system 100 generates the second chart 408 based on the generated expected metric. In various implementations, generating the chart 412 includes calculating the metric associated with each possible slice (e.g., each possible combination of dimensions, etc.). Additionally or alternatively, the data analysis system 100 may calculate the metric associated with a portion of the possible slices. In various implementations, the time series data 402 is hierarchical data and the data analysis system 100 treats the hierarchical data as a chart.

[0043] At step 430, the data analysis system 100 can analyze the graph 412 to identify the graph of interest 414. The graph of interest 414 can be selected from the graph 412. In various implementations, step 430 includes identifying a first graph 406 and a second graph 408 corresponding to timestamps that represent significant shifts in the metrics associated with one or more nodes. For example, the data analysis system 100 can identify a set of graphs corresponding to timestamps where the metrics associated with one or more nodes of the observed graph are outside the range of the expected metrics in the corresponding nodes of the expected graph (e.g., upper bound and / or lower bound, etc.). In some implementations, step 430 includes calculating one or more eigenvectors associated with the graph 412. For example, the data analysis system 100 can generate eigenvectors associated with each graph in the graph 412 and compare the cosine similarity of each observed graph (e.g., the first graph 406, etc.) and the expected graph (e.g., the second graph 408, etc.) corresponding to each timestamp to identify the timestamp with the minimum cosine similarity, which can indicate the presence of an anomaly. Additionally or alternatively, other graph mining techniques can be used (e.g., generating PageRank vectors, treating the graph as a circuit and calculating the resistance associated with edge weights, etc.). In some various implementations, the data analysis system 100 compares the angles between the vectors associated with each graph (e.g., representing cosine similarity, etc.). In various implementations, the graph of interest 414 includes an anomaly. For example, the graph 412 can include a first set of graphs with a first first graph 406 (e.g., the observed graph) and a first second graph 408 (e.g., the expected graph), and a second set of graphs with a second first graph 406 (e.g., the observed graph) and a second second graph 408 (e.g., the expected graph). To continue this example, the data analysis system 100 can calculate the PageRank vectors corresponding to the four graphs, compare the cosine similarity between the PageRank vectors associated with the first first graph 406 and the first second graph 408, and the PageRank vectors associated with the second first graph 406 and the second second graph 408, and identify the first set of graphs or the second set of graphs as containing an anomaly based on the comparison of the cosine similarity of the PageRank vectors. In some implementations, the graph of interest 414 includes two graphs, one of the first graphs 406 (e.g., the observed graph) and one of the second graphs 408 (e.g., the expected graph). In some implementations, step 430 includes measuring the entropy of the graph 412 and determining the graph of interest 414 based on the measured entropy.

[0044] At step 440, the data analysis system 100 may analyze the graph of interest 414 to identify one or more nodes of interest 418. The nodes of interest 418 may correspond to nodes that significantly contribute to the deviation of the observed metric value from the expected metric value corresponding to the timestamp. For example, at step 430, the data analysis system 100 may identify a graph of interest 414 corresponding to a significant deviation in the observed impression rate compared to the expected impression rate associated with the content item at a specific timestamp, and at step 440, the data analysis system 100 may determine a node with the dimension "country_id = 2152, serving_restriction = SERVING_RESTRICTION_NO_RESTRICTION" that significantly contributes to the deviation. Additionally or alternatively, step 430 may include identifying an anomaly associated with the content item (e.g., a change in the served content item, conversion rate, etc.). For example, step 430 may include identifying a significant decrease in conversions associated with a content group and determining that a specific region and operating system primarily contribute to the decrease. As another example, step 430 may include identifying a significant increase in the customer lifetime value associated with an account with a set of characteristics and determining the specific characteristics that primarily contribute to the increase. In various implementations, the nodes of interest 418 are associated with the root cause of the identified anomaly. In various implementations, step 440 includes comparing the PageRank vectors associated with the graph of interest 414. For example, the data analysis system 100 may perform a component-by-component comparison of the PageRank vectors associated with a specific first graph 406 (e.g., observed graph) and a specific second graph 408 (e.g., expected graph) associated with the identified graph of interest 414 to identify one or more nodes of interest 418 with differences. In some implementations, the data analysis system 100 identifies the nodes of interest 418 based on a threshold difference.

[0045] Specific reference Figure 4C, which shows a selected graph 416 according to an illustrative implementation. The selected graph 416 may correspond to one of the graphs 414 of interest. For example, the selected graph 416 may be a specific first graph 406 corresponding to the identified graph 414 of interest. The selected graph 416 may include a data structure 422 having nodes 426 and edges 428. In various implementations, the nodes 426 are associated with slices of time series data. For example, a node 426 labeled "I" may represent time series data 402 corresponding to "time = 1 / 20 / 2020, device = 1, software version = 0, and mobile device = 1" and having a metric value "memory usage = 10". In various implementations, the data structure 422 may include computational slices corresponding to all possible higher-dimensional slices. For example, a node 426 labeled "G" may represent time series data 402 corresponding to "time = 1 / 20 / 2020, software version = 0, and mobile device = 1", and a node 426 labeled "C" may represent time series data 402 corresponding to "time = 1 / 20 / 2020, mobile device = 1". In various implementations, the top node 426 of the data structure 422 is the root node 424. For example, the root node 424 may be the parent node of nodes 426 labeled "A, B, C, and D". Still referring to Figure 4C , step 440 may include identifying a node 418 of interest. In various implementations, the node 418 of interest is associated with the root cause of an anomaly. For example, a node 426 labeled "E" may represent "time = 1 / 20 / 2020, software version = 1, and mobile device = 0", which may indicate that the combination of software version and mobile device type causes an anomaly, such as a peak in memory usage. In some implementations, step 440 includes measuring the entropy of the graph 414 of interest and determining the node 418 of interest based on the measured entropy.

[0046] Figure 5Illustrated is a description of a computing system 500, which can be used, for example, to implement any illustrative system described in the present disclosure (e.g., data analysis system 100, etc.). The computing system 500 includes a bus 505 or other communication component for transmitting information and a processor 510 coupled to the bus 505 for processing information. The computing system 500 also includes a main memory 515, such as random access memory (“RAM”) or other dynamic storage device, coupled to the bus 505 to store information and instructions to be executed by the processor 510. The main memory 515 can also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 510. The computing system 500 can further include a read-only memory (“ROM”) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for the processor 510. A storage device 525, such as a solid-state device, disk, or optical disc, is coupled to the bus 505 for persistently storing information and instructions.

[0047] The computing system 500 can be coupled via the bus 505 to a display 535 for displaying information to a user, such as a liquid crystal display or an active matrix display. An input device 530, such as a keyboard including alphanumeric keys and other keys, can be coupled to the bus 505 for transmitting information and command selections to the processor 510. In another implementation, the input device 530 has a touchscreen display 535. The input device 530 can include a cursor control, such as a mouse, trackball, or cursor direction keys, for transmitting direction information and command selections to the processor 510 and for controlling the movement of a cursor on the display 535.

[0048] In some implementations, the computing system 500 can include a communication adapter 540, such as a networking adapter. The communication adapter 540 can be coupled to the bus 505 and can be configured to enable communication with a computing or communication network 545 and / or other computing systems. In various illustrative implementations, any type of networking configuration can be implemented using the communication adapter 540, such as wired (e.g., via Ethernet), wireless (e.g., via WiFi, Bluetooth, etc.), pre-configured, ad hoc, LAN, WAN, etc.

[0049] In accordance with various implementations, the processes of the illustrative implementations described herein may be implemented by a computing system 500 in response to execution of instruction arrangements contained in main memory 515 by a processor 510. Such instructions may be read into main memory 515 from another computer-readable medium, such as a storage device 525. Execution of the instruction arrangements contained in main memory 515 causes the computing system 500 to perform the illustrative processes described herein. One or more processors in a multiprocessing arrangement may also be employed to execute the instructions contained in main memory 515. In alternative implementations, hardwired circuitry may be used in place of or in combination with software instructions to implement the illustrative implementations. Accordingly, implementations are not limited to any specific combination of hardware circuitry and software.

[0050] Although example processing systems have been described in Figure 5 , implementations of the subject matter and functional operations described in this specification may be performed using other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them.

[0051] In addition to the foregoing description, controls may be provided for a user to select which, if any, and when, the systems, programs, or features described herein may implement collecting user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location); and whether content or communications are sent to the user from a server. Additionally, certain data may be processed in one or more ways before it is stored or used, so as to remove personal identifying information. For example, a user's identity may be processed so that personal identifying information about the user cannot be determined, or the user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level) so that a user's specific location cannot be determined. Accordingly, a user may have control over what information is collected about the user, how that information is used, and what information is provided to the user. Where the systems described herein collect personal information about a user or use personal information, a user may be provided with an opportunity to control whether the program or feature collects user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location). Additionally or alternatively, certain data may be processed in one or more ways before it is stored or used, so as to remove personal identifying information.

[0052] Implementations of the subject matter and the operations described in this specification can be realized in digital electronic circuitry, or in computer software embodied in a tangible medium, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more computer storage media for execution by, or to control the operation of, a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to the appropriate receiver apparatus for execution by the data processing apparatus. A computer-readable storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be the source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The computer storage medium can be tangible and / or non-transitory.

[0053] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on or received from one or more computer-readable storage devices.

[0054] The term “data processing apparatus” or “computing device” encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or multiple or combinations of the foregoing. The apparatus can include special-purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus can also include code that creates an execution environment for the computer programs being discussed, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and the execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0055] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and the computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. The program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files that store one or more modules, subroutines, or portions of code). The computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0056] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuitry—e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)—and the apparatus can also be implemented as special-purpose logic circuitry. The circuits used herein can be implemented using hardware circuitry (e.g., FPGA, ASIC, etc.), software (instructions stored on one or more computer-readable storage media and executable by one or more processors), or any combination thereof.

[0057] Processors suitable for executing computer programs include, for example, any one or more processors of both general and special purpose microprocessors as well as any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. Essential elements of a computer are a processor for performing operations in accordance with the instructions and one or more memory devices for storing the instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as, magnetic disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to one or more mass storage devices to receive data therefrom or to transfer data thereto or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant ("PDA"), a mobile audio or video player, a game console, a global positioning system ("GPS") receiver, or a portable storage device (such as, a universal serial bus ("USB") flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as, EPROM, EEPROM, and flash memory devices; magnetic disks, such as, internal hard disks or removable disks; magneto-optical disks; and, CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0058] For providing interaction with a user, implementations of the subject matter described in this specification may be practiced using a computer having: a display device, such as, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and, a keyboard and a pointing device such as a mouse or a trackball by which the user may provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input received from the user may be in any form, including sound, speech, or tactile input. Additionally, a computer may interact with the user by: sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from a web browser on a user's client device.

[0059] Implementations of the subject matter described in this specification can be executed using a computing system that includes, for example, backend components such as a data server, or includes middleware components such as an application server, or includes frontend components such as a client computer having a graphical user interface or a web browser through which a user can interact with implementations of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication - for example, a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0060] A computing system can include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. In some implementations, the server sends data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., results of user interaction) can be received at the server from the client device.

[0061] Although this specification contains many details of specific implementations, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular implementations of a particular invention. Certain features described in this specification in the context of separate implementations can also be implemented in combination or in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable sub-combination. Additionally, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excluded from the combination, and the claimed combination can be directed to a sub-combination or variation of a sub-combination. Additionally, features described with respect to a particular heading can be used with respect to and / or in conjunction with illustrative implementations described under other headings; headings - if provided - are included only for purposes of readability and should not be construed as limiting any features provided with respect to such headings.

[0062] Similarly, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described implementations should not be construed as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products embodied on a tangible medium.

[0063] Accordingly, particular implementations of the subject matter have been described. Other implementations are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the particular order shown or sequential order to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for identifying contributing causes of an anomaly, comprising: Receiving, by one or more processing circuits, a time series data set representing a time-varying metric, wherein the time series data has at least two dimensions; For each of two or more timestamps in the time series data set, generating, by the one or more processing circuits, first and second charts that represent: (i) the metric at that timestamp, (ii) the at least two dimensions at that timestamp, and (iii) the associations between the metrics at that timestamp, wherein the metric of the first chart includes the metric corresponding to that timestamp, and wherein the metric of the second chart is based on an expected value generated from the received time series data set; Analyzing, by the one or more processing circuits, the first and second charts associated with each timestamp to identify a specific timestamp that includes an anomaly; and Analyzing, by the one or more processing circuits, the first and second charts associated with the identified specific timestamp to identify the nodes that contribute to causing the anomaly.

2. The method according to claim 1, wherein, The first chart includes a plurality of nodes having metrics associated with the metric at that timestamp, each node corresponding to one or more of the at least two dimensions, and wherein the first chart further includes edges connecting two of the plurality of nodes, the edges representing the associations between the two nodes; and wherein the second chart includes a plurality of nodes having expected values associated with the metric at that timestamp, each node corresponding to one or more of the at least two dimensions, and wherein the second chart further includes edges connecting two of the plurality of nodes, the edges representing the associations between the two nodes.

3. The method according to claim 1, wherein, Analyzing the first and second charts associated with each timestamp includes: generating, by the one or more processing circuits, first and second feature vectors associated with the first and second charts of the specific timestamp, and comparing the angles between the first and second feature vectors.

4. The method according to claim 2, wherein Analyzing the first and second charts associated with the identified specific timestamp includes: performing, by the one or more processing circuits, an operation using a first feature vector associated with the first chart and a second feature vector associated with the second chart to generate a result, wherein the result includes the importance of each of the plurality of nodes of the first chart in contributing to the anomaly.

5. The method according to claim 4, wherein, The operation is vector subtraction, and wherein the result is a vector.

6. The method according to claim 1, wherein Analyzing the first and second charts associated with the identified specific timestamp includes: measuring, by the one or more processing circuits, the entropy of the first and second charts.

7. The method according to claim 1, the method further comprising: Determining, by the one or more processing circuits, the root cause of the anomaly based on the identified nodes, wherein the root cause is associated with the dimensions of the identified nodes.

8. The method according to claim 7, the method further comprising: Determining, by the one or more processing circuits, a solution for the anomaly based on the root cause.

9. The method according to claim 1, wherein, Generating the first and second diagrams includes: Generating a plurality of first nodes by the one or more processing circuits; Iteratively generating parent nodes by the one or more processing circuits, wherein each parent node is connected to a child node by an edge; and Iteratively assigning weights to the edges connecting each parent node and each child node by the one or more processing circuits.

10. The method according to claim 1, wherein, The at least two dimensions describe characteristics of entities that generate the metric values in the time series dataset.

11. The method according to any one of claims 1 to 10, wherein, Generating the expected value based on a Bayesian analysis of the time series dataset.

12. A system for identifying contributing causes of anomalies, the system comprising: One or more processing circuits, including one or more processors and one or more memories, the one or more memories having instructions stored thereon that, when executed by the one or more processors, cause the one or more processing circuits to: Receive a time series dataset representing metric values that vary over time, wherein the time series data has at least two dimensions; For each of two or more timestamps in the time series dataset, generate first and second diagrams that represent: (i) the metric value at that timestamp, (ii) the at least two dimensions at that timestamp, and (iii) the association between the metric values at that timestamp, wherein the metric value of the first diagram includes the metric value corresponding to that timestamp, and wherein the metric value of the second diagram is based on an expected value generated from the received time series dataset; Analyze the first and second diagrams associated with each timestamp to identify specific timestamps that include anomalies; and Analyze the first and second diagrams associated with the identified specific timestamps to identify the nodes that contribute to causing the anomalies.

13. The system according to claim 12, wherein, The first diagram includes a plurality of nodes having metric values associated with the metric value at that timestamp, each node corresponding to one or more of the at least two dimensions, and wherein the first diagram further includes edges connecting two nodes among the plurality of nodes, the edges representing the association between the two nodes; and Wherein the second diagram includes a plurality of nodes having expected values associated with the metric value at that timestamp, each node corresponding to one or more of the at least two dimensions, and wherein the second diagram further includes edges connecting two nodes among the plurality of nodes, the edges representing the association between the two nodes.

14. The system according to claim 12, wherein, Analyzing the first and second diagrams associated with each timestamp includes: generating first and second feature vectors associated with the first and second diagrams of the specific timestamp, and comparing the angles between the first and second feature vectors.

15. The system according to claim 13, wherein Analyzing the first and second graphs associated with the identified specific timestamp includes: performing an operation using a first eigenvector associated with the first graph and a second eigenvector associated with the second graph to generate a result, where the result includes the importance of each of the plurality of nodes of the first graph in contributing to the anomaly.

16. The system according to claim 15, wherein, The operation is vector subtraction, and where the result is a vector.

17. The system according to claim 12, wherein Analyzing the first and second graphs associated with the identified specific timestamp includes: measuring the entropy of the first and second graphs.

18. The system according to claim 12, wherein, Generating the first and second graphs includes: Generating a plurality of first nodes; Iteratively generating parent nodes, where each parent node is connected to a child node by an edge; and Iteratively assigning weights to the edges connecting each parent node and each child node.

19. The system according to claim 12, wherein, The at least two dimensions describe characteristics of an entity that generates the metric values in the time series dataset.

20. The system according to claim 12, wherein, Generating the expected value based on a Bayesian analysis of the time series dataset.

21. The system according to any one of claims 12 to 20, wherein, The one or more processing circuits are further configured to determine a root cause of the anomaly based on the identified nodes, where the root cause is associated with a dimension of the identified nodes.

22. The system according to claim 21, wherein, The one or more processing circuits are further configured to determine a solution for the anomaly based on the root cause.

23. One or more computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to: Receive a time series data set representing a metric that varies over time, where, The time series data has at least two dimensions, and where the at least two dimensions describe characteristics of an entity that generates the metric values in the time series dataset; For each of two or more timestamps in the time series dataset, generate a first and a second graph, the first and second graphs representing: (i) the metric value at that timestamp, (ii) the at least two dimensions at that timestamp, and (iii) the association between the metric values at that timestamp, where the metric value of the first graph includes the metric value corresponding to that timestamp, and where the metric value of the second graph is based on an expected value generated from the received time series dataset; Analyze the first and second graphs associated with each timestamp to identify a specific timestamp including an anomaly; and Analyze the first and second graphs associated with the identified specific timestamp by generating first and second eigenvectors associated with the first and second graphs of the specific timestamp and comparing the angles between the first and second eigenvectors to identify the nodes contributing to the anomaly.

24. The one or more computer-readable storage media according to claim 23, wherein, Generating the expected value based on a Bayesian analysis of the time series dataset.

25. The one or more computer-readable storage media according to claim 23, wherein, The one or more processors are further configured to determine a root cause of the anomaly based on the identified nodes, where the root cause is associated with a dimension of the identified nodes.

26. The one or more computer-readable storage media according to claim 25, wherein, The one or more processors are further configured to determine a solution for the anomaly based on the root cause.

Images