Model-driven data pipeline

Abstract

A bound system model may be accessible, and the bound system model includes associations between model elements of an abstract system model representing the deployed system infrastructure and inventory elements contained in the data source inventory. A graphical user interface (GUI) may provide options for visually constructing a metric pipeline template associated with the bound system model. The definition of the metric pipeline template may be determined based on user input. The metric pipeline template may include multiple nodes that process inputs and generate derived metrics. Links between data sources associated with the deployed system infrastructure and the metric pipeline template may be established based on user input. Derived metrics may be generated based on metric data obtained based on the links established between the data sources and the metric pipeline template.

Description

TECHNICAL FIELD

[0001] RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 588,227 filed on October 5, 2023, entitled "MODEL RESOLUTION, MODEL-DRIVEN DASHBOARDING, AND DATA PROCESSING", and U.S. Patent Application No. 18 / 900,285 filed on September 27, 2024, entitled "MODEL-DRIVEN DATA PIPELINES", all of which are assigned to the assignee of the present application and are hereby incorporated by reference in their entireties into the present specification.

[0002] The present disclosure relates generally to modeling and management of information technology (IT) system infrastructures. BACKGROUND ART

[0003] Businesses generally rely on robust monitoring of their information technology (IT) infrastructure to proactively identify and resolve problems before they impact customers or business operations. A key aspect of such monitoring involves analyzing system logs. System logs often provide a detailed record of events, errors, warnings, and other diagnostic information across various components of the IT infrastructure. By centralizing and analyzing these logs, often using a log management platform, businesses can gain visibility into the health and performance of their systems. Alerts can be configured to notify the IT team when specific log events or patterns are detected that may indicate an ongoing problem. While automated tools can be used to perform aspects of log analysis, manual log analysis by skilled engineers remains a crucial part of troubleshooting complex IT infrastructure. Generally, engineers tasked with troubleshooting IT infrastructure evaluate logs generated by components of the IT infrastructure to identify error messages, pinpoint root causes of problems, uncover hidden correlations between events, and trace issues across interconnected components or services. [Overview of the Initiative]

[0004] One or more computer systems may be configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action when it is running. One or more computer programs (or engines) may be configured to perform a particular operation or action by including instructions that cause the device to perform an action when executed by a data processing device. One common embodiment includes a method for modeling a deployed system infrastructure of a cloud computing environment. The method also includes a step of generating an abstract system model by one or more processors, which may include one or more model elements corresponding to one or more infrastructure elements of the deployed system infrastructure, wherein each of the one or more model elements specifies one or more selectors, and for each of the one or more model elements, each selector from each of the one or more selectors of the model element includes an indication of at least one infrastructure element from the one or more infrastructure elements associated with the model element. The method also includes a step of building an inventory of the one or more infrastructure elements by one or more processors. The method also includes the step of identifying, for at least one of the one or more model elements, the at least one infrastructure element of the inventory associated with the at least one model element, as indicated by the one or more selectors for each of the at least one model element, by the one or more processors.The method also includes the step of generating a bound system model by the one or more processors, which may include one or more associations between the one or more model elements of the abstract system model and the inventory of the one or more infrastructure elements, wherein the one or more associations are determined at least in part on the identified at least one infrastructure element of the inventory associated with the at least one model element. The method also includes the step of outputting data by the one or more processors indicating the status of the deployed system infrastructure, incidents occurring within the deployed system infrastructure, or both, the data being generated using the bound system model. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0005] The implementation may include one or more of the following features: In the method, the one or more infrastructure elements may include: one or more physical elements of the deployed system infrastructure, one or more virtual elements of the deployed system infrastructure, or a combination thereof. The step of building the inventory of the one or more infrastructure elements may include: obtaining information about the one or more infrastructure elements from services hosting the deployed system infrastructure, application programming interfaces (APIs), or a combination thereof. Each of the one or more selectors of the at least one model element includes one or more aspects of ancestor elements in the indication of the at least one infrastructure element associated with the at least one model element, and the one or more aspects of ancestor elements are used to determine the one or more associations between the one or more model elements of the abstract system model and the inventory of the one or more infrastructure elements. The one or more aspects of ancestor elements may include: the labels of the ancestor elements, the properties of the ancestor elements, or a combination thereof. The step of identifying the at least one infrastructure element of the inventory associated with the at least one model element may include: (i) evaluating a first portion of the one or more selectors for each of the at least one model element that does not refer to the ancestor element; (ii) following action (i), evaluating a second portion of the one or more selectors for each of the at least one model element that does refer to the ancestor element; and (iii) repeating actions (i) and (ii) until all of the one or more selectors for each of the at least one model element have been evaluated. The step of identifying the at least one infrastructure element of the inventory associated with the at least one model element may include: using label-based matching, queries, heuristics, or a combination thereof.Each of the one or more selectors of the at least one model element includes one or more variables that replace one or more aspects of the ancestor element of the at least one model element in the indication of the at least one infrastructure element associated with the at least one model element. The at least one model element may include a stream of telemetry associated with an ancestor element, the stream of telemetry may include metrics, logs, event traces, or a combination thereof. Child elements may include relationships between abstract system model elements and / or inventory elements. The method may include a step of updating the bound system model in response to detected changes in the inventory of the one or more infrastructure elements. The method may include a step of maintaining a history indicating when the bound system model was updated. Implementations of the described technique may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0006] One or more computer systems may be configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action when it is running. One or more computer programs may be configured to perform a particular operation or action by including instructions that cause the device to perform an action when executed by a data processing device. One general embodiment includes the step of providing; the providing step includes defining at least one selector, the at least one selector which may be used by the one or more processors to bind one or more sets of infrastructure elements of the deployed system infrastructure to a first model element of the one or more model elements; and defining at least one indicator which may be used by the one or more processors to identify at least one stream of metric data about the first model element from a metric data source. The providing step also includes the step of having the one or more processors bind the one or more sets of infrastructure elements to the first model element in response to receiving user input defining the at least one selector. The providing step also includes the step of having one or more processors identify the at least one stream of metric data for the first model element in response to receiving user input defining the at least one indicator. The providing step also includes the step of having one or more processors update the GUI to include: a set of one or more infrastructure elements bound to the first model element, and a representation of the at least one indicator. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0007] An implementation may include one or more of the following features: A method wherein the GUI is further configured to enable a user to define the state of the first model element based on metric data from the at least one stream of metric data. The method may include, in response to receiving user input defining the state of the first model element: a step of the one or more processors determining that the first model element is in the state; and a step of the one or more processors updating the GUI to include a representation of the first model element being in the state. The state of the first model element is based on whether a condition is met, and the one or more processors use the metric data to determine whether the condition is met. The GUI is configured to enable a user to define a relationship between the first model element and a second model element of the one or more model elements, and the method may further include, in response to receiving user input defining the relationship, a step of the one or more processors updating the GUI to include a representation of the relationship. The relationship indicates how the state of the first model element affects the second model element, how the state of the second model element affects the first model element, or both. The method wherein the one or more model elements further include one or more child elements of the first model element, and the relationship further indicates how the state of the one or more child elements affects the second model element, how the state of the second model element affects the one or more child elements, or both. The GUI is configured to allow a user to view historical data indicating the state of the first model element at a previous point in time. The metric data may include: data indicating the root cause of an incident in the deployed system infrastructure, the cost associated with operating the deployed system infrastructure, or any combination thereof.The metric data source may include a metric data source for the set of one or more infrastructure elements, a metric data source other than the set of one or more infrastructure elements, or both. To enable the user to define the at least one selector, the GUI is configured to display an inventory of infrastructure elements, including the set of one or more infrastructure elements. The GUI is configured to provide apparent states that simulate failure conditions or incidents based on user input specifying information about a virtual system state. The GUI is configured to enable the user to view virtual states of a deployed system infrastructure by adjusting one or more states of the one or more model elements. Implementations of the techniques described may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0008] A system of one or more computers may be configured to perform a specific operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action when it is running. A system of one or more computer programs may be configured to perform a specific operation or action by including instructions that cause the device to perform an action when it is executed by a data processing device. One common embodiment includes an access step. The access step also includes a step of using the one or more processors to provide a graphical user interface (GUI), the GUI providing options for visually constructing a metric pipeline template associated with the bound system model. The access step also includes a step of using the one or more processors to determine the definition of the metric pipeline template, at least in part, based on user input via the GUI, the metric pipeline template may include a plurality of nodes that process inputs and generate derived metrics. The access step also includes a step of using the one or more processors to establish a link between the metric pipeline template and at least one data source associated with the deployed system infrastructure, based on user input via the GUI. The access step also includes a step of generating one or more derived metrics using the one or more processors, at least in part, based on the metric data obtained based on the link established between the at least one data source and the metric pipeline template. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0009] An implementation may include one or more of the following features: The GUI provides an option for defining the metric pipeline template as one or more input nodes, one or more analysis nodes, and one or more output nodes; the step of establishing the link between the at least one data source associated with the deployed system infrastructure and the metric pipeline template may include: defining the at least one data source as a source connection for the at least one input node included in the metric pipeline template; the at least one data source is an indicator defined to select a set of metrics from the one or more data sources associated with the deployed system infrastructure; the one or more analysis nodes include a metric behavior node configured to apply a metric behavior to transform a first time series into a second time series; the metric behavior includes one of the following: maximum value in a time window, minimum value in a time window, standard deviation across a time window, percentile across a time window, increase across a time window, or rate of increase across a time window; the one or more analysis nodes include a combiner node configured to apply a metric behavior to combine multiple time series into a single value. The metric operation includes converting a sum, average, or count to a rate. The link between at least one data source associated with the deployed system infrastructure and the metric pipeline template is defined for a first model element of the abstract system model. The one or more derived metrics are provided in association with the first model element of the abstract system model, and the GUI provides one or more graphs plotting the one or more derived metrics over a time window. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0010] This summary is not intended to identify any important or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate parts of the entire specification of this disclosure, any or all of the drawings, and each claim. The matters described herein, along with other features and examples, are described in more detail in the following specification, claims, and accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram of an exemplary computing environment according to several embodiments.

[0012] [Figure 2] This is a block diagram of components of an infrastructure management system according to several embodiments.

[0013] [Figure 3] This is a block diagram of an illustrative abstract system model according to several embodiments.

[0014] [Figure 4] This is a block diagram of an exemplary operation that may be performed by an infrastructure management system according to several embodiments.

[0015] [Figure 5A] The following are example model element definitions according to several embodiments. [Figure 5B] The following shows relevant JSON code according to several embodiments.

[0016] [Figure 6A] The following are example model element definitions according to several embodiments. [Figure 6B] The following shows relevant JSON code according to several embodiments.

[0017] [Figure 7]Illustrative model resolution processes related to model element definitions of FIGS. 5A to 5B in accordance with some embodiments are shown.

[0018] [Figure 8] Another illustrative model resolution process related to model element definitions of FIGS. 6A to 6B in accordance with some embodiments is shown.

[0019] [Figure 9] This is an illustrative flowchart for modeling a deployed system infrastructure in accordance with some embodiments.

[0020] [Figure 10A] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10B] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10C] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10D] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10E] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10F] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10G] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10H] An example related to defining aspects of a system model and generating a dynamic dashboard in accordance with some embodiments is shown. [Figure 10I]This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10J] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10K] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10L] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10M] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10N] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10O] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard. [Figure 10P] This document defines several embodiments of the system model and provides examples related to generating a dynamic dashboard.

[0021] [Figure 11] This is an illustrative flowchart for model-driven incident analysis according to several embodiments.

[0022] [Figure 12A] Examples related to incident analysis according to several embodiments are shown. [Figure 12B] Examples related to incident analysis according to several embodiments are shown. [Figure 12C] Examples related to incident analysis according to several embodiments are shown.

[0023] [Figure 13]This is an illustrative flowchart for generating derived metrics by applying a model-driven metric pipeline according to several embodiments.

[0024] [Figure 14A] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14B] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14C] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14D] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14E] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14F] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14G] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14H] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments. [Figure 14I] This document presents examples related to applying a model-driven metric pipeline to generate derived metrics, according to several embodiments.

[0025] [Figure 15A] The following illustrates some of the processes according to several embodiments.

[0026] [Figure 15B]The following illustrates some of the processes according to several embodiments.

[0027] [Figure 15C] The following illustrates some of the processes according to several embodiments.

[0028] [Figure 16] This is a block diagram illustrating an example computing system.

[0029] Similar reference numerals in various drawings indicate similar elements according to the implementation of a particular example. In addition, multiple instances of an element may be indicated by appending a letter or hyphen and a second number to the first number of the element. For example, multiple instances of element 110 may be indicated as 110-1, 110-2, 110-3, etc., or as 110a, 110b, 110c, etc. When such an element is referred to using only the first number, any instance of the element should be understood (for example, element 110 in the above example would refer to elements 110-1, 110-2, and 110-3, or elements 110a, 110b, and 110c). [Modes for carrying out the invention]

[0030] In modern cloud-based system infrastructure, applications are often designed using a distributed architecture where various components and processes run across different physical and cloud-based systems. This architectural approach enables greater flexibility, scalability, and resilience compared to traditional monolithic systems.

[0031] Efficiently monitoring and managing such system infrastructure is challenging due to the large number of components running various processes, yet it is necessary to maximize system uptime. Furthermore, monitoring and managing the system infrastructure should be timely to reduce downtime. However, traditional methods that rely on manual analysis of system logs to identify problems are limited and typically result in a longer mean time to insight metric, which is a measure of the amount of time required from the time an alert is generated to the identification of a problem within the deployed system infrastructure. Given the increasing cybersecurity threats to both physical and cloud-based computing systems, there is an urgent need for improved solutions to better visualize and manage deployed system infrastructure so that any incident can be proactively addressed with a shorter mean time to insight metric.

[0032] The claimed solutions described herein provide improved, computer-technology-based solutions to overcome problems that arise particularly in the domain of computer networks. For example, Figure 1 is a block diagram of an exemplary computing environment 100 according to an embodiment described herein. The computing environment 100 may enable entities (e.g., users, companies, organizations, etc.) to build and deploy system infrastructure in physical, cloud, or hybrid computing environments.

[0033] In this example, the computing environment 100 includes an infrastructure management system 102, user devices 104, a cloud resource provider 106, a cloud resource provider 108, and on-premises resources 110, all of which can communicate electronically over one or more computer networks 120 (e.g., the Internet).

[0034] The cloud resource provider 106 may be a cloud computing provider that provides functionality for provisioning cloud computing instances that run various applications or services, such as web services 106a, storage services 106b, order processing 106c, databases 106d, inventory management systems, recommendation engines, payment processing, shipping processing, and data analysis.

[0035] The cloud resource provider 108 may be different cloud computing providers that provide similar functionality for provisioning cloud computing instances to run various applications or services. The on-premises resource 110 may be one or more computing systems housed in a location outside the cloud.

[0036] Each resource (e.g., a cloud instance, a server, etc.) and any application or service running on that resource may output various types of metric data, including, in particular, CPU utilization, memory usage, disk input / output (I / O), network traffic, log messages (or application logs) generated by applications running on the resource, and log messages generated by the operating system running on the resource.

[0037] For example, the system infrastructure on which an e-commerce application runs may consist of multiple collaborating infrastructure elements. Front-end processes, such as a website or mobile app, may present products, handle user interactions, and provide a user interface to facilitate the shopping experience. These front-end components can be deployed on instances running web services 106a via a cloud resource provider 106. The back-end processes of the e-commerce application may run on multiple instances from different cloud resource providers to handle critical tasks such as order processing 106c, databases 106d, inventory management, shipping processing, payment processing, and data analysis.

[0038] Over time, the number of instances allocated to a particular application or service may change. For example, instances may be added when demand surges and removed when demand is low. Such iterative changes within a deployed system infrastructure can make it difficult to maintain a real-time model of the system infrastructure that allows for low-level insights and management of any instance within the deployed system infrastructure, including any applications and services running on those instances.

[0039] The infrastructure management system 102 may be configured to provide functions for modeling and managing the deployed system infrastructure, among other operations, to enable fine-grained insights and management of instances, applications, and services. Further details describing the infrastructure management system 102 are provided with reference to Figure 2.

[0040] Figure 2 shows a block diagram of an exemplary infrastructure management system 202 according to several embodiments. The infrastructure management system 202 may be implemented in a computing system including at least one processor, memory, and a communication interface. The computer system can run software such as system infrastructure management software, which performs any number of functions described in relation to Figure 2.

[0041] The infrastructure management system 202 includes an interface engine 204, an inventory engine 206, a model engine 208, a model explorer engine 210, an incident engine 212, and a metric pipeline engine 214.

[0042] The interface engine 204 may provide an interface (e.g., a graphical user interface) that enables the user to perform various functions described herein.

[0043] In some embodiments, the interface engine 204 may provide an interface that allows the user to specify any data source associated with the deployed system infrastructure. The data source may provide system logs from a physical server or any information that can be retrieved via an application programming interface (API). For example, the deployed system infrastructure may run on instances from a specific cloud resource provider. In this example, the user may interact with the interface to identify the cloud resource provider and provide any relevant details (e.g., credentials, APIs, etc.) necessary to retrieve information about those instances. The data source may be used to build an inventory of elements associated with the deployed system infrastructure, as described below in relation to the inventory engine 206.

[0044] In some embodiments, the interface engine 204 may provide a model editor in which the user can specify (or define) an abstract system model of the deployed system infrastructure. The abstract system model may be represented as a block diagram consisting of model elements. The model elements may represent applications, services, or logical entities associated with the deployed system infrastructure.

[0045] For example, Figure 3 shows an exemplary abstract system model 300 for a deployed system infrastructure supporting an e-commerce application. In Figure 3, the abstract system model 300 includes model elements corresponding to various logical components of the system infrastructure. For example, model element 304 corresponds to a front-end function that interacts with a user device 302. The front-end may include various cloud-based resources, such as compute instances, that provide functionality for hosting the e-commerce application. Furthermore, model element 306 corresponds to an inventory processing function that supports the system infrastructure, model element 308 corresponds to a database function that supports the system infrastructure, model element 310 corresponds to an order processing function that supports the system infrastructure, and model element 312 corresponds to a payment function that supports the system infrastructure. The interface engine 204 may provide an interface that allows the user to define selectors and indicators for each model element, as discussed herein.

[0046] The inventory engine 206 may be configured to build an inventory of the deployed system infrastructure, as shown in the example in Figure 4. The inventory 402 may identify infrastructure elements 406 associated with the deployed system infrastructure. For example, infrastructure elements may refer to physical or virtual computing systems such as cloud compute instances. The inventory engine 206 may build the inventory 402 based on any data source 404 identified by the user. For example, the inventory engine 206 may make calls via various cloud-based application programming interfaces (APIs) to identify compute instances present in the deployed system infrastructure at a given point in time. For example, the APIs may be provided by cloud resource providers (e.g., Amazon Web Services®, Azure®, Google Cloud Platform®, etc.) or deployment frameworks such as Kubernetes.

[0047] Inventory 402 may also identify a list 408 of metrics that can be retrieved from each of the infrastructure elements 406. For example, the list 408 of metrics may include any information that can be retrieved from the instance via a ListMetrics API call. For instance, the list 408 of metrics may include, among other things, information about CPU utilization, memory usage, disk I / O, network traffic, cloud metrics, web services, inventory management, shipping processing, payment processing, databases, and storage services.

[0048] In some embodiments, inventory 402 may be defined as a schema containing a set of inventory elements. For example, inventory 402 may include the following elements:

[0049] ID: Unique ID,

[0050] Source ID: Identifies the data source from which the inventory element was identified.

[0051] Type: The type of inventory element, such as an AWS EC2 instance.

[0052] Name: The human-readable name of the inventory element, and

[0053] Key: A unique key for the inventory element, appropriate for its type.

[0054] In addition, for each inventory element, the inventory engine 206 may track a set of labels for the inventory element over time in the form of inventory detail fields. Inventory detail fields allow properties such as tags or other fields to change over time. For example, inventory details may include:

[0055] Element ID: Inventory element ID,

[0056] Starting point: When an inventory element is discovered,

[0057] End point: The point at which the inventory element is removed, and

[0058] Label: A set of key / value pairs associated with an inventory element.

[0059] Generally, for a given instance at a given point in time, there exists a single inventory detail item. Furthermore, the inventory detail may be active over a certain interval [start time, end time]. In various embodiments, the inventory schema may be used to determine the state of inventory 402 at a particular point in time by looking at the start and end times in the inventory detail.

[0060] Once constructed, inventory 402 represents all potential sources of information (or telemetry) that are available for access and processing to facilitate real-time monitoring and management of the deployed system infrastructure. In various embodiments, inventory engine 206 may update inventory 402 periodically (e.g., every second, every minute, every hour, etc.) by polling data sources so that any changes in the deployed system infrastructure, such as the addition or removal of instances, can be recognized in real time or near real time.

[0061] The model engine 208 may be configured to perform various functions related to the modeling and management of the deployed system infrastructure.

[0062] In various embodiments, the model engine 208 may allow the user to define selectors provided in the abstract system model 300. For example, a model element may be associated with one or more selectors. In some embodiments, the selectors act as live queries that can be used to match the model element to an infrastructure element included in the inventory 402.

[0063] Model elements may be associated with logical entities in the deployed system architecture, as defined in the abstract system model 300, such as front-end services. Furthermore, model elements may also be associated with any individual instances assigned to logical entities. For example, model element 304 may be associated with a front-end service, as well as any cloud instances running to support the front-end service.

[0064] When defining selectors for model elements, users may define matching criteria that can be used to match the selectors to infrastructure elements in inventory 402. For example, Figure 5A shows an exemplary model element definition 502 for a service 504 called FooService. In this example, a selector 506 is defined for service 504. Selector 506 is associated with a matching criterion 508 for matching selector 506 to infrastructure elements in inventory 402. In various embodiments, the matching criteria may be expressed as label-based matches (or queries) and / or heuristics. In the example in Figure 5A, the matching criterion 508 is defined to identify any EC2 cloud computing instance in the deployed system infrastructure associated with a "service" label having the value "foo". As an example, the model element definition 502 may be expressed in JavaScript Object Notation (JSON) 510, as shown in Figure 5B.

[0065] In various embodiments, model elements may be organized hierarchically. That is, a model element may have one or more “parent” (or ancestor) model elements and / or “child” model elements. A selector may use one or more aspects of the ancestor model elements to help match one or more child model elements. For example, the matching criteria for a selector may utilize any combination of labels, properties, or characteristics associated with the ancestor model elements.

[0066] The model engine 208 may also allow users to define indicators for model elements. In some embodiments, indicators act as live queries that can be used to retrieve information associated with one or more infrastructure elements, such as metrics, data, or other arbitrary telemetry that may be available from a specified data source. Such information may indicate the health of a given infrastructure element, as well as the health of the services supported by that infrastructure element.

[0067] For example, Figure 6A shows an exemplary model element definition 602 for a service 604 called FooService. In this example, as described above with reference to Figure 5A, the selector 606 is defined to match infrastructure elements in inventory 402 based on matching criteria 608.

[0068] Model element definition 602 also includes indicator 612 that specifies a CloudWatch metric 614 called "CPUUtilization" which can be used to measure the health of service 604. The CPUUtilization metric can provide various information about the utilization of CPU resources allocated to an instance. To measure the overall health of service 604, indicator 612 needs to be configured to retrieve the CPUUtilization metric from all instances associated with service 604. Therefore, indicator 612 is associated with another selector 616. Selector 616 may be associated with a criterion 618 which can be used to match selector 616 to elements in inventory 402. In this example, matching criterion 618 is defined to identify a metric contained in inventory 402 that identifies the metric as type "CloudWatch Metric" and has a property with the label "CPUUtilization". Matching criterion 618 also specifies the expression "InstanceId=${parent.InstanceId}". This expression leverages the hierarchical context of the model element to allow retrieval of the corresponding CPUUtilization metric from each instance running service 604. In other words, since service 604 is bound to the underlying instance using selector 606, the CPUUtilization metric for each instance can be easily identified using the ${parent.instanceID} expression. As an example, model element definition 602 can be represented in JavaScript Object Notation (JSON) 630, as shown in Figure 6B. In general, indicators can be defined for any metric, log, event trace, or other stream of telemetry.

[0069] Once model elements are defined, the model engine 208 may be configured to perform model resolution. In various embodiments, model resolution 416 includes combining inventory elements from inventory 402 with an abstract system model 412. For example, as shown in Figure 4, inventory elements may be combined with the abstract system model 412 based on model definitions 414 (e.g., model element definitions, selectors, indicators, etc.). The result of this combination is a bound system model 418 that maps model elements defined for the abstract system model 412 to the corresponding inventory elements from inventory 402. Model resolution may be performed hierarchically such that the result of one inventory match may influence the result of another inventory match.

[0070] In various embodiments, the bound system model 418 may consist of three types of bindings: element bindings, data (or metric) bindings, and relational bindings. For example, the bound system model 418 may consist of a set of element bindings that map inventory elements to model elements. For example, element bindings may be associated with the following properties:

[0071] ID: Unique ID of the element binding,

[0072] ModelElementID: ID of the abstract model element,

[0073] InventoryElementID: The ID of the inventory element in the mapping.

[0074] InventoryDetailID: The detail ID of the inventory element at the time of model resolution, and

[0075] ParentID: If applicable, this property contains the ID of the element binding for the parent model element.

[0076] The bound system model 418 may include a set of data (or metric) bindings that map metrics or other data from the inventory 402 to element bindings. For example, a data binding may refer to a metric for an indicator. In some embodiments, a data binding may also refer to an associated element binding to which the data binding is associated. For example, a data binding may be associated to the following properties:

[0077] ID: Unique ID for data binding,

[0078] IndicatorID: Indicator ID,

[0079] ElementBindingID: The ID of the parent element binding.

[0080] MetricID: The ID of the inventory element for the metric in the mapping.

[0081] MetricDetailID: The detail ID of the inventory element at the time of model resolution, and

[0082] MetricSummarization: A type of metric summary that may be needed when searching for metrics.

[0083] Furthermore, the bound system model 418 may include a set of relation bindings that identify relationships between element bindings. Relation bindings may be used for model relationships that have selectors. For example, relation bindings may be associated with the following properties:

[0084] ID: Unique ID of the relationship binding,

[0085] RelationshipID: The ID of the relationship object that created this relationship binding.

[0086] SourceModelElementID: Model element ID for the source of the relationship.

[0087] DestElementBindingID: The destination of the relationship, which is the element binding. Therefore, the destination is a specific bound inventory element within the destination model element.

[0088] RelationshipID: The ID of the relationship object that created this relationship binding.

[0089] PropagationTo: Represents any condition to propagate from source to destination along this binding, and

[0090] PropagationFrom: Represents any condition that propagates from the destination to the source along this binding.

[0091] As an example, Figure 7 shows an exemplary model resolution that can be performed by the model engine 208 with reference to the model element definition 502. In this example, the model engine 208 found a set of inventory elements 704, 706 that match the selector 506. For example, the model engine 208 may determine the matching inventory elements 704, 706 based on a matching criterion associated with the selector 506. In this example, the matching criterion is defined to identify inventory elements associated with the type "EC2 instance" and the label "service=foo," which matches the properties 704a, 706a associated with inventory elements 704, 706. Inventory elements 704, 706 may be represented as element bindings 702 in the bound system model 418. Each of the inventory elements 704, 706 is an actual instance bound to the model element definition 502.

[0092] As another example, Figure 8 shows another illustrative model resolution that can be performed by the model engine 208 with reference to model element definition 602. In this example, the model engine 208 found a set of inventory elements 804, 814 that match selector 606. For example, the model engine 208 may determine the matching inventory elements 804, 814 based on matching criteria associated with selector 606. In this example, the matching criteria are defined to identify inventory elements associated with type "EC2 instance" and label "service=foo", which matches properties 804a, 814a associated with inventory elements 804, 814. Inventory elements 804, 814 may be represented as element bindings 802 in the bound system model 418. Each of the inventory elements 804, 814 is an actual instance bound to model element definition 602.

[0093] In addition, in the example in Figure 8, the model engine 208 performs an additional model resolution to resolve selector 616 associated with indicator 612 for the metric "CPUUtilization," as described above. Selector 616 is associated with matching criterion 618 for inventory elements associated with type "CloudWatch Metric" and label "name=CPUUtilization." Matching criterion 618 also includes the expression "InstanceId=${parent.InstanceId}" used to specify the InstanceId value during model resolution. The model engine 208 may be configured to resolve selectors and indicators in the model based on their hierarchical context. For example, when resolving data bindings, the model engine 208 may determine the respective context for each matched inventory element. In the example in Figure 8, the model engine 208 may perform model resolution twice, once for each EC2 instance matched based on selector 606. When performing model resolution for a given matched instance, the model engine 208 may access labels from the parent of the matched instance and apply those labels using variable substitution. The syntax ${parent.InstanceId} indicates that the value of the label InstanceId is used from the parent context.

[0094] Therefore, Figure 8 shows that the model engine 208 performs a model resolution process for each indicator and inventory element binding. The model resolution performed by the model engine 208 applies indicator selectors using the context from the inventory element binding. The result is a data binding that binds the correct CPUUtilization metric to each model binding. This same hierarchical model resolution process used for binding can also be used for component model elements. In various embodiments, a component model element is a model element logically nested inside a parent element, and the model resolution of the component element can use the parent's context as a variable for model resolution.

[0095] In some embodiments, when performing model resolution, the model engine 208 may resolve the model based on the hierarchical context by, for example, first evaluating selectors that do not have references to ancestor elements, then evaluating selectors that do have references to ancestor elements when it is possible to evaluate them, and repeating these operations as necessary to evaluate all selectors in the model.

[0096] The model engine 208 may output a bound system model 418 upon completion of model resolution. The bound system model 418 may consist of a set of model bindings that map elements in inventory 402 to elements in abstract system model 412. The bound system model 418 corresponds to a point-in-time snapshot of the system infrastructure, so that the state of inventory 402 at the time the bound system model 418 was created is used in all subsequent uses of the bound system model 418.

[0097] The bound system model 418 output by the model engine 208 may include associations between model elements of the abstract system model 412 and infrastructure elements contained in the inventory 402, including countless element bindings, data bindings, and relational bindings. In various embodiments, the model engine 208 regenerates the bound system model 418 whenever a change is detected in the inventory 402. For example, if the inventory 402 changes due to the addition or deletion of instances, the model engine 208 performs model resolution again so that the bound system model 418 is the most recent model of the abstract system model 412 with the corresponding telemetry.

[0098] The model engine 208 may be configured to evaluate various model relationships, including single static elements, highly available services, and single dynamic elements.

[0099] For example, a single static element may not dynamically map to anything. While typically not used to describe physical objects, single static elements can be useful when describing things that are treated as black boxes from the user's perspective. For single static elements, Model Engine 208 can still access telemetry indicating their health, and such insights can be applied to potentially identify root cause failures in many situations. Some examples of single static elements include AWS Availability Zones (AZs), Regions, or Services, and Cockroach Cloud.

[0100] Model elements representing highly available services can be particularly useful for monitoring infrastructure health. Here, there is a natural top-level whole element representing the overall service status, while selectors match to the individual elements that collectively provide that service. In this case, the model engine 208 can access telemetry related to the health of each individual element, as well as telemetry related to the overall health of the service. The whole telemetry can be generated based on some combination of telemetry from individual service instances, or it can be obtained from some other natural source of such data, such as metrics from a load balancer or service mesh. For such highly available services, the bound system model can model how the whole service may degrade due to failures in individual components, and how failures in the whole service may affect other systems that may depend on it. Some examples of highly available services may include customer-created microservices running as a Kubernetes deployment, where individual Kubernetes pods are bound to the service. Another example is a database deployed on an EC2 instance.

[0101] A single dynamic element may have components found within an inventory that have only a single instance. For example, a component may be a logical object that does not correspond to a physical infrastructure, or it may not be a highly available service. For this type of object, the model engine 208 may have access to telemetry related to the health of a particular object, which is the same as the health of the element as a whole. Here, the model engine 208 may utilize the resolution context to match against the correct telemetry, so the inventors may not be able to simply use top-level indicators alone. This type of element is qualitatively different from others because a single matched element is treated as essentially the top-level element. Some examples of single dynamic elements include AWS Elastic Kubernetes Clusters and AWS Elastic Load Balancers.

[0102] The Model Explorer Engine 210 of the Infrastructure Management System 202 may provide a Model Explorer tool (or interface) for exploring bound system models. The Model Explorer tool may enable users to access dashboard interfaces, incident alerts and summaries, and metrics, among other types of information, as discussed herein. For example, the Model Explorer Engine 210 may generate a visual representation of a bound system model corresponding to a deployed system infrastructure. For example, the visual representation may be provided in a dynamic dashboard. The dashboard may be configured to model the system infrastructure based on attributes defined for inventory elements, such as relationships, states, and conditions. Based on these attributes, the Model Explorer Engine 210 may generate a dynamic dashboard representing the system infrastructure, which may be provided to the user in a graphical user interface (GUI). Unlike conventional dashboards, which typically plot some value over time, the dashboard generated by the Model Explorer Engine 210 may represent the dynamic nature of various aspects of the system infrastructure. Further applications of the Model Explorer Engine 210 are described below with reference to the illustrative flowchart 900 and Figures 10A-10P in Figure 9.

[0103] The incident engine 212 may be configured to generate a model-driven incident summary and a condition graph whenever an incident is detected in the deployed system infrastructure. The incident summary may be presented in an interface provided by the interface engine 204. Further details regarding the incident engine 212 are described below with reference to the illustrative flowchart 1100 and Figures 12A-12C in Figure 11.

[0104] The metric pipeline engine 214 may provide functionality for generating modeling and model-driven derived metrics. In various embodiments, derived metrics may be metrics generated by performing some kind of query or calculation on other metrics or data. Derived metrics can have many use cases. As an example, a user may simply want to see some calculated value when accessing and exploring their bound system model. In this case, as the user drills down into a model element, the user may see a graph of calculated derived metrics associated with the model element or a particular bound inventory element. As another example, when performing incident analysis, a user may need to transform or combine metrics to obtain values ​​that can be mapped to statuses using thresholds. This may include things like transforming counts into rates, calculating averages, or calculating sums across some data. Further details regarding the metric pipeline engine 214 are described below with reference to the illustrative flowchart 1300 and Figures 14A–14I in Figure 13.

[0105] Figure 9 shows an illustrative flowchart 900 for modeling a deployed system infrastructure and generating a dashboard. In step 902, the infrastructure management system 202 provides interfaces (e.g., graphical user interface, application programming interface) that enable the user to define data sources associated with the deployed system infrastructure. For example, Figure 10A shows an illustrative interface 1002 that enables the user to add a data source. Interface 1002 enables the user to specify the name of the data source, the source type (e.g., cloud computing provider), the source configuration list of application programming interfaces (APIs) to use (e.g., EC2, CloudWatch, Elastic Kubernetes Service, Elastic Load Balancer, etc.), and the credentials (e.g., access key) to access the data source. An example in Figure 10B shows another illustrative interface 1004 that is checked when a data source is added. Interface 1004 may also provide a synchronization status for each of the applicable APIs, indicating whether the infrastructure management system 202 was able to connect to the respective endpoint for each of the APIs based on the provided credentials.

[0106] In step 904, the infrastructure management system 202 may build an inventory based on the added data sources as described above. Figure 10C shows an example interface 1006 that may be provided. Interface 1006 may, among other things, show a list of data sources and inventory details such as CanonicalHostedZoneId, DNSName, IpAddressType, KubernetesIngressName, KubernetesIngressNamespace, LoadBalancer, LoadBalancerMetricLabel, LoadBalancerName, Scheme, Tags, Type, Vpcid, etc. The infrastructure management system 202 may provide another interface 1008 that shows a list of inventory elements, as shown in the example in Figure 10D. Interface 1008 may also identify a list of infrastructure elements and a list of metrics that may be derived from those infrastructure elements, as determined based on the data sources.

[0107] In step 906, the infrastructure management system 202 may provide an interface that allows the user to define an abstract system model of the deployed system infrastructure, as described above. For example, the user may interact with interface 1010 to define an abstract system model. In Figure 10E, the user creates a first element that references an entity named "Bob" 1012 and a second element that references an entity named "Alice" 1014, both associated with the abstract system model. Both the first element 1012 and the second element 1014 may represent different logical entities associated with the deployed system infrastructure.

[0108] The user may also specify attributes for each element. For example, Figure 10F shows an exemplary interface 1020 that may be provided by the infrastructure management system 202. Interface 1020 may be provided when the user selects an option to define attributes associated with the first element 1012. In this example, interface 1020 may provide options 1022 for defining one or more selectors, indicators, indicator metrics, conditions, relationships, propagations, and states, among other attributes.

[0109] In some embodiments, the state may measure something related to the element (e.g., health, cost, etc.). Such a state may be used to determine how the state of a given element may affect other elements, such as parent elements affecting child elements, based on the hierarchy associated with the elements in the abstract system model.

[0110] In some embodiments, the relationship between two elements may indicate that the first element influences the second element, the second element influences the first element, or both the first and second elements influence each other. Such relationships model how actions related to one element may affect another. These relationships can be used in conjunction with rules. For example, a user may define a rule to attribute costs to the number of instances required for entity Bob and the number of instances required for entity Alice. This rule may specify the relationship between Bob and Alice, the ratio of the number of instances required for Bob to the number of instances required for Alice, and a function for determining the costs associated with the instances. In this example, the infrastructure management system 202 may process the rule to report the costs associated with running Bob and Alice at any given point in time. Thus, if it becomes necessary to reduce computational costs, the risks of reducing the number of instances of Bob and Alice, and the effort involved in each of their associated entities can be determined. For example, Figure 10G shows an illustrative interface 1026 in which the relationship between Bob and Alice is specified. In the example in Figure 10G, the given relationship shows that Alice influences Bob, which is visually represented by arrow 1028 in the abstract system model.

[0111] Figure 10H shows an exemplary interface 1030 that may be provided when a user selects an option to define conditions for an element. In some embodiments, the conditions may be bound to a data source. In this example, interface 1030 provides a list of conditions associated with a database associated with the deployed system infrastructure. Some exemplary conditions that may be applied include "Database Instance Degraded," "Database Instance Quota Exceeded," and "Database Service Degraded."

[0112] Figure 10I shows an illustrative interface 1036 showing a database element 1038 associated with the condition “Database Instance Degradation” 1040. A condition can be defined by a range value, a type value, a severity value, and a description value. In this example, condition 1040 is defined by the range value “Instance”, the type value “Impact”, and the severity value “Normal”. In this example, database element 1038 has the condition “Database Instance Degradation” with an impact of normal severity at the per-instance level. Essentially, this condition is standalone and does not need to act as a trigger (e.g., a “happiness” state). In contrast, Figure 10J shows an illustrative interface 1042 showing the condition “Database Instance Limit Exceeded” 1044 associated with trigger 1046. Trigger 1046 can match against a data source, which may differ from the selector data source of database element 1038. In this case, database element 1038 has a Kubernetes data source, but trigger 1046 is configured to use data from a Prometheus data source to determine whether a condition is met. The ability to configure triggers in this way offers many advantages. For example, triggers can be defined so that the existence of an element (e.g., "an instance exists") is obtained from one data source, and the element's metric (e.g., CPUUtilization) is obtained from another data source.

[0113] Figure 10K shows an illustrative interface 1052 that illustrates propagation 1054 associated with database element 1038. In this example, database element 1038 propagates to the application service. For example, Figure 10L shows an illustrative interface 1060 that illustrates database element 1038 propagating to application service element 1062 when database element 1038 is in a certain state, as shown in the illustrative interface 1068 of Figure 10M. In this example, based on the defined propagation, if the database is degraded, the latency of the application service instance will be high.

[0114] In step 908, the infrastructure management system 202 may perform a model solution as described herein.

[0115] In step 910, the infrastructure management system 202 may generate a dynamic dashboard that allows the user to visualize an abstract model representing the deployed system infrastructure and related information. For example, Figure 10N shows an illustrative interface 1070 showing a dynamic dashboard 1072 representing the system infrastructure. The dynamic dashboard 1072 shows corresponding elements 1074 representing various components of the system infrastructure, including the "AWS ELB Demo App" element, the "App Service (Target Group)" element, the "App Service" element, and the "Database" element. Each element may provide various related information, such as conditions, propagation (outgoing, internal, incoming), the number of associated instances, and any related conditions and indicators. In this example, the infrastructure management system 202 generates a dynamic dashboard, but other applications are intended. For example, once model resolution is complete, the infrastructure management system 202 may be applied for incident analysis or to generate derived metrics, as described herein.

[0116] Figure 10O illustrates the dynamic functionality of the dynamic dashboard 1072. A user interacting with the dynamic dashboard 1072 may select element 1076 corresponding to “App Services”. In response, the dynamic dashboard 1072 may update to display the inventory element bound to element 1076. In this example, the bound inventory element includes instance 1078 in addition to various related information. The “App Services” element 1076 is a top-level model element representing the overall health of the highly available app services. Instance 1078 is matched to the app services element 1076 based on the selector and is updated in real time.

[0117] In some embodiments, when a user drills down into a model element, the user can see graphs of any metrics and any derived metrics associated with the model element or a specific bound inventory element. For example, Figure 10P shows further dynamic functionality of the dynamic dashboard 1072. In this example, the dynamic dashboard 1072 provides access to graphs about various metrics associated with instance 1078. In this example, the graph plots request latency over a certain time window. The time window can be adjusted to visualize changes in the metrics over a desired period. Many variations are possible.

[0118] Figure 11 shows an illustrative flowchart 1100 for model-driven incident analysis. In step 1102, the infrastructure management system 202 provides a graphical user interface 1202 that enables access to the Model Explorer tool, as shown in the example in Figure 12A. Interface 1202 allows the user to visualize the elements of a bound system model 1204 that represent the deployed system infrastructure. The bound system model 1204 includes a data service leader element 1206 responsible for inventory and data ingestion, as well as directing model and data pipeline processing. The bound system model 1204 also includes a periodic task worker element 1208 responsible for performing tasks to synchronize inventory, pull metrics, and perform model resolution. Furthermore, the bound system model 1204 includes a Clickhouse element 1210. In this example, Clickhouse is a third-party cloud service with limited telemetry. The Clickhouse service can be observed indirectly by monitoring errors from other services provided by the bound system model 1204.

[0119] In step 1104, the infrastructure management system 202 may analyze metric data from various data sources to identify any incidents that may occur. In various embodiments, incidents may be triggered on an element-by-element basis, based on the respective metrics associated with those elements. For example, an incident may be triggered when one or more metrics meet a threshold over a certain time window. In some embodiments, incidents may be linked to model root causes.

[0120] In the example in Figure 12A, an incident may occur due to a configuration error that causes queries in Clickhouse to fail only from a specific Kubernetes pod, resulting in ingestion and data processing failures. In this example, the infrastructure management system 202 may detect the incident based on a root cause model that identifies relationships between model elements, including causal and impact edges between model elements.

[0121] The root cause can be modeled in interface 1222, as shown in the example in Figure 12B. In Figure 12B, the data service leader element 1206 is associated with impact condition 1224, which is modeled as “Ingest Broken” (critical impact). Furthermore, the periodic task worker element 1208 can be associated with the following modeled impacts: “Failure pulling metrics” (overall) 1226, “Failure pulling metrics” (instance) 1228, and “Clickhouse queries failing” (instance) 1230. The periodic task worker element 1208 can be associated with the following indicators: “Rate of errors pulling metrics” (for each instance), “Sum of rate of errors pulling metrics” (overall), and “Rate of errors querying clickhouse” (for each instance). Furthermore, the periodic task worker element 1208 can be associated with the following propagation: "Metric pull failure" (overall) propagates to "Ingestion corruption", and "Metric pull failure" (instance) propagates to "Metric pull failure" (overall). The Clickhouse element 1210 can be associated with the following modeled root cause: "Clickhouse query failure" (overall) 1232. The Clickhouse element 1210 can be associated with the following indicator: "Sum of rate of errors querying Clickhouse" (aggregated across all pods using Clickhouse). Furthermore, the Clickhouse element 1210 can be associated with the following propagation: "Clickhouse query failure" (overall) propagates to "Metric pull failure" (instance).

[0122] In some embodiments, when an incident occurs in the deployed system infrastructure shown in Figure 12A, the infrastructure management system 202 may analyze information or outputs (e.g., metrics) generated by the elements with respect to any modeled root cause, for example, based on associated selectors and indicators. In various embodiments, the incident may be modeled based on other criteria, such as the costs associated with operating the deployed system infrastructure.

[0123] In step 1106, based on the incident analysis, the infrastructure management system 202 may provide incident details in one or more interfaces. For example, in some embodiments, the infrastructure management system 202 may generate and provide an executive summary 1232, as shown in the example in Figure 12C. The executive summary 1232 may provide a hierarchical overview 1234 of the elements involved in the incident. In this example, the executive summary 1232 shows that the ingestion corruption condition 1236 is triggered by “critical impact”. Furthermore, the root cause 1238 shows a Clickhouse query failure on a periodic task worker instance. Furthermore, the root cause 1240 is also shown because the Clickhouse aggregate metric is triggered by a generally high error rate for all queries to Clickhouse. Many variations are possible. In some embodiments, a timeline selector 1242 may be used to access information (e.g., root cause, impact, etc.) at a specific point in time or over a period of time.

[0124] In some embodiments, the infrastructure management system 202 may provide a condition graph in addition to the executive summary 1232. The condition graph may be represented similarly to the example in Figure 12B and may show a chain of causal relationships about propagation conditions related to an incident. For example, the condition graph may show that a "Clickhouse query failure" is triggered for the entire Clickhouse, which propagates to a "Clickhouse query failure" being triggered for a periodic task worker instance, which propagates to a "metric pull failure" for the periodic task worker instance, which propagates to a "metric pull failure" for the entire periodic task worker, which ultimately propagates to an "ingestion corruption" for the entire data service leader.

[0125] In some embodiments, the infrastructure management system 202 may be configured to provide apparent states that simulate failure conditions or incidents based on user input specifying information about the virtual state of the deployed system infrastructure. For example, the virtual state may be created via a GUI that includes options for adjusting the state of model elements associated with the deployed system infrastructure.

[0126] Figure 13 shows an illustrative flowchart 1300 for applying a model-driven metrics pipeline to generate derived metrics. In step 1302, the infrastructure management system 202 provides a graphical user interface 1402 that allows access to a bound system model 1404 representing the deployed system infrastructure, as shown in the example in Figure 14A. The interface 1402 allows the user to visualize the elements of the bound system model 1404.

[0127] The infrastructure management system 202 may provide an option 1406 for adding a new metric pipeline that can generate derived metrics by applying any number of operations and combinations to outputs generated by any number of elements. In various embodiments, a derived metric may be a metric generated by performing some kind of query or calculation on other metrics or data. A derived metric in a bound system model may be the result of processing one or more other metrics into a new metric. For example, an indicator in the system model may be defined to select a set of metrics, and the user can then specify a metric pipeline that should be performed on the metrics selected by the indicator to calculate the final derived metric.

[0128] In step 1304, the infrastructure management system 202 may be used to define a metric pipeline template that can be bound to various dynamic data sources and used in the context of a hierarchical model. Notably, the underlying data sources may be associated with different types of databases, each having its own query language. However, according to the embodiments herein, the user does not need to understand how to implement such a query language in order to build a metric pipeline template. Rather, the user simply identifies the data source from which the metric data should be retrieved, for example, using associated selectors or indicators.

[0129] In some embodiments, a metric pipeline may consist of one or more analysis nodes. An analysis node may consist of (1) a set of inputs having IDs and names, where the inputs represent a set of time-series streams; (2) a combiner, where the combiner takes in multiple time-series streams and combines them into a single result through some operation. One example of a combiner would be to take the sum of all input metrics at each point in time. Some combiners can take in any number of streams in their inputs, while others require exactly one stream to make sense; and (3) a set of input actions, where the input actions take in the inputs, apply a list of metric actions to each input metric, and then map the output to the inputs of the combiner. A metric action is an operation that takes in a single time series and transforms it into another time series. Furthermore, a metric pipeline may also consist of: (1) a set of inputs, similar to a node; (2) an output, the ID of an analysis node, which is used as the overall output of the pipeline; (3) a set of input connections connecting the inputs to the inputs of an analysis node; and (4) a set of node connections connecting the outputs of an analysis node to the inputs of other analysis nodes.

[0130] For example, Figure 14B shows an illustrative interface 1410 for defining a metric pipeline 1412. Interface 1410 allows a user to graphically model a pipeline for processing metrics from various data sources. In this example, the metric pipeline 1412 includes multiple analysis nodes, including a pipeline input node 1412a, a metric action node 1412b, a combiner node 1412c, and a pipeline output node 1412d. Each node (e.g., action, combiner, etc.) may be associated with metadata (or properties) containing information about its name, description, and the inputs and parameters required for its definition.

[0131] For example, Figure 14C shows an illustrated interface 1410 updated to provide details about metric operation 1414 associated with metric operation node 1412b. In this example, metric operation 1414 is defined such that its input 1414a is received from a "Pipeline input" node 1412a (i.e., source connection). Importantly, input 1414a is not associated with a static resource, but rather is dynamically resolved based on information provided by any element or indicator linked to the "Pipeline input" node 1412a. Thus, metric pipeline 1412 can be a template that can be reused in different scenarios and for different input sources.

[0132] In Figure 14C, metric operation 1414 is also defined so that metric operation 1414b (e.g., "maximum value within the time window") is performed on input data received from the "pipeline input" node 1412a over a window time frame (e.g., a 5-minute window). Furthermore, according to the definition of metric pipeline 1412, any result determined by metric operation 1414b is provided to one or more output connections 1414c. In this example, the result is provided as input to combiner node 1412c, which is configured to apply a summation combiner to the input.

[0133] Figure 14D shows List 1416 of examples of predefined metric behaviors that may be associated with metric behavior node 1412b, including, in particular, “sum over a time window,” “rolling average over a time window,” and “standard deviation over a time window.” List 1416 is provided as an example, and in various embodiments, users may be able to custom define their own metric behaviors that are applied by a given metric node.

[0134] The infrastructure management system 202 may provide options for building more complex metric pipelines by piping outputs from existing nodes to new nodes. For example, in Figure 14E, interface 1410 provides option 1420, which allows outputs from combiner node 1412c to be connected to new or existing nodes. In another example, Figure 14F shows option 1422 for connecting a "pipeline input" node 1412a to a new metric operation node or a new combiner node. In this example, a new node is created to calculate the "rate" metric operation, as shown in Figure 14G. In this example, outputs from the "pipeline input" node 1412a are provided to the "max" metric operation node 1412b and the new "rate" metric operation node 1424.

[0135] Figure 14H shows another exemplary metric pipeline 1432 that can be created and applied to a bound system model. In Figure 14H, the metric pipeline 1432 includes an input node 1432a that outputs to a first metric action node 1432b (e.g., "min" action) and a second metric action node 1432c (e.g., "max" action). The outputs from the metric action nodes 1432b and 1432c are provided to a combiner node 1432d. The combiner node 1432d is configured to perform the standard deviation of all input time series at each point in time, provided by the metric action nodes 1432b and 1432c. The combiner node 1432d is further configured to provide its output to a pipeline output node 1432e. Again, many variations are possible.

[0136] In some embodiments, the infrastructure management system 202 may utilize metric groups. A metric group allows for the selection of a set of metric time-series streams and can be used in a metric pipeline. A metric group may consist of: (1) a set of selectors to match for the metrics to include; (2) a summarization type indicating how the matched metrics should be summarized when reducing the cardinality of the data; (3) an interpolation type indicating a policy on how to interpolate between two values ​​in the stream; (4) an extrapolation type indicating how to extrapolate data past the end of the stream (unless the data window has expired); and (5) a data window after which the metric should be considered missing. If the data stream is missing, it is assumed to take a specified missing value.

[0137] In some embodiments, the infrastructure management system 202 may allow indicators to be composed of derived metrics. That is, instead of using selectors directly, indicators may instead constitute a set of metric groups. If the user specifies only a single metric group having a single metric, this can be used directly as the indicator value. The user may also include an analysis pipeline to apply to the metric group. If this is done, a pipeline connection connecting the metric group to the input of the analysis pipeline may be included. The user can then visualize all the matched primary metrics, as well as the derived metrics which are the results of the calculated pipeline, in the interface provided by the infrastructure management system 202.

[0138] In some embodiments, the infrastructure management system 202 may implement two selector types: an inventory selector and an indicator selector. In some embodiments, the inventory selector matches against an inventory using filtering by data source and inventory label. In some embodiments, the indicator selector selects a metric obtained as a result of a target indicator, rather than directly matching against an inventory. If the target indicator is associated with a pipeline, any metric generated by the pipeline may be used.

[0139] In step 1306, the infrastructure management system 202 may be used to link a metric pipeline template to a source connection. For example, Figure 14I shows an illustrative interface 1442 that illustrates a bound system model 1444 of a deployed system infrastructure. In this example, the regions of interface 1442 provide access to various properties associated with the “app service” element 1446, such as details, selectors, conditions, relationships, propagations, and indicators. In this example, an indicator 1448 that provides “request latency” metric data may be linked to the pipeline. For example, a user may select one or more pipelines from a menu 1450 that displays available pipeline templates. Once linked, metric data from indicator 1448 may be provided as input to the linked pipeline, which processes the input data and outputs one or more derived metrics.

[0140] In step 1308, the infrastructure management system 202 may generate derived metrics according to a linked metric pipeline template. In some embodiments, derived metrics may be provided as outputs to other connections, such as other indicators. Thus, derived metrics can be reflected in any interface (e.g., a dashboard) and used to better manage the deployed system infrastructure, as described herein.

[0141] In some embodiments, the infrastructure management system 202 may provide functionality for chaining pipelines. For example, the infrastructure management system 202 may allow users to chain pipelines within a model and between different models. For example, a practical application of chained pipelines involves aggregating instance-scoped indicators based on multiple metrics into a single global-scoped indicator. To achieve this, instance-scoped indicators can be created first, and selectors that match all metrics of interest may be defined. If multiple metrics are expected to match, a combined pipeline may be created on the indicator (or a predefined pipeline may be referenced). A global-scoped indicator may then be created on the same element (or any other element). The global-scoped indicator may use an indicator type selector that references the instance-scoped indicator. For the link from instance to global, a combined pipeline is typically required. Another practical application of chained pipelines is that indicators can be selected for indicators on other model elements. Here, indicators for elements where relationships exist may be used as input. In this way, the infrastructure management system 202 can traverse along model graph edges to find the relevant metrics.

[0142] Figure 15A illustrates the process in several embodiments. In step 1502, an abstract system model is generated which includes one or more model elements corresponding to one or more infrastructure elements of the deployed system infrastructure. Each model element of the one or more model elements specifies one or more selectors, and for each model element of the one or more model elements, each selector of each of the one or more selectors of the model element includes an indication of at least one infrastructure element from the one or more infrastructure elements associated with the model element. In step 1504, an inventory of one or more infrastructure elements is constructed. In step 1506, for at least one model element of the one or more model elements, at least one infrastructure element of the inventory associated with the at least one model element is identified as indicated by each of the one or more selectors of the at least one model element. In step 1508, a bound system model is generated which includes one or more associations between one or more model elements of the abstract system model and the inventory of one or more infrastructure elements. One or more associations are determined at least partially based on at least one identified infrastructure element of the inventory associated with at least one model element. In step 1510, data showing the status of the deployed system infrastructure, incidents occurring within the deployed system infrastructure, or both are output as generated using the bound system model.

[0143] Figure 15B illustrates an illustrative process according to several embodiments. In step 1522, a graphical user interface (GUI) is provided which includes a graphical representation of one or more model elements of the system model of the deployed system infrastructure. The GUI is configured to allow the user to: define at least one selector, the at least one selector which includes rules for binding a set of one or more infrastructure elements of the deployed system infrastructure to a first model element of one or more model elements; and define at least one indicator for identifying at least one stream of metric data for the first model element from a metric data source. In step 1524, in response to receiving user input defining at least one selector, the set of one or more infrastructure elements is bound to the first model element. In step 1526, in response to receiving user input defining at least one indicator, at least one stream of metric data is identified for the first model element. In step 1528, the GUI is updated to include a representation of the set of one or more infrastructure elements bound to the first model element, and at least one indicator.

[0144] Figure 15C illustrates an illustrative process according to several embodiments. In step 1532, a bound system model is accessed, which includes one or more associations between one or more model elements of an abstract system model representing the deployed system infrastructure and one or more inventory elements included in the inventory. The inventory may be loaded based on one or more data sources associated with the deployed system infrastructure. In step 1534, a graphical representation of the abstract system model is provided in a graphical user interface (GUI). The GUI may provide options for defining (or visually constructing) a metric pipeline template associated with the bound system model. In step 1536, the definition of the metric pipeline template is determined at least in part based on user input via the GUI, and the metric pipeline template includes a plurality of nodes that process inputs and generate derived metrics. In step 1538, a link between at least one data source associated with the deployed system infrastructure and the metric pipeline template is established based on user input via the GUI. In step 1540, one or more derived metrics are generated at least in part based on inputs received based on the link established between at least one data source and the metric pipeline template.

[0145] Figure 16 is a block diagram illustrating a digital device in one example. The digital device may read instructions from a machine-readable medium and execute those instructions by a processor to perform machine processing tasks as described herein, such as the engine operations discussed above. Specifically, Figure 16 shows a schematic representation of a machine in an exemplary form of a computer system 1600 in which instructions 1624 (e.g., software) can be executed to cause the machine to perform any one or more of the methodologies described herein. In alternative embodiments, the machine may operate as a standalone device or be connected to other machines (e.g., networked) via, for example, the Internet. In a networked deployment, the machine may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0146] A machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA®), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions 1624 (whether sequential or not) that specify actions to be taken by such machine. Furthermore, although only a single machine is shown, the term “machine” shall also be interpreted to include any set of machines that individually or collectively execute instructions 1624 to perform any one or more of the methodologies discussed herein.

[0147] The illustrated computer system 1600 includes a processor 1602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination thereof), main memory 1604, and static memory 1606, which are configured to communicate with each other via a bus 1608. The computer system 1600 may further include a graphics display unit 1610 (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system 1600 may also include an alphanumeric input device 1612 (e.g., a keyboard), a cursor control device 1614 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing device), a data store 1616, a signal generation device 1618 (e.g., a speaker), and a network interface device 1620, which is also configured to communicate via the bus 1608.

[0148] The data store 1616 includes a machine-readable medium 1622 in which instructions 1624 (e.g., software) embodying any one or more of the methodologies or functions described herein are stored. The instructions 1624 (e.g., software) may reside entirely or at least partially in the main memory 1604 or in the processor 1602 (e.g., in the processor's cache memory) during their execution by the computer system 1600, and the main memory 1604 and the processor 1602 also constitute the machine-readable medium. The instructions 1624 (e.g., software) may be transmitted or received over the network 1626 via the network interface 1620.

[0149] Although the machine-readable medium 1622 is shown to be a single medium in the exemplary embodiment, the term “machine-readable medium” should be interpreted to include a single or multiple mediums (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instruction 1624). The term “machine-readable medium” should also be interpreted to include any medium capable of storing instructions for machine execution (e.g., instruction 1624) and causing a machine to execute any one or more of the methodologies disclosed herein. The term “machine-readable medium” includes, but should not be limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0150] In this description, the term “engine” refers to computational logic for providing a specified function. An engine can be implemented in hardware, firmware, and / or software. When an engine described herein is implemented as software, it can be implemented as a standalone program, but can also be implemented by other means, for example, as part of a larger program, as any number of separate programs, or as one or more statically or dynamically linked libraries. It will be understood that a named engine described herein represents one embodiment, and other embodiments may include other engines. In addition, other embodiments may not require an engine described herein, and / or may distribute the described functions among engines in different ways. In addition, functions that belong to more than one engine can be incorporated into a single engine. In one embodiment in which an engine is implemented by software, they are stored on a computer-readable persistent storage device (e.g., a hard disk), loaded into memory, and executed by one or more processors as described above in relation to Figure 16. Alternatively, hardware or software engines may be stored elsewhere in the computing system.

[0151] As referenced herein, a computer or computing system includes hardware elements used in the operations described herein, whether or not there is a specific reference to such elements in Figure 16, for example, one or more processors, high-speed memory, hard disk storage and backup, network interfaces and protocols, input devices for data entry, and output devices for displaying, printing, or otherwise presenting data. Numerous variations are possible from the system architectures specified herein. The entities of such systems and their respective functions may be combined or redistributed.

Claims

1. A step of using one or more processors to access a bound system model which includes one or more associations between one or more model elements of an abstract system model representing a deployed system infrastructure and one or more inventory elements included in the inventory, wherein the inventory is loaded based on one or more data sources associated with the deployed system infrastructure; The step of providing a graphical user interface (GUI) using one or more of the aforementioned processors, wherein the GUI provides options for visually constructing a metric pipeline template associated with the bound system model; A step of using one or more of the aforementioned processors to determine the definition of the metric pipeline template, at least in part, based on user input via the GUI, wherein the metric pipeline template includes a plurality of nodes that process inputs and generate derived metrics; A step of using one or more of the aforementioned processors to establish a link between at least one data source associated with the deployed system infrastructure and the metric pipeline template based on user input via the GUI; and A step of generating one or more derived metrics using one or more processors, at least partially based on metric data obtained based on the link established between the at least one data source and the metric pipeline template. Methods that include...

2. The method according to claim 1, wherein the GUI provides options for defining the metric pipeline template as one or more input nodes, one or more analysis nodes, and one or more output nodes.

3. The step of establishing the link between the at least one data source associated with the deployed system infrastructure and the metric pipeline template is: The step of defining the at least one data source as a source connection for the at least one input node included in the metric pipeline template. The method according to claim 2, including the method described in claim 2.

4. The method according to claim 3, wherein the at least one data source is an indicator defined to select a set of metrics from the one or more data sources associated with the deployed system infrastructure.

5. The method according to claim 2, wherein the one or more analysis nodes include a metric action node configured to apply a metric action that transforms a first time series into a second time series.

6. The method according to claim 5, wherein the metric operation includes one of the following: maximum value in a time window, minimum value in a time window, standard deviation across a time window, percentile across a time window, increase across a time window, or rate of increase across a time window.

7. The method according to claim 2, wherein the one or more analysis nodes include a combiner node configured to apply a metric action to combine multiple time series into a single value.

8. The method according to claim 7, wherein the metric operation includes: converting a sum, an average, or a count to a rate.

9. The method according to claim 1, wherein the link between at least one data source associated with the deployed system infrastructure and the metric pipeline template is defined for a first model element of the abstract system model.

10. The method according to claim 9, further comprising providing the one or more derived metrics in a GUI that visually displays the one or more model elements of the abstract system model representing the deployed system infrastructure, wherein the one or more derived metrics are provided in association with the first model element of the abstract system model, and the GUI provides one or more graphs plotting the one or more derived metrics over a time window.

11. A system comprising at least one processor, and the system: Accessing a bound system model that includes one or more associations between one or more model elements of an abstract system model representing a deployed system infrastructure and one or more inventory elements included in the inventory, wherein the inventory is populated based on one or more data sources associated with the deployed system infrastructure; To provide a graphical user interface (GUI), wherein the GUI provides options for visually constructing metric pipeline templates associated with the bound system model; The definition of the metric pipeline template is determined at least in part based on user input via the GUI, wherein the metric pipeline template includes a plurality of nodes that process inputs and generate derived metrics; Based on user input via the GUI, establish a link between at least one data source associated with the deployed system infrastructure and the metric pipeline template; and To generate one or more derived metrics based at least partially on metric data obtained based on the link established between the at least one data source and the metric pipeline template. A system including memory that stores instructions for executing a function.

12. The system according to claim 11, wherein the GUI provides options for defining the metric pipeline template as one or more input nodes, one or more analysis nodes, and one or more output nodes.

13. Establishing the link between the at least one data source associated with the deployed system infrastructure and the metric pipeline template is: Define the at least one data source as a source connection for the at least one input node included in the metric pipeline template. The system according to claim 12, including the above.

14. The system according to claim 13, wherein the at least one data source is an indicator defined to select a set of metrics from the one or more data sources associated with the deployed system infrastructure.

15. The system according to claim 12, wherein the one or more analysis nodes include a metric action node configured to apply a metric action that transforms a first time series into a second time series.

16. A non-temporary computer-readable storage medium containing instructions, wherein the instructions are executed by one or more hardware processors: A procedure for accessing a bound system model, which includes one or more associations between one or more model elements of an abstract system model representing a deployed system infrastructure and one or more inventory elements included in the inventory, wherein the inventory is loaded based on one or more data sources associated with the deployed system infrastructure; A procedure for providing a graphical user interface (GUI), wherein the GUI provides options for visually constructing a metric pipeline template associated with the bound system model; A procedure for determining the definition of the metric pipeline template based at least in part on user input via the GUI, wherein the metric pipeline template includes a plurality of nodes that process inputs and generate derived metrics; A procedure for establishing a link between at least one data source associated with the deployed system infrastructure and the metric pipeline template based on user input via the GUI; and A procedure for generating one or more derived metrics based at least partially on metric data obtained based on the link established between the at least one data source and the metric pipeline template. A non-temporary, computer-readable storage medium that triggers the execution of a set of operations, including [specific actions].

17. The non-temporary computer-readable storage medium according to claim 16, wherein the GUI provides an option for defining the metric pipeline template as one or more input nodes, one or more analysis nodes, and one or more output nodes.

18. The procedure for establishing the link between the at least one data source associated with the deployed system infrastructure and the metric pipeline template is as follows: A procedure for defining the at least one data source as a source connection for the at least one input node included in the metric pipeline template. A non-temporary computer-readable storage medium according to claim 17, including the following:

19. The non-temporary computer-readable storage medium according to claim 18, wherein the at least one data source is an indicator defined to select a set of metrics from the one or more data sources associated with the deployed system infrastructure.

20. The non-temporary computer-readable storage medium according to claim 17, wherein the one or more analysis nodes include a metric operation node configured to apply a metric operation that converts a first time series to a second time series.

Images