Machine learning for metrology acquisition

Abstract

The present disclosure relates to machine learning for metrics collection. A performance monitoring system includes a metrics collector configured to receive telemetry data including metrics related to a network of computing devices via a metrics exporter. A metrics time series database stores the metrics. An alert rule evaluator service is configured to evaluate rules using the stored metrics. The performance monitoring system can include a machine learning module and be configured to determine optimized metrics collection sampling intervals and rule evaluation intervals and automatically determine recommended alert rules.

Description

Machine learning for metric collection

[0001] Related applications

[0002] This application claims priority to U.S. Application No. 17 / 810,178, filed June 30, 2022, and Indian Provisional Patent Application No. 202241022566, filed April 16, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to computer networks, and more specifically to improving the acquisition and evaluation of telemetry data in computer networks. Background Technology

[0004] In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing and / or storage capacity to run a variety of applications. For example, a data center includes facilities hosting applications and servers for its subscribers (i.e., the data center's customers). A data center may, for example, host all infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage servers and application servers (computer nodes) are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers offer infrastructure that is widely distributed throughout the world, allowing subscriber support equipment to reside in various physical hosting facilities.

[0005] The connectivity between the server and the switching infrastructure occurs at the hardware module (called a Network Interface Card (NIC)). Traditional NICs include Application-Specific Integrated Circuits (ASICs) to perform packet forwarding, and these ASICs include some basic Layer 2 / Layer 3 (L2 / L3) functions. In a traditional NIC, packet processing, supervision, and other higher-level functions (known as the "data path") are performed by the host CPU, i.e., the CPU of the server including the NIC. As a result, CPU resources in the server are shared by the applications running on that server and also by the data path processing. For example, in a 4-core x86 server, one of these cores can be reserved for the data path, leaving three cores (or 75% of the CPU) for applications and the host operating system.

[0006] Performance monitoring systems monitor data center performance. Telemetry data includes various metrics about network components / nodes, which can be transmitted to the metric collectors of a typical centralized remote performance monitoring system for evaluation according to various rules. This allows users (such as network administrators) to measure and evaluate many different performance metrics about the network, such as CPU utilization, memory utilization, total network devices and applications, link and node utilization, network congestion, etc.

[0007] Some NIC vendors have begun incorporating additional processing units into the NIC itself to offload some of the data path processing from the host CPU to the NIC. The processing unit within the NIC can be, for example, a multi-core ARM processor, with some hardware acceleration provided by a data processing unit (DPU), a field-programmable gate array (FPGA), and / or an ASIC. NICs that include this type of enhanced data path processing capability are often referred to as smart NICs and can provide additional processing capacity to assist in telemetry data transmission. Summary of the Invention

[0008] Generally, techniques for using machine learning to compute infrastructure performance monitoring systems are described to provide improved metric collection sampling intervals, improved rule evaluation intervals, and / or ongoing rule recommendations, thereby conserving network resources and providing more important data for a better understanding of the network.

[0009] The performance monitoring system includes a data acquisition unit for collecting telemetry data (metrics) related to computing nodes in the network, and an alarm rule evaluator service for analyzing the telemetry data based on alarm rules that determine whether an alarm should be generated based on the telemetry data. Telemetry data is collected at predetermined sampling intervals, and rules are evaluated at predetermined rule evaluation intervals.

[0010] The performance monitoring system described in this disclosure reduces the storage space required to store metrics and avoids unnecessary collection of metric data that is irrelevant to a given use case and / or context, rather than collecting telemetry data at a fixed static rate (where the collector uses a static sampling interval to collect metrics). That is, collectors using static sampling intervals to collect metrics may have disadvantages related to the amount of storage space required to store metrics, especially in large data centers with many compute nodes and many alert rules. Other disadvantages may include the collection of metrics that may be irrelevant to the user, the need for significant computational power to search for potentially irrelevant metrics, and the possibility of redundant metrics when the metric value does not change much over time. Therefore, when collecting more relevant metrics, the metric collector may use the same sampling interval to avoid collecting many less useful metrics. As metrics are collected in scaled network environments, the problems listed above may become more pronounced and problematic. In an example, a performance monitoring system implementing various aspects of the techniques described in this disclosure can leverage machine learning to determine the relevance of one or more metric attributes representing the usefulness of a metric to the user, in order to predict metric weights and an optimized collection sampling rate corresponding to that metric. In this respect, the performance monitoring system described in this disclosure can efficiently identify and optimize the acquisition sampling rate of metrics, rather than using a fixed static time interval to acquire metrics.

[0011] Furthermore, the periodic rule evaluation process involves computationally intensive tasks such as querying massive amounts of telemetry data, aggregating telemetry data, and comparing the aggregated data with multiple thresholds. When configuring a large number of rules in a computationally limited environment, a rule evaluation process using a static evaluation interval may struggle to properly evaluate the rules. Additionally, processing too many rules in an overloaded system can waste computational resources. To avoid these scaling issues, administrators typically limit the number of rules by configuring or increasing the rule evaluation interval (i.e., reducing the rule evaluation rate).

[0012] By employing optimized rule evaluation intervals (i.e., using different rule evaluation intervals) that can evaluate rules at different frequencies, such as based on their past evaluation success or failure (hit or miss) rates, the performance monitoring system described in this disclosure can avoid the aforementioned problems of static evaluation intervals. When a rule is missed for a long time, a solution based on a fixed evaluation rate will waste resources because the probability of a hit evaluation in the near future is low.

[0013] In contrast, performance monitoring systems offer several advantages. They implement an intelligent, machine learning-based process for rule evaluation, where alert rules are periodically evaluated to optimize evaluation intervals; these alert rules may change over time due to variations in network conditions. Evaluation intervals can be assigned based on the defined weights of the rules. These weights indicate rule priority and may be inversely proportional to the desired rule evaluation interval. In other words, higher rule weights correspond to shorter evaluation intervals, and lower rule weights to longer evaluation intervals. Rule weights can be predicted using machine learning models and past rule evaluation data.

[0014] The performance monitoring system optimizes the sampling interval for metric acquisition and the rule evaluation interval for evaluating rules, enabling the compute nodes of the monitored network and the performance monitoring system itself to operate more efficiently based on reduced computational resource consumption (such as processing cycles, memory, and memory bus bandwidth) and associated power demand.

[0015] In one example, this disclosure describes a method comprising: acquiring telemetry data including network-related metrics of a computing device by a performance monitoring system, wherein, for each metric, at each of a plurality of time points according to a first acquisition sampling interval, a metric value associated with a corresponding metric name of the metric is acquired; evaluating an alarm rule by the performance monitoring system using the acquired telemetry data, wherein evaluating a first metric of the first rule includes: comparing the metric value associated with a corresponding metric name of the first metric with a corresponding threshold of the first rule at the plurality of evaluation time points to generate a metric attribute; determining a predicted metric weight for the first metric by the performance monitoring system based on the metric attribute; and determining a second acquisition sampling interval for the first metric by the performance monitoring system based on the predicted metric weight.

[0016] In another example, this disclosure describes a performance monitoring system including: a memory; and one or more memories communicating with the memory; the one or more memories are configured to execute a collector and an alarm rule evaluator service, wherein the collector is configured to receive telemetry data via a metric derivative, the telemetry data including network-related metrics of a computing device, wherein for each metric, a metric value associated with a corresponding metric name is configured to be collected at each of a plurality of time points based on a corresponding first predetermined collection sampling interval; wherein the alarm rule evaluator service is configured to evaluate rules using the collected telemetry data; wherein, in order to evaluate a first metric of a first rule, the alarm rule evaluator service uses the metric value associated with the corresponding metric name of the first metric to compare the corresponding metric value with a corresponding threshold of the first rule at each of the plurality of rule evaluation time points, generates an alarm regarding whether a comparison condition of the first rule is met, and stores the generated alarm in the memory; wherein the collector is configured to access the generated alarm in the memory to calculate a metric attribute of the first metric, use the metric attribute of the first metric to determine a predicted metric weight, and determine a second collection sampling interval for the first metric based on the determined metric weight.

[0017] Details of one or more embodiments of this disclosure will be set forth in the following drawings and description. Other features, objects, and advantages of the invention will become apparent from the description, drawings, and claims. Attached Figure Description

[0018] Figure 1 is a block diagram illustrating an exemplary network system with a data center, in which examples of the technologies described herein can be implemented.

[0019] Figure 2 is a block diagram illustrating an exemplary computing device according to the technology described herein, which uses a network interface card with a separate processing unit to perform services managed by an edge service controller.

[0020] Figure 3 is a design diagram of a data center with servers according to the technology described in this disclosure, wherein each server includes a network interface card with a separate processing unit controlled by an edge service controller.

[0021] Figure 4 is a block diagram illustrating an exemplary performance monitoring service with telemetry services according to the technology described in this disclosure, the telemetry services including telemetry acquisition services in a network and / or a data center.

[0022] Figure 5 illustrates a performance monitoring system according to the technology described in this disclosure, which communicates with a metric exporter for acquiring telemetry data and includes an alarm rule evaluator service for evaluating rules using the telemetry data.

[0023] Figure 6 shows an example of a performance monitoring system with an intelligent data acquisition device according to the technology described in this disclosure.

[0024] Figure 7 is an exemplary flowchart of determining the metric weights and corresponding new sampling intervals for metric acquisition according to the technology described in this disclosure.

[0025] Figure 8 is an exemplary sequence diagram for determining a new sampling interval for metric acquisition according to the technology described in this disclosure.

[0026] Figure 9 is an example of a performance monitoring system for evaluating intervals according to the adjustment rules of the technology described in this disclosure.

[0027] Figure 10 is an exemplary sequence diagram of the performance monitoring system of Figure 9 according to the technology described in this disclosure, which provides additional details about the interactions between the various components.

[0028] Figure 11 is an exemplary flowchart illustrating the operation of an alarm rule evaluator service according to the technology described herein.

[0029] Figure 12 is an example of a performance monitoring system for recommendation rules based on the technology described in this disclosure.

[0030] Figure 13 is an exemplary sequence diagram of the performance monitoring system of Figure 12 according to the technology described in this disclosure, which provides additional details about the interaction between the various components.

[0031] Figure 14 is an exemplary flowchart illustrating the operation of the performance monitoring system of Figure 12 according to the technology described in this disclosure.

[0032] Similar reference characters denote similar elements throughout the specification and drawings. Detailed Implementation

[0033] Figure 1 is a block diagram illustrating an exemplary network system 8 with computing infrastructure, in which examples of the techniques described herein can be implemented. Generally, data center 10 provides an operating environment for applications and services to one or more customer sites 11 (shown as "customer 11"), which have one or more customer networks coupled to the data center via a service provider network 7.

[0034] Data center 10 can, for example, host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. Service provider network 7 is coupled to public network 4, which can represent one or more networks managed by other providers, and thus can form part of a large public network infrastructure (e.g., the Internet). Public network 4 can represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider operating mobile service provider network 7, an enterprise IP network, or some combination thereof.

[0035] Although customer site 11 and public network 4 are primarily shown and described as edge networks of service provider network 7, in some examples, one or more of customer site 11 and public network 4 may be tenant networks within data center 10 or another data center. For example, data center 10 may host multiple tenants (customers), each of which is associated with one or more virtual private networks (VPNs), and each of these tenants (customers) may be located within customer site 11.

[0036] Service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public network 4. Service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. Service provider network 7 may implement Multiprotocol Label Switching (MPLS) forwarding and, in such instances, may be referred to as an MPLS network or MPLS backbone network. In some instances, service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, served by one or more service providers.

[0037] In some examples, data center 10 may represent one of many geographically distributed network data centers. As illustrated in the example of Figure 1, data center 10 may be a facility that provides network services to customers. The customers of the service provider may be a collection of entities, such as enterprises and governments, or individuals. For example, the network data center may host network services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although shown as a separate edge network of service provider network 7, elements of data center 10, such as one or more physical network functions (PNFs) or virtualized network functions (VNFs), may be included within the core of service provider network 7.

[0038] In this example, data center 10 includes storage and / or compute servers interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, wherein servers 12A-12X (hereinafter referred to as "server 12") are shown coupled to top-of-rack switches 16A-16N (hereinafter referred to as "TOR switches 16"). Server 12 may also be referred to herein as a "host" or "host device". Data center 10 may include a number of additional servers coupled to other TOR switches 16 of data center 10. Each host device in such a data center may run one or more virtual machines, PODs, or other expandable virtual execution elements, which may be referred to as workloads. Clients in the data center can typically access these workloads and are able to install applications and perform other operations using these workloads. Workloads running on different host devices but accessible to a specific client are organized into virtual networks. Each client typically has at least one virtual network. These virtual networks are also referred to as overlay networks.

[0039] In some cases, clients in a data center may experience network problems such as increased latency, packet loss, low network traffic, or slow workload processing. Deploying workloads in a large, multi-tenant data center can complicate the resolution of these issues. Telemetry data, such as that provided by telemetry services and analyzed by performance monitoring systems, can be used to help resolve problems in the data center.

[0040] Edge service controller 28 may include a performance monitoring system (shown in more detail in Figures 5, 6, 9, and 12) having a collector for acquiring telemetry data and an alarm rule evaluator that analyzes the telemetry data according to alarm rules, which are based on whether an alarm should be generated based on the telemetry data, as will be further explained below. The performance monitoring system may also include a telemetry service, such as shown in Figure 4, which may include a metric collector and allow users to create alarm rules for network monitoring. The performance monitoring system may include one or more machine learning components and may be configured to provide adaptive sampling intervals for acquiring telemetry data, adaptive rule evaluation intervals, and / or recommended alarm rules that provide a better understanding of the network.

[0041] In the example shown, servers 12A and 12X are directly coupled to TOR switch 16, while servers 12B, 12D, and 12C are not directly coupled to the TOR switch. Servers 12B, 12D, and 12C can reach TOR switch 16 and IP fabric 20 via servers 12A or 12X. The switching fabric 14 in the example shown includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N, which are coupled to the distribution layer (here, "rack switch 18") of rack-mounted (or "spine" or "core") switches 18A-18M. Although not shown, data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices (such as cellular phones or personal digital assistants), wireless access points, bridges, cable modems, application accelerators, or other network devices.

[0042] In this example, TOR switch 16 and rack switch 18 can provide redundant (multi-homed) connectivity to IP fabric 20 and service provider network 7 to server 12 in some cases. Rack switch 18 aggregates traffic and provides connectivity between TOR switches 16. TOR switch 16 can be a network device providing Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching capabilities. Each of TOR switch 16 and rack switch 18 can include one or more processors and memory and be capable of executing one or more software processes. Rack switch 18 is coupled to IP fabric 20, which can perform Layer 3 routing to route network traffic between data center 10 and customer site 11 by service provider network 7. The switching fabric of data center 10 is merely an example. Other switching fabrics may have more or fewer switching layers, for example.

[0043] The terms "packet flow," "traffic flow," or simply "flow" refer to a group of packets originating from a specific source device or endpoint and destined for a specific destination device or endpoint. A single flow of packets can be identified by a 5-tuple: for example, <source network address, destination network address, source port, destination port, protocol>. This 5-tuple typically identifies the packet flow corresponding to a received packet. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet refers to either <source network address, destination network address> or <source network address, source port> for that packet. The source port refers to a transport layer (e.g., TCP / UDP) port. A "port" might refer to the physical network interface of the NIC.

[0044] Each of the servers 12 can be a computer node, application server, storage server, or other type of server. For example, each of the servers 12 can represent a computing device, such as an x86 processor-based server, configured to operate according to the techniques described herein. Server 12 can provide Network Functions Virtualization Infrastructure (NFVI) for an NFV architecture.

[0045] Server 12 can host endpoints for one or more virtual networks, which operate on the physical network represented here by IP structure 20 and switching structure 14. Although a data center-based switching network is primarily described, other physical networks, such as service provider network 7, may also be underlying one or more virtual networks. Endpoints may include, for example, virtual machines, containerized applications, or applications that run natively on an operating system or bare metal.

[0046] Each of servers 12 includes at least one network interface card (NIC) among NICs 13A-13X (collectively referred to as "NIC 13"), each of NICs 13A-13X including at least one port, the at least one port exchanging packets via one or more communication links coupled to the NIC port. For example, server 12A includes NIC 13A.

[0047] In some examples, each of the NICs 13 provides one or more virtual hardware components for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of the physical NIC 13 (“physical function”). For example, in Single Root I / O Virtualization (SR-IOV) described in the Peripheral Component Interface (PCI) Special Interest Group (SIG) SR-IOV specification, the PCIe physical function of a network interface card (or “network adapter”) is virtualized to present one or more virtual network interface cards as “virtual functions” used by their respective endpoints running on server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and the virtual function is an example of a virtual hardware component. As another example, one or more servers 12 can implement, for example, the available paravirtualization framework Virtio for the Linux operating system, which provides emulated NIC functions as a virtual hardware component. For example, one or more servers 12 can implement Open vSwitch to perform distributed virtual multi-tier switching between one or more virtual NICs (vNICs) for hosted virtual machines, where such vNICs can also represent a virtual hardware component. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function and can provide SR-IOV with direct process user-space access based on the Data Plane Development Kit (DPDK).

[0048] In some examples, including the one shown in Figure 1, one or more of the NICs 13 may include multiple ports. The NICs 13 may be connected to each other via the ports and communication links of the NICs 13 to form a NIC structure 23 with a NIC structure topology. The NIC structure 23 is a collection of NICs 13 connected to at least one of the NICs 13 and coupled to each other via communication links.

[0049] Each of the NICs 13 includes a processing unit 25 to offload all aspects of the data path. The processing unit in the NIC can be, for example, a multi-core ARM processor with hardware acceleration provided by a data processing unit (DPU), a field-programmable gate array (FPGA), and / or an ASIC. The NIC 13 may optionally be referred to as a Smart NIC or a Genius NIC.

[0050] According to various aspects of the technology described in this disclosure, the edge service platform utilizes the processing unit 25 of NIC 13 to enhance the processing and networking capabilities of the switching fabric 14 and / or the server 12 including NIC 13.

[0051] In addition, the edge service controller 28 can manage the API-driven deployment of services 233 on NIC 13; the addition, deletion, and replacement of NIC 13 within the edge service platform; the monitoring of services 233 and other resources on NIC 13; and the management of connectivity between the various services 233 running on NIC 13. Furthermore, the edge service controller 28 may include a performance monitoring system 500 and a telemetry service 440 (shown in Figure 3), which can be used to collect metrics from DPU 25 using drag-and-drop or push-based queries. The performance monitoring system 500 can take various forms, such as the performance monitoring systems 600, 900, and 1200 shown in Figures 6, 9, and 12.

[0052] Edge service controller 28 may transmit information describing services, the topology of NIC architecture 23, or other information about the edge service platform available on NIC 13 to the orchestration system (not shown) of network controller 24. Exemplary orchestration systems include OpenStack, VMware vCenter, or Microsoft System Center. Exemplary network controller 24 includes controllers for Juniper Networks or Tungsten Fabric's Contrail system. Network controller 24 may be a network architecture manager. Additional information regarding controller 24 cooperating with data center 10 or other software-defined networks is found in International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled "Determining Physical Paths for Virtual Network Packet Flows," and U.S. Patent Application No. 14 / 226,509, filed March 26, 2014, entitled "Tunneled Packets for Virtual Networks," each of which is incorporated herein by reference as if fully disclosed herein.

[0053] In some examples, the edge service controller 28 can be programmed to provide telemetry data upon request for the processing unit 25. The edge service controller 28 also performs performance monitoring functions, including evaluating metrics, assessing the required telemetry data according to alarm rules, and using machine learning to determine optimized telemetry data acquisition rates, rule evaluation rates, and providing alarm rule recommendations.

[0054] Figure 2 is a block diagram illustrating an exemplary server 12 according to the technology described herein. The exemplary server 12 uses a network interface card (NIC) with a separate processing unit to perform services managed by an edge service platform. The server 12 of Figure 2 may represent a real or virtual server and may represent an exemplary instance of any of the servers 12A-12X of Figure 1. In this example, server 12 includes a bus 242 coupling to hardware components of server 12, such as an SR-IOV-enabled network interface card (NIC) 13, a storage disk 246, and a microprocessor 210. In some cases, the front-side bus may couple the microprocessor 210 and the storage device 244. In some examples, bus 242 may couple the storage device 244, the microprocessor 210, and the NIC 13. Bus 242 may represent a Peripheral Component Interface (PCI) Faster (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 242 control DMA transfers between components coupled to bus 242.

[0055] Microprocessor 210 may include one or more processors, each of which includes an independent execution unit (“processing core”) to execute instructions conforming to an instruction set architecture. The execution unit may be implemented as a separate integrated circuit (IC) or may be integrated into one or more multi-core processors (or “many-core” processors), with each multi-core processor (or “many-core” processor) implemented using ICs (i.e., chip multiprocessors).

[0056] Disk 246 represents a computer-readable storage medium, which includes volatile and / or non-volatile media, removable and / or non-removable media, and communication media, implemented in methods and techniques for storing information (such as processor-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices, or any other medium that can be used to store desired information and can be accessed by microprocessor 210.

[0057] Main memory 244 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM) or static RAM (SRAM), flash memory, or any other form of removable storage medium that can be used to carry or store desired program code and program data in the form of instructions or data structures and is accessible by a computer. Main memory 144 provides a physical address space consisting of addressable memory locations.

[0058] Network interface card (NIC) 13 includes one or more interfaces 232 configured to exchange packets using links of the underlying physical network. Interface 232 may include a port interface card with one or more network ports. NIC 13 also includes on-card memory 227, for example, to store packet data. Direct memory access transfers between NIC 13 and other devices coupled to bus 242 may be read from / written to memory 227.

[0059] The memory 244, NIC 13, storage disk 246, and microprocessor 210 provide an operating environment for a software stack that can execute the hypervisor 214 and one or more virtual machines 228 managed by the hypervisor 214.

[0060] Generally, virtual machines provide a virtualization / guest operating system for executing applications in an isolated virtual environment. Because virtual machines are virtualized from the physical hardware of the host server, the executed applications are isolated from the hardware of the host and other virtual machines.

[0061] An alternative to virtual machines is virtualized containers, such as those provided by the open-source DOCKER container application. Similar to virtual machines, each container is virtualized and remains isolated from the host and other containers. However, unlike virtual machines, each container can omit a single operating system and only provide an application suite and proprietary application libraries. Containers are executed by the host as isolated user-space instances and can share the operating system and common libraries with other containers running on the host. Therefore, containers may require less processing power, storage, and network resources compared to virtual machines. As used in this article, containers may also be referred to as virtualization engines, virtual private server silos, or jails. In some instances, the techniques described in this article pertain to containers and virtual machines or other virtualization components.

[0062] Although the virtual network endpoints in Figure 2 are shown and described relative to virtual machines and other operating environments, containers (e.g., DOCKER containers) can execute virtual network endpoints. An operating system kernel (not shown in Figure 2) can execute in kernel space 243 and may include, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel, which are available from Microsoft.

[0063] Server 12 executes hypervisor 214 to manage virtual machines 228. Exemplary hypervisors include kernel-based virtual machines (KVM) for Linux kernels, Xen, ESXi (available from VMware), Windows Hyper-V (available from Microsoft), and other open-source proprietary hypervisors. Hypervisor 214 may represent a virtual machine manager (VMM).

[0064] Virtual machine 228 can host one or more applications, such as Virtual Network Functions (VNF) instances. In some examples, virtual machine 228 can host one or more VNF instances, where each of the VNF instances is configured to apply network functions to packets.

[0065] Hypervisor 214 includes physical driver 225 to utilize the physical functions provided by network interface card 13. In some cases, network interface card 13 may also implement SR-IOV, enabling the sharing of physical network functions (I / O) among virtual machines 228. Each port of NIC 13 may be associated with a different physical function. Shared virtual devices (also known as virtual functions) provide dedicated resources, allowing each of the virtual machines 228 (and its corresponding guest operating system) to access the dedicated resources of NIC 13, which thus appears as a dedicated NIC for the virtual machines 228. Virtual function 217 may be a lightweight PCIe function that shares physical resources with physical functions and other virtual functions. According to the SR-IOV standard, NIC 13 can have thousands of available virtual functions; however, for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0066] Virtual machine 228 includes a corresponding virtual NIC 229 directly presented in the guest operating system of virtual machine 228, thereby enabling direct transfer of virtual functions allocated to the virtual machine between NIC 13 and virtual machine 228 via bus 242. This reduces the overhead involved in software-based, VIRTIO, and / or vSwitch implementations of the hypervisor 214, where the hypervisor 214 storage address space in memory 244 stores packet data, and the copying of packet data from NIC 13 to the hypervisor 214 storage address space and from the hypervisor 214 storage address space to the virtual machine 228 storage address space consumes microprocessor 210 cycles.

[0067] NIC 13 may also include a hardware-based Ethernet bridge or embedded switch 234. The Ethernet bridge 234 can perform Layer 2 forwarding between the virtual and physical functions of NIC 13. Therefore, in some cases, the bridge 234 provides hardware acceleration for packet forwarding between virtual machines 228 via bus 242 and between the hypervisor 214, which accesses physical functions via physical driver 225, and any of the virtual machines 228. The embedded switch 234 may be physically decoupled from the processing unit 25.

[0068] Server 12 may be coupled to a physical network switching infrastructure including an overlay network that extends the network infrastructure from physical switches to software or "virtual" routers, including virtual router 220, coupled to the physical servers in the switching infrastructure. The virtual router may be a process, thread, or component thereof executed by a physical server (e.g., server 12 of Figure 1), which dynamically creates and manages one or more virtual networks that can be used for transport between virtual network endpoints. In one example, the virtual router uses an overlay network to implement each virtual network, which provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server executing the endpoint. Each virtual network may use its own addressing and security scheme and may be considered orthogonal to the physical network and its addressing scheme. Various techniques may be used to transport packets within and across virtual networks via the physical network. At least some of the functions of the virtual router may be implemented as one of services 233.

[0069] In the exemplary computing device / server 12 of FIG2, the virtual router 220 executes within the management program 214 which uses physical functions for I / O, but the virtual router 220 may execute within one of the management program, the host operating system, the host application, the virtual machine 228 and / or the NIC 13 processing unit 25.

[0070] Generally, each virtual machine 228 can be assigned a virtual address for use in a corresponding virtual network, where each virtual network can be associated with a different virtual subnet provided by the virtual router 220. The virtual machine 228 can assign its own virtual Layer 3 (L3) IP address, for example, for sending and receiving communications, but may not know the IP address of the server 12 running on it. Thus, the "virtual address" is an address specific to the application, different from the logical address specific to the underlying physical computer system, such as the server 12.

[0071] In one implementation, server 12 includes a Virtual Network (VN) agent (not shown) that controls virtual network overlap for server 12 and adjusts the routing of data packets within server 12. Generally, the VN agent communicates with a virtual network controller for multiple virtual networks, which generates commands to control packet routing. The VN agent can act as a proxy for control plane messages between virtual machine 228 and a virtual network controller (such as controller 24 or 28). For example, a virtual machine can request to send a message using its virtual address via the VN agent, and the VN agent can then send a message and, in response to a received message for the virtual address of the virtual machine, request to initiate a first message. In some cases, virtual machine 228 can trigger program or function calls presented by the application programming interface of the VN agent, and the VN agent can also handle message encapsulation, including addressing.

[0072] In one example, network packets, such as Layer 3 (L3) IP packets or Layer 2 (L2) Ethernet packets generated or consumed by instances of applications executed by virtual machines 228 within a virtual network domain, can be encapsulated within another packet (e.g., another IP or Ethernet packet) transmitted over the physical network. Packets transmitted within the virtual network may be referred to herein as “internal packets,” while physical network packets may be referred to herein as “external packets” or “tunneled packets.” Encapsulation and / or decapsulation of virtual network packets within physical network packets can be performed by virtual router 220. This functionality is referred to herein as tunneling and can be used to create one or more overlay networks. Besides IP-in-IP, other exemplary tunneling protocols that can be used include IP over Generic Routing Encapsulation (GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE (MPLSoGRE), or MPLS over User Datagram Protocol (UDP) (MPLSoUDP), etc.

[0073] As described above, the virtual network controller can provide a logical centralized controller for facilitating the operation of one or more virtual networks. The virtual network controller can, for example, maintain a routing information base, such as one or more routing tables, storing routing information for the physical network and one or more overlay networks. The virtual router 220 of the hypervisor 214 implements network forwarding tables (NFTs) 222A-222N for N virtual networks; the virtual router 220 operates as a tunnel endpoint for the network forwarding tables (NFTs) 222A-222N. Generally, each NFT 222 stores forwarding information for the corresponding virtual network and identifies where data packets will be forwarded and whether packets will be encapsulated in a tunneling protocol, such as with a tunnel header that may include one or more headers for different layers of the virtual network protocol stack. Each of the NFTs 222 can be an NFT for a different routing instance (not shown) implemented by the virtual router 220.

[0074] According to the technology described in this disclosure, an edge service platform includes, for example, an edge service controller 28 that utilizes a processing unit 25 of NIC 13 to enhance the processing and networking capabilities of server 12. The processing unit 25 includes processing circuitry 231 to execute services programmed by the edge service controller 28. The processing circuitry 231 can represent any combination of processing cores, ASICs, FPGAs, or other integrated circuits and programmable hardware. In an example, the processing circuitry can include a system-on-a-chip (SoC) having, for example, one or more cores, a network interface for high-speed packet processing, one or more acceleration engines for dedicated functions (e.g., security / cryptography, machine learning, storage), programming logic, integrated circuits, etc. Such a SoC may be referred to as a data processing unit (DPU). The DPU can be an example of the processing unit 25.

[0075] In the exemplary NIC 13, the processing unit 25 executes the operating system kernel 237 and user space 241 to provide services. The kernel 237 may be a Linux kernel, a Unix or BSD kernel, a real-time OS kernel, or another kernel used to manage the hardware resources of the processing unit 25 and manage the user space 241.

[0076] Service 233 may include networking, security, storage, data processing, coprocessing, machine learning, telemetry (such as telemetry service 233 of Figure 3), and / or other services. Service 233 and ESP agent 236 may include executable instructions. Processing unit 25 may execute service 233 and edge service platform (ESP) agent 236 as processes and / or execute service 233 and edge service platform (ESP) agent 236 within a virtual execution element (such as a container or virtual machine). As described elsewhere herein, service 233 may enhance the processing power of the host processor (e.g., microprocessor 210) by, for example, causing server 12 to offload packet processing, security, or other operations that would also be performed by the host processor.

[0077] Processing unit 25 executes Edge Services Platform (ESP) agent 236 to exchange data and uses Edge Services Controller 28 for Edge Services Platform to control the data. Although shown in user space 241, in some instances, ESP agent 236 may be a kernel module of kernel 237.

[0078] For example, ESP agent 236 can collect telemetry data generated by service 233 and send it to the ESP controller (another way of describing the edge service controller 28 shown in the example of Figure 1). The telemetry data describes the service and / or resource availability in the network, server 12, and / or processing unit 25 (such as memory or processor and / or core utilization). As another example, ESP agent 236 can receive service codes from the ESP controller to execute any of the services 233, configure the service configuration of any of the services 233, inject packets or other data into the network.

[0079] The edge service controller 28 manages the operations of the processing unit 25 by, for example, compiling and configuring services 233 executed by the processing unit 25; deploying services 233; adding, deleting, and replacing NIC 13 within the edge service platform; monitoring services 233 and other resources on NIC 13; and managing the connectivity between the various services 233 running on NIC 13. Exemplary resources on NIC 13 include memory 227 and processing circuitry 231.

[0080] Figure 3 is a conceptual diagram illustrating a data center with computing nodes according to the technology described herein. Each server includes a network interface card with a separate processing unit controlled by an edge service platform 300. The edge service platform 300 may include a network automation platform 306 and an orchestrator 304. The rack of the computing node may correspond to server 12 in Figure 1, and switches 16A / 18A and 16B / 18B may correspond to switches 16 and 18 in structure 14 of Figure 1. The processing unit 25, shown as a data processing unit (DPU), may include agents 236 and services (such as service 233 in Figure 2), which may represent software. The service 233 executed by the processing unit 25 may include network service 233A, L4-L7 service 233B, telemetry service 233C, and Linux+SDK (Software Development Kit) service 233D.

[0081] As described more fully herein, processing unit 25 can send telemetry data (shown as telemetry data 312) and other information for the NIC including this processing unit to compiler 304 of edge service platform 300 via agent 236 and telemetry service 233C. Compiler 304 may represent an example of edge service controller 28 of Figure 1 and may include performance monitoring system 500 (shown in more detail in Figure 4), which includes telemetry service 440 (shown in more detail in Figure 4). Performance monitoring system 500 can receive telemetry data, including metrics, from a large number of agents 236 associated with a large number of hosts (another way of referring to server 12) via telemetry service 440.

[0082] A network automation platform 306, representing an example of controller 24 in Figure 1, connects to and manages network devices (e.g., server 12 and / or switches 16 / 18) and compiler 304. The network automation platform 306 can, for example, deploy network devices, configure and manage the network. A performance monitoring system 500 can extract telemetry data, analyze it, and provide indications of network status. Various APIs can provide a user interface to the network automation platform and / or performance monitoring system, enabling, for example, access and automatic configuration of intent-based policies regarding network operation and performance.

[0083] Figure 4 illustrates a scalable, microservice-based telemetry service 440 capable of acquiring time-series telemetry data from computing devices, such as agent 236 of Figure 3, and enabling different consumers to obtain telemetry data through subscription services. Telemetry service 440 may be part of performance monitoring system 500, or part of controller 28 or controller 24. Consumers of telemetry data may be other shared services included in performance monitoring system 500, as detailed in Figure 5.

[0084] Administrators or applications can express telemetry acquisition requests as "intents," which define how telemetry data will be acquired in high-level "natural language." A telemetry intent compiler receives telemetry intents and translates them into abstract telemetry configuration parameters, which provide a categorical description of the desired telemetry data, also known as metrics or performance measurements. Telemetry service 440 can identify a set of devices from which it will acquire telemetry data based on the telemetry intents. For each device, the telemetry service can determine the device's capabilities regarding telemetry data acquisition. This capability may include the telemetry protocols supported by the device. The telemetry service can create protocol-specific devices based on the abstract telemetry configuration parameters and the telemetry protocols supported by the device. Devices in a network system that support a specific telemetry protocol can be assigned to telemetry collectors (metric collectors) that support distributed telemetry protocols.

[0085] Telemetry service 440 can be implemented as a collection of fault-tolerant and scalable microservices. New instances of microservices can be created in response to the growing demand for telemetry acquisition services.

[0086] Specifically, the exemplary data center 400 may include telemetry services 440 in network 405 and / or within one or more data centers. The data center 400 of Figure 4 can be described as an example or alternative implementation of the data center 10 of Figure 1. One or more aspects of Figure 4 may be described herein within the context of Figure 1.

[0087] While data centers, such as those shown in Figures 1 and 4, can be operated by any entity, some data centers are operated by service providers whose business models may involve providing computing power to customers or clients, often through co-leasing. For this reason, data centers typically contain a massive number of compute nodes or host devices. For efficient operation, these hosts must be interconnected and connected to the outside world, and this capability is provided via physical devices that can be interconnected in a leaf-spine topology. The collection of these physical devices (such as network devices and host devices) forms the underlying network.

[0088] In some examples, data center 10 may represent one of many geographically distributed network data centers. In the example of Figure 4, data center 400 includes a set of storage systems, application servers, compute nodes, or other devices, including devices 410A-410N (collectively referred to as “device 410”, meaning any number of devices). Devices 410 may be interconnected via the high-speed switching structure 14 of Figure 1 provided by one or more physical network switches and routers.

[0089] Device 410 may represent any of several different types of devices (core switch, spine network device, leaf network device, edge network device, or other network devices), but in some examples, one or more devices 410 may represent physical compute nodes and / or storage nodes in a data center. For example, one or more of devices 410 may provide an operating environment for executing one or more customer-specific applications or services. Optionally, or additionally, one or more of devices 410 may provide an operating environment for one or more virtual machines or other virtualization instances (such as containers). In some examples, one or more of devices 410 may optionally be referred to as a host compute device, host, or server. Thus, device 410 may execute one or more virtualization instances, such as virtual machines, containers, or other virtual execution environments for running one or more applications or services (such as virtualized network functions (VNFs)).

[0090] Generally, each of the devices 410 can be any type of device that can operate on a network and generate data (e.g., connectivity data, streaming data, sFlow data, resource utilization data) accessible by telemetry or other means, and may include any type of computing device, sensor, camera, node, monitoring device, or other device. Furthermore, some and all of the devices 410 may represent components of another device that can generate data acquired by telemetry or other means. For example, some and all of the devices 410 may represent physical or virtual devices such as switches, routers, hubs, gateways, and security devices (such as firewalls, intrusion detection and / or intrusion prevention devices).

[0091] Telemetry service 440 can configure device 410 (and / or other devices) to generate and provide telemetry data related to the operation of these devices. This data may include process usage data, memory usage data, network usage data, error counts, etc. Telemetry service 440 can be configured to collect telemetry data from device 410 using protocols supported by device 410. Applications, processes, threads, etc., can subscribe to the collected telemetry data to be notified when telemetry data becomes available to one or more devices supporting the network.

[0092] User interface device 429 can be implemented as any suitable device for presenting output and / or accepting user input. For example, user interface device 429 may include a display. User interface device 429 may be a computing system, such as a mobile or non-mobile computing device operated by a user and / or administrator 428. In some examples, user interface device 429 may be physically separate from controller 24 and / or located in a different location from controller 24. In such examples, user interface device 429 may communicate with controller 24 via a network or other communication tools. In other examples, user interface device 149 may be a local peripheral device of controller 24 or 28, or may be integrated into controller 24 or 28.

[0093] In some aspects, user interface device 429 can communicate with telemetry service 440 or its components to configure telemetry service 440, thereby configuring the device to use high-level claims of intent to provide telemetry data and receive telemetry data from the device and other components of data center 10 via telemetry service 440. In some aspects, telemetry service 440 can be configured by an application or service that uses telemetry data obtained via telemetry service 440. For example, performance monitoring system 500 or its components of FIG. 5 can be configured to acquire and provide telemetry data from device 410, such as at a desired acquisition rate. In some cases, telemetry data includes metrics (performance measurements) for different aspects of the host device, and the metrics (performance measurements) can be acquired for each metric as a series of metrics, obtained at each of a plurality of specific times according to a specific sampling rate, and the metric is associated with a metric name corresponding to the metric.

[0094] Telemetry Service 440 provides a shareable telemetry data acquisition service to collect telemetry data from multiple devices in a network system according to protocols supported by the devices. The acquired telemetry data can be used to perform anomaly detection and generate alerts to monitor cloud computing infrastructure available to multiple applications and tenants at a cloud scale.

[0095] Administrator 128 can input data using UI device 129, which represents telemetry acquisition requests as "intents" defined in high-level "natural language". Telemetry service 440 can receive the data representing the intent and translate the high-level intent into abstract telemetry configuration parameters that can be programmably processed by the telemetry controller of telemetry service 440. The telemetry controller can create protocol-specific telemetry configurations for the device based on the abstract telemetry configuration parameters and the telemetry protocols supported by the device.

[0096] As mentioned above, in some situations, clients in a data center may experience network problems such as increased latency, packet loss, low network traffic, or slow workload processing. Deploying workloads in a large, multi-tenant data center can complicate the resolution of these problems. Telemetry data, such as that provided by Telemetry Service 440, can be used to help resolve issues in the data center.

[0097] In the example of Figure 4, network 405 connects telemetry service 440, host device 410A, and host devices 410B-410N. Host devices 410A, 410B-410N can be collectively referred to as "host devices 410", which means any number of host devices 410.

[0098] Each of the host devices 410 can be an example of device 12N in FIG1; however, in the example of FIG4, each of the host devices 410 is implemented as a server or host device operating as a physical or virtualized computing node or a storage node in a virtualized data center, which is distinct from a network device. As further described herein, one or more of the host devices 410 (e.g., host device 410A of FIG4) can execute multiple virtual computing instances, such as virtual machine 428. Furthermore, one or more of the host devices 410 (e.g., one or more of host devices 410B-410N of FIG4) can execute applications or services on non-virtualized, single-tenant, and / or bare metal servers. Thus, the example of FIG4 illustrates a network system that may include a combination of virtualized server devices and bare metal server devices.

[0099] It is also connected to a user interface device 129 that can be operated by an administrator 128. In some examples, the user interface device 129 may present one or more user interfaces on a display device associated with the user interface device 129.

[0100] Network 405 may correspond to either the switching structure 14 and / or the service provider network 7 of Figure 1, or alternatively, it may correspond to a combination of the switching structure 14, the service provider network 7, and / or another network. Although not shown in Figure 4, network 405 may also include some of the components of Figure 1, the SDN controller 24, and the edge service controller 28.

[0101] Network 405 shows spine devices 402A and 402B (collectively referred to as "spine devices 402", and any number of spine devices 402), and leaf devices 403A, 403B, and 403C (collectively referred to as "leaf devices 403", and any number of leaf devices 403). Although network 405 is shown with spine devices 402 and leaf devices 403, network 405 may also include other types of devices, including core switches, edge devices, top-of-rack switches, and other devices (such as those shown in Figure 1).

[0102] Generally, a Network 405 can be the Internet, or it can include or represent any public or private communications network or other network. For example, a Network 405 can be a cellular network, ZigBee, Bluetooth, Near Field Communication (NFC), satellite, enterprise, service provider, and / or other types of networks capable of transmitting data between computing systems, servers, and computing devices. One or more of the client devices, server devices, and other devices may use any suitable communication technology to send and receive data, commands, control signals, and / or other information on network 405. Network 405 may include one or more network hubs, network switches, network routers, satellite dish antennas, or any other network devices. Such devices or components may be operatively internally coupled to enable the exchange of information between computers, devices, or other components (e.g., between one or more client devices or systems and one or more server devices or systems). Each of the devices or systems shown in Figure 4 is operatively coupled to network 405 using one or more network links. The link coupling such a device or system to network 405 may be Ethernet, Asynchronous Transfer Mode (ATM), or other types of network connections, and such connections may be wireless and / or wired connections. One or more of the devices or systems shown in Figure 4 or on network 405 may be located at a remote location relative to one or more other illustrated devices or systems.

[0103] Each of the host devices 410 represents a physical computing device or compute node or storage node that provides an execution environment for virtual hosts, virtual machines, containers, and / or other real or virtualized computing resources. In some examples, each of the host devices 410 may be a component of a cloud computing system, server farm, and / or server cluster (or a portion thereof) that provides services to client devices and other devices or systems.

[0104] Specific aspects of host device 410 are described herein with respect to host device 410A. Other host devices 410 (e.g., host devices 410B-410N) may be described in the same manner and may also include components with the same reference numerals, which denote the same, similar, or corresponding components, means, modules, functions, and / or features. Therefore, the description herein with respect to host device 410A can correspondingly be applied to one or more other host devices 410 (e.g., host devices 410B-410N).

[0105] In the example of Figure 4, host device 410A includes underlying physical computing hardware, which includes one or more processors 413, one or more communication units 415, one or more input devices 416, one or more output devices 417, and one or more storage devices 420. In the illustrated example, storage device 420 may include kernel module 422 and virtual router module 424. Storage device 420 may also include virtual machines 428A-428N (collectively referred to as “virtual machine 428”, and representing any number of virtual machines 428), which, when present, may execute on top of or be controlled by a hypervisor (not shown). One or more of the devices, modules, storage areas, and other components of host device 410A may be interconnected to enable inter-component communication (physical, communicative, and / or operational). In some examples, this connectivity may be provided via a communication channel (e.g., communication channel 412), system bus, network connection, inter-process communication data structure, or any other method for transferring data.

[0106] Processor 413 can perform functions and / or execute instructions associated with host device 410A. Communication unit 415 can communicate with other devices or systems on behalf of host device 410A. One or more input devices 416 and output devices 417 can represent any other input and / or output devices associated with host device 410A. Storage device 420 can store information used for processing during operation of host device 410A.

[0107] The virtual router module 424 can execute multiple routing instances for the corresponding virtual network within the data center 10 (Figure 1) and can route packets to the appropriate virtual machine executing within the operating environment provided by device 410. When used for infrastructure employing Contrail SDN, the virtual router module 424 can also be responsible for capturing overlay stream data, such as Contrail stream data.

[0108] Virtual machines 428A-428N (collectively referred to as "virtual machine 428", meaning any number of virtual machines 428) can represent exemplary instances of virtual machine 428. Host device 410A can partition the virtual and / or physical address space provided by storage device 420 into a user space for running user processes. Host device 410A can also partition the virtual and / or physical address space provided by storage device 420 into a protected kernel space that is not accessible to user processes.

[0109] Each of the virtual machines 428 can represent a tenant virtual machine running a client application (such as a web server, database server, enterprise application, or a managed virtualization service for creating service chains). In some cases, any one or more of the host devices 410 or other computing devices directly host the client application, i.e., they are not used as virtual machines (e.g., one or more of host devices 410B-410N, such as host device 410B and host device 410N). Although one or more aspects of this disclosure have been described with respect to virtual machines or virtual hosts, the techniques described herein with respect to such virtual machines or virtual hosts according to one or more aspects of this disclosure can also be applied to containers, applications, processes, or other execution units (virtualized or non-virtualized) executing on host device 410.

[0110] In the example of Figure 4, one or more processors 443 may execute telemetry service 440 to perform the operations categorized herein as telemetry service 440, which may be stored in memory (such as storage device 450). Telemetry service 440 may include one or more communication units 445, one or more input devices 446, and one or more output devices 447. Storage device 450 may include intent service 418, telemetry controller 421, telemetry subscription service 408, and telemetry controller 510.

[0111] One or more of the devices, modules, storage areas, and other components of telemetry service 440 can be interconnected to enable inter-component communication (physical, communicative, and / or operational). In some examples, this connectivity can be provided via one or more of a communication channel (e.g., communication channel 442), a system bus, a network connection, inter-process communication data structures, and any other methods for transmitting data.

[0112] One or more processors 443 may be part of the NIC of FIG1, and / or may include processing circuitry that performs operations according to one or more aspects of this disclosure. Examples of processors 443 include microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device.

[0113] One or more communication units 445 of the telemetry service 440 can communicate with devices outside the telemetry service 440 by sending and / or receiving data, and in some aspects, can operate as both input and output devices. In some examples, the communication unit 445 can communicate with other devices, such as the encoder 304 and agent 302 shown in the example of FIG3, via a network.

[0114] One or more storage devices 450 within service 440 may store information to be processed during operation of service 440. Storage devices 450 may store program instructions and / or data associated with one or more modules described in accordance with one or more aspects of this disclosure. One or more processors 443 and one or more storage devices 450 may provide an operating environment or platform for such modules, which may be implemented as software, but in some examples may include any combination of hardware, firmware, and software. One or more processors 443 may execute instructions, and one or more storage devices 450 may store instructions and / or data of one or more modules. The combination of processors 443 and storage devices 450 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. Processors 443 and / or storage devices 450 may also be operatively coupled to one or more other software and / or hardware components, including, but not limited to, one or more components of alarm service 440 and / or shown as one or more devices or systems connected to telemetry service 440.

[0115] In some examples, one or more storage devices 450 are implemented via telemetry memory, which may mean that the primary purpose of one or more storage devices is not long-term storage. The storage device 450 of the telemetry service 440 can be configured as volatile memory for short-term storage of information, so that the stored content is not retained if deactivated. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, the storage device 450 also includes one or more computer-readable storage media. The storage device 450 can be configured to store a larger amount of information than volatile memory. The storage device 450 can also be configured as non-volatile storage space for long-term storage of information and retains the information after an activation / deactivation cycle. Examples of non-volatile storage elements include magnetic hard disks, optical disks, flash memory, or forms including electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory.

[0116] Intent service 418 receives telemetry intent 430, which expresses a high-level telemetry request for generating and acquiring telemetry data. Telemetry intent 430 can be expressed in natural language. For example, telemetry intent 430 could be “Acquire CPU resource usage metrics from all devices at 1-minute intervals.” Another example is “Acquire memory resource usage from devices Router A, Router B, and Router C.” Intent service 418 can translate telemetry intent 430 into one or more low-level telemetry commands and protocols that can implement telemetry intent 430. In some cases, the device can support more than one telemetry protocol. In this case, intent service can use protocol translation of telemetry intent 430 to select a protocol based on criteria such as the priority assigned to the protocol, the device's capabilities regarding the protocol, and the overhead associated with the protocol. Furthermore, in some aspects, intent service 418 can mediate intents for multiple applications requesting telemetry data from the same device. Intent service 418 is capable of sending low-level telemetry commands (conforming to the selected protocol) and instructions for the selected protocol to telemetry controller 441 to escalate telemetry acquisition for affected devices.

[0117] Telemetry controller 441 is capable of receiving low-level telemetry commands and instructions for selected protocols. In some aspects, telemetry controller 441 maintains up-to-date telemetry requirements for each device. Telemetry controller 441 is capable of providing telemetry acquisition devices 214 for devices such as leaf devices 203 and spine devices 202, as specified by telemetry commands and protocols translated from telemetry intent 430.

[0118] Telemetry subscription service 408 receives a request to subscribe to telemetry data generated by the device. In some aspects, in response to receiving a subscription, if a telemetry collector has not yet been provided to the device, telemetry controller 441 may provide a telemetry collector 510.

[0119] Telemetry collector 510 collects telemetry data from the device. The telemetry collector 510 can store the collected data in a cache or database (not shown in Figure 4 for illustrative purposes). Telemetry service 440 can provide the collected data to applications or services that have subscribed to the data.

[0120] Figure 5 illustrates a performance monitoring system according to the technology described in this disclosure. The performance monitoring system includes a collector for acquiring telemetry data and an alarm rule evaluator service for evaluating rules using the telemetry data. As shown in Figure 5, the metric collector of the performance monitoring system 500 can acquire telemetry data via a metric exporter 504. In the example, system 500 can be a consumer of telemetry data acquired by telemetry service 440 and can implement services and rules that can be used to acquire and / or subscribe to telemetry data. The performance monitoring system 500 can analyze the telemetry data according to alarm rules, which determine whether an alarm should be generated based on the telemetry data, as will be further explained below. In addition, the performance monitoring system 500 may include one or more machine learning components (such as machine learning component 521) and can be configured to provide an adaptive sampling interval for acquiring telemetry data, provide an adaptive rule evaluation interval, and / or be able to recommend alarm rules that provide a better understanding of the network.

[0121] Referring to Figure 5, applications and services running within a workload cluster (such as cluster 502) are configured to export various metrics of the network to a performance monitoring system 500 via one or more metric exporters 504A-504C. The performance monitoring system 500 may be an example of controller 24 and / or edge service controller 28, which may be configured to control the cluster of virtual machines 502 and communicate with multiple metric exporters 504. System 500 may include a metric collector 510 for collecting telemetry data, a metric time-series database (TSDB) 508 for storing telemetry data, a metric queryer 512 for receiving queries from users about metrics, and an alarm rule evaluator service 514 for evaluating alarm rules (such as user-created rules from users, such as network administrators). The metric collector 510, as shown in Figure 5, periodically discovers the metric exporters 504 and collects metrics, for example, by using a drag-and-drop method, wherein the performance monitoring system 500 determines the collection time. Applications and services on the network display their internal metrics through metric exporter 504, which can be an agent 236 that performs the export function. Metric exporter 504 can be embedded in the application / service or run alongside the application / service and display the metrics using an HTTP endpoint.

[0122] More specifically, metric collector 510 can automatically discover metric exporters 504 in the network (such as data center 10) and collect all metrics displayed by these exporters. Metric collector 510 periodically collects metrics at configured time intervals, which define the corresponding sampling rate. Previous systems typically used fixed time intervals to collect metrics. The collected metrics include metric values ​​associated with metric names and can be stored in a metric time-series database 508, where metrics are timestamped. Time-series data is typically stored as dense, high-precision data points, which can then be downsampled and discarded. Furthermore, TSDB can provide features for generating aggregated data sequences over time. Some advantages of time-series databases when used to store time-sorted data are that the time-sorted nature of the data compresses the data and reduces storage space (e.g., disk or solid-state drive) footprint. Time-series databases can be SQL (relational) or NoSQL (non-relational) in the architecture. NoSQL databases can operate better at scale within a cluster.

[0123] The metric queryer 512 is configured to interact with the time-series database 508 to access collected metrics. Metric data is accessed using the query language provided by the metric queryer 512, which offers an HTTP-based interface to users. In case of problems, users can manually query the collected metrics through the metric queryer interface, which supports the query language, enabling users to construct complex queries and access metric data.

[0124] When a user wants to monitor a metric or a set of metrics offline or in the background, the user can automate metric monitoring by creating specific metric evaluation rules (referred to as alert rules (also known as event rules)). These alert rules contain various conditions relating to the metric to be evaluated relative to a set of thresholds. Each alert rule can include the name of the corresponding metric, the threshold, and the comparison condition. For example, a user can configure an alert rule to alert the user when the system's CPU usage exceeds 80%. These rules can be evaluated periodically using the collected metric data by the alert rule evaluator service 514 at predetermined time intervals; and if the comparison condition is met, referred to herein as a hit, an alert is generated for the user. The alert rule evaluator service 514 includes an event reporter 516, an alert rule evaluator 518, and an alert rule database 520. More specifically, the alert rule evaluator 518 periodically reads the user-created rules from the alert rule database 520 and evaluates the rule representation relative to the metric data accessed through the metric queryer 512. The evaluation essentially determines whether the metric meets the conditions specified by the user as a rule; if so, an alarm is generated by the alarm rule evaluator 518 and transmitted to the user and / or stored by the event reporter 516.

[0125] When the alarm rule evaluator 518 evaluates a rule and determines that the comparison condition is met or true, an alarm is generated and the rule evaluation is considered a rule hit; and if no alarm is generated, the rule evaluation is considered a rule miss.

[0126] Self-learning metric collector

[0127] Metric acquisition units that use static predefined time intervals instead of dynamic time intervals (variable sampling rates) to obtain metrics generally work better, but they have the following disadvantages:

[0128] a. More storage space is needed.

[0129] b. Frequently collect metrics that users rarely access.

[0130] c. Analyzing metrics requires greater computing power because finding relevant metrics requires covering a large amount of metric data.

[0131] d. When the metric does not change much over time, redundant metric collection will occur.

[0132] Therefore, when collecting more useful metrics, the metric collector may use the same time intervals to avoid collecting many less useful metrics. As metrics are collected in scaled network environments, the problems listed above become more apparent and problematic.

[0133] In the example, a machine learning-based intelligent approach is employed to train a machine learning model using various analytics (obtained using telemetry data). This machine learning model then uses additional telemetry data to make predictions, such as the weights of predicted metrics or the weights of predictions for rules. Using machine learning, a metric collector (e.g., metric collector 510) can learn how to identify the usefulness of a metric based on its relevance to the user, or an alarm rule evaluator service (e.g., alarm rule evaluator service 514) can learn how to identify the relevance or weight of a metric.

[0134] For example, various metric attributes (which can be aggregated in some way) can be used to measure the relevance of a metric to a user to determine a metric relevance value, also referred to as metric weight in this paper. In the example, a higher metric weight indicates a greater likelihood that a user is interested in that metric; conversely, a lower metric weight indicates a less likely user is interested in that metric. A higher metric weight can then be used to calculate the updated sampling interval, which allows the collector to sample that metric more frequently (i.e., a higher metric weight increases the sampling frequency and decreases the sampling interval).

[0135] Figure 6 illustrates an example of a performance monitoring system 600 according to the technology described herein, including an intelligent collector 610 that utilizes machine learning via a machine learning module 621 to determine an improved metric collection rate. The intelligent collector 610 receives metrics from metric exporters 604A-604C of cluster 602. The intelligent collector 610 includes a metric sampler 630 for accessing metrics stored in a database 608; a metric metadata synchronizer 632; and a metric metadata inventory database 638; wherein the metric metadata may include metric relevance data and historical relevance data for each evaluated metric.

[0136] The intelligent collector 610 also includes a metric group discovery service 636 for discovering one or more corresponding metrics (or rules) related to a metric (or rule); a metric weight predictor 640 for predicting metric weights based on a machine learning model; a metric variance detector 634 for determining how a metric changes over a period of time; a key metric discovery service 642 for identifying key metrics (or rules); and a metric access rate calculator 644 for determining the metric and the access rate associated with the metric. First, the intelligent collector 610 can sample all exported metrics within the network at a predetermined default sampling interval. After each sampling iteration, a learning process can be triggered to learn more suitable sampling intervals for the exported metrics using the model of the machine learning module 621.

[0137] More specifically, the performance monitoring system 600 may include a metric time-series database 608, a metric queryer 612, and a query history database 618. Users can access the metric queryer 612 to query metrics stored in the metric time-series database 608. The query history of the metric queryer can be stored in the query history database 618. The performance monitoring system 600 also includes an alarm rule evaluator service 614 (operating in a similar manner to the alarm rule evaluator service 514 in Figure 5), and includes an event reporter 616 for reporting alarms / events based on rule evaluation results; an alarm rule evaluator 624 for evaluating rules to generate rule evaluation results; an alarm rule database 620 for storing alarm rules; and an alarm rule history database 622 for storing alarm rule evaluation results including alarms / events.

[0138] Figure 8 is an exemplary sequence diagram for determining a new sampling interval for metric acquisition according to the technology described in this disclosure. As shown in Figure 8, the intelligent collector 610 can initially collect all metrics from the metric exporter 604 at a default interval (e.g., a predetermined default sampling rate). For each metric, the corresponding metric value with its associated metric name is stored in the metric TSDB 608. The alarm rule evaluator service 614 evaluates alarm rules using the stored metrics, and, along with storing query history in database 618, stores events / alarms in the alarm history database 622.

[0139] The metric intelligent collector 610 is capable of determining various metric attributes. For example, the metric intelligent collector 610 can read its query history for a given metric and associated alert rules, use calculator 644 to determine the corresponding metric access rate, and store the access rate in the metric metadata database 638. The intelligent collector 610 can read the metric event history, determine the metric threshold hit rate, and store the threshold hit rate. The metric group discovery service 636 can discover one or more related metrics associated with a given metric, wherein these related metrics define a group, evaluate the related metrics and associated rules in the group, and determine the group access rate based on the evaluation of the rules associated with the related metrics in the group. Other different metric attributes can also be determined as being related to the expected sampling rate of a given metric or metric group. Using the determined metric attributes, the intelligent collector uses metric weight predictor 640 to determine the predicted metric weights and associated alert rules for the given metric. Using the predicted metric weights, the updated collection sampling rate can be determined and subsequently used by metric sampler 630 to collect additional metrics. In this way, a given metric can be collected using the regular sampling rate for a specified metric or related metric group.

[0140] In the example, metric weights can be determined based on various consideration / metric attributes, such as consideration / metric attributes represented by the following guidelines:

[0141] a) If a user has already accessed a metric (at a certain frequency) through a query, the user is likely to access the same metric again in the near future.

[0142] b) If a user has already accessed a metric through a query, the user may also be interested in the metric.

[0143] Relevant metrics can be identified based on various factors, including:

[0144] 1. If a user frequently or simultaneously accesses two metrics within a certain time window, these metrics may be considered related.

[0145] 2. If a user accesses two metrics when a system event occurs, then these metrics may be considered related.

[0146] 3. If two metrics share a common metric label (e.g., CPU-related metrics), then these metrics can be considered related.

[0147] 4. If two metrics are derived using the same or separate software components or modules, then these metrics may be considered related.

[0148] Based on the above considerations, a set of metrics related attributes, such as metric access rate, metric threshold hit rate, related metric group access rate, and metric variance, can be calculated for each metric and used to calculate the weight of the metric, as described below.

[0149] a) Metric Access Rate: This can be defined as the ratio between the access rate and the sampling rate, indicating how many times the metric has been accessed compared to its sampling rate within a given time period, used to evaluate alarm rules. A higher value indicates that users are more interested in the metric, while a lower value indicates that users are less interested. In other words, the metric access rate is the number of times it is accessed within a fixed duration. The metric sampling rate is the number of samples taken within a fixed duration. The metric access rate can be determined using the metric access rate calculator 644.

[0150] Access rate = Access rate / Sampling rate

[0151] (b) The metric threshold hit rate can be defined as the number of times a metric crosses the threshold set by the user in the corresponding alert rule, divided by the number of samples. Generally, users set thresholds for useful metrics in alert rules to monitor system behavior. The metric threshold hit rate can be determined using the results of the alert rule evaluator service 614 and the smart collector 610.

[0152] Threshold hit rate = Number of times the threshold was crossed / Number of samples

[0153] c) Related metric group access rate: This can be defined as the average access rate of each metric in a related metric group. Related metrics can be identified using metric labels, origin, and query history. Two metrics can be considered related metrics when they share a common label, originate from the same exporter / software component, or are queried together. The group access rate can be calculated by averaging the access rates of the individual metrics in the group. This metric attribute can be determined using the metric group discovery service 636 and the metric access rate calculator 644.

[0154] The access rate of a metric group is the average of the access rates of all metrics in the group.

[0155] If there are more relevant groups, it is possible to determine the access rate of the relevant metric group, where the access rate of the relevant metric group = the average of the access rates of all groups.

[0156] d) Critical Metric Labeling: When a metric is part of a critical event, it can be automatically labeled as critical. Critical metric labels can be attached to these types of metrics, and when attached in this way, maximum weight can be assigned to the metric, resulting in more frequent sampling of the metric. Some examples of metrics that can be applied to include: packet dropping, CRC error counting, etc. In some cases where metrics are manually labeled as critical or system metrics, a minimum threshold interval (i.e., a predetermined high sampling rate) is automatically considered. This can be determined by the critical metric discovery service 642.

[0157] e) Metric variance: can be defined as the difference between two metrics sampled within a specific time window. Metric variance indicates how a metric changes over time. Metrics with lower variance can be considered to have lower weight and will be sampled at higher intervals. Metric variance can be determined using a metric variance detector 634.

[0158] By using the relevance attributes of the metric, its weight can be predicted using the machine learning model of the metric weight predictor 640 and the machine learning module 621. The machine learning module 621 is capable of reading historical data related to the metric, such as the historical relevance attributes of the metric, and training the metric weight machine learning model. The intelligent collector training process includes determining the relevance attributes of the metric; feeding the relevance attributes into the machine learning model; predicting the metric weight; and determining the updated sampling interval (corresponding to the sampling frequency).

[0159] Specifically, the metric weights can be calculated using linear regression with metric attributes R1, R2, R3, and R4, as follows:

[0160] The access rate is R1.

[0161] The threshold hit rate is R².

[0162] The access rate for the relevant metrics group is R3.

[0163] The variance is measured as R4

[0164] W1 = a + b(R1)

[0165] W2 = a + b(R2)

[0166] W3 = a + b(R3)

[0167] W4 = a + b(R4)

[0168] Where a=(ΣWi)(ΣRi*2)-(ΣRi)(ΣWiRi) / n(ΣRi*2)-(ΣRi)2b=n(ΣWiRi)-(ΣRi)(ΣWi) / n(ΣRi*2)-(ΣRi*)2 the weight of the metric can be calculated by averaging the predicted weights.

[0169]

[0170] Here, n is the number of samples. Using the calculated metric weights (or determining the metric as the key), the update sampling interval / sampling rate can be determined. Subsequently, the smart collector 610 can use the new sampling interval to obtain the metric value for this metric.

[0171] Figure 7 is an exemplary flowchart of an intelligent collector 610 of a performance monitoring system 600, according to the technology described herein, for determining metric weights and corresponding updated sampling intervals for metric collection. At 702, the intelligent collector 610 reads metrics from a metric database 708. At 704, the intelligent collector may determine one or more of the following: metric access rate, metric variance, whether the metric is a critical metric, and metric threshold hit rate. At 708, a determination is made regarding whether query history for other metrics can be obtained. If yes, the process proceeds to 712. If no, the process proceeds to 710.

[0172] At 712, the query history is fed into the machine learning model, where one or more relevant metric groups can be identified at 714 using the metric group discovery service 636. At 716, the smart collector can use the metric access rate receiver 644 to calculate the metric access rate for the metrics in one or more groups. At 710, the relevance attributes calculated at 704 or 716 are stored; and at 720, this metric relevance data can be stored, for example, in a database 638, and the process proceeds to 718. At 718, a determination is made as to whether the next metric should be evaluated. If not, the process ends. If another metric is to be evaluated, the process proceeds to 702 to repeat steps 702-718 for the next metric.

[0173] To predict metric weights and calculate the updated sampling interval, at 730, the machine learning module can read historical relevance data for the metric. At 732, a machine learning model for predicting metric weights and sampling intervals is trained using the historical relevance attributes. At 734, the relevance attributes for a specific metric are read; and at 736, these relevance attributes are passed to the machine learning model. At 738, metric weights are predicted; and at 740, if the metric is a critical metric, the sampling interval is calculated, such as by dividing the default sampling interval by the predicted metric weights, or by sampling using a predetermined minimum sampling interval (maximum frequency). At 742, additional metrics are evaluated using steps 734-740, and the process ends after all metrics have been evaluated.

[0174] The following includes exemplary pseudocode for implementing the above techniques:

[0175]

[0176]

[0177]

[0178] Machine Learning for Telemetry Rule Evaluation

[0179] As mentioned earlier, the background service (referred to as the Alert Rule Evaluator Service) periodically evaluates alert rules. Some performance monitoring systems use static time intervals to evaluate alert rules. A rule evaluation is considered a rule hit when the Alert Rule Evaluator Service generates an alert for the user due to a true comparison condition; otherwise, if no alert is generated, it is considered a rule miss. The periodic rule evaluation process involves computationally intensive tasks such as querying massive amounts of telemetry data, aggregating telemetry data, and comparing the aggregated data with multiple thresholds.

[0180] When configuring a large number of rules in a computationally constrained environment, a rule evaluation process using static time intervals cannot properly evaluate the rules. Furthermore, processing rules on an overloaded system can waste computational resources. To avoid these scaling issues, administrators typically limit the number of rules they can configure or increase the rule evaluation interval.

[0181] An optimized rule evaluation cycle means being able to evaluate rules at different frequencies based on their past success or failure (hit or miss) rates. This means that when a rule has been missing for a long time, a solution based on a fixed evaluation rate will waste resources because the probability of a successful evaluation in the near future is low.

[0182] In the example, a performance monitoring system utilizing a machine learning-based intelligent approach is used for rule evaluation. This method evaluates rules periodically at optimized evaluation intervals, and these alert rules may change over time as network conditions change. The evaluation interval for rules can be assigned based on their predetermined weights. These weights indicate the priority of the rule and may be inversely proportional to the desired evaluation interval. In other words, a higher rule weight corresponds to a lower evaluation interval, and vice versa. Rule weights can be predicted using machine learning models and past rule evaluation data.

[0183] Figure 9 illustrates an example of a performance monitoring system 900 according to the technology described herein. The performance monitoring system 900 utilizes machine learning via a machine learning module 921 to determine the evaluation interval for assessing improvements to alert rules. The performance monitoring system includes a metric collector 910, a metric TSDB 908, a metric queryer 912, and an alert rule evaluator service 914. As shown in Figure 9, the alert rule evaluator service includes an alert rule database 920, an alert rule evaluator 924, an event reporter 916, a rule evaluation history database 928, an alert rule history analyzer 926, and a rule weight predictor 922. The alert rule evaluator service 914 can operate in a similar manner to alert rule evaluator services 514 and 614 as described herein, and may include additional capabilities using machine learning. Service 914 can store rule evaluation results in the rule evaluation history DB 928, which may be a persistent database; and the machine learning module 921 can use these results to derive analyses related to alert rule evaluation, such as by predicting rule weights. As mentioned earlier, when an alarm rule evaluation generates an alarm (by comparing a metric value with a rule threshold and determining that the rule comparison condition is true), the evaluated rule is considered a rule hit. When no alarm is generated, it means that the rule comparison condition is not true, and the rule is considered a rule miss. By analyzing the rules and their corresponding metrics over time, such as by determining the hits and misses of a series of collected metrics over a first rule evaluation interval and predicting the rule weights, the updated evaluation interval can be determined based on the predicted rule weights.

[0184] Specifically, the machine learning module 921 can interact with or be incorporated into the alarm rule evaluator service 914. The alarm rule history analyzer 926 can analyze the rule evaluation history and derive data rule analysis, as described below. The alarm rule history analyzer 926 can provide (or, in other words, pass on, possibly as a reference to a memory location (e.g., an indicator) for storing such analysis) analysis to the rule weight predictor 922, which can process the analysis to determine rule weights and thus determine the corresponding updated evaluation interval for each rule evaluation. For example, the following rule analysis can be used:

[0185] a) Rule hit rate (RHR) can be defined as the number of successful evaluations (alerts / hits) in the total rule evaluation.

[0186] RHR = Alarms generated by the rule / Total number of times the rule is evaluated

[0187] b) The Relevant Rule Hit Rate (RRHR) can be defined as the success rate of evaluations of other alarm rules related to a given alarm rule. For example, if two rules contain metrics from the same source in the network, these two rules can be considered relevant rules. This metric provides information about the overall health of the system. A higher RRHR value indicates poor system health and a higher probability that the rule will generate an alarm. The Relevant Rule Hit Rate can be calculated as follows:

[0188] RRHR = Total hit rate of all relevant rules / Total number of evaluations of relevant rules

[0189] c) The near-miss rate (RCMR) can be defined as the number of times an alert fails to be generated because the evaluated metric is less than, but the margin is close to, the rule threshold. In other words, the rule is evaluated as a value that is only slightly less than the rule threshold. The miss margin can be calculated as a percentage relative to the rule threshold.

[0190] Miss margin = 100 – [(Evaluation value * 100) / Rule threshold]

[0191] To mark an assessment as close to a miss, a miss margin value can be compared to an acceptable margin limit, and an assessment is considered close to a miss assessment when the margin falls within the acceptable margin limit. For example, if the rule threshold is 20 and the rule's assessment value is 18, then the miss margin = 100 – [(18*100) / 20], which is 10%. This means that the rule assessment fails to generate an alarm due to this 10% amount. To consider an assessment as close to a miss, let's assume 20% is used as the miss margin threshold. Therefore, a 10% miss margin makes an assessment considered close to a miss assessment. The rule close to miss rate can be calculated as follows:

[0192] RCMR = Near misses / Total number of assessments

[0193] d) Rule metric criticality: Rule metrics can be considered critical, such as when a tag is attached to a rule, when any metric included in a rule is part of a critical event and is tagged by the user, or when runtime metric metadata is marked as critical. For example, when a user observes a log event related to packet loss, this event can be labeled as critical, and all network-related metrics can also be considered critical metrics. When a metric is labeled as critical, the rule weights related to that metric can be set to their maximum value, allowing the rule evaluator to evaluate these rules more frequently (i.e., using smaller evaluation intervals).

[0194] Using the rule analysis calculated above, the rule weight predictor 922 can predict the weights for each rule. Depending on the rule weights, a new evaluation time interval can be derived for each rule using an inverse relationship.

[0195] Rule analysis can be periodically calculated over a predetermined period of time to generate rule weight predictions, such as using the following:

[0196] The rule hit rate is R1

[0197] R1 = (Alarm count / Evaluation count)

[0198] The rule hit rate is R2.

[0199] R2 = (Relevant rule hit rate / Relevant rule evaluation count)

[0200] The rule has a near miss rate of R3

[0201] R3 = (Near miss assessment count / Assessment count)

[0202] The rule weights can be predicted using the linear regression formula and the individual ratios R1, R2, and R3 calculated above.

[0203] W1 = a + b(R1)

[0204] W2 = a + b(R2)

[0205] W3 = a + b(R3)

[0206] Among them, a=(ΣWi)(ΣRi*2)-(ΣRi)(ΣWiRi) / n(ΣRi*2)-(ΣRi)2b=n(ΣWiRi)-(ΣRi)(ΣWi) / n(ΣRi*2)-(ΣRi*)2

[0207] The average of the predicted weights can be considered as the regular weights.

[0208]

[0209] Then, the alarm rule evaluator service 914 can calculate a new evaluation interval for the rule based on the following rule weights:

[0210] Updated evaluation interval = (default or first rule evaluation interval / rule weight).

[0211] The Alert Rule Evaluator Service 914 can use the updated evaluation interval to evaluate rules using newly collected metrics.

[0212] In the example, the alarm rule evaluation interval is independent of the sampling rate. In some cases, performance monitoring systems can use the rule evaluation interval to coordinate the metric sampling rate. If a rule is evaluated less frequently than before, it may be desirable to slow down the acquisition of the relevant metric within the same timeframe; if a rule is evaluated more frequently than before, it may be desirable to increase the acquisition rate of the relevant metric (reduce the sampling interval).

[0213] Figure 10 illustrates an exemplary sequence diagram of the performance monitoring system 900 of Figure 9. Alarm rules, such as user-created alarm rules, can be created and stored in an alarm rule database 920. An alarm rule evaluator service 914 can read rules from the alarm rule database 920 and can evaluate rules by accessing a metric queryer 912 to receive corresponding metric values ​​for the metric and by comparing the metric values ​​with the corresponding thresholds of the rules to determine the hit count and miss count for each rule using a first evaluation interval. The evaluation results can be stored in a rule evaluation database 928. The alarm rule evaluator service 914 can discover relevant rules for the metric and for the specified rule, and calculate the relevant rule hit count and miss count. The alarm rule service can also calculate rule hit rate, relevant rule hit rate, and near hit rate. These rule attributes and rule history can be used as training data for a machine learning model for a rule weight predictor. The machine learning model can use the rule attributes corresponding to the first evaluation interval to predict rule weights, determine an updated rule evaluation interval based on the predicted rule weights, and then use the updated rule evaluation interval to evaluate the rules.

[0214] Figure 11 is a flowchart illustrating the operation of the alarm rule evaluator service 1114 according to the present disclosure. First, at 1100, the alarm rule evaluator service 1114 can determine if a rule history exists. If not, at 1108, a first evaluation interval can be used to evaluate the rule, with each rule evaluated multiple times and the evaluation results recorded in the history database. If a rule history exists, at 1102, the rule history corresponding to the rule can be checked; and at 1104, rule evaluation analysis (rule attributes) can be determined and used as training data to update the rule weight predictor of the machine learning module 921. At 1106, it can be determined whether the rule includes a key metric. If it is determined that the rule has a key metric, at 1108, the maximum weight can be assigned to the rule; and at 1116, the updated evaluation interval can be determined as a function of the predicted weights. In the example, a predetermined minimum evaluation interval can be used for the key metric. If it is determined that a rule does not have a key metric, then at 1112, rule evaluation analysis / rule attribute can be used to determine the rule weights for the rule; at 1114, the predicted weights can be assigned to the rule; and at 1116, the updated evaluation interval can be determined as a function of the predicted weights. In the example, the updated evaluation interval is inversely proportional to the predicted weights of the rule. At 1118, the rule can then be evaluated using the updated evaluation interval; and at 1120, these new evaluation results can be stored in the rule evaluation history database 928. The actions described in Figure 11 can continue to optimize the rule evaluation interval as a condition for monitoring network changes, building upon the previous steps.

[0215] The following includes exemplary pseudocode for implementing the above techniques:

[0216] pseudocode

[0217]

[0218]

[0219]

[0220]

[0221] Self-learning telemetry alarm rule recommender

[0222] Figure 12 illustrates an example of a performance monitoring system 1200 according to the technology described herein, which provides recommended alert rules for a network of performance monitoring computing devices. The performance monitoring system 1200 may be similar in many respects to monitoring systems 500, 600, and 900 shown herein, and may also include various other components of these systems. As shown in Figure 12, system 1200 may include a metric collector 1210, a metric TSDB 1208, a metric queryer 1212, an alert rule evaluator service 1214, and a machine learning module 1222. The alert rule evaluator service 1214 may include an alert rule database 1220, an alert rule evaluator 1218, and an event reporter 1216. The alert rule evaluator service 1214 may store alert rule evaluation results in a persistent database, and the machine learning module 1222 may use this data to derive analysis of user-created alert rules and ad-hoc rules to automatically generate recommended alert rules, thereby fine-tuning the alert generation information to be more relevant to the user.

[0223] For example, if a network system has high CPU utilization, administrators typically look for applications or modules that consume the most CPU resources or perform the most CPU-intensive operations. After performing this analysis, administrators usually use relevant metrics to create one or more alert rules to catch the high CPU problem before it recurs and may take action to prevent excessive system CPU usage.

[0224] Manually creating alert rules can be time-consuming and may require administrators to analyze metric data and attempt to identify suspicious metrics that might be related to the problem they are trying to diagnose. This can become even more difficult when the volume of telemetry data is large. Due to the time delay in implementing user-created alert rules, the process of manually creating a suitable set of alert rules to diagnose a problem can be time-consuming, inefficient, and in some cases, unsuccessful. For example, the defect / problem may no longer exist when the administrator begins the investigation or adds a new rule.

[0225] The machine learning-based intelligent alert rule creation method according to this disclosure can automatically discover relevant metrics related to existing rule metrics and recommend additional alert rules for future problem analysis. The recommended alert rules can be implemented automatically or may require user approval, providing a method to alleviate the burden of manual rule creation while conserving network resources by providing relevant alert rules that offer meaningful network-related information.

[0226] Alarm rules, such as user-created alarm rules, can be stored in the alarm rule database 1220. To evaluate alarm rules, the alarm rule evaluator service 1214 reads the alarm rule and its associated metric name, accesses the metric queryer 1212 to receive the corresponding metric value from the metric time series database 1208 for the metric name in the rule, compares the metric value with the corresponding rule threshold, and can provide an alarm via the event reporter 1216 when a rule hit occurs. The evaluation results, including hits and misses, can be recorded in the rule evaluation history database (not specifically shown in Figure 12).

[0227] Figure 13 shows an exemplary sequence diagram of the performance monitoring system 1200 of Figure 12. The alarm rule evaluator service 1214 can read user-created alarm rules and their associated metric names. The alarm rule evaluator service discovers a set of relevant metrics related to the metrics of the user-created alarm rules and uses the discovered relevant metrics to create temporary relevant rules, as described in more detail below, and automatically generates temporary rules based on analysis. Temporary relevant rules can be stored, such as in an alarm rule database 1220. User-created alarm rules are evaluated using corresponding metrics from a metric time-series database 1208. If a rule miss occurs, the miss count is incremented; and if a rule hit occurs, the rule hit count is incremented, and an alarm can be generated. The evaluation count (total number of evaluations) of user-created alarm rules can also be tracked. Automatically generated temporary relevant rules can also be evaluated using corresponding metrics from the metric time-series database 1208. For each temporary relevant rule, corresponding rule attributes, such as one or both of temporary hit rate and temporary miss rate, and other possible rule relevance attributes, such as relevance, can be determined, as described below. The machine learning module 1222 can predict the weight of each automatically generated temporary rule. When the predicted weight of a temporary rule is greater than (or greater than or equal to) a predetermined acceptable value, the temporary rule can be recommended to the user; and when the predicted weight is less than (or less than or equal to) the acceptable value, the temporary rule can be discarded and not provided to the user as a recommended rule.

[0228] Figure 14 is a flowchart illustrating exemplary operations performed by the alarm rule evaluator service and machine learning module of the performance monitoring system of Figure 12 according to the technology described herein to create and evaluate temporary rules. At 1402, the alarm rule evaluator service 1214 can read user-created alarm rules from a user rule table (which may be an alarm rule database 1220). At 1404, the alarm rule evaluator service 1214 finds relevant metrics and creates one or more temporary relevant rules at 1406. At 1408, the temporary relevant rules may be stored, for example, in a temporary automatic rule table (which may be an alarm rule database 1220). At 1410, for each created temporary rule, the temporary rule can be read and evaluated to determine the hit count, miss count, and evaluation count; and at 1412, the hit rate and miss rate for the temporary rule can be calculated. At 1414, the relative hit rate and relative miss rate of the temporary rule can be calculated. At 1416, this information is stored, for example, in a temporary rule evaluation result table. Each of the temporary rules associated with the user-created rule can be evaluated according to steps 1410-1416. At 1418, if there are additional user-created rules, each of these rules can have one or more temporary related rules, which are then determined to be evaluated in the same manner as described above.

[0229] A machine learning module 1212 can be trained to predict alert rule weights for temporary rules to determine the most relevant temporary rules. For example, as shown in Figure 14, at 1420, the machine learning module can read historical evaluation data for each temporary rule; and at 1422, a rule recommender machine learning model can be trained. At 1424, the relative hit rate and relative miss rate of the temporary rules can be read from memory; and at 1426, the relative hit rate and relative miss rate of the temporary rules can be fed into the machine learning module, which uses one or more analysis data from the rule relevance attributes described below to determine the predicted weights for each corresponding temporary alert rule. At 1428, a prediction can be made about whether the rule is the most relevant. If so, the rule is added to the list of recommended rules. A prediction can also be made about whether the rule is the least relevant. If so, the rule is discarded and not recommended. Several sets of temporary rules associated with alert rules created by other users can be evaluated in the same manner.

[0230] When a user creates an alert rule, a set of temporary alert rules can be automatically created; and these rules can be evaluated at several times the time interval of the associated user-created rules. For example, if the evaluation interval for a user-created rule is 30 seconds, the evaluation interval for the temporary rules can be 30*T seconds, where T can be a predetermined value, a random variable, or an exponential variable, etc.

[0231] When evaluating user-created alert rules that occur concurrently, the machine learning model identifies the most relevant rule among temporary rules and transforms this temporary rule into a regular rule for future failure analysis.

[0232] A set of metrics can be identified as related metrics for each metric in a user-created rule, such as when the related metrics originate from the same service, component, or module as the metrics in the user-created rule, and / or share common metric tags. Tags can be used as keywords and can act as indicators of different types of metrics. A set of metric aggregation and comparison operators can be used to translate the identified related metrics into a temporary set of rules.

[0233] For example, if a user has created an alert rule to monitor time points where the average aggregate value of the metric "system_cpu_usage" is greater than 80%, the alert rule's metric can be derived from a system resource monitoring agent running on the machine and can be tagged "cpu". There may be several other processes running on the same machine, and these processes derive their metrics for CPU utilization and CPU-intensive operations (such as encryption / decryption counts). These other metrics are typically tagged "cpu", "cpu_intense_op", etc.

[0234] Suppose that some of the metrics exported by these processes are as follows.

[0235] 1. Metric #1: Metric Name: app_x_cpu_usage; Metric Label: label1 = cpu

[0236] 2. Metric #2: Metric name: app_x_encrypt_op_count, Metric tag:

[0237] label2 = cpu_intense_op

[0238] 3. Metric #3: Metric name: app_y_net_if_down_count, Metric label: label1 = net_err

[0239] 4. Metric #4: Metric name: app_y_cpu_usage, Metric label: label1 = cpu

[0240] The metric identification process identifies metrics 1, 2, and 4 as relevant metrics because these metrics either originate from the same machine or share common labels. Following this, a set of ad hoc rules can be created for user labels, using different combinations of aggregation and comparison functions. Thresholds for ad hoc alert rules can be calculated based on instrument metadata related to the metrics. For example, the instrument metadata for the metric "app_x_encrypt_op_count" would be an approximate percentage of CPU that each operation will consume.

[0241] In the example above, a temporary rule can be created as shown below.

[0242] Temporary rule-1: The average value of "app_x_cpu_usage" is greater than 80%.

[0243] Temporary rule-2: The average value of "app_x_cpu_usage" is less than 40%.

[0244] Temporary rule -3: The value of "app_x_encrypt_op_count" is greater than 500

[0245] Temporary rule -4: The value of "app_x_encrypt_op_count" is less than 100

[0246] Temporary rule -5: The average value of "app_y_cpu_usage" is greater than 80%.

[0247] Temporary rule -6: The average value of "app_y_cpu_usage" is less than 40%.

[0248] When a rule created by an associated user is matched, the newly created temporary rule can be evaluated. After each evaluation of a temporary rule, a set of evaluation attributes can be calculated and assigned to each temporary rule. These evaluation attributes can indicate the rule's effectiveness in generating alerts. The evaluation attributes are calculated as follows.

[0249] Temporary rule hit rate (PRHR): This indicates the frequency with which a temporary rule satisfies its rule condition (rule hit). The temporary rule hit rate can be calculated as follows:

[0250] PRHR = Hit Count / Number of Temporary Rule Evaluations

[0251] Temporary rule miss rate (PRMR): This indicates the frequency with which temporary rules fail to meet their conditions (rule miss). The temporary rule miss rate can be calculated as follows:

[0252] PRMR = Miss Count / Number of Temporary Rule Evaluations

[0253] This attribute plays a crucial role in learning which temporary rule is the least relevant and discarding this irrelevant rule in future evaluations.

[0254] Relative Temporary Rule Hit Rate (RPRHR): This indicates how frequently a temporary rule satisfies its rule conditions relative to an associated user-created rule. The relative temporary rule hit rate is calculated as follows:

[0255] RPRHR = Temporary rule hit rate / User-created rule hit rate

[0256] Relative Temporary Rule Miss Rate (RPRMR): This indicates the frequency with which a temporary rule fails to meet its rule conditions relative to the associated user-created rules. The Relative Temporary Rule Miss Rate is calculated as follows:

[0257] RPRMR = Temporary rule miss rate / User rule miss rate

[0258] This property plays a crucial role in learning the least relevant temporary rules and discarding them in future evaluations.

[0259] The evaluation attributes described above can be used to predict the weight of each temporary rule. The predicted weight indicates the relativity of the temporary rule to the user-created rule. A higher weight indicates that the rule is most relevant, and vice versa.

[0260] A simple linear regression machine learning model can be used to predict rule weights. For example, a weight is predicted relative to each relevant attribute, and the average of these weights is considered the rule weight.

[0261] Assumption:

[0262] The temporary rule hit rate is R1

[0263] R1 = (Number of hits / Number of rule evaluations)

[0264] The temporary rule miss rate is R2.

[0265] R2 = (Number of misses / Number of rule evaluations)

[0266] The relative temporary rule hit rate is R3

[0267] R3 = (Temporary rule hit rate / User rule hit rate)

[0268] The relative temporary rule miss rate is R4

[0269] R4 = (Temporary rule miss rate / User rule hit rate)

[0270] The rule weights are predicted using a linear regression formula and the single ratio calculated above.

[0271] W1 = a + b(R1)

[0272] W2 = a + b(1 / R2)

[0273] W3 = a + b(R3)

[0274] W4 = a + b(1 / R4)

[0275] Among them, a=(ΣWi)(ΣRi*2)-(ΣRi)(ΣWiRi) / n(ΣRi*2)-(ΣRi)2b=n(ΣWiRi)-(ΣRi)(ΣWi) / n(ΣRi*2)-(ΣRi*)2

[0276] The average of the predicted weights can be used as the rule weights and compared with a predetermined threshold to determine whether the proposed rule is relevant or irrelevant.

[0277]

[0278] The following includes exemplary pseudocode for implementing the above techniques:

[0279] pseudocode

[0280]

[0281]

[0282]

[0283]

[0284] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Individual features of the different examples 500, 600, 900, and 1200 of the system can be incorporated into a single performance monitoring system. Individual features described as modules, units, or components can be implemented together as integrated logic devices, or can be implemented individually as discrete but interoperable logic devices or other hardware devices. In some cases, individual features of the electronic circuitry can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0285] If implemented in hardware, this disclosure may relate to apparatuses, such as processors or integrated circuit devices (such as integrated circuit chips or chipsets). Optionally, or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable storage medium including instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable storage medium may store such instructions, which are executed by a processor.

[0286] Computer-readable media may form part of a computer program product, which may include packaging material. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable and programmable read-only memory (EEPROM), flash memory, magnetic data storage media, or optical data storage media. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0287] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may mean that the storage medium is not embodied as a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., in RAM or cache).

[0288] The code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent discrete or integrated logic circuits. Therefore, the term "processor," as used herein, can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure can be provided in software modules or hardware modules.

Claims

1. A performance monitoring method, comprising: The performance monitoring system collects telemetry data including network-related metrics of the computing device, wherein, for each metric, at each of a plurality of time points according to a first sampling interval, a metric value associated with a corresponding metric name of the metric is collected; the performance monitoring system uses the collected telemetry data to evaluate alarm rules, including evaluating the first alarm rule at a plurality of evaluation time points by comparing the metric value associated with the corresponding metric name of the first metric with a corresponding threshold of the first alarm rule of the alarm rule; generating metric attributes related to the alarm rule based on counts related to evaluating the first alarm rule; determining a predicted metric weight for the first metric based on the metric attributes; and determining a second sampling interval for the first metric based on the predicted metric weight.

2. The method according to claim 1, wherein, The second sampling interval is different from the first sampling interval; the method further includes the performance monitoring system using the second sampling interval to subsequently collect the corresponding metric value for the first metric.

3. The method according to claim 1, wherein, The performance monitoring system determines whether the first metric is a critical metric, and if the first metric is a critical metric, a predetermined minimum sampling interval is used for the second acquisition sampling interval.

4. The method according to any one of claims 1 to 3, wherein, The metric attributes include at least one of the following: metric access rate, metric threshold hit rate, metric variance, and metric group access rate.

5. The method according to any one of claims 1 to 3, wherein, Collecting the telemetry data includes: executing a drag-and-drop-based acquisition process to collect the telemetry data.

6. The method according to any one of claims 1 to 3, wherein, Determining the metric weights of the prediction includes executing a machine learning module to determine the metric weights of the prediction.

7. The method according to any one of claims 1 to 3, wherein, Determining the metric weights of the predictions includes performing regression analysis to determine the metric weights of the predictions.

8. The method according to any one of claims 1 to 3, further comprising: Based on the previous steps, update the current sampling interval for the first metric.

9. The method according to any one of claims 1 to 3, further comprising: Identify one or more related metrics associated with the first metric, wherein the metric attributes include a first metric access rate and a group metric access rate for the one or more related metrics.

10. The method according to any one of claims 1-3, wherein, The metric attribute is related to the access rate of the metric of the alert rule, and the count is a count of the number of times the metric is accessed within a specified time period.

11. The method according to any one of claims 1-3, wherein, The metric attribute is the access rate, which is the ratio of the access rate to the sampling rate.

12. The method according to any one of claims 1-3, wherein, The metric attribute is related to the hit rate of the metric of the alert rule, and wherein the count is a count of the number of times the metric value associated with the corresponding metric name of the first metric is greater than the corresponding threshold of the first alert rule.

13. The method according to any one of claims 1-3, further comprising instructing a metric derivator in the network to provide the first metric according to the second acquisition sampling interval.

14. A computer-readable storage medium, encoded with instructions for causing one or more programmable processors to perform the method according to any one of claims 1 to 13.

15. A performance monitoring system, comprising: Memory; and one or more memories, communicating with the memories, the one or more memories being configured to execute a collector and an alarm rule evaluator service, wherein the collector is configured to receive telemetry data via a metric derivative, the telemetry data including network-related metrics of the computing device, wherein for each metric, a metric value associated with a corresponding metric name is configured to be collected at each of a plurality of time points based on a corresponding first predetermined collection sampling interval; wherein the alarm rule evaluator service is configured to evaluate alarm rules using the collected telemetry data, wherein, in order to evaluate a first alarm rule of the alarm rules, the alarm rule evaluator service will compare the first metric with the first alarm rule. The collector compares the corresponding metric value associated with the corresponding metric name. At each of the multiple rule evaluation time points, the corresponding metric value is compared with the corresponding threshold of the first alarm rule to generate an alarm regarding whether the comparison condition of the first alarm rule is met, and the generated alarm is stored in the memory. The collector is configured to: access the generated alarm in the memory to calculate the metric attribute of the first metric associated with the alarm rule based on the count associated with the evaluation of the first alarm rule, determine the predicted metric weight using the metric attribute of the first metric, and determine a second collection sampling interval for the first metric based on the determined metric weight.

16. The performance monitoring system according to claim 15, wherein, The collector is configured to subsequently acquire the metric value of the first metric using the second acquisition sampling interval.

17. The performance monitoring system according to claim 15, wherein, The collector is configured to determine whether the first metric is a critical metric, and if the first metric is a critical metric, to use a predetermined minimum sampling interval for the second acquisition sampling interval.

18. The performance monitoring system according to any one of claims 15 to 17, wherein, The metric attributes include at least one of the following: access rate, variance, and threshold hit rate.

19. The performance monitoring system according to any one of claims 15 to 17, wherein, The collector also includes a machine learning module configured to determine the metric weights for the prediction.

20. The performance monitoring system according to any one of claims 15 to 17, wherein, The collector is configured to perform regression analysis to determine the weights of the predicted metrics.

21. The performance monitoring system according to any one of claims 15 to 17, wherein, The collector is configured to update the current sampling interval for the first metric on a forward basis.

22. The performance monitoring system according to any one of claims 15 to 17, wherein, The collector includes: a key metric discovery service for identifying one or more related metrics associated with the first metric, wherein the metric attributes are a first metric access rate and a group metric access rate for the one or more related metrics.

23. The performance monitoring system according to any one of claims 15 to 17, further comprising: The system includes a time-series metric database and a query history database, wherein the one or more processors are further configured to execute a metric queryer that communicates with the alarm rule evaluator service and the time-series metric database. The metric queryer is capable of receiving user queries related to metrics and storing the user queries in the query history database. The collector accesses the query history database to determine the metric weight and the second collection sampling interval.

Images