Proxy for aggregating telemetry stream data

Abstract

The present disclosure relates to a proxy for aggregating telemetry stream data. A method includes subscribing, by a proxy, to telemetry stream data from each of a plurality of network devices, and receiving, by the proxy, a plurality of streams of telemetry stream data from the plurality of network devices. Each of the plurality of streams corresponds to a different network device of the plurality of network devices. The method further includes aggregating, by the proxy, data from at least one of the plurality of streams of telemetry stream data received over a time period, and transmitting, by the proxy, the aggregated telemetry stream data to a network analyzer device at an end of the time period and / or when the data from the at least one stream exceeds a data threshold.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Application No. 17 / 456,105, filed November 22, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to computer networks, and more specifically, to the management of network devices. Background Technology

[0004] These network devices typically include mechanisms for configuring the device locally or remotely, such as management interfaces. By interacting with the management interface, clients can perform configuration tasks and execute operational commands to collect and view operational data from the managed device. For example, clients can configure the device's interface cards, adjust parameters of supported network protocols, specify physical components within the device, modify routing information maintained by the router, access software modules and other resources residing on the device, and perform other configuration tasks. Furthermore, clients can allow users to view current operating parameters, system logs, network connection-related information, network activity or other status information, and view and react to event information received from the device.

[0005] Network configuration services can be performed by multiple different devices, such as routers with service cards and / or dedicated service devices. These services include connectivity services such as Layer 3 Virtual Private Network (L3VPN), Virtual Private Local Area Network (VPLS), and peer-to-peer (P2P) services. Other services include network configuration services such as Dot1q VLAN services. Network Management Systems (NMS) and NMS devices (also known as controllers or controller devices) can support these services, allowing administrators to easily create and manage these advanced network configuration services.

[0006] Specifically, user configurations for a device can be termed "intents." Intent-based network systems allow administrators to describe the expected network / compute / storage state. User intents can be categorized as stateful intents (e.g., business policies) or stateless intents. Stateful intents can be parsed based on the current state of the network. Stateless intents can be a completely declarative way of describing the expected network / compute / storage state without considering the current network state.

[0007] Intents can be represented as an intent data model, which can be modeled using a unified graph model. The intent data model can be represented as a connection graph so that stateful intents can be implemented across business computing architectures. For example, the data model can be represented using data structures, such as a connection graph with vertices connected by edges and reference edges. Controller devices can model the intent data model as a unified graph model. In this way, stateful intents can be implemented on the intent data model. When intents are modeled using a unified graph model, extending new intent support can extend the graph model and compilation logic.

[0008] To configure a device to execute intents, users (such as network administrators) can write conversion programs to translate high-level configuration instructions (e.g., instructions based on the intent data model, which can be represented as a unified graph model) into low-level configuration instructions (e.g., instructions based on the device configuration model). As part of configuration service support, users / administrators can provide the intent data model and the mapping between the intent data model and the device configuration model.

[0009] Sensor devices can collect Internet Protocol (IP) network traffic as it enters and / or leaves an interface, and can output telemetry packets based on this IP network traffic. By analyzing the data provided by these telemetry packets, users (e.g., network administrators) can determine telemetry flow data such as the source and destination of network traffic, the service class of the network traffic, and the cause of network traffic congestion. User and / or controller devices can use intents to modify the configuration of network devices using the telemetry flow data. Summary of the Invention

[0010] Generally, this disclosure describes techniques for aggregating telemetry stream data generated by network devices. Some network analyzer devices (e.g., devices configured to aggregate packetized telemetry stream data) can receive telemetry stream data from each network device (e.g., a cluster (pod, container pool)). However, a system that sends telemetry stream data from each network device to the network analyzer may not scale well. For example, as the number of network devices increases, the processing burden on the network analyzer to handle the telemetry stream data increases. Furthermore, as the number of network devices increases, the storage burden on the network analyzer to store the telemetry stream data also increases.

[0011] Instead of relying on telemetry stream data from each network device or a relatively small sample from each network device, and as described herein, the system may include a software agent, or simply a "proxy," arranged to aggregate telemetry stream data from a set of network devices. For example, the agent may use a publish-subscribe model to subscribe to telemetry stream data from each network device in a set of network devices. In this example, the agent may receive a set of telemetry stream data streams from a set of network devices. The agent may aggregate telemetry stream data from at least one stream (e.g., one or two streams from a set of streams). In response to the end of a time period for receiving said at least one stream and when the telemetry stream data exceeds a data threshold, the agent may send the aggregated telemetry stream data to a network analyzer device. This reduces the number of samples in the telemetry stream data, which can reduce the processing burden on the network analyzer and / or reduce the bandwidth used to send the telemetry stream data. For example, the agent may reduce 10,000 samples of telemetry stream data from network devices to 1 to 10 samples of aggregated telemetry stream data. For cloud-based network analyzers located far from network devices (such as Software as a Service (SaaS) network analyzer devices), reducing the amount of telemetry stream data sent over the network to the network analyzer can be helpful. Furthermore, reducing the number of samples in the telemetry stream data can reduce the data storage burden on the network analyzer.

[0012] In one example, a method includes having an agent subscribe to telemetry stream data from each of a plurality of network devices. The method also includes having the agent receive streams of multiple telemetry stream data from the multiple network devices. Each of the multiple streams corresponds to a different one of the multiple network devices. The method further includes having the agent aggregate data from at least one of the multiple telemetry stream data streams received within a time period. The method also includes having the agent send the aggregated telemetry stream data to a network analyzer device at the end of the time period and / or when the data from at least one stream exceeds a data threshold.

[0013] In another example, the device includes processing circuitry and memory. The memory includes instructions that, when executed, cause the processing circuitry to subscribe to telemetry stream data from each of a plurality of network devices and to receive streams of multiple telemetry stream data from the multiple network devices. Each of the multiple streams corresponds to a different one of the multiple network devices. The instructions also cause the processing circuitry to aggregate data from at least one of the multiple telemetry stream data streams received within a time period. The instructions further cause the processing circuitry to send the aggregated telemetry stream data to a network analyzer device at the end of the time period and / or when the data from at least one stream exceeds a data threshold.

[0014] In one example, a non-transient computer-readable storage medium includes one or more instructions that cause processing circuitry to subscribe to telemetry stream data from each of a plurality of network devices and to receive streams of multiple telemetry stream data from the plurality of network devices. Each of the multiple streams corresponds to a different one of the plurality of network devices. The instructions also cause the processing circuitry to aggregate data from at least one of the multiple telemetry stream data streams received within a time period. The instructions further cause the processing circuitry to send the aggregated telemetry stream data to a network analyzer device at the end of the time period and / or when the data from at least one stream exceeds a data threshold.

[0015] Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, the drawings, and the claims. Attached Figure Description

[0016] Figure 1 This is a block diagram illustrating an example system in which the techniques described herein can be implemented.

[0017] Figure 2 This is a block diagram of an example computing device (e.g., a host) according to the techniques described in this disclosure, the example computing device including a cluster for implementing an agent configured to aggregate telemetry streaming data from network devices.

[0018] Figure 3 This is a block diagram of an example computing device operating as an instance of a network analyzer according to the techniques described in this disclosure.

[0019] Figure 4 This is a flowchart illustrating an example of aggregated telemetry stream data according to the techniques described in this disclosure.

[0020] Similar reference characters refer to similar elements throughout the diagram and description. Detailed Implementation

[0021] Figure 1 This is a block diagram illustrating an example system 100 in which the techniques described herein can be implemented. Generally, network devices 104A to 104N (collectively referred to as "network devices 104") can be arranged as a "cluster". Each cluster (collectively referred to as "cluster 102") in clusters 102A to 102N can represent a group of, for example, approximately 100 managed network devices. Network devices 104 can stream telemetry data to network analyzer device 108, for example, using a network flow collection protocol such as NetFlow or a packet sampling protocol such as sampled flow ("sFlow"). Network devices 104A can represent, for example, hundreds or thousands of devices.

[0022] Network analyzer device 108 can be configured to receive telemetry flow data from network device 104. In some examples, the telemetry flow data output by network device 104A can indicate a set of samples of traffic flows of packets exchanged by network device 104A. For example, the telemetry flow data can represent a set of packet samples flowing from a source IP address to a destination IP address. The telemetry flow data can be NetFlow compatible or can use other flow protocols, such as sflow, Juniper flow, or other flow protocols. Network analyzer device 108 can be configured to use the telemetry flow data to monitor traffic flows to provide visibility into network usage. For example, network analyzer device 108 can use the telemetry flow data to perform one or more of the following: troubleshooting network problems, controlling congestion, performing security and audit analysis, and performing route profiling.

[0023] Network device 104 may include, for example, routers, switches, gateways, bridges, hubs, servers, firewalls or other intrusion detection systems (IDS) or intrusion prevention systems (IDP), computing devices, computing terminals, printers, other network devices, or combinations thereof. Network device 104 may include one or more sensor devices configured to generate telemetry packets indicating telemetry stream data for multiple packets output by corresponding network elements. In some examples, each device in network device 104 may be associated with a corresponding set of sensor devices separate from network device 104 (e.g., a discrete circuit board or a discrete processing device). Although described herein as packets to be transmitted, transmitted, or otherwise supported, network device 104 may transmit data according to any other discrete data unit defined by any other protocol, such as a unit defined by the Asynchronous Transfer Mode (ATM) protocol or a datagram defined by the User Datagram Protocol (UDP). The communication link interconnecting network device 104 may be a physical link (e.g., an optical link, a copper link, etc.), a wireless link, or any combination thereof.

[0024] Network device 104 can be connected to network analyzer device 108 via public network 103 (e.g., the Internet). The public network may include, for example, one or more client computing devices. The public network can provide access to web servers, application servers, public databases, media servers, end-user devices, and other types of network resource devices and content. The public network can provide access to network analyzer device 108. Network analyzer device 108 may represent one or more computing devices (e.g., servers, computers, or the cloud).

[0025] Cluster 102 can be an example of a Kubernetes cluster and a virtual network endpoint. A cluster is one or more logically related containers ( Figure 1 (Not shown in the diagram) is a group of shared storage units for containers and options regarding how to run containers. In the case of instantiation for execution, a cluster can alternatively be referred to as a "cluster replica." For example, each container in cluster 102A could be an example of a virtual execution element. Containers in a cluster always reside on a single server, are co-scheduled, and run in a shared context. The shared context of a cluster can be a set of Linux namespaces, cgroups, and other isolation aspects. Within the cluster context, individual applications may apply further sub-isolation. Typically, containers in a cluster have a common IP address and port space and are able to detect each other via the localhost. Because they have a shared context, containers in a cluster also communicate with each other using inter-process communication (IPC). Examples of IPC include System V semaphores or POSIX shared memory. Generally, containers that are members of different clusters have different IP addresses and cannot communicate via IPC without a configuration that enables this feature. Containers that are members of different clusters typically communicate with each other via the cluster IP address.

[0026] In some cases, all clusters within all namespaces generated in a Kubernetes cluster may be able to communicate with each other, and network addresses for all clusters can be assigned from the cluster subnet specified by the orchestrator. When a user creates an isolated namespace for a cluster, the orchestrator and network controller can create new cluster virtual networks and new shared service virtual networks for the new isolated namespace. Clusters within the isolated namespaces generated in the Kubernetes cluster extract network addresses from the new cluster virtual networks, and the corresponding services of these clusters extract network addresses from the new service virtual networks.

[0027] Software proxies 106A through 106N (collectively referred to as "proxies 106"), and also simply "proxies 106" herein, can be configured to aggregate data from network device 104. Although Figure 1 The example illustrates agent 106A as implemented in a cluster, but in some examples, agent 106 may be implemented, for example, on a virtual machine and / or a bare metal server. Agent 106 may represent a compute node of cluster 102. For example, agent 106A may be implemented as a compute node of cluster 102A using processing circuitry, and agent 106N may be implemented as a compute node of cluster 102N using processing circuitry.

[0028] Agent 106 and network device 104 can be physically located away from network analyzer device 108. In some examples, agent 106 can be located at the network edge of public network 103. Agent 106 can send aggregated telemetry flow data to network analyzer device 108 via public network 103.

[0029] To reduce network traffic transmitted over public network 103, each cluster 102 may include a corresponding agent 106 configured to collect and aggregate telemetry flow data from various managed network devices 104 within cluster 102. For example, cluster 102A may include agent 106A, which may be configured to collect and aggregate telemetry flow data from network device 104A. That is, an agent for one cluster in cluster 102 (e.g., agent 106A) may be configured to subscribe to telemetry flow data from each network device in the cluster. The software agent may then aggregate this data and push the aggregated data to a cloud-based collector at regular time intervals (e.g., thirty-second intervals). Specifically, the aggregation may consist of multiple distinct flows, for example, originating from different network devices within the cluster. For example, the software agent may mathematically combine values ​​from different flows corresponding to common characteristics. In some examples, the software agent may be further configured to also perform analysis and / or aggregation on the aggregated data.

[0030] exist Figure 1 In the example, agent 106 subscribes to a stream of telemetry stream data 110 (also referred to herein as "telemetry stream data 110") from network device 104. For example, each network device 104A may send telemetry stream data 110 (e.g., sFlow data) as a sample once at each interval (e.g., a 1-second interval). Agent 106 may aggregate telemetry stream data 110. For example, agent 106A may aggregate telemetry stream data 110 from a single stream.

[0031] In some examples, agent 106A can aggregate telemetry stream data 110 from two or more streams associated with a unique 5-tuple stream. For example, agent 106A can aggregate telemetry stream data 110 that includes metadata indicating frequently used 5-tuples (e.g., source IP address, source TCP / UDP port, destination IP address, destination TCP / UDP port, and IP protocol) over a time period (e.g., 1 minute, 15 minutes, or 1 hour) to generate aggregated data 112A. For example, agent 106A can sum the transmission and / or reception rates of all samples (e.g., sFlow samples) from telemetry stream data 110, which includes metadata indicating common 5-tuples for that time period.

[0032] Agent 106 can send aggregated data 112A to 112N (aggregated data 112) to network analyzer device 108. For example, agent 106A can send aggregated data 112A indicating a sample (e.g., an sFlow sample) from telemetry flow data 110, which includes common 5-tuples received during that time period. Network analyzer device 108 can use the aggregated data 112 to analyze the characteristics of system 100, for example, to modify the configuration of network device 104 to implement additional or alternative network services, to determine whether to add, remove, or modify network devices or links between network devices, or other such decisions.

[0033] Aggregating data from two or more streams can help reduce the amount of data transmitted to network analyzer device 108, which can reduce bandwidth usage to network analyzer device 108 within public network 103. Furthermore, aggregating data from two or more streams can help reduce the amount of data that needs to be processed by network analyzer device 108, which can reduce the processing resource burden on network analyzer device 108. For example, agent 106 can reduce 10,000 samples streaming from network device 104 to 1 to 10 samples for network analyzer device 108 to process. Additionally, aggregating data from two or more streams can help reduce the amount of data that needs to be stored by network analyzer device 108, thereby reducing the required storage space. While data can be aggregated at network analyzer device 108 before storage, the techniques of this disclosure employ multiple distributed agents 106 to perform aggregation at the network edge (which may be cloud-based) where the streaming records are created, away from network analyzer device 108. This approach offers advantages over aggregating streaming data at network analyzer device 108 because less data is sent over public network 103 and less data needs to be processed by network analyzer device 108. Instead, the burden of aggregating streaming data is handled by multiple distributed agents 106 at the network edge.

[0034] Figure 2 This is a block diagram of an example computing device (e.g., a host) according to the technology described in this disclosure, which includes a cluster for implementing an agent configured to aggregate telemetry streaming data from network devices. Clusters 202A to 202B can be represented in more detail. Figure 1 Example instance of cluster 102. Figure 1 Agent 106 can be implemented as containers 229A to 229B. For example, one or more containers 229A can implement agent 250.

[0035] Figure 2 The computing device 200 can represent a real or virtual server, and can represent Figure 1This is an example instance of any server. In this example, computing device 200 includes a bus 242 that couples hardware components to the hardware environment of computing device 200. Bus 242 couples a network interface card (NIC) 230, a disk 246, and one or more microprocessors 210 (hereinafter referred to as "microprocessor 210"). NIC 230 may support SR-IOV. In some cases, a front-side bus may couple microprocessor 210 and memory device 244. In some examples, bus 242 may couple memory device 244, microprocessor 210, and NIC 230. Bus 242 may represent a Peripheral Component Interface (PCI) Fast (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 242 control DMA transfers between components coupled to bus 242.

[0036] Microprocessor 210 may include one or more processors, each processor including an independent execution unit to execute instructions of a compatible instruction set architecture, the instructions being stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC) or may be combined within one or more multi-core processors (or "multi-core" processors), each multi-core processor being implemented using a single IC (i.e., a chip multiprocessor).

[0037] Disk 246 represents a computer-readable storage medium, including volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information, such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile optical disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by the microprocessor 210.

[0038] Main memory 244 includes one or more computer-readable storage media, which may include random access memory (RAM) of various forms such as dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media, which can be used to carry or store required program code and program data in the form of instructions or data structures, and can be accessed by a computer. Main memory 244 provides a physical address space consisting of addressable memory locations.

[0039] Network interface card (NIC) 230 includes one or more interfaces 232 configured to exchange packets using links of the underlying physical network. Interface 232 may include a port interface card with one or more network ports. NIC 230 may also include on-card memory, for example, for storing packet data. Direct memory access transfers between NIC 230 and other devices coupled to bus 242 can read from / write to NIC memory.

[0040] Memory 244, NIC 230, storage disk 246, and microprocessor 210 provide an operating environment for the software stack including the operating system kernel 214 executing in kernel space. Kernel 214 may represent, for example, Linux, the Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel, available from Microsoft Corp. In some cases, the operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi provided by VMware, Windows Hyper-V provided by Microsoft, and other open-source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). The operating system including kernel 214 provides an execution environment for one or more processes in user space 245.

[0041] Kernel 214 includes physical driver 225 for use with network interface card 230. Network interface card 230 may also implement SR-IOV to enable sharing of physical network functions (I / O) between one or more virtual execution elements, such as containers 229A to 229B or one or more virtual machines. Figure 2 (Not shown in the image). Shared virtual devices, such as virtual functions, can provide dedicated resources, allowing each virtual execution element to access the dedicated resources of NIC 230, thus appearing as a dedicated NIC for each virtual execution element. Virtual functions can represent physical functions used with physical drive 225 as well as lightweight PCIe functions that share physical resources with other virtual functions. For an SR-IOV-enabled NIC 230, the NIC 230 may have thousands of available virtual functions according to the SR-IOV standard, but for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0042] Clusters 202A to 202B can be represented in more detail. Figure 1 An example instance of cluster 102. Cluster 202A includes one or more containers 229A, and cluster 202B includes one or more containers 229B. Container platform 204 can be represented in more detail. Figure 1 An example instance of a container platform. Container platform 204 includes container engine 208, orchestration agent 209, service agent 211, and network module 206. Network module 206 can represent... Figure 1 Example instance of the network module.

[0043] Container engine 208 includes code executable by microprocessor 210. Container engine 208 can be one or more computer processes. Container engine 208 runs containerized applications in the form of containers 229A to 229B. Container engine 208 can represent Dockert, rkt, or other container engines used to manage containers. Typically, container engine 208 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating containers. A container is an executable instance of an image. Based on instructions from orchestration agent 209, container engine 208 can obtain images and instantiate them into executable containers 229A to 229B in cluster 202A to 202B.

[0044] Service agent 211 includes code executable by microprocessor 210. Service agent 211 can be one or more computer processes. Service agent 211 monitors the addition and removal of services and endpoint objects, and it maintains the network configuration of computing device 200 to ensure communication between the cluster and containers, for example, using services. Service agent 211 can also manage IP tables to capture traffic to the virtual IP address and port of the service and redirect the traffic to the proxy port of the cluster supported by the proxy. Service agent 211 can represent kube-proxy for a small node of a Kubernetes cluster. In some examples, container platform 204 does not include service agent 211 or service agent 211 is disabled to support the configuration of virtual router 220 and cluster 202 by network module 206.

[0045] The orchestration agent 209 includes code executable by the microprocessor 210. The orchestration agent 209 can be one or more computer processes. The orchestration agent 209 can represent a kubelet of a small node in a Kubernetes cluster. The orchestration agent 209 is an orchestrator (e.g., Figure 1The orchestration agent 209 acts as an agent for the orchestration engine 208, receiving container specification data for the container and ensuring that the container is executed by the compute device 200. The container specification data can be a manifest file sent from the orchestrator to the orchestration agent 209 or received indirectly via a command-line interface, HTTP endpoint, or HTTP server. The container specification data can also be a cluster specification (e.g., a PodSpec – a YAML (yet another markup language) or JSON object describing the cluster) for use with container 229 in cluster 202. Based on the container specification data, the orchestration agent 209 instructs the container engine 208 to acquire and instantiate the container image of container 229 for the compute device 200 to execute.

[0046] Orchestration agent 209 instantiates network module 206 to configure one or more virtual network interfaces for each cluster 202. For example, orchestration agent 209 receives container specification data for cluster 202A and instructs container engine 208 to create cluster 202A with container 229A based on the container specification data of cluster 202A. Orchestration agent 209 also invokes network module 206 to configure virtual network interface 212 for cluster 202A corresponding to the virtual network of VRF 222A. In this example, clusters 202A and 202B are virtual network endpoints corresponding to VRF 222A and VRF 222B, respectively.

[0047] Container 229A can achieve Figure 1 One or more software agents 106. For example, the software agent of container 229A can be configured to aggregate from... Figure 1 Data from network device 104. Container 229A is used for illustrative purposes only in the following examples.

[0048] To reduce network traffic transmitted over public network 103, cluster 202A may include an agent 250, implemented by container 229A and configured to collect and aggregate telemetry flow data from various managed network devices of cluster 202A. For example, agent 250 may be configured to collect and aggregate telemetry flow data from network devices. Subscriber 252 may be configured to subscribe to telemetry flow data from each network device in cluster 202A. For example, each network device in cluster 202A may be configured to send (e.g., using sFlow) telemetry data to an IP address corresponding to (e.g., matching) agent 250. Thus, each network device in cluster 202A may be configured to send telemetry data generated by its respective network device to agent 250, which is treated as a collector device (e.g., sFlow compliant).

[0049] Agent 250 can store telemetry stream data from each network device in cluster 202A at data store 254. Aggregator 256 can aggregate the telemetry stream data stored at data store 254. Sender 260 (e.g., a gRPC sender) can send the aggregated data to a cloud-based collector (e.g., a gRPC receiver) at configurable periodic intervals (e.g., 30-second intervals). Specifically, aggregation can be the aggregation of multiple different streams, for example, originating from different network devices within or outside the cluster. For example, aggregator 256 can aggregate telemetry stream data from a single stream across multiple time interval samples within a given time period. In some examples, aggregator 256 can mathematically combine values ​​from different streams corresponding to common characteristics between the streams. In some examples, container 229A may optionally include analyzer 258, which is configured to also perform analysis and / or synthesis on the aggregated data.

[0050] At the end of a time period and / or when data from at least one stream (e.g., one or more streams) exceeds a data threshold, transmitter 260 may send aggregated telemetry stream data to network analyzer device 108. For example, aggregator 256 may aggregate data from at least one of multiple streams of telemetry stream data received within a time period. However, if transmitter 260 determines that data from at least one stream is greater than the data threshold before the end of the time period, transmitter 260 may send the aggregated telemetry stream data in response to determining that data from at least one stream is greater than the data threshold. If, at the end of the time period, transmitter 260 determines that data from at least one stream is less than the data threshold, transmitter 260 may send the aggregated telemetry stream data in response to the end of the time period.

[0051] The agent implemented by container 229A can output aggregated telemetry flow data that includes metadata. For example, the agent can output aggregated telemetry flow data that includes one or more of a tenant identifier and a location identifier associated with the telemetry flow data received from network device 104. For example, agent 250 can establish a gRPC connection to network analyzer device 108 (e.g., a SaaS controller). In this example, the gRPC connection from agent 250 to network analyzer device 108 can be per agent, and agent 250 can use the gRPC connection to identify the location identifier and / or tenant identifier.

[0052] In some examples, agent 250 can be configured to select between aggregated stream mode and debug mode. For example, agent 250 can receive instructions from network analyzer device 108 instructing the agent to operate in aggregated stream mode. In this example, agent 250 can collect and aggregate telemetry stream data from various managed network devices in cluster 202A and “summarize” the aggregated telemetry stream data to network analyzer device 108. For example, agent 250 can perform one or more of the following: subscribing to telemetry stream data, receiving telemetry stream data streams, aggregating data from at least one stream of telemetry stream data, and sending the aggregated stream data in response to receiving instructions to operate in aggregated stream mode.

[0053] Conversely, in response to receiving an instruction from network analyzer device 108 instructing agent 250 to operate in debug mode, agent 250 (e.g., subscriber 252) can collect telemetry stream data from various managed network devices in cluster 202A and forward all data without aggregation and latency. For example, agent 250 can receive telemetry packets from network device 104A and can forward telemetry packets from the network device without first collecting telemetry packets within a time period. For example, before the end of a configured time period and before data exceeds a data threshold, in response to receiving an instruction to operate in debug mode, agent 250 can send the telemetry stream data subsequently received from network device 104 to network analyzer device 108 without aggregation. The time period, the configured time period, and / or the data threshold can be configured, for example, by an administrator.

[0054] In some examples, aggregator 256 can be configured to correlate and / or enrich “mixed” types of telemetry data. For example, analyzer 258 can correlate data from different vendor devices and / or supporting different protocols (e.g., routers using NetFlow, routers using JFlow, or bare-metal servers using another streaming protocol). For example, when analyzer 258 receives telemetry streaming data from network devices, the telemetry streaming data can be in different formats. In this case, analyzer 258 can normalize the telemetry streaming data in different formats into a common format. For example, analyzer 258 can convert telemetry streaming data in different formats from different types of sources (e.g., sFlow, JFlow, etc.) into telemetry streaming data in a common format for storage.

[0055] For example, aggregator 256 can enrich mixed-type telemetry data with the same identifier. For example, aggregator 256 can use identifiers to associate a first stream compatible with two or more streams of a first protocol with a second stream compatible with two or more streams of a second protocol. The first protocol and the second protocol can be different.

[0056] Sending aggregated streaming data over public network 103 by transmitter 260 may be more secure than sending the raw telemetry streaming data itself, because if intercepted, the data being sent may have lower visibility into the actual data of the specific network device. For example, transmitter 260 can provide a reliable and secure return route for information flows from a customer's premises to the cloud. For example, network device 104A can send streaming data via unencrypted UDP-based packets, and transmitter 260 can use gRPC aggregated streaming packets, which may be more suitable (e.g., more secure and / or more reliable) for transmitting data to the cloud (e.g., network analyzer device 108) than unencrypted UDP-based packets. Analyzer 258 can generate snapshots of the data (e.g., a 15-minute portion of the telemetry streaming data) for the user to help ensure that the telemetry streaming data provided to network analyzer device 108 is accurate.

[0057] Figure 3 This is a block diagram of an example computing device operating as an instance of a network analyzer according to the techniques described in this disclosure. Computing device 300 is an example instance of a controller for virtualizing computing infrastructure. Figure 3 The computing device 300 can represent one or more real or virtual servers.

[0058] Although the telemetry stream data receiver 320, telemetry stream data enrichment manager 322, and telemetry stream data storage 324 are described and illustrated as being executed by a single computing device 300, they can be distributed among multiple computing devices 300 constituting a computing system or hardware / server cluster. In other words, each of the multiple computing devices 300 can provide a hardware operating environment for one or more instances of any one or more of the telemetry stream data receiver 320, telemetry stream data enrichment manager 322, and telemetry stream data storage 324.

[0059] The telemetry stream data receiver 320 can be configured to receive data from... Figure 1 Agent 106 receives aggregated telemetry stream data. For example, telemetry stream data receiver 320 may include a gRPC (gRPC Remote Procedure Call) edge receiver configured to receive aggregated telemetry stream data from the gRPC edge transmitter of agent 106. In some examples, telemetry stream data receiver 320 may be configured to receive telemetry stream data directly from network device 104. For example, telemetry stream data receiver 320 may be configured to receive data from... Figure 1Agent 106 receives one or more telemetry packets indicating aggregated telemetry flow data. The telemetry packets may be compatible with, for example, NetFlow, OpenConfig, Juniper Telemetry Interface (JTI) Native, netconf, Simple Network Management Protocol (SNMP), syslog, and sFlow. The aggregated telemetry flow data may have associated transmission and / or reception rates. The aggregated telemetry flow data may include metadata indicating tuples (e.g., 5-tuples or 3-tuples) associated with the aggregated telemetry flow data. In some examples, the metadata included in the aggregated telemetry flow data may include one or more of a tenant identifier and a location identifier.

[0060] The telemetry stream data enrichment manager 322 can be configured to convert aggregated telemetry stream data into the system format of the network analyzer 308. For example, the telemetry stream data enrichment manager 322 can convert sflow UDP datagrams into protocol buffers (protobuf) for Kafka processing. The telemetry stream data enrichment manager 322 can convert protobuf into SQL format for storage at the telemetry stream data store 324. For example, the telemetry stream data enrichment manager 322 can be configured to convert device identifiers indicated in the aggregated telemetry stream data into network device identifiers. The telemetry stream data enrichment manager 322 can be configured to associate aggregated telemetry stream data with servers (e.g., VMware or hypervisor). The telemetry stream data enrichment manager 322 can maintain aggregated telemetry stream data for one or more network devices, server sets, instance sets, network device-to-bare metal server (BMS) connections, or item lists. The telemetry stream data enrichment manager 322 can be configured to store the converted aggregated telemetry stream data in the telemetry stream data store 324.

[0061] According to the technology disclosed herein, agent 106 can aggregate data from at least one (e.g., one or more) of multiple streams of telemetry stream data received within a time period. Aggregating data from two or more streams can help reduce the amount of data transmitted to network analyzer 308, which can reduce the bandwidth usage of the network sending the streams to network analyzer 308. Furthermore, aggregating data from two or more streams can help reduce the amount of data to be processed by network analyzer 308, which can reduce the processing burden on network analyzer 308. For example, agent 106 can reduce 10,000 samples streaming from network device 104 to 1 to 10 samples for network analyzer 308 to process. Additionally, aggregating data from two or more streams can help reduce the amount of data to be stored by telemetry stream data storage 324 of network analyzer 308.

[0062] In this example, computing device 300 includes a bus 342 that couples hardware components to the hardware environment of computing device 300. Bus 342 couples to a network interface card (NIC) 330, a storage disk 346, and one or more microprocessors 310 (hereinafter referred to as "microprocessor 310"). In some cases, a front-side bus may couple to microprocessor 310 and memory device 344. In some examples, bus 342 may couple to memory device 344, microprocessor 310, and NIC 330. Bus 342 may represent a Peripheral Component Interface (PCI) Fast (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 342. In some examples, components coupled to bus 342 control DMA transfers between components coupled to bus 342.

[0063] The microprocessor 310 may include one or more processors, each processor including an independent execution unit to execute instructions of a compatible instruction set architecture, the instructions being stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC) or may be combined within one or more multi-core processors (or "multi-core" processors), each multi-core processor being implemented using a single IC (i.e., a chip multiprocessor).

[0064] Disk 346 represents a computer-readable storage medium, including volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information, such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by the microprocessor 310.

[0065] Main memory 344 includes one or more computer-readable storage media, which may include random access memory (RAM) of various forms such as dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media, which can be used to carry or store required program code and program data in the form of instructions or data structures, and can be accessed by a computer. Main memory 344 provides a physical address space consisting of addressable memory locations.

[0066] Network interface card (NIC) 330 includes one or more interfaces 332 configured to exchange packets using links of the underlying physical network. Interface 332 may include a port interface card with one or more network ports. NIC 330 may also include on-card memory, for example, for storing packet data. Direct memory access transfers between NIC 330 and other devices coupled to bus 342 can read from / write to NIC memory.

[0067] Memory 344, NIC 330, storage disk 346, and microprocessor 310 provide an operating environment for the software stack including the operating system kernel 314, which executes in kernel space. Kernel 314 may represent, for example, Linux, the Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel, available from Microsoft Corp. In some cases, the operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi provided by VMware, Windows Hyper-V provided by Microsoft, and other open-source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). The operating system including kernel 314 provides an execution environment for one or more processes in user space 345. Kernel 314 includes physical drive 325 for use with network interface card 230.

[0068] Computing device 300 can be coupled to a physical network switching infrastructure, which includes an overlay network and software or "virtual" routers that extend the switching infrastructure from physical switches to physical servers coupled to the switching infrastructure, such as... Figure 2 The virtual router 220. The computing device 300 can use one or more dedicated virtual networks to configure small nodes of the cluster.

[0069] Figures 1 to 3Various components, functional units, and / or modules shown and / or shown or described elsewhere in this disclosure may perform the described operations using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and / or executing on one or more computing devices. For example, a computing device may execute one or more such modules using multiple processors or multiple devices. A computing device may execute one or more such modules as a virtual machine executing on the underlying hardware. One or more such modules may execute as one or more services of an operating system or computing platform. One or more such modules may execute as one or more executable programs at the application layer of a computing platform. In other examples, the functionality provided by the modules may be implemented by dedicated hardware devices. Although certain modules, data stores, components, programs, executable files, data items, functional units, and / or other items included within one or more storage devices may be shown individually, one or more of these items may be combined and operate as a single module, component, program, executable file, data item, or functional unit. For example, one or more modules or data stores may be combined or partially combined so that they operate as a single module or provide functionality. Furthermore, one or more modules may operate in combination with each other, for example, one module may act as a service or extension of another module. Furthermore, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may include multiple components, subcomponents, modules, submodules, data stores, and / or other components or modules or data stores not shown. Additionally, each module, data store, component, program, executable file, data item, functional unit, or other item shown in the storage device may be implemented in various ways. For example, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may be implemented as part of an operating system executing on a computing device.

[0070] Figure 4 This is a flowchart illustrating an example of the aggregation of telemetry stream data according to the techniques described in this disclosure. Figure 4 For reference only Figures 1 to 3 The description is for illustrative purposes only.

[0071] exist Figure 4 In the example, agent 106 subscribes to telemetry stream data 110 (e.g., sFlow data) from network device 104 (402). For example, agent 106 may subscribe to receive a stream of telemetry stream data 110. The telemetry data stream may include telemetry packets compatible with NetFlow, OpenConfig, Juniper Telemetry Interface (JTI) natively, netconf, SNMP, syslog, or sFlow.

[0072] Agent 106 receives multiple streams (404) of telemetry stream data from multiple network devices 104. Each of the multiple streams may correspond to a different one of the multiple network devices 104. For example, agent 106A may receive a first stream of telemetry stream data from a first network device 104A, a second stream of telemetry stream data from a second network device 104B, and so on. For example, each network device 104A may send telemetry stream data 110 (e.g., sFlow packets) to include a set of samples once at each interval (e.g., a 1-second interval).

[0073] Agent 106 aggregates telemetry stream data 110 (406) based on at least one of multiple streams (e.g., one or more streams) received within a time period. This time period can be, for example, configured by an administrator. For instance, agent 106 can aggregate telemetry stream data from a single stream. In some examples, agent 106 can aggregate telemetry stream data from two or more streams based on a 5-tuple (source IP address, source TCP / UDP port, destination IP address, destination TCP / UDP port, and IP protocol) or a 3-tuple (source IP address, destination IP address, and IP protocol). For example, agent 106A can sum the transmission and / or reception rates from all samples of telemetry stream data 110, which includes metadata indicating a common 5-tuple for that time period.

[0074] For example, agent 106A can determine that the second flow of telemetry flow data includes a destination IP address that matches the destination IP address of the first flow of telemetry flow data and a source IP address that matches the source IP address of the first flow. In this example, agent 106A can aggregate the first and second flows based on determining that the second flow includes a destination IP address that matches the destination IP address of the first flow and a source IP address that matches the source IP address of the first flow. Agent 106A can determine that the second flow of telemetry flow data is associated with an IP protocol that matches the IP protocol of the first flow. In this example, agent 106A can further aggregate the first and second flows based on determining that the second flow of telemetry flow data is associated with an IP protocol that matches the IP protocol of the first flow.

[0075] In some examples, agent 106A can determine that a second stream of telemetry stream data is assigned a destination port that matches the destination port of the first stream, and determine that a second stream of telemetry stream data is assigned a source port that matches the source port of the first stream. In this example, agent 106A can further aggregate the first stream and the second stream based on the determination that the second stream of telemetry stream data is assigned a destination port that matches the destination port of the first stream, and based on the determination that the second stream of telemetry stream data is assigned a source port that matches the source port of the first stream.

[0076] Agent 106 sends aggregated telemetry flow data to network analyzer device 108 (408) at the end of a time period and / or when data from at least one flow (e.g., one or more flows) exceeds a data threshold. For example, at the end of a time period, agent 106A may determine that data from at least one flow is less than a data threshold. The data threshold may be configured, for example, by an administrator. In this example, agent 106A may send aggregated telemetry flow data in response to the end of a time period. For example, agent 106A may generate an sFlow packet including each corresponding sample set for each sFlow packet received from network device 104. In some examples, agent 106A may determine that data from at least one flow is greater than a data threshold before the end of a time period. In this example, agent 106A may send aggregated telemetry flow data in response to determining that data from at least one flow is greater than a data threshold.

[0077] Network analyzer device 108 can use aggregated data 112 to analyze the characteristics of system 100, such as modifying the configuration of network device 104, implementing additional or alternative network services, determining whether to add, remove or modify network devices or links between network devices, or other such decisions.

[0078] In some examples, agent 106A can be configured to select between aggregated flow mode and debug mode. For example, agent 106A can receive instructions from network analyzer device 108 instructing the agent to operate in aggregated flow mode. In this example, agent 106A can collect and aggregate telemetry flow data from network device 104A and "summarize" the aggregated telemetry flow data to network analyzer device 108. However, in response to receiving instructions from network analyzer device 108 instructing agent 106A to operate in debug mode, agent 106A can collect telemetry flow data from network device 104A and forward the data without aggregation and without delay. For example, agent 106A can receive telemetry packets from network device 104A and forward the telemetry packets from the network device to network analyzer device 108 without collecting telemetry packets within a time period.

[0079] The techniques described in this disclosure can be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, aspects of the techniques can be implemented within one or more processors, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuits, and any combination of these components. The terms "processor" or "processing circuitry" generally refer to any of the aforementioned logic circuits, alone or in combination with other logic circuits, or any other equivalent circuitry. Control units, including hardware, can also perform one or more of the techniques of this disclosure.

[0080] Such hardware, software, and firmware can be implemented within the same device or in separate devices to support the various operations and functions described in this disclosure. Furthermore, any of the described units, modules, or components can be implemented together or separately as discrete but interoperable logical devices. Describing different features as modules or units is intended to highlight different functional aspects and does not necessarily mean that these modules or units must be implemented by separate hardware or software components. Rather, the functionality associated with one or more modules or units can be performed by separate hardware or software components, or integrated within common or separate hardware or software components.

[0081] The techniques described in this disclosure can also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium containing instructions. Instructions embedded or encoded in a computer-readable medium can cause a programmable processor or other processor to perform the method, for example, when the instructions are executed. Computer-readable media can include non-transitory computer-readable storage media and transient communication media. Tangible and non-transitory computer-readable storage media can include random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory, hard disk, CD-ROM, floppy disk, magnetic tape cassette, magnetic media, optical media, or other computer-readable storage media. The term "computer-readable storage medium" refers to a physical storage medium, not a signal, carrier, or other transient medium.

Claims

1. A method comprising: The agent subscribes to telemetry stream data from each of multiple network devices. The agent receives multiple streams of telemetry data from the plurality of network devices, wherein each of the multiple streams corresponds to a different network device among the plurality of network devices; The agent aggregates data from at least one of the plurality of streams of telemetry stream data received within a time period; Before the end of the time period, the agent determines that the data of the at least one stream is greater than a data threshold; and In response to determining that the data of the at least one stream is greater than the data threshold, the agent sends the aggregated telemetry stream data to the network analyzer device.

2. The method according to claim 1, wherein, Aggregating the data from the at least one stream includes aggregating the telemetry stream data from a single stream.

3. The method according to claim 1, wherein, Aggregating the data from the at least one stream includes aggregating the telemetry stream data from two or more streams associated with a unique 5-tuple stream.

4. The method according to claim 1, wherein, The agent and the network device are physically located away from the network analyzer device and at the network edge of a public network, and the aggregation of telemetry flow data to the network analyzer device includes sending the aggregation of telemetry flow data to the network analyzer device via the public network.

5. The method of claim 1, further comprising performing the steps of claim 1 in response to receiving an instruction to operate in aggregate stream mode.

6. The method of claim 1, further comprising: Before the configured time period ends and before the data exceeds the data threshold, in response to receiving an instruction to operate in debug mode, the agent sends the stream of telemetry flow data subsequently received from the plurality of network devices to the network analyzer device without aggregation.

7. The method of claim 1, further comprising: At the end of the time period, the agent determines that the data from the at least one stream is less than the data threshold, wherein sending the aggregated telemetry stream data is in response to the end of the time period.

8. The method according to any one of claims 1, 3 to 7, further comprising: The second stream of the telemetry data determined by the agent includes: a destination IP address matching the destination IP address of the first stream of the telemetry data, and a source IP address matching the source IP address of the first stream; and The aggregation of data from the at least one stream includes: aggregating the first stream and the second stream based on determining that the second stream includes a destination IP address that matches the destination IP address of the first stream and a source IP address that matches the source IP address of the first stream.

9. The method of claim 8, further comprising: The agent determines that the second stream of the telemetry stream data is associated with an IP protocol that matches the IP protocol of the first stream; as well as The aggregation of data from the at least one stream includes: further aggregating the first stream and the second stream based on determining that the second stream of the telemetry stream data is associated with an IP protocol that matches the IP protocol of the first stream.

10. The method of claim 8, further comprising: The agent determines that the second stream of the telemetry stream data is assigned a destination port that matches the destination port of the first stream; The agent determines that the second stream of the telemetry stream data is assigned a source port that matches the source port of the first stream; as well as The aggregation of the data from the at least one stream includes: further aggregating the first stream and the second stream based on determining that the second stream of the telemetry stream data is assigned a destination port that matches the destination port of the first stream, and based on determining that the second stream of the telemetry stream data is assigned a source port that matches the source port of the first stream.

11. The method according to any one of claims 1 to 7, wherein, Sending the aggregated telemetry stream data includes sending telemetry packets compatible with protocols selected from the group consisting of: NetFlow, OpenConfig, Juniper Telemetry Interface JTI Local, netconf, Simple Network Management Protocol SNMP, syslog, and sFlow.

12. The method according to any one of claims 1 to 7, wherein, Aggregating the data from at least one of the plurality of streams of the telemetry stream data includes: associating a first stream of the at least one stream that is compatible with a first protocol with a second stream of the at least one stream that is compatible with a second protocol using an identifier.

13. An apparatus comprising: Processing circuitry; as well as The memory includes instructions that, when executed, cause the processing circuitry to: Subscribe to telemetry stream data from each of multiple network devices; Multiple streams of telemetry data are received from the plurality of network devices, wherein each of the plurality of streams corresponds to a different network device among the plurality of network devices; Aggregate data from at least one of the plurality of streams of telemetry stream data received within a time period; Before the end of the time period, it is determined that the data in at least one stream is greater than a data threshold; and In response to determining that the data of the at least one stream is greater than the data threshold, the aggregated telemetry stream data is sent to the network analyzer device.

14. The device according to claim 13, wherein, The device and the network device are physically located away from the network analyzer device and at the network edge of a public network, and wherein, in order to send the aggregated telemetry flow data to the network analyzer device, the instruction causes the processing circuitry to send the aggregated telemetry flow data to the network analyzer device through the public network.

15. The device according to claim 13, wherein, The instruction further causes the processing circuit to perform the steps of claim 14 in response to receiving an instruction to operate in aggregate stream mode.

16. The device according to claim 13, wherein, The instruction also causes the processing circuitry, in response to receiving an instruction to operate in debug mode, to send a stream of telemetry data subsequently received from the plurality of network devices to the network analyzer device without aggregation, before the end of the configured time period and before the data exceeds the data threshold.

17. The device according to any one of claims 13 to 15, wherein, The instruction further causes the processing circuit to: determine at the end of the time period that the data from the at least one stream is less than the data threshold, wherein the instruction causes the processing circuit to send the aggregated telemetry stream data in response to the end of the time period.

18. A computer-readable storage medium encoded with instructions that, when executed, cause one or more programmable processors to perform the method of any one of claims 1 to 12.

Images