Managing anomalous network telemetry message flows

Abstract

Embodiments of the present disclosure relate to managing anomalous network telemetry message flows. A process includes monitoring a plurality of message streams sent by a respective network device of a plurality of network devices. The plurality of message flows are associated with reporting network telemetry metrics to a network management system. The process includes determining that a given message stream of the plurality of message streams exhibits an unexpected behavior. The process includes, in response to determining that a given message stream exhibits an unexpected behavior, determining an aggregated available bandwidth for message streams of the plurality of message streams that each exhibit an expected behavior. The process includes, in response to determining that the given message stream exhibits unexpected behavior, modifying a bandwidth of the given message stream based on aggregated available bandwidths.

Description

Background Art

[0001] Enterprises can use cloud-based network management systems to collect, log, visualize, and analyze network telemetry metrics. Analysis of network telemetry metrics can be beneficial for a variety of purposes, such as identifying network configuration issues, identifying network performance issues, detecting network device failures, identifying security vulnerabilities, and detecting security attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a block diagram of a computer network with a central network management system (NMS) cluster that manages abnormal network telemetry message flows according to an example implementation.

[0003] Figure 2 is a flow diagram depicting a process for detecting anomalous network telemetry message flows according to an example implementation.

[0004] Figure 3 is a flow diagram depicting a process for adaptively managing network telemetry subscriptions to manage abnormal network telemetry message flows according to an example implementation.

[0005] Figure 4 is a flow chart depicting a process for interpolating missing information content in a network telemetry message stream associated with an adaptive subscription, according to an example implementation.

[0006] Figure 5 is a flow diagram depicting a process for managing a network telemetry message flow that exhibits unexpected behavior according to an example implementation.

[0007] Figure 6 is an illustration of machine readable instructions that, when executed by a machine, cause the machine to manage exception message flows according to an example implementation.

[0008] Figure 7 is a block diagram of a system for managing a message flow having a rate that is an outlier, according to an example implementation. DETAILED DESCRIPTION

[0009] A network management system (NMS) cluster may include a central NMS component (e.g., a cloud-based component) that manages network devices (also referred to herein as "managed network devices") of a particular network device deployment. In an example, a network device deployment may include a collection of network devices associated with a particular geographic location. For example, a particular NMS cluster may include a network device deployment that includes network devices in one or more branch networks located on the same campus, in the same data center, or in the same country.

[0010] The central NMS component may include a central server, and the managed network devices may generate and send messages (referred to herein as "network telemetry messages") to the central server. The messages may contain information about network telemetry metrics, and a particular network device may publish information about a particular network metric in a sequence of messages referred to as a "network telemetry message stream." Generally, a network telemetry message stream may provide information about network telemetry metrics from which insights about the state of the network may be determined directly or indirectly. Here, "network state" refers to one or more characteristics of a network device (e.g., a network switch, router, or gateway) that forms a network structure, a portion of a network structure (e.g., a particular subnet), or a client device connected to the network structure.

[0011] Abnormal conditions may cause one or more network devices to generate abnormally large network telemetry message rates, which may overwhelm the telemetry communication network. According to an example implementation, the central server of the NMS cluster detects unexpected or abnormal network telemetry message flows and takes message throttling measures to limit the bandwidth of the corresponding abnormal message flows. In the example, the central server can replace the subscription based on the push model of the abnormal network telemetry message flow with an adaptive subscription that constrains the bandwidth of the message flow (compared with the basic subscription). In the example, the adaptive subscription can increase the reminder interval (for periodic subscriptions) or limit the time when the network device can provide messages (for subscriptions when changes). In another example, for abnormal network telemetry flow message flows controlled by polling, the central server can increase the polling interval. According to the example implementation, the central server can determine that the message flow is abnormal based on the message rate of a specific network telemetry message flow. According to the example implementation, in order to determine whether the message flow is abnormal, the central server can compare the message rate of a specific network telemetry message flow with one or more local message rate baselines and / or one or more global message rate baselines.

[0012] In an example, the network telemetry metric may indicate the occurrence of a particular event (e.g., a network disassociation or a network association). In another example, the network telemetry metric may be a particular statistic (e.g., the ingress bandwidth of a network device associated with a particular virtual local area network (VLAN) or the egress bandwidth associated with a particular class of network traffic). In another example, the network telemetry metric may be an event log record. In another example, the network telemetry metric may be a trace.

[0013] A managed network device may publish information about a certain network telemetry metric based on a subscription. A subscription corresponds to a push-based telemetry reporting model because the network device initiates the publication of the network telemetry metric information, as compared to polling the network device for network telemetry metric information (based on a pull-based network telemetry reporting model). More specifically, a network device may publish information about a specific network telemetry metric based on the parameters of a subscription. These parameters correspond to details about the covered network telemetry metric and how the publication occurs. In an example, a subscription may identify a specific network telemetry metric, define when a specific network device (publisher) publishes information about the network telemetry metric, and specify one or more recipients (subscribers) for the published information.

[0014] In an example, a subscription may identify a single network telemetry metric. In another example, a subscription may identify multiple network telemetry metrics, and thus cover multiple network telemetry metrics. In another example, a subscription may identify a tree structure of network telemetry metrics. In an example, a subscription may specify a content format of a publication generated by a network device. A publication may be in the form of a network telemetry message generated by a network device and sent to a (multiple) recipient device.

[0015] The sequence of network telemetry messages provided by a network device is referred to herein as a "network telemetry message stream". In an example, the network device may generate the network telemetry message stream based on information stored in the network device and defining subscription aspects (referred to herein as "subscription information"). In another example, the network device may generate the network telemetry message stream in response to the network device being polled by a central server for network telemetry information, rather than generating the network telemetry message stream in response to a subscription.

[0016] Subscriptions may be on-change subscriptions or periodic subscriptions. For on-change subscriptions, when a change in a network telemetry metric covered by the subscription changes, the network device immediately publishes the change (e.g., sends a message containing content representing the change). For periodic subscriptions, the network device may publish changes to the covered telemetry metric according to a periodic schedule. The period of the periodic schedule may be referred to as a "prompt interval." In an example, each publication (network telemetry message) may contain data representing any (multiple) changes to the covered telemetry metric within the most recent prompt interval.

[0017] As an example of an abnormal scenario that may cause one or more network devices to generate an abnormally large rate of network telemetry messages, a password (e.g., expiration of a pre-shared key on a network switch or client device) may expire and result in a large number of network disassociation events (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 disassociation events). A change-once subscription for a network device may specify that the network device is to send a network telemetry message for each network disassociation event, and due to the expiration of a password, the network device may send a large number of network telemetry messages to report network disassociation events. For example, a network device may experience less than 100 disassociation events per minute on average, but due to the expiration of a password, the network device may experience 100,000 or more disassociation events per minute. Furthermore, due to the expiration of the same password, multiple network devices may publish messages corresponding to disassociation events, exacerbating the problem.

[0018] According to example implementations, the NMS cluster manages network telemetry message flows with abnormal or unexpected behavior (referred to herein as "abnormal network telemetry message flows") in a manner that does not overwhelm the network telemetry communication network, while still retaining network telemetry information associated with the message flows. More specifically, according to some implementations, a central server of the NMS cluster detects or identifies abnormal network telemetry message flows, and the central server intelligently throttles the identified abnormal network telemetry message flows until the message flow again exhibits normal or expected behavior.

[0019] According to an example implementation, the central server determines the bandwidth available for the detected anomalous network telemetry message flow (referred to herein as “B r According to an example implementation, taking into account the bandwidth of non-abnormal network traffic (e.g., application bandwidth, non-abnormal network telemetry message flow, and NMS management traffic), B r Bandwidth is the remaining bandwidth available for the network telemetry communication network. The central server throttles the abnormal network telemetry message flow to constrain the corresponding bandwidth consumed by the message flow to B r Bandwidth. The way the central server performs throttling depends on the network telemetry publishing model. For abnormal network telemetry message flows controlled by subscriptions (or "push-based models"), the central server replaces the basic subscription (e.g., default subscription) of the message flow with an adaptive subscription that constrains the bandwidth of the message flow. For abnormal network telemetry message flows controlled by polling (or "pull-based models"), the central server increases the polling period to constrain the bandwidth of the message flow.

[0020] According to an example implementation, upon successful login to a managed network device, the central server may send data (via one or more messages) to the network device representing a telemetry metric subscription for the network device (referred to as "subscription information"). The subscription information corresponds to a basic subscription (also referred to as a "non-adaptive" subscription or a "default" subscription) of the network device. After being configured with the subscription information, the network device may then begin generating and sending a network telemetry message stream corresponding to the parameters of the basic subscription.

[0021] According to an example implementation, the central server includes a message rate tracking engine for identifying an abnormal network telemetry message flow based on a corresponding message frequency or message rate of the abnormal network telemetry message flow. In addition, according to an example implementation, the central server includes a subscription management engine for r Bandwidth management manages network telemetry subscriptions of managed network devices in a manner that throttles abnormal network telemetry message flows. More specifically, according to an example implementation, in response to a message rate tracking engine determining that a particular network telemetry message flow is abnormal, a subscription management engine (via one or more messages) updates a corresponding basic subscription to change the basic subscription to an adaptive subscription. Relative to the basic subscription, the adaptive subscription constrains the bandwidth of the abnormal network telemetry message flow. This constraint is referred to herein as "throttling" the abnormal network telemetry message flow to produce a corresponding "throttled" network telemetry message flow. According to an example implementation, when a previously abnormal network telemetry message flow has returned to exhibiting expected behavior, the subscription management system (via one or more messages) restores the adaptive subscription of the message flow back to the corresponding basic subscription. In addition, if the available bandwidth of the abnormal network telemetry message flow changes (for example, B r If there is a change in bandwidth or a change in the number of abnormal network telemetry message flows), the subscription management engine can adjust the throttling parameters of the message flow accordingly (e.g., increase or decrease the allocated bandwidth).

[0022] In an example, a network device may provide a network telemetry message stream according to a specific on-change subscription. According to the on-change subscription, the network device may generate network telemetry messages in response to an event or state change of a specific telemetry metric. For this example, a message rate tracking engine may determine that the message stream exhibits unexpected or abnormal behavior based on a comparison of the message rate of the network telemetry message stream from the specific network device with one or more baselines. In response to this determination, the subscription management engine may then replace the corresponding on-change subscription with an adaptive on-change subscription, so that the network device thereafter generates a network telemetry message stream according to the adaptive on-change subscription.

[0023] Continuing with this example, the subscription management engine may determine the B available for all abnormal network telemetry message streams. rbandwidth, and the subscription management engine can determine the bandwidth of all abnormal messages with the same message subtype (referred to herein as "B amt Bandwidth”). In this case, a “message subtype” refers to a category or group associated with a particular network telemetry metric. In an example, the subscription management engine may determine that a particular message subtype associated with a particular anomalous network telemetry message flow has an associated weight (referred to herein as “W a Weight”), so that W of all message subtypes a The weights add up to 1. The subscription management engine can then determine the bandwidth available for the message subtype (referred to herein as "B amt Bandwidth”) is B r Bandwidth W a The weighted portion, or B amt =B r *W a The subscription management engine can then constrain the bandwidth of the abnormal message flow to the allocated bandwidth from B amt Bandwidth AllocationUnusual network telemetry message flow for this example.

[0024] In an example, for abnormal network telemetry messages with a subscription based on changes, the subscription management engine can replace the subscription based on changes with an adaptive subscription based on changes. The adaptive subscription based on changes identifies an allowed time window. The allowed time window can be part of a regulation time period that is measured by the network device and controls when the network device is allowed to send network telemetry messages during the regulation time period. For example, the network device may be prohibited from sending any network telemetry messages during the portion of the regulation time period that is outside the allowed time window. The ratio of the duration of the allowed time window to the regulation period corresponds to the degree of throttling imposed by the adaptive subscription based on changes. In contrast, the network device is allowed to send a message each time the change of the covered network telemetry metric changes based on changes.

[0025] In an example, in order to constrain the bandwidth of a message stream associated with a periodic subscription, the subscription management engine may replace a basic periodic subscription with an adaptive periodic subscription. Through the periodic subscription, a network device may generate a network telemetry message when a change (e.g., a state change or an event occurs) occurs during a recent reminder interval. In an example, the network device may generate messages based on the basic periodic subscription to generate a corresponding network telemetry message stream, and the message stream may have an unexpectedly large message rate. In response to the detection of an abnormal message stream, the subscription management engine may replace the basic periodic subscription with an adaptive periodic subscription that constrains the bandwidth of the message stream by increasing the reminder interval.

[0026] According to an example implementation, a message rate tracking engine determines whether a given network telemetry message flow is abnormal based on a comparison of the message rate of the given network telemetry message flow with one or more message rate baselines. In an example, the message rate tracking engine can determine a local message rate baseline and determine whether a given message rate is an outlier compared to the local message rate baseline. In this case, a "local" message rate baseline refers to a baseline determined from observed message rates of a collection of managed network devices that share certain common attributes.

[0027] In an example, a local message rate baseline may be associated with a particular message subtype and may be derived from observed message rates of all managed network devices of the same NMS cluster (e.g., compared to managed network devices from multiple NMS clusters). In another example, a local message rate baseline may be associated with a particular message subtype and may be derived from observed message rates of all managed network devices of the same local branch network. In another example, a local message rate baseline may be associated with a particular message subtype and may be derived from observed message rates of all managed network devices affiliated with the same NMS client identification (ID).

[0028] The message rate tracking engine may determine multiple local baselines for the same message subtype. In an example, the message rate tracking engine may determine multiple local message rate baselines for a given message subtype from observed message rates for the same set of managed network devices by applying different baseline algorithms. In another example, the message rate tracking engine may determine multiple local message rate baselines for a given message subtype over different observation periods (e.g., baselines corresponding to respective observation periods of one hour, one day, and three days).

[0029] In another example, the message rate tracking engine can determine a global message rate baseline and determine whether a given message rate is an outlier compared to a global message rate baseline. In this case, the "global" message rate baseline refers to a baseline for a particular message subtype that is determined based on observed message rates for network devices belonging to multiple local groups.

[0030] In an example, a global message rate baseline for a particular message subtype may be derived from observed message rates of managed network devices distributed across multiple local branch networks. In another example, a global message rate baseline for a particular message subtype may be derived from observed message rates of managed network devices associated with multiple NMS customer IDs. In another example, a global message rate baseline for a particular message subtype may be derived from observed message rates of all managed network devices that are similarly categorized (e.g., network devices of the same model or generation).

[0031] According to an example implementation, the message rate tracking engine can use, for example, multiple baseline algorithms and / or different global groupings of network devices to determine multiple global message rate baselines for a given message subtype. In addition, according to an example implementation, the message rate tracking engine can determine multiple global message rate baselines for different corresponding observation periods.

[0032] According to an example implementation, a message rate tracking engine may apply a specific baseline policy for the purpose of determining whether a particular network telemetry message flow is abnormal. A baseline policy is a rule or set of rules that defines how different outlier determinations affect the decision of whether a corresponding message flow is considered abnormal. In an example of a baseline policy, the message rate tracking engine may compare the message rate of a network telemetry flow with multiple message rate baselines, and based on these comparisons, if the message rate is an outlier according to all baselines, determine that the network telemetry message flow is abnormal. In another example of a baseline policy, the policy may specify that if the message rate is an outlier compared to any of the multiple baselines, the network telemetry message flow is considered abnormal. In another example, the baseline policy may specify a specific combination logic for combining multiple outlier determinations to determine whether a network telemetry message flow is considered abnormal. In another example, the policy may specify that if the message rate is an outlier compared to a specific baseline, the network telemetry message flow is considered abnormal, regardless of whether the message rate is considered an outlier compared to other baselines.

[0033] According to an example implementation, a network management system includes an interpolation engine that extrapolates network telemetry content that may be lost in a throttled network telemetry message stream. In this manner, the interpolation engine processes the content of the throttled network telemetry message stream to extrapolate content that would otherwise be reported if the message stream was not throttled. In an example, if the reporting periods for a basic periodic subscription and an adaptive periodic subscription are 60 seconds and 130 seconds, respectively, the interpolation engine can provide a message stream corresponding to a 60-second prompt interval (processing and logging from a central server) based on content from a message stream corresponding to a 130-second prompt interval. Thus, the interpolation engine can provide customer-desired reporting for a throttled message stream.

[0034] refer to Figure 1As a more specific example, according to some implementations, the computer network 100 includes one or more NMS clusters 112. Each NMS cluster 112 includes a central NMS resource 170 and one or more network device deployments 118. The network device deployment 118 includes a network device 114 (also referred to herein as a "managed network device 114"). In an example, the network device deployment 118 may correspond to a local branch network (e.g., a local area network (LAN)), such as a network corresponding to a specific building, a group of buildings, a campus, an edge computer system, or a data center. In another example, the network device deployment 118 may include multiple local branch networks. The network device deployment 118 may be associated with a specific geographic location, such as a campus site, a data center site, a city, a state, a country, or other geographic site. According to some implementations, the central NMS resource 170 may be a cloud-based resource that may be located in one or more data centers. In an example, a specific network device deployment 118 may be associated with a specific NMS customer ID.

[0035] According to an example implementation, the network devices 114 and the central NMS resources 170 may communicate via a network fabric 164. According to an example implementation, the network fabric 164 may be associated with one or more types of communication networks, such as, by way of example, a Fibre Channel network, a Compute Express Link (CXL) fabric, a dedicated management network, a local area network (LAN), a WAN, a global network (e.g., the Internet), a wireless network, or any combination thereof.

[0036] According to an example implementation, the central NMS resources 170 include a central server 176 that provides one or more NMS services 187. The (multiple) NMS services 187 can receive as input network telemetry metrics transmitted via network telemetry message streams provided by the network devices 114. Due to various scenarios, sometimes, one or more network telemetry message streams can exhibit unexpected behavior and can therefore be considered abnormal. In order to manage abnormal network telemetry message streams, the central server 176 includes a message rate tracking engine 178 and a subscription management engine 180.

[0037] In the context used herein, "managing" an abnormal network telemetry message flow refers to performing one or more actions to identify and / or regulate an abnormal network telemetry message flow. In an example, managing an abnormal network telemetry message flow may include identifying one or more characteristics (e.g., message rate or bandwidth) of an unexpected message flow (e.g., the message rate is an outlier compared to a baseline). In another example, managing an abnormal network telemetry message flow includes throttling the message flow. In another example, managing an abnormal network telemetry message flow includes increasing a reminder interval for periodic subscriptions to the message flow. In another example, for a message flow associated with a subscription on change, managing an abnormal network telemetry message flow includes identifying an allowed time window that limits when a network device can publish a message. In another example, managing an abnormal network telemetry message flow includes limiting the bandwidth of the message flow. In another example, managing an abnormal network telemetry message flow includes increasing the period of polling a network device that provides the flow. In another example, managing the abnormal network telemetry message flow includes identifying that a previously identified abnormal network telemetry message flow is no longer abnormal and modifying a throttling constraint imposed on the message flow (e.g., reducing a reminder interval, no longer binding a report to an allowed time window, reverting an adaptive subscription back to a basic subscription, or increasing a polling rate). In another example, managing the abnormal network telemetry message flow includes interpolating data for the message flow that would otherwise be lost due to throttling of the message flow.

[0038] According to an example implementation, the message rate tracking engine 178 identifies an abnormal network telemetry message flow based on the corresponding message rate of the message flow. According to an example implementation, the subscription management engine 180 throttles the identified abnormal network telemetry message flow by replacing the basic subscription of the flow with an adaptive subscription. Figure 1 As depicted, according to an example implementation, the central server 176 may include an interpolation engine 182 that provides extrapolated information for a throttled network telemetry message stream.

[0039] The network NMS services 187 may include any of a number of different services for visualizing, analyzing, logging, collecting, querying, and / or monitoring network telemetry metrics associated with the managed network devices 114. In an example, the central server 176 may provide data to a dashboard such as a graphical user interface (GUI) 167 that allows an IT administrator to monitor network telemetry-affiliated events, traces, and conditions, and allows an IT administrator to submit queries to the central server 176 for network device deployment information (e.g., network telemetry information, configuration information, and other information related to the network device deployment 118). In an example, the GUI 167 may be provided by specific client software executing on the management node 165, or as another example, may be provided by an Internet browser executing on the management node 165. In an example, the GUI 167 may allow an IT administrator to manage various aspects of the cluster 112, such as configuring the network devices 114, providing configuration data (e.g., configuration files) to be transmitted to the network devices 114, initiating firmware upgrades on the network devices 114, establishing a basic subscription for network telemetry reports, and various other management-related actions.

[0040] In an example, the NMS service 187 may identify potential or actual network device failure issues based on network telemetry metrics. In another example, the NMS service 187 may identify potential or actual network performance issues (e.g., issues with network devices or subnets) based on network telemetry metrics. In another example, the NMS service 187 may oversee remedial actions for correcting network issues. In another example, the NMS service 187 may identify performance issues with client devices (e.g., servers) connected to the network device deployment 118 or a portion thereof. In another example, the NMS service 187 may log network events. In another example, the NMS service 187 may service responses to queries related to obtaining information about the network device deployment 118. In another example, the NMS service 187 may provide recommended solutions for the identified network issues. In an example, the recommendation may be a recommended reconfiguration, upgrade, or replacement for one or more network devices 114. In another example, the recommendation may be a recommended reconfiguration for a specific network subnet. In another example, the recommendation may be a recommended firmware upgrade for a specific network device 114 or a group of network devices 114. According to some implementations, the central server 176 includes one or more core network management system engines 184 for providing network management system services 187 .

[0041] According to an example implementation, the central NMS resources 170 include an activation server 174. In an example, when the network device 114 is first connected to the network device deployment 118, a dynamic host protocol configuration (DHCP) server can provide the network device 114 with an Internet Protocol (IP) address of the activation server 174 (e.g., providing the IP address as a DHCP option). The activation server 174 and its other functions authenticate the network device 114, and upon successful authentication, the activation server 174 provides the network device 114 with network artifacts (e.g., an IP address and credentials) for connecting to the central server 176.

[0042] According to an example implementation, the central server 176 configures the newly connected network device 114 to provide or publish network telemetry messages by providing subscription information 116 to the network device 114. The subscription information 116 contains entries corresponding to basic subscriptions for different network telemetry metrics to regulate the publication of information about metric values ​​by the network device 114. In an example, among other attributes, a given entry of the subscription information 116 may also identify a specific message subtype (to accordingly identify a specific network telemetry metric), identify whether the subscription is a change-time subscription or a periodic subscription, identify a reminder interval for a periodic subscription, and identify an IP address of a recipient of the network telemetry message stream.

[0043] In an example, for a periodic subscription, the subscription information 116 may include data representing the following items:

[0044] Cust1 Sequence 1 Sub-ID1 60 default

[0045] For this example, "Cust1" represents a specific customer ID, "Serial 1" represents a specific network device serial number, and "Sub-ID1" represents a specific message subtype. In addition, for this example, the entry corresponds to a periodic subscription with a reminder interval of 60 seconds, and "Default" represents a basic subscription.

[0046] In another example, for another periodic subscription, the subscription information 116 may include data representing the following items:

[0047] Cust2 Sequence 2 Sub-ID1 10 default Cust1 Sequence 1 Sub-ID1 60 default

[0048] For this example, the periodic subscription has a reminder interval of 10 seconds.

[0049] In another example, for subscription on change, the subscription information 116 may include data representing the following items:

[0050] Cust3 Sequence 5 Sub-ID3 0 default

[0051] Here, "0" in the Notification Interval field indicates subscribing when changes occur.

[0052] The example subscription mapping entry above is a basic subscription, as indicated by "default" in the corresponding status field of the entry. According to an example implementation, the status field may indicate that the subscription is an adaptive subscription. For example, the example subscription mapping entry based on changes may be changed to an adaptive subscription, as indicated by data in the entry indicating that the corresponding subscription is adaptive:

[0053] Cust3 Sequence 5 Sub-ID3 0 Adaptive

[0054] As described herein, to manage abnormal network telemetry message flows, the subscription management engine 180 may modify subscription information 116 entries in one or more network devices 114. To track and manage subscriptions of the network devices 114, the subscription management engine 180 may maintain data representing a subscription mapping 181. In an example, the subscription management engine 180 may replace a basic subscription associated with the message flow with an adaptive subscription to throttle the message flow in response to detection of a particular abnormal network telemetry message flow. In another example, abnormal network telemetry message flow management may include the subscription management engine 180 replacing an adaptive subscription corresponding to the message flow with a basic subscription in response to a network telemetry message flow previously identified as abnormal no longer exhibiting abnormal behavior.

[0055] In the context used herein, "network telemetry measurement" refers to information or content from which the understanding of the state of a network, network device, or client device connected to (multiple) network devices can be determined directly or indirectly. In an example, network telemetry data may include statistics of periodic measurements of network or network device measurements or measurements. In an example, the statistic may be the traffic volume of a specific sampling period (e.g., the traffic volume of a specific VLAN). In other examples, the statistic may be periodically measured egress bandwidth, ingress bandwidth, latency, round-trip time, or the use of resources or the activity of a host. In other examples, network telemetry data may represent the value of a counter of a network device, a configuration setting of a network device, an event log record of a network device, a state snapshot of a network device, a configuration snapshot of a network device, or other information about a network device. Network telemetry data may represent an event. In an example, a network device may send a network telemetry message that is triggered by certain change events (e.g., a network telemetry message triggered by the sending of a disassociation message by a network device). Typically, network telemetry data may represent information about the control, management, and / or data plane of a network.

[0056] In the context used in this article, "network device" refers to actual or physical electronic components that enable data communication between other components. In an example, a network device can be a switch that operates at the second level (L2) of an open system interconnection (OSI) model to connect components of a computer network together. In another example, a network device can be a third level (L3) switch that connects two components of a computer network together and connects a computer network together. In other examples, a network device can be a component, a processor device, a network interface controller (NIC) or a structure switch including one or more devices in the above-mentioned devices of a gateway, a multicast router, a bridge, a Z generation or a computing high-speed link (CXL) network. A network device can be a wired or wireless device.

[0057] According to some implementations, a server (such as activation server 174 or central server 176) may correspond to machine-readable instructions (or "software") executed on one or more nodes 188 of central NMS resources 170. In the context used herein, a "node" refers to a processor-based entity with an associated set of hardware and software resources. Figure 1 As depicted, the node 188 may have one or more associated hardware processors 190 (e.g., one of one or more central processing unit (CPU) cores and / or one of a plurality of graphics processing unit (GPU) cores) and associated memory 192. The memory 192 is a non-transitory storage medium that may be formed of a semiconductor memory device, a memristor-based memory device, a magnetic memory device, a phase change memory device, a combination of devices of one or more of these memory technologies, and the like. The memory 192 may represent a collection of memories of both volatile memory devices and non-volatile memory devices. According to some implementations, the memory 192 may store machine-readable instructions that, when executed by the one or more hardware processors 190, cause the (multiple) hardware processors 190 to form instances of components of the activation server 174 and the central server 176. In an example, the memory 192 may store machine-readable instructions that, when executed by the one or more hardware processors 190, cause the (multiple) hardware processors 190 to form instances of the message rate tracking engine 178, the subscription management engine 180, the interpolation engine 182, and one or more core network management system engines 184.

[0058] In an example, a node 188 may be an actual or physical entity, such as a computer platform or a portion of a computer platform (e.g., corresponding to a set of CPU cores or a portion of a CPU core). In this case, a "computer platform" is a processor-based electronic device with an associated operating system. In an example, a computer platform may be a rack server or a blade server. In another example, a node 188 may be a virtual entity, such as a virtual machine, that is an abstraction of physical hardware and software resources. Depending on a specific implementation, a plurality of nodes 188 may be located on one or more virtual machines or physical machines. In addition, according to an example implementation, a node 188 may be distributed across virtual machines or physical machines located in different geographic locations (e.g., located in different data centers).

[0059] As used herein, an "engine" may refer to one or more circuits. For example, a circuit may be a hardware processing circuit that may include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit (e.g., a programmable logic device (PLD) such as a complex PLD (CPLD)), a programmable gate array (e.g., a field programmable gate array (FPGA)), an application specific integrated circuit (ASIC), or another hardware processing circuit. An "engine" may refer to a combination of one or more hardware processing circuits and machine-readable instructions (software and / or firmware) executable on the one or more hardware processing circuits. According to some implementations, one or more engines of a central NMS resource 170 (such as a message rate tracking engine 178, a subscription management engine 180, an interpolation engine 182, and / or (multiple) core NMS engines 184) may be formed by one or more hardware processors 190 that execute machine-readable instructions.

[0060] According to some implementations, the central NMS resources 170 may include multiple activation servers 174 and multiple central servers 176 for the same NMS cluster 112. In an example, for high availability (HA), a given service container for a particular cluster 112 may contain an HA group of activation servers 174 and an HA group of central servers 176, such that if a given active server 174, 176 fails, another server 174, 176 may take over for the failed server 174, 176 in the HA group.

[0061] Figure 2 A process 200 for detecting abnormal network telemetry message flows according to an example implementation is depicted. In an example, the process 200 may be performed by Figure 1 The message rate tracking engine 178 of is executed. The process 200 depicts abnormal network telemetry message flow detection for a particular network telemetry metric (also referred to herein as corresponding to a particular "message subtype").

[0062] refer to Figure 2 According to an example implementation, process 200 includes updating (block 204) a set of observed message rates for an incoming network telemetry message stream for a particular network telemetry metric. The network telemetry message stream may be received at a particular virtual or physical network interface of a central network and management system. In this case, a "set" of observed message rates refers to a set or group of message rates over a particular time period. In an example, process 200 may collect message rates over a sliding or moving time window. In another example, process 200 may collect message rates over a particular fixed adjustment period.

[0063] As depicted in block 208, one or more local message rate baselines and one or more global message rate baselines may be updated using an updated set of observed message rates. In an example, block 208 may include determining a local message rate baseline based on the message rates of network devices in the same local branch network. In another example, block 208 may include determining a local message rate baseline based on the message rates of network devices associated with the same customer identifier. In another example, block 208 may include determining a global message rate baseline based on the message rates of network devices across multiple local branches. In another example, block 208 may include determining a global message rate baseline based on the message rates of network devices across multiple customers. In another example, block 208 may include determining multiple local message rate baselines and / or multiple global message rate baselines over different time periods. In another example, block 208 may include determining multiple local message rate baselines and / or multiple global message rate baselines using different baseline algorithms. In another example, block 208 may include determining multiple local message rate baselines and / or different global message rate baselines for different groups of network devices.

[0064] Determination of a global or local baseline may include applying an outlier detection algorithm to a particular set of message rates (e.g., message rates corresponding to network devices in the same local branch network, or message rates corresponding to network devices associated with the same customer identifier). Depending on the particular implementation, any of a plurality of outlier detection algorithms may be applied to the set of message rates to determine the message rate baseline. In an example, the outlier detection algorithm may use a supervised machine learning based model or may use an unsupervised machine learning based model.

[0065] In an example, block 208 may apply a K-nearest neighbor (KNN) algorithm to a set of message rates. In an example, a distance may be determined for each message rate of the set, such as a distance determined based on a difference between the message rate and each of the K closest message rates of the set. The resulting set of distances may include a cluster of KNN distances corresponding to a baseline. The baseline may correspond to a particular range of distances. In an example, a particular threshold distance (e.g., a distance determined based on a statistical measurement or based on another criterion) may define the boundaries of the baseline, such that message rates corresponding to distances exceeding the threshold distance may be considered outliers.

[0066] According to another implementation, block 208 of process 200 may apply other outlier detection algorithms besides the KNN algorithm to derive a baseline and identify outliers. In another example, an interquartile range baseline algorithm may be used to derive a baseline and detect message rate outliers relative to the baseline. In another example, an isolation forest algorithm may be used to derive a baseline and detect message rate outliers relative to the baseline. In another example, a median absolute deviation algorithm may be used to derive a baseline and detect message rate outliers relative to a local baseline.

[0067] Regardless of the particular baseline algorithm(s) used, process 200 includes determining (decision box 212) whether one or more message rate outliers exist relative to the corresponding local baseline(s). If one or more message rate outliers are detected, process 200 may include logging the detected message rate anomaly(s), as depicted by box 213. Logging 213 may include, for example, logging information identifying the message stream(s) corresponding to the message rate outliers. According to decision box 214, process 200 includes determining where any message rate outliers exist relative to the corresponding global baseline(s). If one or more message rate outliers are detected, process 200 may include logging the detected message rate outliers, as depicted by box 215. Logging 213 may include, for example, logging information identifying the message stream(s) corresponding to the message rate outliers.

[0068] Process 200 next includes determining (block 220) whether any of the network telemetry message flows are anomalous based on applying one or more baseline policies to the detected outlier(s). In an example, a particular baseline policy may classify a particular message flow as anomalous based on whether the corresponding message rate is considered an outlier according to a particular message rate baseline. In another example, a particular policy may classify a particular message flow as anomalous if the corresponding message rate is considered an outlier according to a certain set of baselines. In another example, a particular policy may classify a particular message flow as anomalous if the corresponding message rate is considered an outlier according to any of the applied local and / or global baseline algorithms.

[0069] According to some implementations, process 200 may include evaluating whether a throttled network telemetry message stream should still be considered abnormal. In an example, the process may evaluate the extrapolated message rate for the throttled message stream for the purpose of determining whether the extrapolated message rate is considered an outlier. To this end, process 200 may, for example, obtain an extrapolated message rate from an interpolation engine (e.g., Figure 1 An interpolation engine 182) receives an extrapolated rate, which estimates an extrapolated message flow of a throttled message flow based on an observed throttled message flow.

[0070] According to an example implementation, block 220 may include updating a list of abnormal network telemetry message flows. The update may include actions such as adding, retaining, and / or removing network telemetry message flows from the list.

[0071] Figure 3 Depicted is an adaptive subscription management process 300 for managing abnormal network telemetry message flows according to an example implementation. In an example, the process 300 may be performed by Figure 1 The subscription management engine 180 executes.

[0072] refer to Figure 3 , process 300 includes identifying (block 304) any abnormal network telemetry message flows. In an example, block 304 may include accessing a list of abnormal network telemetry message flows, the list being provided by a user such as Figure 1The message rate tracking engine 178 or the like is provided by a message rate tracker. Process 300 includes determining (decision box 306) whether any adaptive subscription is to be restored to a corresponding basic subscription. In an example, the subscription management engine may maintain a list of adaptive subscriptions in a subscription map for the purpose of allowing identification of any network telemetry message streams that were previously determined to be abnormal and have corresponding adaptive subscriptions, but the network telemetry message streams may no longer be considered abnormal. If any adaptive subscription is to be restored to a corresponding basic subscription, a modification is made according to box 310 of process 300. In an example, changing a subscription from an adaptive subscription to a basic subscription may include the subscription management engine sending a message to a network device to cause the network device to update a corresponding subscription entry stored in the network device.

[0073] According to block 324, process 300 includes determining the B available for the abnormal network telemetry message flow during the t minute adjustment period. r According to an example implementation, process 300 determines B in block 324 as follows: r First, process 300 determines the total available bandwidth (referred to herein as “B a Bandwidth”). In the example, for a regulation period of t minutes, B a The bandwidth may be the maximum bandwidth (referred to herein as "B m bandwidth”) (which is the bandwidth consumed by the application service) minus the bandwidth of the NMS cluster management data (referred to herein as “B c bandwidth") and the remaining miscellaneous general data bandwidth (also referred to herein as "B g bandwidth”). Therefore, B a The bandwidth can be described as B a =B m -B c -B g The process 300 may determine the duration t of the throttling period based on one or more characteristics, such as the frequency of the message flow and the message subtype. Generally, a long duration for t provides a better chance of extrapolation in case of missed events.

[0074] According to an example implementation, in process 300, B r The total bandwidth is B a bandwidth minus the bandwidth consumed by normal non-anomalous network telemetry metric flows (referred to as “B an Bandwidth”), or B r =B a -B an .

[0075] In determining B rAfter bandwidth, process 300 next includes executing subprocess 326 to manage subscription of abnormal network telemetry message flow so that the abnormal network telemetry message flow is constrained to B r Subprocess 326 includes determining (block 328) the bandwidth available for the abnormal message flow of the next message subtype. amt Bandwidth. According to an example implementation, process 300 performs the following operations by using the bandwidth assigned to the message subtype. a Weight pair B r Bandwidth weighting to determine the bandwidth available for a particular subtype amt Bandwidth, or B amt =B r *W a According to an example implementation, process 300 assigns W to each message subtype. a Weight. Since W a Weights, some message subtypes can be assigned higher values ​​and importance than other message subtypes. a The weights may be based on one or more criteria, such as the relative importance of the client, the relative importance of the message subtype, and the characteristics of the network device. a The sum of the weights equals 1.

[0076] According to decision block 336, process 300 includes determining whether any subscriptions are to be modified. In an example, if the particular abnormal network telemetry message stream currently has a basic subscription, then according to decision block 336, it may be determined that the basic subscription is to be replaced with an adaptive subscription. In another example, if the particular abnormal network telemetry message stream currently has an adaptive subscription, and B amt If the bandwidth has not changed since the last adjustment cycle, then according to decision block 336, a determination may be made not to further modify the adaptive subscription. In another example, if a particular abnormal network telemetry message stream currently has an adaptive subscription, and B amt If the bandwidth has changed since the last adjustment cycle, then according to decision box 336, a decision based on the new B amt Bandwidth is used to modify the determination of adaptive subscription.

[0077] If a decision is made in decision block 336 to modify any subscription, then the subscription(s) are modified according to block 340. According to an example implementation, block 340 may involve: determining the bandwidth of the abnormal message flow without throttling (referred to herein as “unthrottled bandwidth”) for a given subscription to be modified; determining B amt bandwidth to the unthrottled bandwidth, and then adapt subscriptions based on that ratio to constrain the flow of abnormal messages to B amt Bandwidth. When the subscription is no longer being modified for a particular message type, process 300 includes determining (decision block 344 ) whether there are any remaining message subtypes to process, and if so, control returns to block 328 .

[0078] In a more specific example of updating subscriptions to throttle abnormal network telemetry message flows, 100 network devices may have a subscription on change to report IEEE 802.11 disassociation messages in response to network disassociation events. For this example, for non-abnormal message flows, there are an average of 80 IEEE 802.11 disassociation messages per minute. In a five-minute time period, there may be a corresponding average number of 400 IEEE 802.11 disassociation messages. For this example, there are 95 network devices that send the expected average number of IEEE 802.11 disassociation messages in corresponding normal or non-abnormal message flows. Due to abnormal events (e.g., expired passwords), five network devices provide IEEE 802.11 disassociation message flows with a corresponding message rate of 130,000 messages per five minutes, and are considered abnormal message flows for this example. Therefore, for this example, there are a total of 61,757,600 IEEE 802.11 disassociation messages (abnormal and non-abnormal) in a five-minute period: 130,000*5(number of messages for abnormal message flow)+400*95(number of messages for non-abnormal message flow)=61,757,600 events. For example, the message size of a message reporting an IEEE 802.11 disassociation is 128 bytes. Therefore, the total bandwidth of IEEE 802.11 disassociation messages for all 100 network devices in a five-minute period is 128*61757600=7,904,972,800 bytes.

[0079] Continuing with this example, the B available for the exception message flow amt The bandwidth can be derived as follows. B at the interface during a five minute period m The maximum application traffic bandwidth is 4 gigabytes (GB) or 4,294,967,296 bytes. c The NMS cluster management bandwidth is 5,242,880 bytes. g The miscellaneous data bandwidth is 0,242,880 bytes. Therefore, the B a The total available bandwidth is B m -B c -B g = 4,239,481,536 bytes. Therefore, the B available for the abnormal message flow amt Bandwidth is the total bandwidth remaining for the message flow, which is 38,864,000 bytes, or approximately 38 megabytes (MB). For this example, the W of the disassociated event message subtype is a The relative weight is 1 / 5 or .20.

[0080] For this example, the unthrottled bandwidth of the abnormal message flow in a five-minute period is 130,000 events * 5 devices, and each 802.11 disassociation message is 128 bytes, which is 83,200,000 bytes, or about 83MB. The bandwidth throttling ratio for this example is 0.45, or 38 / 83. Therefore, the basic subscription can be replaced with an adaptive subscription that constrains the message flow to a ratio of 0.45 of its unthrottled bandwidth. For subscriptions that are adaptively changed, the subscription identifier allows a time window so that the ratio of the allowed time to the total adjustment period is 0.45. For adaptive periodic subscriptions, the subscription specifies a new prompt interval so that the ratio of the new prompt interval to the old prompt interval is 1 / 0.45, or 2.22.

[0081] Figure 4 A process 400 for interpolating or extrapolating the content of a throttled network telemetry message stream is depicted. In this manner, the extrapolated content corresponds to content that is estimated to already be present in the non-throttled message stream but not in the throttled message stream. In an example, the process 400 may be performed by Figure 1 The interpolation engine 182 is executed.

[0082] refer to Figure 4 According to an example implementation, process 400 includes receiving (block 404) a message of a network telemetry message stream. According to decision block 408, a determination is made as to whether the network telemetry message stream corresponds to an adaptive subscription. In an example, decision block 408 may include a data interpolation engine that consults a subscription map maintained by a network management system. Figure 4 As depicted, if the network telemetry message stream does not correspond to an adaptive subscription, then according to block 420, the message stream is provided unchanged to one or more core network management system engines.

[0083] If, according to decision block 408, it is determined that the network telemetry message stream corresponds to an adaptive subscription, then according to block 412, process 400 includes estimating lost content that would have been transmitted in the corresponding non-throttled network telemetry message stream and reconstructing (block 416) the non-throttled network telemetry message stream.

[0084] In an example, the network telemetry message stream may correspond to an adaptive periodic subscription having a reminder interval with a longer duration than the reminder interval of the corresponding basic subscription. In an example, the original reminder interval of the basic subscription may be 60 seconds, and the reminder interval of the adaptive periodic subscription may be 130 seconds. Based on the content present in the received message of the throttled network telemetry message stream, block 412 may include determining the network telemetry report at a 60-second reminder interval. In addition, in an example, block 412 may include determining the content of the next upcoming corresponding 60-second reminder interval. Using the extrapolated data, block 416 includes reconstructing the message stream according to the 60-second reminder interval to provide a presentation of the message stream without throttling. In another example, the network telemetry message stream may correspond to an adaptive subscription when changing. For this example, block 412 may include reconstructing a message sequence corresponding to a non-throttled basic subscription based on the content in the throttled message stream. In addition, according to the example implementation, process 400 may include estimating the message rate of the reconstructed message stream so that the message rate can be evaluated for the purpose of determining whether the message stream is still considered abnormal.

[0085] Other implementations are within the scope of the following claims. For example, Figure 1 The subscription management engine 180 is an example of a network telemetry management engine that adjusts the bandwidth of an abnormal network telemetry message stream. According to another example implementation, the network telemetry management engine may adjust the bandwidth of an abnormal network telemetry message stream generated by polling. More specifically, according to an example implementation, a central server of a network telemetry management system cluster may use a pull-based telemetry reporting model. In this way, the central server may include a polling engine that polls network devices at a periodic polling cycle to obtain network telemetry metric information. In an example, the periodic polling cycle may vary depending on the requested network metric information. According to an example implementation, the network telemetry management engine of the central server may adjust the bandwidth of a specific abnormal network telemetry message stream to reduce the width of the message stream. In an example, the network telemetry management engine may determine B for an abnormal network telemetry message stream. amt Bandwidth, the abnormal network telemetry message flow has a current bandwidth (referred to as "B anom ”) and is currently being polled in the first polling cycle (referred to as “T1”). The network telemetry management engine may replace the T1 polling cycle with a B anom The bandwidth is reduced to B amt The network telemetry management engine may determine the T2 polling period as follows: T2 = T1 * (B anom / B amt ).

[0086] refer to Figure 5According to an example implementation, process 500 includes monitoring (block 504) by a message rate tracking engine a plurality of message streams sent by respective network devices in a plurality of network devices. The plurality of message streams are associated with reporting network telemetry metrics to a network management system. In an example, the message streams may be generated and sent by a network device based on a subscription when it changes. In another example, the message streams may be generated and sent by a network device based on a periodic subscription. In an example, a plurality of network devices may correspond to a network device deployment of an NMS cluster. In an example, the network devices may be part of the same branch network. In another example, the network devices are distributed across a plurality of branch networks. In an example, a message rate tracker may be a component of a collection of central NMS components. In an example, the collection of central NMS components may be cloud-based. In an example, a message rate tracking engine may be part of a central server. In an example, a message rate tracking engine may correspond to machine-readable instructions executed by one or more processors.

[0087] Process 500 includes determining (block 508) by a message rate tracker that a given message flow among a plurality of message flows exhibits unexpected behavior. In an example, determining that a given message flow exhibits an unexpected message flow may include determining one or more message rate baselines and comparing the message rate of the given message flow to the (multiple) baselines. In an example, determining that a given message flow exhibits an unexpected message flow may include determining whether the message rate of the given message flow is an outlier compared to the message rate baseline. In an example, determining whether a message is an outlier may include applying a KNN baseline algorithm. In an example, determining whether a message rate is an outlier may include applying an interquartile range baseline algorithm. In an example, determining whether a message rate is an outlier may include applying an isolation forest baseline algorithm. In an example, determining whether a message rate is an outlier may include applying a median absolute deviation baseline algorithm. In an example, determining whether a given message flow exhibits unexpected behavior may include applying a baseline strategy based on (multiple) outlier detections.

[0088] Process 500 includes: in response to determining that a given message stream exhibits unexpected behavior, determining (block 512) by a subscription manager an aggregate available bandwidth for message streams in a plurality of message streams that respectively exhibit expected behavior. According to an example implementation, determining the aggregate available bandwidth may take into account a maximum application bandwidth. According to an example implementation, determining the aggregate available bandwidth may take into account a cluster management bandwidth.

[0089] Process 500 includes: in response to determining that a given message stream exhibits unexpected behavior, determining (block 508) by a subscription manager an adjusted bandwidth for the given message stream based on the aggregate available bandwidth. In an example, determining the adjusted bandwidth includes determining the bandwidth for the given message stream based on the aggregate available bandwidth and a weight. In an example, the weight corresponds to a message subtype associated with the given message stream.

[0090] The process 500 includes, in response to determining that a given message stream exhibits unexpected behavior, modifying (block 512) by a subscription manager a subscription of a given network device to a network management service based on the regulated bandwidth. In an example, the subscription is a time-of-change subscription, and modifying the subscription includes specifying an allowed time window for the given message stream in an adaptive subscription to constrain bandwidth consumed by the given message stream to the regulated bandwidth. In an example, the subscription is a periodic subscription, and modifying the subscription includes specifying a reminder interval in the adaptive subscription that is longer than a reminder interval in a corresponding basic subscription to constrain bandwidth consumed by the given message stream to the regulated bandwidth.

[0091] refer to Figure 6 According to an example implementation, a non-transitory storage medium 600 stores machine-readable instructions 610, which, when executed by a machine, causes the machine to receive an indication of an anomaly of a first message stream in a plurality of message streams. The plurality of message streams are provided by a plurality of network devices over a network to report network telemetry metrics according to corresponding subscriptions. In an example, the instruction 610 may be executed on one or more nodes of a collection of central NMS resources. In an example, the collection of central NMS resources may be cloud-based. In an example, the central resources may be associated with a central server of an NMS cluster. In an example, a message stream may be generated and sent by a network device according to a subscription when it changes. In another example, a message stream may be generated and sent by a network device according to a periodic subscription. In an example, a plurality of network devices may correspond to a network device deployment of an NMS cluster. In an example, the network device may be part of the same branch network. In another example, the network device is distributed across a plurality of branch networks.

[0092] In an example, determining that a message flow is abnormal may include determining one or more message rate baselines and comparing the message rate of the message flow to the (multiple) baselines. In an example, determining that a message flow is abnormal may include determining that the message rate of the message flow is an outlier compared to the message rate baseline. In an example, determining whether the message rate is an outlier may include applying a KNN baseline algorithm. In an example, determining whether the message rate is an outlier may include applying an interquartile range baseline algorithm. In an example, determining whether the message rate is an outlier may include applying an isolation forest baseline algorithm. In an example, determining whether the message rate is an outlier may include applying a median absolute deviation baseline algorithm. In an example, determining that a message flow is abnormal includes applying a baseline strategy based on one or more outlier detections.

[0093] Instructions 610, when executed by the machine, cause the machine to determine, in response to receiving the indication, a first bandwidth available for network telemetry communication of anomaly messages in the network. According to an example implementation, determining the first bandwidth may take into account a maximum application bandwidth. According to an example implementation, determining the first bandwidth may take into account a cluster management bandwidth.

[0094] Instructions 610 also cause the machine to determine, in response to receiving the indication, a second bandwidth available for a first network device in the plurality of network devices based on the first bandwidth. In an example, determining the second bandwidth includes determining a bandwidth for the message flow based on the aggregated available bandwidth and a weight. In an example, the weight corresponds to a message subtype associated with the message flow.

[0095] The instructions 610, when executed by the machine, also cause the machine to replace a first subscription associated with the first network device with a second adaptive subscription corresponding to the second bandwidth in response to receiving the indication. In an example, the first subscription is a time-of-change subscription, and the second adaptive subscription specifies an allowed time window for constraining the message flow to the second bandwidth. In an example, the subscription is a periodic subscription, and the second adaptive subscription specifies a reminder interval in the adaptive subscription that is longer than the reminder interval in the first subscription to constrain the bandwidth consumed by the message flow to the second bandwidth.

[0096] refer to Figure 7 According to an example implementation, a computer system 700 includes a network telemetry message for tracking engine 704 and network telemetry management engine 708. The network telemetry message rate tracking engine 704 monitors a plurality of message streams sent by respective network devices in a plurality of network devices. The plurality of message streams are associated with reporting network telemetry metrics to a network management system. The network telemetry message rate tracking engine 704 determines that a rate of a given message stream in the plurality of message streams is an outlier. In an example, the message stream may be generated and sent by the network device according to a subscription when it changes. In another example, the message stream may be generated and sent by the network device according to a periodic subscription. In an example, the plurality of network devices may correspond to a network device deployment of an NMS cluster. In an example, the network device may be part of the same branch network. In another example, the network device is distributed across a plurality of branch networks. In an example, the network telemetry message rate tracking engine 704 may be a component of a collection of central NMS components. In an example, the collection of central NMS components may be cloud-based. In an example, the network telemetry message rate tracking engine 704 may be part of a central server. In an example, the message rate tracking engine may correspond to machine-readable instructions executed by one or more processors.

[0097] In an example, the network telemetry message rate tracking engine 704 determines that the rate is an outlier by applying a KNN baseline algorithm. In an example, the network telemetry message rate tracking engine 704 determines that the rate is an outlier by applying an interquartile range baseline algorithm. In an example, the network telemetry message rate tracking engine 704 determines that the rate is an outlier by applying an isolation forest baseline algorithm. In an example, the network telemetry message rate tracking engine 704 determines that the rate is an outlier by applying a median absolute deviation baseline algorithm. In an example, the network telemetry message rate tracking engine 704 determines that the rate is an outlier by applying a baseline strategy based on (multiple) outlier detections.

[0098] The network telemetry management engine 708 determines an aggregate available bandwidth for message flows having corresponding expected rates in the plurality of message flows in response to the network telemetry message tracking engine determining that the rate is an outlier. The network telemetry management engine 708 determines an adjusted bandwidth for a given message flow based on the aggregate available bandwidth in response to the network telemetry message tracking engine 704 determining that the rate is an outlier. The network telemetry management engine 708 modifies a bandwidth associated with the given message flow based on the adjusted bandwidth in response to the network telemetry message tracking engine 704 determining that the rate is an outlier.

[0099] In an example, a given network device may provide a given message in response to a pull model. In an example, according to the pull model, the network telemetry management system cluster polls the given network device at a first periodic rate to cause the given network device to provide a given message stream. The first periodic polling rate has an associated first polling period. In an example, the network telemetry management engine may modify a bandwidth associated with the given message stream by reducing the first periodic polling rate to a second periodic polling rate. The second periodic polling rate has an associated second period that is greater than the first period.

[0100] In an example, a given network device may provide a given message stream in response to a push model or a subscription. In an example, the subscription is a time-on-change subscription, and the modified subscription specifies an allowed time window that constrains the message stream to a second bandwidth. In another example, the subscription is a periodic subscription, and the modified subscription specifies a reminder interval in an adaptive subscription that is longer than a reminder interval in a first subscription to constrain bandwidth consumed by the message stream to the second bandwidth.

[0101] According to an example implementation, determining that a given message flow exhibits unexpected behavior includes comparing a message rate of the given message flow to a message rate threshold. A particular advantage is that anomalous network telemetry flows can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0102] According to an example implementation, a message rate threshold is determined. Determining the message rate threshold includes applying a baseline algorithm to a plurality of message rates associated with network telemetry metrics and associated with a plurality of network devices of a local network containing a given network device. A particular advantage is that anomalous network telemetry flows can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0103] According to an example implementation, a message rate threshold is determined. Determining the message rate threshold includes applying a baseline algorithm to a plurality of message rates associated with network telemetry metrics and associated with a plurality of network devices located in at least one other local network other than the local network containing the given network device. A particular advantage is that anomalous network telemetry flows can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0104] According to an example implementation, a second network device of a plurality of network devices transmits messages of a network telemetry message stream according to a periodic subscription interval. The process includes: in response to determining that a given message stream exhibits unexpected behavior, modifying a subscription of a second network device of the plurality of network devices to increase a periodic subscription interval at which the second network device transmits messages of a corresponding message stream. A particular advantage is that anomalous network telemetry streams can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0105] According to an example implementation, the process further includes: in response to determining that a given message flow exhibits unexpected behavior, modifying, by the subscription manager, a second subscription to a periodic message flow associated with reporting a second network telemetry metric to the network management system. Modifying the second subscription includes increasing the period of the periodic subscription from a first reporting time interval to a second reporting time interval to reduce a bandwidth of the periodic message flow. Of particular advantage, anomalous network telemetry flows can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0106] According to an example implementation, the process also includes receiving, by the network management system, a periodic message stream corresponding to the second reporting time interval to provide the received periodic message stream. The process also includes interpolating, by the network management system, content missing from the received periodic message stream, the content corresponding to a difference between the first reporting time interval and the second reporting time interval. A particular advantage is that anomalous network telemetry streams can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0107] According to an example implementation, a received periodic message stream includes a first sample of a network telemetry statistic. Interpolation includes estimating an additional second sample of the network telemetry statistic, the second sample corresponding to a difference between the first reporting time interval and the second reporting time interval. A particular advantage is that anomalous network telemetry streams can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting communication of network telemetry metric information from the managed network.

[0108] According to an example implementation, the process includes receiving, by a network management system, a given message stream having an adjusted bandwidth to provide a received given message stream. The process includes interpolating, by the network management system, content lost in the received given message stream, the content corresponding to a difference between a bandwidth of the given message stream prior to adjustment of the bandwidth and the adjusted bandwidth. A particular advantage is that anomalous network telemetry streams can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0109] According to an example implementation, modifying a previous subscription allows a given network device to report network telemetry metrics in response to a change in state of the network telemetry metrics. Modifying the subscription includes constraining the reporting of network telemetry metrics by the network device to a time window having a duration corresponding to the regulated bandwidth. A particular advantage is that anomalous network telemetry flows can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly limiting the communication of network telemetry metric information from the managed network.

[0110] According to an example implementation, the process further includes: in response to expiration of the time window, determining whether the given flow continues to exhibit unexpected behavior. The process includes: in response to expiration of the time window and in response to determining that the given message flow continues to exhibit unexpected behavior, constraining the reporting of network telemetry metrics by the network device to another time window. A particular advantage is that anomalous network telemetry flows can be identified and throttled in an intelligent manner that takes into account bandwidth constraints of the network without unduly restricting the communication of network telemetry metric information from the managed network.

[0111] The detailed description set forth herein refers to the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the foregoing description to refer to the same or similar parts. However, it should be clearly understood that the drawings are for illustration and description purposes only. Although several examples are described in this document, modifications, adaptations, and other implementations are also possible. Therefore, the detailed description does not limit the disclosed examples. On the contrary, the proper scope of the disclosed examples may be defined by the appended claims.

[0112] The terms used herein are only for the purpose of describing specific examples, and are not intended to be limiting terms. As used herein, unless the context clearly indicates otherwise, the singular forms "one", "one" and "said" should also include plural forms. The term "multiple" used herein is defined as two or more than two. The term "another" used herein is defined as at least the second or more. Unless otherwise specified, the term "connection" used herein is defined as connection, whether direct connection without any intervening element, or indirectly through the connection of at least one intervening element. Two elements can be connected mechanically, electrically or communicatively through a communication channel, path, network or system. The term "and / or" used herein refers to and encompasses any and all possible combinations of related listed items. It should also be understood that although the terms first, second, third, etc. can be used to describe various elements in this article, these elements should not be limited by these terms, because unless otherwise specified or the context indicates otherwise, these terms are only used to distinguish one element from another element. As used herein, the term "including" means including but not limited to, and the term "including" means including but not limited to. The term "based on" means at least partially based on.

[0113] While the present disclosure has been described with respect to a limited number of implementations, those skilled in the art having the benefit of this disclosure will appreciate numerous modifications and variations thereof. The appended claims are intended to cover all such modifications and variations.

Claims

1. A method comprising: monitoring, by a message rate tracking engine, a plurality of message flows sent by respective ones of a plurality of network devices, wherein the plurality of message flows are associated with reporting network telemetry metrics to a network management system; determining, by the message rate tracking engine, that a given message flow among the plurality of message flows exhibits unexpected behavior; as well as In response to determining that the given message flow exhibits the unexpected behavior: Determining, by a subscription management engine, aggregate available bandwidth for message flows in the plurality of message flows that respectively exhibit expected behavior; determining, by the subscription management engine, an adjusted bandwidth for the given message flow based on the aggregate available bandwidth; as well as A subscription of the given network device to the network management system is modified by the subscription management engine based on the adjusted bandwidth.

2. The method of claim 1 , wherein determining that the given message flow exhibits the unexpected behavior comprises: The message rate of the given message flow is compared to a message rate threshold.

3. The method of claim 2, further comprising determining a message rate threshold, wherein determining the message rate threshold comprises: A baselining algorithm is applied to a plurality of message rates associated with the network telemetry metric and associated with a plurality of network devices of a local network including the given network device.

4. The method of claim 2, further comprising determining a message rate threshold, wherein determining the message rate threshold comprises: A baselining algorithm is applied to a plurality of message rates associated with the network telemetry metric and associated with a plurality of network devices located in at least one other local network other than the local network containing the given network device.

5. The method of claim 2, wherein a second network device among the plurality of network devices sends a message of the network telemetry message stream according to a periodic subscription interval, the method further comprising: In response to determining that the given message flow exhibits the unexpected behavior, a subscription of a second network device of the plurality of network devices is modified to increase a periodic subscription interval at which the second network device sends messages of the corresponding message flow.

6. The method according to claim 1, further comprising: In response to determining that the given message stream exhibits the unexpected behavior, the subscription manager modifies a second subscription to a periodic message stream associated with reporting a second network telemetry metric to the network management system, wherein modifying the second subscription includes increasing a period of the periodic subscription from a first reporting time interval to a second reporting time interval to reduce a bandwidth of the periodic message stream.

7. The method according to claim 6, further comprising: receiving, by the network management system, the periodic message stream corresponding to the second reporting time interval to provide the received periodic message stream; as well as Content lost in the received periodic message stream is interpolated by the network management system, the content corresponding to a difference between the first reporting time interval and the second reporting time interval.

8. The method according to claim 6, wherein: The received periodic message stream includes a first sample of network telemetry statistics; The interpolation includes estimating an additional second sample of the network telemetry statistic, the second sample corresponding to a difference between the first reporting time interval and the second reporting time interval.

9. The method according to claim 1, further comprising: receiving, by the network management system, the given message stream having the adjusted bandwidth to provide a received given message stream; as well as Content lost in the received given message stream is interpolated by the network management system, the content corresponding to a difference between a bandwidth of the given message stream prior to the adjustment of the bandwidth and the adjusted bandwidth.

10. A method according to claim 1, wherein modifying the previous subscription allows the given network device to report the network telemetry metric in response to a change in the state of the network telemetry metric, and modifying the subscription includes constraining the network device's reporting of the network telemetry metric to a time window having a duration corresponding to the adjusted bandwidth.

11. The method of claim 10, further comprising, in response to expiration of the time window: determining whether the given message flow continues to exhibit unexpected behavior; and Responsive to determining that the given message flow continues to exhibit the unexpected behavior, restricting reporting of the network telemetry metric by the network device to another time window.

12. A non-transitory storage medium for storing machine-readable instructions that, when executed by a machine, cause the machine to: receiving an indication of an anomaly in a first message flow of a plurality of message flows, wherein the plurality of message flows are provided by a plurality of network devices over a network to report network telemetry metrics according to corresponding subscriptions; as well as In response to receiving the indication: determining a first bandwidth available for network telemetry communications of exception messages in the network; determining a second bandwidth available for a first network device among the plurality of network devices based on the first bandwidth; as well as A first subscription associated with the first network device is replaced with a second adaptive subscription corresponding to the second bandwidth.

13. The storage medium of claim 12, wherein the instructions, when executed by the machine, further cause the machine to: determine the first bandwidth based on a third bandwidth corresponding to application traffic in the communication network and a fourth bandwidth for non-abnormal network telemetry message communications in the communication network.

14. The storage medium according to claim 12, wherein: The first message flow is associated with a network telemetry category; and The instructions, when executed by the machine, further cause the machine to determine the second bandwidth based on a weight associated with the network telemetry category.

15. The storage medium according to claim 12, wherein: The first subscription comprises a periodic subscription specifying a first periodic reporting time interval; and The instructions, when executed by the machine, further cause the machine to determine a second reporting time interval based on the second bandwidth; and The second reporting time interval is included in the second subscription.

16. The storage medium according to claim 12, wherein: The first subscription includes an on-change subscription, the on-change subscription specifying reporting of a state change in response to the change occurring; and The instructions, when executed by the machine, further cause the machine to determine a time window to limit the reporting based on the second bandwidth; and The time window is identified in the second subscription.

17. A computer system comprising: Network telemetry message rate tracking engine for: monitoring a plurality of message flows sent by respective ones of a plurality of network devices, wherein the plurality of message flows are associated with reporting network telemetry metrics to a network management system; as well as determining that a rate of a given message flow among the plurality of message flows is an outlier, wherein the given message flow has an associated bandwidth; as well as a network telemetry management engine for, in response to the network telemetry message tracking rate engine determining that the rate is an outlier: determining an aggregate available bandwidth for message flows of the plurality of message flows having corresponding expected rates; as well as The bandwidth associated with the given message flow is adjusted based on the aggregate available bandwidth.

18. The computer system of claim 17, wherein: A given network device among the plurality of network devices provides the given message stream in response to a subscription; and The network telemetry management engine modifies the subscription to adjust the bandwidth associated with the given message flow.

19. The computer system of claim 17, wherein: A given network device of the plurality of network devices responds to being polled at a first rate to provide the given message stream; and The network telemetry management engine modifies the polling so that the given network device is polled at a second rate that is less than the first rate to regulate the bandwidth associated with the given message flow.

20. The computer system of claim 17, wherein the network telemetry message tracking engine is configured to further apply a baseline algorithm to a plurality of message rates associated with the network telemetry metric to determine a rate baseline, and to determine that the rate of the given message flow is an outlier in response to a comparison of the rate of the given message flow with the rate baseline.

Images