Application service level desired health and performance

Abstract

Embodiments of the present disclosure relate to application service level desired health and performance. Techniques for monitoring application performance in a computer network are described. For example, a network management system (NMS) includes a memory storing path data received from a plurality of network devices, the path data reported by each of the plurality of network devices for one or more logical paths over a wide area network (WAN) from a physical interface of a given network device. Further, the NMS may include processing circuitry in communication with the memory and configured to determine one or more application health evaluations for the one or more applications based on the path data, where the one or more application health evaluations are associated with one or more application time periods for the site, and in response to determining the at least one fault state, outputting a notification including an identification of a root cause of the at least one fault state.

Description

[0001] This application is a divisional application of the invention patent application with application date of January 17, 2023, application number 2023800229520, and invention name “Application Service Level Expected Health and Performance”. Technical Field

[0002] The present disclosure relates generally to computer networks, and more particularly to monitoring and / or managing application performance in a computer network. Background Art

[0003] A computer network is a collection of interconnected computing devices that can exchange data and share resources. Example computing devices include routers, switches, and other layer 2 network devices operating within layer 2 (L2) (i.e., the data link layer) of the Open Systems Interconnection (OSI) reference model, and layer 3 (L3) network devices operating within layer 3 (i.e., the network layer) of the OSI reference model. Network devices within a computer network typically include a control unit that provides control plane functions for the network device and a forwarding component for routing or switching data units. Summary of the invention

[0004] In general, the present disclosure describes techniques for monitoring application performance and managing application problems that may affect the user experience in a network. A cloud-based network management system (NMS) receives one or more of application performance data, path data, and network device data from a network device. The application performance data indicates one or more of jitter, delay, loss, and application service. The path data indicates one or more aspects of application and / or network performance monitored on each logical path between network devices on a network, which may be a wide area network (WAN), such as a broadband network, a long-term evolution (LTE) network, or a multi-protocol label switching (MPLS) network. The network device data indicates the performance of network devices (e.g., routers or gateways) in the WAN. The NMS includes an application health service level expectation (SLE) measurement engine that determines one or more application health assessments based on the path data received from the network device. Based on the application health assessment, the NMS can identify a success or failure state associated with the application, identify the root cause of one or more failure states, and / or automatically recommend or invoke one or more remedial measures to resolve the identified failure state.

[0005] A given network device may establish multiple logical paths (e.g., peer paths or tunnels) with multiple other network devices over a WAN over a single physical interface. Each of the network devices may include a software agent or other module configured to report path data collected at the logical path level to an NMS in the cloud, and / or the path data may be retrieved from the network device by the NMS via an application programming interface (API) or open configuration protocol. The cloud-based NMS may store path data received from the network devices over time and thereby provide a history of application performance in the network, including on a per-site basis.

[0006] In an example where the network device includes a session-based router, a given session-based router can establish multiple peer paths with multiple other session-based routers over a WAN on a single physical interface. Each of the session-based routers can include a software agent embedded in the session-based router that is configured to report path data collected at the peer path level to the NMS in the cloud. In an example where the network device includes a packet-based router, a given packet-based router can establish multiple tunnels with multiple other packet-based routers over a single physical interface. Each of the packet-based routers can collect data at the tunnel level, and the tunnel data can be retrieved by the NMS via an API or open configuration protocol, or the tunnel data can be reported to the NMS by a software agent or other module running on the packet-based router.

[0007] According to the disclosed technology, the application health SLE measurement engine is configured to monitor the health of logical paths from network devices on a WAN and detect network failures and performance degradations that may affect the user experience. The application health SLE measurement engine uses the measurement unit of "application minutes" to measure the health status (e.g., good or bad) of each user of each application per minute, which is multiplied by the number of active users transmitting traffic through each path during the time interval as a user impact measurement. The application health SLE measurement engine can aggregate the path data received from the network device at a selected level of granularity (e.g., site level, application level, or network device level) over a selected time period (e.g., today, the past 7 days, etc.) to compile application minute measurements. The application health SLE measurement engine can determine the success or failure status associated with one or more classifiers and classify the determined failure status. Although described in "minutes", any time period can generally be used, and "time period" can be used instead of "minutes" throughout the description. Using minutes as time periods can have some advantages, but different time periods consistent with the technology of the present disclosure can be used.

[0008] The techniques of the present disclosure provide one or more technical advantages and practical applications. These techniques enable a cloud-based NMS to automatically monitor and quantify the health of applications based on path data received from network devices over time. For example, the NMS can store path data in a microservice cloud infrastructure without scaling limitations. Thus, the stored path data can provide an application performance history of network devices, which can enable an application health SLE metrics engine to identify application performance degradation and distinguish it from network failures in a way that may not be detected from an assessment based on a shorter data "snapshot", such as performed by the network device itself.

[0009] In addition, the NMS can provide users with visibility into the application health of the network by generating and outputting notifications, including identification of the root causes of any identified fault conditions. For example, the NMS can generate data representing a user interface for display on a user interface device operated, for example, by a network administrator of the network. The user interface can present the results of the root cause analysis, including classifiers of the determined fault conditions, and a timeline of faulty application minutes for each classifier during a selected time period and a selected level of granularity (e.g., site level or application level). The NMS can also generate and output notifications, such as to a network administrator of the network, and recommend execution of one or more remedial actions to resolve the determined fault conditions. In other examples, the NMS can instead automatically invoke one or more remedial actions to resolve the determined fault conditions.

[0010] In some examples, a network management system (NMS) for an enterprise network includes a memory that stores path data received from a plurality of network devices operating as network gateways for the enterprise network, the path data reported by each of the plurality of network devices for one or more logical paths from a physical interface of a given network device over a wide area network (WAN). Additionally, the NMS may include a processing circuit system in communication with the memory and configured to: determine one or more application health assessments for one or more applications based on the path data, wherein the one or more application health assessments are associated with one or more application time periods for a site, and in response to determining at least one fault state, output a notification including an identification of a root cause of the at least one fault state.

[0011] In some examples, a method includes determining, by a processing circuit system of a network management system (NMS) of an enterprise network and based on path data, one or more application health assessments for one or more applications, wherein the one or more application health assessments are associated with one or more application time periods for a site, and wherein a memory of the NMS is configured to store path data received from a plurality of network devices operating as network gateways for the enterprise network, the path data reported by each of the plurality of network devices for one or more logical paths from a physical interface of a given network device over a wide area network (WAN); and in response to determining at least one fault state, outputting, by the processing circuit system, a notification including an identification of a root cause of the at least one fault state.

[0012] In some examples, a non-transitory computer-readable medium includes instructions for causing one or more processors to perform the following: determining one or more application health assessments for one or more applications based on path data, wherein the one or more application health assessments are associated with one or more application time periods for a site, and wherein a memory of a network management system (NMS) is configured to store path data received from a plurality of network devices operating as network gateways for an enterprise network, the path data reported by each of the plurality of network devices for one or more logical paths from a physical interface of a given network device over a wide area network (WAN); and in response to determining at least one fault state, outputting a notification including an identification of a root cause of the at least one fault state.

[0013] In some examples, a network management system (NMS) for an enterprise network includes a memory that stores historical application performance data for one or more applications, historical path data for one or more logical paths from each of a plurality of network devices on a wide area network (WAN), and historical network device data for a plurality of network devices. Additionally, the NMS includes a processing circuit system in communication with the memory and configured to: identify pattern data indicative of one or more previous network problems from the historical application performance data, the historical path data, and the historical network device data, predict one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site, determine at least one configuration change for one or more network devices of the network site to remedy the potential network problem, and output a notification of at least one configuration change.

[0014] In some examples, a method includes identifying, by a processing circuit system of a network management system (NMS), pattern data from historical application performance data of one or more applications, historical path data of one or more logical paths from each of a plurality of network devices on a wide area network (WAN), and historical network device data of a plurality of network devices, wherein the path data indicates one or more previous network problems, wherein a memory of the NMS is configured to store the historical application performance data, the historical path data, and the historical network device data; predicting, by the processing circuit system, one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site; determining, by the processing circuit system, at least one configuration change for one or more network devices of the network site to remedy the potential network problem, and outputting, by the processing circuit system, a notification of at least one configuration change.

[0015] In some examples, a non-transitory computer-readable medium includes instructions for causing one or more processors to perform the following: identify pattern data from historical application performance data of one or more applications, historical path data for one or more logical paths from each of a plurality of network devices on a wide area network (WAN), and historical network device data for a plurality of network devices, wherein the path data indicates one or more previous network problems, wherein a memory of a network management system (NMS) is configured to store the historical application performance data, the historical path data, and the historical network device data; predict one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site; determine at least one configuration change for one or more network devices of the network site to remedy the potential network problem; and output a notification of at least one configuration change.

[0016] The details of one or more examples of the technology of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the technology will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figures 1A to 1D is a block diagram illustrating an example network system including a network management system (NMS) configured to monitor network performance and manage network failures in an enterprise network based on one or more application health assessments, in accordance with one or more techniques of this disclosure.

[0018] Figure 2 is a block diagram illustrating an example network device in accordance with techniques of this disclosure.

[0019] Figure 3An example NMS configured in accordance with one or more techniques of this disclosure is shown.

[0020] Figure 4 is a conceptual diagram illustrating an example of multiple sessions of multiple applications for measuring application characteristics in application minutes, in accordance with one or more techniques of this disclosure.

[0021] Figure 5 is a block diagram illustrating a system for analyzing one or more application time periods in accordance with one or more techniques of this disclosure.

[0022] Figure 6 is a block diagram illustrating a system for providing one or more recommendations and actions based on network health, in accordance with one or more techniques of this disclosure.

[0023] Figure 7 is a flow diagram illustrating example operations for classifying one or more application minutes in accordance with one or more techniques of this disclosure.

[0024] Figure 8 is a block diagram illustrating a system for determining top applications in accordance with one or more techniques of this disclosure.

[0025] Fig. 9 is a conceptual diagram illustrating a user interface screen including a first example root cause analysis performed by an NMS, in accordance with one or more techniques of this disclosure.

[0026] Fig.10 is a conceptual diagram illustrating a user interface screen including a second example root cause analysis performed by an NMS, in accordance with one or more techniques of this disclosure.

[0027] Fig.11 is a flow diagram illustrating example operations for one or more application health assessments in accordance with one or more techniques of this disclosure.

[0028] Fig.12 is a flow diagram illustrating example operations for predicting network problems and determining configuration changes in accordance with one or more techniques of this disclosure.

[0029] Like reference numerals refer to like elements throughout the drawings and the description. DETAILED DESCRIPTION

[0030] Figures 1A to 1D is a block diagram illustrating an example network system 100 including a network management system (NMS) 130 configured to monitor network performance and manage network failures in an enterprise network based on one or more application health assessments, in accordance with one or more techniques of this disclosure.

[0031] Figure 1A is a block diagram illustrating an example network system 100A according to techniques of the present disclosure. Figure 1A In the example of FIG. 1 , network system 100A includes networks 102A-102D (collectively, “network 102”) configured to provide wide area network (WAN) connections to different customer networks 104A-104B (“customer networks 104”) of an enterprise network. In some examples, network 102 is a service provider network. Although in Figure 1A In the example of FIG. 1 , the network system 100A is illustrated as including multiple interconnected networks 102 , but in other examples, the network system 100A may instead include a single network that provides connections between customer networks 104 .

[0032] Network devices 110A-110I (collectively, "network devices 110") of network 102 provide access to network 102 via customer edge devices 116A-116C (collectively, "CE devices 116") for source devices 112A and 112B (collectively, "source devices 112") and destination device 114 associated with customer network 104. The communication links between network devices 110 may be Ethernet, ATM, or any other suitable network connection.

[0033] The network device conductor 120 is a centralized management and policy engine that provides orchestration, management, and zero-touch provisioning for distributed network devices 110 while maintaining a network-wide multi-tenant service and policy data model. The network device conductor 120 can be considered an orchestrator. In some examples, the network device conductor 120 also provides monitoring and analysis for the network devices 110, while in other examples, monitoring and analysis of the network devices 110 and / or CE devices 116 are provided solely by the NMS 130. In some examples, the NMS 130 provides WAN assurance services to the network 102 and provides wireless assurance and / or wired assurance services to the customer networks 104. Figure 1A In the example of , NMS 130 includes a virtual network assistant 133 that can provide machine learning-based analysis of data collected by NMS 130 from network devices 110 of network 102 for WAN assurance services, and can provide machine learning-based analysis of data collected by NMS 130 from CE devices 116 or other customer devices within customer network 104 for wireless assurance and / or wired assurance services.

[0034] For purposes of illustration, the CE device 116 and the network device 110 are discussed herein as routers. However, the techniques of the present disclosure may be implemented using any network device, such as a switch, a router, a gateway, or other suitable network device that can send and receive network traffic. The customer networks 104 may be, for example, a network of geographically separated sites for an enterprise network. Each of the customer networks 104 may include additional customer devices, such as one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls), intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices (such as cellular phones or personal digital assistants), wireless access points, bridges, cable modems, application accelerators, or Figure 1A Other network devices not shown. Figure 1A The configuration of the network system 100A shown is merely an example. For example, the network system 100A may include any number of customer networks 104. However, for ease of description, Figure 1A Only customer networks 104A-104B are shown.

[0035] Networks 102 represent one or more publicly accessible computer networks owned and operated by one or more service providers. The service providers are typically large telecommunications entities or companies. Each of networks 102 is typically a large layer 3 (L3) computer network, where references to layers followed by numbers refer to corresponding layers in the Open Systems Interconnection (OSI) model. Each network 102 is an L3 network because it natively supports L3 operations described in the OSI model. Common L3 operations include operations performed in accordance with L3 protocols, such as the Internet Protocol (IP). L3 is also referred to as the "network layer" in the OSI model, and in this disclosure, the term L3 may be used interchangeably with the phrase "network layer."

[0036] Although not shown, each network 102 may be coupled to one or more networks managed by other providers and may thus form part of a large-scale public network infrastructure (e.g., the Internet). Thus, customer networks 104 may be viewed as edge networks of the Internet. Each network 102 may provide computing devices within customer network 104 (such as source device 112 and destination device 114) access to the Internet and may allow computing devices within customer network 104 to communicate with each other.

[0037] Although additional network devices are not shown for ease of explanation, network system 100A may include additional network and / or computing devices, such as, for example, one or more additional switches, routers, hubs, gateways, security devices (such as firewalls), intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices (such as cellular phones or personal digital assistants), wireless access points, bridges, cable modems, application accelerators, or other network devices. In addition, although the elements of network system 100A are illustrated as being directly coupled, one or more additional network elements may be included along any communication link between network devices 110 such that the network elements of computer network system 100A are not directly coupled.

[0038] Each network 102 typically provides a number of residential and business services to customer networks 104, including residential and business-type data services (commonly referred to as “Internet services” because these data services allow access to a collection of publicly accessible networks known as the Internet), residential and business-type telephone and / or voice services, and residential and business-type television services.

[0039] In some examples, the network device 110 includes a packet-based router that employs a packet-based or flow-based routing scheme to forward packets according to a defined network path established by a centralized controller (such as a software-defined network (SDN) controller) that performs path selection and traffic engineering. A given network device (e.g., network device 110A) in the network devices 110 that include a packet-based router that operates as a network gateway for the customer network 104A can establish multiple tunnels, such as Internet Protocol Security (IPsec) tunnels, with one or more other packet-based routers (e.g., network device 110I) that operate as network gateways for other sites of the enterprise network (e.g., customer network 104B) over a wide area network. As described herein, each of the packet-based routers can collect data at the tunnel level, and the tunnel data can be retrieved by the NMS 130 via an API or open configuration protocol, or the tunnel data can be reported to the NMS 130 by a software agent or other module running on the packet-based router.

[0040] In other examples, the network device 110 includes a session-based router that employs a stateful session-based routing scheme that enables each network device 110 to independently perform path selection and traffic engineering. The use of session-based routing can enable the network device 110 to avoid using a centralized controller (such as an SDN controller) to perform path selection or traffic engineering. In this way, the network device 110 can be more efficient and scalable for large networks where an SDN controller cannot be used. In addition, the use of session-based routing can enable the network device 110 to avoid using tunnels, thereby saving a large amount of network resources by eliminating the need to perform encapsulation and decapsulation at the tunnel endpoints. In some examples, the network device 110 implements session-based routing as a security vector routing (SVR) provided by Juniper Networks. A given network device (e.g., network device 110A) of network devices 110 including a session-based router operating as a network gateway for customer network 104A can establish multiple peer paths over a WAN with one or more other session-based routers (e.g., network device 110I) operating as network gateways for other sites of an enterprise network (e.g., customer network 104B). As described herein, each of the session-based routers can include a software agent embedded in the session-based router that is configured to report path data collected at the peer path level to NMS 130.

[0041] A network session (also referred to herein as a "session") includes a forward packet flow originating from a first device and destined for a second device and / or a reverse packet flow originating from a second device and destined for the first device. A session can be bidirectional in that a session can include packets traveling in both directions between a first device and a second device (e.g., a forward packet flow and a reverse packet flow).

[0042] For example, when network device 110A receives a packet of a flow originating from source device 112A and destined for destination device 114, network device 110A determines whether the packet belongs to a new session (e.g., the "first" packet or "lead" packet of a session). In some examples, network device 110A determines whether the source address, source port, destination address, destination port, and protocol of the first packet match an entry in the session table. If such an entry does not exist, network device 110A determines that the packet belongs to a new session and creates an entry in the session table. In addition, if the packet belongs to a new session, network device 110A generates a session identifier for the session. The session identifier may include, for example, the source address and source port of source device 112A, the destination address and destination port of destination device 114, and the protocol used by the first packet. Network device 110A may use the session identifier to identify subsequent packets belonging to the session.

[0043] In some examples, network device 110 performs stateful routing for a session. This means that network device 110 forwards each packet of a forward packet flow of the session sequentially and along the same forward network path. As described herein, the "same" forward path refers to the same network device 110 that forms a segment or at least a portion between the device from which the packet originated and the device to which the packet is destined (and not necessarily the entire network path between the device from which the packet originated and the device to which the packet is destined). In addition, network device 110 forwards each packet of a return flow of the session sequentially and along the same return network path. The forward network path of the forward packet flow and the return network path of the return flow can be the same path or different paths. By ensuring that each packet of the flow is forwarded sequentially and along the same path, network device 110 maintains the state of the entire flow at each network device 110, thereby enabling the use of stateful packet services, such as deep packet inspection (DPI).

[0044] exist Figure 1AIn the example of , a stateful routing session can be established from the ingress network device 110A through the intermediate network devices 110B-110H to the egress network device 110I. In this example, the network device 110A determines that the first packet is an unmodified packet and is the first packet of the new session. The network device 110A modifies the first packet to include metadata (e.g., the original source address, source port, destination address, and destination port) that specifies the session identifier. The network device 110A replaces the header of the modified first packet to specify the source address, source port, destination address, and destination port, wherein the source address is the address of the network device 110A, the source port is the port via which the network device 110A forwards the modified first packet to the destination device 114, the destination address is the address of the next hop to which the network device 110A forwards the first packet (e.g., the address of the network device 110B), and the destination port is the port of the next hop to which the network device 110A forwards the first packet (e.g., the port of the network device 110B).

[0045] The network device 110A may also identify a network service associated with the session. For example, the network device 110A may compare one or more of the source address, source port, destination address, or destination port for the session with a service address and port information table to identify the service associated with the session. Examples of network services include Hypertext Transfer Protocol (HTTP), firewall services, proxy services, packet monitoring or measurement services, etc. For example, if the source port and / or destination port for the session is 80, the network device may determine that the session is associated with HTTP. In other examples, the network device 110A may determine that one or more of the source address, source port, destination address, or destination port for the session belongs to an address block or port block indicating that a particular service is associated with the session.

[0046] In some examples, network device 110A uses the network service determined for the session to select a forward path to forward the first packet and each subsequent packet to destination device 114. In this way, network device 110A can perform service-specific path selection to select a network path that best suits the service requirements. Compared to network topologies that use an SDN controller to perform path selection, each network device 110 performs path selection. In addition, the use of session-based routing enables each network device 110 to make routing decisions at the service or application level, compared to traditional network devices that can only make routing decisions at the flow level.

[0047] Network device 110A forwards the modified first packet to network device 110B. In addition, network device 110A stores the session identifier for the session so that upon receiving subsequent packets for the session, network device 110A can identify the subsequent packets as belonging to the same session and forward the subsequent packets along the same path as the first packet.

[0048] Intermediate network device 110B receives the modified first packet and determines whether the modified first packet includes a portion of metadata specifying a session identifier. In response to determining that the modified first packet includes metadata specifying the session identifier, intermediate network device 110B determines that network device 110B is not an ingress device, such that network device 110B does not append metadata specifying the session identifier.

[0049] As described above with respect to network device 110A, network device 110B determines whether the packet belongs to a new session (e.g., is the "first" packet or "leader" packet of a session) by determining whether the source address, source port, destination address, destination port, and protocol of the first packet match an entry in the session table. If such an entry does not exist, network device 110B determines that the packet belongs to a new session and creates an entry in the session table. In addition, if the packet belongs to a new session, network device 110B generates a session identifier for the session. The session identifier used by network device 110B to identify the session for the first packet may be different from the session identifier used by network device 110A to identify the same session for the first packet, because each network device 110A, 110B uses the header source address, source port, destination address, and destination port of the first packet to generate the session identifier, and when each network device 110 forwards the first packet along the forward path, this information is modified by each preceding network device 110. Additionally, each network device 110 may store the header information to identify the previous network device 110 (or "waypoint") and the next network device 110 (or "waypoint") so that each network device 110 may reconstruct the same forward and reverse paths for each subsequent packet of the session.

[0050] Network device 110B replaces the header of the modified first packet to specify a source address, a source port, a destination address, and a destination port, wherein the source address is the address of network device 110B, the source port is the port via which network device 110B forwards the modified first packet to destination device 114, the destination address is the address of the next hop to which network device 110B forwards the first packet (e.g., the address of network device 110C), and the destination port is the port of the next hop to which network device 110B forwards the first packet (e.g., the port of network device 110C). Network device 110B forwards the modified first packet to network device 110C. In addition, network device 110B stores a session identifier for the session so that upon receiving subsequent packets for the session, network device 110B can identify the subsequent packets as belonging to the same session and forward the subsequent packets along the same path as the first packet.

[0051] Subsequent intermediate network devices 110C-110H process the modified first packet in a similar manner as network devices 110A and 110B, such that network devices 110 forward subsequent packets of the session along the same path as the first packet. In addition, each network device 110 stores a session identifier for the session, which may include an identification of a previous network device 110 along the network path. Thus, each network device 110 may forward packets of a reverse packet flow of the session along the same network path back to source device 112A using the session identifier.

[0052] The network device 110 that can forward packets for the forward packet flow for the session to the destination for the packet flow is an egress or "terminal" network device. In the above example, the network device 110I is a terminal network device because the network device 110I can forward the packet to the CE device 116C for forwarding to the destination device 114. The network device 110I receives a modified first packet that includes metadata (e.g., the original source address, source port, destination address, and destination port) that specifies the session identifier. The network device 110I identifies the modified first packet as being destined for a service that terminates at the network device 110I by determining that the destination source address and destination source port specified in the metadata of the modified leader packet correspond to a destination that is reachable by the network device 110I (e.g., a destination device 114 that is reachable via the CE device 116C). The network device 110I restores the original first packet by removing the metadata from the modified first packet and modifying the header of the first packet to specify the original source address, source port, destination address, and destination port. Network device 110I forwards the recovered first packet to CE device 116C for forwarding to destination device 114 .

[0053] Additional information about session-based routing and SVR is described in the following: U.S. Patent No. 9,729,439, entitled “COMPUTER NETWORK PACKET FLOW CONTROLLER,” issued on August 8, 2017; U.S. Patent No. 9,729,682, entitled “NETWORK DEVICE AND METHOD FOR PROCESSING ASESSION USING A PACKET SIGNATURE,” issued on August 8, 2017; U.S. Patent No. 9,762,485, entitled “NETWORK PACKET FLOW CONTROLLER WITH EXTENDED SESSION MANAGEMENT,” issued on September 12, 2017; U.S. Patent No. 9,762,485, entitled “ROUTER WITH OPTIMIZED U.S. Patent No. 9,871,748, entitled “STATISTICAL FUNCTIONALITY”; U.S. Patent No. 9,985,883, issued on May 29, 2018, entitled “NAME-BASEDROUTING SYSTEM AND METHOD”; U.S. Patent No. 10,200,264, issued on February 5, 2019, entitled “LINK STATUS MONITORING BASED ON PACKET LOSS DETECTION”; U.S. Patent No. 10,277,506, issued on April 30, 2019, entitled “STATEFUL LOAD BALANCING IN A STATELESS NETWORK”; U.S. Patent No. 10,277,506, issued on October 1, 2019, entitled “NETWORK PACKET FLOWCONTROLLER WITH EXTENDED SESSION No. 10,432,522, entitled “IN-LINE PERFORMANCE MONITORING” and U.S. Patent Application Publication No. 2020 / 0403890, published on December 24, 2020, and entitled “IN-LINE PERFORMANCE MONITORING,” the entire contents of which are incorporated herein by reference.

[0054] In some examples, to implement session-based routing, each network device 110 maintains a local repository of service and topology state information for each other network device 110. The service and topology state information includes services reachable from each network device 110, and the network topology from each network device used to reach these services. Each network device 110 can transmit changes in services reachable from the network device 110 and / or changes in the network topology used to reach the services from the network device to a central repository, such as a server. In addition, each network device 110 can receive service and topology state information for each other network device 110 in the computer network system 100A from the central repository.

[0055] In the above example, the network device 110A receives a packet, determines a session for a packet flow including the packet, determines a service associated with the session, and selects a network path for forwarding the packet. The network device 110A can use its local copy of the service and topology state information for each network device 110 to select a network path for forwarding the packet. For example, the network device 110A can use the identified service associated with the packet and the network topology for reaching the identified service to select a network path that meets the service level agreement (SLA) requirements or other performance requirements of the service. The network device 110A can then forward the packet and subsequent packets for the flow along the selected path. In this way, the network device 110A can perform service-specific path selection because the network device 110 can use the service-specific criteria associated with the packet to select the network path that best suits the requirements of the service.

[0056] Additional information regarding the exchange of service and topology state information is described in U.S. Patent Application Publication No. 2020 / 0366590, entitled “CENTRAL AUTHORITY FOR SERVICE AND TOPOLOGY EXCHANGE,” published on November 19, 2020; U.S. Patent Application Publication No. 2020 / 0366599, entitled “SOURCE-BASED ROUTING,” published on November 19, 2020; U.S. Patent Application Publication No. 2020 / 0366598, entitled “SERVICE AND TOPOLOGY EXCHANGE PROTOCOL,” published on November 19, 2020; U.S. Patent Application Publication No. 2020 / 0366589, entitled “ROUTING USING SEGMENT-BASED METRICS,” published on November 19, 2020; and U.S. Patent Application Publication No. 2020 / 0366589, entitled “NETWORK CONTROL SYSTEM,” filed on July 31, 2018. NEIGHBORHOODS FORESTABLISHING COMMUNICATION RELATIONSHIPS BETWEEN COMMUNICATION INTERFACES IN AN ADMINISTRATIVE DOMAIN”, which is incorporated herein by reference in its entirety.

[0057] According to the techniques of the present disclosure, NMS 130 is configured to monitor application performance and manage application failures that may affect user experience in the enterprise network (e.g., experience of source device 112 and / or destination device 114 in customer network 104) based on application data and path data received from one or more network devices 110 operating as gateways for an enterprise network. NMS 130 receives path data from network devices 110 and stores the received path data over time in a database 135. The path data indicates one or more aspects of network performance monitored on each logical path (e.g., a peer path or tunnel) between network devices 110 on a WAN (e.g., a broadband network, a long term evolution (LTE) network, or a multi-protocol label switching (MPLS) network). NMS 130 includes a virtual network assistant 133 having an application health service level expectation (SLE) metric engine 134 that determines one or more application health assessments based on the path data received from the network devices 110. Based on the application health assessment, NMS 130 may identify success or failure states associated with application interfaces and / or paths, identify root causes of one or more failure states, and / or automatically recommend or invoke one or more remedial actions to resolve the identified failure states.

[0058] In some examples, the application health service level expectation (SLE) metric engine 134 is configured to determine one or more application health assessments for one or more applications based on the path data. In some examples, the application health assessment may represent an assessment of the quality of service provided by the application to the client. In some examples, the source device 112 may represent one or more client devices, and the destination device 114 may represent an application server. In some examples, the source device 112 may represent one or more application servers, and the destination device 114 may represent a client device. In any case, the application server may provide services to the client device via the path formed by the network device 110. This means that the path data indicated by the network device 110 may indicate the "health" of the application. The health of the application may represent the quality of service received by the client from the application server through the path formed by the network device 110 at any given time.

[0059] The term "path data" may be used herein to describe one or more data indicating the characteristics of a network connection between two or more nodes of an indication network. That is, the path data may include data indicating the quality of a network connection (e.g., a path via a network device 110) through which a client device communicates with an application server. For example, the path data may include delay values, jitter values, and loss values. Delay, jitter, and loss may indicate the quality of a network connection because these parameters indicate the speed at which a packet traverses a path and the reliability at which a packet traverses a path and reaches a destination. For example, a delay may represent the amount of time a packet takes to travel from one network node to another network node, or the amount of time a packet takes to travel back and forth. Jitter may represent the variability of delays in a packet stream. Losses may represent the rate at which a packet does not reach a destination or arrives at a destination with a delay. This indicates that the application health SLE engine 134 may analyze delays, jitters, and losses to determine the health of one or more applications.

[0060] In addition to or in lieu of delay, jitter, and loss, the path data may also include one or more other types of data. For example, the path data may include a first time to data packet value, data indicating one or more transmission control protocol (TCP) resets, data indicating one or more TCP retransmissions, data indicating the amount of bandwidth used by one or more applications, mean opinion score (MOS) data, low-level transport layer metrics, or any combination thereof. The path data evaluated by the application health SLE engine 134 may include any type of data corresponding to any portion of the path between the client device and the application server. The application health SLE engine 134 may analyze the path data to determine the quality or health of the path.

[0061] The network device 110 may send the path data to the NMS 130. The NMS 130 may store the path data in a database 135. In some examples, the path data may correspond to a point in time or a time period. That is, the quality of service provided by the application server to the client device may change (e.g., improve or decrease) over a period of time, and the change in quality may be reflected in the path data. For example, the delay on the path of the network device 110 may increase over a period of time, indicating that the quality of service provided via the path is decreasing. The application health SLE engine 134 may monitor the quality of the application over time based on the path data.

[0062] In some examples, one or more application health assessments performed by the application health SLE engine 134 are associated with one or more application time periods corresponding to the application. For example, an application time period may correspond to a time period during which an application server provides services to a client device via a network path on the network device 110. In some examples, an application health assessment may include an assessment of one or more continuous application time periods. For example, an application server may provide services through the network device 110 for a duration that includes a sequence of continuous application time periods. The application health SLE engine 134 may perform an application health assessment to determine the quality of service during each application time period of a sequence of continuous application time periods. In some examples, each application time period of the sequence of application time periods includes the same amount of time. An application time period may represent the quality of application service over a period of time. In some examples, an application time period may be referred to herein as an "application minute". An application time period is not limited to indicating application health within a one-minute time period. An application time period may indicate the health of an application within any time period.

[0063] In some examples, the application health assessment may include a classification of whether an application time period is "good" or "bad". In some examples, good application time periods are referred to as high-quality application time periods. In some examples, bad application time periods may be referred to herein as low-quality application time periods. In some examples, the application health assessment may include more classifications than good or bad. For example, in some examples, the application health assessment may include a classification of whether an application time period is good, average, or bad. It may be beneficial for the application health SLE engine 134 to determine the quality of each application time period in order to identify low-quality application time periods, thereby identifying these time periods and identifying the reasons for poor network service.

[0064] In some examples, the application health SLE engine 134 may apply one or more thresholds to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period. In some examples, the one or more thresholds may include a threshold corresponding to each path data type in a set of path data types. For example, when the path data includes delay, jitter, and loss, the application health SLE engine 134 may apply a delay threshold, a jitter threshold, and / or a loss threshold to classify the application time period as high quality or low quality. In some examples, in addition to or in lieu of delay, jitter, and loss, the application health SLE engine 134 may also apply one or more thresholds corresponding to the path data type. In some examples, the application health SLE engine 134 may determine one or more thresholds based on the path data and historical application performance data. For example, the application health SLE engine 134 may identify a delay threshold based on a historical delay value corresponding to the application.

[0065] In some cases, the application health SLE engine 134 can classify each low-quality (e.g., "bad") application time period based on the identified application-related problem classifier based on the path data stored in the database 135. In some examples, in addition to whether the application is good or bad, the application-related problem classifier can also include other information. For example, the application-related problem classifier can include a slow application problem classifier corresponding to a slow response time from an application server, a TCP retransmission problem classifier corresponding to one or more retries from a client device or application server caused by service unavailability, an application bandwidth problem classifier corresponding to a bandwidth threshold amount required by the application corresponding to the corresponding application time period, or an application disconnection problem classifier corresponding to frequent disconnections from a client device or application server. That is, the application-related problem classifier can indicate the cause or source of the poor application service. In some examples, the application health SLE engine 134 can apply one or more thresholds to classify each low-quality application time period based on the identified application-related problem classifier.

[0066] In some examples, NMS 130 can execute a machine learning model to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period. In some examples, NMS 130 can execute a machine learning model to classify each low-quality (e.g., "bad") application time period with an application-related problem classifier. In some examples, NMS 130 can train one or more machine learning models based on the path data stored in database 135.

[0067] The NMS 130 can provide a cloud service that brings automated operations and service levels to the enterprise access layer at the WAN edge and, when used in conjunction with wired and wireless assurance, transforms all operations covering switches, IoT devices, access points, servers, printers, etc. The gateway provides rich streaming telemetry data that enables insights into gateway health metrics and anomaly detection. After the network device 110 is logged into the cloud-based NMS 130 (as described in more detail below), the network device 110 provides streaming telemetry data to the cloud-based NMS 130.

[0068] In some examples, telemetry data includes data related to application response time and WAN link and gateway health measurements. For gateway health, data may include, for example, CPU utilization, memory utilization, link utilization, temperature, fans, power. For WAN link health, data may include, for example, IPSec information, routing protocol information, and WAN interface information. Application experience information may include, for example, latency, jitter, packet loss, round-trip time, and other metrics on a per-application basis.

[0069] In some examples, NMS 130 is configured to identify pattern data indicating one or more previous network problems from historical application performance data, historical path data, and historical network device data. In some examples, the historical application performance data includes performance data corresponding to one or more applications. In some examples, the historical path data includes performance data corresponding to a network path between two or more network devices. In some examples, the historical network device data corresponds to performance data corresponding to one or more network devices (e.g., a gateway device).

[0070] In some examples, NMS 130 may predict one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with the network site. For example, historical application performance data, historical path data, and historical network device data may include one or more trends corresponding to healthy network performance and one or more trends corresponding to poor network performance. In some examples, historical application performance data for one or more applications includes one or more application health assessments associated with one or more application time periods for the network site. In some examples, NMS 130 may determine that one or more potential network problems may occur or are occurring based on the identified pattern data and current application performance data, current path data, and current network device data. NMS 130 may determine at least one configuration change for one or more network devices (e.g., network device 110) of the network site to remedy the potential network problem.

[0071] NMS 130 may output a notification of at least one configuration change. In some examples, the notification includes a recommendation to update a configuration of one or more network devices of network device 110 according to the at least one configuration change. In some examples, the notification includes an instruction to update a configuration of one or more network devices of network device 110 according to the at least one configuration change, and the one or more network devices may implement the at least one configuration change. In some examples, NMS 130 may automatically update a configuration of one or more network devices of network device 110 according to the at least one configuration change. In some examples, outputting a notification of at least one configuration change may include outputting the notification via a user interface for display on a user interface device of an administrator associated with system 100A.

[0072] In some examples, NMS 130 is configured to generate a machine learning model based on identified pattern data associated with a specific application deployed at a network site. NMS 130 can generate a machine learning model to reflect one or more patterns corresponding to good network performance and one or more patterns corresponding to poor network performance. NMS 130 can use the machine learning model to predict potential network problems for a specific application deployed at a network site. In some examples, the machine learning model used by NMS 130 to determine potential network problems is different from the machine learning model that NMS 130 can use to classify application time periods.

[0073] In some examples, the machine learning model used by NMS 130 to determine potential network problems may include a self-learning model. NMS 130 may provide a continuous feedback mechanism for the determined at least one configuration change and the subsequently collected application performance data, path data, and network device data. In other words, the machine learning model may accept as input feedback data indicating whether the configuration change suggested by the machine learning model results in an improvement in network performance.

[0074] In some examples, at least one configuration change for one or more network devices of network device 110 includes a routing threshold configuration based on a service level agreement (SLA). In some examples, at least one configuration change for one or more network devices of network device 110 includes a packet path configuration change. In some examples, at least one configuration change for one or more network devices of network device 110 includes one or more other types of configuration changes.

[0075] Figure 1B It is a graphic Figure 1A A block diagram of additional example details of a network system 100B is shown. In this example, Figure 1BThe NMS 130 is illustrated and is configured to operate according to an artificial intelligence / machine learning based computing platform that provides wireless network 173 and wired LAN 175 ( Figure 1B ) to cloud-based application services 181 ( hosted by computing resources within data center 179 Figure 1B The far right of the diagram shows the complete automation, insight, and assurance of the network (e.g., Wireless Assurance, Wired Assurance, and / or WAN Assurance). Figure 1A , the user device 171 may include one or more of the source device 112 and the destination device 114, and the wired LAN 175 hosting the wireless network 173 may include one or more customer networks 104 of the enterprise network.

[0076] As described herein, NMS 130 provides an integrated set of management tools and implements various technologies of the present disclosure. In general, NMS 130 can provide a cloud-based platform for wireless network data collection, monitoring, activity logging, reporting, predictive analysis, network anomaly identification, and alarm generation. For example, NMS 130 can be configured to actively monitor and adaptively configure network system 100B to provide autonomous driving capabilities. In addition, VNA 133 includes a natural language processing engine to provide AI-driven support and troubleshooting, anomaly detection, AI-driven location services, and AI-driven RF optimization using reinforcement learning.

[0077] like Figure 1B As shown in the example of , the AI-driven NMS 130 also provides configuration management, monitoring, and automated supervision of a software-defined wide area network (SD-WAN) 177, which operates as an intermediate network to communicatively couple the wireless network 173 and the wired LAN 175 to the data center 179 and application services 181. In general, the SD-WAN 177 provides seamless, secure, traffic-engineered connectivity between the "spoke" routers 187A of the edge wired network 175 of a hosted wireless network 173 (such as a branch or campus network) (such as the customer network 104 as a site for the enterprise network in FIG. 1 ) and the "hub" routers 187B further up the cloud stack toward the cloud-based application services 181. Again, referring to Figure 1A , routers 187A, 187B may include a network device 110 that operates as a network gateway for an enterprise network.

[0078] SD-WAN 177 typically operates and manages an overlay network on an underlying physical wide area network (WAN) that provides connectivity to geographically separate customer networks (e.g., Figure 1AIn other words, the SD-WAN 177 can extend SDN capabilities and / or session-based routing or SVR capabilities to the WAN to allow the network to decouple the underlying physical network infrastructure from the virtualized network infrastructure and applications so that the network can be configured and managed in a flexible and scalable manner.

[0079] In some examples, the underlying routers of the SD-WAN 177 can implement a stateful session-based routing scheme, in which routers 187A, 187B dynamically modify the contents of the original packet headers sourced by the user device 171 to direct traffic along a selected path (e.g., peer path 189) to the application service 181 without the use of tunnels and / or additional labels. In this way, routers 187A, 187B can be more efficient and scalable for large networks because the use of tunnel-free session-based routing can enable routers 187A and 187B to obtain a large amount of network resources by eliminating the need to perform encapsulation and decapsulation at the tunnel endpoints. In addition, in some examples, each router 187A, 187B can independently perform path selection and traffic engineering to control the packet flow associated with each session without the need to use a centralized SDN controller for path selection and label distribution. In some examples, routers 187A, 187B implement session-based routing as SVR provided by Juniper Networks.

[0080] Additional information about session-based routing and SVR is described in the following: U.S. Patent No. 9,729,439, entitled “COMPUTER NETWORK PACKET FLOW CONTROLLER,” issued on August 8, 2017; U.S. Patent No. 9,729,682, entitled “NETWORK DEVICE AND METHOD FOR PROCESSING A SESSIONUSING A PACKET SIGNATURE,” issued on August 8, 2017; U.S. Patent No. 9,762,485, entitled “NETWORK PACKET FLOW CONTROLLER WITH EXTENDED SESSION MANAGEMENT,” issued on September 12, 2017; U.S. Patent No. 9,762,485, entitled “ROUTER WITH OPTIMIZED U.S. Patent No. 9,871,748, entitled “STATISTICAL FUNCTIONALITY”; U.S. Patent No. 9,985,883, issued on May 29, 2018, entitled “NAME-BASEDROUTING SYSTEM AND METHOD”; U.S. Patent No. 10,200,264, issued on February 5, 2019, entitled “LINK STATUS MONITORING BASED ON PACKET LOSS DETECTION”; U.S. Patent No. 10,277,506, issued on April 30, 2019, entitled “STATEFUL LOAD BALANCING IN A STATELESS NETWORK”; U.S. Patent No. 10,277,506, issued on October 1, 2019, entitled “NETWORK PACKET FLOWCONTROLLER WITH EXTENDED SESSION No. 10,432,522, entitled “IN-LINE PERFORMANCE MONITORING” and U.S. Patent Application Publication No. 2020 / 0403890, published on December 24, 2020, and entitled “IN-LINE PERFORMANCE MONITORING,” the entire contents of which are incorporated herein by reference.

[0081] In some examples, the AI-driven NMS 130 can enable intent-based configuration and management of the network system 100B, including enabling the construction, presentation, and execution of intent-driven workflows for configuring and managing devices associated with the wireless network 173, the wired LAN network 175, and / or the SD-WAN 177. For example, declarative requirements express the desired configuration of network components without specifying the exact local device configuration and control flow. By using declarative requirements, it is possible to specify what should be done, rather than how it should be done. Declarative requirements can be contrasted with imperative instructions that describe the exact device configuration syntax and control flow used to implement the configuration.

[0082] By utilizing declarative requirements rather than imperative instructions, users and / or user systems can reduce the burden of determining the exact device configuration required to achieve the desired results of the user / system. For example, when using various types of devices from different vendors, it is often difficult and cumbersome to specify and manage the exact imperative instructions for configuring each device of the network. As new devices are added and device failures occur, the types and types of network devices may change dynamically. It is often difficult to manage various types of devices from different vendors, grammars, and software versions with different configuration protocols to configure a cohesive device network. Therefore, by only requiring the user / system to specify declarative requirements that specify the desired results applicable to various types of devices, the management and configuration of network devices becomes more efficient. Additional example details and techniques of intent-based network management systems are described in U.S. Patent No. 10,756,983 entitled "Intent-based Analytics" and U.S. Patent No. 10,992,543 entitled "Automatically generating an intent-based network model of an existing computer network", both of which are incorporated herein by reference.

[0083] According to the technology described in the present disclosure, NMS 130 is configured to monitor application performance and manage application failures that may affect the user experience in the enterprise network based on path data received from one or more network devices operating as network gateways (e.g., routers 187A, 187B) for the enterprise network. NMS 130 receives path data from routers 187A, 187B through the underlying physical WAN, the path data indicating one or more aspects of application performance monitored on each logical path 189 (e.g., peer path or tunnel) between routers 187A, 187B in SD-WAN 177, and stores the path data in database 135 over time.

[0084] The NMS 130 includes a virtual network assistant 133 having an application health SLE metric engine that can be configured to determine one or more application health assessments based on the path data in the database 135. The application health SLE metric engine can aggregate the path data at a selected level of granularity (e.g., site level or network device level) over a selected time period. The application health SLE metric engine can determine a success or failure status associated with one or more classifiers.

[0085] Application health SLE can measure the real-time experience of applications. Application health SLE can be different from WAN link SLE. For example, a low-quality network connection can provide a high-quality user experience for non-real-time applications such as YouTube or File Transfer Protocol (FTP) applications, but provide a low-quality user experience for applications based on real-time protocols such as Voice over Internet Protocol (VoIP) and Web Real-Time Communications (WebRTC).

[0086] In some examples, the unit of measurement for the Application Health SLE is "session minutes". For example, a Wi-Fi Successful Connection SLE can measure client connection sessions, e.g., whether a pre-connection session (based on state machine logic) takes 5 seconds or 5 minutes, they are weighted equally. In comparison, the Application Health SLE measures streaming session minutes, so each streaming session has a different weight based on its duration, e.g., if a Zoom call session lasts 30 minutes, it is measured as 30 different session minutes, while a 1 minute HTTP session is only measured as 1 session minute.

[0087]

[0088]

[0089] Table 1

[0090] Table 1 provides a mapping of one or more SLE names to measurement units. For example, the "Connection Time" SLE name corresponds to the "Client (Connection Session)" measurement unit, and the "Application Health" SLE name corresponds to the "Session (Flow) Minutes" measurement unit.

[0091] Figure 1C is a diagram illustrating one or more techniques according to the present disclosure Figure 1B 100B is a block diagram of additional example details of a regional hub deployment model of a network system 100B. Specifically, Figure 1C Picture shows Figure 1B177. In the example shown, the SD-WAN 177 includes spoke routers 187A within a branch office that are connected to a hub router 187B in a data center via logical paths 189 on an underlying physical WAN (e.g., an MPLS network 188). The SD-WAN 177 also includes hosted or software as a service (SaaS) applications.

[0092] When troubleshooting SD-WAN issues, it can be helpful to separate the problem into three parts: 1) branch office, 2) logical path (e.g., peer path or tunnel) on the WAN, such as MPLS, LTE, or broadband network, and 3) application services, including internally hosted applications (e.g., in a data center) and SaaS applications. NMS 130 can be configured to track the time connection topology of these three segments for each customer deployment, and also detect various types of user-impacting problems in the virtual network assistant 133. By joining the connection topology with the corresponding events occurring in each segment, the virtual network assistant 133 of NMS 130 can be able to pinpoint the location and root cause of different user-impacting SD-WAN problems. Examples of user-impacting problems for the branch office segment can include device health, bad cables, and configuration issues (e.g., maximum transmission unit (MTU)). Examples of user-impacting problems for the logical path segment can include link connectivity and link performance degradation. Examples of user-impacting problems for the application service segment can include service reachability and service performance.

[0093] According to the techniques described in the present disclosure, the virtual network assistant 133 of the NMS 130 has a WAN link health SLE metric engine that is configured to monitor the health of logical paths from spoke routers (e.g., logical path 189 from router 187A) and detect network failures and performance degradations that may affect user experience. The WAN link health SLE metric engine uses a measurement unit of user path minutes to measure the health status (e.g., success and failure) of each user of each logical path per minute, which is multiplied by the number of active users passing through each path during the time interval as a user impact measurement. The WAN link health SLE metric engine can aggregate path data received from network devices (e.g., routers 187A, 187B) at a selected granularity level (e.g., site level or network device level) over a selected time period. The WAN link health SLE metric engine can determine a success or failure status associated with one or more of service provider reachability, physical interface operation, or logical path performance based on the aggregated path data, and classify the determined failure status. Some examples of failure conditions (ie, what conditions should be considered a failed user path minute) are as follows: ISP unreachable, logical path down, logical path performance degraded, interface oversubscribed, interface error, and / or interface signal strength weak / unstable.

[0094] Several high-level design considerations are described herein. In some examples, the WAN link health SLE measurement engine is configured to measure the health of logical path segments on WAN 188 (which may be broadband, LTE, or MPLS) between a spoke router 187A in a branch office and a hub router 187B in a data center, but may not measure the health of connections from the data center to application servers or the health of the application services themselves. In some examples, the WAN link health SLE measurement engine is configured to measure the health of logical path segments from spoke routers (e.g., spoke router 187A) in a branch office, but may not measure the health of hub routers (e.g., hub router 187B) in the data center.

[0095] Network devices may collect logical path statistics via Bidirectional Forwarding Detection (BFD) probes, which are typically sent via low priority traffic classes. Therefore, logical path statistics may not always represent the real user experience at different application levels. For example, for a best-effort traffic class, a logical path may have lower performance and therefore be determined to have poor or failed user path minutes, but since the user application sessions are sent via higher priority traffic classes, the poor performance of the best-effort traffic class may not have any real impact on the user. In some cases, this may result in the discovery of a "bad WAN link health SLE" but a "good application health SLE". In addition, network devices (e.g., session-based routers) may treat all available links (e.g., LTE, broadband, or MPLS) as active links and may monitor logical path statistics on each link. Therefore, the WAN link health SLE measurement engine can detect and report link failures even if no user traffic was sent over a particular link during the failure interval.

[0096] In the SD-WAN regional hub deployment model, since branch office traffic can go directly to the Internet for SaaS applications and can also be routed to the data center for internal applications, it can be beneficial to configure the branch office router's probe destination accordingly to cover both internal and Internet applications.

[0097] Figure 1D is a diagram illustrating one or more techniques according to the present disclosure Figure 1B A block diagram of additional example details of the spoke hub deployment model of the network system 100B is provided. Specifically, Figure 1D Picture shows Figure 1B 177. In the example shown, the SD-WAN 177 includes spoke routers 187A within a branch office that are connected to a hub router 187B in a data center via a point-to-point connection 189 on the underlying physical WAN.

[0098] In some examples, for the spoke hub deployment model, all branch office traffic is routed to the data center before entering the Internet. In some examples, the packet path from the branch office to the data center and to the Internet is divided into two parts, which are analyzed separately. These segments can include a first segment from the branch office to the data center (e.g., between router 187A and router 187B), and a second segment between the data center and the Internet. In some examples, each spoke router can be configured to detect a hub router in the data center, and the hub router can be configured to detect different application services in the Internet. In some examples, the NMS 130 can analyze each of the first segment and the second segment to identify the root cause of one or more bad application time periods identified by the application health SLE engine 134.

[0099] Figure 2 is a block diagram illustrating an example network device 200 according to the techniques of the present disclosure. In general, the network device 200 may be Figure 1A A network device among the network devices 110 or Figure 1B-Figure 1D 187A, 187B. In this example, network device 200 includes interface cards 226A-226N ("IFC 226") that receive packets via incoming links 228A-228N ("incoming links 228") and send packets via outgoing links 230A-230N ("outgoing links 230"). IFC 226 is typically coupled to links 228, 230 via a plurality of interface ports. Network device 200 also includes a control unit 202 that determines a route for a received packet and forwards the packet accordingly via IFC 226.

[0100] The control unit 202 may include a routing unit 204 and a forwarding unit 222. The routing unit 204 operates as a control plane for the network device 200 and includes an operating system that provides a multi-tasking operating environment for the execution of multiple concurrent processes. The routing unit 204 communicates with other routers, such as Figure 1A Network equipment 110 to establish and maintain computer networks, such as Figure 1A-Figure 1CThe network system 100 is configured to transmit network traffic between one or more client devices. The routing protocol daemon (RPD) 208 of the routing unit 204 executes software instructions to implement one or more control plane network protocols 212. For example, the protocol 212 may include one or more routing protocols, such as the Internet Group Management Protocol (IGMP) 221 and / or the Border Gateway Protocol (BGP) 220, to exchange routing information with other routing devices and update the routing information base (RIB) 206, the Multi-Protocol Label Switching (MPLS) protocol 214, and other routing protocols. The protocol 212 may also include one or more communication session protocols 223, such as TCP, UDP, TLS, or ICMP. The protocol 212 may also include one or more performance monitoring protocols, such as BFD 225.

[0101] The RIB 206 may describe the topology of the computer network in which the network device 200 is located, and may also include routes through a shared tree in the computer network. The RIB 206 describes various routes within the computer network, and the appropriate next hop for each route, i.e., the adjacent routing devices along each route. The routing unit 204 analyzes the information stored in the RIB 206 and generates forwarding information for the forwarding unit 222 stored in the forwarding information base (FIB) 224. The FIB 224 may associate, for example, a network destination with a specific next hop and a corresponding IFC 226 and a physical output port of an output link 230. The FIB 224 may be a radix tree, a series of tables, a complex database, a linked list, a radix tree, a database, a flat file, or various other data structures programmed into a dedicated forwarding chip.

[0102] FIB 224 may also include a lookup structure. Given a key (such as an address), the lookup structure may provide one or more values. In some examples, the one or more values ​​may be one or more next hops. The next hop may be implemented as a microcode that performs one or more operations when implemented. One or more next hops may be "chained" so that a set of chained next hops perform a set of operations for corresponding different next hops when implemented. Examples of such operations may include applying one or more services to a packet, dropping a packet, and / or forwarding a packet using an interface identified by one or more next hops.

[0103] The session information 235 stores information for identifying a session. In some examples, the session information 235 takes the form of a session table. For example, the service information 232 includes one or more entries specifying a session identifier. In some examples, the session identifier includes one or more of a source address, a source port, a destination address, a destination port, or a protocol associated with a forward flow and / or a reverse flow of the session. As described above, when the routing unit 204 receives a packet of a forward packet flow originating from a client device (e.g., the source device 112A of FIG. 1 ) and destined for another client device (e.g., the destination device 114 of FIG. 1 ), the routing unit 204 determines whether the packet belongs to a new session (e.g., the "first" packet or the "leading" packet of the session). In order to determine whether the packet belongs to a new session, the routing unit 204 determines whether the session information 235 includes an entry corresponding to the source address, source port, destination address, destination port, and protocol of the first packet. If the entry exists, the session is not a new session. If the entry does not exist, the session is new, and the routing unit 204 generates a session identifier for the session and stores the session identifier in the session information 235. Thereafter, routing unit 204 may use the session identifier stored in session information 235 to identify subsequent packets belonging to the same session.

[0104] The service information 232 stores information that the routing unit 204 can use to identify the service associated with the session. In some examples, the service information 232 takes the form of a service table. For example, the service information 232 includes one or more entries that specify a service identifier, and one or more of a source address, source port, destination address, destination port, or protocol associated with the service. In some examples, the routing unit 204 can query the service information 232 using one or more of the source address, source port, destination address, destination port, or protocol of the session of the received packet to determine the service associated with the session. For example, the routing unit 204 can determine the service identifier based on the correspondence between the source address, source port, destination address, destination port, or protocol in the service information 232 and the source address, source port, destination address, destination port, or protocol specified by the session identifier. The routing unit 204 retrieves one or more service policies 234 corresponding to the identified service based on the service associated with the packet. The service policy may include, for example, a path failover policy, a dynamic host configuration protocol (DHCP) tag policy, a traffic engineering policy, a priority of network traffic associated with the session, and the like. Routing unit 204 applies one or more service policies 234 corresponding to the services associated with the slave packet to the packet.

[0105] In some examples, the network device 200 may include a session-based router that employs a stateful session-based routing scheme that enables the routing unit 204 to independently perform path selection and traffic engineering. The use of session-based routing can enable the network device 200 to avoid using a centralized controller (such as an SDN controller) to perform path selection and traffic engineering, and avoid the use of tunnels. In some examples, the network device 200 can implement session-based routing as a security vector routing (SVR) provided by Juniper Networks. In the case where the network device 200 includes a session-based router that operates as a network gateway for a site of an enterprise network, the network device 200 can establish multiple peer paths through the underlying physical WAN with one or more other session-based routers that operate as network gateways for other sites of the enterprise network.

[0106] Although the network device 200 is primarily described herein as a session-based router, in other examples, the network device 200 may include a packet-based router, wherein the routing unit 204 employs a packet-based or flow-based routing scheme to forward packets according to a defined network path, such as a path established by a centralized controller that performs path selection and traffic engineering. In the case where the network device 200 includes a packet-based router that operates as a network gateway for a site of an enterprise network, the network device 200 may establish multiple tunnels with one or more other packet-based routers that operate as network routers for other sites of the enterprise network through the underlying physical WAN.

[0107] According to the technology of the present disclosure, path data may include periodically reported data and event-driven data. The control unit 202 of the network device 200 is configured to collect logical path statistics via BFD 225 detection and data extracted from messages and / or counters at the logical path (e.g., peer path or tunnel) level. In some examples, the control unit 202 is configured to collect statistics and / or sample other data according to a first periodic interval (e.g., every 3 seconds, every 5 seconds, etc.). The control unit 202 may store the collected and sampled data as path data in, for example, a buffer. In some examples, the path data agent 238 may periodically create statistical data packets according to a second periodic interval (e.g., every 3 minutes). The collected and sampled data periodically reported in the statistical data packet may be referred to as "oc-stats" in this article. In some examples, the statistical data packet may also include details about the client and associated client session connected to the network device 200. The path data agent 238 may then report the statistical data packet to the NMS 130 in the cloud. In other examples, the NMS 130 may request, retrieve, or otherwise receive statistics packets from the network device 200 via an API, an open configuration protocol, or another communication protocol 223. The statistics packets created by the path data agent 238 or another module of the control unit 202 may include a header identifying the network device 200 and statistics and data samples for each logical path from the network device 200. In another example, the path data agent 238 reports event data to the NMS 130 in the cloud in response to the occurrence of certain events at the network device 200 when the event occurs. Event-driven data may be referred to herein as "oc-events."

[0108] Figure 3 An example network management system (NMS) 300 configured according to one or more techniques of the present disclosure is shown. NMS 300 may be used to implement, for example Figure 1A-Figure 1D In these examples, NMS 300 is responsible for monitoring and managing Figure 1A The network devices 110A to 110I of the network 102, Figures 1B to 1D Router 187A, 187B or Figure 2 One or more of the network devices 200.

[0109] In this example, the NMS 300 receives path data collected by the network devices 110A-110N. The path data may include statistics and data samples periodically reported at the logical path (e.g., peer path or tunnel) level, such as telemetry data and data extracted from messages and / or counters. In some examples, the path data may also include details about clients connected to the network device 110. In other examples, the path data may include event-driven data reported in response to the occurrence of certain events at the network device 110. The NMS 300 uses the path data to calculate one or more SLE metrics in order to monitor the health of the logical paths from the network devices 110 through the underlying physical WAN and detect network failures and performance degradations that may affect the user experience. In some examples, the NMS 300 may be a Figure 1A-Figure 1C A server that is part of a microservices cloud infrastructure within or accessible by the network system 100.

[0110] In some examples, in addition to monitoring network devices 110, NMS 300 is also responsible for monitoring and managing one or more wireless or wired networks (e.g., Figure 1B In this example, the NMS 300 also receives information from user devices (e.g., wireless networks 173 and wired LANs 175), as well as network devices of monitoring service providers or other networks. Figure 1B The user device 171 of the access point receives data collected by the access point, such as data used to calculate one or more SLE metrics, and analyzes the data for cloud-based management of the wireless network. In this way, a single NMS 300 can be used to manage both the network devices 110 and the wireless network for an end-to-end WAN assurance system viewable via a single cloud-based WAN assurance portal, where the network devices may include virtualized network devices (e.g., software-based routers executed on virtual machines or containers).

[0111] The NMS 300 includes a communication interface 330, one or more processors 306, a user interface 310, a memory 312, and a database 318. The various elements are coupled together via a bus 314, through which the various elements can exchange data and information. The processor(s) 306 execute software instructions, such as instructions defining a software or computer program, stored in a computer-readable storage medium such as the memory 312, such as a non-transitory computer-readable medium including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions for causing the one or more processors 306 to perform the techniques described herein.

[0112] The communication interface 330 may include, for example, an Ethernet interface. The communication interface 330 couples the NMS 300 to a network and / or the Internet, such as any of the network(s) 102 shown in FIG. 1 , and / or any wide area network or local area network. The communication interface 330 includes a receiver 332 and a transmitter 334, through which the NMS 300 transmits / receives data and information to / from any network device 110 and / or any other device or system forming part of a network 102 or 104, such as shown in FIG. 1 . For example, the data and information received by the NMS 300 may include SLE-related or event log data received from the network device 110, and is used by the NMS 300 to remotely monitor the performance of the network device 110 and the network 102. In some examples, the NMS may also transmit data to any network device 110 via the communication interface 330 to remotely manage the network 102.

[0113] The memory 312 includes one or more devices configured to store programming modules and / or data associated with the operation of the NMS 300. For example, the memory 312 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions for causing the one or more processors 306 to perform the techniques described herein.

[0114] In this example, the memory 312 includes the API 220, a virtual network assistant (VNA) / AI engine 350 including an application health SLE engine 352 and a root cause analysis engine 370, and an ML model 356. The NMS 300 may also include any other programming modules, software engines, and / or interfaces configured for remote monitoring and management of the network devices 110, including remote monitoring and management of any network device 110. The NMS 300 may also include any other programming modules, software engines, and / or interfaces configured for remote monitoring and management of wireless networks, including remote monitoring and control of any access point.

[0115] The VNA / AI engine 350 analyzes the path data 318 received from the network device 110 and its own data to identify when an unexpected or abnormal state is encountered in one of the networks 102. For example, the VNA / AI engine 350 can use the root cause analysis module 354 to identify the root cause of any unexpected or abnormal state. In some examples, the root cause analysis module 354 uses artificial intelligence-based technology to help identify the root cause of any (multiple) bad SLE metrics at one or more networks 102. In addition, the VNA / AI engine 350 can automatically call one or more corrective actions, which are intended to address the identified (multiple) root causes of one or more bad SLE metrics. Examples of corrective actions that can be automatically called by the VNA / AI engine 350 may include, but are not limited to, calling an API 320 to restart one or more network devices 110. Corrective actions may also include restarting switches and / or routers, calling new software to download network devices, switches or routers, and the like. These corrective actions are for example purposes only, and the present disclosure is not limited in this regard. If automatic corrective actions are not available or do not adequately address the root cause, the VNA / AI engine 350 can proactively provide notifications, including recommended corrective actions for IT personnel to take to resolve the network error.

[0116] In some examples, the VNA / AI engine 350 can construct, train, apply, and retrain supervised and / or unsupervised ML model(s) 356 on event data (e.g., SLE metrics 316) to determine whether the collected network event data represents abnormal behavior that needs to be further analyzed by the root cause analysis 354 of the VNA / AI engine 350 to facilitate the identification and resolution of faults. The VNA / AI engine 350 can then apply the ML model 356 to data streams and / or logs of newly collected data (e.g., path data 318) of various network event types (e.g., connection events, and / or statistics and data extracted from messages, counters, etc.) to detect whether the currently observed network event data with the incoming data stream indicates normal operation of the system, or whether the incoming network event data indicates an atypical system behavior event or trend corresponding to a faulty network that needs to be mitigated.

[0117] When the application of the ML model 356 to the path data 318 indicates that mitigation is needed, the VNA / AI engine 350 can invoke the root cause analysis 354 to identify the root cause of the abnormal system behavior and trigger automatic or semi-automatic corrective actions where possible. In this way, the VNA / AI engine 350 can construct and apply the ML model 356 based on the specific complex network to determine whether to perform further resource-intensive analysis on the incoming path data streams collected (e.g., in real time) from network devices within the complex network system.

[0118] According to the technology of the present disclosure, the application health SLE engine 352 determines one or more application health assessments for one or more applications. For example, the application health SLE engine 352 can analyze SLE-related data (e.g., path data 318) collected by the network device 110. For example, the NMS 300 receives path data 318 from the network device 110, which indicates one or more aspects of network performance monitored on each logical path (e.g., peer path or tunnel) between the network devices 110 in the SD-WAN on the underlying physical WAN, and stores the path data 318 in the database 315 over time. The path data 318 may include periodically reported data and event-driven data. For example, the NMS 300 may receive path data 318 as a statistical data packet from each network device 110 at periodic intervals (e.g., every 3 minutes). The portion of the path data 318 that is periodically reported in the statistical data packet may be referred to as "oc-stats" herein. In some examples, the statistical data packet may also include details about the client and related client sessions connected to the network device 110. The statistics data packet received from each network device 110 may include a header identifying the corresponding network device 110 and a plurality of statistics and data samples for each logical path. In some examples, the path data 318 may include event-driven data received from the network device 110 in response to the occurrence of certain events at the network device 110 when the event occurs. The portion of the path data 318 that includes event-driven data may be referred to herein as "oc-events". In some examples, the NMS 300 may store the path data 318 in a database of a microservice cloud infrastructure with no scaling limitations.

[0119] In some examples, path data 318 may include data indicating the quality of a network connection (e.g., a path via network device 110) through which a client device communicates with an application server. For example, path data may include delay values, jitter values, and loss values. Delay, jitter, and loss may indicate the quality of a network connection because these parameters indicate the speed at which packets traverse a path and the reliability with which packets traverse a path and arrive at a destination. For example, delay may represent the amount of time it takes for a packet to travel from one network node to another, or the amount of time it takes for a packet to travel back and forth. Jitter may represent the variability of delays in a packet stream. Losses may represent the rate at which packets do not arrive at a destination or arrive at a destination with delays. This indicates that the application health SLE engine 352 may analyze delays, jitter, and loss to determine the health of one or more applications.

[0120] In addition to or in lieu of delay, jitter, and loss, the path data may also include one or more other types of data. For example, the path data may include a first time to data packet value, data indicating one or more TCP resets, data indicating one or more TCP retransmissions, data indicating the amount of bandwidth used by one or more applications, MOS data, low-level transport layer metrics, or any combination thereof. The path data evaluated by the application health SLE engine 352 may include any type of data corresponding to any portion of the path between the client device and the application server. The application health SLE engine 352 may analyze the path data to determine the quality or health of the path.

[0121] The NMS 300 may execute the application health SLE engine 352 to determine one or more application health assessments of the one or more applications based on the path data 318. In some examples, each of the one or more application health assessments is associated with one or more application time periods. The application time period corresponds to the performance of the application over a period of time.

[0122] Application health SLEs (e.g., WAN assurance), WAN actions (e.g., bad WAN links, bad application services), and conversational interfaces (e.g., gateway troubleshooting, application session troubleshooting) are all target network issues that cause application user experience issues, but they are slightly different:

[0123] Application Health SLE can cover both persistent and intermittent WAN links located in spoke routers, hub routers, or application servers. In some examples, the Application Health SLE engine 352 can detect network problems, including jitter, delay, and loss. The affected items can include interfaces, applications, or users / clients. WAN actions can include persistent WAN link network problems that can lead to poor application user experience, such as slow application servers, slow spoke hub tunnels, etc. The conversation interface can include persistent and intermittent network problems on end-to-end networks (including wireless, LAN, WAN, and data center SaaS).

[0124] See U.S. patent application Ser. No. 17 / 554,928, filed on December 17, 2021, entitled “NETWORK PERFORMANCE MONITORING AND FAULT MANAGEMENT BASED ON WIDE AREA NETWORK LINK HEALTH ASSESSMENTS” [Attorney Docket No. 2014-475US01], the entire contents of which are incorporated herein by reference.

[0125] In some cases, the application health SLE engine 352 may identify one or more streaming sessions corresponding to the user experience that needs to be tracked, and the application health SLE engine 352 may determine whether the streaming session is "good" or "bad" in terms of the end user experience based on the streaming session. In addition, for each "bad" user experience streaming session, the application health SLE engine 352 may determine the root cause of the bad user experience. For example, the application health SLE engine 352 may identify which part of the network causes the bad user experience.

[0126] The application health SLE engine 352 can troubleshoot one or more poor user experience streaming sessions by separating network problems from application problems. For example, if the connection corresponding to an application (e.g., a VoIP application) is poor, the application health SLE engine 352 can determine whether the problem is caused by the network link or not. The application health SLE engine 352 can identify the problem in the network that caused the poor flow. For example, the application health SLE engine 352 can identify the node where the problem exists.

[0127] In some examples, the suggestion and prediction engine 353 can provide one or more suggestions for resolving network problems. For example, the suggestion and prediction engine 353 can output a suggestion to increase site bandwidth based on identifying a trend of poor application performance. The suggestion and prediction engine 353 can predict one or more applications that are expected to have problems in the network in the future. The one or more suggestions output by the suggestion and prediction engine 353 can include configuration changes to overcome application health issues, such as changing the vector of a service and changing the quality of service (QoS) of a service.

[0128] It may be beneficial for the NMS 300 to identify one or more top (e.g., most popular, most used) application lists. This may help the NMS 300 predict when certain applications may encounter problems.

[0129] In some examples, the suggestion and prediction engine 353 is configured to identify pattern data indicating one or more previous network problems from the application performance data, the historical path data, and the historical network device data. The suggestion and prediction engine 353 can predict one or more potential network problems based on the identified pattern data and the current application performance data, the current path data, and the current network device data associated with the network site.

[0130] In some examples, the suggestion and prediction engine 353 can determine at least one configuration change for one or more network devices of the network site to remedy the potential network problem. In some examples, the suggested configuration change can represent a configuration change to one or more network devices, such as (network device 110 and / or router 187). The suggestion and prediction engine 353 can output a notification of the at least one configuration change.

[0131] Due to the variety of applications, it may be difficult for each customer to manually define and maintain a list of most interesting applications. Based on historical telemetry data, it may be beneficial to dynamically learn and update the list of most interesting applications for each customer and each site periodically. Popularity and application category are two factors used to determine the list of popular applications.

[0132] Popularity can be determined based on flow count and flow duration. For example, the text below can indicate the top ten applications based on one day's data, ranked only by flow count. In some examples, the top ten applications account for the majority (e.g., 85%) of the total flow sessions. In some examples, one or more flow sessions correspond to short-lived infrastructure flows, such as Domain Name System (DNS) flows, Internet Control Message Protocol (ICMP) flows, and Network Time Protocol (NTP) flows, which may not be sensitive to the actual user experience. As shown below, BitTorrent can be ranked second in the list of the top ten applications due to the large number of short-lived connection attempt flows generated by P2P protocols.

[0133]

[0134] The following text may indicate the top ten applications based on one day's data, ranked only by flow duration (e.g., flow minutes). The following list may include long-lived flows such as Secure Sockets Layer (SSL) flows and Secure Shell Protocol (SSH) flows, which may be relevant to user experience, but lack user-sensitive applications.

[0135]

[0136] The following text includes a list of the top ten applications ranked by application category (e.g., games, multimedia, messaging) and stream duration (e.g., streams longer than one minute). The list includes one or more long-lived streaming applications that are sensitive to user experience. In some examples, the list below includes these applications that account for a relatively small percentage (e.g., 3%) of the total number of streaming sessions.

[0137]

[0138] The most interesting applications can be dynamically updated at each level of site based on historical usage patterns. In some examples, the most interesting applications can be selected based on popularity and application category. In some cases, popularity can be determined based on flow count and / or flow duration. In some cases, the top ten applications selected based on popularity and category can significantly reduce the load and cost of application health, WAN health, and gateway health.

[0139] In some examples, streaming sessions can be the highest volume source of telemetry data in an SD-WAN, and for high-end devices deployed in the core or data center, not every streaming session is sent to the cloud. Since only the most interesting application sessions can be consumed in SLE and Marvis, filtering out irrelevant streams at an early stage in the telemetry pipeline can be preferred to save resources in both the device and the cloud. In some examples, streaming events can be collected because of the ability to send update events for long-lived sessions.

[0140] In some examples, one or more network devices can support customized flow filtering logic on the device. The network device can filter the flow session directly inside the router, so that the router can save resources on the device and in the cloud. This is a more preferred and scalable solution.

[0141] In some examples, one or more network devices may not provide a per-session level performance metric mean opinion score (MOS). In some examples, the determination may be made at a per-application level rather than a per-session level. For example, if there are ten concurrent video calls corresponding to a same branch site and passing through a network device, then the ten calls may be determined to be all "good" or all "bad" without determining whether each individual call is good or bad.

[0142] In some examples, the application health SLE engine can track three different network performance metrics, including jitter, delay, and loss, to determine session quality. In some examples, one or more applications have requirements for a good user experience. For example, a file transfer application may be more tolerant of high jitter and high delay than a VoIP application or a gaming application.

[0143] Applications may not have vendor-specified network performance metrics. Rather than requiring customers to manually specify thresholds for each application (as we do in Wi-Fi SLE), the NMS 300 can dynamically learn network performance thresholds for each application. Application performance requirements may be independent of the customer or geographic location. The NMS 300 can learn these thresholds by combining customer data collected in the NMS world. For each application, the system can analyze the distribution of network performance metrics collected from different customer sites and use statistical methods to determine a global threshold.

[0144] In some examples, thresholds for different network performance metrics including jitter, delay, and loss can be determined at a per-application level and can be shared among different customers. The thresholds can be determined based on application vendor-specified requirements and thresholds dynamically learned from NMS customer data.

[0145] Based on determining that the streaming session is associated with a poor user experience, the VNA / AI engine 350 can use root cause analysis 354 to identify the root cause of the poor user experience. For example, the VNA / AI engine 350 can identify the portion of the network that has performance issues. In some examples, WAN assurance identifies WAN-side network issues that may have one or more segments with one or more deployment models. It is important to intelligently set up probes to cover different segments to pinpoint the root cause.

[0146] In some examples, the most interesting application learning workflow for each site may require performing cloud-based learning and baselining work, including learning a list of most interesting applications for each site based on historical streaming session data. In some examples, user-defined applications can determine the final list, which is pulled down to the streaming event data filtering, querying using GraphDB to identify the role of each router (spoke, hub) in the deployment, and dynamically determining the probe target IP based on the device role and other per-application properties learned from the data: probe packet size, probe packet protocol.

[0147] In some examples, each application performance learning workflow can perform a per-application network performance learning job that determines the quality of each session minute in the application health SLE. This is a global learning job for the entire NMS world, and its results apply to all different sites.

[0148] In some examples, the graph correlation and ingestion workflow can process and correlate telemetry data collected from different devices (e.g., client devices, access points (APs), switches, routers, and third-party application services) to build a temporal network topology for each streaming session. The graph correlation and ingestion workflow can ingest all relevant data into ArangoDB in batch mode.

[0149] In some examples, the application health topology represents an example topology of the application health SLE engine 352. The application health topology can correlate the stream analysis and real-time performance monitoring (RPM) statistics in the oc-stats analysis in the data stream processing, and use the most interesting application list from the previous offline job and the performance threshold of each application to determine the quality of each streaming session minute. The results of the application health topology can be used for application SLE aggregation.

[0150] In some examples, a network administrator may be interested in determining the performance of one or more applications in a network. In some examples, application performance may depend on underlying network conditions. In some cases, one or more applications may perform poorly even when the underlying network conditions are of high quality. In some cases, one or more applications may perform well even when the underlying network conditions are of low quality. It may be beneficial for the system to distinguish application-related issues from network issues in order to troubleshoot issues encountered by a user.

[0151] In some examples, the application health SLE engine 352 can identify application-related issues as separate from underlying network issues. For example, the application health SLE engine 352 can identify "application minutes" as a unit for measuring the performance of an application over a period of time (e.g., one minute). Application minutes are not limited to one-minute time periods. In some examples, application minutes may be referred to herein as "application time periods." Application time periods may represent the performance of an application over any time period (e.g., 10 seconds, 30 seconds, 1 minute, 5 minutes, 1 hour, or any other time period).

[0152] In one example, an application minute can be identified by four tuples, including the IP address of the client device corresponding to the application minute, the name of the application (e.g., Teams, YouTube, Zoom, Spotify), the network path corresponding to the application minute (e.g., local routing, peer path), and the traffic class corresponding to the application (e.g., low quality of service, medium quality of service, high quality of service).

[0153] Figure 4 is a conceptual diagram illustrating an example of multiple sessions of multiple applications for measuring application characteristics in application minutes according to one or more techniques of the present disclosure. For example, Figure 4 Included are a client device 402 , a network device 404 , a first application 406 , and a second application 408 .

[0154] In some examples, application minute 410 may correspond to a link between client device 402 and first application 406 via network device 404. During application minute 410, first session 412 may exist between client device 402 and first application 406. In some examples, application minute 420 may correspond to a link between client device 402 and second application 408 via network device 404. During application minute 420, first session 422 may exist between client device 402 and second application 408, and second session 424 may exist between client device 402 and second application 408. In some examples, application minute 430 may correspond to a link between client device 402 and first application 406 via network device 404. During application minute 430, second session 432 may exist between client device 402 and first application 406, and third session 434 may exist between client device 402 and first application 406.

[0155] App minutes can represent the user experience regarding the amount of time a user spends on an app. Figure 4 As shown, a single application minute can have one or more data sessions flowing as part of the same application minute. For example, application minute 410 can correspond to a single session 412, and application minute 420 can correspond to two sessions 422 and 424. In some examples, "session minutes" or "session flow minutes" can represent application minutes that use only a single session.

[0156] Although described herein as "minutes", application minutes may correspond to any time period, and throughout the description, the phrase "time period" may replace "minutes". Using minutes as the time period for application minutes may have some advantages, but different time periods may be used according to the technology of the present disclosure.

[0157] User experience statistics for each session corresponding to each application can be collected and aggregated into time period based "buckets" corresponding to application minutes. Based on the collected data, application minutes can be classified as "good" or "bad" and further divided into different classifier buckets for ease of use.

[0158] Figure 5 is a block diagram illustrating a system 500 for analyzing one or more application time periods according to one or more techniques of the present disclosure. Figure 5As shown, system 500 includes devices 502A-502N (collectively referred to as "devices 502"), pipeline 504, network data 506, database 507, machine learning system 508 including machine learning engine 509 and machine learning model 510, application SLE engine 512, and classifier 514. In addition, system 500 includes application servers 522A-522N (collectively referred to as "application servers 522"), user data 524, application information extractor 526, and database 528.

[0159] In some examples, device 502 may correspond to a device (e.g., a network device) within a WAN, through which a client device establishes a connection with an application server, and through which the application server provides services to the client device. Fig.10 That is, device 502 may be part of a path within the WAN through which source device 112A communicates with destination device 114. In some examples, Figure 1A The source device 112 is a client device, and the destination device 114 represents an application server. Figure 1A The source device 112 may represent an application server, and the destination device 114 may represent a client device. In some examples, the device 502 may correspond to Figure 1B-Figure 1D router 187 and / or any other device within SD-WAN 177.

[0160] The network data 504 may include data indicating the quality of a network connection (e.g., a WAN) through which a client device communicates with an application server. For example, the network data 504 may include delay, jitter, and loss. Delay, jitter, and loss may indicate the quality of a network connection because these parameters indicate how packets travel in a network connection. In order to ensure that the application server provides a high quality of service to the client device, packets must traverse the network connection in an efficient manner. Delay, jitter, and loss may provide a picture of the efficiency with which packets traverse the network connection. In some examples, delay, jitter, and loss are forwarding plane manager (FPM) values.

[0161] Forwarding plane manager (FPM) metrics may represent in-band statistics used to determine the state of a peer path between two network devices. When a session smart router transmits session data, it may add additional information to determine peer router data. More details about the forwarding plane manager are described in U.S. Patent No. 11,075,824, filed on July 27, 2021, entitled “IN-LINE PERFORMANCE MONITORING,” [Attorney Docket No. 2014-332US01], the entire contents of which are incorporated herein by reference.

[0162] Details of using session information to obtain application records are described in U.S. patent application Ser. No. 17 / 452,344, filed on October 26, 2021, entitled “APPLICATION RECORDS USING SESSION INFORMATION,” [Attorney Docket No. 2014-494US01], the entire contents of which are incorporated herein by reference.

[0163] The delay represents the amount of time it takes for a data packet to travel from point to point on a network. In some examples, the delay may describe the amount of time it takes for a packet to travel between any two nodes within the network. In some examples, the delay may describe the amount of time it takes for a packet to make a round trip from a first device to a second device and back to the first device. In some examples, the delay may describe the amount of time it takes for a first packet to travel from a first device to a second device and for a second packet to travel from the second device back to the first device in response to the second device receiving the first packet. The delay value corresponding to the application minute may correspond to the median or average of the amount of time it takes for a packet to travel between any two points in a network connection during the application minute. For example, during the application minute, a sequence of packets may travel between a first network device and a second network device. The delay corresponding to the application minute may represent the average amount of time or mean amount of time it takes for a packet in a sequence of packets to travel from a first network device to a second network device. In some examples, a higher delay time may correspond to a lower quality of service, while a lower delay time may correspond to a higher quality of service.

[0164] Jitter may indicate the degree to which the amount of time a packet takes to travel between two nodes in a network connection is inconsistent. For example, during an application minute, a sequence of packets may travel between a first network device and a second network device. The jitter corresponding to the application minute may indicate the level of variability in the amount of time it takes each packet in the sequence of packets to travel from the first network device to the second network device. In some examples, higher jitter may correspond to a lower quality of service, while lower jitter may correspond to a higher quality of service.

[0165] Losses (also referred to herein as "packet loss") may correspond to a rate at which packets of a packet stream do not arrive at a destination, arrive at a destination out of order, or arrive at a destination too late. In some examples, a high quality service may involve a steady stream of packets, where packets predictably arrive at a destination in order and on time. In some examples, a low quality service may involve an "unstable" stream of packets with packet loss. Thus, a higher rate of packet loss may correspond to a lower quality of service, while a lower rate of packet loss may correspond to a higher quality of service.

[0166] Pipeline 506 can pass data to one or more engines for processing. For example, pipeline 506 can pass network data 504 and / or other data to database 507. In some examples, database 507 can be stored in NMS 130. In some examples, database 507 is stored within database 135 of NMS 130. In some examples, database 507 is stored separately from database 135 within NMS 130. Pipeline 506 can represent any type of connection (e.g., wired or wireless) configured to pass data to database 507 or any other device. For example, Figure 1A The NMS 130 is illustrated as including connections to each of the network devices 110. These connections may be part of a pipeline 506 that delivers the network data 504 to the databases of the NMS 130 and / or other devices.

[0167] The machine learning system 508 may include a machine learning engine 509 and a machine learning model 510. In some examples, the machine learning engine 509 may be configured to execute the machine learning model 510 to generate one or more outputs based on input data. In some examples, the input data may represent data from one or more databases (e.g., database 507 and / or database 528). In some examples, the machine learning model 510 may be stored by a network management system (e.g., NMS 130), and the network management system may execute the machine learning model 510 using the machine learning engine 509.

[0168] In some examples, the machine learning engine 509 can execute the machine learning model 510 to generate an output based on the input data stored by the database 507. In some examples, the database 507 can include network data 504 corresponding to one or more application time periods. The application time period can represent a time period in which a client device receives a service from an application server. As described above, the network data 504 can include data indicating the quality of one or more application minutes. The machine learning engine 509 can execute the machine learning model 510 based on the network data 504 to generate an output indicating whether each of the one or more application minutes is of high quality (e.g., good) or low quality (e.g., bad).

[0169] In some examples, each of the one or more applications may have a different tolerance for each FPM metric. For example, an application may correspond to a threshold delay value, a threshold jitter value, and a threshold loss value. If one or more of the delay value, jitter value, and loss value corresponding to an application minute exceeds the corresponding threshold delay value, threshold jitter value, and threshold loss value, the user may experience poor service. The machine learning engine 509 may execute a machine learning model 510 to determine whether one or more application minutes are of high quality or low quality based on the network data 504 including delay, jitter, and loss data.

[0170] When the machine learning engine 509 executes the machine learning model 510 to generate an output that classifies each of the one or more application minutes as high quality or low quality, the machine learning system 508 can send the output to the application SLE engine 512. The application SLE engine 512 can receive the output from the machine learning system 508. In some examples, the application SLE engine 512 can additionally or alternatively receive data from the pipeline 506, including the network data 504 corresponding to the one or more application minutes. The application SLE engine 512 can generate an output based on the output from the machine learning system 508 and / or the data received from the pipeline 506, which further classifies each application minute determined to be of low quality (e.g., bad).

[0171] The application SLE engine 512 may classify each application minute determined to be of low quality (e.g., bad) as being associated with one or more problem classifiers from a set of problem classifiers. In some examples, the set of problem classifiers may include a slow application problem classifier corresponding to a slow response time from an application server, a TCP retransmission problem classifier corresponding to one or more retries from a client device or application server caused by service unavailability, an application bandwidth problem classifier corresponding to a bandwidth below a threshold amount of bandwidth required by an application corresponding to a corresponding application time period, and an application disconnection problem classifier corresponding to frequent disconnections from a client device or application server.

[0172] In some examples, the slow response time, retransmission frequency, amount of bandwidth, and application disconnection can be part of the network data 504 received by the application SLE engine 512. In some examples, the application SLE engine 512 can determine any one or a combination of the slow response time, retransmission frequency, amount of bandwidth, and application disconnection based on FPM metrics (such as delay, jitter, and loss) of the network data 504. In any case, the application SLE engine 512 can further classify each low-quality (e.g., bad) application minute with one or more problem classifiers from a set of problem classifiers. In some examples, the classifier 514 includes a problem classification for each low-quality (e.g., bad) application minute.

[0173] The system 500 can determine an initial classification of whether each of the one or more application minutes is good or bad, and determine a further problem classification for each bad application minute. That is, the system 500 can identify bad application minutes and provide an indication of the cause of the low quality application service. This means that the system 500 can provide information to the network administrator about what caused the user to experience poor application service from one or more applications.

[0174] Some applications can receive user feedback. In some cases, VoIP applications (e.g., Microsoft Teams, Zoom) can output prompts to ask users to provide feedback on the quality of service experienced by the user. For example, the application can output a request to ask the user to rate a video call from one to five stars. In some examples, the application can output a request to ask the user to describe one or more problems with the service. In some examples, the application can output a list of possible problems and ask the user to select from the list the problem that the user encountered during use of the application. In some examples, the machine learning system 508 can receive user feedback data corresponding to one or more applications and generate output based on the user feedback data.

[0175] In some cases, application server 522 can provide services to one or more user devices. In some examples, application server 522 can be Figure 1A In some examples, the application server 522 may be an example of a source device 112A and / or a destination device 114. Figure 1B An example of a multi-cloud application 181 and services can be provided to one or more user devices 171 via the SD-WAN 177.

[0176] The user data 524 may represent user feedback corresponding to one or more applications. The user data 524 may include user feedback indicating the quality of user experience when the application is used. For example, the user data 524 may include user feedback about one or more calls made through a VoIP application or another application requesting user feedback. The application information extractor 526 may receive the user data 524 from the application server 522 and forward the user data 524 to the database 528. In some examples, the machine learning engine 509 may be configured to execute the machine learning model 510 to generate an output based on the data (including the user data 524) received from the database 528. That is, the machine learning engine 509 may be configured to execute the machine learning model 510 to determine whether each of the one or more application minutes is of high quality (e.g., "good") or low quality (e.g., "bad"). In some examples, the machine learning system 508 may classify each of the one or more application minutes based on data from the database 528 and the database 507. That is, in addition to analyzing network data 504 in database 507, machine learning system 508 may also analyze user data 524 in database 528 to classify each of the one or more application minutes.

[0177] The machine learning system 508 need not analyze the user data 524 stored in the database 528 to classify the application minutes as high quality or low quality. In some examples, the user data 524 may represent supplemental data that the machine learning system 508 analyzes in addition to the network data 504 stored in the database 507. For example, there may be both user data 524 and network data 504 corresponding to the same one or more application minutes, and the machine learning system 508 may analyze both the user data 524 and the network data 504 to classify the one or more application minutes. In some examples, the machine learning system 508 may analyze the network data 504 to classify the one or more application minutes without analyzing any user data. In some examples, the machine learning system 508 may analyze the user data 524 to classify the one or more application minutes without analyzing any network data.

[0178] In some examples, application server 522 can host one or more different types of applications, including VoIP applications or collaborative SaaS applications (e.g., Zoom, Microsoft Teams, Webex) data center applications, productivity SaaS applications (e.g., Google Workspace), video streaming applications (such as YouTube), or any combination thereof. In some examples, different types of applications can be configured to tolerate different amounts of network delay, jitter, and loss. The following table indicates one or more maximum network performance parameter values ​​corresponding to different applications.

[0179]

[0180] Table 2

[0181] In some examples, the delay value, jitter value, and / or loss value may indicate a poor connection between the client device and the application server. In some examples, one or more disconnections or timeouts may be indicated by the delay value, jitter value, and / or loss value. In some examples, slow loading times may be indicated by the delay value, jitter value, and / or loss value.

[0182] In some examples, the system 500 may evaluate one or more application minutes based on the type of application associated with the one or more application minutes. For example, the system 500 may analyze application minutes corresponding to a video streaming application based on a bandwidth level corresponding to the application minute, a delay or round trip time (RTT) corresponding to the application minute, and a packet loss corresponding to the application minute. In some examples, the system 500 may analyze application minutes corresponding to a web browsing application based on a bandwidth level corresponding to the application minute, a delay or RTT corresponding to the application minute, and a packet loss corresponding to the application minute. In some examples, the system 500 may analyze application minutes corresponding to an interactive application based on a bandwidth level corresponding to the application minute, a delay or RTT corresponding to the application minute, and a packet loss corresponding to the application minute.

[0183] In some cases, the system 500 may evaluate one or more application minutes based on the data path routing of the application associated with the one or more application minutes. In some examples, the system 500 may analyze the application minutes corresponding to the client-to-gateway data path routing based on the bandwidth level corresponding to the application minute, the delay or RTT corresponding to the application minute, and the packet loss corresponding to the application minute. In some examples, the system 500 may analyze the application minutes corresponding to the data path routing between gateway routers based on the bandwidth level corresponding to the application minute, the delay or RTT corresponding to the application minute, and the packet loss corresponding to the application minute. In some examples, the system 500 may analyze the application minutes corresponding to the gateway router-to-application server data path routing based on the bandwidth level corresponding to the application minute, the delay or RTT corresponding to the application minute, and the packet loss corresponding to the application minute.

[0184] Each application may have a different tolerance for each of the above application-related problem classifiers in a set of application-related problem classifiers. For example, a set of application-related problem classifiers may include a slow application problem classifier, a TCP retransmission problem classifier, an application bandwidth problem classifier, and an application disconnection problem classifier. In some examples, one application may have a greater tolerance for low bandwidth than another application.

[0185] The machine learning model 510 can analyze application minutes based on rich network information, such as latency, jitter, loss, low-level transport layer metrics, link metrics, and labeled data about real user experience extracted from application servers. In some examples, the machine learning model 510 can predict real-time application experience, label application minutes as good or bad, and classify them into different classifiers.

[0186] In some examples, an application health SLE (e.g., application SLE engine 512 and / or machine learning system 508) can measure a real-time user experience corresponding to one or more applications. Application health SLE can be different from WAN link SLE. A low-quality network connection may cause some types of applications to perform poorly without affecting one or more other types of applications. For example, non-real-time applications such as YouTube or FTP can perform well even if the network connection quality is low, but real-time applications such as VoIP applications and WebRTC applications may perform poorly when the network connection quality is low.

[0187] The application health SLE (e.g., the application SLE engine 512 and / or the machine learning system 508) can associate one or more FPM (e.g., delay, jitter, loss) metrics with one or more session flow records. The application health SLE can mark flows as good or bad based on parameters in in-band peer path statistics (e.g., FPM statistics) and TCP statistics. That is, the application health SLE can determine application performance at a per-device, per-site, or organizational level.

[0188] Application Health SLE can use session statistics from network devices to classify one or more application minutes. For example, application name, service name, and path are examples of session statistics. Application Health SLE can also use advanced metrics, such as FPM statistics such as TCP retransmissions and TCP first time to data packets. Application Health SLE can use FPM statistics to calculate delay, jitter, and loss between two or more network nodes. FPM metrics can be determined using in-band metadata attached to packet samples passed between nodes.

[0189] In some examples, applications can have different tolerance limits for latency, jitter, and loss. High latency can cause poor performance for real-time streaming applications, while jitter can cause poor performance for non-real-time streaming applications (such as YouTube). Within the same category of live streaming, tolerance limits can vary from application to application. In some examples, the system can use a threshold-based approach to classify packet streams as good, fair, or poor.

[0190] In some examples, one or more session smart routers may provide FPM data including delay, jitter, and loss. For example, each of the network devices 110 may be configured to send FPM data to the NMS 130. In some examples, the NMS 130 may be configured to store the FPM data in a database 135. In some examples, the network device may provide session records for one or more services. In some examples, the NMS 130 may receive session data for all flows. This configuration may be set as a default configuration or enabled as part of a loading process. Configuration-based filtering logic may be supported. The NMS 130 may receive application profile data. The application profile data may be tracked using the TCP retransmission counts present in the session statistics. The total number of packets transmitted may be used to calculate the percentage of retransmissions within a given period of time.

[0191] In some examples, peer path metrics and session flow records may indicate gateway-to-gateway flows and gateway-to-application flows. For example, a gateway-to-gateway flow may extend from router 187A to router 187B, and a gateway-to-application flow may extend from router 187B to multi-cloud application 181. In the first stage, the application SLE may depend on the peer path metrics and session flow records. In the second stage, there may be correlations with the wireless SLE data in the application SLE. In the second stage, the SLE application may use application profile data instead of session records to establish correlations. In the third stage, the application health SLE may add machine learning to the 'flow marking function'.

[0192] Figure 6 is a block diagram illustrating a system 600 for providing one or more recommendations and actions based on network health in accordance with one or more techniques of the present disclosure. Figure 6 As shown, system 600 may include health data 602A-602N (collectively referred to as "health data 602"). System 600 may apply one or more thresholds 604 to health data 602 and output health data 502 to database 610. In some examples, system 600 may change peer paths or hardware 606 based on applying one or more thresholds 604 to health data 602. System 600 may propose solutions 608 to one or more problems indicated by health data 602 and output solutions 608 to database 610.

[0193] The system 600 can use the machine learning engine 612 to execute the machine learning model 614 to generate an output. In some examples, the output from the machine learning model 614 is received by the prediction engine 622. The devices 618A-618N can output device data to the prediction engine 622 and the database 632. The prediction engine 622 can generate one or more recommendations and actions 630 based on the output from the machine learning model 614 and the device data, and output the one or more recommendations or actions to the database 632.

[0194] In some examples, the system 600 may include an analysis phase, a prediction phase, and a suggestion / action phase. The system 600 may examine historical patterns in application health SLEs, gateway health SLEs, and WAN link SLEs to make proactive suggestions for configuration. In some examples, the application health SLEs, gateway health SLEs, and WAN link SLEs may be included in health data 602.

[0195] In some examples, the system 600 is configured to identify pattern data indicative of one or more previous network problems from historical application performance data, historical path data, and historical network device data. In some examples, the historical application performance data, historical path data, and historical network device data are part of health data 602 of the system 600. In some examples, to identify pattern data indicative of one or more previous network problems, the system 600 may apply one or more thresholds 604. In some examples, to identify pattern data indicative of one or more previous network problems, the system 600 may change peer paths or hardware 606, and / or output one or more solutions 608. In some examples, the historical application performance data for one or more applications includes one or more application health assessments associated with one or more application time periods for a network site.

[0196] The prediction engine 622 of the system 600 can predict one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with the network site. In some examples, the prediction engine 622 can receive output from the machine learning model 614 and receive data from one or more devices 618.

[0197] The system 600 may generate one or more recommendations and actions 630 based on the output from the prediction engine 622. For example, the system 600 may determine at least one configuration change for one or more network devices (e.g., network device 110) of a network site to remedy a potential network problem. In some examples, the system 600 may output a notification of at least one configuration change to the one or more network devices. In some examples, the at least one configuration change of the one or more network devices includes a routing threshold configuration based on a service level agreement (SLA). In some examples, the at least one configuration change of the one or more network devices includes a packet path configuration change. In some examples, the notification includes a recommendation to update the configuration of the one or more network devices of the network site according to the at least one configuration change. In some examples, the NMS 300 is configured to automatically update the configuration of the one or more network devices of the network site according to the at least one configuration change. In some examples, in order to output the notification of the at least one configuration change, the system 600 is configured to output the notification via a user interface for display on a user interface device of an administrator associated with the network site.

[0198] In some examples, the NMS (e.g., NMS 130) is configured to generate a machine learning model 614 based on pattern data associated with a specific application deployed at the network site. In some examples, the machine learning engine 612 can execute the machine learning model 614 to predict potential network problems for the specific application deployed at the network site using the machine learning model. In some examples, the machine learning model 614 is different from Figure 5 The machine learning model 510. In some examples, the machine learning model 614 may include a self-learning model. The system 600 is configured to provide a continuous feedback mechanism of the determined at least one configuration change and the subsequently collected application performance data, path data, and network device data.

[0199] Figure 7 is a flowchart illustrating example operations for classifying one or more application minutes in accordance with one or more techniques of the present disclosure. Figures 1A to 1D The NMS 130 and its components are described in detail below. Figure 7 The techniques of may be performed by different components of NMS 130, or by additional or alternative network systems.

[0200] The NMS 130 may determine whether the delay value of the application minute 701 is greater than the delay threshold (702). If the delay value of the application minute 701 is greater than the delay threshold ("yes" at block 702), the NMS 130 may classify the application minute 701 as high delay (704). If the delay value of the application minute 701 is not greater than the delay threshold ("no" at block 702), the NMS 130 may proceed to block 706.

[0201] The NMS 130 may determine whether the jitter value of the application minute 701 is greater than the jitter threshold (706). If the jitter value of the application minute 701 is greater than the jitter threshold ("yes" at block 706), the NMS 130 may classify the application minute 701 as high jitter (708). If the jitter value of the application minute 701 is not greater than the jitter threshold ("no" at block 706), the NMS 130 may continue to block 710.

[0202] The NMS 130 may determine whether the loss value of the application minutes 701 is greater than the loss threshold (710). If the loss value of the application minutes 701 is greater than the loss threshold ("yes" at block 710), the NMS 130 may classify the application minutes 701 as high loss (712). If the loss value of the application minutes 701 is not greater than the loss threshold ("no" at block 710), the NMS 130 may continue to block 714.

[0203] The NMS 130 may determine whether the first time-to-data packet value of the application minute 701 is greater than the first time-to-data packet threshold (714). If the first time-to-data packet value of the application minute 701 is greater than the first time-to-data packet threshold ("yes" at block 714), the NMS 130 may classify the application minute 701 as a slow application (716). If the first time-to-data packet value of the application minute 701 is not greater than the first time-to-data packet threshold ("no" at block 714), the NMS 130 may proceed to block 718.

[0204] NMS 130 may determine whether the TCP reset check value of application minute 701 is greater than the TCP reset check packet threshold (718). If the TCP reset check value of application minute 701 is greater than the TCP reset check threshold ("yes" at block 718), NMS 130 may classify application minute 701 as a slow application (720). If the TCP reset check value of application minute 701 is not greater than the TCP reset check threshold ("no" at block 718), NMS 130 may proceed to block 722.

[0205] The NMS 130 may determine whether the bandwidth value of the application minute 701 is less than the bandwidth threshold (722). If the bandwidth value of the application minute 701 is less than the bandwidth threshold ("Yes" at block 722), the NMS 130 may classify the application minute 701 as low bandwidth (724). If the bandwidth value of the application minute 701 is not less than the bandwidth threshold ("No" at block 722), the NMS 130 may proceed to block 726.

[0206] The NMS 130 may determine whether the retransmission value of the application minute 701 is greater than the retransmission threshold (726). If the retransmission value of the application minute 701 is greater than the retransmission threshold ("Yes" at block 726), the NMS 130 may classify the application minute 701 as high retransmission (728). If the retransmission value of the application minute 701 is not greater than the retransmission threshold ("No" at block 726), the NMS 130 may proceed without classifying the application minute 701 (730).

[0207] Figure 8 8 is a block diagram illustrating a system 800 for determining top applications according to one or more techniques of the present disclosure. Figure 8 As shown, system 800 includes devices 802A-802N (collectively referred to as “devices 802 ”), application data 804 , pipeline 806 , database 808 , machine learning engine 810 , top application 812 , application SLE engine 814 , and user interface 816 .

[0208] In some examples, it may be beneficial to identify the top applications on which the Application SLE is calculated to improve the user experience of these applications. In some examples, industry top applications may be identified by one parameter (e.g., number of users, amount of bandwidth used). In some examples, it may be beneficial to consider various factors to obtain a list of top applications.

[0209] In some examples, the system 800 can identify the top applications based on one or more parameters, such as duration of use, number of times the application is used, number of unique clients using the application in the network, scope of use of the application (e.g., all day, only at lunchtime, only at night), and amount of bandwidth consumed. In some examples, the system 800 can assign weights to each parameter and determine an application index or application score based on the use of the application. The application index can be a number between 0 and 1.0. An application index of 0 indicates a very low priority application, and an application index of 1 indicates a very high priority application.

[0210] In some examples, it may be beneficial to automate the process of identifying applications that users care about on a per-site basis. A "site" may include a set of devices 802, including one or more client devices, routers, switches, APs. The set of applications will then be fed into the application SLE engine 814 to come up with a machine learning model to calculate the application SLE.

[0211] The following equation represents an example of an application index calculated based on one or more parameters.

[0212]

[0213]

[0214] In some examples, device 802 may correspond to a device (e.g., a network device) within a WAN, through which a client device establishes a connection with an application server, and through which the application server provides services to the client device. Fig.10 That is, device 802 may be part of a path within the WAN through which source device 112A communicates with destination device 114. In some examples, Figure 1A The source device 112 is a client device, and the destination device 114 represents an application server. Figure 1A The source device 112 may represent an application server, and the destination device 114 may represent a client device. In some examples, the device 802 may correspond to Figure 1B-Figure 1D router 187 and / or any other device within SD-WAN 177.

[0215] In some examples, application data 804 includes data corresponding to one or more applications, such as duration of use, number of times an application is used, number of unique clients of an application, scope of use of an application, amount of bandwidth used by an application, and an application index of an application. In some examples, application data 804 includes separate application data for each of the one or more applications.

[0216] The pipeline 806 can pass data to one or more engines for processing. For example, the pipeline 806 can pass network data 804 and / or other data to a database 808. In some examples, the database 808 can be stored in the NMS 130. In some examples, the database 808 is stored within the database 135 of the NMS 130. In some examples, the database 808 is stored separately from the database 135 within the NMS 130. The pipeline 806 can represent any type of connection (e.g., wired or wireless) configured to pass data to the database 808 or any other device.

[0217] In some examples, the NMS 130 can use the machine learning engine 810 to execute one or more machine models. For example, the NMS 130 can use the machine learning engine 810 to execute a machine learning model to identify one or more top applications 812 based on the application data 804 stored in the database 808. The application SLE engine 814 can output the one or more top applications 812 to the user interface 816.

[0218] In some examples, the system 800 can learn one or more top applications based on the session frequency of the service, the session length, and the number of bytes transmitted for the service or application. However, it may not represent the set of services that the customer is interested in. The session flow record can have a session identifier service name field, which can be used to identify the configured service. Only configured services learned from the session flow record are used. The system 800 (e.g., the machine learning engine 810) can learn the top applications based on one or more criteria and filter the results based on the top applications. The system 800 can provide a list of recommended applications to the user.

[0219] Fig. 9 9 is a conceptual diagram illustrating a user interface screen 900 including a first example root cause analysis performed by NMS 300, in accordance with one or more techniques of this disclosure. Fig. 9As shown, user interface screen 900 includes gateway health indicator 902A, WAN link health indicator 902B, and application health indicator 902C. User interface screen 900 includes response time classifier 904A, disconnection classifier 904B, and capacity classifier 904C corresponding to application health indicator 902C. User interface screen 900 includes slow link subclassifier 906A, slow application subclassifier 906B, and unstable link subclassifier 906.

[0220] In some examples, the user interface screen 900 may indicate one or more applications corresponding to the selected one or more classifiers. For example, the application health indicator 902C, the response time classifier 904A, and the slow link subclassifier 906A may be selected on the user interface screen 900. The user interface screen 900 may indicate a first application 912A and a second application 912B. The user interface screen 900 may indicate a first overall impact 914A corresponding to the first application 912A and a second overall impact 914B corresponding to the second application 912B.

[0221] Fig.10 1 is a conceptual diagram illustrating a user interface screen 1000 including a second example root cause analysis performed by NMS 300, in accordance with one or more techniques of this disclosure. Fig.10 As shown, the user interface screen 1000 includes a gateway health indicator 1002A, a WAN link health indicator 1002B, and an application health indicator 1002C. The user interface screen 1000 includes a delay classifier 1004A, a jitter classifier 1004B, a loss classifier 1004C, and an application service classifier 1004D. The user interface screen 1000 includes a slow application classifier 1006A, an application disconnection classifier 1006B, an application bandwidth classifier 1006C, and a retransmission classifier 1006D corresponding to the application service classifier 1004D. The user interface screen 1000 may include a first data type 1012A and a second data type 1012B corresponding to the application health indicator 1002C, the application service classifier 1004D, and the application bandwidth classifier 1006C. In some examples, the first data type 1012A may correspond to multiple faults within a period of time. In some examples, the second data type 1012B may correspond to the number of connected clients within a period of time.

[0222] Fig.11 is a flowchart illustrating example operations for performing one or more application health assessments in accordance with one or more techniques of the present disclosure. Figure 1A-Figure 1D and Figure 3 The NMS 130 and its components are described in detail below. Fig.11The techniques of may be performed by different components of NMS 130, or by additional or alternative network systems.

[0223] NMS 130 may receive path data from a plurality of network devices operating as network gateways for an enterprise network (1102). In some examples, the network devices may include Figure 1A One or more of the network devices 110. In some examples, the network devices may include Figure 1B-Figure 1D One or more of the routers 187. In some examples, the network devices may include Figure 2 The NMS 130 may include an application health SLE engine 134 configured to determine one or more application health assessments of one or more applications based on the path data, wherein the one or more application health assessments are associated with one or more application time periods for the site (1104). In response to determining at least one fault state, the NMS 300 is configured to output a notification including an identification of a root cause of the at least one fault state (1106).

[0224] Fig.12 is a flow chart illustrating example operations for predicting network problems and determining configuration changes in accordance with one or more techniques of this disclosure. Figure 1A-Figure 1D and Figure 3 The NMS 130 and its components are described in detail below. Fig.11 The techniques of may be performed by different components of NMS 130, or by additional or alternative network systems.

[0225] NMS 130 may identify pattern data indicative of one or more previous network problems from historical application performance data, historical path data, and historical network device data (1202). NMS 130 may predict one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site (1204). NMS 130 may determine at least one configuration change for one or more network devices of the network site to remediate the potential network problem (1206). NMS 130 may output a notification of at least one configuration change (1206).

[0226] The following numbered clauses may demonstrate one or more aspects of the present disclosure.

[0227] Clause 1. A network management system (NMS) for an enterprise network, the network management system comprising: a memory storing path data received from a plurality of network devices operating as network gateways for the enterprise network, the path data reported by each of the plurality of network devices for one or more logical paths from a physical interface of a given network device over a wide area network (WAN); and a processing circuit system communicating with the memory and configured to: determine one or more application health assessments for one or more applications based on the path data, wherein the one or more application health assessments are associated with one or more application time periods for a site, and in response to determining at least one fault state, output a notification including an identification of a root cause of the at least one fault state.

[0228] Clause 2. An NMS according to Clause 1, wherein the processing circuit system is configured to: select a subset of the multiple applications based on one or more characteristics indicating that the subset of the multiple applications has an impact on the user experience of the client device; and determine one or more application health assessments for each application of the subset of the multiple applications based on path data.

[0229] Clause 3. An NMS according to any one of clauses 1 to 2, wherein the processing circuit system is configured to apply one or more thresholds to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period.

[0230] Clause 4. The NMS of clause 3, wherein the processing circuitry is configured to determine the one or more thresholds based on the path data and historical application performance data.

[0231] Clause 5. The NMS of any of clauses 3 to 4, wherein the processing circuitry is further configured to classify each low-quality application time period based on the identified application-related problem classifier.

[0232] Clause 6. An NMS according to clause 5, wherein the application-related problem classifier includes one of the following: a slow application problem classifier corresponding to a slow response time from an application server; a transmission control protocol (TCP) retransmission problem classifier corresponding to one or more retries from a client device or the application server caused by service unavailability; an application bandwidth problem classifier corresponding to a bandwidth that is lower than a threshold amount of bandwidth required by an application, wherein the threshold amount of bandwidth corresponds to a corresponding application time period; or an application disconnection problem classifier corresponding to frequent disconnections from the client device or the application server.

[0233] Clause 7. The NMS of any one of clauses 1 to 6, wherein the path data comprises delay, jitter, and loss corresponding to each logical path from each of the plurality of network devices on the WAN.

[0234] Clause 8. An NMS according to any one of clauses 1 to 7, wherein the memory is also configured to store a machine learning model, and wherein the processing circuit system is also configured to: execute the machine learning model to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period based on the path data.

[0235] Clause 9. An NMS according to Clause 8, wherein the memory is also configured to store user feedback data corresponding to the one or more application time periods, and wherein the processing circuit system is also configured to: execute the machine learning model to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period based on the path data and the user feedback data.

[0236] Clause 10. An NMS according to any one of clauses 8 to 9, wherein the processing circuit system is further configured to execute the machine learning model to classify each low-quality application time period based on the identified application-related problem classifier.

[0237] Clause 11. A method comprising:

[0238] Determining, by a processing circuit system of a network management system (NMS) of an enterprise network and based on path data, one or more application health assessments for one or more applications, wherein the one or more application health assessments are associated with one or more application time periods for a site, and wherein a memory of the NMS is configured to store the path data received from a plurality of network devices operating as network gateways for the enterprise network, the path data reported by each of the plurality of network devices for one or more logical paths from a physical interface of a given network device over a wide area network (WAN); and in response to determining at least one fault state, outputting, by the processing circuit system, a notification including an identification of a root cause of the at least one fault state.

[0239] Clause 12. The method according to Clause 11 further includes: selecting, by the processing circuit system, a subset of the multiple applications based on one or more characteristics indicating that the subset of the multiple applications has an impact on the user experience of the client device; and determining, by the processing circuit system, one or more application health assessments for each application of the subset of the multiple applications based on the path data.

[0240] Clause 13. The method of any one of clauses 11 to 12, further comprising applying, by the processing circuit system, one or more thresholds to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period.

[0241] Clause 14. The method of clause 13, further comprising determining, by the processing circuitry, the one or more thresholds based on the path data and historical application performance data.

[0242] Clause 15. The method of any of Clauses 11 to 14, further comprising classifying, by the processing circuitry, each low-quality application time period based on the identified application-related problem classifier.

[0243] Clause 16. A method according to clause 15, wherein the application-related problem classifier includes one of the following: a slow application problem classifier corresponding to a slow response time from an application server; a Transmission Control Protocol (TCP) retransmission problem classifier corresponding to one or more retries from a client device or the application server caused by service unavailability; an application bandwidth problem classifier corresponding to a bandwidth that is lower than a threshold amount of bandwidth required by an application, wherein the threshold amount of bandwidth corresponds to a corresponding application time period; or an application disconnection problem classifier corresponding to frequent disconnections from the client device or the application server.

[0244] Clause 17. A method according to any one of clauses 11 to 16, wherein the memory is also configured to store a machine learning model, and wherein the method further comprises: executing the machine learning model by the processing circuit system so as to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period based on the path data.

[0245] Clause 18. A method according to Clause 17, wherein the memory is also configured to store user feedback data corresponding to the one or more application time periods, and wherein the method further comprises: the processing circuit system executes the machine learning model to classify each of the one or more application time periods as a high-quality application time period or a low-quality application time period based on the path data and the user feedback data.

[0246] Clause 19. The method of any one of clauses 17 to 18 further comprises executing, by the processing circuit system, the machine learning model to classify each low-quality application time period based on the identified application-related problem classifier.

[0247] Clause 20. A non-transitory computer-readable medium comprising instructions for causing one or more processors to perform the following: determining one or more application health assessments for one or more applications based on path data, wherein the one or more application health assessments are associated with one or more application time periods for a site, and wherein a memory of a network management system (NMS) is configured to store the path data received from a plurality of network devices operating as network gateways for an enterprise network, the path data being reported by each of the plurality of network devices for one or more logical paths from a physical interface of a given network device over a wide area network (WAN); and in response to determining at least one fault state, outputting a notification including an identification of a root cause of the at least one fault state.

[0248] Clause 21. A network management system (NMS) for an enterprise network, the network management system comprising: a memory storing historical application performance data of one or more applications, historical path data of one or more logical paths from each of a plurality of network devices on a wide area network (WAN), and historical network device data of the plurality of network devices; and a processing circuit system communicating with the memory and configured to: identify pattern data indicating one or more previous network problems from the historical application performance data, the historical path data, and the historical network device data, predict one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site, determine at least one configuration change for one or more network devices of the network site to remedy the potential network problem, and output a notification of the at least one configuration change.

[0249] Clause 22. An NMS according to Clause 21, wherein, in order to predict the one or more potential network problems, the processing circuit system is configured to: generate a machine learning model based on identified pattern data associated with a specific application deployed at the network site; and use the machine learning model to predict the potential network problem of the specific application deployed at the network site.

[0250] Clause 23. An NMS according to Clause 22, wherein the machine learning model includes a self-learning model, and wherein the processing circuit system is configured to provide a continuous feedback mechanism of at least one determined configuration change and subsequently collected application performance data, path data, and network device data.

[0251] Clause 24. The NMS of any one of clauses 21 to 23, wherein the at least one configuration change of the one or more network devices comprises a routing threshold configuration based on a service level agreement (SLA).

[0252] Clause 25. The NMS of any one of Clauses 21 to 24, wherein the at least one configuration change of the one or more network devices comprises a packet path configuration change.

[0253] Clause 26. The NMS of any one of Clauses 21 to 25, wherein the historical application performance data of the one or more applications comprises one or more application health assessments associated with one or more application time periods for the network site.

[0254] Clause 27. The NMS of any of Clauses 21 to 26, wherein the notification includes a recommendation to update the configuration of the one or more network devices of the network site in accordance with the at least one configuration change.

[0255] Clause 28. The NMS of any of Clauses 21 to 27, wherein the processing circuitry is configured to automatically update configuration of the one or more network devices of the network site according to the at least one configuration change.

[0256] Clause 29. The NMS of any of clauses 21 to 28, wherein to output the notification of the at least one configuration change, the processing circuitry is configured to output the notification via a user interface for display on a user interface device of an administrator associated with the network site.

[0257] Clause 30. An NMS according to any one of clauses 21 to 29, wherein the historical path data comprises delay data, jitter data and loss data.

[0258] Item 31. A method comprising: identifying, by a processing circuit system of a network management system (NMS), pattern data from historical application performance data of one or more applications, historical path data of one or more logical paths from each of a plurality of network devices on a wide area network (WAN), and historical network device data of the plurality of network devices, wherein the path data indicates one or more previous network problems, wherein a memory of the NMS is configured to store the historical application performance data, the historical path data, and the historical network device data; predicting, by the processing circuit system, one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site; determining, by the processing circuit system, at least one configuration change for one or more network devices of the network site to remedy the potential network problem, and outputting, by the processing circuit system, a notification of the at least one configuration change.

[0259] Clause 32. A method according to clause 31, wherein predicting the one or more potential network problems includes: generating a machine learning model by the processing circuit system based on identified pattern data associated with a specific application deployed at the network site; and using the machine learning model by the processing circuit system to predict the potential network problem of the specific application deployed at the network site.

[0260] Clause 33. A method according to clause 32, wherein the machine learning model includes a self-learning model, and wherein the method also includes a continuous feedback mechanism provided by the processing circuit system of at least one determined configuration change and subsequently collected application performance data, path data, and network device data.

[0261] Clause 34. The method of any one of clauses 31 to 33, wherein the at least one configuration change to the one or more network devices comprises a routing threshold configuration based on a service level agreement (SLA).

[0262] Clause 35. The method of any one of clauses 31 to 34, wherein the at least one configuration change to the one or more network devices comprises a packet path configuration change.

[0263] Clause 36. The method of any one of clauses 31 to 35, wherein the historical application performance data for the one or more applications comprises one or more application health assessments associated with one or more application time periods for the network site.

[0264] Clause 37. The method of any one of Clauses 31 to 36, wherein the notification includes a recommendation to update the configuration of the one or more network devices of the network site in accordance with the at least one configuration change.

[0265] Clause 38. The method of any one of Clauses 31 to 37, further comprising automatically updating a configuration of the one or more network devices of the network site based on the at least one configuration change.

[0266] Clause 39. The method of any one of clauses 31 to 38, wherein outputting the notification of the at least one configuration change comprises outputting the notification by the processing circuitry via a user interface for display on a user interface device of an administrator associated with the website.

[0267] Item 40. A non-transitory computer-readable medium comprising instructions for causing one or more processors to perform the following: identifying pattern data from historical application performance data of one or more applications, historical path data of one or more logical paths from each of a plurality of network devices on a wide area network (WAN), and historical network device data of the plurality of network devices, wherein the path data indicates one or more previous network problems, wherein a memory of a network management system (NMS) is configured to store the historical application performance data, the historical path data, and the historical network device data; predicting one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site; determining at least one configuration change for one or more network devices of the network site to remedy the potential network problem; and outputting a notification of the at least one configuration change.

[0268] The techniques described in this disclosure may be implemented at least in part in hardware, software, firmware, or any combination thereof. For example, various aspects of the techniques may be implemented in one or more processors, including one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuit systems, and any combination of such components. The term "processor" or "processing circuit system" may generally refer to any of the above-mentioned logic circuit systems (alone or in combination with other logic circuit systems) or any other equivalent circuit systems. A control unit including hardware may also perform one or more of the techniques of the present disclosure.

[0269] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any of the above-mentioned units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. Describing different features as modules or units is intended to highlight different functional aspects, and does not necessarily mean that such modules or units must be implemented by separate hardware or software components. On the contrary, the functions associated with one or more modules or units may be performed by separate hardware or software components, or integrated in common or separate hardware and software components.

[0270] The techniques described in the present disclosure may also be embodied or encoded in a computer-readable medium (such as a computer-readable storage medium) containing instructions. The instructions embedded or encoded in the computer-readable storage medium may cause a programmable processor or other processor to perform the method, for example, when the instructions are executed. The computer-readable storage medium may include a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, a magnetic medium, an optical medium, or other computer-readable medium.

[0271] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A network management system for an enterprise network, the network management system comprising: a memory storing historical application performance data of one or more applications, historical path data of one or more logical paths from each of a plurality of network devices over a wide area network, and historical network device data of the plurality of network devices; as well as a processing circuit system in communication with the memory and configured to: identifying pattern data from the historical application performance data, the historical path data, and the historical network device data indicative of one or more previous network problems, predicting one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with the network site, determining at least one configuration change to one or more network devices of the network site to remedy the potential network problem, and A notification of the at least one configuration change is output.

2. The network management system of claim 1 , wherein to predict the one or more potential network problems, the processing circuit system is configured to: generating a machine learning model based on the identified pattern data associated with a specific application deployed at the network site; and The machine learning model is used to predict the potential network problem for the specific application deployed at the network site.

3. A network management system according to claim 2, wherein the machine learning model includes a self-learning model, and wherein the processing circuit is configured to provide a continuous feedback mechanism of the at least one configuration change determined and the application performance data, path data, and network device data subsequently collected. 4 . The network management system according to claim 1 , wherein the at least one configuration change for the one or more network devices comprises a routing threshold configuration based on a service level agreement. 5 .

5. The network management system according to any one of claims 1 to 3, wherein the at least one configuration change for the one or more network devices comprises a packet path configuration change.

6. The network management system of any one of claims 1 to 3, wherein the historical application performance data for one or more applications comprises one or more application health assessments associated with one or more application time periods for the network site.

7. The network management system according to any one of claims 1 to 3, wherein the notification comprises a recommendation to update the configuration of the one or more network devices of the network site according to the at least one configuration change.

8. The network management system of any one of claims 1 to 3, wherein the processing circuitry is configured to automatically update the configuration of the one or more network devices of the network site according to the at least one configuration change.

9. A network management system according to any one of claims 1 to 3, wherein in order to output the notification of the at least one configuration change, the processing circuit system is configured to: output the notification via a user interface for display on a user interface device of an administrator associated with the network site.

10. The network management system according to any one of claims 1 to 3, wherein the historical path data comprises delay data, jitter data and loss data.

11. A network management method, comprising: identifying, by processing circuitry of a network management system, pattern data from historical application performance data for one or more applications, historical path data for one or more logical paths from each of a plurality of network devices over a wide area network, and historical network device data for the plurality of network devices, wherein the path data is indicative of one or more prior network problems, wherein a memory of the network management system is configured to store the historical application performance data, the historical path data, and the historical network device data; predicting, by the processing circuitry, one or more potential network problems based on the identified pattern data and current application performance data, current path data, and current network device data associated with a network site; determining, by the processing circuitry, at least one configuration change for one or more network devices of the network site to remedy the potential network problem, and A notification of the at least one configuration change is output by the processing circuitry.

12. The network management method of claim 11, wherein predicting the one or more potential network problems comprises: generating, by the processing circuitry, a machine learning model based on the identified pattern data associated with a particular application deployed at the network site; as well as The machine learning model is used by the processing circuitry to predict the potential network problem for the specific application deployed at the network site.

13. The network management method of claim 12, wherein the machine learning model comprises a self-learning model, and wherein the method further comprises: A continuous feedback mechanism of the determined at least one configuration change and subsequently collected application performance data, path data, and network device data is provided by the processing circuitry.

14. The network management method according to any one of claims 11 to 13, wherein the at least one configuration change for the one or more network devices comprises a routing threshold configuration based on a service level agreement.

15. The network management method according to any one of claims 11 to 13, wherein the at least one configuration change for the one or more network devices comprises a packet path configuration change.

16. The network management method of any one of claims 11 to 13, wherein the historical application performance data of the one or more applications includes one or more application health assessments associated with one or more application time periods for the network site.

17. The network management method according to any one of claims 11 to 13, wherein the notification comprises a recommendation to update the configuration of the one or more network devices of the network site according to the at least one configuration change.

18. The network management method according to any one of claims 11 to 13, further comprising automatically updating configurations of the one or more network devices of the network site according to the at least one configuration change.

19. The network management method according to any one of claims 11 to 13, wherein outputting the notification of the at least one configuration change comprises: The notification is output by the processing circuitry via a user interface for display on a user interface device of an administrator associated with the website.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to be configured as the network management system of any one of claims 1 to 10 or to implement the method of any one of claims 11 to 19.

Images