Detecting network faults using machine learning models

Abstract

The invention relates to network fault detection using a machine learning model. A network management system includes a memory configured to store a machine learning model and processing circuitry in communication with the memory. The processing circuitry is configured to receive network data from a plurality of network devices of an internal network managed by a network management system. The internal network may be connected to an external network isolated from the network management system. Further, the processing circuitry is configured to apply the machine learning model to detect a network fault within the external network based on the network data, and to perform a corrective action to remedy the network fault within the external network.

Description

[0001] priority

[0002] This application claims the benefit of U.S. Application No. 18 / 343,360, filed on June 28, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to computer networks and, more particularly, to monitoring and troubleshooting computer networks. Background Art

[0004] Commercial venues or sites, such as offices, hospitals, airports, stadiums, or retail stores, typically install a complex wireless network system (including a network of wireless access points (APs)) throughout the venue to provide wireless network services to one or more wireless client devices (or simply "clients"). An AP is a physical electronic device that enables other devices to wirelessly connect to a wired network using a variety of wireless network protocols and technologies, such as wireless local area network protocols that comply with one or more of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (i.e., "WiFi"), Bluetooth / Bluetooth Low Energy (BLE), mesh network protocols (e.g., ZigBee), or other wireless network technologies.

[0005] Many different types of wireless client devices (e.g., laptops, smartphones, tablets, wearable devices, appliances, and Internet of Things (IoT) devices) incorporate wireless communication technology and can be configured to connect to a wireless access point when the device is within range of a compatible wireless access point in order to access a wired network. In the case where the client device runs a cloud-based application (e.g., a Voice over Internet Protocol (VoIP) application, a streaming video application, a gaming application, or a video conferencing application), data is exchanged from the client device during an application session through one or more APs and one or more wired network devices (e.g., switches, routers, and / or gateway devices) to reach a cloud-based application server. Summary of the invention

[0006] In general, the present disclosure describes one or more techniques for detecting one or more network failures in an external network using a machine learning model. In some examples, a network management system (NMS) can provide network management services, such as wide area network (WAN) assurance services, for an internal network. The internal network may include multiple network devices that operate to provide WAN interconnection between devices of one or more external networks (e.g., one or more client devices of a customer network with one or more service provider devices of a cloud service provider network). That is, a client device can initiate a session with a service provider device via multiple network devices of the internal network. An administrator of the internal network can control multiple paths through the internal network. These paths can facilitate the flow of network traffic through the internal network. For example, a client device can connect to an access point and initiate a session with a service provider device such as a server of a video streaming platform. The path through one or more network devices of the internal network can be part of an end-to-end pathway that connects a client device to a service provider device.

[0007] When a client device is communicating with a service provider device via a network connection within a plurality of network devices of an internal network, one or more network failures may occur outside the internal network, which interrupts a session between the client device and the service provider device. For example, the client device may be connected to a network device of the internal network via one or more access points or other computing devices of an external network. Additionally or alternatively, the service provider device may be connected to a network device of the internal network via one or more access points or other computing devices of the external network. A network failure may occur at an access point, computing device, or interface located in the external network, such that a session between the client device and the service provider device is interrupted even if a network path through one or more network devices of the internal network is fully operational.

[0008] For example, a client device may establish a session with a service provider device via an end-to-end path that includes a path through an internal network. If a network failure located between the internal network and the service provider device occurs during a session, the network device within the internal network may attempt to send network traffic to the service provider device but fail. Due to the network failure, the service provider device may not be able to receive network traffic from the network device of the internal network, or may not be able to send return network traffic back to the internal network for forwarding to the client device. This means that data indicating network traffic sent and received at the network device of the internal network can indicate whether there is a network failure associated with the service provider device. Other types of data that may indicate a network failure include Transmission Control Protocol (TCP) data indicating the flow of TCP packets at the network device and new session data indicating the number of new sessions established via the network device.

[0009] The technology of the present disclosure may provide one or more improvements to computer-related fields of computer networks that are integrated into practical applications. For example, a network management system may use a machine learning model to detect one or more network failures outside an internal network based on network data received from network devices of the internal network. Since the network data may include information indicating network traffic and new sessions at the network devices, the network management system may use a machine learning model to process the network data to determine whether one or more network devices within the internal network have failed to exchange network traffic or establish new sessions with devices outside the internal network. By using a machine learning model to detect network failures outside the internal network, the network management system may detect network failures more accurately than a system that does not use a machine learning model to analyze network data.

[0010] In one example, a network management system includes a memory configured to store a machine learning model; and a processing circuit in communication with the memory. The processing circuit is configured to: receive network data from a plurality of network devices of an internal network managed by the network management system, wherein the internal network is connected to an external network isolated from the network management system; apply the machine learning model to detect a network failure within the external network based on the network data; and perform a corrective action to remedy the network failure within the external network.

[0011] In another example, a method includes receiving, by a processing circuit of a network management system, network data from a plurality of network devices of an internal network managed by the network management system, wherein the internal network is connected to an external network isolated from the network management system, and wherein the processing circuit communicates with a memory of the network management system, the memory being configured to store a machine learning model; applying, by the processing circuit, the machine learning model to detect a network fault within the external network based on the network data; and performing, by the processing circuit, a corrective action to remedy the network fault within the external network.

[0012] In another example, a computer-readable medium includes instructions that, when executed by a processing circuit, cause the processing circuit to: receive network data from a plurality of network devices of an internal network managed by a network management system, wherein the internal network is connected to an external network isolated from the network management system; apply a machine learning model to detect a network failure within the external network based on the network data; and apply a corrective action to remedy the network failure within the external network.

[0013] The details of one or more examples of the technology of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of these technologies will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A is a block diagram of an example network system including a network management system (NMS), in accordance with one or more techniques of this disclosure.

[0015] Figure 1B is a diagram showing one or more techniques according to the present disclosure Figure 1A A block diagram of a network system with further example details.

[0016] Figure 1C is a diagram showing one or more techniques according to the present disclosure Figure 1A A block diagram of a network system with further example details.

[0017] Figure 2 is a block diagram illustrating an example network device in accordance with one or more techniques of this disclosure.

[0018] Figure 3 is a block diagram of an example NMS in accordance with one or more techniques of this disclosure.

[0019] Figure 4 is a block diagram illustrating an example user equipment (UE), in accordance with one or more techniques of this disclosure.

[0020] Figure 5 is a block diagram illustrating an example network node in accordance with one or more techniques of this disclosure.

[0021] Figure 6 is a block diagram of an example AP device in accordance with one or more techniques of this disclosure.

[0022] Figure 7 is a flow diagram illustrating example operations for detecting one or more network failures using a machine learning model in accordance with one or more techniques of this disclosure.

[0023] Figure 8is a flow diagram illustrating example operations for training a machine learning model to detect network failures in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0024] Figure 1A 1 is a block diagram of an example network system 100 including a network management system (NMS) 130 according to one or more techniques of the present disclosure. The example network system 100 includes a plurality of sites 102A-102N (collectively referred to as "sites 102"), where a network service provider manages one or more wireless networks 106A-106N (collectively referred to as "wireless networks 106"), respectively. Figure 1A 1 , each of stations 102 is respectively illustrated as including a single one of wireless networks 106, but in some examples each of stations 102 may include multiple wireless networks, and the present disclosure is not limited in this respect.

[0025] Each of the sites 102 includes a plurality of network access server (NAS) devices (eg, access points (APs), switches or routers ( Figure 1A 1 to 142A-M. Similarly, site 102N includes multiple APs 142N-1 to 142N-M. Each AP 142 may be any type of wireless access point, including but not limited to a commercial or enterprise AP, a router, or any other device connected to a wired network and capable of providing wireless network access to client devices within the site.

[0026] Each of the sites 102 also includes a plurality of client devices, also referred to as user equipment (UE), generally referred to as UE or client devices, which represent various wireless-enabled devices within each site. For example, a plurality of UEs 148A-1 to 148A-K are currently located at site 102A. Similarly, a plurality of UEs 148N-1 to 148N-K are currently located at site 102N. The UEs located at site 102 may be collectively referred to as "UE 148". In some examples, the UEs may be referred to as "client devices". Each of the UEs 148 may be any type of wireless client device, including but not limited to a mobile device, such as a smartphone, tablet or laptop computer, a personal digital assistant (PDA), a wireless terminal, a smart watch, a smart ring, or other wearable device. The UE 148 may also include a wired client device, such as an Internet of Things (IoT) device such as a printer, a security device, an environmental sensor, or any other device connected to a wired network and configured to communicate via one or more wireless networks 106.

[0027] In order to provide wireless network services to UE 148 and / or communicate over wireless network 106, AP 142 and other wired client devices at site 102 are directly or indirectly connected to one or more network devices (e.g., switches, routers, etc.) via physical cables (e.g., Ethernet cables). Figure 1A In the example of , site 102A includes switch 146A to which each of APs 142A-1 through 142A-M at site 102A is connected. Similarly, site 102N includes switch 146N to which each of APs 142N-1 through 142N-M at site 102N is connected. Figure 1A 146, and all APs 142 of a given site 102 are connected to the single switch of switches 146, but in other examples, each site 102 may include more or fewer switches and / or routers. In addition, the APs and other wired client devices of a given site may be connected to two or more switches and / or routers. In addition, two or more switches of a site may be connected to each other and / or to two or more routers, for example, via a mesh or partial mesh topology in a hub-and-spoke architecture. In some examples, the interconnected switches and routers include a wired local area network (LAN) located at the site 102 hosting the wireless network 106.

[0028] The example network system 100 also includes various network components for providing network services within the wired network, including, for example, an authentication, authorization, and accounting (AAA) server 110 for authenticating users and / or UEs 148, a dynamic host configuration protocol (DHCP) server 116 for dynamically allocating a network address (e.g., an IP address) to the UE 148 upon authentication, a domain name system (DNS) server 118 for resolving domain names into network addresses, a plurality of switches 126A-126Y (collectively referred to as “switches 126”), a plurality of servers 128A-128X (collectively referred to as “servers 128”) (e.g., network servers, database servers, file servers, service provider servers, etc.), and an NMS 130. In some examples, the server 128 may represent a service provider device configured to provide one or more services to the UE 148. For example, the external network 125 may include the switches 126 and 128 and may be configured to provide services to the site 102. In some examples, the external network 125 is managed by a third party outside of the NMS 130 or an administrator of the NMS 130. like Figure 1AAs shown, various devices and systems of network system 100 are coupled together via network 120 (eg, the Internet and / or an intranet). In some examples, network 120 may include network devices 122A-122I (collectively referred to as "network devices 122").

[0029] In some examples, network device 122 may include multiple network devices that form multiple paths through network 120 as an internal network. Each of the multiple paths through the internal network may be part of one or more end-to-end paths between UE 148 and server 128. In some examples, administrator device 111 of network 120 may control the configuration of the paths through network 120. Since, in some cases, each of the multiple paths through the internal network formed by network device 122 may be part of one or more end-to-end paths connecting one or more UEs of UE 148 with one or more servers of server 128, the administrator of network 120 may control the configuration of the end-to-end paths by controlling the configuration of the paths through the internal network. For example, a first path through the internal network including network device 122 may be part of a first end-to-end path connecting UE 148A-1 with server 128B, and a second path through the internal network including network device 122 may be part of a second end-to-end path connecting UE 148N-1 with server 128X. This means that the configuration of the first path and the configuration of the second path may affect the configuration of the first end-to-end path and the configuration of the second end-to-end path, respectively.

[0030] exist Figure 1A In the example of , NMS 130 is a cloud-based computing platform that manages wireless networks 106 at one or more sites 102. As further described herein, NMS 130 provides a set of integrated management tools and implements various technologies of the present disclosure. Generally, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analysis, network anomaly identification, and alarm generation. In some examples, NMS 130 outputs notifications, such as alarms, warnings, graphical indicators on dashboards, log messages, text / short message service (SMS) messages, email messages, etc., and / or suggestions about wireless network problems to site or network administrators ("administrators") interacting with and / or operating management devices 111. In addition, in some examples, NMS 130 operates in response to configuration inputs received from administrators interacting with and / or operating administrator devices 111.

[0031] Administrators and administrator devices 111 may include IT personnel and administrator computing devices associated with one or more sites 102. Administrator device 111 may be implemented as any suitable device for presenting output and / or accepting user input. For example, administrator device 111 may include a display. Administrator device 111 may be a computing system, such as a mobile or non-mobile computing device operated by a user and / or administrator. According to one or more aspects of the present disclosure, administrator device 111 may, for example, represent a workstation, a laptop or notebook computer, a desktop computer, a tablet computer, or any other computing device that can be operated by a user and / or present a user interface. Administrator device 111 may be physically separated from NMS 130 and / or located in a different location so that administrator device 111 can communicate with NMS 130 via network 120 or other communication means.

[0032] In some examples, one or more NAS devices (e.g., AP 142, switch 146, or router) can be connected to edge devices 150A-150N (collectively, "edge devices 150") via physical cables (e.g., Ethernet cables). Edge devices 150 include cloud-managed wireless LAN controllers. Each edge device 150 may include an internal device at site 102 that communicates with NMS 130 to extend some microservices from NMS 130 to the on-premises NAS devices, while using NMS 130 and its distributed software architecture for scalable and resilient operations, management, troubleshooting, and analysis.

[0033] Each network device of the network system 100 (e.g., servers 110, 116, 122, and / or 128, network devices 122, AP 142, UE 148, switches 126, and / or 146, and any other server or device attached to or forming part of the network system 100) may include a system log or error log module, wherein each of these network devices records the state of the network device, including normal operating states and error conditions. Throughout this disclosure, one or more devices of the network system 100 (e.g., servers 110, 116, 118, and / or 128, switches 126, AP 142, UE 148, and switches 146) may be considered a "third party" network device when owned and / or associated with an entity other than the NMS 130, such that the NMS 130 does not receive, collect, or otherwise access state and other data recorded by the third party network devices. In some examples, edge device 150 may provide an agent through which recorded states and other data of third-party network devices may be reported to NMS 130. In some examples, one or more devices of network system 100, such as servers 110, 116, 118, and / or 128, switch 126, AP 142, UE 148, and switch 146, may be owned and / or associated with the same entity associated with NMS 130. In some examples, network device 122 may be owned and / or associated with the same entity associated with NMS 130.

[0034] NMS 130 may include processing circuitry 132 and memory 133. Processing circuitry 132 may include fixed function circuitry and / or programmable processing circuitry. Processing circuitry 132 may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or analog logic circuitry. In some examples, processing circuitry 132 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, GPUs, TPUs, one or more ASICs, and one or more FPGAs, as well as other discrete or integrated logic circuitry, which may be physically located in one or more devices at one or more physical locations.

[0035] Processing circuitry 132 may be capable of processing instructions stored in memory 133. In some examples, memory 133 includes a computer-readable medium that includes instructions that, when executed by processing circuitry 132, cause NMS 130 and processing circuitry 132 to perform the various functions attributed to them herein. Memory 133 may include any volatile, nonvolatile, magnetic, optical, or electrical medium, such as random access memory (RAM), read-only memory (ROM), nonvolatile RAM (NVRAM), electrically erasable programmable ROM (EEPROM), ferroelectric RAM (FRAM), dynamic random access memory (DRAM), flash memory, or any other digital medium.

[0036] In computer networking, a "black hole" is a network failure in which the network discards or "abandons" incoming or outgoing network traffic on an end-to-end path between a source device and a destination device without alerting the source device that the network traffic did not reach its intended recipient. A network failure such as a black hole can interrupt a session between a source device and a destination device, causing the destination device to stop providing one or more services associated with the network traffic to the source device. Many factors can cause network failures such as black holes, including misconfiguration of one or more forwarding path elements (e.g., next hops) of a network device, failure of a WAN link, failure of a server, or failure of any connection between nodes of a network, etc.

[0037] In some examples, the processing circuit 132 is configured to use the virtual network assistant (VNA) 134 and the machine learning model 135 to identify one or more network failures that occur outside the network managed by the NMS 130, such as one or more network failures that occur within the external network 125 that is not managed by the NMS 130. The network device 122 and the NMS 130 may be part of the same internal network, where an administrator of the internal network controls the network device 122 and the NMS 130. In some examples, the site 102 including the AP 142, the switch 146, and the UE 148 is outside the internal network corresponding to the network device 122 managed by the NMS 130. In some cases, one or more of the switches 146 may be located within the internal network corresponding to the network device 122 and the NMS 130. In some examples, the switch 126 and the server 128 may be located outside the internal network corresponding to the network device 122 and the NMS 130. In some examples, one or more of the switches 126 may be located within the internal network corresponding to the network device 122 and the NMS 130.

[0038] An administrator of the internal network may control network device 122 to form a plurality of network paths connecting one or more UEs 148 with one or more servers 128. Since server 128 may be configured to provide one or more services to UE 148, it may be beneficial for network device 122 to form a plurality of network paths connecting server 128 with UE 148. An end-to-end path between one of UEs 148 and one of servers 128 may include one or more paths through the internal network including network device 122, and one or more paths through an external network isolated from the internal network corresponding to network device 122 and NMS 130. In an example where site 102, switch 126A, and server 128 are located outside of an internal network corresponding to network device 122 and NMS 130, the end-to-end path between UE 148A-1 and server 128A may include one or more paths between UE 128A-1 and network device 122A through an external network isolated from the internal network and via AP 142 and switch 146A, a path through network device 122 within the internal network, and one or more paths between network device 122 and server 128A outside the internal network via switch 126 and connection 129.

[0039] Although the network device 122 may form multiple network paths connecting the server 128 with the UE 148 through the network 120, one or more network failures that interrupt one or more sessions between the server 128 and the UE 148 via the network device 122 may occur outside the internal network 120 associated with the network device 122 and the NMS 130. That is, the network failure may occur in a device, interface, or network connection that is not controlled by an administrator of the internal network associated with the network device 122 and the NMS 130. For example, the switch 126 and the server 128 may be outside the internal network associated with the network device 122 and the NMS 130. This means that the switch 126, the server 128, and the connections 129A-129D between the switch 126 and the server 128 may be outside the internal network. The network failure that occurs at the switch 126, the server 128, or the connection 129 may interrupt one or more sessions between the server 128 and the UE 148 via the multiple network connections formed by the network device 122.

[0040] In some examples, processing circuitry 132 of NMS 130 may be configured to apply machine learning model 135 of VNA 134 to identify one or more network faults located outside of internal network 120 associated with network device 122 and NMS 130. For example, processing circuitry 132 may apply machine learning model 135 to identify one or more network faults occurring at switch 126, server 128, connection 129, or any other device or connection outside of internal network associated with network device 122 and NMS 130 based on data received from within internal network 120. By applying machine learning model 135 to identify network faults occurring outside of internal network 120 based on data received from within internal network 120, NMS 130 may improve the manner in which network device 122 facilitates sessions between server 128 and UE 148 as compared to a network system that does not use machine learning models to identify network faults outside of the internal network.

[0041] NMS 130 can use machine learning models and network features to detect network failures such as black holes, and remedy or avoid detected network failures. Example features can relay what the route of the packet is to the customer, such as how the packet flows, the next hop, the service, etc. Compared with a system that does not use a machine learning model to detect network failures, the use of a machine learning model 135 can increase the speed and / or accuracy of NMS 130 detecting one or more network failures. For example, a system that uses a heartbeat mechanism to detect network failures may not be able to detect network failures as quickly, accurately, or with high confidence as NMS 130 detects network failures using a machine learning model 135. This is because the machine learning model 135 can detect network failures by analyzing the traffic and session data 136 received from the network device 122, which includes much more data than just the heartbeat signal. In addition, the machine learning model 135 can be trained using a large cache of training data 138, so that the machine learning model 135 is configured to distinguish multiple network data patterns associated with network failures and / or multiple network data patterns associated with healthy network operations.

[0042] In order to apply the machine learning model 135 to detect a network failure, the processing circuit 132 may be configured to determine that one or more of the network devices 122 cannot exchange network traffic with a server in the server 128 located outside the internal network. For example, the processing circuit 132 may apply the machine learning model 135 to determine that the network device 122I cannot communicate with the server 128A via the switch 126A and the connection 129A. When the network device 122I cannot communicate with the server 128A, this means that there may be a network failure corresponding to the switch 126A, the server 128A, the connection 129A, or any combination thereof. In some examples, the processing circuit 132 may be configured to apply the machine learning model 135 to identify the location of the network failure outside the internal network. For example, the machine learning model 135 may output information indicating that the network failure exists at the connection 129A, the switch 126A, or another location. In some examples, when NMS 130 applies machine learning model 135 to detect a network fault, machine learning model 135 may output the detected network fault with a confidence score indicating the confidence that machine learning model 135 accurately detected the network fault.

[0043] In some examples, a network failure located outside of the internal network may be located along one or more paths through an external network that is part of one or more end-to-end paths that also include a path through the internal network that includes network device 122. That is, network traffic may travel through a path of the internal network that includes network device 122 to a path through the external network that includes a network failure that prevents network traffic from traversing the end-to-end path that includes both the internal network path and the external network path. Even if an administrator of the internal network corresponding to network device 122 and NMS 130 can control network device 122 and the internal network path but cannot control the external network path, the administrator can reroute network traffic through network device 122 to avoid the network failure within the external network path and / or reconfigure the internal network path so that network traffic avoids the network failure within the external network path. NMS 130 is not limited to rerouting network traffic in response to detecting a network failure. In some examples, NMS 130 may, in response to detecting a network failure, control network device 122 to avoid forwarding network traffic to an address associated with the network failure, withdraw routing to the address associated with the network failure, send a diagnostic message to the address associated with the network failure to confirm the network failure within an external network, and output an indication or notification of the network failure to an administrator, or any combination thereof.

[0044] In some examples, it may be beneficial to reroute network traffic to server 128A or another server via an external network path that does not have a network failure. For example, when processing circuit 132 applies machine learning model 135 to determine that there is a network failure at connection 129A that prevents network device 122A from communicating with server 128A, processing circuit 132 may route network traffic from network device 122I to server 128A via switch 126B and connection 129B, rather than routing network traffic via switch 125A and connection 129A. In another example, the same service provider may use both server 128A and server 128B to provide at least some of the same services to UE 148. In this example, based on determining that there is a network failure at connection 129A, processing circuit 132 may route network traffic from network device 122I to server 128B via switch 126C and connection 129C, rather than routing network traffic to server 128A via switch 125A and connection 129A. In any case, based on applying the machine learning model 135 to identify one or more network failures located outside the internal network, the NMS 130 can control the network device 122 to reroute traffic to avoid the network failure outside the internal network without interrupting one or more sessions between the UE 148 and the server 128A, or reducing the interruption of one or more sessions between the UE 148 and the service 128A.

[0045] Rerouting network traffic is not the only remedial action that NMS 130 is configured to take in response to identifying one or more network failures. In some examples, NMS 130 may output one or more messages in response to identifying one or more network failures. For example, when NMS 130 identifies a network failure associated with server 128A, NMS 130 may output a message (e.g., a ping) to server 128A and / or a service provider associated with server 128B indicating that NMS 130 detected the network failure. In some examples, NMS 130 may save information indicating the detected network failure to memory 133.

[0046] In some examples, the NMS 130 receives network traffic and session data 136 from the network devices 122. In some examples, the term “network traffic and session data” may be referred to herein as “network data.” In some examples, the network traffic and session data 136 may include network traffic and session data corresponding to each of the network devices 122. In some examples, the network traffic and session data corresponding to each of the network devices 122 may include network traffic transmission (T) indicating an amount of network traffic transmitted from an interface of the network device. x) data, and network traffic received (R ) data indicating the amount of network traffic received by the network interface of the network device x ) data. Additionally or alternatively, the network traffic and session data corresponding to each of the network devices 122 may include transmission control protocol (TCP) data indicating an amount of TCP synchronization (SYN) network traffic received and / or transmitted from the network device, an amount of TCP synchronization acknowledgement (SYN-ACK) network traffic received and / or transmitted from the network device, and an amount of TCP acknowledgement (ACK) network traffic received and / or transmitted from the network device. Additionally or alternatively, the network traffic and session data corresponding to each of the network devices 122 may include new session data indicating a number of new sessions that failed to be established via an interface of the network device, or any combination thereof.

[0047] If any Internet Protocol (IP) address within network system 100 is inaccessible, this may result in changes to network traffic transmission data, network traffic reception data, TCP data, new session data, or any combination thereof. For example, when a TCP session is not accessible relative to a device associated with an IP address, the TCP session may be inaccessible. x Data is increasing and R x When data is decreasing or nonexistent, this may indicate one or more network failures corresponding to the device. In another example, when one or more TCP resets relative to a device associated with an IP address increase, this may indicate one or more network failures relative to the device. In some examples, one or more failures to establish a new session involving a device associated with an IP address may indicate one or more network failures corresponding to the device.

[0048] In some examples, all edge routers within the internal network corresponding to the network device 122 and the NMS 130 use the same gateway IP address to reach the destination device (e.g., a server in the server 128). When the machine learning model 135 determines that a connection, device, or server outside the internal network is associated with a network failure, the NMS 130 can notify the devices within the internal network to stop using the connection, device, or server associated with the IP address to communicate with the destination device. In some examples, the NMS 130 can reroute the network traffic to the destination device via another external network path that is not associated with the identified network failure. In some examples, the NMS 130 can reroute the network traffic to another server that is configured to provide the same or similar services as the destination device associated with the network failure.

[0049] In some examples, network traffic and session data received from network devices in network devices 122 may be organized according to the respective servers in servers 128 with which the data is associated. For example, network traffic and session data received from network device 122I may include a set of data including a T indicating the amount of network traffic transmitted from an interface of network device 122I to server 128A. x data and R indicating the amount of network traffic received by network device 122I from server 128A x Additionally or alternatively, the network traffic and session data received from network device 122I may include TCP data indicating a number of TCP SYN packets sent by network device 122I to server 128A, a number of TCP SYN-ACK packets received by network device 122I from server 128A, a number of TCP ACK packets sent by network device 122I to server 128A, or any combination thereof. In some examples, the network traffic and session data received from network device 122I may include new session data indicating a number of new sessions established with server 128A via network device 122A according to any communication session protocol including TCP, UDP, TLS, ICMP, or any combination thereof.

[0050] Because network traffic and session data 136 may include various data, including data indicating network traffic levels and the number of new sessions, network traffic and session data 136 indicates the manner and extent to which each of network devices 122 communicates with each of servers 128 via each of a plurality of end-to-end paths between servers 128 and UEs 148 via network device 122. In some examples, network devices 122 and NMS 130 may be located within the same internal network, and the plurality of internal network paths through network devices 122 may interact with a plurality of external network paths outside the internal network to form a plurality of end-to-end paths between servers 128 and UEs 148. That is, an end-to-end path between a server of servers 128 and a UE of UEs 148 may include one or more internal network paths through network device 122 and one or more external network paths outside the internal network.

[0051] To apply the machine learning model 135, the processing circuitry 132 of the NMS 130 may use the network traffic and session data 136 as inputs to the machine learning model, and the machine learning model 135 may generate an output based on the network traffic and session data 136 indicating one or more network failures outside the internal network corresponding to the network devices 122 and the NMS 130. That is, the machine learning model 135 may identify one or more network failures outside the internal network based on data received from the network devices 122 located within the internal network.

[0052] In some cases, the machine learning model 135 can identify network failures based on one or more trends in the network traffic and session data 136. For example, when the T x Data and R x When trends in the data indicate high levels of network traffic transmitted from one or more network devices 122 to server 128B and low levels of network traffic received by one or more network devices 122 from server 128B, this may indicate that a network failure exists between one or more network devices 122 and server 128B. In another example, when trends in the TCP data in the network traffic and session data 136 indicate high levels of TCP SYN messages transmitted to server 128B and low levels of TCP SYN-ACK messages received from server 128B, this may indicate that a TCP session between network device 122 and server 128B failed to be established, which may indicate a network failure. In another example, when trends in the new session data in the network traffic and session data 136 indicate a low number of new sessions established between server 128B and UE 148 via network devices 122 of the internal network, this may indicate that a network failure exists between network device 122 and server 128B.

[0053] In some examples, machine learning model 135 can detect one or more network failures that may occur at a future time based on network traffic and session data 136. For example, machine learning model 135 can determine that the probability that a network failure will occur at connection 129A is greater than a probability threshold. NMS 130 can reroute network traffic based on determining that the probability that a network failure will occur at connection 129A is greater than the probability threshold.

[0054] The memory 133 of the NMS 130 can be configured to store the training data 138. In some examples, the training data 138 includes multiple sets of training network data. In some examples, each set of training network data in the multiple sets of training network data can include data similar in kind to the network traffic and session data 136 received from the network device 122. For example, each set of training network data in the multiple sets of training network data can include network traffic transmission data, network traffic reception data, TCP data, new session data, or any combination thereof corresponding to the network device. In some examples, each set of training network data in the multiple sets of training network data is associated with network status information indicating whether the set of training network data is associated with one or more network failure events.

[0055] The processing circuit 132 of the NMS 130 may be configured to train the machine learning model 135 using supervised learning based on multiple sets of training network data and network state information corresponding to each set of training network data in the multiple sets of training network data. In some examples, to train the machine learning model 135, the processing circuit 132 may train the machine learning model 135 to identify one or more patterns in the training data that are associated with network failures and / or one or more patterns in the training data that are not associated with network failures.

[0056] NMS 130 may train machine learning model 135 using historical data of routing parameters at the internal network level network device interfaces (e.g., gateway routers and / or destination IP addresses) to determine one or more routing parameters (e.g., R x Data, T x In some examples, NMS 130 may train machine learning model 135 to identify spikes in routing parameters compared to baseline levels that indicate network failures, including black holes. This means that machine learning model 135 may identify network failures using patterns associated with network failures learned by machine learning model 135 when trained using training data 138. For example, to learn patterns corresponding to network failures such as black holes, machine learning model 135 may analyze average uplink and downlink traffic at one or more points in time, analyze new session data, analyze TCP resets, or any combination thereof.

[0057] In some examples, processing circuitry 132 may receive information indicating whether machine learning model 135 accurately detected a network failure based on network traffic and session data 136. Processing circuitry 132 may augment training data 138 with network data and network status information indicating whether machine learning model 135 accurately detected a network failure based on network traffic and session data 136. In some cases, processing circuitry 132 may retrain machine learning model 135 based on the augmented training data using supervised learning. That is, processing circuitry 132 may continuously retrain the machine learning model based on predictions and determinations made by machine learning model 135.

[0058] According to one specific embodiment, the computing device is part of the NMS 130. According to other implementations, the NMS 130 may include one or more computing devices, dedicated servers, virtual machines, containers, services, or other forms of environments for executing the techniques described herein. Similarly, the computing resources and components that implement the VNA 134 may be part of the NMS 130, may be executed on other servers or execution environments, or may be distributed to nodes within the network 120 (e.g., routers, switches, controllers, gateways, etc.).

[0059] Although the techniques of the present disclosure are described in this example as being performed by NMS 130, the techniques described herein may be performed by any other computing device, system, and / or server, and the present disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the techniques of the present invention may reside in a dedicated server, or be included in any other server other than NMS 130, or may be distributed throughout network 100, and may or may not form part of NMS 130.

[0060] Figure 1B is a diagram showing one or more techniques according to the present disclosure Figure 1A A block diagram of a network system with further example details is provided. In this example, Figure 1B FIG. 1 shows an NMS 130 configured to operate in accordance with an artificial intelligence / machine learning based computing platform that provides comprehensive automation, insight, and assurance (Wi-Fi assurance, wired assurance, and WAN assurance) ranging from “clients” (e.g., devices connected to the wireless network 106 and wired LAN 175) to Figure 1B 148) to the “cloud”, e.g., a cloud-based application service 181 (which may be hosted by computing resources within a data center 179) Figure 1B In some examples, wired LAN 174 may be a switch (e.g., Figure 1AIn some examples, data center 179 may include Figure 1A One or more switches of the switch 126 and / or Figure 1A One or more servers of the server 128. The cloud-based application service 181 may represent a server corresponding to Figure 1A The services are provided by one or more service providers of server 128.

[0061] As described herein, NMS 130 provides a set of integrated management tools and implements various technologies of the present disclosure. Typically, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analysis, network anomaly identification, and alarm generation. For example, NMS 130 can be configured to actively monitor and adaptively configure network system 100 to provide self-driving capabilities. In addition, VNA 134 includes a natural language processing engine to provide AI-driven support and troubleshooting, anomaly detection, AI-driven location services, and artificial intelligence (AI)-driven radio frequency (RF) optimization with reinforcement learning. For example, VNA 134 includes a machine learning model 135, which is configured to identify one or more network faults based on network traffic and session data 136 stored in memory 133 of NMS 130. In some examples, processing circuit 132 can use training data 138 to train machine learning model 135.

[0062] like Figure 1B As shown in the example of , the NMS 130 also provides configuration management, monitoring, and automated supervision of a software-defined wide area network (SD-WAN) 177, which operates as an intermediate network that communicatively couples the wireless network 106 and the wired LAN 175 to the data center 179 and the cloud-based application services 181. Typically, the SD-WAN 177 provides seamless, secure traffic engineering connections between the "hub and spoke" routers 187A of the wired LAN 175 hosting the wireless network 106 (e.g., a branch or campus network) and the "hub" routers 187B further up in the cloud stack toward the cloud-based application services 181. The SD-WAN 177 typically operates and manages an overlay network on the underlying physical WAN that provides connectivity to geographically separated customer networks. In other words, the SD-WAN 177 extends software-defined networking (SDN) capabilities to the WAN and allows the network to decouple the underlying physical network infrastructure from the virtualized network infrastructure and applications so that the network can be configured and managed in a flexible and scalable manner.

[0063] In some examples, the underlying routers of the SD-WAN 177 can implement a stateful, session-based routing scheme, in which the routers 187A, 187B dynamically modify the contents of the original packet header initiated by the UE 148 to direct traffic to the application service 181 along a selected path (e.g., path 189) without the need to use tunnels and / or additional labels. In this way, the routers 187A, 187B can be more efficient and scalable for large networks because the use of tunnel-free, session-based routing can enable the routers 187A, 187B to realize considerable network resources by eliminating the need to perform encapsulation and decapsulation at the tunnel endpoints. In addition, in some examples, each router 187A, 187B can independently perform path selection and traffic engineering to control the packet flow associated with each session without the need to use a centralized SDN controller for path selection and label distribution. In some examples, the routers 187A, 187B implement session-based routing as Security Vector Routing (SVR) provided by Juniper Networks.

[0064] In some examples, SD-WAN 177 including routers 187A, 187B and path 189 may be Figure 1A 187B and a plurality of paths including path 189. That is, path 189 may be an example of a path through SD-WAN 177. In some examples, NMS 130, SD-WAN 177, a plurality of routers including routers 187A, 187B, and a plurality of paths including path 189 may be part of an internal network managed by an administrator. For example, an administrator may control a plurality of routers including routers 187A, 187B and configure a plurality of paths including path 189 to connect UE 148 to cloud-based application service 181 via SD-WAN 177. In some examples, one or more other devices may be located outside the internal network corresponding to NMS 130, SD-WAN 177, a plurality of routers including routers 187A, 187B, and a plurality of paths including path 189. For example, UE 148 , wireless network 106 , wired LAN 175 , data center 179 , and cloud-based application services 181 may be located outside of an internal network.

[0065] In some examples, the AI-driven NMS 130 can implement intent-based configuration and management of the network system 100, including implementing the construction, presentation, and execution of intent-driven workflows for configuring and managing devices associated with the wireless network 106, the wired LAN 175, and / or the SD-WAN 177. For example, declarative requirements express the desired configuration of network components without specifying the exact local device configuration and control flow. By utilizing declarative requirements, it is possible to specify what should be done, rather than how it should be done. Declarative requirements can be contrasted with imperative instructions that describe the exact device configuration syntax and control flow that implement the configuration. By utilizing declarative requirements rather than imperative instructions, the burden of determining the exact device configuration required to achieve the desired results of the user / system is reduced for the user and / or user system. For example, when utilizing a variety of different types of devices from different manufacturers, it is often difficult and cumbersome to specify and manage precise imperative instructions to configure each device of the network. The types and categories of network devices can change dynamically as new devices are added and device failures occur. Managing a variety of different types of devices from different vendors with different configuration protocols, syntaxes, and software versions to configure a cohesive network of devices is often difficult to achieve. Therefore, by requiring the user / system to specify only declarative requirements that specify the desired results that apply to a variety of different types of devices, management and configuration of network devices becomes more efficient.

[0066] In general, the NMS 130 may provide network management services, such as management, monitoring, and automated supervision services, for the SD-WAN based on network data received from components of the SD-WAN 177, such as the routers 187. Traditionally, the NMS may not receive network data for external or third-party networks, such as cloud service provider networks supporting cloud-based application services 181. As a result, a traditional NMS may not have insight into the causes of network failures within such external networks, and therefore have limited ability to remediate failures occurring within such external networks, which have an adverse effect on the forwarding of traffic within the SD-WAN 177, such as blackholing of customer traffic occurring within such external networks.

[0067] According to the techniques described in this disclosure, the NMS 130 is configured to receive network traffic and session data 136 from a plurality of routers including routers 187A, 187B. In some examples, the network traffic and session data 136 may include data sets from each of the plurality of routers including routers 187A, 187B inside the SD WAN 177. The processing circuit 132 may apply the machine learning model 135 of the VNA 134 to identify one or more network failures occurring outside the SD-WAN 177 (e.g., within a network supporting cloud-based application services 181) based on the network traffic and session data 136 received from the plurality of routers including routers 187A, 187B within the internal network. The processing circuit 132 of the NMS 138 may be configured to train the machine learning model 135 based on the training data 138.

[0068] Figure 1C is a diagram showing one or more techniques according to the present disclosure Figure 1A A block diagram of a network system 100 is provided for further example details. Figure 1C As shown, network system 100 includes networks 121A-121D (collectively referred to as "network 121") configured to provide WAN connections to site 102. In some examples, network 121 may be Figure 1A 1. Network device 122 of network 121 provides site 102 with access to network 121. Network system 100 includes a set of connections 123A-123I (collectively, "connections 123"). Each connection in connections 123 can connect a network device of network device 122 to another network device in network devices 122 or another device external to network device 122. Network device 122 of network 121 can provide site 102 with access to one or more services from a service provider associated with server 128 including server 128A and server 128B. Figure 1C As shown, server 128A is connected to network device 1221 via connection 123G, switch 126A, and connection 129A, and server 128A is connected to network device 122F via connection 123H, switch 126B, and connection 129B. Server 128B is connected to network device 122E via connection 1231, switch 126C, and connection 129C.

[0069] For ease of illustration, network 121A is depicted as having two network devices, network 121B is depicted as having two network devices, network 121C is depicted as having two network devices, and network 121D is depicted as having three network devices, but each of networks 121 may include any number of network devices. Figure 1CThe configuration of the site 102, network device 122, switch 126, and server 128 shown in FIG. 1 is an example configuration, but the technology of the present disclosure is not limited to Figure 1C For example, another example of network 121B may include three network devices connected between network device 122 and network device 122E, instead of Figure 1C The two network devices shown in .

[0070] Connection 123 may form multiple internal network paths through network device 122. For example, there is an internal network path between site 102 and switch 126A via connection 123A, network device 122A, connection 123B, network device 1220, connection 123D, network device 120E, network device 121F, connection 123F, network device 112G, network device 122H, network device 122, and connection 123G. There is another internal network path between site 102 and switch 126C via connection 123A, network device 122A, connection 123C, network device 123C, connection 123E, network device 122, and connection 123I. Figure 1C As shown, there are many possible internal network paths through network device 122.

[0071] In some examples, the internal network path through network device 122 via connection 123 may be located within the internal network corresponding to network device 122 and NMS 130. That is, an administrator of internal network 121 (or NMS 130) may configure network device 122 and the internal network path through network device 122 via connection 123. When one or more network failures occur at network device 122 and / or connection 123, the administrator of internal network 121 (or NMS 130) may resolve these network failures by controlling network device 122 and / or connection 123. When one or more network failures occur outside internal network 121 (e.g., behind switch 126 relative to network device 122), it may be beneficial for NMS 130 to detect these failures and reroute network traffic through network device 122 to avoid the network failure outside the internal network. This is because an administrator of internal network 121 and network devices 122 (or NMS 130 ) may not be able to control one or more devices or connections that are the source of a network failure outside the internal network, but the administrator does have control over network traffic through internal network 121 .

[0072] In some examples, site 102, switch 126, server 128, and connection 129 may be located outside of an internal network corresponding to network device 122 and NMS 130. In some examples, site 102, switch 126, server 128, and connection 129 may form multiple external network paths that, together with the internal network path through network device 122, form multiple end-to-end paths between UE 148 and server 128. For example, an end-to-end path between UE 148A-1 and server 128B may include an external network path extending from UE 148A-1 to network device 122 via AP 142 and switch 146A, and an internal network path through network device 122 via connection 123A, network device 122A, connection 123C, network device 123C, connection 123E, network device 122E, connection 123I, switch 126C, and connection 129C. Figure 1C As shown, there are many other examples of end-to-end paths connecting site 102 and server 128.

[0073] NMS 130 can be applied Figure 1A to Figure 1B The machine learning model 135 of the embodiment of the present invention can be used to identify one or more network failures located outside the internal network 121. For example, the machine learning model 135 can determine that there is a network failure in the external network 125 at the connection 129A between the switch 126A and the server 128A, and the network failure is interrupting one or more sessions between the server 128A and the site 102 via the network device 122. Based on the determination that there is a network failure at the connection 129A, the NMS 130 can reroute the traffic to the server 128A via the switch 126B and the connection 129B. In other words, because the network failure at the connection 129B prevents the server 128A from exchanging network traffic on the connection 129A, the NMS 130 can instead route the traffic to the server 128A via the connection 123H, the switch 126B, and the connection 129B. In another example, the server 128A and the server 128B can be associated with the same service provider. In this example, NMS 130 may reroute traffic to server 128B via connection 1231, switch 126C, and connection 129C based on determining that a network failure exists at connection 129A.

[0074] Figure 2 2 is a block diagram illustrating an example network device 200 according to one or more techniques of the present disclosure. In general, the network device 200 may be Figure 1A and Figure 1C One of the network devices 122 or Figure 1B187B. In this example, network device 200 includes interface cards 226A-226N ("IFC 226") that receive packets via input links 228A-228N ("input links 228") and send packets via output links 230A-230N ("output links 230"). IFC 226 is typically coupled to links 228, 230 via a plurality of interface ports. Network device 200 also includes a control unit 202 that determines the routing of received packets and forwards the packets accordingly via IFC 226.

[0075] The control unit 202 may include a routing engine 204 and a packet forwarding engine 222. The routing engine 204 serves as a control plane for the network device 200 and includes an operating system that provides a multi-tasking operating environment to execute multiple concurrent processes. The routing engine 204 is coupled to other routers (e.g., Figure 1A and Figure 1C 122) to establish and maintain a computer network (e.g., Figures 1A to 1C The network system 100 of the routing engine 204 is used to transmit network traffic between one or more client devices. The routing protocol daemon (RPD) 208 of the routing engine 204 executes software instructions to implement one or more control plane network protocols 212. For example, the protocol 212 may include one or more routing protocols, such as the Multi-Protocol Label Switching (MPLS) protocol 214, the Border Gateway Protocol (BGP) 220, the Internet Group Management Protocol (IGMP) 221, which is used to exchange routing information with other routing devices and update the routing information base (RIB) 206 and other routing protocols. The protocol 212 may also include one or more communication session protocols 223, such as TCP, User Datagram Protocol (UDP), Transport Layer Security (TLS), or Internet Control Message Protocol (ICMP).

[0076] The RIB 206 may describe the topology of the computer network in which the network device 200 is located, and may also include routes through a shared tree in the computer network. The RIB 206 describes various routes within the computer network, as well as the appropriate next hop for each route, i.e., the adjacent routing devices along each route. The routing engine 204 analyzes the information stored in the RIB 206 and generates forwarding information for the packet forwarding engine 222 stored in the forwarding information base (FIB) 224. The FIB 224 may associate, for example, a network destination with a specific next hop and a corresponding IFC 226 and a physical output port for an output link 230. The FIB 224 may be a radix tree, a series of tables, a complex database, a linked list, a radix tree, a database, a flat file, or various other data structures programmed into a dedicated forwarding chip.

[0077] FIB 224 may also include a lookup structure. Once a keyword (e.g., an address) is given, the lookup structure may provide one or more values. In some examples, the one or more values ​​may be one or more next hops. The next hop may be implemented as a microcode that performs one or more operations when executed. One or more next hops may be "chained," such that a set of chained next hops perform a set of operations for each different next hop when executed. Examples of such operations may include applying one or more services to a packet, abandoning a packet, and / or forwarding a packet using an interface and / or interface identified by one or more next hops.

[0078] The network traffic transmission data 232 may include information corresponding to the amount of network traffic forwarded by the network device 200 for each of a set of endpoint devices. For example, the network traffic transmission data 232 may include information corresponding to the amount of network traffic forwarded by the network device 200 along one or more end-to-end paths at which Figure 1A Server 128A is an endpoint device, and corresponds to information about the amount of network traffic forwarded by network device 200 along one or more end-to-end paths at which Figure 1A The server 128B is an endpoint device, etc. In some examples, the amount of network traffic indicated by the network traffic transmission data 232 may indicate a total number of packets, a rate at which packets are sent, a number of packets transmitted in a period of time, or any combination thereof.

[0079] Network traffic reception data 234 may include information corresponding to the amount of network traffic received by network device 200 from each of a set of endpoint devices. For example, network traffic reception data 234 may include information corresponding to the amount of network traffic received by network device 200 from server 128A along one or more end-to-end paths. Figure 1A Server 128A is an endpoint device, and information corresponding to the amount of network traffic received by network device 200 from server 128B along one or more end-to-end paths at which Figure 1A The server 128B is an endpoint device, etc. In some examples, the amount of network traffic indicated by the network traffic reception data 234 may indicate a total number of packets, a rate at which packets are received, a number of packets received in a period of time, or any combination thereof.

[0080] The network traffic transmission data 232 and the network traffic reception data 234 may indicate one or more network failures associated with one or more endpoint devices of a set of endpoint devices. For example, if the network traffic transmission data 232 indicates that a large amount of network traffic is forwarded along one or more end-to-end paths associated with the endpoint device, and the network traffic reception data 234 indicates that a small amount or no amount of network traffic is received from the endpoint device along the one or more end-to-end paths, this may indicate that the network failure associated with the endpoint device is preventing network traffic from reaching the endpoint device and / or preventing the endpoint device from sending network traffic along the one or more end-to-end paths.

[0081] In some examples, in order to communicate with a client device (e.g., Figure 1A UE 148) and service provider equipment (e.g. Figure 1A In order to establish a session between the client device and the service provider device (one of the servers 128 of the client device), the client device and the service provider device may perform a three-way handshake. For example, the client device may send a SYN packet to the service provider device via the end-to-end path between the client device and the service provider device. Upon receiving the SYN packet, the service provider device may send a SYN-ACK packet to the client device via the end-to-end path in response to the SYN packet. When the client device receives the SYN-ACK packet, the client device may send an ACK packet to the service provider device via the end-to-end path, thereby completing the three-way handshake and establishing a session according to TCP.

[0082] TCP data 236 may include information indicating a number of TCP SYN packets forwarded along one or more end-to-end paths associated with an endpoint device via network device 200, a number of TCP SYN-ACK packets forwarded along one or more end-to-end paths associated with an endpoint device via network device 200, a number of multiple TCP ACK packets forwarded along one or more end-to-end paths associated with an endpoint device via network device 200, or any combination thereof. TCP data 236 may include information corresponding to each of the multiple endpoint devices (e.g., Figure 1A That is, the TCP data 236 may indicate whether the TCP three-way handshake is successfully completed for each of the plurality of endpoint devices.

[0083] When one or more TCP handshakes involving an endpoint device do not complete successfully, this may indicate that there is a network failure corresponding to the endpoint device. Figure 1C Connection 123G, switch 126A, and connection 129A forward a large number of TCP SYN packets along one or more end-to-end paths to server 128A, and via Figure 1C When connection 129A, switch 126B, and connection 123G in receive a small or no amount of TCP SYN-ACK packets from server 128A along one or more end-to-end paths, this may indicate that there is a network failure in switch 126A, connection 129A, and / or server 128A. This is because switch 126A, connection 129A, and server 128A represent a portion of one or more end-to-end paths between server 128A and UE 148, which is outside the internal network corresponding to network device 200.

[0084] The new session data 238 may indicate a number of new sessions that failed to be established between each of the plurality of service provider devices (e.g., each of the servers 128) and one or more clients of the plurality of client devices (e.g., UE 148) via the network device 200. As used herein, the term "new session" may refer to a new session according to TCP and / or a new session according to one or more other communication session protocols (e.g., UDP, TLS, and ICMP). For example, the new session data 238 may indicate a number of new sessions that failed to be established between the server 128A and one or more of the UEs 148 via the network device 200, a number of new sessions created between the server 128B and one or more of the UEs 148 via the network device 200B, and the like. The "number" of new sessions that failed to be established may refer to the total number of new sessions that failed to be established, the rate at which new sessions failed to be established, the number of new sessions that failed to be established over a period of time, or any combination thereof. In some examples, new session data 238 may indicate a number of new sessions established via network device 200 between each of a plurality of service provider devices (eg, each of servers 128 ) and one or more of a plurality of client devices (eg, UE 148 ).

[0085] The network device 200 may output the network traffic transmission data 232, the network traffic reception data 234, the TCP data 236, the new session data 238, or any combination thereof to a network management system (e.g., Figures 1A to 1CIn some examples, the network device 200 may continuously output the network traffic transmission data 232, the network traffic reception data 234, the TCP data 236, the new session data 238, or any combination thereof to the network management system. In some examples, the network device 200 may output the network traffic transmission data 232, the network traffic reception data 234, the TCP data 236, the new session data 238, or any combination thereof to the network management system in a block sequence, and output each block of the block sequence according to an output frequency (e.g., one block per second, one block per minute, one block per hour). In some examples, the network device 200 may output the network traffic transmission data 232, the network traffic reception data 234, the TCP data 236, the new session data 238, or any combination thereof to the network management system in response to receiving a request from the network management system.

[0086] Since the network device 200 can represent Figure 1A and Figure 1C The network devices 122 may be one of the network devices 122, and thus the network traffic transmission data 232, the network traffic reception data 234, the TCP data 236, and the new session data 238 may represent network traffic and session data corresponding to only one of the network devices 122. Each of the network devices 122 may collect and store the network traffic and session data, and each of the network devices 122 may output the network traffic and session data to the NMS 130. In some examples, the network traffic transmission data 232, the network traffic reception data 234, the TCP data 236, and the new session data 238 may be different from the network traffic and session data corresponding to one or more other network devices of the network device 122.

[0087] Figure 3 is a block diagram of an example NMS 300 according to one or more techniques of the present disclosure. NMS 300 may be used to implement, for example Figures 1A to 1C In this example, NMS 300 is responsible for monitoring and managing Figure 1A One or more wireless networks 106 at the site 102. In some examples, the NMS 300 is responsible for monitoring and managing Figure 1A In some examples, NMS 300 is responsible for monitoring and managing network devices 122 in network 120. Figure 1A The switch 126 and the server 128 are configured as follows.

[0088] In some examples, the NMS 300 and the network device 122 are located within an internal network controlled by an administrator. That is, the administrator device 111 can output one or more instructions based on receiving instructions from the administrator to control, configure, or otherwise manage the network device 122 and / or the NMS 130. For example, the administrator device 111 can output one or more instructions to configure multiple internal network paths through the network device 122 as part of multiple end-to-end paths. In some examples, each of the multiple end-to-end paths can connect one or more UEs 148 to one or more servers of the server 128A. However, at least a portion of the end-to-end path between a UE in the UE 148 and a server in the server 128 can be located outside the internal network corresponding to the network device 122 and the NMS 300. For example, the switch 126, the server 128, the connection 129, the AP 142, and the switch 146 can be located outside the internal network. This means that although the end-to-end path between UE 148A-1 and server 128A traverses an internal network path passing through network device 122A within the internal network, in some cases, the end-to-end path between UE 148A-1 and server 128A may traverse one or more external network paths including AP 142A-1, switch 146A, switch 126A, and connection 129A located outside the internal network.

[0089] NMS 300 includes processing circuitry 306, memory 308, user interface 310, communication interface 312, and database 318. Memory 308 is configured to store application programming interface (API) 322 and VNA 360 including machine learning model 362 and model training engine. Communication interface 330 includes receiver 324 and transmitter 326. Database 318 is configured to store network traffic and session data 370 and training data 380. Network traffic and session data 370 includes network traffic transmission data 372, network traffic reception data 374, TCP data 376, and new session data 378. Although database 318 is Figure 3 308 is shown as being separate from memory 308 , but in some examples, memory 308 is configured to store database 318 .

[0090] The various components are coupled together via bus 314, and the various components can exchange data and information via bus 314. Processing circuit 306 may be Figure 1A to Figure 1B The memory 308 may be an example of a processing circuit 132 of FIG. Figure 1A to Figure 1B An example of memory 133 of VNA 360 may be Figure 1A to Figure 1B An example of a VNA 134 of FIG. 360 is shown in FIG. 361 . The machine learning model 362 may be Figure 1A to Figure 1BAn example of a machine learning model 135 of the network traffic and session data 370 may be Figure 1A to Figure 1B An example of network traffic and session data 136. Training data 380 may be Figure 1A to Figure 1B An example of 138 training data.

[0091] In some examples, NMS 300 receives data from one or more of network devices 122, switches 126, servers 128, APs 142, switches 146, UEs 148, routers 187, and other network nodes within network system 100, which data may be used to calculate one or more metrics corresponding to network system 100. NMS 300 may analyze the data for cloud-based monitoring and / or management of wireless network 106 at site 102, monitoring and / or management of network devices 122, and monitoring and / or management of switches 126 and servers 128. In some examples, NMS 300 may be Figure 1A part of another server as shown, or part of any other server.

[0092] Processing circuit 306 may include fixed function circuits and / or programmable processing circuits. Processing circuit 306 may include any one or more of a microprocessor, a controller, a DSP, a GPU, a TPU, an ASIC, an FPGA, or an equivalent discrete or analog logic circuit. In some examples, processing circuit 306 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, GPUs, TPUs, one or more ASICs, or one or more FPGAs, as well as other discrete or integrated logic circuits, which may be physically located in one or more devices at one or more physical locations. Processing circuit 306 may execute software instructions stored in a computer-readable storage medium (e.g., memory 308), for example, software instructions for defining software or a computer program, such as a non-temporary computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to enable processing circuit 306 to perform the techniques described herein.

[0093] Processing circuit 306 may be capable of processing instructions stored in memory 308. In some examples, memory 308 includes a computer-readable medium that includes instructions that, when executed by processing circuit 306, cause NMS 300 and processing circuit 306 to perform various functions attributed to them herein. Memory 308 may include any volatile, non-volatile, magnetic, optical, or electrical medium, such as RAM, ROM, NVRAM, EEPROM, FRAM, DRAM, flash memory, or any other digital medium. Memory 308 may include one or more devices configured to store programming modules and / or data associated with the operation of NMS 300. For example, memory 308 may include a computer-readable storage medium, such as a storage device (e.g., a disk drive or optical drive) or a memory (e.g., flash memory or RAM) or a non-transitory computer-readable medium that stores instructions to cause processing circuit 306 to perform the techniques described herein.

[0094] A user, such as an administrator, may interact with the NMS 300 through the user interface 310. The user interface 310 may include a display, such as a liquid crystal display (LCD), a light emitting diode (LED) display, or other type of screen, with which the processing circuit 306 may present information related to the NMS 300, network devices, or other devices of the network system 100. In addition, the user interface 310 may include an input mechanism for receiving input from a user. The input mechanism may include, for example, any one or more of a button, a keypad (e.g., an alphanumeric keypad), a peripheral pointing device, a touch screen, or another input mechanism that allows a user to navigate and provide input in a user interface presented by the processing circuit 306 of the NMS 300. In other examples, the user interface 310 also includes audio circuitry for providing audible notifications, instructions, or other sounds to the user, receiving voice commands from the user, or both. The memory 308 may include instructions for operating the user interface 310.

[0095] The communication interface 312 may include, for example, an Ethernet interface. The communication interface 312 couples the NMS 300 to a network and / or the Internet, for example, Figure 1A The communication interface 312 includes a receiver 324 and a transmitter 326, and the NMS 300 transmits the data to the client device server 110, 116, 118, the network device 122, the server 128, the switch 146, the UE 148 and / or the like via the receiver 324 and the transmitter 326. Figures 1A to 1C100. In some embodiments, the NMS 300 may transmit data and information to or receive data and information from any of the other network nodes, devices, or systems that are part of the illustrated network system 100. In some scenarios described herein where the network system 100 includes "third-party" network devices owned and / or associated with an entity other than the NMS 300, the NMS 300 does not receive, collect, or otherwise access network data from the third-party devices. For example, when the switch 126 and / or the server 128 are third-party devices located outside the internal network corresponding to the NMS 300 and the network devices 122, the NMS 300 does not receive, collect, or otherwise access network data from the switch 126 and / or the server 128. Additionally or alternatively, when the AP 142, the switch 146, and the UE 148 are third-party devices located outside the internal network corresponding to the NMS 300 and the network devices 122, the NMS 300 does not receive, collect, or otherwise access network data from the AP 142, the switch 146, and the UE 148.

[0096] The data and information received by NMS 300 via communication interface 312 may include, for example, network traffic and session data 370. Figure 3 As shown, network traffic and session data 370 includes network traffic transmission data 372, network traffic reception data 374, TCP data 376, and new session data 378. In some examples, network traffic transmission data 372 may include network traffic transmission information from each of one or more network devices, including corresponding Figure 2 The network traffic transmission information 232 of the network device 200. In some examples, the network traffic reception data 374 may include network traffic reception information from each of the one or more network devices, including corresponding Figure 2 The network traffic receiving information 234 of the network device 200. In some examples, the TCP data 376 may include TCP data from each of the one or more network devices, including corresponding Figure 2 TCP data 236 of network device 200. In some examples, new session data 378 may include new session data from each of the one or more network devices, including corresponding Figure 2 New session data 238 of network device 200.

[0097] In some examples, network traffic transmission data 372 may indicate an amount of network traffic transmitted by each of network devices 122 via each of a plurality of internal network paths through network device 122. In some examples, network traffic reception data 374 may indicate an amount of network traffic received by each of network devices 122 via each of a plurality of internal network paths through network device 122. In some examples, TCP data 376 may indicate a number of TCP SYN packets, TCP SYN-ACK packets, and / or TCP ACK packets forwarded by each of network devices 122 via each of a plurality of internal network paths through network device 122. In some examples, new session data 378 may indicate a number of new sessions that failed to be established via each of network devices 122 corresponding to each of a plurality of internal network paths.

[0098] In some examples, as network device 122 receives network traffic, transmits network traffic, forwards TCP handshake packets, and establishes new sessions, NMS 300 can continuously receive network traffic and session data 370. In some examples, NMS 300 can output a request for network traffic and session data 370 and receive network traffic and session data 370 in response to the request from network device 122. In some examples, NMS 300 can receive network traffic and session data 370 in a sequence of blocks, with each block in the sequence of blocks arriving according to a scheduled frequency (e.g., every second, every minute, or every hour).

[0099] Processing circuit 306 may apply machine learning model 362 to detect a network failure outside of an internal network corresponding to NMS 300 and network device 122 based on network traffic and session data 370. That is, machine learning model 362 may process network traffic and session data 370 to generate an output indicating a network failure. In some examples, to apply machine learning model 362 to detect a network failure, processing circuit 306 is configured to determine that one or more network devices 122 are unable to exchange network traffic with a server in servers 128 via one or more external network paths. In some examples, to apply machine learning model 362 to detect a network failure, processing circuit 306 is further configured to determine that the network failure is associated with a network connection in a plurality of network connections (e.g., connection 129) outside of the internal network along the external network path, wherein the network connection is between the server in server 128 and network device 122.

[0100] In some examples, the machine learning model 362 can generate an output indicating the precise location of the detected network failure. Since the NMS 300 receives network traffic and session data 370 from multiple network devices (e.g., each network device 122), the machine learning model 362 can analyze the network traffic and session data 370 corresponding to multiple internal network paths through the multiple network devices. These multiple internal network paths through the multiple network devices can be associated with multiple external network paths outside the internal network corresponding to the NMS 300 and the network devices 122. Multiple end-to-end paths can exist between the UE 148 and the server 128, each of the multiple end-to-end paths including one or more internal network paths and one or more external network paths. The machine learning model 362 can identify the location corresponding to the network failure outside the internal network based on the network traffic and session data 370 received from within the internal network.

[0101] When network traffic and session data 370 indicates Figure 1C One or more sessions with server 128A of the embodiment of the present invention are operable via one or more paths connected to server 128A through connection 123H, switch 126B, and connection 129B, and network traffic and session data 370 indicates that the network traffic and session data 370 are connected to server 128A of the embodiment of the present invention. Figure 1C When one or more sessions to server 128A via one or more end-to-end paths connected to server 128A through connection 123G, switch 126A, and connection 129A are inoperable, machine learning model 362 can determine that the network fault is located at connection 129A and / or switch 126A. This is because machine learning model 362 will determine that there is no network fault at server 128A because the session with server 128A via switch 126B and connection 129B is active. Because machine learning model 362 analyzes data collected from multiple network devices within the internal network that form multiple end-to-end paths connected to one or more devices outside the internal network, machine learning model 362 can more accurately identify the location of the network fault compared to a system that does not use a machine learning model to analyze data from network devices within the internal network.

[0102] The machine learning model 362 can identify network failures using trends associated with network failures that the machine learning model 362 learns when trained using the training data 380. For example, the machine learning model 362 can identify trends in the network traffic transmission data 372 and the network traffic reception data 374 that indicate a high level of network traffic transmission to a server in the servers 128 outside the internal network and a low level of network traffic reception from the server. The machine learning model 362 is not limited to identifying levels of network traffic transmission and / or reception associated only with a server. The machine learning model 362 can analyze network traffic based on one or more external network paths to the server. For example, the machine learning model 362 can determine that there is a high level of network traffic transmission to a server in the servers 128 that includes the internal network. Figure 1C high level network traffic transmission to and low level network traffic reception from server 128A via one or more paths of connection 123H, switch 126B, and connection 129B, and via one or more paths including Figure 1C The machine learning model 362 can determine based on the identified patterns that there is a network failure associated with the switch 126B and / or the connection 129B, and there is no network failure associated with the switch 126A and / or the connection 129A.

[0103] In some examples, the machine learning model 362 can identify a network failure by identifying one or more trends associated with the TCP data 376. TCP is a protocol that can establish a session between two endpoint devices. To establish a TCP session, a first endpoint device can send a TCP SYN packet to a second endpoint device, the second endpoint device can send a TCP SYN-ACK packet to the first endpoint device, and the first endpoint device can send a TCP ACK packet to the second endpoint device. This exchange of SYN, SYN-ACK, and ACK packets is sometimes referred to as a "three-way handshake." When a TCP session fails, one or more endpoint devices can attempt to reestablish the session by initiating another three-way handshake. This can be referred to as a "TCP reset." The machine learning model 362 can identify a trend indicating that there is an increased amount of TCP resets associated with one of the servers 128 and / or an increased amount of TCP / IP resets associated with one or more external network paths associated with one of the servers 128. In some examples, machine learning model 362 may determine that there is an increased volume of TCP SYN packets sent to a server in servers 128 via one or more external network paths, and a decreased volume or no volume of TCP SYN-ACK packets received from the server in response to the SYN packets. Machine learning model 362 may determine that there is a network failure in one or more external network paths based on identifying the trend.

[0104] The machine learning model 362 is not limited to analyzing the TCP three-way handshake to identify network failures. In some examples, the machine learning model 362 may be configured to analyze new session data 378 corresponding to one or more protocols other than TCP (e.g., UDP, TLS, ICMP, or any combination thereof) to identify network failures. That is, for each of the network devices 122, the new session data 378 may indicate the number of new sessions that failed to be established according to any communication session protocol via the network device corresponding to each of the servers 128. In some examples, the machine learning model 362 may identify a pattern indicating that the number of new sessions established between a server in the server 128 and the UE 148 via one or more internal network paths and / or external network paths has decreased from a baseline level to zero or near zero. The machine learning model 362 may determine that there is a network failure at the server 128A or at one or more external network paths connected to the server 128A based on identifying the trend.

[0105] The processing circuit 306 of the NMS 300 can apply the model training engine 364 to train the machine learning model 362 using the training data 380. In some examples, the training data 380 can include multiple sets of training data. In some examples, each set of training data in the multiple sets of training data can include network traffic and session data, which includes at least one of network traffic transmission data, network traffic reception data, TCP data, and new session data. In this way, each set of training data in the multiple sets of training data can include data that is similar in kind to the data from the training data. Figure 1A Training data 380 may include data similar to data present in network traffic and session data 370 received by network device 122. In some examples, training data 380 may include multiple sets of training network data corresponding to network traffic and session data received from network device 122, separate from network traffic and session data 370, but this is not required. Training data 380 may include multiple sets of training network data corresponding to any network device within any network.

[0106] Each set of training data in the plurality of sets of training data 380 may include information indicating whether the set of training data is associated with a network failure. In some examples, additionally or alternatively, each set of training data in the plurality of sets of training data 380 may be associated with information whether the set of training data is associated with a network failure. For example, the plurality of sets of training data may be associated with information indicating a type of network failure associated with the data (e.g., a black hole or another type of network failure), a location of the network failure associated with the data (e.g., at a server, at a switch, at a connection), a severity of the network failure (e.g., a complete network failure or a partial network failure), or any combination thereof.

[0107] Based on the information associated with each of the multiple sets of training data and the multiple sets of training information based on the training data 380, the processing circuit 306 can apply the model training engine 364 to train the machine learning model 362 using supervised learning. Supervised machine learning involves using a labeled set of training data to train a machine learning model based on one or more patterns between the labels of the training data and the training data itself. For example, information indicating whether a set of training data is associated with a network failure is a label. In the process of training the machine learning model 362, the model training engine 364 can identify one or more patterns of the training data 380 associated with a network failure and / or identify one or more patterns of the training data 380 associated with a healthy network. The machine learning model 362 can learn these patterns when it is trained by the model training engine 264, so that once trained, the machine learning model 363 can detect network failures based on these patterns.

[0108] The model training engine 364 is not limited to using supervised learning to generate the machine learning model 364. Example machine learning techniques that can be used to generate the machine learning model 352 include various learning styles, such as supervised learning, unsupervised learning, and semi-supervised learning. Example algorithm types include Bayesian algorithms, Markov models, Hawkes processes, clustering algorithms, decision tree algorithms, regularization algorithms, regression algorithms, instance-based algorithms, artificial neural network algorithms, deep learning algorithms, dimensionality reduction algorithms, etc. Various examples of specific algorithms include Bayesian linear regression, enhanced decision tree regression, neural network regression, back propagation neural network, self-attention model, convolutional neural network (CNN), long short-term network (LSTM), prior algorithm, K-means clustering, K-nearest neighbor (kNN), learning vector quantization (LVQ), self-organizing map (SOM), local weighted learning (LWL), ridge regression (Ridgeregression), least absolute shrinkage and selection operator (LASSO), elastic net (Elastic net), least angle regression (LARS), principal component analysis (PCA) and principal component regression (PCR).

[0109] In some examples, when the NMS 300 uses the machine learning model 362 to identify a network based on the network traffic and session data 370, the NMS 300 may determine whether the machine learning model 363 accurately identified the network fault. For example, when the NMS 300 determines that the machine learning model 362 correctly identified the network fault, correctly identified the location of the network fault, incorrectly identified the network fault, incorrectly identified the location of the network fault, or any combination thereof, the NMS 300 may save the information together with the network traffic and session data 370 used to identify the network fault. In some examples, the NMS 300 may save the network traffic and session data 370 used to identify the network fault and the information indicating the accuracy of the identification to the training data 380.

[0110] In some examples, the model training engine 364 can retrain the machine learning model 362 based on the training data 380, which has been augmented with the network traffic and session data 370 for identifying network failures and information indicating the accuracy of the identification. In some examples, the NMS 300 can continuously receive information from Figure 1A The NMS 300 may further include a processor 364 and a processor 365 that receives new network traffic and session data 370 received from the network devices 122, continuously augments the training data 380 using the network traffic and session data 370 and information indicating the accuracy of the network fault identification, and continuously uses the model training engine 364 to retrain the machine learning model 362 using the augmented training data 380. In other words, the NMS 300 may improve the machine learning model 362 as it is used.

[0111] Figure 4 An example UE device 400 is shown, in accordance with one or more techniques of this disclosure. Figure 4 The example UE device 400 shown in FIG. 4 may be used to implement the Figure 1A Any UE 148 shown and described. UE device 400 may include any type of wireless client device, and the present disclosure is not limited in this respect. For example, UE device 400 may include a mobile device, such as a smart phone, a tablet or laptop computer, a personal digital assistant (PDA), a wireless terminal, a smart watch, a smart ring, or any other type of mobile or wearable device. In some examples, UE device 400 may also include a wired client device, such as an IoT device, such as a printer, a security sensor or device, an environmental sensor, or any other device connected to a wired network and configured to communicate via one or more wireless networks.

[0112] UE device 400 includes wired interface 430, wireless interfaces 420A-420C, processing circuit 406, memory 408, and user interface 410. The various components are coupled together via bus 414, and the various components can exchange data and information through the bus. Wired interface 430 represents a physical network interface and includes a receiver 432 and a transmitter 434. If necessary, wired interface 430 can be used to communicate via a cable (e.g., Figure 1A An Ethernet cable 144) couples the UE device 400 directly or indirectly to a wired network device within the wired network, such as, Figure 1A A switch 146.

[0113] The first, second and third wireless interfaces 420A, 420B and 420C include receivers 422A, 422B and 422C, respectively. Each receiver includes a receiving antenna, and the UE device 400 can receive signals from a wireless communication device (e.g., Figure 1A AP142, Figure 6 The first, second and third wireless interfaces 420A, 420B and 420C further include transmitters 424A, 424B and 424C, respectively, each of which includes a transmitting antenna, and the UE device 400 can send wireless signals to wireless communication devices (e.g., Figure 1A AP 142 Figure 6The first wireless interface 420A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz), and the second wireless interface 420B may include a Bluetooth interface and / or a Bluetooth low energy interface. The third wireless interface 420C may include, for example, a cellular interface, through which the UE device 400 may connect to a cellular network.

[0114] Processing circuit 406 may include fixed function circuits and / or programmable processing circuits. Processing circuit 406 may include any one or more of a microprocessor, a controller, a DSP, a GPU, a TPU, an ASIC, an FPGA, or an equivalent discrete or analog logic circuit. In some examples, processing circuit 406 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, GPUs, TPUs, one or more ASICs, or one or more FPGAs, and other discrete or integrated logic circuits, which may be physically located in one or more devices at one or more physical locations. Processing circuit 406 executes software instructions stored in a computer-readable storage medium (e.g., memory 408), such as software instructions for defining software or computer programs, which medium, for example, includes a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or a non-temporary computer-readable medium storing instructions to enable processing circuit 406 to perform any other type of volatile or non-volatile memory of the technology described herein.

[0115] Processing circuit 406 may be capable of processing instructions stored in memory 408. In some examples, memory 408 includes a computer-readable medium that includes instructions that, when executed by processing circuit 406, cause UE device 400 and processing circuit 406 to perform various functions attributed to them herein. Memory 408 may include any volatile, non-volatile, magnetic, optical, or electrical medium, such as RAM, ROM, NVRAM, EEPROM, FRAM, DRAM, flash memory, or any other digital medium. Memory 408 includes one or more devices configured to store programming modules and / or data associated with the operation of UE device 400. For example, memory 408 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause processing circuit 406 to perform the techniques described herein.

[0116] The user can interact with the UE device 400 through the user interface 410. The user interface 410 may include a display such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED), or other types of screens, and the processing circuit 406 may use the display to present information related to the UE device 400 and / or one or more services received by the UE device 400. In addition, the user interface 410 may include an input mechanism for receiving input from the user. The input mechanism may include, for example, any one or more of a button, a keypad (e.g., an alphanumeric keypad), a peripheral pointing device, a touch screen, or another input mechanism that allows the user to navigate and provide input in a user interface presented by the processing circuit 406 of the UE device 400. In other examples, the user interface 410 also includes an audio circuit for providing audible notifications, instructions, or other sounds to the user, receiving voice commands from the user, or both. The memory 408 may include instructions for operating the user interface 410.

[0117] In this example, memory 408 includes operating system 440, application programs 442, communication module 444, configuration settings 450, and data 454. Communication module 444 includes program code that, when executed by processing circuit 406, enables UE device 400 to communicate using any of wired interface 430, wireless interfaces 420A-420B, and / or cellular interface 450C. Configuration settings 450 include any device settings for UE device 400 settings for each of wireless interfaces 420A-420B and / or cellular interface 420C.

[0118] The data 454 may include, for example, a status / error log that includes a list of events specific to the UE device 400. The events may include a log of normal events and error events, depending on the log level based on instructions from the NMS 130. The data 454 may store any data used and / or generated by the UE device 400, such as data used to calculate one or more metrics or identify relevant behavioral data, which is collected by the UE device 400 and transmitted directly to the NMS 130 or to any AP 142 in a wireless network in the wireless network 106 for further transmission to the NMS 130.

[0119] As described herein, UE device 400 may measure and report network data from data 454 to NMS 130. The network data may include event data, telemetry data, and / or other data. In some examples, the network data may include data corresponding to one or more sessions between UE device 400 and a service provider device such as a server in server 128. For example, UE device 400 may form one or more sessions with a service provider device such as a server of a video streaming service. Data 454 may include information corresponding to one or more sessions, including information indicating the quality of one or more sessions and / or data corresponding to one or more failed sessions. The network data may include various parameters indicating the performance and / or status of a wireless network.

[0120] Figure 5 is a block diagram illustrating an example network node 500 according to one or more techniques of the present disclosure. In one or more examples, the network node 500 implements an Figure 1A A device or server of the network 120, such as the switch 146, the AAA server 110, the administrator device 111, the DHCP server 116, the DNS server 118, the switch 126, the network server 128, the switch 146, or a device or server supporting the wireless network 106, the wired LAN 175 or the SD-WAN 177 or Figure 1B Another network device in one or more of the data centers 179, such as a router 187.

[0121] In this example, the network node 500 includes a wired interface 502 (e.g., an Ethernet interface), a processing circuit 506, a memory 508, and an input / output 508 (e.g., a display, a button, a keyboard, a keypad, a touch screen, a mouse, etc.) coupled together via a bus 514, and the various elements can exchange data and information through the bus. The wired interface 502 couples the network node 500 to a network, such as an enterprise network. Although only one interface is shown by way of example, the network node can and typically does have multiple communication interfaces and / or multiple communication interface ports. The wired interface 502 includes a receiver 520 and a transmitter 522.

[0122] The processing circuit 506 may include fixed function circuits and / or programmable processing circuits. The processing circuit 506 may include any one or more of a microprocessor, a controller, a DSP, a GPU, a TPU, an ASIC, an FPGA, or an equivalent discrete or analog logic circuit. In some examples, the processing circuit 506 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, GPUs, TPUs, one or more ASICs, or one or more FPGAs, and other discrete or integrated logic circuits, which may be physically located in one or more devices at one or more physical locations. The processing circuit 506 may execute software instructions stored in a computer-readable storage medium (e.g., memory 508), such as those used to define software or computer programs, which medium, for example, includes a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or a non-temporary computer-readable medium storing instructions to enable the processing circuit 506 to perform the technology described herein.

[0123] Processing circuit 506 may be capable of processing instructions stored in memory 508. In some examples, memory 508 includes a computer-readable medium that includes instructions that, when executed by processing circuit 506, cause network node 500 and processing circuit 506 to perform various functions attributed to them herein. Memory 508 may include any volatile, non-volatile, magnetic, optical, or electrical medium, such as RAM, ROM, NVRAM, EEPROM, FRAM, DRAM, flash memory, or any other digital medium. Memory 508 may include one or more devices configured to store programming modules and / or data associated with the operation of network node 500. For example, memory 508 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause processing circuit 506 to perform the techniques described herein.

[0124] Memory 508 stores executable software applications 532, data 534, operating system 540, and NMS agent 544. Data 534 may include system logs and / or error logs that store event data (including behavioral data) for network node 500. In examples where network node 500 comprises a "third party" network device, the same entity does not own or have access to both the AP or wired client device and network node 500. Thus, in examples where network node 500 is a third party network device, NMS 130 does not receive, collect, or otherwise access network data from network node 500.

[0125] In an example where the network node 500 includes a server, the network node 500 can receive data and information via a receiver 520, the data and information including, for example, operation-related information, such as a handshake request, a session request, a registration request, an AAA service, a DHCP request, a simple notification service (SNS) lookup, and a web page request; and send data and information via a transmitter 522, the data and information including, for example, configuration information, authentication information, web page data, services, etc.

[0126] In an example where the network node 500 includes a wired network device, the network node 500 can be connected to one or more APs or other wired client devices, such as IoT devices, via a wired interface 502. For example, the network node 500 can include multiple wired interfaces 502 and / or the wired interface 502 can include multiple physical ports to connect to multiple APs or other wired client devices inside the site via corresponding Ethernet cables. In some examples, each AP or other wired client device connected to the network node 500 can access the wired network via the wired interface 502 of the network node 500. In some examples, one or more of the APs or other wired client devices connected to the network node 500 can each draw power from the network node 500 via a corresponding Ethernet cable and a Power over Ethernet (PoE) port of the wired interface 502.

[0127] In examples where network node 500 includes a session-based router that employs a stateful, session-based routing scheme, network node 500 can be configured to independently perform path selection and traffic engineering. The use of session-based routing can enable network node 500 to avoid using a centralized controller (e.g., an SDN controller) to perform path selection and traffic engineering, and to avoid the use of tunnels. In some examples, network node 500 can implement session-based routing as Security Vector Routing (SVR) provided by Juniper Networks, Inc. In examples where network node 500 includes a session-based router (e.g., an SDN controller) that operates as a network gateway for an enterprise network site, network node 500 can implement session-based routing as a security vector router (SVR) provided by Juniper Networks, Inc. Figure 1B In the case of router 187A), network node 500 may communicate with each other through an underlying physical WAN (e.g., Figure 1B SD-WAN 177) establishes multiple peer paths with one or more other session-based routers (e.g., Figure 1B 189), these routers serve as network gateways for other sites in the enterprise network (e.g. Figure 1B The network node 500 operating as a session-based router can collect data at the peer path level and report the peer path data to the NMS 130.

[0128] In examples where network node 500 includes a packet-based router, network node 500 may employ a packet-based or flow-based routing scheme to forward packets according to defined network paths established, for example, by a centralized controller that performs path selection and traffic engineering. In examples where network node 500 includes a packet-based router (e.g., Figure 1B In the case of router 187A), network node 500 may communicate with each other through an underlying physical WAN (e.g., Figure 1B SD-WAN 177) with one or more other packet-based routers operating as network gateways for other sites of the enterprise network (e.g., Figure 1B Router 187B) establishes multiple tunnels (for example, Figure 1B Path 189). The network node 500 operating as a packet-based router can collect data at the tunnel level, and the tunnel data can be retrieved by the NMS 130 via an API or open configuration protocol, or the tunnel data can be reported to the NMS 130 by the NMS agent 544 or another module running on the network node 500.

[0129] The data collected and reported by the network node 500 may include periodically reported data and event-driven data. The network node 500 is configured to collect logical path statistics via bidirectional forwarding detection (BFD) detection and data extracted from messages and / or counters at the logical path (e.g., peer path or tunnel) level. In some examples, the network node 500 is configured to collect statistics and / or sample other data according to a first periodic interval (e.g., every 3 seconds, every 5 seconds, etc.). The network node 500 may store the collected and sampled data as path data in, for example, a buffer.

[0130] In some examples, the network node 500 optionally includes an NMS agent 544. The NMS agent 544 may periodically create a statistical data package according to a second periodic interval (e.g., every 3 minutes). In some examples, the statistical data package may also include details about the clients connected to the network node 500 and the relevant client sessions. The NMS agent 544 may then report the statistical data package to the NMS 130 in the cloud. In other examples, the NMS 130 may request, retrieve, or otherwise receive the statistical data package from the network node 500 via an API, an open configuration protocol, or another communication protocol. The statistical data package created by the NMS agent 544 or another module of the network node 500 may include a header identifying the network node 500 and statistics and data samples for each logical path from the network node 500. In other examples, when an event occurs, in response to the occurrence of some event at the network node 500, the NMS agent 544 reports event data to the NMS 130 in the cloud.

[0131] Figure 6 is a block diagram of an example AP device 600 , in accordance with one or more techniques of this disclosure. Figure 6 The example AP device 600 shown in FIG. 1 may be used to implement the Figure 1A Any AP 142 shown and described. AP device 600 may include, for example, a Wi-Fi, Bluetooth, and / or Bluetooth Low Energy (BLE) base station or any other type of wireless access point.

[0132] exist Figure 6 In the example of FIG. 6 , the AP device 600 includes a wired interface 630, wireless interfaces 620A-620B, processing circuitry 606, memory 608, and input / output 210 coupled together via a bus 614, and the various components can exchange data and information through the bus. The wired interface 630 represents a physical network interface and includes a receiver 632 and a transmitter 634 for sending and receiving network communications (e.g., packets). The wired interface 630 directly or indirectly couples the AP device 600 to a wired network device within a wired network, such as a wired network device, via a cable such as an Ethernet cable. Figure 1A One of the switches 146.

[0133] The first and second wireless interfaces 620A and 620B represent wireless network interfaces and include receivers 622A and 622B, respectively, each of which includes a receiving antenna, via which the AP device 600 can receive signals from wireless communication devices (e.g., Figure 1A The first and second wireless interfaces 620A and 620B further include transmitters 624A and 624B, respectively, each of which includes a transmitting antenna, and the AP device 600 can transmit a wireless communication signal to a wireless communication device (e.g., Figure 1A In some examples, the first wireless interface 620A may include a Wi-Fi 802.11 interface (eg, 2.4 GHz and / or 5 GHz), and the second wireless interface 620B may include a Bluetooth interface and / or a BLE interface.

[0134] Processing circuit 606 may include fixed function circuits and / or programmable processing circuits. Processing circuit 606 may include any one or more of a microprocessor, a controller, a DSP, a GPU, a TPU, an ASIC, an FPGA, or an equivalent discrete or analog logic circuit. In some examples, processing circuit 606 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, GPUs, TPUs, one or more ASICs, or one or more FPGAs, and other discrete or integrated logic circuits, which may be physically located in one or more devices at one or more physical locations. Processing circuit 606 may execute software instructions stored in a computer-readable storage medium (e.g., memory 608), such as software instructions for defining software or computer programs, such as a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or a non-temporary computer-readable medium storing instructions to enable processing circuit 606 to perform the technology herein, such as any other type of volatile or non-volatile memory.

[0135] The processing circuit 606 may be capable of processing instructions stored in the memory 608. In some examples, the memory 608 includes a computer-readable medium that includes instructions that, when executed by the processing circuit 606, cause the network node 500 and the processing circuit 606 to perform various functions attributed to them herein. The memory 608 may include any volatile, non-volatile, magnetic, optical, or electrical medium, such as RAM, ROM, NVRAM, EEPROM, FRAM, DRAM, flash memory, or any other digital medium. The memory 608 may include one or more devices that are configured to store programming modules and / or data related to the operation of the AP device 600. For example, the memory 608 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause the processing circuit 606 to perform the techniques described herein.

[0136] In this example, the memory 608 stores executable software, including an API 640, a communication manager 642, configuration settings 650, a device status log 652, data 654, and a log controller 655. The device status log 652 includes a list of events specific to the AP device 600. The events may include a log of normal events and error events, such as memory status, restart or reboot events, crash events, cloud disconnection events with self-recovery, low link speed or link speed swing events, Ethernet port status, Ethernet interface packet errors, upgrade failure events, firmware upgrade events, configuration changes, etc., as well as time and date stamps for each event. The log controller 655 determines the logging level of the device based on instructions from the NMS 130. The data 654 may store any data used and / or generated by the AP device 600, including data collected from the UE 148 (e.g., data used to calculate one or more metrics), which is sent by the AP device 600 for the NMS 130 to perform cloud-based management of the wireless network 106.

[0137] Input / output (I / O) 610 represents physical hardware components that can interact with a user, such as buttons, displays, etc. Although not shown, memory 608 typically stores executable software for controlling a user interface with respect to input received via I / O 610. Communication manager 642 includes program code that, when executed by processing circuit 606, allows AP device 600 to communicate with UE 148 and / or network 120 via any wired interface 630 and / or wireless interface 620A-620B. Configuration settings 650 include any device settings of AP device 600, such as radio settings for each wireless interface 620A-620B. These settings can be manually configured, or can be remotely monitored and managed by NMS 130 to optimize wireless network performance periodically (e.g., hourly or daily).

[0138] As described herein, the AP device 600 can measure network data from the status log 652 and report it to the NMS 130. The network data may include event data, telemetry data, and / or other data. The network data may include various parameters indicating the performance and / or state of the wireless network. These parameters may be measured and / or determined by one or more UE devices and / or one or more APs in the wireless network. The NMS 130 may determine one or more metrics based on data received from the AP in the wireless network and store the metrics in a memory. In an example where the AP device 600 includes a "third-party" network device, the same entity does not own or cannot access both the NMS 130 and the AP device 600. Therefore, in an example where the AP device 600 is a third-party network device, the NMS 130 does not receive, collect, or otherwise access network data from the AP device 600.

[0139] Figure 7 is a flowchart illustrating example operations for detecting one or more network failures using a machine learning model in accordance with one or more techniques of the present disclosure. The example operations are relative to Figures 1A to 1C However, Figure 7 The techniques may be performed by different components of sensor network system 100, or by additional or alternative systems.

[0140] NMS 130 may be configured to receive network data (702) from a network device 122 of an internal network managed by network management system 130. In some examples, network device 122 may connect UE 148 with server 128 via one or more paths through an internal network corresponding to network device 122. In some examples, the network data may include network traffic data and / or session data (e.g., Figures 1A to 1C The network traffic and session data 136 may include data corresponding to network traffic that traverses one or more paths through the internal network corresponding to the network device 122. The session data may include data corresponding to one or more sessions between the UE 148 and the server 128 via one or more paths through the internal network. In some examples, the network data received by the NMS 130 includes network data corresponding to each of the network devices 122. The internal network may include the NMS 130 and the network device 122, but one or more devices may be located outside the internal network, within one or more external networks. For example, the server 128 and the UE 148 may be located in an external network.

[0141] The processing circuit 132 of the NMS 130 may be configured to apply a machine learning model to detect a network failure within the external network based on the network data (704). For example, the machine learning model 135 may process the network data to identify a network failure. The machine learning model 135 may be trained to discern one or more patterns in the network data corresponding to a network failure and / or to discern one or more patterns in the network data corresponding to healthy network operation. For example, the machine learning model 135 may be trained using supervised learning based on a set of labeled training data. The training data may include data similar in kind to the network data received from the network device 122 and one or more tags indicating information corresponding to the training data. The training data may include tags indicating whether the data is associated with a network failure or associated with a healthy network.

[0142] In some examples, the external network includes one or more external devices (e.g., one or more of servers 128) and an edge device (e.g., a switch in switches 126) that connect the one or more external network devices to the internal network. To detect a network failure, in some examples, processing circuitry 132 may be configured to apply machine learning model 135 to detect a failure of a link between the edge device and one of the plurality of external network devices (e.g., connection 129A between switch 126A and server 128A). Additionally or alternatively, to detect a network failure, processing circuitry 122 is configured to apply a machine learning model to detect a failure of one of the external network devices (e.g., one of servers 128).

[0143] In some examples, the external network may represent a first external network (e.g., an external network including switch 126A, switch 126B, and server 128A). To detect network failures, processing circuit 122 may be configured to apply machine learning model 135 to detect blackholing of network traffic by a first edge device (e.g., switch 126A of the first external network). In some examples, a second edge device of the first external network (e.g., switch 126B) is not associated with blackholing. In some examples, a second external network (e.g., an external network including switch 126C and server 128B) is not associated with blackholing.

[0144] NMS 130 may perform corrective actions to remedy the network failure within the external network (706). Even though the network failure may be located in the external network, NMS 130 may be configured to reroute network traffic within the internal network so that when the network traffic leaves the internal network, the network traffic does not reach the network failure. For example, when Figure 1CWhen a network failure at connection 129A of the NMS 130 prevents network traffic from reaching server 128A via connection 123G, switch 126A, and connection 129A, NMS 130 may reroute network traffic to server 128A via connection 123H, switch 126B, and connection 129B to circumvent the network failure at connection 129A. Additionally or alternatively, NMS 130 may reroute network traffic to server 128B via connection 123I, switch 126C, and connection 129C to circumvent the network failure at connection 129A when servers 128A and 128B are controlled by the same service provider and / or provide similar services. Rerouting network traffic is not the only corrective action available to NMS 130. For example, NMS 130 may control network device 122 to avoid forwarding network traffic to an address associated with the network failure, withdraw routing to a location associated with the network failure, send a diagnostic message to an address associated with the network failure to confirm the network failure within an external network, or any combination thereof.

[0145] Figure 8 is a flowchart illustrating example operations for training a machine learning model to detect network failures according to one or more techniques of the present disclosure. The example operations are relative to Figures 1A to 1C However, Figure 8 The techniques may be performed by different components of network system 100, or by additional or alternative systems.

[0146] The processing circuit 132 of the NMS 300 is configured to train the machine learning model 135 using the training data 138 to detect one or more network faults associated with each of the plurality of sets of training data (802). Each set of training network data in the plurality of sets of training network data may be associated with network status information indicating whether the set of training network data is associated with one or more network fault events. For example, the network status information may include information indicating whether the set of training network data is associated with a network fault or with healthy network operation, information indicating a nature of the network fault associated with the set of training network data (e.g., a black hole or other type of network fault), information indicating a location of the network fault associated with the set of training network data (e.g., a location related to one or more network devices), or any combination thereof.

[0147] By training the machine learning model 135, the processing circuit 132 of the NMS 130 can enable the machine learning model 135 to identify one or more training network data patterns associated with network failure events. Since each set of training network data in the plurality of sets of training data 138 includes network state information indicating whether the set of training network data is associated with one or more network failure events, the processing circuit 132 of the NMS 130 can train the machine learning model 135 to recognize that the one or more patterns are training data 138 associated with the plurality of sets of training network data that are marked as associated with network failure events. Additionally or alternatively, the processing circuit 132 of the NMS 130 can train the machine learning model 135 to recognize that the one or more patterns are training data 138 associated with the plurality of sets of training network data that are marked as associated with healthy network operations.

[0148] Processing circuitry 132 of NMS 130 may apply machine learning model 135 to detect network failures based on network data received from a plurality of network devices 122 of the internal network (804). In some examples, at block 806, processing circuitry 132 may execute Figure 7 The processing circuit 132 of the NMS 130 may apply the machine learning model 135 to detect network faults based on network data received from the network device 122 using one or more other techniques. The processing circuit 132 of the NMS 130 may receive information indicating whether the detection of the network fault by the machine learning model 135 is accurate (806). The processing circuit 132 of the NMS 130 may train the machine learning model 135 using the network data and the information indicating whether the detection of the network fault by the machine learning model 135 is accurate (808).

[0149] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as separate but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuit may be implemented as one or more integrated circuit devices, e.g., an integrated circuit chip or chipset.

[0150] If implemented in hardware, the present disclosure may be directed to a device, such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store such instructions that are executed by a processor.

[0151] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0152] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may mean that the storage medium is not contained in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache).

[0153] The code or instructions may be software and / or firmware executed by a processing circuit that includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the present disclosure may be provided within a software module or a hardware module.

Claims

1. A computer network management system, comprising: a memory configured to store the machine learning model; as well as a processing circuit in communication with the memory and configured to: receiving network data from a plurality of network devices of an internal network managed by the network management system, wherein the internal network is connected to an external network isolated from the network management system; applying the machine learning model to detect network failures within the external network based on the network data; and Corrective action is performed to remedy the network failure within the external network.

2. The computer network management system according to claim 1, in, The external network includes a plurality of external network devices and an edge device connecting the plurality of external network devices to the internal network, and In order to detect the network failure, the processing circuit is configured to apply the machine learning model to detect a failure of a link between the edge device and one of the multiple external network devices.

3. The computer network management system according to claim 1, in, The external network includes a plurality of external network devices and an edge device connecting the plurality of external network devices to the internal network, and In order to detect the network failure, the processing circuit is configured to apply the machine learning model to detect a failure of one of the multiple external network devices.

4. The computer network management system according to claim 1, in, The external network comprises a first external network, wherein, in order to detect the network failure, the processing circuit is configured to apply the machine learning model to detect a black hole behavior of the first edge device of the first external network on the network traffic, and Wherein, in order to perform the corrective action, the processing circuit is configured to control the multiple network devices to forward network traffic to a second edge device of a second external network instead of the first edge device of the first external network, wherein the second external network is isolated from the network management system and separated from the first external network.

5. The computer network management system according to claim 1, in, The external network includes a plurality of external network devices, a first edge device and a second edge device, wherein the first edge device and the second edge device connect the plurality of external network devices to the internal network. wherein, in order to detect the network failure, the processing circuit is configured to apply the machine learning model to detect the black hole behavior of the first edge device on the network traffic, and Wherein, in order to perform the corrective action, the network management system is configured to control the plurality of network devices to forward network traffic to the second edge device instead of the first edge device.

6. The computer network management system according to claim 1, wherein: The network data includes at least one of the following: Network traffic transmission (T x ) data indicating an amount of network traffic transmitted from each of the plurality of network devices; Network traffic reception (R x ) data indicating an amount of network traffic received by each of the plurality of network devices; Transmission Control Protocol TCP data indicating (1) a volume of TCP synchronization SYN messages transmitted by each of the plurality of network devices, (2) a volume of TCP synchronization acknowledgement SYN-ACK messages received by each of the plurality of network devices, or (3) a volume of TCP reset messages received by each of the plurality of network devices; as well as New session data indicating a number of new sessions between one or more service provider devices and one or more client devices that failed to be established via the network device.

7. The computer network management system according to claim 6, wherein: To apply the machine learning model to detect the network fault, the processing circuit is configured to perform any one or more of the following: Identify T among multiple sets of network parameters x Data and R x a trend in the data, the trend indicating a high level of transmitted network traffic transmitted to a device of the external network and a low level of received network traffic from the device of the external network; identifying a trend in the TCP data, the trend indicating a high level of TCP SYN messages transmitted to the device of the external network and a low level of TCP SYN-ACK messages received from the device of the external network; as well as A trend in the new session data is identified, the trend indicating a high number of new sessions that failed to be established with the device of the external network.

8. The computer network management system according to claim 1, in, To apply the machine learning model to detect the network failure within the external network, the processing circuit is configured to apply the machine learning model to predict a network failure within the external network that may occur at a future time based on the network data; as well as The processing circuit is configured to perform the corrective action before the predicted network failure occurs to remedy the predicted network failure.

9. The computer network management system according to any one of claims 1 to 8, in, The memory is configured to store training data, wherein the training data includes a plurality of sets of training data, each set of training data in the plurality of sets of training data is marked with information indicating whether the set of training data is associated with one or more network failures, and The processing circuit is configured to use the training data to train the machine learning model to detect one or more network faults associated with each set of training data in the multiple sets of training data.

10. The computer network management system according to any one of claims 1 to 8, wherein: The processing circuit is further configured to: receiving information indicating whether the machine learning model accurately detected the network fault; The machine learning model is trained using the network data and the information indicating whether the machine learning model accurately detects the network fault.

11. The computer network management system according to any one of claims 1 to 8, wherein: The internal network is a software-defined wide area network (SD-WAN), and wherein the external network includes one or more of a customer network, an Internet service provider network, and a cloud service provider network connected to the SD-WAN.

12. The computer network management system according to any one of claims 1 to 8, wherein: To perform the corrective action to remedy the network failure within the external network, the processing circuit is configured to perform one or more of the following: controlling the plurality of network devices to avoid forwarding network traffic to an address associated with the network failure; withdrawing routing to the address associated with the network failure; as well as A diagnostic message is sent to the address associated with the network fault to confirm the network fault within the external network.

13. A computer networking method, comprising: receiving, by processing circuitry of a network management system, network data from a plurality of network devices of an internal network managed by the network management system, wherein the internal network is connected to an external network isolated from the network management system, and wherein the processing circuitry is in communication with a memory of the network management system configured to store a machine learning model; applying, by the processing circuitry, the machine learning model to detect network faults within the external network based on the network data; and Corrective action is performed by the processing circuit to remedy the network failure within the external network.

14. The computer networking method according to claim 13, in, The external network includes a plurality of external network devices and an edge device connecting the plurality of external network devices to the internal network, and Wherein, detecting the network failure includes: applying the machine learning model by the processing circuit to detect a failure of a link between the edge device and one of the multiple external network devices.

15. The method according to claim 13, in, The external network includes a plurality of external network devices and an edge device connecting the plurality of external network devices to the internal network, and Among them, detecting the network failure includes: the processing circuit applies the machine learning model to detect the failure of one of the multiple external network devices.

16. The computer networking method according to claim 13, in, The external network comprises a first external network, wherein detecting the network failure comprises applying, by the processing circuit, the machine learning model to detect a black hole behavior of a first edge device of the first external network on network traffic, and Wherein, performing the corrective action includes controlling the multiple network devices by the processing circuit to forward network traffic to a second edge device of a second external network instead of the first edge device of the first external network, wherein the second external network is isolated from the network management system and separated from the first external network.

17. The computer networking method according to claim 16, wherein: The network data includes at least one of the following: Network traffic transmission (T x ) data indicating an amount of network traffic transmitted from each of the plurality of network devices; Network traffic reception (R x ) data indicating an amount of network traffic received by each of the plurality of network devices; Transmission Control Protocol TCP data indicating (1) a volume of TCP synchronization SYN messages transmitted by each of the plurality of network devices, (2) a volume of TCP synchronization acknowledgement SYN-ACK messages received by each of the plurality of network devices, or (3) a volume of TCP reset messages received by each of the plurality of network devices; as well as New session data indicating a number of new sessions between one or more service provider devices and one or more client devices that failed to be established via the network device.

18. The computer networking method according to any one of claims 13 to 17, in, The memory is configured to store training data, wherein the training data includes a plurality of sets of training data, each set of training data in the plurality of sets of training data is marked with information indicating whether the set of training data is associated with one or more network failures, and The method further includes training the machine learning model using the training data to detect one or more network faults associated with each set of training data in the multiple sets of training data.

19. The computer networking method according to any one of claims 13 to 17, wherein: The internal network is a software-defined wide area network (SD-WAN), and wherein the external network includes one or more of a customer network, an Internet service provider network, and a cloud service provider network connected to the SD-WAN.

20. A computer readable medium comprising instructions which, when executed by a processing circuit, cause the processing circuit to: receiving network data from a plurality of network devices of an internal network managed by a network management system, wherein The internal network is connected to an external network isolated from the network management system; applying a machine learning model to detect network failures within the external network based on the network data; as well as Corrective actions are applied to remedy the network failure within the external network.

Images