Network device, computer networking method, and computer readable storage medium

Abstract

The invention relates to a network apparatus, a computer networking method, and a computer readable storage medium. Techniques are disclosed for sending downloadable software packages that can be executed to perform testing of a network device without the need for the network device to update firmware. In one example, a network device obtains a software package including instructions to emulate a network instance, and stores the software package in a volatile memory of the network device. The network device simulates the network instance based on the instructions to obtain data indicative of performance of the network instance. A network device transmits data indicating performance of a network instance via a communication channel established with a network management system (NMS) based on a software package. In some examples, the NMS uses data indicative of performance of the network instance to identify a root cause of a network device failure or remedial action to be performed on the network device.

Description

[0001] This application claims priority to U.S. Application 19 / 093,141, filed March 27, 2025, which claims the benefit of Indian Application 202441038170, filed May 15, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to computer networks, and more specifically, to the monitoring and troubleshooting of computer networks. Background Technology

[0003] Commercial locations or sites (such as offices, hospitals, airports, stadiums, or retail stores) typically install sophisticated wireless network systems (including networks of wireless access points (APs)) throughout the premises to provide wireless network services to one or more wireless client devices (or simply "clients"). An AP is a physical electronic device that uses various wireless network protocols and technologies to enable other devices to wirelessly connect to a wired network, such as Wireless Local Area Network protocols (i.e., "WiFi") conforming to one or more IEEE 802.11 standards, Bluetooth / Bluetooth Low Energy (BLE), mesh networking protocols (such as ZigBee), or other wireless network technologies. Many different types of wireless client devices (such as laptops, smartphones, tablets, wearables, appliances, and Internet of Things (IoT) devices) incorporate wireless communication technologies and can be configured to connect to a compatible wireless access point to access a wired network when the device is within range of the access point. Summary of the Invention

[0004] In general, this disclosure describes techniques for sending downloadable software packages to a network device, which can be executed to perform tests on the network device without requiring firmware updates performed by the network device. In one example, a network management system (NMS) identifies triggering events associated with a network device among a plurality of network devices managed by the NMS. Based at least in part on the identified triggering events, the NMS selects a software package that includes instructions for the network device to simulate a network instance. The NMS sends the selected software package to the network device (or the network device obtains it from the NMS).

[0005] The network device receives a software package containing instructions for simulating a network instance and stores the package in the network device's volatile memory. The network device simulates the network instance based on the instructions to obtain data indicating the network instance's performance. The network device transmits the data indicating the network instance's performance via a communication channel established with the NMS based on the software package. In some examples, the NMS uses the data indicating the network instance's performance to identify the root cause of a network device failure or remedial actions to be performed on the network device.

[0006] The technology disclosed herein provides one or more technical advantages and practical applications. For example, the technology disclosed herein enables the dynamic deployment of downloadable network performance modules that configure network devices within an enterprise network to perform network functions, such as network testing, without requiring the installation of firmware that would typically disrupt the network. Furthermore, the technology disclosed herein enables network devices to download and execute software modules, thereby enabling network testing functions that were previously impossible for the network device to perform, without requiring the network device to update or reinstall firmware, operating system, restart or reboot the network device, or interrupt network traffic forwarding. Because such network devices eliminate the need to reinstall or update firmware to perform such network tests, network devices as described herein can enable a wider range of network testing without incurring significant administrative overhead to update such devices or imposing additional network downtime to perform such tests. Therefore, the technology disclosed herein can enhance the ability of network devices to perform network performance monitoring, troubleshooting, diagnosis, analysis, and remediation without disrupting normal network operations or network traffic forwarding.

[0007] In one example, this disclosure describes a network apparatus including: volatile memory; and processing circuitry in communication with the storage medium, the processing circuitry being configured to: obtain a software package including instructions for simulating a network instance; store the software package in the volatile memory; simulate the network instance based on the instructions specified by the software package to obtain data indicating the performance of the network instance; and transmit the data indicating the performance of the network instance via a communication channel established between the software package and a network management system (NMS).

[0008] In another example, this disclosure describes a network management system (NMS) including: processing circuitry communicating with a storage medium, the processing circuitry being configured to: identify trigger events associated with a network device among a plurality of network devices managed by the network management system; select a software package from a plurality of software packages based at least in part on the identified trigger events, the selected software package including instructions for simulating a network instance; and send the selected software package to the network device to cause the network device to simulate the network instance, thereby obtaining data indicative of the performance of the network instance.

[0009] Details of one or more examples of the technology disclosed herein are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of these technologies will be apparent from the specification, drawings, and claims. Attached Figure Description

[0010] Figure 1A This is a block diagram of an exemplary network system based on one or more technologies of this disclosure, the exemplary network system being configured to provide downloadable software packages to perform test functions or other functions to monitor network performance.

[0011] Figure 1B It is shown Figure 1A A block diagram showing other exemplary details of the network system.

[0012] Figure 2 This is a block diagram of an exemplary access point device according to one or more technologies of this disclosure.

[0013] Figure 3 This is a block diagram of an exemplary network management system based on one or more technologies of this disclosure.

[0014] Figure 4 This is a block diagram of an exemplary user equipment device according to one or more technologies of this disclosure.

[0015] Figure 5 This is a block diagram of an exemplary network node (such as a router or switch) according to one or more technologies of this disclosure.

[0016] Figure 6 This is a flowchart illustrating exemplary operations of providing a downloadable software package to perform testing functions or other functions to monitor network performance according to one or more technologies disclosed herein. Detailed Implementation

[0017] This document discloses techniques for providing downloadable software packages that include instructions for enabling network devices to perform test functions or monitor network performance (collectively referred to herein as "network performance monitoring functions") without requiring the network device to reinstall or update the firmware or operating system executed by the network device. The downloadable software package may be referred to herein as a "network performance monitoring module" or a "downloadable mini-module." As an example, the software package enables a network device (e.g., an access point, router, switch, etc.) to simulate a network instance. In some examples, the network device simulates the operation of one or more client devices accessing the network via the network device. The network device collects data associated with the simulated network instance, which may be sent to a network management system (NMS) configured to manage the network. The NMS can use the data associated with the simulated network instance to analyze the performance of the network and / or the network device.

[0018] In some examples, the network device may obtain downloadable software packages from the NMS or a server storing downloadable packages. Instructions may be specified, for example, the network performance monitoring function to be performed by the network device (e.g., the specific type of network test, the data type to be collected, and / or measurements, etc.), instructions to configure communication channels to transmit data collected by the network device to the NMS, and / or the data format used to transmit the collected data (e.g., JSON). The data format may be specified, for example, the message payload, which includes information such as the name of the network performance monitoring function, the identifier of the network performance monitoring function, one or more categories of the collected data (e.g., Kafka topics), and / or the values ​​of the collected data (e.g., latency, jitter, packet loss, etc.).

[0019] The network device mounts downloadable software packages as temporary files (e.g., tmp) in a random access memory (RAM) file system (“ramfs”) or other volatile memory. The network device's administrator or user can invoke the downloadable network performance monitoring module via, for example, a command-line interface (CLI) or other interface. The administrator or user can specify trigger commands or other information for the downloadable network performance monitoring module to execute network performance monitoring functions. These trigger commands specify, for example, the identifier of the network performance monitoring function, the name of the network performance monitoring function, one or more categories of data collected (e.g., Kafka topics), the schedule for executing the network performance monitoring function (e.g., the frequency of test execution), the network configuration for executing the network performance monitoring function (e.g., VLANs), and / or the resource requirements for executing the network performance monitoring function (e.g., memory and / or CPU limits).

[0020] Network devices can perform network performance monitoring functions and collect data related to the performance of these functions. For example, a network device can simulate a network instance of a synthetic client device performing a network function (e.g., requesting access to an application server in the network) and generate a report of the collected data associated with the network function. The report can represent a JSON file specifying the values ​​of the collected data. As an example, a network device can perform a speed test by sending and receiving network traffic to and from a test server and measuring the download and upload speeds of the network traffic (e.g., latency, round-trip times per minute (rpm), throughput, etc.) and generate a report specifying the measured download and upload speeds. As another example, a network device can perform continuous pings to a specific destination (e.g., a domain and / or server) by measuring the round-trip time (RTT) of network traffic transmitted from the network device (e.g., average RTT, maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of packets received, number of packets transmitted, etc.). The network functions described above are just some examples of functions performed by a downloadable network performance module. Alternatively or additionally, downloadable software packages may include instructions to perform other functions typically performed by the NMS, such as specifying AI / ML models to enable network devices to examine data collected locally on the enterprise network. In some examples, reports may additionally or alternatively specify information about active and / or downloadable network performance modules installed on network devices, such as runtime statistics (e.g., CPU utilization, memory utilization, version, etc.) for each active and / or downloadable network performance module installed on the network device.

[0021] Network devices transmit collected data to the NMS via communication channels. The NMS can determine network performance based on the collected data. In some examples, the NMS determines that there are problems or anomalies within the network based on the collected data. Based at least in part on the determination that there are problems or anomalies, the NMS performs actions to remedy the detected problems, such as generating a notification specifying the detected problems and / or recommendations to remedy them, or automatically performing remedial actions (e.g., resetting network devices, changing the configuration of network devices, etc.).

[0022] Figure 1A This is a block diagram of an exemplary network system 100 according to one or more technologies of this disclosure, configured to provide downloadable software packages to perform testing functions or other functions to monitor network performance. The exemplary network system 100 includes multiple sites 102A to 102N, at which a network service provider manages one or more wireless networks 106A to 106N, respectively. Although in Figure 1AIn the present invention, each site 102A to 102N is shown as comprising a single wireless network 106A to 106N, but in some examples each site 102A to 102N may include multiple wireless networks, and the present disclosure is not limited in this respect.

[0023] Each site 102A to 102N includes multiple network access server (NAS) devices, such as access points (APs) 142, switches 146, and routers (not shown). For example, site 102A includes multiple APs 142A-1 to 142A-N. Similarly, site 102N includes multiple APs 142N-1 to 142N-M. Each AP 142 can be any type of wireless access point, including but not limited to commercial or enterprise APs, routers, or any other device connected to a wired network and capable of providing wireless network access to client devices within the site. References to “N” or “M” can represent any number. References to “N” do not need to be the same number for different components. Similarly, references to “M” do not need to be the same number for different components.

[0024] Each site 102A to 102N also includes multiple client devices, also known as user equipment devices (UEs), generally referred to as UEs or client devices 148, representing various wireless-enabled devices within each site. For example, multiple UEs 148A-1 to 148A-N are currently located at site 102A. Similarly, multiple UEs 148N-1 to 148N-M are currently located at site 102N. Each UE 148 can be any type of wireless client device, including but not limited to mobile devices (such as smartphones, tablets or laptops, personal digital assistants (PDAs), wireless terminals, smartwatches, smart rings, or other wearable devices). UE 148 may also include wired client-side devices, such as IoT devices (such as printers), security devices, environmental sensors, or any other device connected to a wired network and configured to communicate over one or more wireless networks 106.

[0025] To provide wireless network services to UE 148 and / or communicate via wireless network 106, AP 142 and other wired client-side devices at site 102 are directly or indirectly connected to one or more network devices (e.g., switches, routers, etc.) via physical cables (e.g., Ethernet cables). Figure 1A In the example, site 102A includes switch 146A, and each of APs 142A-1 to 142A-N at site 102A is connected to this switch. Similarly, site 102N includes switch 146N, and each of APs 142N-1 to 142N-M at site 102N is connected to this switch 146N. Although in Figure 1AThe diagram shows what appears to be a single switch 146 for each site 102, with all APs 142 at a given site 102 connected to this single switch 146. However, in other examples, each site 102 may include more or fewer switches and / or routers. Additionally, APs and other wired client devices at a given site may connect to two or more switches and / or routers. Furthermore, two or more switches at a site may be interconnected with each other and / or connected to two or more routers, for example, via a mesh or partial mesh topology in a hub-and-spoke architecture. In some examples, the interconnected switches and routers comprise a wired local area network (LAN) at site 102 hosting the wireless network 106.

[0026] The exemplary network system 100 also includes various network components for providing network services within a wired network. For example, it includes an authentication, authorization, and accounting (AAA) server 110 for authenticating users and / or UE 148; a dynamic host configuration protocol (DHCP) server 116 for dynamically assigning network addresses (e.g., IP addresses) to UE 148 during authentication; a domain name system (DNS) server 122 for resolving domain names to network addresses; multiple servers 128A to 128N (collectively referred to as "Server 128") (e.g., web servers, database servers, file servers, etc.); and a network management system (NMS) 130. Figure 1A As shown, various devices and systems of network 100 are coupled together via one or more networks 134 (e.g., the Internet and / or corporate intranets).

[0027] exist Figure 1A In the examples, NMS 130 is a cloud-based computing platform for managing one or more wireless networks 106A to 106N at sites 102A to 102N. As further described herein, NMS 130 provides an integrated set of management tools and implements various technologies disclosed herein. Typically, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analytics, network anomaly identification, and alarm generation. In some examples, NMS 130 outputs notifications (such as alarms, warnings, graphical indicators on dashboards, log messages, text / SMS messages, email messages, etc.) and / or suggestions regarding wireless network issues to site or network administrators (“administrators”) who interact with and / or operate administrator device 111. Additionally, in some examples, NMS 130 operates in response to configuration input received from administrators who interact with and / or operate administrator device 111.

[0028] The administrator and administrator device 111 may include IT personnel and administrator computing devices associated with one or more sites 102. The administrator device 111 may be implemented as any suitable device for presenting output and / or accepting user input. For example, the administrator device 111 may include a display. The administrator device 111 may be a computing system, such as a mobile or non-mobile computing device operated by a user and / or an administrator. According to one or more aspects of this disclosure, the administrator device 111 may, for example, represent a workstation, a laptop or notebook computer, a desktop computer, a tablet computer, or any other computing device that can be operated by a user and / or present a user interface. The administrator device 111 may be physically separate from and / or located in a different location from the NMS 130, such that the administrator device 111 can communicate with the NMS 130 via network 134 or other communication means.

[0029] In some examples, one or more of the NAS devices (e.g., AP 142, switch 146, and router) may be connected to edge devices 150A through 150N via physical cables (e.g., Ethernet cables). Edge device 150 includes a cloud-managed wireless local area network (LAN) controller. Each edge device 150 may include a locally deployed device at site 102 that communicates with NMS 130 to extend some microservices from NMS 130 to the locally deployed NAS device, while using NMS 130 and its distributed software architecture for scalable and resilient operation, management, troubleshooting, and analysis.

[0030] Each of the network devices in network system 100 (e.g., servers 110, 116, 122 and / or 128, AP 142, UE 148, switch 146, and any other servers or devices attached to or forming part of network system 100) may include a system log or error log module, wherein each of these network devices records the status of the network device, including normal operating status and error status. Throughout this disclosure, one or more of the network devices in network system 100 (e.g., servers 110, 116, 122 and / or 128, AP 142, UE 148, and switch 146) may be considered “third-party” network devices when owned and / or associated with an entity different from NMS 130, such that NMS 130 does not directly receive, collect, or otherwise access the status and other data recorded by the third-party network devices. In some examples, edge device 150 may provide a proxy through which the status and other data recorded by the third-party network devices may be reported to NMS 130.

[0031] In some examples, the NMS 130 monitors network data 137 (e.g., one or more Service Level Expectation (SLE) metrics) received from wireless networks 106A to 106N at each site 102A to 102N, and manages network resources (such as APs 142 at each site) to deliver a high-quality wireless experience to end users, IoT devices, and clients at the sites. For example, the NMS 130 may include a Virtual Network Assistant (VNA) 133 that implements an event processing platform for providing real-time insights and simplified troubleshooting for IT operations, and automatically takes corrective action or provides recommendations to proactively resolve wireless network issues. For example, the VNA 133 may include an event processing platform configured to process hundreds or thousands of concurrent streams of network data 137 from sensors and / or agents associated with nodes within AP 142 and / or network 134. For example, the VNA 133 of the NMS 130 may include underlying analytics and network error detection engines and alerting systems according to various examples described herein. The underlying analytics engine of VNA 133 can apply historical data and models to inbound event streams to calculate assertions, such as the identification of anomalies or the prediction of events constituting network error states. Furthermore, VNA 133 can provide real-time alerts and reports to notify site or network administrators of any predicted events, anomalies, or trends via administrator device 111, and can perform root cause analysis and automatic or assisted error remediation. In some examples, the VNA 133 of NMS 130 can apply machine learning techniques to identify the root causes of error states detected or predicted from the flow of network data 137. If the root cause can be resolved automatically, VNA 133 can invoke one or more corrective actions to correct the root cause of the error state, thereby automatically improving underlying SLE metrics and also automatically enhancing the user experience.

[0032] Further details of the operations implemented by the VNA 133 of the NMS 130 are described in the following documents: U.S. Patent No. 9,832,082, published November 28, 2017, entitled "Monitoring Wireless Access Point Events"; U.S. Publication No. US 2021 / 0306201, published September 30, 2021, entitled "Network System Fault Resolution Using a Machine Learning Model"; U.S. Patent No. 10,985,969, published April 20, 2021, entitled "Systems and Methods for a Virtual Network Assistant"; and U.S. Patent No. 10,958,585, published March 23, 2021, entitled "Methods and Apparatus for Facilitating Fault Detection and / or Predictive Fault". The following patents are incorporated herein by reference in their entirety: “Detection (Method and Apparatus for Facilitating Fault Detection and / or Predictive Fault Detection)”; U.S. Patent No. 10,958,537, issued March 23, 2021, entitled “Method for Spatio-Temporal Modeling”; and U.S. Patent No. 10,862,742, issued December 8, 2020, entitled “Method for Conveying AP Error Codes Over BLE Advertisements”.

[0033] In operation, NMS 130 observes, collects, and / or receives network data 137, which may take the form, for example, data extracted from messages, counters, and statistics. According to one particular implementation, a computing device is part of NMS 130. According to other implementations, NMS 130 may include one or more computing devices, dedicated servers, virtual machines, containers, services, or other forms of environments for performing the techniques described herein. Similarly, computing resources and components implementing VNA 133 may be part of NMS 130, may run on other servers or execution environments, or may be distributed across nodes within network 134 (e.g., routers, switches, controllers, gateways, etc.).

[0034] According to one or more techniques of this disclosure, the NMS 130 is configured to perform scheduling and / or orchestration of tests performed by devices at a site. In this example, the VNA 133 of the NMS 130 includes a network test transport module 135 configured to provide software packages to one or more devices (such as AP 142) to enable AP 142 to simulate network instances and use the simulated network instances to perform one or more network performance tests to obtain data indicative of the performance of the network instances. The NMS 130 can receive the data indicative of the performance of the network instances from AP 142 for fault detection, troubleshooting and root cause analysis, and to perform remedial measures on network system 100, as described in more detail below.

[0035] In one example, the network test transport module 135 of the VNA 133 of the NMS 130 identifies a trigger event. In some examples, the trigger event may include a network event indicating a potential problem that may require additional testing. For example, the network test transport module 135 may determine network anomalies that may require additional testing based on network data of AP 142A-1 and / or one or more client devices or other network devices associated with AP 142A-1. In some examples, the trigger event may be based on a schedule for performing tests. In other examples, the trigger event may be input received from administrator device 111. Based at least in part on the identified trigger event, the network test transport module 135 selects a software package from a plurality of downloadable software packages 136. Each software package includes a downloadable network performance monitoring module, which includes, for example, instructions for causing AP 142 to perform network performance monitoring functions. For example, the software package may include instructions for causing AP 142 to perform network performance tests by simulating network instances. The network test transmission module 135 sends (or the AP 142A-1 obtains from the network test transmission module 135) a selected software package to enable the AP 142A-1 to simulate a network instance to obtain data indicating the performance of the network instance.

[0036] In this example, the network test transmission module 135 stores the selected software package and sends the selected software module to the AP 142. In other examples, another device may store the selected software package, such as a server, software package repository, content delivery network, etc. In this case, the network test transmission module 135 can enable the AP 142 to obtain the selected software package from, for example, a server or software package repository storing the selected software package.

[0037] The AP 142A-1 acquires and stores a software package containing instructions for simulating network instances. In some examples, the AP 142A-1 optionally stores the software package in volatile memory, such that the software package is not retained when the AP 142A-1 is powered off, restarted, or rebooted. In some examples, the AP 142A-1 optionally stores the software package in non-volatile memory, such that the software package is retained after the AP 142A-1 is powered off, restarted, or rebooted.

[0038] AP 142A-1 simulates a network instance based on instructions specified by the software package to obtain data indicating the network instance's performance. AP 142A-1 establishes a communication channel with NMS 130 based at least in part on configuration information for the communication channel specified by the software package. AP 142A-1 transmits data indicating the network instance's performance to NMS 130 via the established communication channel.

[0039] The NMS 130 receives data indicating the performance of network instances from the AP 142A-1 via a communication channel established by the network device based on selected software packages. In some examples, the VNA 133 of the NMS 130 performs fault diagnosis, troubleshooting, or automatic fault recovery based at least in part on the received data indicating the performance of network instances. In some examples, the VNA 133 performs traffic engineering, network path selection, or network traffic load balancing based at least in part on the received data indicating the performance of network instances.

[0040] Although the techniques of this disclosure as performed by an access point device (e.g., AP 142A-1) are described in this example, the techniques described herein can be performed by any other device, such as client device 148, switch 146, router, and other devices.

[0041] Although the technology of this disclosure is described in this example as being performed by NMS 130, the technology described herein can be performed by any other computing device, system, and / or server, and this disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the technology of this disclosure may reside in a dedicated server, or be included in any other server besides NMS 130, or may be distributed throughout network 100, and may or may not be part of NMS 130.

[0042] Figure 1B It shows Figure 1A A block diagram illustrating other exemplary details of the network system. In this example, Figure 1BThe NMS 130 is shown, which is configured to operate based on an AI / machine learning-based computing platform that provides connectivity across "clients" (e.g., connected to wireless network 106 and wired LAN 175). Figure 1B The user device 148 (far left) is connected to the "cloud" (e.g., it can be connected to the data center 179). Figure 1B The cloud-based application services hosted on the far right side of the computing resources within the cloud offer comprehensive automation, insights, and assurance (WiFi assurance, wired assurance, and WAN assurance).

[0043] As described herein, NMS 130 provides an integrated set of management tools and implements various technologies disclosed herein. Typically, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analytics, network anomaly detection, and alarm generation. For example, the network management system 130 can be configured to proactively monitor and adaptively configure network 100 to provide automated capabilities. Furthermore, VNA 133 includes a natural language processing engine to provide AI-driven support and troubleshooting, anomaly detection, AI-driven location services, and AI-driven radio frequency (RF) optimization with reinforcement learning.

[0044] like Figure 1B As illustrated in the example, the AI-driven NMS 130 also provides configuration management, monitoring, and automated supervision of a software-defined wide area network (SD-WAN) 177, which operates as an intermediate network coordinating wireless network 106 and wired LAN 175 to data center 179 and application services 181. Typically, the SD-WAN 177 provides a seamless, secure, traffic-engineered connection between a “spoke” router 187A hosting wireless network 106 on a wired network 175 (such as a branch or campus network) and a “hub” router 187B further up towards cloud-based application services 181. The SD-WAN 177 typically operates and manages overlay networks on the underlying physical wide area network (WAN), which provides connectivity to geographically separated customer networks. In other words, the SD-WAN 177 extends software-defined networking (SDN) capabilities to the WAN and allows the network to decouple the underlying physical network infrastructure from virtualized network infrastructure and applications, enabling flexible and scalable network configuration and management.

[0045] In some examples, the underlying routers of the SD-WAN 177 can implement a stateful, session-based routing scheme, where routers 187A and 187B dynamically modify the contents of the raw packet headers originating from client device 148 to direct traffic along a selected path (e.g., path 189) to application service 181 without using channels and / or additional labels. In this way, routers 187A and 187B can be more efficient and scalable for large networks because using channelless, session-based routing allows routers 187A and 187B to utilize significant network resources by eliminating the need for encapsulation and decapsulation at channel endpoints. Furthermore, in some examples, each router 187A and 187B can independently perform path selection and traffic engineering to control the packet flow associated with each session, without requiring a centralized SDN controller for path selection and label distribution. In some examples, routers 187A and 187B implement session-based routing as Secure Vector Routing (SVR) provided by Juniper Networks.

[0046] Additional information regarding session-based routing and SVR is described in the following patents: U.S. Patent No. 9,729,439, entitled "Computer Network Packet Flow Controller," published August 8, 2017; U.S. Patent No. 9,729,682, entitled "Network Device and Method for Processing a Session Using a Packet Signature," published August 8, 2017; U.S. Patent No. 9,762,485, entitled "Network Packet Flow Controller with Extended Session Management," published September 12, 2017; and U.S. Patent No. 9,762,485, entitled "Router with Optimized Statistical..." U.S. Patent No. 9,871,748, entitled "FUNCTIONALITY (Router with Optimized Statistics)," published January 16, 2018; U.S. Patent No. 9,985,883, entitled "NAME-BASED ROUTING SYSTEM AND METHOD," published May 29, 2018; U.S. Patent No. 10,200,264, entitled "LINK STATUS MONITORING BASED ON PACKET LOSS DETECTION," published February 5, 2019; U.S. Patent No. 10,277,506, entitled "STATEFUL LOAD BALANCING IN A STATELESS NETWORK," published April 30, 2019; and U.S. Patent No. 10,277,506, entitled "NETWORK PACKET FLOWCONTROLLER WITH EXTENDED SESSION," published April 30, 2019. The entire contents of each of these patents are incorporated herein by reference in their entirety: U.S. Patent No. 10,432,522, issued October 1, 2019, entitled “Network Packet Flow Controller with Extended Session Management”; and U.S. Patent No. 11,075,824, issued July 27, 2021, entitled “In-Line Performance Monitoring”.

[0047] In some examples, the AI-driven NMS 130 can enable intent-based configuration and management of network system 100, including the construction, presentation, and execution of intent-driven workflows for configuring and managing devices associated with wireless network 106, wired LAN network 175, and / or SD-WAN 177. For example, declarative requirements represent the desired configuration of network components without specifying exact local device configurations and control flows. By utilizing declarative requirements, what should be done is specified, rather than how it should be done. Declarative requirements contrast with imperative instructions, which describe the exact device configuration syntax and control flow to achieve the configuration. By utilizing declarative requirements instead of imperative instructions, users and / or user systems are relieved of the burden of determining the exact device configurations required to achieve the user / system's desired results. For example, when utilizing various types of devices from different vendors, specifying and managing exact imperative instructions for configuring each device in the network is often difficult and cumbersome. The types and kinds of devices in the network can change dynamically with the addition of new devices and device failures. Managing and configuring cohesive networks of devices from different vendors with varying configuration protocols, syntaxes, and software versions is often challenging. Therefore, managing and configuring network devices becomes more efficient by requiring only declarative requirements from the user / system (which specify the expected results applicable to various device types). Further exemplary details and techniques of intent-based network management systems are described in U.S. Patent No. 10,756,983, entitled "Intent-based Analytics," and U.S. Patent No. 10,992,543, entitled "Automatically generating an intent-based network model of an existing computer network," each of which is incorporated herein by reference.

[0048] According to one or more techniques of this disclosure, the NMS 130 is configured to perform scheduling and / or orchestration of tests performed by devices at a site. In this example, the VNA 133 of the NMS 130 includes a network test transport module 135 configured to provide software packages (e.g., one or more downloadable software packages 136) to one or more devices (such as client device 148, NAS devices in wireless network 106 and / or wired network 175, or devices in SD-WAN 177) to enable the devices to simulate network instances based on the one or more downloadable software packages 136 provided by the network test transport module 135 and to perform one or more network performance tests using the simulated network instances to obtain data indicative of the performance of the network instances. The NMS 130 can receive the data indicative of the performance of the network instances from the devices performing the one or more downloadable software packages 136 for fault detection, troubleshooting and root cause analysis, and to perform remedial measures on network system 100, as described in more detail below.

[0049] In one example, the network test transport module 135 of the VNA 133 of the NMS 130 identifies a triggering event that may indicate a potential problem that may require additional testing. For example, the network test transport module 135 may determine network anomalies that may require additional testing based on network data associated with one of the network devices, client device 148, AP 142 of wireless network 106, and wired network 175 or SD-WAN 177. Based at least in part on the identified triggering event, the network test transport module 135 selects a software package from a plurality of downloadable software packages 136. Each software package includes a downloadable network performance monitoring module that includes, for example, instructions for causing a device (e.g., one of the APs 142 of wireless network 106) to perform network performance monitoring functions. For example, the software package may include instructions for causing one of the APs 142 of wireless network 106 to perform network performance testing by simulating a network instance. The network test transmission module 135 sends a selected software package to one of the APs 142 of the wireless network 106 to enable the APs 142 of the wireless network 106 to simulate a network instance to obtain data indicating the performance of the network instance.

[0050] One of the APs 142 in the wireless network 106 receives and stores a software package containing instructions for simulating a network instance. In some examples, the AP 142 optionally stores the software package in volatile memory, such that the software package is not retained when the AP 142 is powered off, restarted, or rebooted. In some examples, the AP 142 optionally stores the software package in non-volatile memory, such that the software package is retained after the AP 142 is powered off, restarted, or rebooted.

[0051] AP 142 simulates a network instance based on instructions specified by the software package to obtain data indicating the performance of the network instance. AP 142 establishes a communication channel with NMS 130 based at least in part on configuration information for the communication channel specified by the software package. AP 142 sends the data indicating the performance of the network instance to NMS 130 via the established communication channel.

[0052] NMS 130 receives data indicating the performance of network instances from AP 142 via a communication channel established by the network device based on selected software packages. In some examples, VNA 133 of NMS 130 performs fault diagnosis, troubleshooting, or automatic fault recovery based at least in part on the received data indicating the performance of network instances. In some examples, VNA 133 performs traffic engineering, network path selection, or network traffic load balancing based at least in part on the received data indicating the performance of network instances.

[0053] Although the technology of this disclosure is described in this example as being performed by NMS 130, the technology described herein can be performed by any other computing device, system, and / or server, and this disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the technology of this disclosure may reside in a dedicated server, or be included in any other server besides NMS 130, or may be distributed throughout network 100, and may or may not be part of NMS 130.

[0054] Figure 2 This is a block diagram of an exemplary access point (AP) device 200 based on one or more technologies of this disclosure. Figure 2 The exemplary access point 200 shown can be used to implement as described herein. Figure 1A Any of the APs shown and described in AP 142. Access point 200 may include, for example, a Wi-Fi, Bluetooth and / or Bluetooth Low Energy (BLE) base station or any other type of wireless access point.

[0055] exist Figure 2 In the example, access point 200 includes a wired interface 230, wireless interfaces 220A to 220B, one or more processors 206, memory 212, and input / output 210 coupled together via a bus 214, through which various components can exchange data and information. The wired interface 230 represents a physical network interface and includes a receiver 232 and a transmitter 234 for sending and receiving network communications (e.g., data packets). The wired interface 230 couples access point 200 directly or indirectly to wired network devices within a wired network, such as Ethernet cables, via cables (e.g., Ethernet cables). Figure 1A One of the switches in the 146.

[0056] The first wireless interface 220A and the second wireless interface 220B represent wireless network interfaces, and respectively include receiver 222A and receiver 222B, each receiver including a receiving antenna, and the access point 200 can receive signals from a wireless communication device (such as...) via the receiving antenna. Figure 1A The access point 200 receives wireless signals from the UE 148. The first wireless interface 220A and the second wireless interface 220B also include transmitters 224A and 224B, respectively. Each transmitter includes a transmitting antenna, and the access point 200 can transmit signals to wireless communication devices (such as…) via the transmitting antenna. Figure 1A The UE 148 in the example transmits wireless signals. In some examples, the first wireless interface 220A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz), and the second wireless interface 220B may include a Bluetooth interface and / or a Bluetooth Low Energy (BLE) interface.

[0057] Processor 206 is a programmable, hardware-based processor configured to execute software instructions (such as software instructions for defining software or computer programs) stored in a computer-readable storage medium (such as memory 212), such as a non-transitory computer-readable medium including storage devices (e.g., disk drives or optical disc drives) or memories (such as flash memory or RAM) or any other type of volatile or non-volatile memory, the stored instructions causing one or more processors 206 to perform the techniques described herein.

[0058] Memory 212 includes one or more means configured to store programming modules and / or data associated with the operation of access point 200. For example, memory 212 may include a computer-readable storage medium (such as a non-transitory computer-readable medium) including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, storing instructions to cause one or more processors 206 to perform the techniques described herein.

[0059] In this example, memory 212 stores executable software, including an application programming interface (API) 240, a communication manager 242, configuration settings 250, a device status log 252, a data storage device 254, and a log controller 255. The device status log 252 includes a list of events specific to access point 200. Events may include logs of normal and error events, such as, for example, memory status, reboot or restart events, crash events, self-recovery events due to cloud disconnection, low link speed or link speed fluctuation events, Ethernet port status, Ethernet interface packet errors, upgrade failure events, firmware upgrade events, configuration changes, etc., along with the time and date stamp for each event. The log controller 255 determines the device's logging level based on instructions from NMS 130. Data storage device 254 may store any data used and / or generated by access point 200, including data collected from UE 148 (such as data used to calculate one or more SLE metrics), which is sent by access point 200 for cloud-based management of the wireless network 106A by NMS 130.

[0060] Input / output (I / O) 210 represents physical hardware components that enable user interaction, such as buttons, displays, etc. Although not shown, memory 212 typically stores executable software for controlling the user interface regarding input received via I / O 210. Communication manager 242 includes program code that, when executed by processor 206, allows access point 200 to communicate with UE 148 and / or network 134 via interface 230 and / or any of the interfaces 220A to 220C. Configuration settings 250 include any device settings for access point 200, such as radio settings for each of the wireless interfaces 220A to 220C. These settings can be configured manually or remotely monitored and managed by NMS 130 to optimize wireless network performance periodically (e.g., hourly or daily).

[0061] As described herein, AP device 200 can measure and report network data from status log 252 to NMS 130. Network data may include event data, telemetry data, and / or other SLE-related data. Network data may include various parameters indicating the performance and / or status of the wireless network. These parameters may be measured and / or determined by one or more UE devices and / or one or more APs in the wireless network. NMS 130 can determine one or more SLE metrics based on the SLE-related data received from the APs in the wireless network and store the SLE metrics as network data 137. Figure 1A ).

[0062] According to the technology described in this disclosure, the AP device 200 obtains a software package that enables it to perform network performance tests by simulating network instances to obtain data indicative of the network instance's performance, without requiring the AP device 200 to update its firmware or operating system, or to reboot or restart. Therefore, the AP device 200 can simulate network instances to obtain data indicative of the network instance's performance without adversely affecting the forwarding of network traffic by the AP device 200. Furthermore, such technology avoids the additional management overhead required to upgrade the firmware or operating system of a conventional AP to enable it to perform additional or new network performance monitoring functions not currently programmed into the conventional AP.

[0063] For example, according to the techniques described herein, processor 206 is transmitted from a wired interface 230 or a wireless interface 220. Figure 1A and Figure 1B The NMS 130's VNA 133 obtains a software package. The software package includes a downloadable network performance monitoring module 258, which includes, for example, instructions for causing the processor 206 to perform network performance monitoring functions. For example, the network performance monitoring module 258 may include instructions for causing the processor 206 to perform network performance testing by simulating network instances to obtain data 257 indicative of the performance of the simulated network instances.

[0064] In some examples, the software package includes one or more of the following: network performance monitoring functions or network tests to be performed by the simulated network instance, configuration data for establishing a communication channel with NMS 130 for transmitting network performance monitoring data 257, a schedule for performing network performance monitoring functions or network tests, a virtual local area network (VLAN) to be used during the performance of network performance monitoring functions or network tests, or one or more resource requirements to be applied during the performance of network performance monitoring functions or network tests.

[0065] In some examples, the package specifies the data format for network performance monitoring data 257. The data format can specify, for example, the name or identifier of the network performance monitoring function or network test to be performed by the simulated network instance, one or more categories of network performance monitoring data 257, or one or more fields of latency, jitter, and packet loss values ​​to be specified by network performance monitoring data 257. In some examples, the data format is specified as JavaScript Object Notation (JSON), YAML, XML, CSV, or other data formats.

[0066] Processor 206 stores network performance monitoring module 258 in memory 212. Typically, processor 206 executes firmware configured to perform one or more network functions of AP 200, which is stored in non-volatile memory of memory 212. In some examples, processor 206 optionally stores network performance monitoring module 258 in volatile memory of memory 212 such that network performance monitoring module 258 is not retained when AP 200 is powered off, restarted, or rebooted. In some examples, processor 206 optionally stores network performance monitoring module 258 in non-volatile memory of memory 212 such that network performance monitoring module 258 is retained after AP 200 is powered off, restarted, or rebooted.

[0067] Processor 206 simulates a network instance based on instructions specified by network performance monitoring module 258 to obtain data (network performance monitoring data 257) indicating the performance of the network instance. For example, network performance monitoring module 258 can cause processor 206 to execute a simulated network instance 256 of the simulated network instance. For example, network performance monitoring module 258 may include configuration information for configuring the simulated network instance 256. Such configuration information may include, for example, network performance monitoring functions or network tests to be performed by the simulated network instance, configuration data of communication channels, data format of network performance monitoring data 257, schedule for performing network performance monitoring functions or network tests, VLANs to be used during the performance of network performance monitoring functions or network tests, or one or more resource requirements to be applied during the performance of network performance monitoring functions or network tests.

[0068] Processor 206 executes program instructions based on network performance monitoring module 258, which cause AP device 200 to execute simulated network instance 256 to simulate a network instance. In some examples, processor 206 executes simulated network instance 256 to execute a synthetic client device that establishes a connection with the simulated network instance of AP device 200 and / or forwards network data to the simulated network instance. In some examples, processor 206 executes simulated network instance 256 to execute a synthetic server, to which the simulated network instance forwards network data.

[0069] In some examples, to simulate network instances, processor 206 simulates one or more client devices (such as...) Figure 1A and Figure 1BThe client device 148) accesses network 134 via AP 200. In some examples, processor 206 performs network tests on a simulated network instance 256 to obtain data 257 indicative of the network instance's performance. For example, network testing may include speed testing. In this example, the simulated network instance 256 performs speed testing by exchanging network traffic between the simulated network instance 256 and a server and measuring at least one of the upload and download speeds of the network traffic.

[0070] As another example, network testing can include ping messages. In this example, simulated network instance 256 exchanges ping messages with the server and measures one or more of the following: average round-trip time (RTT), maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of packets received, and number of packets transmitted.

[0071] As another example, network testing can include authentication, authorization, and accounting (AAA) testing. In this example, a simulated network instance 256 attempts to initiate an AAA session with an AAA server (such as an off-network network access control (NAC) remote authentication dial-up user service (RADIUS) server) to obtain data useful for troubleshooting potential configuration problems accessing the AAA server, whether those configuration problems are due to a misconfigured AP 142, a problem with the AAA server, or a misconfigured network policy, etc.

[0072] Processor 206 can monitor traffic exchanged with simulated network instance 256 to generate network performance monitoring data 257. Data 257 indicating the performance of the network instance may include, for example, metrics and / or performance-related information of the network instance, such as latency, jitter, packet loss, CPU utilization, memory utilization, or version identifiers of AP 200, virtual client functionality, and / or network performance monitoring module 258.

[0073] In some examples, processing circuitry 206 may at least partially simulate network instance 256 to obtain network performance monitoring data 257 based on a trigger command. For example, processing circuitry 206 may respond to a trigger command received from a user (such as an administrator), or respond to a trigger command received from... Figure 1A and Figure 1B The NMS 130 receives the trigger command to simulate network instance 256.

[0074] As described above, processor 206 can download and execute network performance monitoring module 258 and simulate network instance 256 while continuously exchanging network traffic or otherwise interrupting normal forwarding of network traffic by AP 200. Additionally, in some examples, processor 206 can download and execute network performance monitoring module 258 and simulate network instance without requiring processor 206 to update the firmware of AP 200 stored in non-volatile memory of memory 212.

[0075] Processor 206 communicates with, at least in part, via wired interface 230 or wireless interface 220 based on configuration information of the communication channel specified by the software package. Figure 1A and Figure 1B The VNA 133 establishes a communication channel. The processor 206 sends network performance monitoring data 257 to the NMS 130 via the established communication channel. For example, the software package may specify interfaces 230 and 220 for establishing a communication channel with the NMS 130, network protocols for communicating via the communication channel, one or more network links for establishing a communication channel with the NMS 130, etc.

[0076] AP device 200 can send signals to NMS (e.g., Figure 1A The NMS 130 sends network performance monitoring data 257, or the NMS can retrieve network performance monitoring data from the AP device 200, and the NMS can perform measures based on the network performance monitoring data 257 obtained from the AP device 200, such as generating and sending a notification including an indication of the performance of the simulated network instance 256 and / or an indication of the root cause of the AP 200 failure (such as a network configuration problem, an AP configuration problem, or other problems causing problems within the network) (e.g., for display on the display device of the administrator device 111 or sent as a message to the administrator), or performing remedial measures to mitigate or resolve the problem of the AP 200 or another AP within the network, such as configuring the operation of the AP device 200 or restarting the AP device 200 to resolve the problem.

[0077] Figure 3 This is a block diagram of an exemplary network management system (NMS) 300 according to one or more technologies of this disclosure. The NMS 300 can be used to implement, for example... Figure 1A and Figure 1B The NMS 130 is used in this example. In such an example, the NMS 300 is responsible for monitoring and managing one or more wireless networks 106A to 106N at sites 102A to 102N.

[0078] The NMS 300 includes a communication interface 330, one or more processors 306, a user interface 310, a memory 312, and a database 318. The various components are coupled together via a bus 314, through which they can exchange data and information. In some examples, the NMS 300 receives data from client devices 148, APs 142, switches 146, and other network nodes within the network 134 (e.g., ...). Figure 1B One or more of the routers (187) receive data, which can be used to calculate one or more SLE metrics and / or update network data 316 in database 318. NMS 300 analyzes this data for cloud-based management of wireless networks 106A to 106N. In some examples, NMS 300 may be... Figure 1A This refers to a portion of another server or any other server shown in the diagram.

[0079] Processor 306 executes software instructions (such as those used to define software or computer programs) stored in a computer-readable storage medium (such as memory 312), such as a non-transitory computer-readable medium including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, the stored instructions causing one or more processors 306 to perform the techniques described herein.

[0080] The communication interface 330 may include, for example, an Ethernet interface. The communication interface 330 couples the NMS 300 to a network and / or the Internet, such as... Figure 1A This refers to any network and / or any local area network (LAN) in network 134 shown. Communication interface 330 includes a receiver 332 and a transmitter 334, through which the NMS 300 receives data from client devices 148, APs 142, switches 146, servers 110, 116, 122, 128, and / or forms networks such as... Figure 1A Any other network node, device, or entity within the network system 100 shown herein may receive or send data and information to any other entity. In some scenarios described herein, network system 100 may include “third-party” network devices that own and / or are associated with entities other than NMS 300, and NMS 300 may not receive, collect, or otherwise access network data from third-party network devices.

[0081] Data and information received by the NMS 300 may include, for example, data from client devices AP 148, AP 142, switch 146, or other network nodes (e.g., Figure 1BThe NMS 300 receives telemetry data, SLE-related data, or event data from one or more of the routers (187), which are used by the NMS 300 to remotely monitor the performance of wireless networks 106A to 106N and application sessions from client devices to cloud-based application servers. The NMS 300 can also send data via communication interface 330 to any of the network devices such as client device 148, AP 142, switch 146, or other network nodes within network 134, or administrator device 111, to remotely manage wireless networks 106A to 106N and a portion of the wired network.

[0082] Memory 312 includes one or more means configured to store programming modules and / or data associated with the operation of NMS 300. For example, memory 312 may include a computer-readable storage medium (such as a non-transitory computer-readable medium) including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause one or more processors 306 to perform the techniques described herein.

[0083] In this example, memory 312 includes API 320, SLE module 322, Virtual Network Assistant (VNA) / AI engine 350, and Radio Resource Management (RRM) engine 360. NMS 300 may also include components configured for use with wireless networks 106A to 106N and wired networks (including AP 142 / 200, switch 146, or other network devices, e.g., Figure 1B Any other programming module, software engine, and / or interface for remote monitoring and management of any of the routers (187).

[0084] SLE module 322 enables the setting and tracking of thresholds for SLE metrics used in each of networks 106A to 106N. SLE module 322 further analyzes SLE-related data collected by APs (such as any of AP 142) from UEs in each of wireless networks 106A to 106N. For example, APs 142A-1 to 142A-N collect SLE-related data from UEs 148A-1 to 148A-N currently connected to wireless network 106A. This data is sent to NMS 300, which is executed by SLE module 322 to determine one or more SLE metrics for each UE 148A-1 to 148A-N currently connected to wireless network 106A. This data, in addition to any network data collected by one or more APs 142A-1 to 142A-N in wireless network 106A, is sent to NMS 300 and stored in database 318 as, for example, network data 316.

[0085] RRM Engine 360 ​​monitors one or more metrics at each site 102A through 102N to understand and optimize the RF environment at each site. For example, RRM Engine 360 ​​can monitor coverage and capacity SLE metrics for wireless network 106 at site 102 to identify potential SLE coverage and / or capacity issues in wireless network 106 and adjust the radio settings of the access points at each site to address the identified issues. For example, RRM Engine 360 ​​can determine the channel and transmit power distribution across all APs 142 in each network 106A through 106N. For example, RRM Engine 360 ​​can monitor events, power, channels, bandwidth, and the number of clients connected to each AP. RRM Engine 360 ​​can also automatically change or update the configuration of one or more APs 142 at site 102 to improve coverage and capacity SLE metrics, thereby providing users with an improved wireless experience.

[0086] The VNA / AI engine 350 analyzes data received from network devices and its own data to identify when an unwanted anomalous state is encountered at one of the network devices. For example, the VNA / AI engine 350 can identify the root cause of any unwanted or anomalous state, such as any poor SLE metric indicating connectivity problems at one or more network devices. Additionally, the VNA / AI engine 350 can automatically invoke one or more corrective actions designed to address the identified root cause of one or more poor SLE metrics. Examples of corrective actions that can be automatically invoked by the VNA / AI engine 350 may include, but are not limited to, invoking the RRM engine 360 ​​to restart one or more APs, adjusting / modifying the transmit power of a specific radio in a specific AP, adding an SSID configuration to a specific AP, changing the channel on an AP or a group of APs, etc. Corrective actions may also include restarting switches and / or routers, invoking the download of new software to APs, switches, or routers, etc. These corrective actions are provided for illustrative purposes only, and this disclosure is not limited in this respect. If automatic corrective actions are unavailable or insufficient to address the root cause, the VNA / AI engine 350 can proactively provide a notification that includes recommended corrective actions to be taken by IT personnel (e.g., site or network administrators using administrator device 111) to resolve the network error.

[0087] In some examples, ML model 380 may include a supervised ML model trained using training data to identify synthetic test time windows and / or synthetic test observers. This training data includes pre-collected, labeled network data received from network devices (e.g., client devices, access points, switches, and / or other network nodes). The supervised ML model may include one of logistic regression, Naive Bayes, support vector machines (SVM), etc. In other examples, ML model 380 may include an unsupervised ML model. Although... Figure 3 Not shown, but in some examples, database 318 may store training data, and VNA / AI engine 350 or dedicated training module may be configured to train ML model 380 based on the training data to determine appropriate weights on one or more features of the training data.

[0088] According to one or more techniques disclosed herein, VNA / AI engine 350 includes network test transport module 352, which is configured to select software packages from downloadable software package 353 to provide to one or more devices (such as AP 142) so that AP 142 can simulate network instances and use the simulated network instances to perform one or more network performance tests to obtain data indicative of the performance of the network instances.

[0089] In one example, the network test transmission module 352 of the NMS 300's VNA / AI engine 350 identifies a trigger event associated with one of the APs 142 managed by the NMS 300. The trigger event can include any network event obtained from the analysis of any data collected and / or measured by any device within the network system 100. In one example, the network test transmission module 352 of the NMS 3030's VNA / AI engine 350 identifies a trigger event that may indicate a potential problem that may require additional testing. For example, the network test transmission module 352 may determine network anomalies that may require additional testing based on network data associated with one of the network devices of client device 148, AP 142 of wireless network 106, and wired network 175 and SD-WAN 177. For example, triggering events may include network device (such as AP 142 or a component of the network device) failure, performance degradation of AP 142 below a performance threshold (such as a Service Level Agreement (SLA) requirement), link failure of AP forwarding traffic, input from a user (such as an administrator of NMS 300), or messages from network devices (such as AP 142) managed by VNA / AI Engine 350.

[0090] Based at least in part on identified triggering events, the network test transmission module 352 selects a software package from a plurality of downloadable software packages 353. In some examples, each downloadable software package 353 includes a downloadable network performance monitoring module, which includes, for example, features for enabling network devices (such as...) Figure 1A and Figure 1B AP 142) includes instructions for performing network performance monitoring functions. For example, the network performance monitoring module may include instructions that cause the network device to perform network performance tests by simulating network instances to obtain data indicating the performance of the simulated network instances.

[0091] In some examples, the triggering event may indicate a specific fault or the type of data to be obtained. In such examples, the network test transmission module 352 selects a software package suitable for diagnosing the specific cause of a fault associated with AP 142. For example, if the triggering event indicates a specific type of fault in AP 142, the network test transmission module 352 selects a software package that enables AP 142 to perform network performance tests to obtain data for diagnosing the root cause of the specific fault or to perform remedial measures to remedy the specific fault. For example, if the triggering event indicates that AP 142's configuration may be incorrect, the network test transmission module 352 selects a software package that enables AP 142 to perform network performance tests by simulating a network instance with disputed configuration parameters to obtain data indicating the performance of a simulated network instance with corresponding configuration parameters.

[0092] As another example, when a triggering event indicates that network performance data is to be obtained, the network test transport module 352 selects a software package that enables AP 142 to perform a speed test by simulating a network instance that uses a second link to forward traffic in order to obtain data indicating the performance of the simulated network instance using the second link.

[0093] As another example, when a triggering event indicates that network performance data is to be obtained, the network test transmission module 352 selects a software package to enable AP 142 to perform a speed test via a simulated network instance. This network instance performs the speed test by exchanging network traffic between the simulated network instance and a server and measuring at least one of the upload and download speeds of the network traffic. In this example, the data indicating the performance of the simulated network instance may include the average, median, or maximum upload or download speed of the network traffic.

[0094] As another example, when a triggering event indicates that network performance data is to be obtained, the network test transport module 352 selects a software package to enable AP 142 to perform a ping test by simulating a network instance that exchanges ping messages with a server and measures performance metrics related to the ping messages. In this example, data indicating the performance of the simulated network instance may include one or more of the following: average round-trip time (RTT), maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of packets received, or number of packets transmitted.

[0095] As another example, in the event that triggers an event indicating that the Authentication, Authorization, and Accounting (AAA) server (such as an Off-Network Access Control (NAC) Remote Authentication Dial-In User Service (RADIUS) server) has failed, the Network Test Transport Module 352 selects a software package that enables AP 142 to perform network performance tests by simulating a network instance that attempts to initiate an AAA session with the AAA server to obtain data useful for troubleshooting potential configuration problems with accessing the AAA server, whether those configuration problems are due to a misconfigured AP 142, a problem with the AAA server, or a misconfigured network policy, etc.

[0096] In some examples, the triggering event may not indicate a specific fault or the type of data to be obtained. In such an example, the network test transport module 352 can select a software package that enables AP 142 to perform a default network performance test by simulating a network instance to obtain various types of data indicative of the performance of the simulated network instance. This data can be used for, for example, traffic engineering, fault analysis and root cause identification, fault remediation, network optimization, etc.

[0097] In some examples, each downloadable package 353 includes one or more of the following: network performance monitoring functions or network tests to be performed by the simulated network instance, configuration data for a communication channel to be established with the NMS 130 for transmitting network performance monitoring data, a schedule for performing network performance monitoring functions or network tests, a virtual local area network (VLAN) to be used during the performance of network performance monitoring functions or network tests, or one or more resource requirements to be applied during the performance of network performance monitoring functions or network tests.

[0098] In some examples, each downloadable package 353 specifies the data format for the network performance monitoring data. The data format can specify, for example, the name or identifier of the network performance monitoring function or network test to be performed by the simulated network instance, one or more categories of the network performance monitoring data, or one or more fields of latency, jitter, and packet loss values ​​to be specified by the network performance monitoring data. In some examples, the data format specifies JavaScript Object Notation (JSON), YAML, XML, CSV, or other data formats.

[0099] In some examples, each downloadable package 353 is specified for emulating one or more client devices (such as...). Figure 1A and Figure 1B The client device 148) provides instructions for accessing network 134 via AP 200. In some examples, each downloadable software package 353 specifies instructions for performing network tests on a simulated network instance to obtain data indicative of the network instance's performance. For example, network tests may include speed tests, and the instructions may cause the simulated network instance to perform speed tests by exchanging network traffic between the simulated network instance and the server and measuring at least one of the upload and download speeds of the network traffic. As another example, network tests may include ping messages, and the instructions may cause the simulated network instance to exchange ping messages with the server and measure one or more of the following: average RTT, maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of packets received, and number of packets transmitted.

[0100] In this example, the network test transmission module 135 stores the downloadable software package 353 and sends the selected downloadable software package 353 to the AP 142. In other examples, another device may store the selected software package, such as a server, software package repository, content delivery network, etc. In this case, the network test transmission module 135 can enable the AP 142 to obtain the selected downloadable software package 353 from, for example, the server or software package repository where the downloadable software package 353 is stored. For example, the network test transmission module 135 may send information to the AP 142 specifying the URL, link, or location where the AP 142 can obtain the selected downloadable software package 353.

[0101] Network test transmission module 135 sends a selected software package to AP 142 to enable AP 142 to simulate a network instance and obtain data indicative of the network instance's performance. NMS 130 receives the performance data indicative of the network instance from AP 142 via a communication channel established by the network device based on the selected software package. In some examples, the performance data indicative of the network instance includes values ​​for one or more of latency, jitter, packet loss, CPU utilization, memory utilization, and version identifier obtained during the simulation of the network instance.

[0102] In some examples, the VNA / AI engine 350 performs fault diagnosis, troubleshooting, or automatic fault remediation based at least in part on received data indicating the performance of a network instance. For example, the VNA / AI engine 350 can use the data indicating the performance of a network instance to determine the root cause of a failure in a network device, such as AP 142 from which it receives data. As another example, the VNA / AI engine 350 can use the data indicating the performance of a network instance to determine the root cause of a failure in another network device with which the VNA / AI engine 350 cannot communicate, such as a peer device of AP 142 from which it receives data. Based on the determined root cause of the failure, the VNA / AI engine 350 can automatically perform remediation measures to remedy the network device failure. In some examples, the VNA / AI engine 350 performs traffic engineering, network path selection, or network traffic load balancing based at least in part on received data indicating the performance of a network instance.

[0103] Figure 4 An exemplary user equipment (UE) device 400 according to one or more technologies of this disclosure is shown. Figure 4 The exemplary UE device 400 shown can be used to implement as described herein. Figure 1A Any of the UE 148 shown and described. UE device 400 may include any type of wireless client device, and this disclosure is not limited in this respect. For example, UE 400 may include mobile devices such as smartphones, tablet computers or laptop computers, personal digital assistants (PDAs), wireless terminals, smartwatches, smart rings, or any other type of mobile or wearable device. In some examples, UE 400 may also include wired client-side devices, such as IoT devices, such as printers, security sensors or devices, environmental sensors, or any other device connected to a wired network and configured to communicate over one or more wireless networks.

[0104] UE device 400 includes a wired interface 430, wireless interfaces 420A–420C, one or more processors 406, memory 412, and user interface 410. The various components are coupled together via bus 414, through which they can exchange data and information. The wired interface 430 represents a physical network interface and includes a receiver 432 and a transmitter 434. If needed, the wired interface 430 can be used via cable (such as…). Figure 1A One of the Ethernet cables 144) directly or indirectly couples the UE 400 to a wired network device (such as...) within the wired network. Figure 1A (One of the switches 146).

[0105] The first wireless interface 420A, the second wireless interface 420B, and the third wireless interface (cellular interface, 420C) each include receivers 422A, 422B, and 422C, respectively. Each receiver includes a receiving antenna, and the UE 400 can receive signals from a wireless communication device (such as...) via the receiving antenna. Figure 1A AP 142, Figure 2 The AP 200, other UEs 148, or other devices configured for wireless communication receive wireless signals. The first wireless interface 420A, the second wireless interface 420B, and the third wireless interface 420C also include transmitters 424A, 424B, and 424C, respectively. Each transmitter includes a transmitting antenna, through which the UE 400 can transmit signals to wireless communication devices (such as AP 200, other UEs 148, or other devices configured for wireless communication). Figure 1A AP 142, Figure 2 The AP 200, other UEs 148, and / or other devices configured for wireless communication transmit wireless signals. In some examples, the first wireless interface 420A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz), and the second wireless interface 420B may include a Bluetooth interface and / or a Bluetooth Low Energy interface. The third wireless interface 420C may include, for example, a cellular interface through which the UE device 400 can connect to a cellular network.

[0106] Processor 406 executes software instructions (such as software instructions for defining software or computer programs) stored in a computer-readable storage medium (such as memory 412), such as a non-transitory computer-readable medium including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, the storage of which causes one or more processors 406 to execute instructions of the techniques described herein.

[0107] Memory 412 includes one or more means configured to store programming modules and / or data associated with the operation of UE 400. For example, memory 412 may include a computer-readable storage medium (such as a non-transitory computer-readable medium) including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, the storage of which enables one or more processors 406 to execute instructions of the techniques described herein.

[0108] In this example, memory 412 includes operating system 440, application 442, communication module 444, configuration settings 450, and data memory 454. Communication module 444 includes program code that, when executed by processor 406, enables UE 400 to communicate using any of wired interface 430, wireless interfaces 420A and 420B, and / or cellular interface 420C. Configuration settings 450 includes any device settings for UE 400 settings for each of wireless interfaces 420A and 420B and / or cellular interface 420C.

[0109] Data storage 454 may include, for example, a status / error log, which includes a list of events specific to UE 400. Depending on the logging level based on instructions from NMS 130, events may include logs of both normal and error events. Data storage 454 may store any data used and / or generated by UE 400 (such as data used to calculate one or more SLE metrics or identify relevant behavioral data), which is collected by UE 400 and transmitted directly to NMS 130 or to either AP 142 in wireless network 106 for further transmission to NMS 130.

[0110] As described herein, UE 400 can measure and report network data from data storage 454 to NMS 130. Network data may include event data, telemetry data, and / or other SLE-related data. Network data may include various parameters indicating the performance and / or status of the wireless network. NMS 130 can determine one or more SLE metrics based on the SLE-related data received from the UE or client device in the wireless network and store the SLE metrics as network data 137. Figure 1A ).

[0111] Optionally, the UE device 400 may include an NMS agent 456. The NMS agent 456 is a software agent of the NMS 130 installed on the UE 400. In some examples, the NMS agent 456 may be implemented as a software application running on the UE 400. The NMS agent 456 collects information from the UE 400, including detailed client device attributes, and insights into the UE 400's roaming behavior. This information provides insights into the client roaming algorithm, as roaming is a decision made by the client device. In some examples, the NMS agent 456 may display client device attributes on the UE 400. The NMS agent 456 sends the client device attributes to the NMS 130 via an AP device connected to the UE 400. The NMS agent 456 may be integrated into a custom application or as part of a location application. The NMS agent 456 may be configured to identify the device connection type (e.g., cellular or Wi-Fi) along with the corresponding signal strength. For example, the NMS agent 456 identifies the access point connection and its corresponding signal strength. NMS agent 456 can store information specifying the APs identified by UE 400 and their corresponding signal strengths. NMS agent 456 or other components of UE 400 also collect information about which APs UE 400 is connected to, and which APs UE 400 is not connected to. UE 400's NMS agent 456 sends this information to NMS 130 via the APs it is connected to. In this way, UE 400 sends not only information about the APs it is connected to, but also information about other APs identified by UE 400 but not connected to, and their signal strengths. The AP then forwards this information to NMS, including information about other APs identified by UE 400 besides itself. This additional level of detail allows NMS 130 and ultimately network administrators to better determine the Wi-Fi experience directly from the client device's perspective.

[0112] In some examples, NMS Agent 456 further enriches the client device data utilized in the service level. For instance, NMS Agent 456 can go beyond basic fingerprinting to provide supplementary details of attributes such as device type, manufacturer, and different versions of the operating system. In detailed client attributes, NMS 130 can display radio hardware and firmware information of the UE 400 received from NMS Agent 456. The more details NMS Agent 456 can extract, the better the VNA / AI engine performs in advanced device classification. NMS 130's VNA / AI engine continuously learns and becomes more accurate in its ability to distinguish between device-specific issues and broader device issues, such as specifically identifying which operating system version is affecting some clients.

[0113] In some examples, NMS agent 456 may display a prompt on user interface 410, instructing the end user of UE 400 to enable location permission before NMS agent 456 can report device location, client information, and network connectivity data to the NMS. NMS agent 456 will then begin reporting connectivity data along with location data to the NMS. In this way, the end user of the client device can control whether NMS agent 456 is able to report client device information to the NMS.

[0114] According to the technology described in this disclosure, the UE device 400 obtains a software package including a network performance monitoring module 458, enabling the UE device 400 to perform network performance testing by simulating a network instance 446 to obtain data indicative of the network instance's performance (network performance monitoring data 448), without requiring the UE device 400 to update its firmware or operating system, or to reboot or restart. Therefore, the UE device 400 can simulate network instance 446 to obtain network performance monitoring data 448 without adversely affecting the forwarding of network traffic by the UE device 400. Furthermore, such technology avoids the additional management overhead required to upgrade the firmware or operating system of a conventional AP to enable such a conventional AP to perform additional or new network performance monitoring functions not currently programmed into the conventional AP.

[0115] For example, according to the techniques described herein, processor 406 receives data from wired interface 430 or wireless interface 420. Figure 1A and Figure 1B The NMS 130 obtains a software package from the VNA 133. The software package includes a downloadable network performance monitoring module 458, which includes, for example, instructions for causing the processor 406 to perform network performance monitoring functions. For example, the software package may include instructions for causing the processor 406 to perform network performance tests by simulating a network instance 446. The processor 406 stores the software package. In some examples, the processor 406 optionally stores the software package in volatile memory of memory 412, such that the software package is not retained when the UE device 400 is powered off, restarted, or restarted. In some examples, the processor 406 optionally stores the software package in non-volatile memory of memory 412, such that the software package is retained after the UE device 400 is powered off, restarted, or restarted.

[0116] Processor 406 simulates network instance 446 based on instructions specified by the software package to obtain data 448 indicative of the network instance's performance. Processor 406 communicates with, at least in part, via wired interface 430 or wireless interface 420 based on configuration information of the communication channel specified by the software package. Figure 1A and Figure 1BA communication channel is established between the VNA 133 and the processor 406. The processor 406 then sends network performance monitoring data 448 to the VNA 133 via the established communication channel.

[0117] Figure 5 This is a block diagram of an exemplary network node 500 according to one or more technologies of this disclosure. In one or more examples, the network node 500 is implemented to be attached to Figure 1A Network devices or servers 134, such as switch 146, AAA server 110 or other NAC server or system, DHCP server 116, DNS server 122, network server 128, etc., or supporting Figure 1B Another network device, such as router 187, of one or more of the following: wireless network 106, wired LAN 175, SD-WAN 177, or data center 179.

[0118] In this example, network node 500 includes a wired interface 502 (e.g., an Ethernet interface), a processor 506, input / output 508 (e.g., a display, buttons, keyboard, keypad, touchscreen, mouse, etc.), and memory 512 coupled together via a bus 514. Various components can exchange data and information through this bus. The wired interface 502 couples network node 500 to a network, such as an enterprise network. Although only one interface is shown by way of example, network nodes can and typically do have multiple communication interfaces and / or multiple communication interface ports. The wired interface 502 includes a receiver 520 and a transmitter 522.

[0119] Memory 512 stores executable software applications 532, operating system 540, and data / information 530. Data 530 may include system logs and / or error logs storing event data (including behavioral data) of network node 500. In the example where network node 500 includes a "third-party" network device, the same entity does not own or access both the AP or wired client-side device and network node 500. Thus, in the example where network node 500 is a third-party network device, NMS 130 does not receive, collect, or otherwise access network data from network node 500.

[0120] In an example where network node 500 includes a server, network node 500 can receive data and information via receiver 520, such as operation-related information (e.g., registration requests, AAA services, DHCP requests, Simple Notification Service (SNS) lookups, and web page requests), and send data and information via transmitter 522, such as configuration information, authentication information, web page data, etc.

[0121] In an example where network node 500 includes a wired network device, network node 500 can be connected to one or more access points (APs) or other wired client-side devices, such as IoT devices, via wired interface 502. For example, network node 500 may include multiple wired interfaces 502, and / or wired interfaces 502 may include multiple physical ports for connection to multiple APs or other wired client-side devices within the site via corresponding Ethernet cables. In some examples, each of the APs or other wired client-side devices connected to network node 500 can access the wired network via wired interface 502 of network node 500. In some examples, one or more APs or other wired client-side devices connected to network node 500 can each draw power from network node 500 via a corresponding Ethernet cable and a Power over Ethernet (PoE) port of wired interface 502.

[0122] In an example where network node 500 includes a session-based router employing a stateful, session-based routing scheme, network node 500 can be configured to perform path selection and traffic engineering independently. Using session-based routing allows network node 500 to avoid using a centralized controller (such as an SDN controller) to perform path selection and traffic engineering, and to avoid using tunnels. In some examples, network node 500 can implement session-based routing as a Security Vector Router (SVR) provided by Juniper Networks, Inc. In an example where network node 500 includes a session-based router operating as a network gateway for sites used in an enterprise network (e.g., Figure 1B In the case of a router 187A, network node 500 can access the underlying physical WAN (e.g., Figure 1B The SD-WAN 177) and one or more other session-based routers (e.g., those operating as network gateways for other sites on the enterprise network) Figure 1B Router 187B) establishes multiple peer paths (e.g., Figure 1B (Logical path 189). As a session-based router, network node 500 can collect peer path level data and report the peer path data to NMS 130.

[0123] In an example where network node 500 includes a packet-based router, network node 500 can employ either packet-based or flow-based routing schemes to forward packets according to network paths defined, for example, by a centralized controller that performs path selection and traffic engineering. In another example, network node 500 may include a packet-based router that operates as a network gateway for a site within an enterprise network (e.g., Figure 1B In the case of a router 187A, network node 500 can access the underlying physical WAN (e.g., Figure 1B The SD-WAN 177) and one or more other packet-based routers (e.g., ) operate as network gateways for other sites on the enterprise network. Figure 1B Router 187B) establishes multiple channels (e.g., Figure 1B (Logical path 189). As a packet-based router, network node 500 can collect data at the channel level, and the channel data can be obtained by NMS 130 via API or open configuration protocol, or the channel data can be reported to NMS 130 by NMS agent 544 or other modules running on network node 500.

[0124] Network data collected and reported by network node 500 may include periodically reported data and event-driven data. Network node 500 is configured to collect logical path statistics via bidirectional forwarding detection (BFD) probes and by extracting data from messages and / or counters at the logical path level (e.g., peer-to-peer paths or channels). In some examples, network node 500 is configured to collect statistics and / or sample other data according to a first periodic interval (e.g., every 3 seconds, every 5 seconds, etc.). Network node 500 may store the collected and sampled data as path data in, for example, a buffer.

[0125] In some examples, network node 500 may optionally include NMS agent 544. NMS agent 544 may periodically create packets of path data at a second periodic interval (e.g., every 3 minutes). The collected and sampled data periodically reported in the statistics packets may be referred to herein as “OC statistics.” In some examples, the statistics packets may also include details about clients connected to network node 500 and associated client sessions. NMS agent 544 may then report the statistics packets to NMS 130 in the cloud. In other examples, NMS 130 may request, retrieve from, or otherwise receive the statistics packets from network node 500 via an API, an open configuration protocol, or another communication protocol. The statistics packets created by NMS agent 544 or another module of network node 500 may include a header identifying network node 500 and statistics and data samples for each logical path from network node 500. In other examples, NMS agent 544 reports event data to NMS 130 in the cloud in response to the occurrence of certain events at network node 500. Event-driven data can be referred to as "oc events" in this article.

[0126] According to the technology described in this disclosure, network node 500 obtains a software package that enables it to perform network performance tests by simulating network instance 534 to obtain data 536 indicative of the performance of network instance 534, without requiring network node 500 to update its firmware or operating system, or to reboot or restart. Therefore, network node 500 can simulate network instance 534 to obtain data 536 without adversely affecting the forwarding of network traffic by network node 500. Furthermore, such technology avoids the additional management overhead required to upgrade the firmware or operating system of a conventional AP to enable it to perform additional or new network performance monitoring functions not currently programmed into the conventional AP.

[0127] For example, according to the techniques described herein, processor 506 is connected via wired interface 502 from... Figure 1A and Figure 1B The software package is obtained from the VNA 133 of the NMS130. The software package includes a downloadable network performance monitoring module 548, which includes, for example, instructions for causing the processor 506 to perform network performance monitoring functions. For example, the software package may include instructions for causing the processor 506 to perform network performance tests by simulating a network instance 534. The processor 506 stores the software package. In some examples, the processor 506 optionally stores the software package in volatile memory of memory 512, such that the software package is not retained when the network node 500 is powered off, restarted, or rebooted. In some examples, the processor 506 optionally stores the software package in non-volatile memory of memory 512, such that the software package is retained after the network node 500 is powered off, restarted, or rebooted.

[0128] Processor 506 simulates network instance 534 based on instructions specified by the software package to obtain data indicative of the network instance's performance (network performance monitoring data 536). Processor 506 communicates with, at least in part, via wired interface 502 based on configuration information of the communication channel specified by the software package. Figure 1A and Figure 1B The VNA 133 establishes a communication channel. The processor 506 sends data 536, indicating the performance of the network instance, to the VNA 133 via the established communication channel.

[0129] Figure 6 This is a flowchart illustrating exemplary operations of providing a downloadable software package to perform testing functions or other functions to monitor network performance according to one or more technologies disclosed herein. Figure 6 It is about Figure 1A and Figure 1B The NMS 130 network test transmission module 135 and Figure 3 The NMS 300 network test transmission module 352 is described in this way.

[0130] According to one or more techniques of this disclosure, the NMS 130 is configured to perform scheduling and / or orchestration of tests performed by devices at a site. In this example, the VNA 133 of the NMS 130 includes a network test transport module 135 configured to provide software packages to one or more devices (such as AP 142) to enable AP 142 to simulate network instances and use the simulated network instances to perform one or more network performance tests to obtain data indicative of the performance of the network instances. The NMS 130 can receive the data indicative of the performance of the network instances from AP 142 for fault detection, troubleshooting and root cause analysis, and to perform remedial measures on network system 100, as described in more detail below.

[0131] like Figure 6 As depicted in the example, the network test transport module 135 of the VNA 133 of the NMS 130 identifies a triggering event (602) associated with one of the APs 142 managed by the NMS 130 (such as AP 142A-1). Based at least in part on the identified triggering event, the network test transport module 135 selects a software package from a plurality of downloadable software packages 136 (604). Each software package includes a downloadable network performance monitoring module, which includes, for example, instructions for causing the AP 142 to perform network performance monitoring functions. For example, the software package may include instructions for causing the AP 142 to perform network performance testing by simulating a network instance. The network test transport module 135 sends the selected software package to the AP 142A-1 to cause the AP 142A-1 to simulate a network instance to obtain data indicating the performance of the network instance (606). Alternatively or separately, the network test transport module 135 may send information to the AP 142A-1 specifying the location where the AP 142A-1 can download or obtain the selected software package.

[0132] AP 142A-1 acquires a software package (608) containing instructions for simulating network instances and stores the software package. In some examples, AP 142A-1 optionally stores the software package in volatile memory such that the software package is not retained when AP 142A-1 is powered off, restarted, or rebooted (610). In some examples, AP 142A-1 optionally stores the software package in non-volatile memory such that the software package is retained after AP 142A-1 is powered off, restarted, or rebooted.

[0133] AP 142A-1 simulates a network instance based on instructions specified by the software package to obtain data indicating the performance of the network instance (612). AP 142A-1 establishes a communication channel with VNA 133 based at least in part on configuration information for the communication channel specified by the software package. AP 142A-1 transmits the data indicating the performance of the network instance to NMS 130 via the established communication channel (614).

[0134] NMS 130 receives data indicating the performance of a network instance from AP 142A-1 via a communication channel established by the network device based on a selected software package (616). In some examples, VNA 133 of NMS 130 performs fault diagnosis, troubleshooting, or automatic fault recovery based at least in part on the received data indicating the performance of the network instance (618). In some examples, VNA 133 performs traffic engineering, network path selection, or network traffic load balancing based at least in part on the received data indicating the performance of the network instance.

[0135] Although the technology of this disclosure is described in this example as being performed by NMS 130, the technology described herein can be performed by any other computing device, system, and / or server, and this disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the technology of this disclosure may reside in a dedicated server, or be included in any other server besides NMS 130, or may be distributed throughout network 100, and may or may not be part of NMS 130.

[0136] The following examples illustrate one or more aspects of this disclosure.

[0137] Example A1. A network apparatus, comprising: volatile memory; and processing circuitry in communication with a storage medium, the processing circuitry being configured to: obtain a software package including instructions for simulating a network instance; store the software package in the volatile memory; simulate the network instance based on the instructions specified by the software package to obtain data indicative of the performance of the network instance; and transmit the data indicative of the performance of the network instance via a communication channel established with a network management system (NMS) based on the software package.

[0138] Example A2. A network device of Example A1, wherein processing circuitry is further configured to execute firmware configured to perform one or more network functions of the network device, the firmware being stored in non-volatile memory of the network device; and wherein the processing circuitry is configured to obtain a software package and, based on instructions specified by the software package, simulate a network instance to obtain data indicative of the performance of the network instance, without updating the firmware stored in the non-volatile memory.

[0139] Example A3. A network apparatus of any one of Examples A1 to A2, wherein the processing circuitry is further configured to forward network traffic; and wherein the processing circuitry is configured to obtain a software package and, based on instructions specified by the software package, simulate a network instance to obtain data indicative of the performance of the network instance, without interrupting the forwarding of network traffic.

[0140] Example A4. A network device of any of Examples A1 to A3, wherein, in order to simulate a network instance to obtain data indicative of the performance of the network instance, the processing circuitry is configured to simulate the operation of one or more client devices accessing the network via the network device.

[0141] Example A5. A network apparatus of any of Examples A1 to A4, wherein, in order to simulate a network instance to obtain data indicative of the performance of the network instance, the processing circuitry is configured to perform a network test on the simulated network instance to obtain data indicative of the performance of the network instance.

[0142] Example A6. A network apparatus of any one of Examples A1 to A5, wherein, in order to simulate a network instance to obtain data indicative of the performance of the network instance, the processing circuitry is configured to perform at least one of the following: simulating a speed test by exchanging network traffic between the network instance and a server and measuring at least one of the upload speed and download speed of the network traffic; or simulating a ping test by exchanging ping messages between the network instance and a server and measuring one or more of the following: average round-trip time (RTT), maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of received packets, and number of transmitted packets.

[0143] Example A7. A network apparatus of any of Examples A1 to A4, wherein, in order to simulate a network instance to obtain data indicative of the performance of the network instance, the processing circuitry is configured to cause the network instance to initiate an AAA session with an authentication, authorization, and accounting (AAA) server.

[0144] Example A8. A network device of any of Examples A1 to A7, wherein the processing circuitry is configured to obtain software packages from the NMS.

[0145] Example A9. A network apparatus of any one of Examples A1 to A8, wherein the software package includes one or more of the following: a network performance monitoring function or network test to be performed by a simulated network instance; configuration data for a communication channel; a data format for data indicating the performance of the network instance; or a schedule for performing the network performance monitoring function or network test; a virtual local area network (VLAN) to be used during the performance of the network performance monitoring function or network test; or one or more resource requirements to be applied during the performance of the network performance monitoring function or network test.

[0146] Example A10. A network apparatus of Example A9, wherein the data format specifies one or more of the following: a name or identifier of a network performance monitoring function or network test to be performed by a simulated network instance; one or more categories of data indicating the performance of the network instance; or one or more values ​​of latency, jitter, and packet loss specified by the data indicating the performance of the network instance.

[0147] Example A11. A network device of any one of Examples A1 to A10, wherein the processing circuitry is configured to simulate a network instance at least in part based on a trigger command received from a user to obtain data indicative of the performance of the network instance.

[0148] Example A12. A network device of any one of Examples A1 to A11, wherein the data indicating the performance of a network instance includes one or more of latency, jitter, packet loss, CPU utilization, memory utilization, or version identifier.

[0149] Example A13. A method performed by a network device as shown in Examples A1 to A12.

[0150] Example A14. A non-transitory computer-readable medium including instructions that, when executed, are configured to cause processing circuitry to perform any of the operations of Examples A1 to A12.

[0151] Example B1. A network management system (NMS) includes: processing circuitry in communication with a storage medium, the processing circuitry being configured to: identify a triggering event associated with a network device among a plurality of network devices managed by the network management system; select a software package from a plurality of software packages based at least in part on the identified triggering event, the selected software package including instructions for simulating a network instance; and send the selected software package to the network device to cause the network device to simulate the network instance, thereby obtaining data indicative of the performance of the network instance.

[0152] Example B2. The NMS of Example B1, wherein the processing circuitry is further configured to receive data indicative of the performance of a network instance from the network device via a communication channel established by the network device based on a selected software package.

[0153] Example B3. The NMS of any one of Examples B1 to B2, wherein the triggering event includes at least one of the following: failure of at least one of the plurality of network devices; performance degradation of at least one of the plurality of network devices; input from the user; or a message from the network device.

[0154] Example B4. An NMS of any of Examples B1 to B3, wherein the processing circuitry is further configured to determine the root cause of a failure of at least one of the plurality of network devices based at least in part on data indicating the performance of the network instance.

[0155] Example B5. The NMS of any one of Examples B2 to B3, wherein the processing circuitry is further configured to perform remedial measures to remedy the failure of at least one of the plurality of network devices based at least in part on data indicating the performance of the network instance.

[0156] Example B6. An NMS of any of Examples B1 through B5, wherein the instructions for simulating a network instance include one or more of the following: instructions for simulating the operation of one or more client devices accessing the network via a network device; instructions for performing network tests on the simulated network instance; instructions for simulating speed tests by exchanging network traffic between the network instance and the server and measuring at least one of the upload speed and download speed of the network traffic; instructions for simulating ping tests by exchanging ping messages between the network instance and the server and measuring one or more of the average round-trip time (RTT), maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of packets received, or number of packets transmitted; or instructions for causing the network instance to initiate an AAA session with an Authentication, Authorization, and Accounting (AAA) server.

[0157] Example B7. A network device of any one of Examples B1 to B6, wherein the software package includes one or more of the following: a network performance monitoring function or network test to be performed by a simulated network instance; configuration data for a communication channel to be established by the network device for exchanging data indicative of the performance of the network instance; a data format for the data indicative of the performance of the network instance; or a schedule for performing the network performance monitoring function or network test; a Virtual Local Area Network (VLAN) to be used during the performance of the network performance monitoring function or network test; or one or more resource requirements to be applied during the performance of the network performance monitoring function or network test.

[0158] Example B8. NMS of Example B7, wherein the data format specifies one or more of the following: the name or identifier of the network performance monitoring function or network test to be performed by the simulated network instance; one or more categories of data indicating the performance of the network instance; or one or more values ​​of latency, jitter, and packet loss specified by the data indicating the performance of the network instance.

[0159] Example B9. NMS of any of Examples B1 to B8, wherein the data indicating the performance of the network instance includes one or more of latency, jitter, packet loss, CPU utilization, memory utilization, and version identifier.

[0160] Example B10. A method performed by a network management system as described in Examples B1 through B9.

[0161] Example B11. A non-transitory computer-readable medium including instructions that, when executed, are configured to cause processing circuitry to perform any of the operations of Examples B1 to B9.

[0162] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Individual features described as modules, units, or components can be implemented together in an integrated logic device or individually as discrete but interoperable logic devices or other hardware devices. In some cases, individual features of an electronic circuit can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0163] If implemented in hardware, this disclosure may relate to devices such as processors or integrated circuit devices such as integrated circuit chips or chipsets. Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store such instructions for processor execution.

[0164] Computer-readable media can form part of a computer program product that may include encapsulating materials. Computer-readable media may include computer data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0165] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" can indicate that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., stored in RAM or cache).

[0166] The code or instructions can be software and / or firmware executed by processing circuitry including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functions described in this disclosure can be set within software modules or hardware modules.

Claims

1. A network device comprising: a volatile memory; and processing circuitry in communication with the storage medium, the processing circuitry configured to: obtain a software package comprising instructions for simulating a network instance; store the software package in the volatile memory; simulate the network instance based on the instructions specified by the software package to obtain data indicative of a performance of the network instance; and send, via a communication channel established with a network management system (NMS) based on the software package, the data indicative of the performance of the network instance.

2. The network device of claim 1, the processing circuitry further configured to execute a firmware configured to perform one or more network functions of the network device, the firmware stored in a non-volatile memory of the network device; wherein and wherein the processing circuitry is configured to obtain the software package and simulate the network instance based on the instructions specified by the software package to obtain the data indicative of the performance of the network instance without updating the firmware stored in the non-volatile memory.

3. The network device of claim 1, the processing circuitry further configured to forward network traffic; and wherein wherein the processing circuitry is configured to obtain the software package and simulate the network instance based on the instructions specified by the software package to obtain the data indicative of the performance of the network instance without interrupting the forwarding of the network traffic. To simulate the network instance to obtain the data indicative of the performance of the network instance, the processing circuitry is configured to simulate an operation of one or more client devices accessing a network via the network device.

4. The network device of claim 1, wherein, To simulate the network instance to obtain the data indicative of the performance of the network instance, the processing circuitry is configured to perform a network test on the simulated network instance to obtain the data indicative of the performance of the network instance.

5. The network device of claim 1, wherein, To simulate the network instance to obtain the data indicative of the performance of the network instance, the processing circuitry is configured to perform one or more of:

6. The network device of claim 1, wherein, simulate a speed test by exchanging network traffic between the network instance and a server and measuring at least one of an upload speed and a download speed of the network traffic; and simulate a ping test by exchanging ping messages between the network instance and the server and measuring one or more of an average round trip time (RTT), a maximum RTT, a minimum RTT, a ping count, a ping interval, a ping request size, a ping field, a packet loss rate, a number of packets received, and a number of packets transmitted. To simulate the network instance to obtain the data indicative of the performance of the network instance, the processing circuitry is configured to cause the network instance to initiate an authentication, authorization, and accounting (AAA) session with an AAA server.

7. The network device of claim 1, wherein, The processing circuitry is configured to obtain the software package from the NMS.

8. The network device of claim 1, wherein, The software package comprises one or more of:

9. The network device of any of claims 1 to 8, wherein, a network performance monitoring function or a network test to be performed by the simulated network instance; configuration data for the communication channel; ​ a data format of data indicative of the performance of the network instance; a schedule of the network performance monitoring function or the network test to be performed; a virtual local area network, VLAN, to be used during performance of the network performance monitoring function or the network test; and one or more resource requirements to be applied during performance of the network performance monitoring function or the network test.

10. The network device of claim 9, wherein, the data format specifies one or more of: a name or identifier of the network performance monitoring function or the network test to be performed by the simulated network instance; one or more categories of data indicative of the performance of the network instance; and one or more of a value of latency, jitter, packet loss specified by the data indicative of the performance of the network instance.

11. The network device of any of claims 1 to 8, wherein, the processing circuitry is configured to simulate the network instance to obtain the data indicative of the performance of the network instance based at least in part on a trigger command received from a user.

12. The network device of any of claims 1 to 8, wherein, the data indicative of the performance of the network instance comprises one or more of latency, jitter, packet loss, CPU usage, memory usage, and a version identifier.

13. A computer networking method comprising: obtaining, by a network device, a software package comprising instructions for simulating a network instance; storing, by the network device, the software package in a volatile memory of the network device; simulating, by the network device and based on the instructions specified by the software package, the network instance to obtain data indicative of a performance of the network instance; and sending, by the network device and via a communication channel established with a network management system, NMS, based on the software package, the data indicative of the performance of the network instance.

14. The computer networking method of claim 13, further comprising: executing, by the network device, firmware configured to perform one or more network functions of the network device, the firmware stored in a non-volatile memory of the network device; wherein obtaining the software package and simulating, based on the instructions specified by the software package, the network instance to obtain the data indicative of the performance of the network instance is performed without updating the firmware stored in the non-volatile memory.

15. The computer networking method of claim 13, wherein the method further comprising forwarding, by the network device, network traffic; and wherein obtaining the software package and simulating, based on the instructions specified by the software package, the network instance to obtain the data indicative of the performance of the network instance is performed without interrupting the forwarding of the network traffic.

16. The computer networking method of claim 13, wherein, simulating the network instance to obtain the data indicative of the performance of the network instance comprises simulating operation of one or more client devices accessing a network via the network device.

17. The computer networking method of claim 13, wherein, simulating the network instance to obtain the data indicative of the performance of the network instance comprises performing a network test on the simulated network instance to obtain the data indicative of the performance of the network instance.

18. The computer networking method of any of claims 13 to 17, wherein, simulating the network instance to obtain the data indicative of the performance of the network instance comprises at least one of: simulating a speed test by exchanging network traffic between the network instance and a server and measuring at least one of upload speed and download speed of the network traffic; and simulating a ping test by exchanging ping messages between the network instance and the server and measuring one or more of average round trip time (RTT), maximum RTT, minimum RTT, ping count, ping interval, ping request size, ping domain, packet loss rate, number of packets received, and number of packets transmitted.

19. The computer networking method of any of claims 13 to 17, wherein, simulating the network instance to obtain data indicative of the performance of the network instance includes causing the network instance to initiate an authentication, authorization, and accounting (AAA) session with an AAA server.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to become configured as a network apparatus according to any of claims 1 to 12, or to become configured to implement a method according to any of claims 13 to 19.

Images