Synthetic test for application connectivity

Abstract

This disclosure relates to synthetic testing for application connectivity. Techniques are disclosed for performing downloadable, automated testing for network connectivity by a network device. A network management system (NMS) sends a test for network connectivity to a network device and causes the network device to perform the test. The network device performs the test for network connectivity and sends corresponding data generated from performing the test to the NMS. The NMS analyzes the data from the network device. If the NMS determines that a particular network device failed the test, the NMS determines a root cause of the failure, selects an action to fix the root cause, and performs the selected action. Such techniques described herein enable a network device to perform network connectivity testing on demand, without pre-configuring the network device to perform such network connectivity testing.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 779,875, filed March 28, 2025, and U.S. Application No. 19 / 573,844, filed March 20, 2026, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Commercial locations or sites, such as offices, hospitals, airports, stadiums, or retail stores, typically install complex wireless network systems throughout the premises, including networks of wireless access points (APs) to provide wireless network services to one or more wireless client devices (or simply "clients"). An AP is a physical electronic device that enables other devices to wirelessly connect to a wired network using various wireless networking protocols and technologies, such as IEEE 802.11-compliant standards (i.e., "WiFi"), Bluetooth / Bluetooth Low Energy (BLE), mesh networking protocols (such as ZigBee), or one or more other wireless networking technologies. Many different types of wireless client devices, such as laptops, smartphones, tablets, wearables, appliances, and Internet of Things (IoT) devices, incorporate wireless communication technologies and can be configured to connect to a compatible wireless access point when within range of the access point to access a wired network. Attached Figure Description

[0003] Figure 1A This is a block diagram of an example network system according to one or more technologies of this disclosure, the network system including a network management system configured to perform automated scheduling and orchestration synthesis tests for network connectivity within a network site.

[0004] Figure 1B It is a diagram. Figure 1A A block diagram providing further examples and details of the network system.

[0005] Figure 2 This is a block diagram of an example access point device based on one or more technologies according to this disclosure.

[0006] Figure 3A This is a block diagram of an example network management system based on one or more technologies disclosed herein.

[0007] Figure 3B This is a block diagram of an example synthetic test module based on one or more techniques of this disclosure.

[0008] Figure 4 This is a block diagram of an example user equipment device according to one or more technologies of this disclosure.

[0009] Figure 5This is a block diagram of an example network node (such as a router or switch) according to one or more technologies of this disclosure.

[0010] Figure 6 The illustration shows an example arrangement of synthetic tests according to one or more techniques of this disclosure.

[0011] Figure 7 The illustration shows an example of obtaining test result data from a specific network device that has performed a synthetic test, according to one or more techniques of this disclosure.

[0012] Figure 8 This is a flowchart illustrating an example operation of a network management system configured to perform automated scheduling and orchestration of network connectivity tests for network sites, according to one or more technologies disclosed herein. Detailed Implementation

[0013] In general, this disclosure describes one or more techniques for performing downloadable, automated tests for network connectivity and application connectivity by network devices, such as access points (APs). For example, the network system may include a network management system (NMS) configured to manage the wireless networks of APs at one or more sites. The NMS instructs the network devices to perform tests for network connectivity. Network connectivity tests may include, for example, synthetic tests of network connectivity performed by a Domain Name Server (DNS), a Dynamic Host Configuration Protocol (DHCP) server, an Address Resolution Protocol (ARP) request, cURL data transfer, network reachability to another network device, or applications performed by the network devices. The NMS obtains data from the network devices that have performed the network connectivity tests. The NMS applies a machine learning system to the data to determine failures in the network connectivity tests performed by the network devices. Based on the type of network connectivity test, the NMS selects actions to address the root causes of the network connectivity test failures. The NMS performs the selected actions.

[0014] Without using the technology disclosed herein, there is a technical problem: customers may find it difficult to diagnose the root cause of network connectivity problems. For example, end users experiencing network connectivity problems may not be able to easily determine whether the root cause is related to the DHCP server, local DNS server, ARP entries, or some other cause. Such network connectivity problems may present similar symptoms, making diagnosis difficult and cumbersome. In systems not using the technology disclosed herein, determining whether a network problem is caused by the DHCP server, local DNS server, ARP entries, or some other cause can be a time-consuming, manual process of checking and verifying the correct configuration and operation of each component.

[0015] According to the technology disclosed herein, an NMS network management system performs network connectivity tests on one or more access points (APs) on one or more VLANs. For example, the network connectivity test may be a composite test of DHCP server problems, DNS problems, ARP entry problems, curl problems, or reachability problems. In response to a failure of the network connectivity test, the network management system performs remedial actions, such as user notifications or actions to change configurations. In some examples, the network system may expand the number of devices performing the composite test and have such network devices perform or re-perform the composite test. In some examples, the remedial actions performed by the network system include checking the logs of recent configuration changes, identifying one or more recent configuration changes that may be the root cause of the network connectivity problem, and including the identification of one or more recent configuration changes in the notification to the user.

[0016] In some examples, the network system may perform synthetic tests of the following types, such as those listed in Table 1 below. In response to the failure of such tests, the network system performs remedial actions as described herein.

[0017] Table 1: Synthetic tests used for network connectivity and corresponding symptoms: Table 1: Synthetic tests for network connectivity and corresponding symptoms

[0018] In some examples, when a request for a synthetic test is initiated, the request is made available on a message service topic. For example, the message service topic for synthetic testing could be a "test-request-" Kafka topic. Subsequently, the actual trigger message is made available on the message service topic during the expected time period. For example, the message service topic for triggering a synthetic test could be a "test-trigger-" Kafka topic. For each MAC and VLAN combination specified in the test trigger, the synthetic test runs on the corresponding AP, and the results are published to the message service topic. For example, the message service topic for synthetic test results could be a "ap-synthetic-test-" Kafka topic. In some examples, there is one synthetic test result per AP-VLAN combination.

[0019] In some examples, NMS may wait up to one minute for test results for each AP-VLAN combination, unless a different delay is specified in the test trigger message. If any results are missing after the one-minute window, NMS considers the test "missing." Therefore, for each test, NMS categorizes each result as successful, failed, or lost. After the waiting period, NMS aggregates the results at both the device and site levels. This provides a comprehensive view of the number of devices tested at both the device and VLAN levels, as well as the counts of successful, failed, and lost tests. In some examples, the aggregated data may also include individual results for each AP-VLAN combination.

[0020] Define the success or failure of Curl and DNS tests. In synthetic testing applications, there are two types of tests: default and custom. The definition of success or failure for these tests depends on the test type. For the default test type, synthetic tests use one or more default Uniform Resource Locators (URLs). The event is considered successful if at least one of the URLs returns a success result. For custom tests, users can define one or more URLs to use. If any test in the custom test fails, NMS treats the event as a failure.

[0021] Once NMS aggregates the test results for a given site, it checks for any scope failures. If the failure conditions are met, NMS triggers a remediation action.

[0022] Connectivity-related actions. For connectivity-related tests (such as DHCP, ARP, DNS, and CURL), NMS triggers a remediation action when the same test fails on all APs used for at least one public VLAN. The issued actions are categorized by site, event type (such as DHCP, ARP, DNS, or CURL), and VLAN level.

[0023] Application-related actions. For application-related tests, especially CURL tests, NMS triggers remediation actions when the test fails on all APs across all VLANs. NMS generates URLs for each test. Because CURL tests are involved and failure is required on all VLANs, NMS triggers actions at both the site and URL levels.

[0024] The technology disclosed herein provides specific improvements to computer-related fields of computer networks, with one or more practical applications. For example, various network connectivity problems, such as DNS, DHCP, ARP, cURL, or inter-network device reachability problems, as well as application connectivity problems, may be resolved with simple configuration changes. However, a common technical problem is that these various network connectivity problems can present similarly and are very cumbersome to distinguish and diagnose, especially when such problems occur across customer sites or VLANs. First, for customers, the technology disclosed herein can allow them to easily identify and resolve problems with their DHCP servers, local DNS servers, ARP entries, etc. Currently, DHCP, DNS, and ARP actions may be available to customers. Second, the technology disclosed herein can also benefit internal administrators (such as customer support, product, and development teams) by helping them locate the root cause of any customer-related problems. The technology disclosed herein can also enable network management systems or network administrators to quickly identify, diagnose, and repair various root causes of network connectivity problems, where such root causes may exhibit similar symptoms in other situations, thereby reducing the complexity and difficulty of diagnosing such network connectivity problems. Furthermore, the technology disclosed herein enables network management systems to autonomously instruct various network devices to perform such network connectivity tests on a per-site, per-VLAN, or per-application basis. This can further aid in identifying, diagnosing, and resolving various different root causes of network connectivity problems. The technology disclosed herein can clearly indicate whether the problem originates from a device within the NMS-managed network or from an external source (e.g., a third party or external network, such as an external service provider network and / or a cloud service provider network).

[0025] Details of one or more examples of the technology disclosed herein are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the technology will be apparent from the description and drawings, as well as from the claims.

[0026] Figure 1A This is a block diagram of an example network system 100 according to one or more technologies of this disclosure, including a network management system (NMS) 130 configured to perform automated scheduling and orchestration of synthetic tests for network connectivity within network sites. The example network system 100 includes multiple sites 102A to 102N, at which a network service provider manages one or more wireless networks 106A to 106N respectively. Although in Figure 1A Each site 102A to 102N is shown as including a single wireless network 106A to 106N, but in some examples, each site 102A to 102N may include multiple wireless networks, and this disclosure is not limited in this respect.

[0027] Each site 102A through 102N includes multiple network access server (NAS) devices, such as access points (APs) 142, switches 146, or routers (not shown). For example, site 102A includes multiple APs 142A-1 through 142A-N. Similarly, site 102N includes multiple APs 142N-1 through 142N-M. Each AP 142 can be any type of wireless access point, including but not limited to commercial or enterprise APs, routers, or any other device connected to a wired network and capable of providing wireless network access to client devices within the site. References to “N” or “M” can represent any number. References to “N” for different elements do not have to be the same number. Similarly, references to “M” for different elements do not have to be the same number.

[0028] Each site 102A to 102N also includes multiple client devices, also known as user equipment units (UEs), typically referred to as UEs or client devices 148, representing various wireless-enabled devices within each site. For example, multiple UEs 148A-1 to 148A-N are currently located at site 102A. Similarly, multiple UEs 148N-1 to 148N-M are currently located at site 102N. Each UE 148 can be any type of wireless client device, including but not limited to mobile devices (such as smartphones, tablets, or laptops), personal digital assistants (PDAs), wireless terminals, smartwatches, smart rings, or other wearable devices. UE 148 may also include wired client-side devices, such as IoT devices (such as printers, security devices, environmental sensors) or any other device connected to a wired network and configured to communicate over one or more wireless networks 106.

[0029] To provide wireless network services to UE 148 and / or communicate via wireless network 106, AP 142 and other wired client-side devices at site 102 are directly or indirectly connected to one or more network devices (e.g., switches, routers, etc.) via physical cables (e.g., Ethernet cables). Figure 1A In the example, site 102A includes switch 146A, and each of APs 142A-1 to 142A-N at site 102A is connected to switch 146A. Similarly, site 102N includes switch 146N, and each of APs 142N-1 to 142N-M at site 102N is connected to switch 146N. Although Figure 1AThe diagram illustrates that each site 102 includes a single switch 146 and all APs 142 in a given site 102 are connected to the single switch 146. However, in other examples, each site 102 may include more or fewer switches and / or routers. Furthermore, APs and other wired client-side devices at a given site may be connected to two or more switches and / or routers. Additionally, two or more switches at a site may be interconnected with each other and / or connected to two or more routers, for example, via a mesh or partial mesh topology in a hub-and-spoke architecture. In some examples, the interconnected switches and routers comprise a wired local area network (LAN) at site 102 carrying a wireless network 106.

[0030] Example network system 100 also includes various network components for providing network services within a wired network, including, as an example, an authentication, authorization, and accounting (AAA) server 110 for authenticating users and / or UE 148; a dynamic host configuration protocol (DHCP) server 116 for dynamically assigning network addresses (e.g., IP addresses) to UE 148 during authentication; a domain name system (DNS) server 122 for resolving domain names to network addresses; multiple servers 128A to 128N (collectively referred to as “Server 128”) (e.g., web server, database server, file server, etc.); and a network management system (NMS) 130. Figure 1A As shown, various devices and systems of network system 100 are coupled together via one or more networks 134 (e.g., the Internet and / or corporate intranets).

[0031] exist Figure 1AIn the examples, NMS 130 is a cloud-based computing platform that manages wireless networks 106A to 106N at one or more of sites 102A to 102N. As further described herein, NMS 130 provides an integrated suite of management tools and implements various technologies disclosed herein. In some examples, NMS 130 may be provided as a service (aaS) because it can be provided as part of a continuous subscription-based service to provide network management, analysis, and troubleshooting tools to customers operating one or more sites 102. In general, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analytics, network anomaly identification, and alarm generation. In some examples, NMS 130 outputs notifications to site or network administrators (“administrators”) who interact with and / or operate administrator device 111, such as alerts, alarms, graphical indicators on dashboards, log messages, text / SMS messages, email messages, and / or recommendations regarding wireless network issues. Additionally, in some examples, the NMS 130 operates in response to configuration input received from an administrator interacting with and / or operating the administrator device 111.

[0032] The administrator and administrator device 111 may include IT personnel and administrator computing devices associated with one or more sites in site 102. Administrator device 111 may be implemented as any suitable device for presenting output and / or accepting user input. For example, administrator device 111 may include a display. Administrator device 111 may be a computing system, such as a mobile or non-mobile computing device operated by a user and / or administrator. Administrator device 111 may, for example, represent a workstation, laptop or notebook computer, desktop computer, tablet computer, or any other computing device that can be operated by a user and / or present a user interface according to one or more aspects of this disclosure. Administrator device 111 may be physically separate from NMS 130 and / or located in a different location than NMS 130, such that administrator device 111 can communicate with NMS 130 via network 134 or other communication methods.

[0033] In some examples, one or more NAS devices (e.g., AP 142, switch 146, or router) in the NAS device suite can be connected to edge devices 150A through 150N via physical cables (e.g., Ethernet cables). Edge device 150 includes a cloud-managed wireless local area network (LAN) controller. Each edge device in edge device 150 may include a local device at site 102 that communicates with NMS 130 to extend certain microservices from NMS 130 to the local NAS device, while utilizing NMS 130 and its distributed software architecture for scalable and resilient operation, management, troubleshooting, and analytics.

[0034] Each network device in network system 100, such as servers 110, 116, 122 and / or 128, AP 142, UE 148, switch 146, and any other server or device attached to or constituting part of network system 100, may include a system log or error log module, wherein each of these network devices records the status of the network device, including normal operating status and error conditions. Throughout this disclosure, one or more network devices in network system 100 (e.g., servers 110, 116, 122 and / or 128, AP 142, UE 148, and switch 146) may be considered a “third-party” network device when owned and / or associated with an entity different from NMS 130, such that NMS 130 does not receive, collect, or otherwise access the recorded status and other data of the third-party network device. In some examples, edge device 150 may provide a proxy through which the recorded status and other data of the third-party network device can be reported to NMS 130.

[0035] In some examples, the NMS 130 monitors network data 137 received from wireless networks 106A to 106N at each site 102A to 102N, such as one or more Service Level Expectations (SLE) metrics, and manages network resources (such as APs 142 at each site) to deliver a high-quality wireless experience to end users, IoT devices, and clients at the sites. For example, the NMS 130 may include a Virtual Network Assistant (VNA) 133, which implements an event processing platform for providing real-time insights and simplified troubleshooting for IT operations, and for automatically taking corrective actions or providing recommendations to proactively resolve wireless network issues. The VNA 133 may, for example, include an event processing platform configured to handle hundreds or thousands of concurrent streams of network data 137 from sensors and / or agents associated with nodes within AP 142 and / or network 134. For example, the VNA 133 of the NMS 130 may include an underlying analytics and network error identification engine, as well as an alerting system, according to various examples described herein. The VNA 133's underlying analytics engine can apply historical data and models to inbound event streams to calculate assertions, such as the predicted occurrence of identified anomalies or events constituting network error conditions. Furthermore, the VNA 133 can provide real-time alerts and reports to notify site or network administrators of any predicted events, anomalies, or trends via administrator device 111, and can perform root cause analysis and automated or assisted problem remediation. In some examples, the VNA 133 of the NMS 130 can apply machine learning techniques to identify the root causes of error conditions detected or predicted from the network data 137 stream. If the root cause can be resolved automatically, the VNA 133 can invoke one or more corrective actions to correct the root cause of the error condition, thus automatically improving underlying SLE metrics and automatically improving the user experience.

[0036] Further examples of the operation implemented by the VNA 133 of the NMS 130 are detailed in U.S. Patent No. 9,832,082, entitled "Monitoring Wireless Access Point Events," published November 28, 2017; U.S. Patent No. 11,570,038, entitled "Network System Fault Resolution Using a Machine Learning Model," published January 31, 2023; U.S. Patent No. 10,985,969, entitled "Systems and Methods for a Virtual Network Assistant," published April 20, 2021; and U.S. Patent No. 10,985,969, entitled "Methods and Apparatus for Facilitating Fault Detection and / or Predictive Fault," published March 23, 2021. The entire contents of all of these patents are described in U.S. Patent No. 10,958,585, entitled “Method and Apparatus for Facilitating Fault Detection and / or Predictive Fault Detection”; U.S. Patent No. 10,958,537, entitled “Method for Spatio-Temporal Modeling”, published March 23, 2021; and U.S. Patent No. 10,862,742, entitled “Method for Conveying AP Error Codes Over BLE Advertisements”, published December 8, 2020, and are incorporated herein by reference.

[0037] In operation, NMS 130 observes, collects, and / or receives network data 135, which may take the form of data extracted from messages, counters, and statistics. Depending on one implementation, a computing device is part of NMS 130. Depending on other implementations, NMS 130 may include one or more computing devices, dedicated servers, virtual machines, containers, services, or other forms of environments for performing the techniques described herein. Similarly, computing resources and components implementing VNA 133 may be part of NMS 130, may run on other servers or execution environments, or may be distributed across nodes within network 134 (e.g., routers, switches, controllers, gateways, etc.).

[0038] Typically, network management systems rely on actual client device usage data to test the network at one or more sites within a network. However, during periods of low or no client device usage (e.g., weekends and / or holidays), almost no actual client device usage data is available for network testing. Conversely, testing the network during periods of high client device usage can disrupt the network. Furthermore, without using the techniques disclosed herein, testing is typically performed by each device in the network, which can lead to prolonged network outages (e.g., hours) and / or significant impacts on network infrastructure, potentially posing a higher risk of causing severe network disruptions.

[0039] In some examples, NMS 130 is configured to automate the scheduling and / or orchestration of synthetic tests for network sites. In this example, VNA 133 of NMS 130 includes a synthetic test module 135, which is configured to determine when and / or which devices perform synthetic tests to detect or confirm problems within network system 100. In some examples, synthetic test module 135 is configured to identify certain network conditions to perform synthetic tests on one or more network devices at network site 102A. As an example, synthetic test module 135 may obtain network data 137 from network devices at network site 102, which may include, for example, the number of UEs 148 actively connecting to AP 142 at site 102 and / or usage data of UEs 148 (e.g., the amount of data traffic sent or received). Synthetic test module 135 may determine time periods with minimal or no client activity (e.g., minimal or no UEs 148 actively connecting to the AP or minimal or no data traffic being sent or received). This time period is referred to as an "idle period". Based on idle periods, the synthetic testing module 135 can determine a suggested time period for performing one or more tests. This suggested time period is referred to as a "synthetic testing time window." As further described below, the synthetic testing module 135 can input network data (e.g., historical data of client activity) into a model (e.g., an ML model) to determine the predicted synthetic testing time window. The synthetic testing time window can be determined for a given site. For example, the synthetic testing time window for a specific site (e.g., 102A) may differ from the synthetic testing time window for another site (e.g., 102N).

[0040] Synthetic test module 135 can select one or more devices (e.g., UE 148, AP 142, switch 146, server 128, or other network devices such as network access control (NAC) servers or systems, routers, and / or gateways) to perform tests. For example, synthetic test module 135 can identify, for instance, a minimum number of network devices to perform synthetic tests to minimize the impact on the customer's network infrastructure or the network service provider's cloud infrastructure, referred to herein as the "synthetic test scope." To select one or more specific network devices to perform synthetic tests, synthetic test module 135 can determine the synthetic test scope based on network topology, adjacency relationships between APs, and / or WLAN and VLAN mappings of wireless networks. Synthetic test module 135 can generate a graph representing one or more of the network topology, adjacency relationships between APs, and / or WLAN and VLAN mappings of wireless networks, and select one or more APs to perform tests based on this graph.

[0041] As an example, the synthesis test module 135 (or another module of the NMS 130) can generate a graph representing the network topology of the site. The graph of the network topology can represent the uplink connections from AP 142 to the network hub (e.g., the root node) used for the site. For example, the graph of the network topology may include representations of uplink connections between AP 142 and access switches, uplink connections between access switches and core switches, and uplink connections between core switches and network hubs. The synthesis test module 135 can perform a graph traversal of the network topology graph to determine the network path to the root (e.g., the path from each AP in AP 142 to the network hub) and to determine if any overlapping network paths exist. Based on the determination of overlapping network paths, the synthesis test module 135 can select one or more APs (e.g., the edge nodes(s) of the graph) to perform tests. For example, suppose three APs in AP 142A (e.g., AP 142A-1, AP 142A-2, and AP 142A-3) are each connected to switch 146A at site 102A, which in turn connects to a network hub. In this example, the synthetic test module 135 can perform a graph traversal of the network topology map of the uplink connections within site 102A to determine the path from each AP in AP 142A to the network hub. The synthetic test module 135 can then determine that AP 142A-1 through AP 142A-3 have overlapping network paths to the network hub. Based on the determination of the overlapping network paths, the synthetic test module 135 can select at least one AP from AP 142A-1 through AP 142A-3 to perform a test.

[0042] The graph may additionally or alternatively include a graph representing the proximity relationships between APs 142. For example, the proximity graph may represent APs 142 that have detected signals from one or more neighboring APs. The proximity graph may be generated based on Received Signal Strength Indicator (RSSI) values ​​(or any other information indicating the communication relationship between APs). The composite test module 135 may determine which APs are strong neighbors (e.g., those with higher RSSI values) and which APs are far neighbors (e.g., those with lower RSSI values) based on the proximity graph. The proximity graph may include one or more strong neighbor groups. The composite test module 135 may select one or more APs from each strong neighbor group to perform tests.

[0043] The diagram may additionally or alternatively include a diagram representing the WLAN and VLAN mappings of a site. The composite test module 135 can determine the APs on each WLAN and one or more VLANs associated with each WLAN based on network data. For example, suppose example site 102A has two WLANs, such as a guest WiFi network on a first set of one or more VLANs (e.g., VLAN A) and an enterprise WiFi network on a second set of one or more VLANs (e.g., VLANs B, C, D, and E). In this example, a first set of one or more APs 142A (e.g., AP 142A-1 and AP 142A-4) may be on the guest WiFi network, a second set of one or more APs 142A (e.g., AP 142A-3 and AP 142A-6) may be on the enterprise WiFi network, and a third set of one or more APs 142A (e.g., AP 142A-2 and AP 142A-4) may be on both the enterprise and guest WiFi networks. The composite test module 135 can select the minimum combination of APs and VLANs that provides the maximum VLAN coverage. For example, the synthetic test module 135 can select AP 142A-1 which is mapped to VLAN A on the guest WiFi network; AP 142A-3 which is mapped to VLANs B, C, D and E on the enterprise WiFi network; and AP 142A-5 which is mapped to VLAN A on both the guest WiFi network and the enterprise WiFi network.

[0044] Alternatively or additionally, the synthetic test module 135 can select one or more devices to perform synthetic tests based on a determined device load. For example, the synthetic test module 135 can determine the load of each AP 142A in site 102A based on network data 137 (e.g., client device traffic, number of client devices connected to the AP). Based on the determined load of each AP 142A, the synthetic test module 135 can compare the load with a predefined network load threshold to determine one or more APs 142A with the lowest load, and can select one or more APs 142A with the lowest load to perform synthetic tests.

[0045] In response to determining the network conditions for performing synthetic testing (e.g., synthetic testing time window and / or synthetic testing range), synthetic testing module 135 can then instruct one or more selected APs to perform synthetic testing. As described above, synthetic testing module 135 can instruct one or more selected APs to actively perform synthetic testing (e.g., before a problem or network event is detected). Additionally or alternatively, synthetic testing module 135 can instruct one or more selected APs to passively perform synthetic testing (e.g., in response to a problem or network event being detected) to confirm previously detected problems. Synthetic testing can test for pre-connection and / or post-connection problems in the network.

[0046] In some examples, synthetic testing module 135 may instruct one or more network devices to perform synthetic tests independently of network conditions. For example, synthetic testing module 135 may instruct one or more network devices to perform synthetic tests based on user input received from an administrator (e.g., using administrator device 111) requesting on-demand synthetic tests.

[0047] Synthetic testing module 135 can obtain data from one or more selected APs performing synthetic testing. This data (referred to herein as "test result data") may, in some examples, indicate a problem at the network site or confirm a previously detected problem at the network site. In response to receiving test result data, synthetic testing module 135 is also configured to perform actions, such as sending a notification to the network site administrator indicating the detected problem, and in some cases, sending one or more recommended actions to prevent or repair the problem. In some examples, synthetic testing module 135 may automatically perform corrective actions (e.g., providing configuration for the AP, restarting the AP, etc.) to prevent or repair the problem.

[0048] In some examples, the network device capable of performing synthetic testing may primarily be an AP at a network site, as APs offer the greatest flexibility in simulating client devices, particularly regarding pre-connection tasks. In other examples, a switch at a network site may be additionally or alternatively used as the network device capable of performing synthetic testing; for example, a switch may be used to simulate client devices to test a Radius server (e.g., AAA server 110). In some examples, APs and / or switches may be configured to simulate post-connection traffic types to extend synthetic testing beyond the pre-connection state of client devices. While the examples above describe network devices running synthetic testing, other devices may be able to perform synthetic testing, such as UE 148, server 128, or other devices managed by NMS 130, such as NAC servers or systems, routers, gateways, etc.

[0049] More information on synthetic testing and automated testing of network devices can be found in U.S. Patent Application Publication No. 2024 / 0223489 entitled “SYNTHETIC TESTING”, filed on December 22, 2023 and published on July 4, 2024, the entire contents of which are incorporated herein by reference.

[0050] According to the technology disclosed herein, the synthetic test module 135 includes a network connectivity test 136. The synthetic test module 135, in conjunction with the network connectivity test 136, performs downloadable, automated tests for network connectivity and application connectivity by network devices (such as AP 142 at site 102).

[0051] In examples of this disclosure, the synthetic test module 135 of the NMS 130 receives a trigger action to perform a network connectivity test. In some examples, the trigger action is to perform the network connectivity test in response to a user input request. In some examples, the NMS 130 issues the trigger action on a scheduled or periodic basis, such as hourly, daily, weekly, etc.

[0052] In response to a triggering action, the network connectivity test 136 of the synthetic test module 135 instructs or causes one or more APs 142 to perform a synthetic test for network connectivity. In some examples, synthetic tests for network connectivity include, for example, synthetic tests of network connectivity such as Domain Name Server (DNS), Dynamic Host Configuration Protocol (DHCP) server, Address Resolution Protocol (ARP) requests, cURL data transfer, network reachability to another network device, or applications performed by network devices.

[0053] The network connectivity test 136 of the synthetic test module 135 obtains data from one or more APs 142 that have performed network connectivity tests. Based on the obtained data, the network connectivity test 136 determines whether the network connectivity test of one or more APs 142 has failed. In some examples, the VNA 133 applies a machine learning system to the data to determine the failure of the network connectivity test of the AP 142.

[0054] Based on the type of network connectivity test that AP 142 failed, VNA 133 selects and executes the action to address the root cause of the test failure. In some examples, to execute the selected action, VNA 133 generates a notification to the administrator indicating which AP 142 failed the test and the type of network connectivity test that failed. The notification may optionally include recommended configuration changes to fix the network connectivity problem of AP 142. In some examples, to execute the selected action, VNA 133 selects and executes a remedial action, such as restarting or rebooting one of the APs in AP 142, adjusting the configuration of one of the APs in AP 142, or reinstalling or restarting an application or service performed by one of the APs in AP 142.

[0055] Although the technology of this disclosure is described in this example as being performed by NMS 130, the technology described herein can be performed by any other computing device(s), system(s), and / or server(s), and this disclosure is not limited thereto. For example, one or more computing devices configured to perform the functions of the technology of this disclosure may reside in a dedicated server or be included in any other server besides NMS 130, or may be distributed throughout the network system 100 and may or may not be part of NMS 130.

[0056] Figure 1B It is a diagram. Figure 1A A block diagram providing further examples of the network system details. In this example... Figure 1B The illustration shows the NMS 130, which is configured to operate based on an AI / machine learning-based computing platform, providing comprehensive automation, insights, and guarantees (WiFi guarantees, wired guarantees, and WAN guarantees) ranging from "clients" (e.g., user equipment 148 connected to wireless network 106 and wired LAN 175). Figure 1B (Leftmost) to the “cloud” (e.g., cloud-based application services 181 that can be hosted by computing resources within data center 179). Figure 1B (far right)

[0057] As described herein, NMS 130 provides an integrated suite of management tools and implements various technologies disclosed herein. In general, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analytics, network anomaly identification, and alarm generation. For example, the network management system 130 can be configured to proactively monitor and adaptively configure network system 100 to provide autonomous driving capabilities. Furthermore, VNA 133 includes a natural language processing engine to provide AI-driven support and troubleshooting, anomaly detection, AI-driven location services, and AI-driven radio frequency (RF) optimization utilizing reinforcement learning.

[0058] like Figure 1B As illustrated in the example, the AI-driven NMS 130 also provides configuration management, monitoring, and automated supervision of the Software-Defined Wide Area Network (SD-WAN) 177, which operates as an intermediate network, communicatively coupling the wireless network 106 and wired LAN 175 to the data center 179 and application services 181. Overall, the SD-WAN 177 provides a seamless, secure, and traffic-engineered connection between the "spoke" routers 187A of the wired network 175 (carrying the wireless network 106, such as a branch or campus network) and the "hub" routers 187B in the cloud stack, which are positioned higher up towards the cloud-based application services 181. The SD-WAN 177 typically operates and manages the overlay network on the underlying physical wide area network (WAN), which provides connectivity to geographically separated customer networks. In other words, SD-WAN 177 extends software-defined networking (SDN) capabilities to WAN and allows (multiple) networks to decouple the underlying physical network infrastructure from the virtualized network infrastructure and applications, so that the network can be configured and managed in a flexible and scalable manner.

[0059] In some examples, the underlying routers of the SD-WAN 177 can implement stateful, session-based routing schemes, where routers 187A and 187B dynamically modify the contents of the original packet headers initiated by client device 148 to direct traffic toward application service 181 along a selected path (e.g., path 189) without requiring tunneling and / or additional labeling. In this way, routers 187A and 187B can be more efficient and scalable for large networks because using tunnelless, session-based routing allows routers 187A and 187B to gain significant network resources by eliminating the need to perform encapsulation and decapsulation at tunnel endpoints. Furthermore, in some examples, each router 187A and 187B can independently perform path selection and traffic engineering to control packet flows associated with each session, without requiring a centralized SDN controller for path selection and label distribution. In some examples, routers 187A and 187B implement session-based routing as Secure Vector Routing (SVR), provided by Juniper Networks.

[0060] Additional information regarding session-based routing and SVR can be found in: U.S. Patent No. 9,729,439, published August 8, 2017, entitled "Computer Network Packet Flow Controller"; U.S. Patent No. 9,729,682, published August 8, 2017, entitled "Network Device and Method for Proceeding a Session Using a Packet Signature"; U.S. Patent No. 9,762,485, published September 12, 2017, entitled "Network Packet Flow Controller with Extended Session Management"; and U.S. Patent No. 9,762,485, published January 16, 2018, entitled "Router with Optimized". U.S. Patent No. 9,871,748, entitled "STATISTICAL FUNCTIONALITY (Router with Optimized Statistical Functionality)"; U.S. Patent No. 9,985,883, published May 29, 2018, entitled "NAME-BASED ROUTING SYSTEM AND METHOD"; U.S. Patent No. 10,200,264, published February 5, 2019, entitled "LINK STATUS MONITORING BASEDON PACKET LOSS DETECTION"; U.S. Patent No. 10,277,506, published April 30, 2019, entitled "STATEFUL LOAD BALANCING IN A STATELESS NETWORK"; and U.S. Patent No. 10,277,506, published October 1, 2019, entitled "NETWORK PACKET FLOW CONTROLLER WITH EXTENDED SESSION". The entire contents of each of the following patents are described in U.S. Patent No. 10,432,522, entitled “Network Packet Flow Controller with Extended Session Management”, and U.S. Patent No. 11,075,824, entitled “IN-LINE PERFORMANCE MONITORING”, published on July 27, 2021, and are incorporated herein by reference.

[0061] In some examples, the AI-driven NMS 130 can enable intent-based configuration and management of network system 100, including enabling intent-driven workflows for building, presenting, and executing configuration and management of devices associated with wireless network 106, wired LAN network 175, and / or SD-WAN 177. For example, declarative requirements express the desired configuration of network components without specifying exact local device configurations and control flows. By utilizing declarative requirements, what should be accomplished is specified, rather than how it should be accomplished. Declarative requirements can contrast with imperative instructions that describe the exact device configuration syntax and control flow for achieving the configuration. By utilizing declarative requirements instead of imperative instructions, users and / or user systems are relieved of the burden of determining exact device configurations to achieve the user / system's desired results. For example, when utilizing various types of devices from different vendors, specifying and managing exact imperative instructions to configure each device in the network is often difficult and cumbersome. As new devices are added and devices fail, the types and kinds of devices in the network can change dynamically. Managing various types of devices from different vendors with different configuration protocols, syntaxes, and software versions to configure a coherent device network is often challenging. Therefore, by using declarative requirements that allow users / systems to specify desired outcomes applicable across a wide variety of different types of devices, the management and configuration of network devices become more efficient. Further illustrative details and techniques of intent-based network management systems are described in U.S. Patent No. 10,756,983 (titled "Intent-based Analytics") and U.S. Patent No. 10,992,543 (titled "Automatically generating an intent-based network model of an existing computer network"), each of which is incorporated herein by reference.

[0062] In some examples, NMS 130 is configured to automate the scheduling and / or orchestration of synthetic tests performed on network sites. As described above, synthetic test module 135 is configured to determine certain network conditions of wireless network 106, wired network 175, and / or SD-WAN 177 to perform synthetic tests. For example, synthetic test module 135 may determine idle periods to perform one or more tests where the impact on the customer's network infrastructure or the network service provider's cloud infrastructure is minimal. In some examples, synthetic test module 135 may select one or more devices to perform tests, such as one or more devices associated with wireless network 106 and / or wired LAN network 175 (e.g., ...). Figure 1AThe network connectivity test 136 of the composite test module 135 can be performed on one or more selected devices (e.g., AP 142), router 187 associated with SD-WAN 177, devices associated with data center 179, or any devices managed by NMS 130. Figure 1A AP 142) performs network connectivity tests to identify, diagnose, and repair network connectivity problems as described in this article.

[0063] Figure 2 This is a block diagram of an example access point (AP) device 200 according to one or more technologies disclosed herein. Figure 2 The example AP 200 shown can be used to implement the implementation described in this article. Figure 1A Any of the APs shown and described in AP 142. AP 200 may include, for example, a Wi-Fi, Bluetooth and / or Bluetooth Low Energy (BLE) base station or any other type of wireless access point.

[0064] exist Figure 2 In the example, AP 200 includes a wired interface 230, wireless interfaces 220A to 220B, one or more processors 206, memory 212, and input / output 210, all coupled together via a bus 214 through which the various components exchange data and information. The wired interface 230 represents a physical network interface and includes a receiver 232 and a transmitter 234 for sending and receiving network communications (e.g., packets). The wired interface 230 couples AP 200 directly or indirectly to a wired network device (e.g., Ethernet cable) via a cable (such as an Ethernet cable). Figure 1A (One of the switches in switch 146).

[0065] The first and second wireless interfaces 220A and 220B represent wireless network interfaces and respectively include receivers 222A and 222B, each including a receiving antenna. The AP200 can receive signals from wireless communication devices (such as…) via the receiving antenna. Figure 1A The first and second wireless interfaces 220A and 220B also include transmitters 224A and 224B, each including a transmitting antenna, which the AP200 can transmit to wireless communication devices (such as UE148) via the transmitting antenna. Figure 1A The UE 148 transmits wireless signals. In some examples, the first wireless interface 220A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz), and the second wireless interface 220B may include a Bluetooth interface and / or a Bluetooth Low Energy (BLE) interface.

[0066] Processor 206 is a programmable, hardware-based processor configured to execute software instructions, such as software instructions used to define software or computer programs, which are stored in a computer-readable storage medium (such as memory 212), such as a non-transitory computer-readable medium including storage devices (e.g., disk drives or optical disc drives) or memories (such as flash memory or RAM) or any other type of volatile or non-volatile memory, the stored instructions causing one or more processors 206 to perform the techniques described herein.

[0067] Memory 212 includes one or more devices configured to store programming modules and / or data associated with the operation of AP 200. For example, memory 212 may include computer-readable storage media, such as non-transitory computer-readable media, including storage devices (e.g., disk drives or optical disk drives) or memories (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions to cause one or more processors 206 to perform the techniques described herein.

[0068] In this example, memory 212 stores executable software, including an application programming interface (API) 240, a communication manager 242, configuration settings 250, a device status log 252, a data storage device 254, and a log controller 255. The device status log 252 includes a list of AP 200-specific events. Events can include logs of both normal and error events, such as, for example, memory status, reboot or restart events, crash events, cloud disconnection events with self-recovery, low link speed or link speed fluctuation events, Ethernet port status, Ethernet interface packet errors, upgrade failure events, firmware upgrade events, configuration changes, etc., along with a time and date stamp for each event. The log controller 255 determines the logging level for the device based on instructions from NMS 130. Data 254 can store any data used and / or generated by AP 200, including data collected from UE 148, such as data used to calculate one or more SLE metrics, transmitted by AP 200 for cloud-based management of wireless network 106A by NMS 130.

[0069] Input / output (I / O) 210 represents physical hardware components that enable user interaction, such as buttons, displays, etc. Although not shown, memory 212 typically stores executable software for controlling the user interface regarding input received via I / O 210. Communication manager 242 includes program code that, when executed by processor(s) 206, allows AP 200 to communicate with UE 148 and / or network(s) 134 via interfaces(s) 230 and / or any of interfaces 220A to 220C. Configuration settings 250 include any device settings for AP 200, such as radio settings for each of the multiple wireless interfaces 220A to 220C. These settings can be configured manually or remotely monitored and managed by NMS 130 to optimize wireless network performance on a periodic (e.g., hourly or daily) basis.

[0070] As described herein, AP device 200 can measure network data and report it from status log 252 to NMS 130. Network data may include event data, telemetry data, and / or other SLE-related data. Network data may include various parameters indicating the performance and / or status of the wireless network. Parameters may be measured and / or determined by one or more UE devices and / or one or more APs in the wireless network. NMS 130 can determine one or more SLE metrics based on SLE-related data received from APs in the wireless network and store the SLE metrics as network data 137. Figure 1A ).

[0071] According to the technology described in this disclosure, AP device 200 includes test engine 256, which is configured to perform tests from synthetic test modules (e.g., Figure 1AThe network connectivity test 136 of the synthetic test module 135 receives one or more network connectivity tests. For example, the AP device 200 may receive a message from the network connectivity test 136 of the synthetic test module 135 specifying the address (e.g., MAC address) of the selected AP device and a payload specifying the test information to be performed by the selected AP device. The test information may include a configuration file or command to perform one or more synthetic tests for network connectivity. In response to the determination message including the address of the AP device 200, the AP device 200 may extract the test information (e.g., command) from the payload of the message and provide the test information to the test engine 256. The test engine 256 may then execute the command received from the network connectivity test 136 of the synthetic test module 135. As an example, the message from the synthetic test module 135 may include test information, such as synthetic tests (or any other tests) performed by the AP device 200 on connectivity with a DNS server or DHCP server, ARP requests, cURL data transfers, network reachability to another network device (such as another AP 142 or server), or network connectivity of an application performed by the AP device 200. Test information may include, for example, configuration information (e.g., commands) for AP device 200 to perform speed tests between AP device 200 and the target device, the VLAN to be used for network connectivity testing, the test protocol, the schedule for performing speed tests, and / or other information that may instruct AP device 200 to perform synthetic network connectivity tests. In some examples, a third-party vendor may provide services for performing the tests. In these examples, synthetic test module 135 may send test information including vendor-specific instructions to enable AP device 200 to collaborate with the third-party vendor providing the test services.

[0072] As part of the command execution, test engine 256 can receive and / or generate test results. Test engine 256 can store the test results in memory 212 as test result data 257. Test engine 256 can send test result data 257 to synthetic test module 135 for further analysis, as further described below.

[0073] Figure 3A This is a block diagram of an example network management system (NMS) 300 according to one or more technologies of this disclosure. The NMS 300 can be used to implement, for example... Figures 1A to 1B The NMS 130 is used in this example. In this type of example, the NMS 300 is responsible for monitoring and managing one or more wireless networks 106A to 106N located at sites 102A to 102N respectively.

[0074] The NMS 300 includes a communication interface 330, one or more processors 306, a user interface 310, a memory 312, and a database 318. The various components are coupled together via a bus 314, through which they can exchange data and information. In some examples, the NMS 300 receives data from client devices 148, APs 142, switches 146, and other network nodes within the network 134 (e.g., ...). Figure 1B The NMS 300 receives data from one or more routers (187), which can be used to calculate one or more SLE metrics and / or update network data 316 in database 318. The NMS 300 analyzes this data for cloud-based management of wireless networks 106A to 106N. In some examples, the NMS 300 may be... Figure 1A This refers to a portion of another server or any other server.

[0075] Multiple processors 306 execute software instructions, such as instructions used to define software or computer programs, which are stored on a computer-readable storage medium (such as memory 312), such as a non-transitory computer-readable medium, including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions to cause one or more processors 306 to perform the techniques described herein.

[0076] The communication interface 330 may include, for example, an Ethernet interface. The communication interface 330 couples the NMS 300 to a network and / or the Internet, such as... Figure 1A Any one of the multiple networks 134 shown, and / or any local area network. Communication interface 330 includes receiver 332 and transmitter 334, through which NMS 300 receives data from any client device 148, AP 142, switch 146, server 110, 116, 122, 128, and / or other entities constituting a network such as... Figure 1A Any other network node, device, or system within the network system 100 shown may receive / send data and information to it. In some scenarios described herein, where network system 100 includes “third-party” network devices owned and / or associated with an entity different from NMS 300, NMS 300 does not receive, collect, or otherwise access network data from third-party network devices.

[0077] Data and information received by NMS 300 may include, for example, data from client device 148, AP 142, switch 146, or other network nodes (e.g., Figure 1BThe NMS 300 uses telemetry data, SLE-related data, or event data received from one or more routers (187) to remotely monitor the performance of wireless networks 106A to 106N and application sessions from client devices to cloud-based application servers. The NMS 300 can further transmit data via communication interface 330 to any network device (such as client device 148, AP 142, switch 146, other network nodes within network 134, and administrator device 111) to remotely manage portions of wireless networks 106A to 106N and the wired network.

[0078] Memory 312 includes one or more devices configured to store programming modules and / or data associated with the operation of NMS 300. For example, memory 312 may include computer-readable storage media, such as non-transitory computer-readable media, including storage devices (e.g., disk drives or optical disk drives) or memories (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions to cause one or more processors 306 to perform the techniques described herein.

[0079] In this example, memory 312 includes API 320, SLE module 322, Virtual Network Assistant (VNA) / AI engine 350, and Radio Resource Management (RRM) engine 360. NMS 300 may also include any other programming modules, software engines, and / or interfaces configured for remote monitoring and management of wireless network 106A to 106N and wired network portions, including AP142 / 200, switch 146, or other network devices (e.g., Figure 1B Remote monitoring and management of any item in the router (187).

[0080] SLE module 322 enables the setting and tracking of thresholds for SLE metrics for each network 106A to 106N. SLE module 322 further analyzes SLE-related data collected by APs (such as any AP 142) from UEs in each wireless network 106A to 106N. For example, APs 142A-1 to 142A-N collect SLE-related data from UEs 148A-1 to 148A-N currently connected to wireless network 106A. This data is transmitted to NMS 300, which, through SLE module 322, determines one or more SLE metrics for each UE 148A-1 to 148A-N currently connected to wireless network 106A. This data, in addition to any network data collected by one or more APs 142A-1 to 142A-N in wireless network 106A, is transmitted to NMS 300 and stored as network data 316 in, for example, database 318.

[0081] RRM Engine 360 ​​monitors one or more metrics for each site 102A through 102N to understand and optimize the RF environment at each site. For example, RRM Engine 360 ​​can monitor coverage and capacity SLE metrics for wireless network 106 at site 102 to identify potential SLE coverage and / or capacity issues in wireless network 106 and adjust the radio settings of the access points at each site to address the identified issues. For example, RRM Engine 360 ​​can determine the channel and transmit power distribution across all APs 142 in each network 106A through 106N. For example, RRM Engine 360 ​​can monitor events, power, channels, bandwidth, and the number of clients connected to each AP. RRM Engine 360 ​​can further automatically change or update the configuration of one or more APs 142 at site 102 to improve coverage and capacity SLE metrics, thus providing an improved wireless experience for users.

[0082] The VNA / AI engine 350 analyzes data received from network devices and its own data to identify when an unexpected anomalous state is encountered at one of the network devices. For example, the VNA / AI engine 350 can identify the root cause of any unexpected or anomalous state, such as any poor SLE metrics indicating connectivity problems at one or more network devices. Furthermore, the VNA / AI engine 350 can automatically invoke one or more corrective actions designed to resolve the identified root cause of one or more poor SLE metrics. Examples of corrective actions that can be automatically invoked by the VNA / AI engine 350 may include, but are not limited to: invoking RRM 360 to restart one or more APs, adjusting / modifying the transmit power of a specific radio in a specific AP, adding an SSID configuration to a specific AP, changing the channel of an AP or set of APs, etc. Corrective actions may also include restarting switches and / or routers, invoking new software downloads to APs, switches, or routers, etc. These corrective actions are given for illustrative purposes only, and this disclosure is not limited in this respect. If automatic corrective actions are unavailable or fail to adequately address the root cause, the VNA / AI engine 350 can proactively provide notifications, including suggested corrective actions, for IT personnel (e.g., site administrators or network administrators using Administrator Device 111) to resolve network errors.

[0083] According to one or more techniques disclosed herein, the VNA 350 includes a synthetic test module 352 configured to automatically schedule and / or orchestrate synthetic tests for network sites, as described below. Figure 3B Further described, the synthetic test module 352 also includes a network connectivity test 353, which enables the orchestration of network connectivity tests for network devices at network sites.

[0084] In some examples, the VNA 350 applies the ML model 380 to data obtained from AP142, which has performed network connectivity tests, to determine failures in the network connectivity tests of one or more network devices. In some examples, the ML model 380 may include a supervised ML model trained using training data, which includes pre-collected, labeled network data from network devices (e.g., client devices, APs, switches, and / or other network nodes) to identify network connectivity problems. For example, the training data may include historical data obtained from network devices that have performed network connectivity tests. The supervised ML model may include one of the following: logistic regression, Naive Bayes, Support Vector Machine (SVM), etc. In other examples, the ML model 380 may include an unsupervised ML model. Although... Figure 3A Not shown, but in some examples, database 318 may store training data, and VNA / AI engine 350 or dedicated training module may be configured to train ML model 380 based on the training data to determine appropriate weights for one or more features across the training data.

[0085] The VNA 350 selects and executes an action to address the root cause of a network connectivity test failure based on the type of failure for one or more network devices. In some examples, to execute the selected action, the VNA 133 generates a notification to the administrator indicating the network device that failed the test and the type of network connectivity test failure. In some examples, the notification indicates the VLAN configured on the network device and that the network connectivity test failed for that VLAN. In some examples, the notification indicates the site to which the network device belongs. The notification may optionally include recommended configuration changes to fix the network connectivity problem with AP 142.

[0086] In some examples, in order to perform the selected action, VNA 133 selects a corrective action and performs the corrective action, such as restarting or rebooting one of APs 142, adjusting the configuration of one of APs 142, or reinstalling or restarting an application or service performed by one of APs 142.

[0087] In some examples, VNA 133 can determine that multiple APs 142 individually fail network connectivity tests related to the shared VLANs configured on the multiple APs 142. This can occur when VLAN configuration causes errors affecting the network connectivity of AP 142 when using that VLAN. In this example, VNA 133 performs remedial actions on the AP 142 configured with the shared VLAN to fix the network connectivity problem.

[0088] In some examples, the NMS 300 may employ a message service 381 for exchanging messages between applications and services executed by the NMS 300. The message service 381 can be used, for example, in implementations where the NMS 300 is distributed across multiple computing systems, data centers, or sites. For instance, the synthetic test module 352 may receive a trigger message from the message service 381, specifying one or more APs 142, one or more VLANs configured on the specified APs 142, and the type of network connectivity test to be performed. The synthetic test module 352 may instruct the specified one or more APs 142 to perform the specified network connectivity test for each VLAN configured on the specified APs 142. In an example where APs 142 are configured with more than one VLAN, the synthetic test module 352 iteratively instructs APs 142 to perform network connectivity tests for each VLAN configured on the APs 142.

[0089] Figure 3B This is a block diagram illustrating an example synthetic test module according to the technology described in this disclosure. Synthetic test module 352 includes one or more software modules and / or engines, which, when powered by one or more processors (e.g., ...), Figure 3A When the (multiple) processors 306) execute, they enable processor scheduling and / or orchestration of synthetic tests for the network site. In this example, Figure 3B It includes a synthetic test time window module 382, ​​a synthetic test range module 384, and a WLAN / VLAN mapping module 386.

[0090] Synthetic test time window module 382 is configured to determine time periods with minimal or no client activity (e.g., idle periods). For example, synthetic test time window module 382 can obtain network data, such as active client counts, from each AP of a site (e.g., AP 142A of site 102A). Based on the obtained network data, synthetic test time window module 382 can determine idle periods as an example. Based on idle periods, synthetic test time window module 382 can determine recommended time periods (e.g., synthetic test time windows) for running one or more tests. Administrators can specify criteria for determining synthetic test time windows for ML models. In some examples, synthetic test time window module 382 outputs predictions of synthetic test time windows (e.g., forecasts). Synthetic test time windows can be determined for a given site. For example, a synthetic test time window for a specific site (e.g., 102A) can differ from a synthetic test time window for another site (e.g., 102N).

[0091] The synthesized test range module 384 is configured to determine the test range for running tests. For example, the synthesized test range module 384 (or another module of the NMS 300) can obtain network data from network devices within the site to generate a graph representing the site's network topology. As an example, the synthesized test range module 384 can obtain network data associated with network nodes within the site. The synthesized test range module 384 can also obtain network data associated with edges of the site's network nodes. The synthesized test range module 384 combines the aforementioned network data to determine connections between all nodes in the site. In response to determining connections between all nodes in the site, the synthesized test range module 384 can generate a graph of the network topology representing uplink connections from AP 142 to the network hub (e.g., the root node) used for the site. For example, the graph of the network topology may include representations of uplink connections between AP 142 and access switches, uplink connections between access switches and core switches, and uplink connections between core switches and network hubs.

[0092] The synthetic test range module 384 can perform a graph traversal of the network topology to determine the network path to the root (e.g., the path from each AP in AP 142 to the network hub) and determine if any overlapping network paths exist. For example, the synthetic test range module 384 can determine the network nodes along the path to the root for each AP. Based on the determination of overlapping network paths, the synthetic test range module 384 can select one or more APs (e.g., edge nodes(s) of the graph) to perform tests.

[0093] The graph can additionally or alternatively include a graph representing the proximity relationships between APs. For example, a proximity graph can represent APs that have detected signals from one or more neighboring APs. The proximity graph can be generated based on RSSI values ​​(or any other information indicating the communication relationship between APs). The synthetic test range module 384 can determine which APs are strong neighbors (e.g., those with higher RSSI values) and which APs are far neighbors (e.g., those with lower RSSI values) based on the proximity graph. The proximity graph can include one or more strong neighbor groups. The synthetic test range module 384 can select one or more APs from each strong neighbor group within the strong neighbor groups to perform tests.

[0094] Alternatively or additionally, the synthetic test range module 384 can select one or more devices to perform synthetic tests based on a determined device load. For example, the synthetic test range module can determine the load of each AP in the network site based on network data 316 (e.g., client device traffic, number of client devices connected to the AP). Based on the determined load of each AP, the synthetic test range module 384 can compare the load with a predefined network load threshold to identify one or more APs with the lowest load, and can select one or more APs with the lowest load to perform synthetic tests.

[0095] Additional or alternative locations may include a map representing the mapping of WLANs and VLANs within the site. The WLAN / VLAN mapping module 386 may also obtain network data associated with the WLAN and VLAN mappings. The WLAN / VLAN mapping module 386 combines the aforementioned network data to determine the VLAN for each AP in the site. In response to determining the VLAN for each AP in the site, the composite test range module 384 can select the AP that provides the maximum WLAN and VLAN combination coverage.

[0096] In some examples, the synthetic test range module 384 can select one or more APs to perform tests based on an aggregation graph representing network topology, proximity relationships, and WLAN and VLAN mappings for the site. In this example, the synthetic test range module 384 can select one or more APs from APs with overlapping network paths, from APs in one or more strong proximity groups, and from APs that provide the minimum AP and VLAN combination that offers the maximum VLAN coverage. In some examples, VLANs are evenly redistributed among the APs selected for the site to ensure a uniform distribution of tests across the APs.

[0097] The synthetic test module 352 may include a test selection module 388, which is configured to select one or more tests to be performed. In some examples, the test selection module 388 may select one or more tests to detect pre-connection (e.g., tests to detect end-to-end connectivity, etc.) and / or post-connection problems (e.g., tests to determine whether the network device is functioning as expected, etc.).

[0098] According to the technology disclosed herein, the test selection module 388 interacts with the network connectivity test 353 to select one or more network connectivity tests. Network connectivity tests may include, for example, DNS test 341, DHCP test 342, ARP test 343, cURL test 344, network device reachability test 345, and / or application reachability test 346.

[0099] In some examples, DNS test 341 includes a test of resolving URLs using a DNS server. A network device may fail DNS test 341 if it fails to resolve URLs using a DNS server for at least one VLAN configured on the network device.

[0100] In some examples, DHCP test 342 includes a test of network connectivity with the DHCP server. A network device may fail DHCP test 342 if it fails to establish reachability with the DHCP server for at least one VLAN configured on the network device.

[0101] In some examples, ARP test 343 includes a test for resolving ARP requests. A network device may fail ARP test 343 if it issues an ARP request and fails to resolve at least one VLAN configured on the network device.

[0102] In some examples, cURL test 344 includes a test of cURL data transmission. The network device may fail cURL test 344 if it attempts to perform cURL data transmission and the data transmission fails or fails to complete for at least one VLAN configured on the network device.

[0103] In some examples, network device reachability test 345 includes a network reachability test between the network device under test and the second network device. The network device under test may fail network device reachability test 345 if it cannot reach the second network device for at least one VLAN configured on the network device.

[0104] In some examples, application reachability test 346 includes testing the network connectivity of an application performed by a network device. The network device under test may fail application reachability test 346 if the application cannot be configured for each VLAN on the switching network device.

[0105] In some examples, the synthetic testing module 352 may include a test activation engine 389 configured to determine whether to actively or passively invoke a synthetic test based on network data 316, such as network events. For example, the test activation module 389 may determine whether to execute a synthetic test based on network data 316, regardless of whether a previously detected problem or network event exists. In some examples, the test activation engine 389 may determine whether to execute a synthetic test in response to the detection of a previous problem or network event to confirm the previously detected problem or network event (e.g., if the data indicates a DHCP connection problem, a DHCP test is selected to confirm the DHCP connection problem).

[0106] In some examples, the synthetic test module 352 can receive user input via user interface 310 to request the execution of synthetic tests on demand. In response, the test activation engine 389 can configure the synthetic test module 352 to execute synthetic tests on demand.

[0107] In some examples, the test activation engine 389 can provide test scheduling to avoid multiple tests executing simultaneously, which could increase network overhead. For example, the test activation engine 389 can determine, based on a request to initiate an on-demand synthesis test, whether the execution of the on-demand synthesis test overlaps with other scheduled tests that are already being executed or scheduled to be executed within the test duration of the on-demand synthesis test. Based on the determination that the execution of the requested on-demand synthesis test overlaps with another test that is already being executed or scheduled to be executed within the test duration of the on-demand synthesis test, the test activation engine 389 can block or reschedule the on-demand synthesis test or other tests respectively, or provide the user with a notification of the overlapping test durations to delay or reschedule the on-demand synthesis test or other tests.

[0108] Synthetic testing module 352 may include test result engine 390, which is configured to acquire network data (referred to herein as "test result data") received from network devices that have performed synthetic tests. In response to acquiring the test result data, test result engine 390 may identify one or more problems based on the test result data. In some examples, test result engine 390 may send the test result data to SLE module 322 to analyze the test result data and determine one or more SLE metrics based on the test result data.

[0109] The synthetic test module 352 may also include an action module 392 that can perform actions in response to identifying one or more problems determined from the test result data. For example, the action module 392 may generate and send a notification to the network site administrator indicating the detected problem, such as generating user interface elements representing the detected problem for display on a display device. In some examples, the action module 392 may automatically correct the detected problem (e.g., configuring the AP to re-establish the DHCP connection if it is a DHCP connection problem, restarting or rebooting the device, adjusting the AP's configuration, etc.).

[0110] In some examples, when a request for a synthetic test is initiated, the request becomes available on a message service topic in message service 381. For example, the message service topic for synthetic testing could be a "test-request-" Kafka topic. Subsequently, the actual trigger message becomes available on the message service topic in message service 381 during the expected time period. For example, the message service topic for triggering a synthetic test could be a "test-trigger-" Kafka topic. For each MAC and VLAN combination specified in the test trigger, the synthetic test runs on the corresponding AP, and the results are published to the message service topic. For example, the message service topic for synthetic test results could be a "ap-synthetic-test-" Kafka topic. In some examples, there is one synthetic test result per AP-VLAN combination.

[0111] In some examples, the composite test module 352 may wait up to one minute for test results for each AP-VLAN combination, unless a different delay is specified in the test trigger message. If any results are missing after the one-minute window, the composite test module 352 considers the test “lost.” Therefore, for each test, the composite test module 352 categorizes each result as successful, failed, or lost. After the waiting period, the composite test module 352 aggregates the results at both the device and site levels. This provides a comprehensive view of the number of devices tested at both the device and VLAN levels, as well as the counts of successful, failed, and lost tests. In some examples, the aggregated data may also include individual results for each AP-VLAN combination.

[0112] The following is an example of aggregated data (including individual results for each AP-VLAN combination): Aggregated test results payload { "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "event_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "status": 1, "event_ts": 0, "trigger_ts": 1727964957081, "trigger_wait_time": 780, "delta_wait_time": 780, "start_ts": 1727964956301, "end_ts": 1727965014319, "who": "VNA", "test_entities": [ { "mac": "a8f7d9816d50", "vlans": [ 1, 2 ], "tests": [ "CURL" ] }, { "mac": "000000000001", "vlans": [ 1, 2 ], "tests": [ "CURL" ] } ], "metadata": { "test_type": "custom", "test_scope": "default", "test_name": "CONNECTIVITY_TEST", "who": " VNA" }, "test_scopes": [ { "entity_type": "ap", "mac": "a8f7d9816d50", "scopes": [ { "mac": "a8f7d9816d50", "delay": 181, "test_name": "CONNECTIVITY_TEST", "test_type": "custom", "urls": [ "https: / / captive.foo.com ", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / squad.bar.com" ], "vlan_ids": [ 1, 2 ] } ] }, { "entity_type": "ap", "mac": "000000000001", "scopes": [ { "mac": "000000000001", "delay": 181, "test_name": "CONNECTIVITY_TEST", "test_type": "custom", "urls": [ "https: / / captive.foo.com ", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / squad.bar.com" ], "vlan_ids": [ 1, 2 ] } ] } ], "tested_devices": [ "000000000001", "a8f7d9816d50" ], "summary_report": { "CURL": 4, "DNS": 4 }, "fail_agg_summary_report": { "CURL": 4 }, "success": 4, "failures": 4, "no_of_devices_tested": 2, "no_of_devices_failed": 2, "no_of_devices_lost": 0, "vlans_set": [ 1, 2 ], "no_of_vlans_tested": 2, "vlans_failed_set": [ 1, 2 ], "no_of_vlans_failed": 2, "no_of_vlans_success": 2, "device_reports": { "000000000001": { "firmware": "0.15.31238", "es_syn_tests": { "00000000-0000-0000-0000-000000000000": [ { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "000000000001", "source": " VNA", "vlan": 1, "test_name": "CURL", "ev_type": "DNS", "ev_status": "SUCCESS", "device_type": "ap", "servers": [ "192.168.2.1" ], "urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / captive.foo.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "DNSIP": "192.168.2.1", "IPs": [ "17.253.97.205", "17.253.97.206" ], "Latency": 14, "URL": "https: / / captive.foo.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "142.250.80.99" ], "Latency": 10, "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "DNSIP": "192.168.2.1", "IPs": [ "13.107.6.156" ], "Latency": 11, "URL": "https: / / business.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "52.123.128.14", "52.123.129.14" ], "Latency": 13, "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "dns" }, { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "000000000001", "source": " VNA", "vlan": 1, "test_name": "CURL", "ev_type": "CURL", "ev_status": "FAILED", "device_type": "ap", "servers": [ null ], "failed_servers": [ null ], "urls": [ "https: / / squad.bar.com", "https: / / captive.foo.com " ], "failed_urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "ClientIP": "192.168.1.16", "Latency": 61, "Response": "200 OK", "URL": "https: / / captive.foo.com " }, { "ClientIP": "192.168.1.16", "Latency": 51, "Error": "Failed http get", "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "ClientIP": "192.168.1.16", "Latency": 335, "Error": "Failed http get", "URL": "https: / / business.com " }, { "ClientIP": "192.168.1.16", "Latency": 157, "Response": "408 Timeout", "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "classifier": "curl", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "curl" }, { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "000000000001", "source": " VNA", "vlan": 2, "test_name": "CURL", "ev_type": "DNS", "ev_status": "SUCCESS", "device_type": "ap", "servers": [ "192.168.2.1" ], "urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / captive.foo.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "DNSIP": "192.168.2.1", "IPs": [ "17.253.97.205", "17.253.97.206" ], "Latency": 14, "URL": "https: / / captive.foo.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "142.250.80.99" ], "Latency": 10, "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "DNSIP": "192.168.2.1", "IPs": [ "13.107.6.156" ], "Latency": 11, "URL": "https: / / business.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "52.123.128.14", "52.123.129.14" ], "Latency": 13, "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "dns" }, { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "000000000001", "source": " VNA", "vlan": 2, "test_name": "CURL", "ev_type": "CURL", "ev_status": "FAILED", "device_type": "ap", "servers": [ null ], "failed_servers": [ null ], "urls": [ "https: / / squad.bar.com", "https: / / business.com ", "https: / / captive.foo.com " ], "failed_urls": [ "https: / / connectivitycheck.gstatic.com / generate_204" ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "ClientIP": "192.168.1.16", "Latency": 61, "Response": "200 OK", "URL": "https: / / captive.foo.com " }, { "ClientIP": "192.168.1.16", "Latency": 51, "Error": "Failed http get", "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "ClientIP": "192.168.1.16", "Latency": 335, "Response": "200 OK", "URL": "https: / / business.com " }, { "ClientIP": "192.168.1.16", "Latency": 157, "Response": "200 OK", "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "classifier": "curl", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "curl" } ] }, "test_vlan": { "1": [ "CURL" ], "2": [ "CURL" ] }, "failed_vlan": { "1": [ "CURL" ], "2": [ "CURL" ] }, "success_vlan": { "1": [ "DNS" ], "2": [ "DNS" ] }, "summary_report": { "success": 2, "failures": 2, "lost": 0, "summary_report_tests": { "CURL": 2, "DNS": 2 }, "failed_report_tests": { "CURL": 2 } }, "vlan_ones": [ 1, 2 ], "is_end_received": true }, "a8f7d9816d50": { "firmware": "0.15.31238", "es_syn_tests": { "00000000-0000-0000-0000-000000000000": [ { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "a8f7d9816d50", "source": " VNA", "vlan": 1, "test_name": "CURL", "ev_type": "DNS", "ev_status": "SUCCESS", "device_type": "ap", "servers": [ "192.168.2.1" ], "urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / captive.foo.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "DNSIP": "192.168.2.1", "IPs": [ "17.253.97.205", "17.253.97.206" ], "Latency": 14, "URL": "https: / / captive.foo.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "142.250.80.99" ], "Latency": 10, "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "DNSIP": "192.168.2.1", "IPs": [ "13.107.6.156" ], "Latency": 11, "URL": "https: / / business.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "52.123.128.14", "52.123.129.14" ], "Latency": 13, "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "dns" }, { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "a8f7d9816d50", "source": " VNA", "vlan": 1, "test_name": "CURL", "ev_type": "CURL", "ev_status": "FAILED", "device_type": "ap", "servers": [ null ], "failed_servers": [ null ], "urls": [ "https: / / squad.bar.com", "https: / / captive.foo.com " ], "failed_urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / captive.foo.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "ClientIP": "192.168.1.16", "Latency": 61, "Response": "408 not OK", "URL": "https: / / captive.foo.com " }, { "ClientIP": "192.168.1.16", "Latency": 51, "Error": "Failed http get", "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "ClientIP": "192.168.1.16", "Latency": 335, "Error": "Failed http get", "URL": "https: / / business.com " }, { "ClientIP": "192.168.1.16", "Latency": 157, "Response": "408 Timeout", "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "classifier": "curl", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "curl" }, { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "a8f7d9816d50", "source": " VNA", "vlan": 2, "test_name": "CURL", "ev_type": "DNS", "ev_status": "SUCCESS", "device_type": "ap", "servers": [ "192.168.2.1" ], "urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / captive.foo.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "DNSIP": "192.168.2.1", "IPs": [ "17.253.97.205", "17.253.97.206" ], "Latency": 14, "URL": "https: / / captive.foo.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "142.250.80.99" ], "Latency": 10, "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "DNSIP": "192.168.2.1", "IPs": [ "13.107.6.156" ], "Latency": 11, "URL": "https: / / business.com " }, { "DNSIP": "192.168.2.1", "IPs": [ "52.123.128.14", "52.123.129.14" ], "Latency": 13, "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "dns" }, { "When": "2024-10-03T14:15:56.301845Z", "org_id": "59829b59-84ca-4916-9999-5126cac6f66a", "site_id": "7bcc62d0-f8f8-4fa5-94fb-bcd9e5188d7f", "mac": "a8f7d9816d50", "source": " VNA", "vlan": 2, "test_name": "CURL", "ev_type": "CURL", "ev_status": "FAILED", "device_type": "ap", "servers": [ null ], "failed_servers": [ null ], "urls": [ "https: / / squad.bar.com", "https: / / captive.foo.com " ], "failed_urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com ", "https: / / captive.foo.com " ], "test_id": "f2d34e0d-ce73-41a1-a92e-95d7605c09a8", "test_detail": { "URLs": [ { "ClientIP": "192.168.1.16", "Latency": 61, "Response": "408 not OK", "URL": "https: / / captive.foo.com " }, { "ClientIP": "192.168.1.16", "Latency": 51, "Error": "Failed http get", "URL": "https: / / connectivitycheck.gstatic.com / generate_204" }, { "ClientIP": "192.168.1.16", "Latency": 335, "Error": "Failed http get", "URL": "https: / / business.com " }, { "ClientIP": "192.168.1.16", "Latency": 157, "Response": "408 Timeout", "URL": "https: / / squad.bar.com" } ] }, "apfw": "0.15.31238", "classifier": "curl", "pcap_id": "00000000-0000-0000-0000-000000000000", "has_pcap": false, "latency": 0, "jitter_ms": 0, "tx_mbps": 0, "rx_mbps": 0, "start": 0, "end": 0, "type": "curl" } ] }, "test_vlan": { "1": [ "CURL" ], "2": [ "CURL" ] }, "failed_vlan": { "1": [ "CURL" ], "2": [ "CURL" ] }, "success_vlan": { "1": [ "DNS" ], "2": [ "DNS" ] }, "summary_report": { "success": 2, "failures": 2, "lost": 0, "summary_report_tests": { "CURL": 2, DNS: 2 }, "failed_report_tests": { "CURL": 2 } }, "vlan_ones": [ 1, 2 ], "is_end_received": true } }, "retry": 0 }

[0113] Define the success or failure of Curl and DNS tests. In synthetic testing applications, there are two types of tests: default and custom. The definition of success or failure for these tests depends on the test type. For the default test type, synthetic tests use one or more default Uniform Resource Locators (URLs). The event is considered successful if at least one of the URLs returns a success result. For custom tests, users can define one or more URLs to use. If any test in the custom test fails, NMS treats the event as a failure. Once NMS aggregates the test results for a given site, it checks for any scoped failures. If the failure conditions are met, NMS triggers a remediation action.

[0114] Connectivity-related actions. For connectivity-related tests (such as DHCP, ARP, DNS, and CURL), when the same test fails on all APs used for at least one public VLAN, the synthetic test module 352 triggers a remedial action. The issued actions are categorized by site, event type (such as DHCP, ARP, DNS, or CURL), and VLAN level. An example of the action payload is illustrated below:

[0115] The following is an example of DHCP actions for VLAN 501: DHCP actions on VLAN 501 { "_index": "entity_suggestion_202411", "_id": "be47404e-6e38-4546-b93c-00c083622e75&dhcp_failure&8bd9ad54-1323-407d-9a29-a10b5fded7e9&vlan=501&dhcp&1731596281706", "_score": null, "_source": { "row_key": "be47404e-6e38-4546-b93c-00c083622e75&dhcp_failure&8bd9ad54-1323-407d-9a29-a10b5fded7e9&vlan=501&dhcp&1731596281706", "org_id": "be47404e-6e38-4546-b93c-00c083622e75", "site_id": "8bd9ad54-1323-407d-9a29-a10b5fded7e9", "entity_type": "site", "entity_id": "8bd9ad54-1323-407d-9a29-a10b5fded7e9&vlan=501&dhcp", "unique_key": "8bd9ad54-1323-407d-9a29-a10b5fded7e9&vlan=501&dhcp", "symptom": "dhcp_failure", "suggestion": "check DHCP Failure", "display_name": "DHCP_FAILURE", "category": "connectivity", "status": "open", "start_time": 1731596281706, "end_time": 1731661921671, "suggestion_time": 1731596385804, "NMS_only": false, "enable_notification": false, "impact_scope": "site", "severity": 60, "duration": 65639, "batch_count": 48, "modification_time": 1731661968328, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "176bd196-bdde-44c6-b641-8fe091aa3b5c", "test_type": "connectivity", "who": "minis", "impacted_vlans": "501" , "impacted_tuple": { "entity_type": "minis test failure", "entity_name": "VLAN

[501] " } } }, "sort": 1731661921671 }

[0116] The following describes an example of an ARP action on VLAN 19: ARP Action on VLAN 19 { "_index": "entity_suggestion_202411", ​​"_id": "47fc80c3-77ea-44ab-aefa-bff44c689fbb&arp_failure&8c76a523-5aee-4618-a73e-e2d8799ef38d&vlan=19&arp&1731057553161", "_score": null, "_source": { "row_key": "47fc80c3-77ea-44ab-aefa-bff44c689fbb&arp_failure&8c76a523-5aee-4618-a73e-e2d8799ef38d&vlan=19&arp&1731057553161", "org_id": "47fc80c3-77ea-44ab-aefa-bff44c689fbb", "site_id": "8c76a523-5aee-4618-a73e-e2d8799ef38d", "entity_type": "site", "entity_id": "8c76a523-5aee-4618-a73e-e2d8799ef38d&vlan=19&arp", "unique_key": "8c76a523-5aee-4618-a73e-e2d8799ef38d&vlan=19&arp", "symptom": "arp_failure", "suggestion": "check ARP Failure", "display_name": "ARP_FAILURE", "category": "connectivity", "status": "open", "start_time": 1731057553161, "end_time": 1731662336532, "suggestion_time": 1731057585225, "NMS_only": false, "enable_notification": false, "impact_scope": "site", "severity": 60, "duration": 604783, "batch_count": 55, "modification_time": 1731662386488, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "2aa0ef4b-db7b-f29b-9d48-720b347f3802", "test_type": "connectivity", "who": "minis", "impacted_vlans": "19" , "failed_servers": "192.168.19.1" , "impacted_tuple": { "entity_type": "minis test failure", "entity_name": "VLAN

[19] " } } }, "sort": 1731662336532 }

[0117] The following describes an example of DNS action on VLAN 900: DNS Action on VLAN 900 { ​​"_index": "entity_suggestion_202411", "_id": "03227f1b-61cc-48a2-90b6-570852676b39&dns_failure&d1bad7cb-3745-41f1-b68c-a7e6b4bef780&vlan=900&dns&1731035405281", "_score": null, "_source": { "row_key": "03227f1b-61cc-48a2-90b6-570852676b39&dns_failure&d1bad7cb-3745-41f1-b68c-a7e6b4bef780&vlan=900&dns&1731035405281", "org_id": "03227f1b-61cc-48a2-90b6-570852676b39", "site_id": "d1bad7cb-3745-41f1-b68c-a7e6b4bef780", "entity_type": "site", "entity_id": "d1bad7cb-3745-41f1-b68c-a7e6b4bef780&vlan=900&dns", "unique_key": "d1bad7cb-3745-41f1-b68c-a7e6b4bef780&vlan=900&dns", "symptom": "dns_failure", "suggestion": "check DNS Failure", "display_name": "DNS_FAILURE", "category": "connectivity", "status": "open", "start_time": 1731035405281, "end_time": 1731662641791, "suggestion_time": 1731035744647, "NMS_only": false, "enable_notification": false, "impact_scope": "site", "severity": 60, "duration": 627236, "batch_count": 648, "modification_time": 1731662686034, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "d3e0c0f6-0d94-4494-955f-b4e1ad39a041", "test_type": "connectivity", "who": "minis", "impacted_vlans": [ "900" ], "failed_servers": [ "10.202.7.22", "10.4.196.41", "172.21.216.41" ], "failed_urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com", "https: / / captive.foo.com" ], "impacted_tuple": [ { "entity_type": "minis test failure", "entity_name": "VLAN

[900] " } ] } }, "sort": [ 1731662641791 ] }

[0118] For synthetic tests of cURL data transfer, the cURL payload and UI state are shown in application-related actions because the category of application action tests is a subset of cURL connectivity actions.

[0119] For application-related tests, especially CURL tests, when a test fails on all APs across all VLANs, the synthetic test module 352 triggers a remediation action. The synthetic test module 352 generates an action for each test URL. Because CURL testing is involved and failure on all VLANs is required, the synthetic test module 352 triggers actions at both the site and URL levels. An example of the action payload for the application action is described below.

[0120] The following are examples of application reachability actions on VLANs 30 and 40: Application reachability actions on VLANs 30 and 40 { "_index": "entity_suggestion_202411", "_id": "872b277a-5535-49ce-97f6-6b5bf1775ace&reachability_failure&eef68fde-ae67-4478-ad21-ce5bacfbcbfd&https: / / prtg.goriv.co / &1731037205285", "_score": null, "_source": { "row_key": "872b277a-5535-49ce-97f6-6b5bf1775ace&reachability_failure&eef68fde-ae67-4478-ad21-ce5bacfbcbfd&https: / / prtg.goriv.co / &1731037205285", "org_id": "872b277a-5535-49ce-97f6-6b5bf1775ace", "site_id": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd", "entity_type": "application", "entity_id": "https: / / prtg.goriv.co / ", "unique_key": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd&https: / / prtg.goriv.co / ", "symptom": "reachability_failure", "suggestion": "check_url_connectivity", "display_name": "APPLICATION_REACHABILITY_FAILURE", "category": "application", "status": "open", "start_time": 1731037205285, "end_time": 1731660841378, "suggestion_time": 1731037485560, "NMS_only": true, "enable_notification": false, "impact_scope": "application", "severity": 60, "duration": 623636, "batch_count": 573, "modification_time": 1731660886589, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "4dc7c0a8-0547-42c7-a5dc-07473f2dfe0c", "test_type": "connectivity", "who": "minis", "impacted_vlans": "40", "30" , "impacted_ap_count": 1, "failure_reason": "empty_response", "impacted_tuple": { "entity_type": "minis test failure", "entity_name": "VLAN [40, 30]" } } , "sort": 1731660841378 }

[0121] The following is an example of a cURL action on VLAN 30: Curl on VLAN 40 { "_index": "entity_suggestion_202411", ​​"_id": "872b277a-5535-49ce-97f6-6b5bf1775ace&curl_failure&eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=30&curl&1731037205285", "_score": null, "_source": { "row_key": "872b277a-5535-49ce-97f6-6b5bf1775ace&curl_failure&eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=30&curl&1731037205285", "org_id": "872b277a-5535-49ce-97f6-6b5bf1775ace", "site_id": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd", "entity_type": "site", "entity_id": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=30&curl", "unique_key": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=30&curl", "symptom": "curl_failure", "suggestion": "check CURL Failure", "display_name": "CURL_FAILURE", "category": "connectivity", "status": "open", "start_time": 1731037205285, "end_time": 1731660841378, "suggestion_time": 1731037485203, "NMS_only": true, "enable_notification": false, "impact_scope": "site", "severity": 60, "duration": 623636, "batch_count": 573, "modification_time": 1731660886092, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "4dc7c0a8-0547-42c7-a5dc-07473f2dfe0c", "test_type": "connectivity", "who": "minis", "impacted_vlans": [ "30" ], "failed_urls": [ "https: / / prtg.goriv.co / ", "apps.goriv.co", "jfrog.goriv.co", "https: / / ipam.site.goriv.co / index.html", "http: / / moab.goriv.co / ", "https: / / anka-controller.tools.uw2.dc.goriv.co / " ], "impacted_tuple": [ { "entity_type": "minis test failure", "entity_name": "VLAN

[30] " } ]} }, "sort": [ 1731660841378 ] }

[0122] The following is another example of cURL actions on VLAN 40: Curl on VLAN 40 { "_index": "entity_suggestion_202411", "_id": "872b277a-5535-49ce-97f6-6b5bf1775ace&curl_failure&eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=40&curl&1731037205285", "_score": null, "_source": { "row_key": "872b277a-5535-49ce-97f6-6b5bf1775ace&curl_failure&eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=40&curl&1731037205285", "org_id": "872b277a-5535-49ce-97f6-6b5bf1775ace", "site_id": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd", "entity_type": "site", "entity_id": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=40&curl", "unique_key": "eef68fde-ae67-4478-ad21-ce5bacfbcbfd&vlan=40&curl", "symptom": "curl_failure", "suggestion": "check CURL Failure", "display_name": "CURL_FAILURE", "category": "connectivity", "status": "open", "start_time": 1731037205285, "end_time": 1731660841378, "suggestion_time": 1731037485421, "NMS_only": true, "enable_notification": false, "impact_scope": "site", "severity": 60, "duration": 623636, "batch_count": 669, "modification_time": 1731660886525, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "4dc7c0a8-0547-42c7-a5dc-07473f2dfe0c", "test_type": "connectivity", "who": "minis", "impacted_vlans": [ "40" ], "failed_urls": [ "10.176.8.49", "10.182.8.49" ], "impacted_tuple": [ { "entity_type": "minis test failure", "entity_name": "VLAN

[40] " } ] } }, "sort": [ 1731660841378 ] }

[0123] The following is another example of application reachability actions on VLAN 1012: Application reachability actions on VLAN 1012 { "_index": "entity_suggestion_202410", "_id": "793de352-344c-4410-9841-97346c7890c7&reachability_failure&de1e2518-fae0-439a-b351-3b8600c4dba3&https: / / connectivitycheck.gstatic.com / generate_204&1728354516871", "_score": null, "_source": { "row_key": "793de352-344c-4410-9841-97346c7890c7&reachability_failure&de1e2518-fae0-439a-b351-3b8600c4dba3&https: / / connectivitycheck.gstatic.com / generate_204&1728354516871", "org_id": "793de352-344c-4410-9841-97346c7890c7", "site_id": "de1e2518-fae0-439a-b351-3b8600c4dba3", "entity_type": "application", "entity_id": "https: / / connectivitycheck.gstatic.com / generate_204", "unique_key": "de1e2518-fae0-439a-b351-3b8600c4dba3&https: / / connectivitycheck.gstatic.com / generate_204", "symptom": "reachability_failure", "suggestion": "check_url_connectivity", "display_name": "APPLICATION_REACHABILITY_FAILURE", "category": "application", "status": "open", "start_time": 1728354516871, "end_time": 1731655797491, "suggestion_time": 1728354632539, "NMS_only": true, "enable_notification": false, "impact_scope": "application", "severity": 60, "duration": 3301280, "batch_count": 1245, "modification_time": 1731655846219, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "063cbf02-4a32-4575-a4aa-08b67c8eb5cf", "test_type": "connectivity", "who": "minis", "impacted_vlans": [ "1012" ], "impacted_ap_count": 2, "failure_reason": "empty_response", "impacted_tuple": [ { "entity_type": "minis test failure", "entity_name": "VLAN

[1012] " } ] } }, "sort": [ 1731655797491 ] }

[0124] The following describes an example of the corresponding cURL action on VLAN 1012. Curl on VLAN 1012 { "_index": "entity_suggestion_202411", "_id": "793de352-344c-4410-9841-97346c7890c7&curl_failure&de1e2518-fae0-439a-b351-3b8600c4dba3&vlan=1012&curl&1731036365284", "_score": null, "_source": { "row_key": "793de352-344c-4410-9841-97346c7890c7&curl_failure&de1e2518-fae0-439a-b351-3b8600c4dba3&vlan=1012&curl&1731036365284", "org_id": "793de352-344c-4410-9841-97346c7890c7", "site_id": "de1e2518-fae0-439a-b351-3b8600c4dba3", "entity_type": "site", "entity_id": "de1e2518-fae0-439a-b351-3b8600c4dba3&vlan=1012&curl", "unique_key": "de1e2518-fae0-439a-b351-3b8600c4dba3&vlan=1012&curl", "symptom": "curl_failure", "suggestion": "check CURL Failure", "display_name": "CURL_FAILURE", "category": "connectivity", "status": "open", "start_time": 1731036365284, "end_time": 1731659221338, "suggestion_time": 1731036525411, "NMS_only": true, "enable_notification": false, "impact_scope": "site", "severity": 60, "duration": 622856, "batch_count": 478, "modification_time": 1731659268751, "snooze_expire_time": 0, "pipeline_type": "STREAMING", "details": { "impacted_entity_count": 1, "test_id": "6f21eaab-18cb-4dbe-a765-8286d2071159", "test_type": "connectivity", "who": "minis", "impacted_vlans": [ "1012" ], "failed_servers": [ "13.107.6.156", "17.253.23.201", "142.251.211.227", "52.123.129.14" ], "failed_urls": [ "https: / / squad.bar.com", "https: / / connectivitycheck.gstatic.com / generate_204", "https: / / business.com", "https: / / captive.foo.com" ], "impacted_tuple": [ { "entity_type": "minis test failure", "entity_name": "VLAN

[1012] " } ] } }, "sort": [ 1731659221338 ] }

[0125] Figure 4 An example user equipment (UE) device 400 according to one or more technologies of this disclosure is shown. Figure 4 The example UE device 400 shown can be used to implement about Figure 1A Any UE in UE 148 shown and described. UE device 400 may include any type of wireless client device, and this disclosure is not limited thereto. For example, UE device 400 may include mobile devices such as smartphones, tablets or laptops, personal digital assistants (PDAs), wireless terminals, smartwatches, smart rings, or any other type of mobile or wearable device. In some examples, UE 400 may also include wired client-side devices, such as IoT devices (such as printers, security sensors or devices, environmental sensors) or any other device connected to a wired network and configured to communicate over one or more wireless networks.

[0126] UE device 400 includes a wired interface 430, wireless interfaces 420A to 420C, one or more processors 406, memory 412, and user interface 410. Various components are coupled together via bus 414, through which they can exchange data and information. The wired interface 430 represents a physical network interface and includes a receiver 432 and a transmitter 434. If desired, the wired interface 430 can be used to couple the UE 400 directly or indirectly to wired network devices (such as Ethernet devices) within a wired network via a cable (such as an Ethernet cable). Figure 1A (One of the switches in switch 146).

[0127] The first, second, and third wireless interfaces 420A, 420B, and 420C respectively include receivers 422A, 422B, and 422C, each receiver including a receiving antenna, through which the UE 400 can receive signals from wireless communication devices (such as…) Figure 1A AP 142, Figure 2 The AP 200, other UEs 148, or other devices configured for wireless communication receive wireless signals. The first, second, and third wireless interfaces 420A, 420B, and 420C further include transmitters 424A, 424B, and 424C, respectively, each including a transmitting antenna, through which the UE 400 can transmit signals to wireless communication devices (such as AP 200, other UEs 148, or other devices configured for wireless communication). Figure 1A AP 142, Figure 2The AP 200, other UEs 148, and / or other devices configured for wireless communication transmit wireless signals. In some examples, the first wireless interface 420A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz), and the second wireless interface 420B may include a Bluetooth interface and / or a Bluetooth Low Energy interface. The third wireless interface 420C may include, for example, a cellular interface through which the UE device 400 can connect to a cellular network.

[0128] Multiple processors 406 execute software instructions, such as instructions used to define software or computer programs, which are stored on a computer-readable storage medium (such as memory 412), such as a non-transitory computer-readable medium, including storage devices (e.g., disk drives or optical disk drives) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions to cause one or more processors 406 to perform the techniques described herein.

[0129] Memory 412 includes one or more devices configured to store programming modules and / or data associated with the operation of UE 400. For example, memory 412 may include computer-readable storage media, such as non-transitory computer-readable media, including storage devices (e.g., disk drives or optical disk drives) or memories (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions to cause one or more processors 406 to perform the techniques described herein.

[0130] In this example, memory 412 includes operating system 440, application 442, communication module 444, configuration settings 450, and data storage device 454. Communication module 444 includes program code that, when executed by processor(s) 406, enables UE 400 to communicate using any of the multiple wired interfaces 430, wireless interfaces 420A to 420B, and / or cellular interface 420C. For each of the multiple wireless interfaces 420A to 420B and / or cellular interface 420C, configuration settings 450 includes any device settings configured for UE 400.

[0131] Data storage device 454 may include, for example, a status / error log, which includes a list of events specific to UE 400. Events may include logs of both normal and error events at the log recording level based on instructions from NMS 130. Data storage device 454 may store any data used and / or generated by UE 400, such as data used to calculate one or more SLE metrics or identify related behavioral data, which is collected by UE 400 and directly transmitted to NMS 130 or to any AP in AP 142 of wireless network 106 for further transmission to NMS 130.

[0132] As described herein, UE 400 can measure network data and report network data from data storage device 454 to NMS 130. Network data may include event data, telemetry data, and / or other SLE-related data. Network data may include various parameters indicating the performance and / or status of the wireless network. NMS 130 can determine one or more SLE metrics based on SLE-related data received from the UE or client equipment in the wireless network and store the SLE metrics as network data 137. Figure 1A ).

[0133] Optionally, the UE device 400 may include an NMS agent 456. The NMS agent 456 is a software agent of the NMS 130 installed on the UE 400. In some examples, the NMS agent 456 may be implemented as a software application running on the UE 400. The NMS agent 456 collects information from the UE 400, including detailed client device characteristics, including insights into the UE 400's roaming behavior. This information provides insights into client roaming algorithms, as roaming is a decision made by the client device. In some examples, the NMS agent 456 may display client device attributes on the UE 400. The NMS agent 456 sends client device characteristics to the NMS 130 via an AP device to which the UE 400 is connected. The NMS agent 456 may be integrated into a custom application or as part of a location application. The NMS agent 456 may be configured to identify the device connection type (e.g., cellular or Wi-Fi) along with the corresponding signal strength. For example, the NMS agent 456 identifies the access point connection and its corresponding signal strength. NMS agent 456 can store information specifying the APs identified by UE 400 and their corresponding signal strengths. UE 400's NMS agent 456 or other components also collect information about which APs UE 400 is connected to, indicating which APs UE 400 is not connected to. UE 400's NMS agent 456 sends this information to NMS 130 via the APs it is connected to. In this way, UE 400 sends information about the APs it is connected to, as well as information about other APs identified by UE 400 but not connected to, and their signal strengths. The APs then forward this information to the NMS, including information about other APs identified by UE 400 (besides themselves). This additional level of granularity allows NMS 130, and ultimately the network administrator, to better determine the Wi-Fi experience directly from the client device's perspective.

[0134] In some examples, NMS Agent 456 further enriches the client device data utilized in the service level. For instance, NMS Agent 456 can go beyond basic fingerprinting to provide supplementary details into the features, such as device type, manufacturer, and different versions of the operating system. Within the detailed client features, NMS 130 can display radio hardware and firmware information of the UE 400 received from NMS Client Agent 456. The more details NMS Agent 456 can extract, the better the VNA / AI engine becomes in advanced device classification. NMS 130's VNA / AI engine continuously learns and becomes more accurate in its ability to distinguish between device-specific issues and broader device issues, such as specifically identifying which OS version is affecting certain clients.

[0135] In some examples, NMS agent 456 may display a prompt on user interface 410, instructing the end user of UE 400 to enable location permissions before NMS agent 456 can report device location, client information, and network connectivity data to NMS. NMS agent 456 will then begin reporting connectivity data along with location data to NMS. In this way, the end user of the client device can control whether NMS agent 456 is enabled to report client device information to NMS.

[0136] According to the technology described in this disclosure, UE device 400 includes a test engine 446 configured to perform one or more network connectivity tests received from a synthetic test module (e.g., synthetic test module 135 of FIG. 1). For example, UE device 400 may receive a message from synthetic test module 135 specifying the address (e.g., MAC address) of the selected UE device and a payload specifying test information to be performed by the selected UE device. In response to determining that the message includes the address of the UE device 400, UE device 400 may extract test information (e.g., commands) from the message payload and provide the test information to test engine 446. Test engine 446 can then execute the commands received from synthetic test module 135.

[0137] As part of the command execution, test engine 446 can receive and / or generate test results. Test engine 446 can store the test results in memory 412 as test result data 448. Test engine 446 can send test result data 448 to synthetic test module 135 for further analysis, as further described below.

[0138] Figure 5 This is a block diagram of an example network node 500 according to one or more techniques of this disclosure. In one or more examples, the network node 500 is implemented to be attached to Figure 1A Network devices or servers 134, such as switch 146, AAA server 110 or other NAC server or system, DHCP server 116, DNS server 122, web server 128, etc., or supporting Figure 1B Another network device, such as router 187, of one or more of the following: wireless network 106, wired LAN 175, SD-WAN 177, or data center 179.

[0139] In this example, network node 500 includes a wired interface 502 (e.g., an Ethernet interface), a processor 506, input / output 508 (e.g., a display, buttons, keyboard, keypad, touchscreen, mouse, etc.), and memory 512, all coupled together via a bus 514 through which various components exchange data and information. The wired interface 502 couples network node 500 to a network, such as an enterprise network. Although only one interface is shown in this example, a network node can and typically has multiple communication interfaces and / or multiple communication interface ports. The wired interface 502 includes a receiver 520 and a transmitter 522.

[0140] Memory 512 stores executable software application 532, operating system 540, and data repository 530. Data / information 530 may include system logs and / or error logs, which store event data, including behavioral data, for network node 500. In the example where network node 500 includes a "third-party" network device, the same entity does not own or have access to both the AP or wired client-side device and network node 500. Therefore, in the example where network node 500 is a third-party network device, NMS 130 does not receive, collect, or otherwise access network data from network node 500.

[0141] In the example where network node 500 includes a server, network node 500 can receive data and information via receiver 520, such as operation-related information, such as registration requests, AAA services, DHCP requests, Simple Notification Service (SNS) lookups, and web page requests, and send data and information via transmitter 522, such as configuration information, authentication information, web page data, etc.

[0142] In an example where network node 500 includes wired network devices, network node 500 can be connected to one or more access points (APs) or other wired client-side devices, such as IoT devices, via wired interface 502. For example, network node 500 may include multiple wired interfaces 502 and / or wired interfaces 502 may include multiple physical ports for connection to multiple APs or other wired client-side devices within the site via appropriate Ethernet cables. In some examples, each AP or other wired client-side device connected to network node 500 can access the wired network via wired interface 502 of network node 500. In some examples, one or more APs or other wired client-side devices connected to network node 500 can each draw power from network node 500 via appropriate Ethernet cables and the Power over Ethernet (PoE) port of wired interface 502.

[0143] In examples where network node 500 includes a session-based router employing a stateful, session-based routing scheme, network node 500 can be configured to perform path selection and traffic engineering independently. Using session-based routing allows network node 500 to avoid using a centralized controller (such as an SDN controller) to perform path selection and traffic engineering, and to avoid using tunnels. In some examples, network node 500 can implement session-based routing as a Security Vector Router (SVR), provided by Juniper Networks. In cases where network node 500 includes a session-based router operating as a network gateway for sites targeting enterprise networks (e.g., Figure 1B Router 187A), network node 500 can interact with one or more other session-based routers (e.g., routers that function as network gateways for other sites on the enterprise network) Figure 1B The router 187B) is located at the underlying physical WAN (e.g., Figure 1B Establish multiple peering paths on SD-WAN 177 (e.g., Figure 1B (Logical path 189). Network node 500, operating as a session-based router, can collect data at the peer path level and report peer path data to NMS 130.

[0144] In the example where network node 500 includes a packet-based router, network node 500 can employ packet-based or flow-based routing schemes to forward packets according to defined network paths (e.g., established by a centralized controller that performs path selection and traffic engineering). In the case where network node 500 includes a packet-based router operating as a network gateway for a site targeting an enterprise network (e.g., ...), Figure 1B Router 187A), network node 500 can interact with one or more other packet-based routers (e.g., routers that function as network gateways for other sites on the enterprise network) Figure 1B The router 187B) is located at the underlying physical WAN (e.g., Figure 1B Establish multiple tunnels over SD-WAN 177 (e.g., Figure 1B (Logical path 189). Network node 500, operating as a packet-based router, can collect data at the tunnel level, and the tunnel data can be retrieved by NMS 130 via API or open configuration protocol, or the tunnel data can be reported to NMS 130 by NMS agent 544 or another application or agent running on network node 500.

[0145] The data collected and reported by network node 500 may include periodically reported data and event-driven data. Network node 500 is configured to collect logical path statistics via bidirectional forwarding detection (BFD) probes and data extracted from messages and / or counters at the logical path level (e.g., peer-to-peer paths or tunnels). In some examples, network node 500 is configured to collect statistics and / or sample other data according to a first periodic interval (e.g., every 3 seconds, every 5 seconds, etc.). Network node 500 may store the collected and sampled data as path data, for example, in a buffer.

[0146] In some examples, network node 500 may optionally include NMS agent 544. NMS agent 544 may periodically create packets of statistics at a second periodic interval (e.g., every 3 minutes). The collected and sampled data periodically reported in the packets of statistics may be referred to herein as “oc-stats”. In some examples, the packets of statistics may also include details of clients connected to network node 500 and associated client sessions. NMS agent 544 may then report the packets of statistics to NMS 130 in the cloud. In other examples, NMS 130 may request, retrieve, or otherwise receive packets of statistics from network node 500 via API, open configuration protocols, or other communication protocols. The packets of statistics created by NMS agent 544 or another module of network node 500 may include headers identifying network node 500 and statistics and data samples from each logical path in the logical path targeting network node 500. In other examples, in response to the occurrence of certain events at network node 500, NMS agent 544 reports event data to NMS 130 in the cloud. Event-driven data can be referred to as "oc-events" in this article.

[0147] According to the technology described in this disclosure, network node 500 includes test engine 534, which is configured to perform tests from synthetic test modules (e.g., Figure 1A The network node 500 receives one or more network connectivity tests from the synthetic test module 135. For example, the network node 500 may receive a message from the synthetic test module 135 specifying the address (e.g., MAC address) of the selected network node and a payload specifying the test information to be performed by the selected network node. In response to determining that the message includes the address of the specified network node 500, the network node 500 may extract the test information (e.g., commands) from the payload of the message and provide the test information to the test engine 534. The test engine 534 can then execute the commands received from the synthetic test module 135.

[0148] As part of the command execution, the test engine 534 can receive and / or generate test results. The test engine 534 can store the test results in memory 512 as test result data 536. The test engine 534 can send the test result data 536 to the synthetic test module 135 for further analysis, as further described below.

[0149] Figure 6 The illustration shows an example arrangement of synthetic tests according to one or more techniques of this disclosure. Figure 3A The NMS 300's synthesis test module 352 can perform... Figure 6 The arrangement of the synthetic tests described in the text.

[0150] In this example, the synthetic test time window module 382, ​​synthetic test range module 384, WLAN / VLAN mapping module 386, and network connectivity test 353 of the synthetic test module 352 can be executed on an analysis engine (e.g., Apache Spark). The module output data of the synthetic test module 352 can be stored in cloud storage devices 604A to 604N (collectively, "cloud storage device 604"). For example, the synthetic test time window determined by the synthetic test time window module 382 can be stored in cloud storage device 604. Similarly, the synthetic test range determined by the synthetic test range module 384 can be stored in cloud storage device 604. Likewise, the WLAN / VLAN mapping provided by the WLAN / VLAN mapping module 386 can be stored in cloud storage device 604. Test service 606 (e.g., Kafka minion) can push specific tests selected by network connectivity test 353 and stored in a data repository (e.g., test data 608) to data pipeline 610 (e.g., Kafka data pipeline) based on synthetic test time windows and / or synthetic test ranges. This data 606 is then streamed to test export service 612 (e.g., a cloud service represented by nodes within a Kafka topology graph), which can then push commands to specific devices, such as one or more APs 142 in this example. Although Figure 6 The illustration shows an example of pushing commands to one or more APs 142, but the synthetic test module 352 can push commands to other devices, such as client devices, network devices, and / or servers, to perform specific tests, causing a specific device to perform a specific test. In this example, the test export service 612 can generate a message including the address (e.g., MAC address) of one or more APs 142 and a payload including a command specifying the specific test to be performed by one or more APs 142. The test export service 612 then sends the message including the command specifying the specific test to one or more APs 142.

[0151] Figure 7 The illustration shows an example of obtaining test result data from a specific network device that has performed a synthetic test, according to one or more techniques of this disclosure.

[0152] In this example, one or more APs 142 perform specific tests, such as network connectivity tests, and send data generated by the execution of these tests (“test result data”). For example, test import service 702 can obtain the test result data and push it to data pipeline 704 (e.g., a Kafka data pipeline), which in turn stores the test result data in cloud storage device 706 and sends it to test event service 708 (e.g., by...). Figure 3B (Executed by action module 392), test event service 708 can identify one or more problems based on test result data and / or send test result data to SLE module 322 to analyze the test result data and determine one or more SLE metrics based on the test result data.

[0153] Figure 8 This is a flowchart illustrating an example operation of a network management system configured to perform automated scheduling and orchestration of network connectivity tests against a network site, according to one or more technologies of this disclosure. More specifically, Figure 8 The description describes the operation of a synthetic test module 135 combined with a network connectivity test 136 to perform downloadable, automated tests for network connectivity and application connectivity performed by network devices (such as AP 142 of site 102). Figure 8 about Figure 1A and Figure 1B NMS 130 and Figure 3A and Figure 3B The synthesis test module 352 of the NMS 300 is described.

[0154] For example, the synthetic test module 135 of the NMS 130 receives a trigger action to perform a network connectivity test. In some examples, the trigger action performs the network connectivity test in response to a user input request. In some examples, the NMS 130 issues the trigger action on a scheduled or periodic basis, such as hourly, daily, weekly, etc. In response to the trigger action, the network connectivity test 136 of the synthetic test module 135 instructs or causes one or more APs 142 to perform a network connectivity test (802). In some examples, the network connectivity test includes, for example, synthetic tests of network connectivity performed by a Domain Name Server (DNS), Dynamic Host Configuration Protocol (DHCP) server, Address Resolution Protocol (ARP) requests, cURL data transfers, network reachability to another network device, or applications performed by a network device.

[0155] The network connectivity test 136 of the synthetic test module 135 obtains data from one or more APs 142 that have performed network connectivity tests. Based on the obtained data, the network connectivity test 136 determines the failure of the network connectivity test for one of the APs 142 (804). In some examples, the VNA 133 applies a machine learning system to the data to determine the failure of the network connectivity test for the AP 142.

[0156] Based on the type of network connectivity test that AP 142 failed, VNA 133 selects an action (806) to fix the root cause of the test failure. VNA 133 executes the selected action (808). In some examples, to execute the selected action, VNA 133 generates a notification to the administrator indicating the AP 142 that failed the test and the type of network connectivity test that failed. The notification may optionally include recommended configuration changes to fix the network connectivity problem of AP 142. In some examples, to execute the selected action, VNA 133 selects a remedial action and executes the remedial action, such as restarting or rebooting one of APs in AP 142, adjusting the configuration of one of APs in AP 142, or reinstalling or restarting an application or service performed by one of APs in AP 142.

[0157] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units, or components can be implemented together in an integrated logic device, or implemented separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit system can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0158] If implemented in hardware, this disclosure may refer to means such as a processor or an integrated circuit device (such as an integrated circuit chip or chipset). Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0159] Computer-readable media can form part of a computer program product, which may include packaging materials. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0160] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" can indicate that the storage medium is not embodied in a carrier or propagating signal. In some examples, non-transitory storage media may store data that can change over time (e.g., in RAM or cache).

[0161] The code or instructions can be software and / or firmware executed by a processing circuitry system including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry systems. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure can be provided within a software module or a hardware module.

Claims

1. A computing system, the computing system comprising a processing circuitry capable of accessing memory, the processing circuitry being configured to: Instruct one or more network devices to perform tests on network connectivity; The failure of the network connectivity test for one of the one or more network devices is determined, at least in part, based on data obtained from the one or more network devices that performed the network connectivity test. At least in part, based on the type of network connectivity test, select actions to address the root cause of the failure in the network connectivity test of the network device; as well as Perform the selected action.

2. The computing system according to claim 1, wherein, To determine the failure of the network connectivity test of the network device, the processing circuitry is configured to apply a machine learning system to the data obtained from the one or more network devices that performed the network connectivity test, in order to determine the failure of the network device in the network connectivity test. The machine learning system is trained using historical data obtained from the one or more network devices that performed the network connectivity test.

3. The computing system according to claim 1, wherein, In order to perform the selected action, the processing circuitry is configured to issue a notification to the messaging service, the notification indicating: The network device that failed the network connectivity test; as well as At least one of the following: the network device is configured with a Virtual Local Area Network (VLAN), or the site to which the network device belongs.

4. The computing system according to claim 1, wherein, To instruct one or more network devices to perform the network connectivity test, the processing circuitry is configured to: Receive a trigger message from the messaging service, the trigger message specifying the one or more network devices and one or more virtual local area networks (VLANs) configured on the one or more network devices; as well as The trigger message instructs the one or more network devices specified therein, with respect to the one or more VLANs configured on the one or more network devices, to perform the network connectivity test.

5. The computing system according to claim 1, wherein, In order to instruct the one or more network devices to perform the network connectivity test, the processing circuitry is configured to iteratively instruct one or more network devices configured with the corresponding VLAN to perform the network connectivity test for each of a plurality of virtual local area network VLANs.

6. The computing system according to claim 1, in, To determine the failure of the network connectivity test for one or more of the network devices, the processing circuitry is configured to determine that the network connectivity test for each of a plurality of network devices configured with a Public Virtual Local Area Network (VLAN) has failed, and In order to perform the selected action, the processing circuitry is configured to perform the selected action on the plurality of network devices configured with the public VLAN.

7. The computing system of claim 1, wherein the one or more network devices include one or more access points (APs).

8. The computing system according to any one of claims 1 to 6, The network connectivity tests mentioned above include tests for resolving Uniform Resource Locator (URL) URLs using Domain Name Servers (DNS). in, To determine the failure of the network connectivity test for the network device, the processing circuitry is configured to determine the failure of the network connectivity test based at least in part on the failure of the test to resolve the URL using DNS for at least one VLAN configured on the network device among a plurality of virtual LAN VLANs.

9. The computing system according to any one of claims 1 to 6, The network connectivity tests mentioned above include tests for network connectivity with a Dynamic Host Configuration Protocol (DHCP) server, and in, To determine the failure of the network connectivity test for the network device, the processing circuitry is configured to determine the failure of the network connectivity test based at least in part on the failure of the test for network connectivity with the DHCP server for at least one VLAN configured on the network device among a plurality of virtual LAN VLANs.

10. The computing system according to any one of claims 1 to 6, The network connectivity tests mentioned above include tests for Address Resolution Protocol (ARP) requests, and in, To determine the failure of the network connectivity test for the network device, the processing circuitry is configured to determine the failure of the network connectivity test based at least in part on the failure of the test for the ARP request of at least one VLAN configured on the network device among a plurality of virtual LAN VLANs.

11. The computing system according to any one of claims 1 to 6, The network connectivity tests mentioned above include cURL data transmission tests, and in, To determine the failure of the network connectivity test for the network device, the processing circuitry is configured to determine the failure of the network connectivity test based at least in part on the failure of the cURL data transmission test for at least one VLAN configured on the network device among a plurality of virtual LAN VLANs.

12. The computing system according to any one of claims 1 to 6, The network connectivity test includes a test of network reachability between the network device and the second network device, and in, To determine the failure of the network connectivity test for the network device, the processing circuitry is configured to determine the failure of the network connectivity test based at least in part on the failure of the test for network reachability between the network device and the second network device for at least one VLAN configured on the network device in a plurality of virtual LAN VLANs.

13. The computing system according to any one of claims 1 to 6, The network connectivity test includes a test of the network connectivity of an application executed by the network device, and in, To determine the failure of the network connectivity test for the network device, the processing circuitry is configured to determine the failure of the network connectivity test based at least in part on the failure of the network connectivity application configured for each VLAN of the network device in a plurality of virtual local area network VLANs.

14. A computer network method, comprising: The processing circuitry of the computing system instructs one or more network devices to perform tests on network connectivity. The processing circuitry system determines, at least in part, the failure of the network connectivity test for one of the one or more network devices based on data obtained from the data obtained from the network devices that performed the network connectivity test; The processing circuitry system selects actions, at least in part, based on the type of network connectivity test, to address the root cause of the failure in the network connectivity test of the network device. as well as The selected action is performed by the processing circuit system.

15. The computer network method of claim 14, wherein determining the failure of the network device in the network connectivity test comprises: The machine learning system is applied to the data obtained from the one or more network devices that performed the network connectivity test to determine the failure of the network device in the network connectivity test. The machine learning system is trained using historical data obtained from the one or more network devices that performed the network connectivity test.

16. The computer network method of claim 14, wherein performing the selected action includes publishing a notification to a messaging service, the notification instructing: The network device that failed the network connectivity test; and At least one of the following: the network device is configured with a Virtual Local Area Network (VLAN), or the site to which the network device belongs.

17. The computer network method of claim 14, wherein instructing the one or more network devices to perform the network connectivity test comprises: Receive a trigger message from the messaging service, the trigger message specifying the one or more network devices and one or more virtual local area networks (VLANs) configured on the one or more network devices; as well as The trigger message instructs the one or more network devices specified therein, with respect to the one or more VLANs configured on the one or more network devices, to perform the network connectivity test.

18. The computer network method of claim 14, wherein instructing the one or more network devices to perform the network connectivity test comprises: Iteratively instructs one or more network devices configured with the corresponding VLAN to perform the network connectivity test for each of the multiple virtual LAN VLANs.

19. The computer network method according to any one of claims 14 to 18, The determination of the failure of the network connectivity test for one or more network devices includes determining the failure of the network connectivity test for each of a plurality of network devices configured with a public virtual local area network (VLAN), and The action of performing the selection includes performing the selection action on the plurality of network devices configured with the public VLAN.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to be configured to perform a computing system according to any one of claims 1 to 13, or to perform a method according to any one of claims 14 to 19.

Images