Detecting degradation in client connectivity and traffic in wireless access point

Abstract

Embodiments of the present disclosure detect degradation in client connectivity and traffic in a wireless access point. Example network management systems, techniques, and computer readable media are described herein. An example network management system is configured to obtain client activity data for one or more access point (AP) devices. The example network management system is configured to determine an anomaly in behavior of one of the one or more AP devices based on the client activity data, based on at least one of a comparison with forecasted client activity data of the AP devices, or a comparison with client activity data of the at least one peer AP device. The network management system is configured to perform an action based on the determined anomaly in the behavior of the AP device.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 486,606, filed on October 13, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to computer networks and, more particularly, to monitoring and troubleshooting wireless access points of a computer network. Background Art

[0004] Commercial venues or sites, such as offices, hospitals, airports, stadiums, or retail stores, often have complex wireless networking systems installed throughout the venue, including a network of wireless access points (APs) to provide wireless networking services to one or more wireless client devices (or simply "clients"). An AP is a physical electronic device that enables other devices to wirelessly connect to a wired network using a variety of wireless networking protocols and technologies, such as wireless local area networking protocols (i.e., "Wi-Fi") that conform to one or more IEEE 802.11 standards, Bluetooth / Bluetooth Low Energy (BLE), mesh networking protocols (such as ZigBee, 5G, etc.), or other wireless networking technologies. Many different types of wireless client devices, such as laptops, smartphones, tablets, wearables, home appliances, and Internet of Things (IoT) devices, incorporate wireless communication technology and can be configured to connect to a wireless access point when the device is within range of a compatible wireless access point in order to access the wired network. In the case where a client device runs a cloud-based application (such as a Voice over Internet Protocol (VOIP) application, a streaming video application, a gaming application, or a video conferencing application), data is exchanged from the client device through one or more APs and one or more wired network devices (e.g., switches, routers, and / or gateway devices) during an application session to reach a cloud-based application server. Summary of the invention

[0005] A wireless network may include one or more network devices (e.g., access point (AP) devices) that provide wireless network services to one or more client devices. As more AP models and firmware versions are deployed in a network, new and / or unknown issues with network experience may arise, such as unexpected loss of connectivity with client devices, inability of client devices to connect, low traffic or throughput, etc. The technology of the present disclosure relates to detection of the health (or lack of health) of AP devices based on client activity data.

[0006] The technology of the present disclosure provides active detection of anomalies in the behavior of AP devices based on client activity data. In some examples, the anomaly in the behavior of the AP device can be an anomaly in the behavior of the AP device's radio. A computing device, such as a network management system (NMS) that provides management of AP devices for one or more sites, may include an inclusive detection engine, a clustering engine, and an exclusive detection engine to actively detect anomalies in the behavior of AP devices based on client activity data and / or perform remedial actions to attempt to resolve any such anomalies. As used herein, a site may include a facility at a specific geographic location, rather than geographically distributed facilities or multiple facilities at different geographic locations.

[0007] In general, the present disclosure describes one or more techniques for a network management system (NMS) to proactively identify anomalies in the behavior of an AP and / or take one or more actions to attempt to correct such abnormal behavior of the AP. For example, the NMS may use one or more machine learning models to predict client activity data for the AP and compare the actual client activity data of the AP with the predicted client activity data of the AP. Additionally or alternatively, the NMS may compare the actual client activity data of the AP with the actual client activity data of a peer AP.

[0008] The client activity data may include, for example, the number of client devices connected to the AP device or the AP device's radio, the length of time each client device is connected to the AP device or the AP device's radio, the amount of client traffic sent by the AP device or the AP device's radio during a time period, and / or the amount of client traffic received by the AP device or the AP device's radio during the time period. In some examples, the NMS may generate one or more scores based on the client activity data and predicted client activity data and / or a comparison of the AP device's client activity data with the client activity data of a peer AP device. The NMS may determine an anomaly in the behavior of the AP device and take action to try to resolve the anomaly. If the action is unsuccessful, the NMS may take (multiple) additional actions to try to resolve the anomaly.

[0009] An example includes a network management system comprising: one or more memories; and one or more processors in communication with the one or more memories and configured to: obtain client activity data of one or more access point (AP) devices; determine an anomaly in the behavior of one of the one or more AP devices based on the client activity data, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of the following: a) determine predicted client activity data for the AP device based on the client activity data and compare the predicted client activity data with actual client activity data for the AP device, or b) compare client activity data of at least one peer AP device with actual client activity data for the AP device; and perform an action based on the determined anomaly in the behavior of the AP device.

[0010] Another example includes a method for managing one or more access points, the method comprising: obtaining client activity data of multiple access point (AP) devices at a site; determining an anomaly in the behavior of one of the multiple AP devices based on the client activity data, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of the following: a) determine predicted client behavior data for the AP device based on the client activity data and compare the predicted client activity data with actual client activity data for the AP device, or b) compare client activity data of at least one peer AP device with actual client activity data for the AP device; and perform an action based on the determined anomaly in the behavior of the AP device.

[0011] Another example includes a computer-readable storage medium comprising instructions that, when executed by a processing circuit system, cause the processing circuit system to: obtain client activity data for multiple access point (AP) devices at a site; determine an anomaly in behavior of an AP device among the multiple AP devices based on the client activity data, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of the following: a) determine predicted client activity data for the AP device based on the client activity data and compare the predicted client activity data with actual client activity data for the AP device, or b) compare client activity data for at least one peer AP device with actual client activity data for the AP device; and perform an action based on the determined anomaly in the behavior of the AP device.

[0012] The disclosed techniques may provide one or more technical advantages and practical applications. For example, by monitoring client activity data of an AP and comparing the activity data with at least one of predicted client activity data or client activity data of a peer AP, an NMS may proactively identify AP anomalies and take actions to resolve such anomalies. Compared to reactively identifying and resolving problems with APs in the network, these techniques may result in more timely resolution of such problems.

[0013] The details of one or more examples of the technology of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the technology will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an example network system in which a network management system identifies one or more contributing characteristics to a problem with an application session and invokes one or more actions to remediate or prevent the problem, in accordance with one or more techniques of this disclosure.

[0015] Figure 2 is a block diagram of an example access point device in accordance with one or more techniques of this disclosure.

[0016] Figure 3 is a block diagram of an example network management system configured to identify one or more contributing characteristics to a problem with an application session and to invoke one or more actions to remedy or prevent the problem, in accordance with one or more techniques of this disclosure.

[0017] Figure 4 is a flow diagram illustrating example operations for reactive problem determination in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0018] Figure 1 1 is a block diagram of an example network system 100 in which a network management system (NMS) 130 determines an anomaly with respect to an access point (AP) and invokes one or more actions to remedy the anomaly, according to one or more aspects of the present disclosure. The example network system 100 includes a plurality of sites 102A through 102N, and a network service provider manages one or more wireless networks 106A through 106N at the plurality of sites 102A through 102N, respectively. Although Figure 1 , each site 102A-102N is illustrated as including a single wireless network 106A-106N, respectively, but in some examples each site 102A-102N may include multiple wireless networks, and the present disclosure is not limited in this respect.

[0019] Each site 102A to 102N includes multiple network access server (NAS) devices, such as access points (APs) 142, switches 146, or routers (not shown) within the edge of the wired network. For example, site 102A includes multiple APs 142A-1 to 142A-M. Similarly, site 102N includes multiple APs 142N-1 to 142N-M. Each AP 142 can be any type of wireless access point, including but not limited to a commercial or enterprise AP, a router, or any other device that is connected to a wired network and is capable of providing wireless network access to client devices within the site. For example, when AP 142 represents a Wi-Fi device, one or more APs 142 may include multiple radios for connecting to client devices, such as a 2.4 GHz radio, a 5 GHz radio, a 6 GHz radio, etc.

[0020] Each site 102A to 102N also includes a plurality of client devices, also referred to as user equipment (UE) devices, generally referred to as UE or client devices 148, representing various wireless-enabled devices within each site. For example, a plurality of UEs 148A-1 to 148A-M are currently located at site 102A. Similarly, a plurality of UEs 148N-1 to 148N-M are currently located at site 102N. Each UE 148 can be any type of wireless client device, including but not limited to mobile devices such as smart phones, tablets or laptops, personal digital assistants (PDAs), wireless terminals, smart watches, smart rings, or other wearable devices. UE 148 can also include wired client devices, for example, IoT devices (such as printers), security devices, environmental sensors, or any other devices connected to a wired network and configured to communicate via one or more wireless networks 106. Although each site 102A to 102N is shown as having one of the wireless networks 106, in some examples, one or more of the sites 102 may include more than one wireless network. For example, APs 142A-1 to 142A-M of site 102A may provide multiple wireless networks, such as a guest network, an employee network, and an employee safety network. In some examples, the techniques of the present disclosure may be applied to wireless networks on a wireless network basis, rather than being applied to APs and / or AP's radios, or may be applied to wireless networks on a wireless network basis in addition to being applied to APs and / or AP's radios. For example, the techniques of the present disclosure may be applied to identify anomalies in an employee safety network based on client activity data of one or more AP devices associated with an employee safety network and / or associated with a peer network (such as a guest network and / or an employee network).

[0021] To provide wireless network services to UE 148 and / or communicate over wireless network 106, AP 142 and other wired client devices at site 102 are connected directly or indirectly to one or more network devices (e.g., switches, routers, etc.) via physical cables (e.g., Ethernet cables). Figure 1 In the example of , site 102A includes switch 146A to which each of APs 142A-1 through 142A-M at site 102A is connected. Similarly, site 102N includes switch 146N to which each of APs 142N-1 through 142N-M at site 102N is connected. Figure 1 146, each site 102 is shown as if it includes a single switch 146 and all APs 142 of a given site 102 are connected to a single switch 146, in some other examples, each site 102 may include more or fewer switches and / or routers. In addition, the APs and other wired client-side devices of a given site may be connected to two or more switches and / or routers. In addition, two or more switches at a site may be interconnected and / or connected to two or more routers, for example, via a mesh or partially meshed topology in a hub-and-spoke architecture. In some examples, the interconnected switches and routers include a wired local area network (LAN) at the site 102 that hosts the wireless network 106.

[0022] The example network system 100 also includes various networking components for providing networking services within the wired network, including, for example, an authentication, authorization, and accounting (AAA) server 110 for authenticating users and / or UEs 148, a dynamic host configuration protocol (DHCP) server 116 for dynamically allocating a network address (e.g., an IP address) to the UE 148 after authentication, a domain name system (DNS) server 122 for resolving domain names to network addresses, a plurality of servers 128A to 128X (collectively referred to as “servers 128”) (e.g., web servers, database servers, file servers, application servers, etc.), and a network management system (NMS) 130. Figure 1 As shown, the various devices and systems of network 100 are coupled together via one or more networks 134 (eg, the Internet and / or a corporate intranet).

[0023] exist Figure 1In an example of the present invention, NMS 130 is a cloud-based computing platform that manages wireless networks 106A to 106N at one or more of sites 102A to 102N. NMS 130 provides an integrated suite of management tools and implements various technologies of the present disclosure. In general, NMS 130 can provide a cloud-based platform for wireless network data acquisition, monitoring, activity logging, reporting, predictive analysis, network anomaly identification, and alarm generation. In some examples, NMS 130 outputs notifications such as alarms, warnings, graphical indicators on dashboards, log messages, text / SMS messages, email messages, etc., and / or suggestions about wireless network problems to a site or network administrator (“admin”) interacting with and / or operating management device 111. In addition, in some examples, NMS 130 operates in response to configuration inputs received from an administrator interacting with and / or operating management device 111. In some examples, NMS 130 actively identifies anomalies associated with any AP 142 (and / or its radio) and takes corrective actions to resolve the anomalies.

[0024] The management device 111 may include an IT staff and / or administrator computing device associated with one or more of the sites 102 and / or switches 146 at the edge of the wired network. The management device 111 may be implemented as any suitable device for presenting output and / or accepting user input. For example, the management device 111 may include a display. The management device 111 may be a computing system, such as a mobile or non-mobile computing device operated by a user and / or by an administrator. The management device 111 may, for example, represent a workstation, a laptop or notebook computer, a desktop computer, a tablet computer, or any other computing device that can be operated by a user and / or present a user interface according to one or more aspects of the present disclosure. The management device 111 may be physically separated from the NMS 130 and / or located at a location different from the location of the NMS 130, so that the management device 111 can communicate with the NMS 130 via the network 134 or other communication means.

[0025] In some examples, one or more of the NAS devices (e.g., AP 142, switch 146, or router) can be connected to edge devices 150A to 150N via physical cables (e.g., Ethernet cables). Edge device 150 includes a cloud-managed wireless local area network (LAN) controller. Each of edge devices 150 can include a resident device at site 102 that communicates with NMS 130 to extend certain microservices from NMS 130 to the resident NAS devices, while using NMS 130 and its distributed software architecture for scalable and resilient operations, management, troubleshooting, and analysis.

[0026] Each network device of the network system 100 (e.g., servers 110, 116, 122 and / or 128, AP 142, UE 148, switch 146, and any other server or device attached to or forming part of the network system 100) may include a system log or error log engine, wherein each of these network devices records the status of the network device, including normal operating status and error conditions.

[0027] In some examples, the NMS 130 monitors network data 138 and client activity data 137 received from the wireless networks 106A to 106N at each site 102A to 102N, respectively, and manages network resources (such as AP 142 at each site) to deliver a high-quality wireless experience to end users, IoT devices, and clients at the site. For example, the NMS 130 can receive client activity data 137 and network data 138 from a site 102 associated with an AP 142. The NMS 130 may include a virtual network assistant (VNA) 133 that implements an event processing platform for providing real-time insights and simplified troubleshooting for IT operations, and automatically takes corrective actions or provides recommendations to proactively resolve wired or wireless network problems. The VNA 133 can, for example, process hundreds or thousands of concurrent streams of network data 138 and client activity data 137 from sensors and / or agents associated with APs 142 within the network 134 and / or nodes within the network 134. For example, the VNA 133 of NMS130 may include an underlying inclusive detection engine, a cluster engine, and an exclusive detection engine according to various examples described herein. These engines may apply historical data and models to identify anomalies, classify the identified anomalies, and take remedial actions to attempt to remedy the anomalies. The VNA 133 may provide real-time alarms and reports to notify any anomalies, trends to a site or network administrator via a management device 111, and may perform root cause analysis and automatic or auxiliary error remediation. In some examples, the VNA 133 of NMS130 may apply machine learning techniques to identify the root cause of anomalies from network data streams 138 and / or client activity data 137. The VNA 133 may take remedial actions to resolve the anomalies, thereby automatically improving the underlying SLE metrics and also automatically improving the user experience.

[0028] In operation, NMS130 observes, collects and / or receives client activity data 137 and network data 138, which may take the form of data extracted from messages, counters and statistics, for example. Network data 138 may include multiple states or parameters indicating one or more aspects of wireless network performance, such as service level expectation / experience (SLE) metrics (e.g., RSSI, jitter, transmitted bytes, radio utilization, total number of clients per AP, number of APs per site), events, etc. According to a specific implementation, the computing device is part of NMS130. According to some other implementations, NMS130 may include one or more computing devices, dedicated servers, processes, virtual machines, containers, services and / or other forms of environments for executing the techniques described herein. Similarly, computing resources and components that implement VNA 133 may be part of NMS130, may be executed on other servers or execution environments, or may be distributed to nodes (e.g., routers, switches, controllers, gateways, etc.) within network 134.

[0029] The client activity data 137 may include the number of client devices connected to an AP device in AP 142 or a radio of an AP device in AP 142, the length of time each client device is connected to an AP device in AP 142 or a radio of an AP device in AP 142, the amount of client traffic sent by an AP device in AP 142 or a radio of an AP device in AP 142 during a time period, the amount of client traffic received by an AP device in AP 142 or a radio of an AP device in AP 142 during the time period, or other information indicating the activity of client devices connected to an AP device or a radio of an AP device.

[0030] The one or more servers 128 may include an application server that provides cloud-based applications. The cloud-based applications may include, for example, video conferencing applications, gaming applications, or other applications. The UE 148 may establish an application session with the application server via a network device (e.g., AP 142, switch 146, etc.) of the network system 100. For example, in a client-to-cloud application session topology of application session 160, the client device 148A-1 may interface with the wireless network 106A, which in turn may communicate with a wired network. The wired network may communicate with a WAN, which may interface with a service provider network (such as network 134) provided by an Internet service provider (ISP), which in turn provides a connection to a third-party application server (e.g., server 128A), which provides instances of applications to the client device.

[0031] NMS 130 may be configured to detect anomalies (e.g., potential problems) in the behavior of AP devices in AP 142 based on information indicating client activity for each AP device (e.g., client activity data 137), such as the number of client devices connected to the AP device or the AP device's radio, the dwell time of each client device connected to the AP device or the AP device's radio (e.g., how long the client device is connected to the AP device), the amount of client traffic sent and / or received by the AP device or the AP device's radio over a time period, etc. For example, NMS 130 may apply one or more machine learning models to the client activity data of one or more APs to predict future client activity of the AP devices. In some examples, NMS 130 may exclude client activity data from certain AP device(s) to be applied to the (multiple) machine learning models, such as client activity data from inactive AP devices, inactive radios, inactive sites, new AP devices, dormant AP devices, dormant radios, dormant sites, and / or low-activity AP devices or radios, because such client activity data may unduly skew the forecast. In some examples, NMS 130 may perform such exclusion based on user selection to exclude client activity data from certain (some) AP devices and / or radio devices. In some examples, NMS 130 may pre-select client activity data from certain (some) sites (e.g., site 102A) or (multiple) organizations (e.g., more than one site may be included) to apply to the forecast model. In some examples, NMS 130 may apply a probabilistic classification algorithm to client activity data to forecast a prediction interval (e.g., a range with a certain probability) or an estimated baseline indicating future observations of client activity (e.g., the predicted number of client devices that will be connected to the AP device). For example, NMS 130 may use quantile simulation to forecast an estimated range or baseline of the number of client devices to be connected to the AP device and / or an estimated range or baseline of the amount of client traffic to be sent or received by the AP device, etc. In some examples, NMS 130 may use conformal prediction to forecast an estimated range or baseline of the number of client devices to be connected to the access point and / or an estimated range or baseline of the amount of client traffic to be sent or received by the AP device. NMS 130 can detect anomalies in the behavior of the AP device by determining whether the incoming client activity data is within a forecasted prediction interval and / or equals or exceeds a baseline. For example, a determination that the incoming client activity data is below a lower prediction interval of the forecasted prediction interval and / or below a baseline can indicate an anomaly in the behavior of the AP device.

[0032] In some examples, NMS 130 can be configured to detect anomalies in the behavior of AP devices based on comparison of client activity data between peer AP devices. In some examples, peer AP devices can include AP devices at the same site (e.g., AP 142A-1 and AP 142A-M can be peer AP devices because they are both at site 102A). In some examples, peer AP devices can include AP devices of the same model, brand, software version, etc. In some examples, peer AP devices can represent neighboring AP devices, such as AP devices that detect signals of at least a predefined signal strength (e.g., RSSI) from each other.

[0033] For example, NMS 130 may obtain client activity data corresponding to peer AP devices (e.g., AP 142A-1 to 142A-M), such as the number of client devices connected to each peer AP device and / or the amount of traffic corresponding to each peer AP device. NMS 130 may compare the client activity data corresponding to the peer AP devices, and based on the comparison, may detect anomalies in the behavior of the AP devices in the peer AP device group. For example, AP 142A-1 and AP 142A-M may be peer devices. In this example, NMS 130 may obtain the number of client devices of UE 148 connected to AP 142A-1 and the number of client devices of UE 148 connected to AP 142A-M. Based on a comparison of the number of client devices connected to AP 142A-1 and the number of client devices connected to AP 142A-M, NMS 130 may determine that there is a difference between the number of client devices connected to AP 142A-1 (e.g., 0 connected client devices during a time period) and the number of client devices connected to AP 142A-M, a second AP device (e.g., 100 connected client devices during the same time period). The difference may be indicative of a potential problem with AP 142A-1. As another example, NMS 130 may obtain client activity data, such as the amount of client traffic handled by AP 142A-1 and the amount of client traffic handled by AP 142A-M during a given time period. Based on a comparison of the amount of client traffic handled by AP 142A-1 (e.g., low traffic throughput) and the amount of client traffic handled by AP 142A-M (e.g., high traffic throughput) during a given time period, NMS 130 AP 142A-1 can determine that there is a difference between the amount of client traffic between peer AP devices, which may indicate a potential problem with AP 142A-1. In some examples, multiple client activity data are used to detect anomalies. For example, NMS 130 can obtain client activity data, such as the number of connected client devices and the amount of client traffic handled by AP 142A-1 during a given time period. If there are a large number of client devices connected to AP 142A-1, but a low amount of client traffic handled by AP 142A-1, NMS 130 can determine that there is an anomaly. In some other examples, NMS 130 can obtain client activity data, such as the number of repeated connections by client devices, and determine that there is an anomaly if there is a certain number of repeated connections. In some examples, NMS 130 may obtain client activity data corresponding to all AP devices at a site to identify whether a potential problem is attributable to a site-level problem (eg, a problem with switch 146A or edge device 150A) rather than an individual AP device 142 .

[0034] NMS 130 may also be configured to determine the scope of the problem (e.g., whether the problem is experienced at an organizational level, a site level, an AP device level, etc.) and to classify AP devices having detected anomalies (e.g., determined from an inclusive detection engine) as AP devices having known problems, AP devices having unknown problems, and / or AP devices that are incorrectly identified as having problems (e.g., false positives).

[0035] For AP devices with known problems, NMS130 can be configured to perform remedial actions associated with such known problems. Remedial actions can include reinitializing the radio of the AP device, resetting the radio of the AP device, switching channels, restarting the device or process associated with monitoring SLE metrics, resetting the AP device, initiating a return authorization (RMA) process, turning off the radio, etc. For example, NMS130 can control the AP device with a known problem to perform remedial actions. If the remedial action fails to correct the anomaly, in some examples, NMS130 can take one or more additional remedial actions to try to resolve the problem. In some examples, remedial actions can be taken in order, for example, from the least interruption to the most interruption until the problem is resolved.

[0036] For AP devices with unknown problems, NMS 130 may send information associated with the AP devices with unknown problems (e.g., client traffic data and / or other information) for further analysis. For example, NMS 130 may send information to management device 111 for analysis by an administrator or technical support. For example, an administrator or technical support may perform a network test to try to identify the problem. If the administrator or technical support does identify the problem, the administrator or technical support may log the problem as known so that NMS 130 may attempt to remedy the now known problem and / or subsequent occurrences of the problem. In some examples, NMS 130 may classify one or more AP devices that are incorrectly identified as having problems and provide feedback (such as training) to a machine learning algorithm that identifies anomalies associated with AP devices.

[0037] Although the techniques of the present disclosure are described in this example as being performed by NMS 130, the techniques described herein may be performed by any other computing device(s), system(s), and / or server(s), and the present disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the techniques of the present disclosure may reside in a dedicated server or be included in any other server that is additional to or different from NMS 130, or may be distributed throughout network 100 (e.g., in edge devices 150), and may or may not form part of NMS 130.

[0038] Figure 2 is a block diagram of an example access point (AP) device 200 configured in accordance with one or more techniques of this disclosure. Figure 2 The example access point 200 shown may be used to implement the Figure 1 Any AP 142 shown and described. Access point 200 may include, for example, a Wi-Fi, 5G, Bluetooth and / or Bluetooth Low Energy (BLE) base station or any other type of wireless access point.

[0039] exist Figure 2 In the example of FIG. 1 , access point 200 includes a wired interface 230, wireless interfaces 220A-220B, one or more processors 206, memory 212, and input / output 210 coupled together via bus 214, through which the various elements can exchange data and information. Wired interface 230 represents a physical network interface and includes a receiver (RX) 232 and a transmitter (TX) 234 for sending and receiving network communications (e.g., packets). Wired interface 230 directly or indirectly couples access point 200 to a wired network device, such as a wireless network, within a wired network via a cable (such as an Ethernet cable). Figure 1 A switch among switches 146.

[0040] The first wireless interface 220A and the second wireless interface 220B represent wireless network interfaces and include receivers 222A and 222B, respectively, each of which includes a receiving antenna, via which the access point 200 can receive wireless communication signals from wireless communication devices such as Figure 1 The first wireless interface 220A and the second wireless interface 220B further include transmitters 224A and 224B, respectively, each of which includes a transmitting antenna, and the access point 200 can transmit the wireless communication device (such as Figure 1 UE 148) sends wireless signals. Therefore, the first wireless interface 220A and the second wireless interface 220B can represent different radios of the access point 200. In some examples, the first wireless interface 220A may include a Wi-Fi 802.11 interface (e.g., 2.4GHz, 5GHz and / or 6GHz) and the second wireless interface 220B may include a Bluetooth interface and / or a Bluetooth low energy (BLE) interface. In some examples, both the first wireless interface 220A and the second wireless interface 220B represent different Wi-Fi 802.11 interfaces. In some examples, the access point 200 may include more than two wireless interfaces.

[0041] The processor(s) 206 are programmable hardware-based processors configured to execute software instructions, such as those used to define software or computer programs, which are stored to a computer-readable storage medium (such as the memory 212), such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause the processor(s) 206 to perform the techniques described herein.

[0042] Memory 212 includes one or more devices configured to store programming engines and / or data associated with the operation of access point 200. For example, memory 212 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause one or more processors 206 to perform the techniques described herein.

[0043] In this example, the memory 212 stores executable software, including an application programming interface (API) 240, a communication manager 242, a configuration / radio settings 250, a device status log 252, and data 254. The device status log 252 includes a list of events specific to the access point 200. The events may include a log of both normal events and error events, such as, for example, memory status, reboot or restart events, crash events, cloud disconnect and self-recovery events, low link speed or link speed oscillation events, Ethernet port status, Ethernet interface packet errors, upgrade failure events, firmware upgrade events, configuration changes, etc., as well as time and date stamps for each event. The log controller 255 determines the logging level for the device based on instructions from the NMS 130. Data 254 may store any data used and / or generated by access point 200, including data collected from UE 148 (such as the number of client devices connected to the AP device, the length of time each client device is connected to the AP device, the amount of client traffic sent by the AP device during a time period, or the amount of client traffic received by the AP device during the time period), which is sent by access point 200 for cloud-based management of wireless network 106A by NMS 130. Data 254 may also store data used to calculate one or more SLE metrics, which is sent by access point 200 for cloud-based management of wireless network 106A by NMS 130.

[0044] Input / output (I / O) 210 represents physical hardware components that can interact with a user, such as buttons, displays, etc. Although not shown, memory 212 typically stores executable software for controlling the user interface with respect to input received via I / O 210. Communication manager 242 includes program code that, when executed by processor(s) 206, allows access point 200 to communicate with UE 148 and / or network(s) 134 via interface(s) 230 and / or any of interfaces 220A to 220C. Configuration settings 250 include any device settings for access point 200, such as radio settings for each of wireless interfaces 220A to 220C. These settings can be manually configured, or can be remotely monitored and managed by NMS 130 to optimize wireless network performance on a periodic (e.g., hourly or daily) basis.

[0045] As described herein, the AP device 200 can measure network data and report it from the status log 252 to the NMS 130. The network data can include event data, telemetry data, and / or other SLE-related data. The network data can include various parameters indicating the performance and / or status of the wireless network. These parameters can be measured and / or determined by one or more UE devices and / or one or more APs in the wireless network. The NMS 130 can determine one or more SLE metrics based on the SLE-related data received from the APs in the wireless network and store the SLE metrics as network data 137 ( Figure 1 ).

[0046] Figure 3 is a block diagram of an example NMS 300 configured to determine anomalies in the behavior of AP devices according to one or more aspects of the present disclosure. The NMS 300 may be used to implement, for example Figure 1 In such an example, the NMS 300 is responsible for monitoring and managing one or more wireless networks 106A to 106N at sites 102A to 102N, respectively.

[0047] The NMS 300 includes a communication interface 330, one or more processors 306, a user interface 310, a memory 312, and a database 318. The various elements are coupled together via a bus 314, and the various elements can exchange data and information through the bus 314. The NMS 300 receives client activity data 316 from one or more of the APs 142 and stores the client activity data 316 in the database 318. For example, the client activity data 316 may include the number of client devices connected to the AP device at a certain time, the length of time each client device is connected to the AP device, the amount of client traffic sent by the AP device during a time period, or the amount of client traffic received by the AP device during the time period. In some examples, the NMS 300 receives data from one or more of the client devices 148, the AP 142, the switch 146, and other network nodes (e.g., routers) within the network 134, which can be used to calculate one or more SLE metrics. The NMS 300 analyzes the data for cloud-based management of the wireless networks 106A to 106N via the AP 142. In some examples, NMS 300 may be Figure 1 or a portion of another server shown in or a portion of any other server.

[0048] The processor(s) 306 execute software instructions, such as those used to define software or a computer program, which are stored to a computer-readable storage medium, such as the memory 312, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, which stores instructions to cause the processor(s) 306 to perform the techniques described herein.

[0049] The communication interface 330 may include, for example, an Ethernet interface. The communication interface 330 couples the NMS 300 to a network and / or the Internet, such as Figure 1 The communication interface 330 includes a receiver 332 and a transmitter 334, through which the NMS 300 receives and sends data and information from and to any of the following: the AP 142, the client device 148, the switch 146, the servers 110, 116, 122, 128, and / or any local area network. Figure 1 Any other network node, device, or system that is part of the illustrated network system 100. In some examples, the NMS 300 can take action to resolve the anomaly via the communication interface 330. For example, the NMS 300 can send a signal to the AP in the APs 142 having the noise problem to change channels.

[0050] The data and information received by the NMS 300 may include, for example, client activity data, telemetry data, SLE-related data, or event data received from one or more of the client devices AP 148, AP 142, switch 146, or other network nodes, which are used by the NMS 300 to remotely monitor the performance of the wireless network 106 and application sessions from client devices to cloud-based application servers. The NMS 300 may also send data and / or commands to any network device (such as client devices 148, AP 142, switch 146, network 134, other network nodes within the management device 111) via the communication interface 330 to remotely manage portions of the wireless network 106 and the wired network. For example, the NMS 300 may send a command to an AP device in AP 142 to take remedial action.

[0051] The memory 312 includes one or more devices configured to store programming engines and / or data associated with the operation of the NMS 300. For example, the memory 312 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including one or more storage devices (e.g., a disk drive or an optical drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause the one or more processors 306 to perform the techniques described herein.

[0052] In this example, the memory 312 includes an API 320, an SLE engine 322, and a virtual network assistant (VNA) / AI engine 350. The SLE engine 322 enables setting and tracking thresholds for SLE metrics for each network 106A to 106N. The SLE engine 322 further analyzes SLE-related data collected by an AP (such as any AP 142 from an AP 142 of a UE in each wireless network 106A to 106N). For example, APs 142A-1 to 142A-M collect SLE-related data from UEs 148A-1 to 148A-M currently connected to the wireless network 106A. The data is sent to the NMS 300, which is executed by the SLE engine 322 to determine one or more SLE metrics for each UE 148A-1 to 148A-M currently connected to the wireless network 106A. This data, in addition to any network data collected by one or more APs 142A- 1 through 142A-M in wireless network 106A, is also sent to NMS 300 and stored in database 318 , for example, as network data 316 .

[0053] The VNA / AI engine 350 analyzes data received from the network devices as well as its own data to identify when an unexpected abnormal state is encountered at one of the network devices. For example, the VNA / AI engine 350 can identify the root cause of any unexpected or abnormal state, such as any (multiple) bad SLE metrics indicating a connection problem at one or more network devices.

[0054] NMS 300 may include VNA / AI engine 350. VNA / AI engine 350 may analyze client activity data 316 to identify anomalies in the behavior of AP 142, and in some examples, take remedial actions to attempt to resolve the anomalies. For example, VNA / AI engine 350 may determine an anomaly in the behavior of AP 142A-1 based on client activity data 316, and in some examples, may perform remedial actions based on the determined anomaly in the behavior of AP 142A-1.

[0055] Examples of remedial actions that can be automatically performed by the VNA / AI engine 350 may include, but are not limited to, reinitializing the radio, resetting the radio, switching channels, resetting a device or process associated with the monitoring SLE, resetting the AP (e.g., AP142A-1), disabling the radio, identifying the AP as a return authorization (RMA) candidate (e.g., the AP itself is defective and should be returned to the manufacturer), etc. These remedial actions are given for exemplary purposes only, and the present disclosure is not limited in this respect. In some examples, if the automatic remedial action is not available or cannot adequately resolve the anomaly, the VNA / AI engine 350 can proactively provide a notification, including a suggested corrective action taken by IT personnel (e.g., a site or network administrator using the management device 111) to resolve the anomaly.

[0056] The VNA / AI engine 350 may include an inclusive detection engine 352, a clustering engine 354, and / or an exclusive detection engine 356. Each of the inclusive detection engine 352, the clustering engine 354, and / or the exclusive detection engine 356, when executed by the processor(s) 306, may utilize one or more machine learning models 380 to perform the actions attributed thereto, as described in the present disclosure.

[0057] The inclusion detection engine 352 may be configured to detect anomalies (e.g., potential problems) in the behavior of one of the AP devices 142 based on the client activity data 316. In some examples, the inclusion detection engine 352 may utilize the Holt-Winter exponential smoothing model of the machine learning model(s) 380 to perform time series and / or temporal detection of anomalies. As an example, the inclusion detection engine 352 may apply a forecasting model (e.g., of the machine learning model(s) 380) to the client activity data 316 to forecast client activity of an AP device (such as AP 142A-1). For example, the inclusion detection engine 352 may utilize historical client activity data of the client activity data 316 from one or more of the APs 142 to forecast client activity of AP 142A-1. In some examples, the inclusion detection engine 352 may utilize only the historical client activity data of AP 142A-1 to forecast client activity of AP 142A-1. In some examples, when forecasting client activity for AP 142A-1, inclusive detection engine 352 may utilize client activity data from neighboring APs or APs at the same site (e.g., site 102A) as AP 142A-1. For example, when forecasting client activity for AP 142A-1, inclusive detection engine 352 may use client activity data for APs 142A-1 through 142A-M (e.g., AP devices at site 102A).

[0058] In some examples, the inclusive detection engine 352 may exclude or not utilize client activity data of certain AP devices when applying the forecast model. For example, the inclusive detection engine 352 may exclude client activity data from inactive AP devices, inactive sites, new AP devices, dormant AP devices, dormant sites, relatively low activity AP devices, etc. In some examples, the inclusive detection engine 352 may exclude client activity data from AP devices obtained during a certain time period, such as weekends, holidays, non-business hours, or other time periods with expected low client activity. In some examples, the administrator may program the NMS 300 via the management device 111 to exclude such client activity data. In this way, the inclusive detection engine 352 may avoid unnecessary downward tilting of the forecast of client activity data for AP devices, which may otherwise cause fewer anomalies to be detected. In some examples, the administrator may pre-select client activity data from certain organizations for the inclusive detection engine 352 to apply to the forecast model. An organization may be an entity associated with one or more sites.

[0059] In some examples, the inclusion detection engine 352 can operate in an offline manner, such as on data that is not received in real time.For example, the inclusion detection engine 352 can determine a forecast of client activity based on multiple weeks (and / or portions thereof) of historical client activity data.

[0060] In some examples, the inclusivity detection engine 352 may apply a probabilistic classification algorithm (e.g., of the machine learning model(s) 380) to the client activity data 316 (or a subset thereof) to forecast a prediction interval (e.g., a range with a certain probability) or an estimated baseline indicating future observations of client activity (e.g., a predicted number of client devices to be connected to the AP device). For example, the inclusivity detection engine 352 may employ quantile simulation to forecast an estimated range or baseline for the number of client devices to be connected to the AP 142A-1, and / or an estimated range or baseline for the amount of client traffic to be sent or received by the AP 142A-1. In some examples, the inclusivity detection engine 352 may use conformal prediction to forecast an estimated range or baseline for the number of client devices to be connected to the access point, and / or an estimated range or baseline for the amount of client traffic to be sent or received by the AP 142A-1. In some examples, the inclusiveness detection engine 352 may apply one or more time series models (e.g., of (multiple) machine learning models 380) to take into account trends and / or seasonality (e.g., typical business hours, holidays, etc.) in the client activity data to determine a forecast baseline and / or prediction interval.

[0061] In some examples, the inclusive detection engine 352 can detect anomalies in the behavior of the AP device by determining whether the incoming client activity data is within a forecasted prediction interval or above a baseline. For example, a determination that the incoming client activity data is below a lower prediction interval of the forecasted prediction interval or below a baseline can indicate an anomaly in the behavior of the AP device. In some examples, the upper limit of the prediction interval (upper_ci) is the 98th percentile, and the lower limit of the prediction interval (lower_ci) is the 2nd percentile. In such an example, if the current value is below the 2% forecast interval (lower_ci), the inclusive detection engine 352 can detect an anomaly.

[0062] In some examples, the inclusive detection engine 352 can be configured to detect anomalies in the behavior of AP 142A-1 based on a comparison of client activity data between AP 142A-1 and a peer AP device. As mentioned herein, AP 142A-1 is a subject AP device, and the subject AP device is an AP device for which anomalies in behavior are determined. The peer AP device can be an AP device that is adjacent to, relatively close to, or within a threshold distance and / or signal strength of the AP device (AP 142A-1) being checked for anomalies, or the peer AP device can be an AP device located at the same site as the AP device being checked for anomalies, such as an AP device to which multiple client devices are connected. In some examples, the peer AP device can be an AP device of the same model, brand, software version, etc. as the subject AP device. In some examples, the peer AP device can be an AP device from which the AP device being checked for anomalies detects a signal of at least a predefined signal strength (e.g., RSSI).

[0063] In these examples, the inclusion detection engine 352 may obtain client activity data corresponding to the peer AP device, such as the client activity data 316. The inclusion detection engine 352 may apply one or more thresholds (such as activity thresholds) to the client activity data corresponding to the peer AP device to determine whether the AP device has an anomaly. There may be thresholds (such as activity thresholds) for each client activity metric that the subject AP device is compared to one or more peer AP devices, and / or there may be thresholds (such as activity thresholds) for combinations of different client activity metrics (such as for scores based on multiple client activity metrics). For example, the inclusion detection engine 352 may compare the client activity data corresponding to the subject AP device and one or more peer AP devices with the subject AP device, and based on the comparison, may detect an anomaly in the behavior of AP 142A-1. For example, AP 142A-1 and AP 142A-M may be neighbors.

[0064] In an example, the number of client devices connected to the AP device can be a client activity metric. In this example, the inclusive detection engine 352 can obtain the number of client devices connected to the AP 142A-1 and the number of client devices connected to the AP 142A-M, and based on a comparison of the number of client devices corresponding to the AP 142A-1 and the number of client devices corresponding to the AP 142A-M, it can be determined that there is a difference (e.g., a difference) between the number of client devices connected to 142A-1 (e.g., 0 connected client devices during a time period) and the number of client devices connected to the AP 142A-M (e.g., 100 connected client devices during the same time period), which may indicate a potential problem with 142A-1, for example, when the difference differs by more than (or is more than or equal to) an activity threshold amount of the client devices.

[0065] In an example, the amount of client traffic of the AP device can be a client activity metric. In this example, the inclusive detection engine 352 can obtain client activity data, such as the amount of client traffic corresponding to 142A-1 and the amount of client traffic corresponding to AP 142A-M, and based on a comparison of the amount of client traffic corresponding to 142A-1 and the amount of traffic corresponding to AP 142A-M, can determine that there is a difference between the amount of client traffic between peer AP devices, which can indicate a potential problem with 142A-1, for example, when the difference differs by more than (or is more than or equal to) an activity threshold amount of client traffic.

[0066] In some examples, the NMS can obtain client activity data corresponding to all AP devices at the site to identify whether a potential problem is attributable to site-level devices rather than individual AP devices (e.g., a problem with an edge device connected to the AP device). The above examples discuss a single peer device. If there are multiple peer devices, the inclusive detection engine 352 can average or otherwise combine client activity data from multiple peer devices as part of the comparison with the AP device for which the presence of a potential anomaly is being performed.

[0067] In some examples, the inclusion detection engine 352 can operate in an offline manner, such as for data that is not received in real time. For example, the inclusion detection engine 352 can compare client activity data of the subject AP device and peer AP devices on an hourly basis.

[0068] In some examples, the inclusion detection engine 352 may determine anomalies in the behavior of the AP device based on a comparison with the past behavior of the AP device. For example, the inclusion detection engine 352 may determine one or more events in the historical client activity data, where the AP device has no connected client devices or a significantly small number of connected client devices (e.g., the number of connected client devices is less than or less than or equal to a threshold) for a relatively long duration (e.g., a threshold duration of at least one hour, a day, etc.). The inclusion detection engine 352 may determine duration information associated with one or more events. For example, the inclusion detection engine 352 may determine the duration information to include the longest duration of one or more events, the average duration of one or more events, etc. The inclusion detection engine 352 may determine an event in the current (e.g., most recently received) client activity data, where the AP device has no connected client devices or a significantly small number of connected client devices for a relatively long duration. The inclusion detection engine 352 may determine the difference between the duration of the event in the current client activity data and the duration information, and determine whether the difference meets the difference threshold. If the difference meets (e.g., is greater than, or greater than or equal to) the threshold, this may indicate that there is an anomaly in the behavior of the AP device.

[0069] Inclusion detection engine 352 may store information associated with AP devices identified as including anomalies in behavior in network data 320. For example, inclusion detection engine may store identification of the AP device, a time associated with the anomaly, and the like in network data 320.

[0070] In some examples, the inclusion detection engine 352 can determine one or more scores associated with a comparison to a forecast, a comparison to (multiple) peer APs, or other techniques for determining anomalies in the behavior of an AP. The inclusion detection engine 352 can utilize the (multiple) scores to determine the occurrence of an anomaly and / or the severity of the anomaly. For example, the inclusion detection engine 352 can determine a combined score based on both the score of the comparison to the forecast and the score of the comparison to (multiple) peer APs.

[0071] In some examples, the upper limit of the prediction interval (upper_ci) is the 98th percentile and the lower limit of the prediction interval (lower_ci) is the 2nd percentile. In such an example, if the current value is below the 2% prediction interval (lower_ci), the inclusiveness detection engine 352 can detect an anomaly.

[0072] In some examples, the inclusiveness detection engine 352 can determine a generalized z-score based on the prediction interval as score = (current value - lower_ci) / (average forecast value - lower_ci), which can be the ratio of the deviation of the current value from the average forecast value to the width of the lower prediction interval. If the score is less than or equal to 0, it means that the current value is below or on the lower 2% boundary. If the score is greater than 0, it means that the current value is within the 98% prediction interval.

[0073] The inclusive detection engine 352 can determine the forecast severity based on the anomaly detected across multiple metrics. For example, a severity value can be added to a severity score based on each metric from which the anomaly is detected. For example, if the anomaly is detected based on forecast comparisons and peer comparisons, the severity value can be added to the severity score and / or the score for each forecast comparison, and the peer comparison can be combined as part of determining the severity score.

[0074] Additionally or alternatively, the severity score may be based on anomalies detected using different types of client activity data. For example, a forecast score based on sent traffic, received traffic, and active client minutes may include three forecast scores that may be utilized to determine a severity score. For example, a peer activity score based on a comparison of client devices to which a peer AP device is connected, sent traffic, and received traffic with those of a detected AP may include three peer activity scores that may be used to determine a severity score. For example, a duration score based on the duration of intervals with low traffic and / or no traffic detected on an AP device may be utilized to determine a severity score. In some examples, a combination of one or more of the forecast score, peer activity score, and / or duration score may be used to determine a severity score.

[0075] For example, the inclusion detection engine 352 may determine the forecast score as forecast score = (current value - lower_ci) / (forecast - lower_ci). The inclusion detection engine 352 may determine the forecast z-score as forecast z_score = 1 / (1 + exp (-abs (forecast score))). The inclusion detection engine 352 may determine the peer score based on the AP client count. For example, the inclusion detection engine may determine the peer score as peer score = (median client minutes) / (median client minutes - 10% quantile). The inclusion detection engine 352 may determine the peer z-score as peer z_score = 1 / (1 + exp (-abs (peer score))). The inclusion detection engine 352 may determine the duration score as duration z_score = 1 / (1 + exp (-abs (duration))).

[0076] In some examples, the inclusion detection engine 352 can determine a cumulative score based on the forecast z score, the peer z score, and / or the duration z score. For example, the inclusion detection engine 352 can determine the combined z score as combined_z_score=100 / norm*sum(score_weight t*z_score), where norm=sum(score_weight) and weight is a machine learning model parameter optimized to reduce false positive detections. For example, an initial setting of weights can include forecast_client_minutes: 1.0, forecast_tx_bytes: 0.3, forecast_rx_bytes: 0.2, peer score: 1.5, duration score: 1.5. It should be noted that such weights can be user-selectable and / or can be adjustable through the training of (multiple) machine learning models 80. In some examples, the inclusion detection engine 352 can send one or more of the scores discussed herein (such as severity scores) to, for example, the management device 111 as part of a report.

[0077] The clustering engine 354 may be configured to determine the scope of the problem (e.g., whether an anomaly is experienced at an organization level, a site level, an AP device level, an AP model type level, an AP firmware version level, etc.), and to classify AP devices with detected anomalies (e.g., determined by the inclusive detection engine 352) as AP devices with known problems, AP devices with unknown problems, and / or AP devices that are incorrectly identified as having problems (e.g., false positives). In some examples, the clustering engine 354 may utilize nearest neighbor and / or mutual information models of the machine learning model(s) 380 when determining the scope of the problem and / or classifying AP devices with detected anomalies.

[0078] In some examples, cluster engine 354 can determine how to classify AP devices based on, for example, AP states that may be received and / or stored in network data 317, SLE events (e.g., as determined by SLE engine 322), cluster error-counters, and / or time series session analysis.

[0079] In some examples, the cluster engine 354 can determine that the AP device has a known problem based on known symptoms experienced by the subject AP device. Example known symptoms can include transmit PHY errors, decryption problems, beacon loss, high noise floor, etc. Such symptoms can be determined by the cluster engine 354, the SLE engine 322, received in network data, or any combination thereof. In some examples, the cluster engine 354 can determine that the AP device has an unknown problem based on the symptoms experienced by the subject AP device are not known symptoms or are not completely within a set of known symptoms.

[0080] For AP devices with known issues, the cluster engine 354 may send an indication to the exclusionary detection engine 356 for the exclusionary detection engine 356 to determine actions to take in attempting to remediate the anomaly.

[0081] For AP devices with unknown problems, cluster engine 354 may send information associated with the AP devices with unknown problems (eg, client behavior data and / or other information) to, for example, management device 111 for further analysis by, for example, technical support or an administrator.

[0082] In some examples, the clustering engine 356 can classify one or more AP devices that are incorrectly identified as problematic. The AP devices that are incorrectly identified as problematic may be AP devices that are actually not abnormal or AP devices that self-correct for their abnormalities. For such AP devices, the clustering engine 356 can ignore the identified abnormalities or can provide feedback to the inclusive detection engine 352, such as in the form of training (multiple) associated machine learning models 380.

[0083] The exclusionary detection engine 356 is configured to perform one or more remedial actions for AP devices with known issues. In some examples, the exclusionary detection engine 356 can use a reinforcement model of the (multiple) machine learning models 380 when determining which remedial actions to perform. The remedial actions may include reinitializing the radio of the AP device, resetting the radio of the AP device, switching the channel of the AP device, restarting the device or process associated with monitoring the SLE metric, resetting the AP device, identifying the AP device as a candidate for the RMA process, or initiating the RMA process, etc. The exclusionary detection engine 356 can initiate the remedial action by issuing a command to the subject AP device using the API 320.

[0084] If the first action taken by the exclusionary detection engine 356 fails to correct the anomaly, the exclusionary detection engine 356 can take another action to correct the anomaly. In some examples, this can continue until the anomaly is corrected or the subject AP device is identified for the RMA process or the RMA process is initiated for the subject AP device. In some examples, the actions taken by the exclusionary detection engine 356 can start with less disruptive to the operation of the wireless network, with each successive action associated with the same subject AP device becoming increasingly disruptive to the operation of the wireless network.

[0085] For example, the exclusionary detection engine 356 can be configured to determine which remedial action to take, and if the action is unsuccessful, which remedial action to take next, based on the output of the inclusive detection engine 352, the cluster engine 354, and / or the client activity data 316. For example, in determining which remedial action to take, the exclusionary detection engine 356 can utilize information (such as the presence of interference, the noise floor, error counts from the AP radio, etc.), which can be stored in the network data 317 and / or determined by the SLE engine 322.

[0086] For example, the exclusionary detection engine 356 can determine the remedial action to be taken based on known symptoms. For example, if the symptoms include that the AP device has no connected clients, the beacon has been lost or is being lost, and the log records a send PHY error, the exclusionary detection engine 356 can determine that the remedial action is to re-initiate the subject AP device. In another example, if the symptoms include that the AP station cache is full and no additional clients are added, the exclusionary detection engine 356 can determine to switch the channel of the subject AP device first. If switching channels does not remedy the anomaly, the exclusionary detection engine 356 can determine to restart the device or process associated with monitoring the SLE metric and / or reset the radio of the subject AP device. Other example symptoms that can be used to determine which action to take can include relatively high background noise, new error counters (such as using a new AP model or new AP firmware), etc.

[0087] The NMS 300 may utilize a self-validating loop to train the (multiple) machine learning models 380. For example, when an anomaly is detected and the action taken mitigates the anomaly, the NMS 300 may reinforce the detection of the anomaly and / or the selection of the action through the training of the (multiple) machine learning models 380, such as adjusting the weights to emphasize the predictions made by the (multiple) machine learning models 380. When an anomaly is detected but still exists after one or more actions are taken, or if the anomaly resolves itself without action, the NMS 300 may de-emphasize the detection of the anomaly and / or the selection of the action through the training of the (multiple) machine learning models 380, such as adjusting the weights to de-emphasize the predictions made by the (multiple) machine learning models 380.

[0088] Although the techniques of the present disclosure are described in this example as being performed by NMS 130, the techniques described herein may be performed by any other computing device(s), system(s), and / or server(s), and the present disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the techniques of the present disclosure may reside in a dedicated server or be included in any other server attached to or in addition to NMS 130, or may be distributed throughout network 100 and may or may not form part of NMS 130.

[0089] Figure 4 is a flow diagram illustrating example operations for reactive problem determination in accordance with one or more techniques of this disclosure. Figure 4 The example operation is about Figure 1 The NMS 130 is described.

[0090] NMS 130 may obtain client activity data 400 for multiple access point (AP) devices at a site. For example, NMS 130 may receive or retrieve client activity data 137 from one or more AP devices of AP 142.

[0091] The NMS 130 may determine an anomaly in the behavior of one of the one or more AP devices based on the client activity data (402). For example, the NMS 130 may determine an anomaly in the behavior of one of the one or more AP devices based on the client activity data, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of: a) determine predicted client activity data of the AP device based on the client activity data, and compare the predicted client activity data with actual client activity data of the AP device, or b) compare client activity data of at least one peer AP device with actual client activity data of the AP device.

[0092] The NMS 130 may perform an action based on the determined anomaly in the behavior of the AP device ( 404 ).For example, the NMS 130 may attempt to resolve the determined anomaly by taking remedial action.

[0093] Although primarily described herein with respect to determining anomalies in the behavior of an AP (which may include the behavior of a radio of the AP), in some examples, the NMS 130 may obtain client activity data for multiple access point (AP) devices at a site. The NMS 130 may determine anomalies in the behavior of a wireless network provided by one or more AP devices based on the client activity data. As part of determining anomalies in the behavior of a wireless network, one or more processors may be configured to at least one of: a) determine predicted client activity data for the wireless network based on the client activity data, and compare the predicted client activity data with actual client activity data for the wireless network, or b) compare client activity data for at least one peer wireless network with actual client activity data for the wireless network. The NMS 130 may perform an action based on the determined anomaly in the behavior of the wireless network. For example, the NMS 130 may attempt to resolve the determined anomaly by taking remedial action.

[0094] In some examples, the actual client activity data includes at least one of the following: the number of client devices connected to the AP device or the radio of the AP device at a certain time, the length of time each client device is connected to the AP device or the radio, the amount of client traffic sent by the AP device or the radio during the time period, or the amount of client traffic received by the AP device or the radio during the time period. In some examples, the NMS 130 may filter out a first portion of the client activity data from the client activity data before determining the abnormality in the behavior of the AP device, so that one or more processors determine the abnormality in the behavior of the AP device based on a second portion of the client activity data, and the second portion of the client activity data does not include the first portion of the client activity data. In some examples, the NMS 130 may determine to filter out the first portion of the client activity data based on the first portion of the client activity data being associated with at least one of the following: an inactive AP device, an inactive radio, an inactive station, a new AP device, a dormant AP device, a dormant radio, a dormant station, or an AP device or radio with an activity level below an activity level threshold.

[0095] In some examples, NMS 130 may pre-select client activity data to be obtained from at least one organization. In some examples, NMS 130 may determine forecasted client activity data for an AP device, and wherein as part of determining the forecasted client activity data for an AP device, NMS 130 may apply a probabilistic classification algorithm to the client activity data to forecast at least one of a prediction interval or an estimated baseline indicating future client activity. In some examples, the probabilistic classification algorithm includes at least one of quantile simulation or conformal prediction. In some examples, wherein as part of determining an anomaly in the behavior of an AP device, NMS 130 may determine at least one of: whether the current client activity data is within the forecasted prediction interval or whether the current client activity data meets the baseline.

[0096] In some examples, NMS 130 can compare client activity data of at least one peer AP device that is at least one of a) located at the same site as the AP device or b) detects a signal of at least a predetermined signal strength from the AP device. In some examples, as part of determining an anomaly in the behavior of the AP device, NMS 130 can determine whether the client activity data of at least one peer AP device differs from the client activity data of the AP device by more than an activity threshold amount.

[0097] In some examples, as part of determining an anomaly in the behavior of an AP device, NMS 130 may determine one or more first events based on historical client activity data, where the number of client devices connected to an AP device or a radio of an AP device is below a connected client device threshold for longer than a length of a time threshold. For example, NMS 130 may determine that there are three events where the number of client devices connected to AP device 142A-1 is below five for longer than one hour. NMS 130 may determine duration information associated with one or more events. For example, NMS 130 may determine that the total duration of the three events is six hours or determine an average of two hours. NMS 130 may determine the difference between the duration information and the duration of a second event where the number of connected client devices connected to the AP device or radio is below a connected client device threshold, the duration of the second event being based on current client activity data. For example, NMS 130 may determine that a second event (e.g., an event indicated in the most recently retrieved client activity data) has a duration of 24 hours where the number of client devices connected to the AP device is below five. The NMS 130 may determine that the difference between the average duration of the first event (e.g., two hours) and the average duration of the second event (e.g., 24 hours) is 22 hours. The NMS 130 may determine that the difference satisfies the difference threshold. For example, the NMS 130 may determine that 22 hours satisfies (e.g., is greater than, or is greater than or equal to) the difference threshold.

[0098] In some examples, NMS 130 may determine predicted behavior of the AP device and compare client activity data of at least one peer AP device. In some examples, as part of determining anomalies in the behavior of the AP device, NMS 130 may determine a composite score from the score associated with the predicted behavior and the score associated with the comparison of client activity data of at least one peer AP device. In some examples, the NMS may send the composite score to an administrator device as part of a report.

[0099] In some examples, NMS130 may determine the identity of each of the multiple AP devices with associated anomalies. In some examples, NMS130 may determine the level of the problem associated with the determined anomaly based on the determination of the identity of each of the multiple AP devices with associated anomalies, wherein the level of the problem includes an organizational level, a site level, or an AP device level. In some examples, NMS130 may classify each of the multiple AP devices with associated anomalies as an AP device with a known problem, an AP device with an unknown problem, or an AP device with a problem that is incorrectly identified. In some examples, the AP device is classified as an AP device with a known problem, and based on the AP device being classified as an AP device with a known problem, the remedial action includes at least one of the following: reinitializing the radio of the AP device, resetting the radio of the AP device, switching the channel of the AP device, restarting the device or process associated with monitoring the SLE metric, resetting the AP device, or initiating a return authorization (RMA) process.

[0100] In some examples, the AP device is classified as an AP device with an unknown problem, and based on the AP device being classified as an AP device with an unknown problem, the remedial action includes sending information associated with the AP device to an administrator device for further analysis. In some examples, the AP device is classified as an AP device that is incorrectly identified as having a problem, and based on the AP device being classified as an AP device that is incorrectly identified as having a problem, the NMS 130 may update a stored state of the AP device to indicate that the AP device has no abnormality.

[0101] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as engines, units, or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuit system may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0102] If implemented in hardware, the present disclosure may be directed to an apparatus, such as a processor or an integrated circuit device (such as an integrated circuit chip or chipset). Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0103] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0104] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagated signal. In some examples, non-transitory storage media may store data that can change over time (e.g., in RAM or cache).

[0105] The code or instructions may be software and / or firmware executed by a processing circuit system that includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described in the present disclosure may be provided within a software engine or a hardware engine.

Claims

1. A network management system, comprising: one or more memories; as well as one or more processors in communication with the one or more memories and configured to: Obtain client activity data of one or more access point AP devices; determining an anomaly in a behavior of an AP device of the one or more AP devices based on the client activity data, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of: a) determine predicted client activity data for the AP device based on the client activity data and compare the predicted client activity data with actual client activity data for the AP device, or b) compare client activity data of at least one peer AP device with the actual client activity data for the AP device; as well as An action is performed based on the determined anomaly in the behavior of the AP device.

2. A network management system according to claim 1, wherein the actual client activity data of the AP device includes at least one of the following: the number of client devices or radios connected to the AP device at a certain time, the length of time each client device is connected to the AP device or the radio, the amount of client traffic sent by the AP device or the radio within a time period, or the amount of client traffic received by the AP device or the radio within the time period.

3. A network management system according to claim 1, wherein the one or more processors are also configured to filter out a first part of the client activity data from the client activity data before determining the abnormality in the behavior of the AP device, so that the one or more processors determine the abnormality in the behavior of the AP device based on a second part of the client activity data, and the second part of the client activity data does not include the first part of the client activity data.

4. A network management system according to claim 3, wherein the one or more processors are configured to determine to filter out the first portion of the client activity data based on the first portion of the client activity data being associated with at least one of the following: an inactive AP device, an inactive radio, an inactive site, a new AP device, a dormant AP device, a dormant radio, a dormant site, or an AP device or radio with an activity level below an activity level threshold. 5 . The network management system of claim 1 , wherein the one or more processors are further configured to pre-select the client activity data to be obtained from at least one organization.

6. A network management system according to claim 1, wherein the one or more processors are configured to determine the predicted client activity data of the AP device, and wherein as part of determining the predicted client activity data of the AP device, the one or more processors are configured to: apply a probabilistic classification algorithm to the client activity data to forecast at least one of a prediction interval or an estimated baseline indicating future client activity.

7. The network management system of claim 6, wherein the probabilistic classification algorithm comprises at least one of quantile simulation or conformal prediction.

8. A network management system according to claim 6, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of the following: determine whether current client activity data is within the forecasted prediction interval or determine whether the current client activity data meets the baseline.

9. A network management system according to claim 1, wherein the one or more processors are configured to: compare the client activity data of at least one peer AP device, and wherein the at least one peer AP device a) is located at the same site as the AP device, or b) detects at least one of a signal of at least a predetermined signal strength from the AP device.

10. A network management system according to claim 9, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are also configured to determine whether client activity data of at least one peer AP device differs from client activity data of the AP device by more than an activity threshold amount.

11. The network management system of any one of claims 1 to 10, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are further configured to: determining one or more first events based on historical client activity data, wherein a number of connected client devices to the AP device or a radio of the AP device is below a connected client device threshold for longer than a time threshold; determining duration information associated with the one or more events; determining a difference between the duration information and a duration of a second event, wherein the number of connected client devices to the AP device or the radio is below the connected client device threshold, the duration of the second event being based on current client activity data; as well as It is determined that the difference satisfies a difference threshold.

12. A network management system according to any one of claims 1 to 10, wherein the one or more processors are configured to determine the predicted behavior of the AP device and compare the client activity data of the at least one peer AP device, and wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to determine a composite score from a score associated with the predicted behavior and a score associated with the comparison with the client activity data of the at least one peer AP device.

13. The network management system of claim 12, wherein the one or more processors are configured to send the composite score to an administrator device as part of a report.

14. The network management system according to any one of claims 1 to 10, wherein the one or more processors are further configured to: determining an identity of each AP device in the plurality of AP devices having an associated anomaly; and Based on the determination of the identity of each of the plurality of AP devices having an associated anomaly, a level of a problem associated with the determined anomaly is determined, wherein the level of the problem comprises an organization level, a site level, or an AP device level.

15. The network management system of claim 14, wherein the one or more processors are further configured to classify each of the plurality of AP devices having an associated anomaly as an AP device having a known problem, an AP device having an unknown problem, or an AP device erroneously identified as having a problem.

16. A network management system according to claim 15, wherein the AP device is classified as an AP device with a known problem, and wherein based on the AP device being classified as an AP device with a known problem, the action includes at least one of the following: reinitializing the radio of the AP device, resetting the radio of the AP device, switching the channel of the AP device, restarting a device or process associated with monitoring a subscriber level experience (SLE) metric, resetting the AP device, or initiating a return merchandise authorization (RMA) process.

17. The network management system of claim 15, wherein the AP device is classified as an AP device with an unknown problem, and wherein based on the AP device being classified as an AP device with an unknown problem, the action comprises: Information associated with the AP device is sent to an administrator device for further analysis.

18. The network management system of claim 15, wherein the AP device is classified as an AP device that is incorrectly identified as having the problem, and wherein based on the AP device being classified as an AP device that is incorrectly identified as having the problem, the actions include: The stored status of the AP device is updated to indicate that there is no abnormality in the AP device.

19. A method of managing one or more access points, the method comprising: Obtain client activity data of multiple access point AP devices at the site; determining an anomaly in a behavior of an AP device among the plurality of AP devices based on the client activity data, wherein as part of determining the anomaly in the behavior of the AP device, the one or more processors are configured to at least one of: a) determine a predicted behavior of the AP device based on the client activity data, and compare the predicted client activity data with actual client activity data of the AP device, or b) compare client activity data of at least one peer AP device with the actual client activity data of the AP device; as well as An action is performed based on the determined anomaly in the behavior of the AP device.

20. A computer readable storage medium encoded with instructions for causing one or more programmable processors to perform the method of claim 19.

Images