Virtual Network Assistant with proactive analytics and correlation engine using ML models

Abstract

Embodiments of the present disclosure relate to a virtual network assistant with an active analysis and correlation engine using an ML model. Techniques for a network management system to process network event data received from an AP device are described. The NMS is configured to dynamically determine, in real time, a minimum (MIN) threshold and a maximum (MAX) threshold for expected occurrence for each event type, wherein the MIN threshold and the MAX threshold define the expected range of occurrence of network events for the corresponding event type. The NMS applies an unsupervised machine learning model to the network event data to determine a predicted occurrence count of network events for each event type in the event type, and identifies one or more network events in the network event as indicating abnormal network behavior based on the predicted occurrence count for each event type and the dynamically determined minimum and maximum thresholds.

Description

[0001] This application claims the benefit of U.S. Application No. 17 / 303,222, filed on May 24, 2021, and U.S. Provisional Patent Application No. 63 / 177,253, filed on April 20, 2021, the entire contents of each of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to computer networks, and more particularly, to machine learning based diagnostics of computer networks and network systems. Background Art

[0003] Wireless access networks utilize a network of wireless access points (APs), which are physical electronic devices that enable other devices to wirelessly connect to a wired network using a variety of wireless networking protocols and technologies, such as wireless local area networking protocols that conform to one or more of the IEEE 802.11 standard (i.e., "WiFi"), Bluetooth / Bluetooth Low Energy (BLE), mesh networking protocols such as ZigBee, or other wireless networking technologies. Many different types of devices, such as laptops, smartphones, tablets, wearable devices, appliances, and Internet of Things (IoT) devices, incorporate wireless communication technologies and can be configured to connect to a wireless access point when the device is within range of a compatible wireless access point in order to access a wired network.

[0004] Wireless access networks, and computer networks in general, are complex systems that may experience transient and / or permanent problems. Some problems may cause noticeable system performance degradation, while other problems may resolve themselves without significantly affecting system-level performance as perceived by users. Some problems may be expected and accepted under high load, and self-healing mechanisms (such as retries, etc.) may make the problem disappear once the load subsides. Summary of the invention

[0005] In general, the present disclosure describes techniques that enable a virtual network assistant (VNA) to execute a proactive analysis and correlation engine (PACE) that is configured to dynamically build and apply unsupervised machine learning-based (“ML-based”) models for reducing or minimizing resources spent on network diagnostics. As described herein, techniques enable the proactive analysis and correlation engine to apply unsupervised ML-based models to collected network event data to determine whether the network event data represents an expected transient network error that can be self-corrected or an abnormal behavior that requires further analysis by the virtual network assistant in order to support resolution of underlying faults in the network system.

[0006] In addition, these techniques implement adaptive closed-loop tuning of unsupervised ML-based network models by utilizing real-time network data of sliding windows divided into time series subgroups for dynamically calculating expected ranges (minimum / maximum expected occurrences) for various types of network events within a defined time period. The ML-based models applied by the VNA are trained to predict the occurrence levels for network events based on actual network event data as training data, which is enhanced with dynamically determined expected ranges for different types of network events. PACE of a network management system (NMS) applies (multiple) ML-based models to network event data received from a network system (excluding the most recent observation time frame), and operates to predict the occurrence levels expected to be seen during the current observation time frame for various types of network events and estimated (predicted) minimum and maximum thresholds for each type of network event, i.e., predicted tolerance ranges for the number of occurrences of each type of network event. Upon determining that the actual observations of network event data during the current observation period deviate beyond the range set by the minimum and / or maximum thresholds estimated (predicted) by the model during that period, the proactive analysis and correlation engine marks those network events as indicating anomalous behavior, thereby triggering a more detailed root cause analysis.

[0007] The technology of the present disclosure provides one or more technical advantages and practical applications. For example, the technology implements automated virtual network assistants that can determine which network problems should be analyzed and which problems should be considered transient problems that can be resolved by themselves and therefore ignored without expending additional computing resources.

[0008] To ensure that complex computer networks meet the needs of their user community, network managers seek to quickly resolve any problems that may arise during system operation. On the other hand, analyzing the network and trying to find the root cause of every problem can lead to a waste of computing resources because the system can over-analyze the root cause of a problem that has been rectified, for example, through a retry mechanism, before or shortly after the results of the network analyzer become available.

[0009] Further, to achieve certain technical efficiencies, the technology implements an automated virtual network assistant based on an unsupervised ML-based model, thereby reducing and / or eliminating the time-consuming work of labeling each message flow and statistics as representing a "good / normal" message flow or a "bad / failed" message flow.

[0010] In one example, the present disclosure relates to a method, comprising: receiving network event data indicating operational behavior of a network, wherein the network event data defines a series of network events of one or more event types; dynamically determining corresponding minimum (MIN) and maximum (MAX) thresholds for each event type and based on the network event data, which thresholds define an expected range of occurrence for the network event. The method also includes: building an unsupervised machine learning model based on the network event data and the dynamically determined minimum and maximum thresholds for each event type, without marking each network event in the network events of the network event data; and after building the unsupervised machine learning model, using the machine learning model to process additional network event data to determine a predicted occurrence count of the network event for each event type in the event type. The method also includes identifying one or more network events in the network events as indicating abnormal network behavior based on the predicted occurrence count and the dynamically determined minimum and maximum thresholds for each event type.

[0011] In another example, the present disclosure relates to a network management system (NMS) that manages one or more access point (AP) devices in a wireless network. The NMS includes a memory that stores network event data received from the AP device, wherein the network event data indicates operational behavior of the wireless network, and wherein the network event data defines a series of network events of one or more event types that change over time. The NMS is configured to apply an unsupervised machine learning model to the network event data to determine the following for a most recent observation time period in an observation time period: (i) a predicted occurrence count of network events for each event type in the event type, and (ii) an estimated minimum (MIN) threshold and an estimated maximum (MAX) threshold for each event type, wherein the MIN threshold and the MAX threshold define an expected range of occurrence of network events for the corresponding event type; and based on the estimated minimum threshold and the estimated maximum threshold and the actual network event data for the most recent observation time period in the observation time period, one or more network events in the network events are identified as indicating abnormal network behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example network system in which a virtual network assistant (VNA) executes a proactive analysis and correlation engine (PACE) configured to dynamically build and apply unsupervised ML-based models for network diagnostics.

[0013] Figure 2 is a block diagram of an example access point device in accordance with one or more techniques of this disclosure.

[0014] Figure 3is a block diagram of an example network management system having a VNA configured to execute PACE configured to dynamically build and apply unsupervised ML-based models for network diagnostics in accordance with one or more techniques of the present disclosure.

[0015] Figure 4 is a block diagram of an example network node, such as a router or a switch, in accordance with one or more techniques of this disclosure.

[0016] Figure 5 is a block diagram of an example user equipment device, in accordance with one or more techniques of this disclosure.

[0017] Figure 6 An example network event table is illustrated.

[0018] Figure 7 The time windows used to measure the variability associated with each network event counter are illustrated.

[0019] Figure 8 A flow chart of a process for estimating expected variability and bounds for network event counters is illustrated.

[0020] Fig. 9 The time series of the values ​​of the VPE counter including the dynamic boundaries is illustrated.

[0021] Fig.10 The time series of the values ​​of the VPE counter including the determination of the dynamic boundaries is illustrated.

[0022] Fig.11 An example diagram illustrating the training of a behavioral model of a system is shown.

[0023] Fig. 12A and Fig. 12B is a graph showing the function of VPE error

[0024] Fig.13 An example of a histogram of prediction errors is shown.

[0025] Fig.14 Illustrated is a flow chart for building a behavioral model of cyber events, including ensuring that the model excludes anomalous behavior.

[0026] Fig.15 A flow chart illustrating a process triggered when the VPE prediction error exceeds a dynamic boundary segment. DETAILED DESCRIPTION

[0027] As described herein, commercial locations (such as offices, hospitals, airports, stadiums, or retail stores) typically install complex wireless network systems throughout the premises, including a network of wireless access points (APs) to provide wireless network services to one or more client devices (or simply "clients") at the site. Clients may include, for example, smart phones or other mobile devices, Internet of Things (IoT) devices, etc. As mobile clients move throughout the premises, these mobile clients may automatically switch from one wireless access point to another wireless access point within range to provide users with seamless network connectivity throughout the premises. Additionally, if a particular wireless access point provides poor coverage, the client device may automatically attempt to connect to a different wireless access point with better coverage.

[0028] In many examples, a wireless network service provider implements a system to monitor and collect one or more network performance metrics to monitor network behavior and measure the performance of a wireless network at a site. For example, a service level expectation (SLE) metric can be used to measure various aspects of wireless network performance. The SLE metric seeks to measure and understand network performance from the perspective of the end-user experience on the network. An example SLE metric is a coverage metric that tracks the number of user minutes in which the received signal strength indicator (RSSI) of a client measured by the access point to which the client is connected is lower than a configurable threshold. Another example SLE metric is a roaming metric that tracks the percentage of successful roaming between two access points within a specified threshold of a client. Other example SLE metrics may include connection time, throughput, successful connection, capacity, AP uptime, and / or any other metric that may indicate one or more aspects of wireless network performance. The threshold may be customized and configured by a wireless service provider to define the service level expectations at a site.

[0029] According to one or more techniques described herein, a virtual network assistant (VNA) executes a proactive analysis and correlation engine (PACE) that is configured to dynamically build and apply unsupervised ML-based models for network diagnostics, i.e., machine learning models trained using unlabeled training data. As described herein, techniques enable the proactive analysis and correlation engine of the virtual network assistant to detect network problems that need to be resolved and support resolution of identified faults in the network system.

[0030] As an example, the present disclosure describes an example embodiment in which a network management system (NMS) of a complex network system (e.g., a wireless network system) implements a proactive analysis and correlation engine to determine whether problems detected in the network system are expected as part of normal operation of the system or are atypical. If the NMS determines that the problems are typical under the observed conditions, then the NMS can be configured to ignore these problems assuming that the problems can be resolved through self-healing mechanisms (such as restart or automatic reconfiguration mechanisms). However, if the NMS determines that the problems are atypical under the observed conditions, then the NMS can be configured to automatically invoke more complex and computationally expensive network analysis to determine the root cause of the problem and automatically perform remediation, such as restarting or reconfiguring one or more network components to restore a satisfactory system-level experience (SLE).

[0031] In various examples described herein, techniques can build unsupervised ML models based on network event data collected by the NMS for the network. For example, according to the techniques described herein, the NMS can automatically generate and retrain unsupervised ML models for the active analysis and correlation engine based on network events extracted from historically observed messages and / or statistics for the network. The active analysis and correlation engine of the NMS can then apply the ML model to logs of newly collected data of data streams and / or various network event types (e.g., statistics, messages, SLE metrics, etc., referred to herein as "PACE" event data or event types) to detect whether the currently observed network event data and the incoming data stream indicate normal operation of the system or whether the incoming network event data indicates an atypical system behavior event or trend corresponding to a failed network that needs to be mitigated.

[0032] As described, when the proactive analysis and correlation engine applies ML models to network event data indicating a need for mitigation, the NMS may invoke a more sophisticated root cause network analysis component of a virtual network assistant (VNA) to identify the root cause of the abnormal system behavior and, if possible, trigger automated or semi-automated corrective actions. In this way, a proactive analysis and correlation engine (PACE) can build and apply ML models based on a particular complex network, wherein PACE is deployed as a mechanism for quickly and efficiently determining whether to perform further resource-intensive analysis on incoming streams of network event data collected (e.g., in real time) from elements within a complex network system.

[0033] Further, in addition to identifying which issues require attention, some examples described herein can be configured to monitor messages exchanged within a complex network system and many operational counters and statistics. During normal operation, the ratios between the values ​​of different counters and statistics can assume values ​​within a specific range of acceptable values, referred to herein as the {Min, Max} range. As described in more detail below, one technical advantage of the techniques described herein is that the NMS can be configured to build an unsupervised ML model so that the ML model automatically determines and adjusts a dynamic {Min, Max} range of acceptable values ​​for network event data (e.g., specific statistics, counters, metrics, etc.) representing normal operation, thereby achieving improved accuracy and reliability of the ML model used to determine whether to trigger a deeper root cause analysis of the network event data.

[0034] Figure 1 is a block diagram of an example network system 100 in which a virtual network assistant (VNA) executes a proactive analysis and correlation engine (PACE) configured to dynamically build, apply, and retrain unsupervised ML-based models for network diagnostics based on real-time collected network data.

[0035] The example network system 100 includes a plurality of sites 102A-102N at which a network service provider manages one or more wireless networks 106A-106N, respectively. Figure 1 In the drawings, each site 102A-102N is shown as including a single wireless network 106A-106N, respectively, in some examples, each site 102A-102N may include multiple wireless networks, and the present disclosure is not limited in this respect.

[0036] Each site 102A-102N includes a plurality of access points (APs), generally referred to as APs 142. For example, site 102A includes a plurality of APs 142A-1 through 142A-N. Similarly, site 102N includes a plurality of APs 142N-1 through 142N-N. Each AP 142 may be any type of wireless access point, including but not limited to a commercial or enterprise AP, a router, or any other device capable of providing wireless network access.

[0037] Each site 102A-102N also includes a plurality of client devices, otherwise referred to as user equipment devices (UE), generally referred to as UE 148, thereby representing various wireless-enabled devices within each site. For example, a plurality of UEs 148A-1 to 148A-N are currently located at site 102A. Similarly, a plurality of UEs 148N-1 to 148N-N are currently located at site 102N. Each UE 148 can be any type of wireless client device, including but not limited to a mobile device (such as a smart phone, tablet computer, or laptop computer), a personal digital assistant (PDA), a wireless terminal, a smart watch, a smart ring, or other wearable device. UE 148 can also include IoT client devices, such as printers, security devices, environmental sensors, or any other device configured to communicate via one or more wireless networks.

[0038] The example network system 100 also includes various networking components for providing networking services within the wired network, including, by way of example, an authentication, authorization, and accounting (AAA) server 110 for authenticating users and / or UEs 148, a dynamic host configuration protocol (DHCP) server 116 for dynamically assigning a network address (e.g., an IP address) to the UE 148 after authentication, a domain name system (DNS) server 122 for resolving domain names to network addresses, a plurality of servers 128 (e.g., web servers, database servers, file servers, etc.), and a network management system (NMS) 136. Figure 1 As shown in FIG, various devices and systems of network 100 are coupled together via one or more networks 134 (e.g., the Internet and / or an intranet). Each of servers 110, 116, 122, and / or 128, AP 142, UE 148, NMS 136, and any other server or device attached to or forming part of network system 100 may include a system log or error log module, wherein each of these devices records the status of the device, including normal operating status and error conditions.

[0039] exist Figure 1 In the example of , NMS 136 is a cloud-based computing platform that manages wireless networks 106A-106N at one or more of sites 102A-102N. As further described herein, NMS 136 provides an integrated suite of management tools and implements various techniques of the present disclosure.

[0040] According to the techniques described herein, NMS 136 monitors the SLE metrics received from the wireless networks 106A-106N at each site 102A-102N, respectively, and manages network resources, such as AP 142 at each site, to provide a high-quality wireless experience to end users, IoT devices, and clients at the site. In general, NMS 136 can provide a cloud-based platform for network SLE data acquisition, monitoring, activity logging, reporting, predictive analysis, network anomaly identification, and alarm generation.

[0041] For example, NMS136 may include a virtual network assistant (VNA) 133 that implements an event processing platform for providing real-time insight and simplified troubleshooting for IT operations, and automatically takes corrective actions or provides recommendations to proactively solve wireless network problems. VNA 133 may, for example, include an event processing platform that is configured to process hundreds of concurrent event streams from sensors and / or agents associated with nodes within AP 142 and / or network 134. For example, according to various examples described herein, VNA 133 of NMS136 may include an underlying analysis and network error identification engine and an alarm system. The underlying analysis engine of VNA 133 may apply historical data and models to inbound event streams to calculate assertions, such as predicted occurrences of identified anomalies or events that constitute network error conditions. Further, VNA 133 may provide real-time alarms and reports to notify administrators of any predicted events, anomalies, trends, and may perform root cause analysis and automated or assisted error remediation.

[0042] Further example details of the operations implemented by the VNA 133 of the NMS 136 are described in U.S. application serial number 14 / 788,489, filed on June 30, 2015, and entitled “Monitoring Wireless Access Point Events”; U.S. application serial number 16 / 835,757, filed on March 31, 2020, and entitled “Network System Fault Resolution Using a Machine Learning Model”; U.S. application serial number 16 / 279,243, filed on February 19, 2019, and entitled “Systems and Methods for a Virtual Network Assistant”; U.S. application serial number 16 / 237,677, filed on December 31, 2018, and entitled “Methods and Apparatus for Facilitating Fault Detection and / or Predictive Fault Detection”; and U.S. application serial number 16 / 237,677, filed on January 18, 2019, and entitled “Method for Spatio-Temporal Serial No. 16 / 251,942, filed on March 8, 2019, and entitled “Method for Conveying AP ErrorCodes Over BLE Advertisements,” the entire contents of all applications are incorporated herein by reference.

[0043] In some examples, the VNA 133 of the NMS 136 can apply machine learning techniques to identify the root cause of the error condition detected or predicted from the event data stream. If the root cause can be automatically resolved, the VNA 133 invokes one or more corrective actions to correct the root cause of the error condition, thereby automatically improving the underlying SLE metrics and also automatically improving the user experience. Further example details of the root cause analysis and automatic correction techniques performed by NMS 136 are described in U.S. application serial number 14 / 788,489, filed on June 30, 2015, and entitled “Monitoring Wireless Access Point Events,” U.S. application serial number 16 / 835,757, filed on March 31, 2020, and entitled “Network System Fault Resolution Using a Machine Learning Model,” U.S. application serial number 16 / 279,243, filed on February 19, 2019, and entitled “Systems and Methods for a Virtual Network Assistant,” U.S. application serial number 16 / 237,677, filed on December 31, 2018, and entitled “Methods and Apparatus for Facilitating Fault Detection and / or Predictive Fault Detection,” U.S. application serial number 16 / 237,677, filed on January 18, 2019, and entitled “Method for No. 16 / 251,942, filed on March 8, 2019, and entitled “Method for Conveying AP Error Codes Over BLE Advertisements,” the entire contents of all applications are incorporated herein by reference.

[0044] In operation, the NMS 136 observes, collects, and / or receives event data 139, which may take the form of data extracted from messages, counters, and statistics, for example. According to one specific implementation, the computing device is part of the network management server 136. According to other implementations, the NMS 136 may include one or more computing devices, dedicated servers, virtual machines, containers, services, or other forms of environments for executing the techniques described herein. Similarly, the computing resources and components that implement the VNA 133 and PACE 135 may be part of the NMS 136, may be executed on other servers or execution environments, or may be distributed to nodes (e.g., routers, switches, controllers, gateways, etc.) within the network 134.

[0045] According to one or more techniques of the present disclosure, the Virtual Network Assistant's Proactive Analysis and Correlation Engine (PACE) 135 to dynamically build, train, apply, and retrain (multiple) unsupervised ML models 137 on event data 139 determines whether the collected network event data represents abnormal behavior that needs to be further analyzed by the VNA 133 to support root cause analysis and troubleshooting. More specifically, the NMS's PACE 135 applies (multiple) ML models 137 to the network event data 139 received from the network system 100 that does not include network events for a recent observation time frame, and the ML models 137 operate to predict the level of occurrence that is expected to be seen for each of the various types of network events during the current observation time frame. Additionally, based on the network event data 139, the ML models 137 predict estimated (predicted) minimum and maximum thresholds for each type of network event within the current observation period, i.e., a predicted tolerance range for the number of occurrences of each type of network event within the period. Upon determining that true (actual) observed deviations of network event data 139 during the current observation period exceed minimum and / or maximum thresholds predicted by ML model 137 during that period, PACE 135 of VNA 133 marks those network events as indicating anomalous behavior, thereby triggering VNA 133 to perform root cause analysis on those events.

[0046] The technology of the present disclosure provides one or more advantages. For example, the technology enables the automated virtual network assistant 133 to accurately determine which potential network problems should be subject to a deeper root cause analysis and which problems should be treated as noise or transient problems that can be resolved in the normal process and can therefore be ignored. Further, in order to achieve certain technical efficiencies, the technology implements an automated virtual network assistant based on an unsupervised machine learning (ML) model, thereby reducing and / or eliminating the time-consuming work of labeling each message flow and statistics as representing a "good / normal" message flow or a "bad / failed" message flow. In addition, the technology supports automatic retraining of ML-based models to adapt to changing network conditions, thereby eliminating false positives that may otherwise occur and lead to over-allocation of resources associated with root cause analysis.

[0047] Although the techniques of the present disclosure are described in this example as being performed by NMS 136, it should be understood that the techniques described herein may be performed by any other computing device(s), system(s), and / or server(s), and the present disclosure is not limited in this regard. For example, one or more computing devices configured to perform the functionality of the techniques of the present disclosure may reside in a dedicated server or be included in any other server other than or in addition to NMS 136, or may be distributed throughout network 100 and may or may not form part of NMS 136.

[0048] In some examples, a network node (e.g., a router or switch within network 134) or even access point 142 may be configured to locally build, train, apply, and retrain unsupervised ML model(s) 137 based on locally collected SLE metrics to determine whether collected network event data should be discarded or whether the data indicates a need to be forwarded to NMS 136 for further root cause analysis on VNA 350 ( Figure 2 ) to support the identification of faults and resolution of abnormal behavior.

[0049] Figure 2 is a block diagram of an example access point (AP) device 200 configured in accordance with one or more techniques of this disclosure. Figure 2 The example access point 200 shown in FIG. 1 may be used to implement the Figure 1 Any AP 142 is shown and described. Access point 200 may include, for example, a Wi-Fi, Bluetooth, and / or Bluetooth Low Energy (BLE) base station or any other type of wireless access point.

[0050] exist Figure 2In the example of FIG. 1 , access point 200 includes a wired interface 230, wireless interfaces 220A-220B, one or more processors 206, memory 212, and user interface 210 coupled together via bus 214, through which the various elements can exchange data and information. Wired interface 230 represents a physical network interface and includes a receiver 232 and a transmitter 234 for sending and receiving network communications, such as packets. Wired interface 230 directly or indirectly couples access point 200 to Figure 1 The first wireless interface 220A and the second wireless interface 220B represent wireless network interfaces and include receivers 222A and 222B, respectively, each of which includes a receiving antenna, via which the access point 200 can receive signals from wireless communication devices such as Figure 1 The first wireless interface 220A and the second wireless interface 220B further include transmitters 224A and 224B, respectively, each of which includes a transmission antenna, and the access point 200 can transmit wireless signals to wireless communication devices (such as Figure 1 UE 148). In some examples, the first wireless interface 220A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz) and the second wireless interface 220B may include a Bluetooth interface and / or a Bluetooth Low Energy (BLE) interface.

[0051] The processor(s) 206 are programmable hardware-based processors configured to execute software instructions, such as those used to define software or computer programs, stored to a computer-readable storage medium, such as the memory 212, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause the processor(s) 206 to perform the techniques described herein.

[0052] Memory 212 includes one or more devices configured to store programming modules and / or data associated with the operation of access point 200. For example, memory 212 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause one or more processors 206 to perform the techniques described herein.

[0053] In this example, the memory 212 stores executable software, including an application programming interface (API) 240, a communication manager 242, configuration settings 250, a device status log 252, and a data storage device 254. The device status log 252 includes a list of events specific to the access point 200. The events may include a log of both normal events and error events, such as, for example, memory status, reboot events, crash events, Ethernet port status, upgrade failure events, firmware upgrade events, configuration changes, etc., as well as a time and date stamp for each event. The log controller 255 determines the logging level of the device based on instructions from the NMS 136. The data 254 may store any data used and / or generated by the access point 200, including data collected from the UE 148, such as data used to calculate one or more SLE metrics, which are transmitted by the access point 200 for cloud-based management of the wireless network 106A by the NMS 136.

[0054] The communication manager 242 includes program code that, when executed by the processor(s) 206, allows the access point 200 to communicate with the UE 148 and / or the network(s) 134 via any of the interface(s) 230 and / or 220A-220C. The configuration settings 250 include any device settings for the access point 200, such as radio settings for each of the wireless interface(s) 220A-220C. These settings may be manually configured or may be remotely monitored and managed by the NMS 136 to optimize wireless network performance on a regular (e.g., hourly or daily) basis.

[0055] Input / output (I / O) 210 represents physical hardware components capable of interacting with a user, such as buttons, a display, etc. Although not shown, memory 212 typically stores executable software for controlling the user interface with respect to input received via I / O 210 .

[0056] As described herein, the AP device 200 can measure SLE related data (i.e., network event data) from the status log 252 and report it to the NMS 136. The SLE related data may include various parameters indicating the performance and / or status of the wireless network. The parameters may be measured and / or determined in the wireless network by one or more UE devices and / or one or more APs 200. The NMS 136 determines one or more SLE metrics based on the SLE related data received from the APs in the wireless network and stores the SLE metrics as event data 139 ( Figure 1). According to one or more techniques of the present disclosure, PACE 135 of NMS 136 analyzes SLE metrics (i.e., event data 139) associated with the wireless network to dynamically build, train, apply, and retrain (multiple) unsupervised ML models 137 to determine whether the collected network event data indicates abnormal behavior that needs to be further analyzed by VNA 133 to support root cause analysis and fault resolution.

[0057] Figure 3 An example network management system (NMS) 300 configured according to one or more techniques of the present disclosure is shown. NMS 300 may be used to implement, for example Figure 1 In this example, NMS 300 is responsible for monitoring and managing one or more wireless networks 106A-106N at sites 102A-102N, respectively. In some examples, NMS 300 receives data collected by AP 200 from UE 148, such as data used to calculate one or more SLE metrics, and analyzes the data for cloud-based management of wireless networks 106A-106N. In some examples, NMS 300 may be Figure 1 or a portion of another server shown in or a portion of any other server.

[0058] NMS 300 includes a communication interface 330, one or more processors 306, a user interface 310, a memory 312, and a database 318. The various elements are coupled together via a bus 314, through which the various elements can exchange data and information.

[0059] The processor(s) 306 execute software instructions, such as those used to define software or a computer program, stored to a computer-readable storage medium, such as the memory 312, such as a non-transitory computer-readable medium including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause the processor(s) 306 to perform the techniques described herein.

[0060] The communication interface 330 may include, for example, an Ethernet interface. The communication interface 330 couples the NMS 300 to a network and / or the Internet, such as Figure 1 The communication interface 330 includes a receiver 332 and a transmitter 334, through which the NMS 300 communicates to / from the AP 142, the servers 110, 116, 122, 128, and / or any other device or system forming part of the network 100 (such as Figure 1106N). The data and information received by the NMS 300 may include, for example, SLE-related or event log data received from the access point 200, which is used by the NMS 300 to remotely monitor the performance of the wireless networks 106A-106N. The NMS may also transmit data to any network device (such as the AP 142 at any network site 102A-102N) via the communication interface 330 to remotely manage the wireless networks 106A-106N.

[0061] The memory 312 includes one or more devices configured to store programming modules and / or data associated with the operation of the NMS 300. For example, the memory 312 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause the one or more processors 306 to perform the techniques described herein.

[0062] In this example, memory 312 includes API 320, SLE module 322, virtual network assistant (VNA) / AI engine 350, radio resource management (RRM) engine 360, and root cause analysis engine 370. NMS 300 may also include any other programming modules, software engines, and / or interfaces configured for remote monitoring and management of wireless networks 106A-106N, including remote monitoring and management of any AP 142 / 200.

[0063] The SLE module 322 is capable of setting and tracking thresholds for the SLE metrics for each network 106A-106N. The SLE module 322 further analyzes the SLE-related data collected by the AP, such as any AP 142 from the UE in each wireless network 106A-106N. For example, APs 142A-1 to 142A-N collect SLE-related data from UEs 148A-1 to 148A-N currently connected to the wireless network 106A. The data is transmitted to the NMS 300, which is executed by the SLE module 322 to determine one or more SLE metrics for each UE 148A-1 to 148A-N currently connected to the wireless network 106A. One or more SLE metrics can also be aggregated to each AP at the site to gain a deeper understanding of the contribution of each AP to the wireless network performance at the site. The SLE metric tracks whether the service level meets the threshold configured for each SLE metric. Each metric can also include one or more classifiers. If the metric does not meet the SLE threshold, then the failure may be attributed to one of the classifiers to further understand where the failure occurred.

[0064] Example SLE metrics and their classifiers that may be determined by the NMS 300 are shown in Table 1.

[0065] Table 1

[0066]

[0067]

[0068] The RRM engine 360 ​​monitors one or more metrics for each site 106A-106N in order to understand and optimize the RF environment at each site. For example, the RRM engine 360 ​​can monitor the coverage and capacity SLE metrics for the wireless network 106 at the site 102 to identify the potential problems of the SLE coverage and / or capacity in the wireless network 106, and adjust the radio settings of the access point at each site to solve the identified problems. For example, the RRM engine can determine the channel and transmission power distribution across all APs 142 in each network 106A-106N. For example, the RRM engine 360 ​​can monitor events, power, channels, bandwidth, and the number of clients connected to each AP. The RRM engine 360 ​​can also automatically change or update the configuration of one or more APs 142 at the site 106, with the purpose of improving the coverage and capacity SLE metrics, and thereby providing an improved wireless experience for users.

[0069] The VNA / AI engine 350 analyzes the data received from the AP 142 / 200 and its own data to identify when an undesirable abnormal state is encountered in one of the wireless networks 106A-106N. For example, the VNA / AI engine 350 can use the root cause analysis module 370 to identify the root cause of any undesirable or abnormal state. In some examples, the root cause analysis module 370 uses artificial intelligence-based technology to help identify the root cause of (multiple) any bad SLE metrics at one or more wireless networks 106A-106N. In addition, the VNA / AI engine 350 can automatically call one or more corrective actions intended to solve the (multiple) identified root causes of one or more bad SLE metrics. Examples of corrective actions that can be automatically called by the VNA / AI engine 350 may include, but are not limited to: calling RRM 360 to restart one or more APs, adjusting / modifying the transmission power of a specific radio in a specific AP, adding SSID configuration to a specific AP, changing the channel on an AP or AP set, etc. Corrective actions can also include restarting switches and / or routers, calling to download new software to APs, switches or routers, etc. These corrective actions are given for example purposes only, and the present disclosure is not limited in this respect.If automatic corrective actions are not available or do not adequately address the root cause, the VNA / AI engine 350 can proactively provide notifications including recommended corrective actions to be taken by IT personnel to resolve the network error.

[0070] In accordance with one or more techniques of the present disclosure, PACE 335 of a virtual network assistant is used to dynamically build, train, apply, and retrain unsupervised ML model(s) 337 on event data (SLE metrics 316) to determine whether the collected network event data indicates abnormal behavior that needs to be further analyzed by root cause analysis 370 of VNA 350 to support identification and resolution of faults.

[0071] The technology of the present disclosure provides one or more advantages. For example, the technology enables the automated virtual network assistant 350 to accurately determine which potential network problems should undergo a deeper root cause analysis 370 and which problems should be treated as noise or transient problems that can be resolved in the normal process and can therefore be ignored. Further, in order to achieve certain technical efficiencies, the technology implements an automated virtual network assistant based on an unsupervised machine learning (ML) model 337, thereby reducing and / or eliminating the time-consuming work of labeling each message flow and statistics as representing a "good / normal" message flow or a "bad / failed" message flow. In addition, the technology supports automatic retraining of ML-based models to adapt to changing network conditions, thereby eliminating false positives that may otherwise occur and cause over-allocation of resources associated with root cause analysis.

[0072] Figure 4 An example user equipment (UE) device 400 is shown. Figure 4 The example UE device 400 shown in FIG. 4 may be used to implement the Figure 1 Any UE 148 shown and described. UE device 400 may include any type of wireless client device, and the present disclosure is not limited in this regard. For example, UE device 400 may include a mobile device (such as a smart phone, tablet computer, or laptop computer), a personal digital assistant (PDA), a wireless terminal, a smart watch, a smart ring, or any other type of mobile or wearable device. UE 400 may also include any type of IoT client device, such as a printer, a security sensor or device, an environmental sensor, or any other connected device configured to communicate over one or more wireless networks.

[0073] According to one or more techniques of the present disclosure, one or more SLE parameter values ​​(i.e., data used by NMS 136 to calculate one or more SLE metrics) are received from each UE 400 in the wireless network. Figure 1 The NMS 136 receives one or more SLE parameter values ​​from the UE 148 in the network 106A-106N. In some examples, the NMS 136 receives the SLE parameter values ​​from the UE 148 on a continuous basis, and the NMS can calculate one or more SLE metrics for each UE on a periodic basis defined by a first predetermined time period (e.g., every 10 minutes or other predetermined time period).

[0074] UE device 400 includes a wired interface 430, wireless interfaces 420A-420C, one or more processors 406, memory 412, and a user interface 410. The various components are coupled together via a bus 414, through which the various components can exchange data and information. The wired interface 430 includes a receiver 432 and a transmitter 434. If necessary, the wired interface 430 can be used to couple the UE 400 to Figure 1 The first wireless interface 420A, the second wireless interface 420B, and the third wireless interface 420C include receivers 422A, 422B, and 422C, respectively, each of which includes a receiving antenna, via which the UE 400 can receive data from a wireless communication device such as Figure 1 AP 142 Figure 2 The first wireless interface 420A, the second wireless interface 420B and the third wireless interface 420C also include transmitters 424A, 424B and 424C, respectively, each of which includes a transmission antenna, and the UE 400 can transmit wireless signals to wireless communication devices (such as AP 200, other UE 148 or other devices configured for wireless communication). Figure 1 AP 142 Figure 2 The first wireless interface 420A may include a Wi-Fi 802.11 interface (e.g., 2.4 GHz and / or 5 GHz) and the second wireless interface 420B may include a Bluetooth interface and / or a Bluetooth low energy interface. The third wireless interface 420C may include, for example, a cellular interface through which the UE device 400 can connect to a cellular network.

[0075] The processor(s) 406 execute software instructions, such as those used to define software or a computer program, stored to a computer-readable storage medium, such as the memory 412, such as a non-transitory computer-readable medium including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause the processor(s) 406 to perform the techniques described herein.

[0076] The memory 412 includes one or more devices configured to store programming modules and / or data associated with the operation of the UE 400. For example, the memory 412 may include a computer-readable storage medium, such as a non-transitory computer-readable medium, including a storage device (e.g., a disk drive or an optical drive) or a memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory, that stores instructions to cause the one or more processors 406 to perform the techniques described herein.

[0077] In this example, the memory 412 includes an operating system 440, an application 442, a communication module 444, a configuration setting 450, and a data storage device 454. The data storage device 454 may include, for example, a status / error log including a list of events and / or SLE-related data specific to the UE 400. Depending on the logging level based on instructions from the network management system, the events may include logs of both normal events and error events. The data storage device 454 may store any data used and / or generated by the UE 400, such as data used to calculate one or more SLE metrics, which are collected by the UE 400 and transmitted to any AP 138 in the wireless network 106 for further transmission to the NMS 136.

[0078] The communication module 444 includes program code that, when executed by the processor(s) 406, enables the UE 400 to communicate using any of the wired interface(s) 430, the wireless interfaces 420A-420B, and / or the cellular interface 450C. The configuration settings 450 include any device settings that the UE 400 sets for each of the wireless interfaces 420A-420B and / or the cellular interface 420C.

[0079] Figure 5 is a block diagram illustrating an example network node 500 configured according to the techniques described herein. In one or more examples, the network node 500 implements a Figure 1 A device or server of the network 134, such as a router, a switch, an AAA server, a DHCP server, a DNS server, a VNA, a Web server, etc., or a network device, such as a router, a switch, etc. In some embodiments, Figure 4 The network node 400 is Figure 1 Server 110, 116, 122, 128 or Figure 1 The network 134 routers / switches.

[0080] In this example, the network node 500 includes a communication interface 502 (e.g., an Ethernet interface), a processor 506, an input / output 508 (e.g., a display, a button, a keyboard, a keypad, a touch screen, a mouse, etc.), a memory 512, and a component 516 assembly, such as an assembly of hardware modules, such as an assembly of circuits, coupled together via a bus 509, and the various elements can exchange data and information through the bus. The communication interface 502 couples the network node 500 to a network, such as an enterprise network. Although only one interface is shown by way of example, it should be recognized by those skilled in the art that a network node can and usually does have multiple communication interfaces. The communication interface 502 includes a receiver 520, via which the network node 500 (e.g., a server) can receive data and information, such as operation-related information, such as registration requests, AAA services, DHCP requests, simple notification service (SNS) lookups, and web page requests. The communication interface 502 includes a transmitter 522, via which the network node 500 (e.g., a server) can send data and information, such as configuration information, authentication information, web page data, etc.

[0081] Memory 512 stores executable software applications 532, operating system 540, and data / information 530. Data 530 includes system logs and / or error logs that store SLE metrics (event data) for node 500 and / or other devices (such as wireless access points) based on logging levels according to instructions from a network management system. In some examples, network node 500 may forward the SLE metrics to a network management system (e.g., Figure 1 Alternatively or additionally, the network node 500 may provide a platform for executing PACE 135 to locally build, train, apply, and retrain (multiple) unsupervised ML models 337 based on the data 530 (SLE metrics) to determine whether the collected network event data should be discarded, or whether the data indicates a need to be forwarded to the NMS 136 for further root cause analysis on the VNA 350 ( Figure 2 ) to support the identification of faults and resolution of abnormal behavior.

[0082] PACE Events

[0083] Figure 6 is a simplified example table 600 of network events collected and used by a PACE, such as PACE 135, according to the techniques described herein. NMS 136 extracts events from messages received from access points 142 and / or network nodes, such as routers and switches of network 134. Messages notifying NMS 136 of the occurrence of network events may arrive at any time after a network event, such as one related to DNS, DHCP, ARP, etc. activity, has been experienced by any component of the network.

[0084] Example message 600 includes an ID that includes an index number for each network event listed in the row of column 620. The simplified table includes 15 network events, but the number of events may be much larger, where the number of events depends on the number of events included in the training database. Column 610 provides indexes for events that can be used to simplify references to specific events. Column 630 of event dictionary table 600 provides text that can be displayed to support IT technicians or system administrators in servicing the system.

[0085] Column 640 includes data specifying the type of network event and helps categorize the event into a particular group of related events, and column 650 provides a more detailed textual description of the nature of each network event. Information from comment column 650 may be useful to technicians who may need to service the system but are not used by the automation system described in more detail below.

[0086] Estimating expected variability and bounds for network event counters

[0087] Figure 7 Illustrated are exemplary time windows 700A, 700B, 700C through 700m+1 that are used by PACE 135 to dynamically measure the variability associated with the counter values ​​for each different type of network event in real time (i.e., as event data 139 is collected within network system 100). In this example, the first time window of time window sequence 700A begins at time t0 and lasts for a duration of w seconds. Similarly, successive time windows begin at time t 0+w starts at time t and continues until time t 0+2w During each time window, PACE 135 monitors the arriving network events and maintains a count of the number of occurrences of each type of network event during the particular time window. In some examples, PACE 135 creates and stores the counts as a vector of network events (referred to herein as a vector of PACE events or VPEs):

[0088] VPE(t) = [c1, c2, c3, … c n Equation 1

[0089] in:

[0090] VPE(t) - network event vector at time t,

[0091] c i - the number of first network events i that occurred during said time

[0092] Window t,

[0093] t - timestamp indicating the start of the time window

[0094] i - index of network event, such as Figure 6 The index of column 610

[0095] n – number of network events.

[0096] As referenced below Figure 8 As explained in more detail in flowchart 800 of FIG. 1 , since the time origin t0 is arbitrarily chosen, PACE 135 checks and determines if the origin of the time window starts at a slightly different time, then the counters c1 to c n In this way, PACE 135 dynamically measures the variability of counter values ​​for a time series of event data, such as a real-time stream of event data received from APs and network nodes of network system 100, using a sliding window of overlapping time windows 700a-700m offset by time increments.

[0097] Figure 8 8 is a flow chart illustrating an example process 800 performed by PACE 135 for estimating expected variability and acceptable bounds for network event counters. The specific value of each event counter parameter / element depends on the specific starting point at time t0, where the time window is set to begin. For example, referring to Figure 7 , each of the event counters is used to establish the network event vector described in Equation 1. The event counters measured by PACE 135 within a time window of time window sequence 700a will be different from the event counters measured within a time window of time window sequence 700b (which begins after delta seconds) or within a time window of time window sequence 700(m+1) (which begins before delta seconds).

[0098] The process 800 performed by PACE 135 for processing event data 139 begins at operation 805 and continues to operation 810, where PACE 135 determines initial time window parameters, such as a starting point t0, a window duration (e.g., 2 weeks), a start time increment increment (e.g., 5 minutes), and a range (limit) within which to vary the start time t0. PACE 135 may determine these values ​​based on configuration data provided by a system administrator, data scientist, or other user. Although the following process refers to specific elements of VPE c i , but it should be understood that PACE 135 performs the same process for each of the n elements of the PACE vector (ie, for each counter).

[0099] Once PACE 135 determines the initial parameters in operation 810, PACE 135 proceeds to step 815 where the system measures, accumulates, and stores the total number of network event counters for each time window in the sequence of time windows. An example of such multivariate time series data generated and stored by PACE 135 is shown in FIG. Fig. 9 In some examples, the measured and observed PACE counters are stored in a table, such as Fig.10 The table shown in the figure.

[0100] In the first iteration through step 820 (which is Fig.10 1050), only the initial values ​​of Table 10 are present, and thus, PACE 135 will be measured by the parameter c shown in column 1015. i (In step 825, c is represented by column 1020. i Max) is set to each c in the corresponding time window i Similarly, in the first iteration through step 830, only c iInitial parameters of t0 exist, and as such, PACE 135 sets CiMin illustrated in column 1025 to the corresponding measured VPE parameter c illustrated by row 1050 within the corresponding time window t0 in operation 835. i These initial parameters are assigned in Fig.10 is shown in .

[0101] According to an example implementation, the VPE counter value obtained when the origin of the time window is at t0 is used as C i * , and the VPE vector constructed from these counter values ​​is called VPE*.

[0102] PACE 135 proceeds to step 840 where a new start time point is set by incrementing or decrementing the start time point by increment seconds. In operation 845, PACE 135 determines whether the new start time of the window is still within the limits used to vary the start time of the time window. Or in other words, whether the time origin is still within the time period [t0-m increment, ..., to, ... t0+m increment]. If so, then PACE 135 loops back to operation 815 and measures VPE values ​​in each new time window that has a different start point than the previous time window. Examples of these measurements are shown in FIG. Fig.10 As shown in row 1055, the measurement or count of each element of the VPE is [c1', c2', ..c n ']. Assuming c1'>c1, PACE 135 sets c1' to c1 max, assuming c2>c2', the method sets c2' to c2 min, and assuming c n '>c1, the method will c n 'Set to c n Similarly, for the time window illustrated by row 1060, assuming c1″>c1′, PACE 135 sets c1″ to c1 max, assuming c2″ falls between c2 and c2′, PACE 135 does not change the entry in the table, and assumes c n ” <c n , PACE 135 will c n ” is set to c1 min.

[0103] PACE 135 continues to iterate through operations 815 , 820 , 825 , 840 , and 845 until the shift of the start point of the time window covers a predetermined time period. The duration is based on the threshold used to vary the start time of the time window set in the initial operation 810 .

[0104] Once the entire range of start times t0 is covered, PACE 135 moves to operation 850. For example, Fig. 9shows the time period covering up to t 0+m增量 time, where m is a predetermined number. A similar time period can be used to shift the start time back to t 0-m增量 (Not shown in the figure.) In step 850, PACE 135 determines the variability of the count for each network event type (ie, SLE parameter), specifically, determines the corresponding ci Max and ci Min values ​​for each VPE counter in each time window based on the values ​​set in the previous step.

[0105] PACE 135 proceeds to operation 855 where VPE*, VPE Max, and VPE Min are stored, thereby augmenting the unlabeled network event data 139 with the determined time-based variability for the network events to produce augmented training data for training an ML model 137 for predicting estimated counts of network events and estimated minimum and maximum thresholds defining corresponding ranges of expected counts for each network type.

[0106] In one example implementation, PACE 135 ends at operation 860. According to another example implementation, PACE 135 may be configured to repeat method 800 every W seconds and continuously generate a time series of corresponding vectors VPE*, VPE Min, and VPE Max for adaptively rebuilding ML model 137 in view of real-time network event data.

[0107] Fig. 9 800 discussed above, obtained by PACE 135. In this example, the x-axis 910 provides time and, in one example implementation, provides time t0 for a corresponding VPE counter value. The y-axis 920 provides a specific c j For simplicity, only a single counter c j Specifically, for each origin time along the x-axis 910, the graph illustrates the values ​​of VPE* 930 and the dynamically determined values ​​of VPE Max 940 and VPE Min 950.

[0108] Training system behavior models

[0109] Fig.11 1 is a block diagram of a component 1100 of PACE 135 operating to train a behavioral ML model 137. As described above, the time series of historical values ​​of the VPE counter 1110 generated by PACE 135 is used as an input to a dynamic boundary determination module 1115. As explained in more detail above, each vector of PACE counters generated by PACE 135 corresponds to a set of sliding time windows separated by w seconds. At this time, the dynamic boundary determination module outputs a value Cj *、C j Max and C j Min. (C j *The value is actually the same as C j Same). As referenced below Fig. 12A and Fig. 12B As explained, these values ​​are used as input to module 1150 to derive a function of the VPE prediction error.

[0110] In operation, PACE 135 applies system behavior model 1120 (i.e., ML model 137) to received event data 139 (referred to as historical VPE counters 1110), which produces as output 1130 an estimated (e.g., “predicted”) value for the current value of VPE:

[0111] [VPE t-k , VPE t-k+1 , …VPE t-1 ] -> Behavioral Model -> Predicted VPE t Equation 2

[0112] in

[0113] [VPE t-k ,VPE t-k+1 ,…VPE t ] is the time series of the historical VPE counter values, and the behavioral model is Fig.11 Module 1120, and

[0114] Predicting VPE t is a predicted value of the VPE counter based on historical values ​​of the VPE counter.

[0115] In some example implementations, the dimension of the estimated (output) VPE vector 1130 may be the same as the value of the input VPE vector 1110. According to another example implementation, the dimension of the estimated (output) VPE vector is less than the dimension of the input vector. For example, assume that the input vector consists of the following two values: the number of clients and the number of clients that failed to associate with the AP. Therefore, the input VPE has a dimension of 2. According to one of the example implementations, the estimated VPE may only contain the number of clients that failed to associate, so the predicted VPE has a dimension of 1.

[0116] The current VPE and the estimated VPE are used as input to the subtraction module 1140. It should be noted that when the dimensions of the VPE and the estimated VPE are different, only the elements common to both vectors are utilized by the subtraction module 1140.

[0117] The output of the subtraction module 1140 is an error vector representing the VPE prediction error as follows:

[0118] VPEt Error = VPE t – Predicting VPE t Equation 3

[0119] in:

[0120] VPE t Error-VPE prediction error

[0121] VPE t - Network element vector at time t

[0122] Predicting VPE t - prediction vector of the network element at time t

[0123] In this way, the VPE prediction error represents the predicted (expected) vector predicted VPE of the network element t (including MAX and MIN components) and the observation vector VPE of the network element t The VPE prediction error is used as an input to module 1150, which produces a vector as a function of the VPE prediction error, which PACE 135 uses to drive the network event behavior model 1120 based on the actual observed network event data 139 ( Figure 1 Adaptation of the ML model 139).

[0124] Fig. 12A and Fig. 12B 12 is a graph illustrating different functions of VPE error. For simplicity, only one dimension of the VPE error vector is depicted. The x-axis 1210 provides the VPE error. The function of the VPE error is provided on the y-axis 1220. The origin of the x-axis is at the point where the predicted VPE value is equal to the VPE* for the corresponding time. Dynamic thresholds VPE Min 1230 and VPE Max 1240 are marked on the x-axis, where VPE Min is less than VPE* and VPE Max is greater than VPE*.

[0125] As from Fig. 12A It can be seen that the VPE error obeys the first function 1250 when it is between VPE Min and VPE Max, and obeys the second function 1260 when the VPE error is greater than VPE Max or less than VPE Min.

[0126] Fig. 12B A specific rectified linear unit (ReLU) function is illustrated. The graph uses the same x and y axes 1210 and 1220 and the same numerical signs for VPE* 1225, VPE min 1230, and VPE Max 1240. The VPE error function is set to zero when the prediction error is as follows:

[0127] VPE Min < VPE Error < VPE Max Equation 4

[0128] And similarly, when the VPE error is greater than VPE Max or less than VPE Min, the VPE error function is set to a linear function of the VPE error. Equation 5 below illustrates an example of such a function.

[0129]

[0130] In either case, a function of the VPE error drives the adaptation of the parameters of the ML model 137 by PACE 135. In one particular example implementation, the system behavior model represented by the ML model 137 is an AI-driven model, such as a long short-term memory (LSTM). Other implementations may utilize other AI and machine learning (ML) algorithms to adapt the system behavior model 1120 ( Figure 1 Parameters of the ML model 137) are adjusted to minimize predicted VPE errors when network event data 139 is received and processed according to the techniques described herein.

[0131] According to one example implementation, the parameters of the system behavior model are continuously adapted based on new values ​​of counters that are streamed to NMS 136. According to another example implementation, the process of creating a system behavior model (adapting the parameters of the model based on counters calculated for historical time series of network event data) is performed periodically (e.g., biweekly) based on recorded network event data. Given that in some deployments, the parameters within the system behavior model (ML 137) may shift rather slowly, if at all, periodically updating the behavior model by PACE 135 may save computing resources and still provide an appropriate representation of network system 100.

[0132] In some examples, to reduce required computing resources, PACE 135 may use only a subset of the VPEs instead of the entire set.

[0133] In this manner, PACE 135 builds and adapts ML models 137 (system behavior models) to capture normal operating ratios between various network events. For example, the number of DNS failures relative to the total number of clients and the total number of failures and successes. Specifically, in one aspect of the system behavior model, PACE 135 updates the model so that the model learns the "normal" number of DNS failures expected for a given total number of clients and the total number of successes. More specifically, in general, NMS 138 and (particularly) event data 139 store information about each individual network event, such as a client's failed attempts, such as attempts to roam, attempts to obtain authentication, attempts to obtain an IP address, etc. PACE 135 applies and adapts ML models 137 as described herein to train the model to accurately predict whether network event data (e.g., failed mobility, failed authentication, etc.) represents a network event within the "normal" range, and so, expected behavior (i.e., expected transient failures due to current operating conditions) or whether one or more of these failures are abnormal and require further analysis and / or mitigation by VNA 133.

[0134] System behavior model updating and verification using error histograms

[0135] According to one specific implementation, the PACE 135 of the NMS 136 builds a system behavior model (ML model 137) based on the event data 139 collected from the plurality of sites 102. Such a system behavior model is referred to herein as a global system behavior model. According to one implementation, once the global model is built, its parameters are fixed, and the VPE data (or just a subset of the data) used to build the global model is run through the model, thereby recording prediction errors. Similarly, according to another implementation, the PACE 135 records the value of the prediction error only if the value of the prediction error is outside the VPE, outside the dynamically determined range [VPE Min, VPE Max].

[0136] Fig.14 is a flowchart 1400 illustrating an example process by which an active analysis and correlation engine (e.g. Figure 1 The PACE 135 of the example adaptively updates the behavior model (e.g., ML model 137) based on the network event ML. For the purpose of example, the example process will be about Figure 1 PACE 135 is described.

[0137] In this example, PACE 135 begins at operation 1405 and proceeds to operation 1410, where PACE 135 receives or otherwise collects a new time series of network event data for a recent time period (e.g., for the last two weeks). At operation 1415, PACE 135 divides the information from the time series into two subsets, also referred to herein as time series subsets, based on timestamps, each having a different date / time range. PACE 135 uses the first subset (e.g., the first two-thirds of the collected network event data) to build a PACE behavior model (ML model 137) in operation 1420 and uses the second subset (e.g., the remaining one-third of the data) to test the built behavior model in operation 1425 and build a prediction error histogram / probability function in operation 1430.

[0138] In some examples, PACE 135 generates and utilizes the prediction error probability function for at least two different technical advantages: a) ensuring that the network events used to construct the PACE behavior model are associated with normal behavior, and b) ensuring that the parameters tuned for the PACE behavior model provide an appropriate representation for the underlying behavior (including rates) of the various network events. As explained in more detail below, the validation process performed by PACE 135 utilizes a histogram of prediction errors, such as the prediction errors at the output of subtraction module 1140. According to yet another implementation, PACE 135 constructs a histogram based on the output of the function of VPE error module 1150. An example of such a prediction error histogram is shown in FIG. Fig.13 13. The x-axis 1310 depicts the prediction error (or in another specific implementation, the amount by which the prediction exceeds the dynamically determined range [VPE Min, VPE Max]). The y-axis 1320 provides the probability of a particular error value occurring during operation across all sites using a second subset of the network event data (e.g., the subset used to test the validity of the PACE behavior model). According to one aspect of the invention, the fact that most errors have small values ​​provides validation that the system model was successfully built.

[0139] In operation 1435, PACE 135 compares the histogram / probability of VPE errors from the newly built model to the error histogram / probability obtained as part of the previous behavioral model training. If, in decision operation 1440, PACE 135 determines that the new error probability distribution / histogram is not similar to the previous error distribution (e.g., via use of the Kullback-Leibler divergence measure as described below), then PACE 135 proceeds to operation 1445 and excludes anomalous / outlier data points / events from the training data. In some examples, PACE 135 may generate an output to notify a user, administrator, or data scientist about the circumstances of the event to guide the user in excluding the anomalous / outlier data points. PACE 135 loops back to operation 1415 and repeats the training process using the cleaned data excluding the outlier events, thereby adaptively retraining the ML model 137 as needed based on the real-time network event data.

[0140] Once PACE 135 determines that the new prediction error is similar to the previous prediction error distribution function (decision operation 1440), PACE 135 terminates the training process and proceeds to operation 1450 and begins processing subsequent event data using the newly constructed VPE behavior model. For example, when the network event system behavior model is updated regularly, such as once every 2 weeks, the currently constructed histogram of prediction errors is compared to the previous prediction errors, such as the prediction error histogram from 2 weeks, 4 weeks, 6 weeks, etc. ago. According to yet another implementation, if the two histograms are highly similar or correlated, then the new network event behavior model is validated and the parameters of the newly constructed system are used. PACE 135 completes the retraining and reconstructing cycle in operation 1455.

[0141] In one specific implementation, PACE 135 performs the comparison of operation 1440 by determining a correlation between past and current prediction errors. If the correlation is greater than a certain threshold, such as 0.8, then PACE 135 determines that the histograms are sufficiently similar. According to another example implementation, the method utilizes information theory to determine the similarity between histograms representing error probabilities. Specifically, the method employs Kullback-Leibler divergence (also known as relative entropy or KLD) to determine a measure of how different one probability distribution is from a second reference probability distribution based on past histograms.

[0142] Use system behavior models to identify behavioral anomalies

[0143] Fig.1515 is a flowchart 1500 illustrating an example process performed by the virtual network assistant 133 for performing root cause analysis of network event data triggered by PACE 135, i.e., when PACE 135 applies ML model 137 and calculates a VPE prediction error that is outside of a dynamically determined range (boundary) for one or more types of network events, thereby indicating that a deeper analysis of anomalous network behavior associated with these network events is required. For purposes of example, the example process will be described with respect to Figure 1 The VNA 133 and PACE 135 are described.

[0144] In this example, the PACE 135 of the VNA 133 starts at operation 1505 and proceeds to operation 1510, where a system behavior model is built and parameters are fixed as described herein. In operation 1515, the PACE 135 feeds real-time VPE information (vectorized input generated from network event data 139) into the ML model 137 and determines the VPE prediction error in step 1520. The PACE 135 proceeds to decision operation 1525 and checks whether the VPE prediction error based on the predicted VPE generated by the ML model 137 is within the dynamically determined range [VPE Min, VPE Max] for each parameter.

[0145] If PACE 135 determines that the VPE prediction error falls within the dynamically determined range, then PACE 135 loops back to operation 1515 and similarly processes any newly received network event data 139. In this way, as long as the prediction error is within the dynamically determined range, the behavior of the entire network system 100 is considered normal, even though individual clients in the system may report failures, such as ARP failures, DHCP failures, or DNS failures. For example, in a system with a large (e.g., 1000 clients), it is normal to experience a few (e.g., 1 or 2) failures (such as the above-mentioned failures).

[0146] However, if the VPE prediction error exceeds the prediction error dynamic range established by [VPE Min, VPE Max], then PACE 135 determines that an anomaly in network behavior has been detected, i.e., the corresponding fault or other network event is occurring at a frequency outside of the expected range. PACE 135 proceeds to operation 1530 and determines the type of network event that exhibits the abnormal behavior. For example, PACE 135 may use Figure 6 Type information 640 of table 600 is provided to identify the network event(s) of a particular type that exhibits the behavior.

[0147] In operation 1535, in response to the identification of one or more abnormal network events by PACE 135, NMS 136 invokes virtual network assistant 133 to perform a deeper analysis of event data 139 to determine the root cause of the detected anomaly. In decision operation 1540, VNA 133 determines whether remedial action can be invoked based on the root cause analysis, such as restarting or reconfiguring one or more APs 142, network nodes, or other components or by outputting scripted actions for an administrator to follow. If remedial corrective action is available, then at operation 1545, VNA 133 invokes remedial action, such as restarting a particular device, component of a device, module of a device, etc. In either case, if remedial action is available or not available, VNA 133 proceeds to operation 1550 and outputs a report / alert to notify a technician of the identified problem and / or corrective action taken by VNA 133 to automatically remedy the underlying root cause of the problem. After invoking the VNA 133 , the PACE 135 loops back to operation 1515 and continues processing the real-time event data 139 received from the network system 100 .

[0148] The techniques described herein may be implemented using software, hardware, and / or a combination of software and hardware. Various examples relate to apparatuses, such as mobile nodes, mobile wireless terminals, base stations, such as access points, communication systems. Various examples also relate to methods, such as methods of controlling and / or operating communication devices (e.g., wireless terminals (UEs), base stations, control nodes, access points, and / or communication systems). Various examples also relate to non-transient machines, such as computers, readable media, such as ROM, RAM, CDs, hard disks, etc., which include machine-readable instructions for controlling a machine to implement one or more steps of a method.

[0149] It should be understood that any specific order or hierarchy of steps in the disclosed process is an example of an example method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged while remaining within the scope of the present disclosure. The attached method claims present elements of the various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0150] In various examples, the devices and nodes described herein are implemented using one or more modules to perform steps corresponding to one or more methods, such as signal generation, transmission, processing and / or receiving steps. Therefore, in some examples, various features are implemented using modules. Such modules can be implemented using software, hardware, or a combination of software and hardware. In some examples, each module is implemented as an individual circuit, wherein the device or system includes a separate circuit for implementing the functions corresponding to each described module. Many of the methods or method steps described above can be implemented using machine executable instructions (such as software, included in a machine readable medium (such as a memory device, such as RAM, a floppy disk, etc.) to control a machine, such as a general-purpose computer with or without additional hardware, to implement all or part of the methods described above in, for example, one or more nodes. Therefore, among other things, various examples relate to a machine readable medium, such as a non-transient computer readable medium, including machine executable instructions for causing a machine (such as a processor and associated hardware) to perform (multiple) one or more steps of the method described above. Some examples relate to a device including a processor, which is configured to implement one, more or all steps of one or more methods of an example aspect.

[0151] In some examples, one or more processors (e.g., CPUs) of one or more devices (e.g., communication devices (such as wireless terminals (UEs)) and / or access nodes) are configured to perform the steps of the methods described as being performed by the devices. The configuration of the processor can be achieved by using one or more modules (e.g., software modules) to control the processor configuration and / or by including hardware (e.g., hardware modules) in the processor to perform the listed steps and / or control the processor configuration. Therefore, some but not all examples relate to a communication device, such as user equipment, having a processor, which includes a module corresponding to each step of the various described methods performed by the device in which the processor is included. In some but not all examples, the communication device includes a module corresponding to each step of the various described methods performed by the device in which the processor is included. The module can be implemented purely in hardware, such as as a circuit, or can be implemented using software and / or hardware or a combination of software and hardware.

[0152] Some examples relate to a computer program product including a computer-readable medium, the computer-readable medium including a code for causing a computer or multiple computers to implement various functions, steps, actions and / or operations (e.g., one or more steps described above). In some examples, the computer program product can and sometimes does include different codes for each step to be performed. Therefore, the computer program product can and sometimes does include codes for each individual step of a method (e.g., a method of operating a communication device (e.g., a wireless terminal or node)). The code may be in the form of machine (e.g., computer) executable instructions stored on a computer-readable medium (such as a RAM (random access memory), a ROM (read-only memory), or other types of storage devices). In addition to relating to a computer program product, some examples relate to a processor configured to implement one or more of the various functions, steps, actions, and / or operations of one or more methods described above. Therefore, some examples relate to a processor configured to implement some or all steps of the method described herein, such as a CPU, a graphics processing unit (GPU), a digital signal processing (DSP) unit, etc. The processor can be used in, for example, a communication device or other device described in this application.

[0153] In view of the above description, many additional variations of the methods and devices of the various examples described above will be obvious to those skilled in the art. Such variations will be considered to be within the scope of the present disclosure. The methods and devices can and in various examples be used together with BLE, LTE, CDMA, orthogonal frequency division multiplexing (OFDM) and / or various other types of communication technologies that can be used to provide a wireless communication link between an access node and a mobile node. In some examples, the access node is implemented as a base station that establishes a communication link with a user equipment device (e.g., a mobile node) using OFDM and / or CDMA. In various examples, the mobile node is implemented as a notebook computer, a personal data assistant (PDA) or other portable device, including a receiver / transmitter circuit and logic and / or routines, for implementing the method.

[0154] In the detailed description, many specific details are set forth in order to provide a thorough understanding of some examples. However, those skilled in the art will appreciate that some examples can be practiced without these specific details. In other cases, well-known methods, procedures, components, units and / or circuits are not described in detail to avoid making the discussion unclear.

[0155] Some examples may be used in conjunction with various devices and systems, such as user equipment (UE), mobile device (MD), wireless station (STA), wireless terminal (WT), personal computer (PC), desktop computer, mobile computer, laptop computer, notebook computer, tablet computer, server computer, handheld computer, handheld device, personal digital assistant (PDA) device, handheld PDA device, in-vehicle device, out-of-vehicle device, hybrid device, vehicle-mounted device, non-vehicle-mounted device, mobile or portable device, consumer device, non-mobile or non-portable device, wireless communication station, wireless communication device, wireless access point (AP), wired or wireless router, wired or wireless modem, video device, audio device, audio and video (A / V) device, wired or wireless network, wireless local area network, wireless video local area network (WVAN), local area network (LAN), wireless LAN (WLAN), personal area network (PAN), wireless PAN (WPAN), etc.

[0156] Some examples may be used in conjunction with: devices and / or networks operating in accordance with existing Wireless Gigabit Alliance (WGA) specifications (Wireless Gigabit Alliance, Inc., WiGig MAC and PHY Specification Version 1.1, April 2011, Final Specification) and / or future versions and / or derivatives thereof; devices and / or networks operating in accordance with existing IEEE 802.11 standards (IEEE 802.11-2012, IEEE Standard for Information Technology--Telecommunications and Information Exchange Between Systems-Local and Metropolitan Area Networks--Specific Requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications, March 29, 2012; IEEE802.11ac-2013 (“IEEE P802.11ac-2013, IEEE Standard for Information Technology-Telecommunications and Information Exchange Between Systems-Local and Metropolitan Area Networks-Specific Requirements-Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications, March 29, 2012); Control (MAC) and Physical Layer (PHY) Specifications-Amendment 4: Enhancements for Very High Throughput for Operation in Bands below 6GHz", December 2013); IEEE 802.11ad ("IEEE P802.11ad - 2012, "IEEE Standard for Information Technology - Telecommunications and Information Exchange Between Systems - Local and Metropolitan Area Networks - Specific Requirements - Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications - Amendment 3: Enhancements for Very High Throughput in the 60GHz Band" (December 28, 2012); IEEE - 802.11REVmc ("IEEE 802.11 - REVmcTM / D3.0, June 2014 draft standard for Information technology - Telecommunications and information exchange between systems Local and metropolitan area networks Specific requirements; Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specification"); IEEE 802.11 - ay (P802.11ay Standard for Information Technology - Telecommunications and Information Exchange Between Systems Local and Metropolitan Area Networks - Specific Requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications - Amendment: Enhanced Throughput for Operation in License - Exempt Bands Above 45GHz)), IEEE 802.11-2016 and / or future versions and / or derivative versions thereof; devices and / or networks, operating according to existing Wireless Fidelity (Wi-Fi) Alliance (WFA) peer-to-peer (P2P) specifications (Wi-Fi P2P technical specification, version 1.5, August 2014) and / or future versions and / or derivative versions thereof; devices and / or networks, operating according to existing cellular specifications and / or protocols (e.g., Third Generation Partnership Project (3GPP), 3GPP Long Term Evolution (LTE)) and / or future versions and / or derivative versions thereof; units and / or devices, which are part of the above networks or operate using any one or more of the above protocols. .

[0157] Some examples may be used in conjunction with: one-way and / or two-way radio communication systems, cellular radiotelephone communication systems, mobile phones, cellular phones, wireless phones, personal communication system (PCS) devices, PDA devices incorporating wireless communication devices, mobile or portable global positioning system (GPS) devices, devices incorporating GPS receivers or transceivers or chips, devices incorporating RFID elements or chips, multiple-input multiple-output (MIMO) transceivers or devices, single-input multiple-output (SIMO) transceivers or devices, multiple-input single-output (MISO) transceivers or devices, devices with one or more internal antennas and / or external antennas, digital video broadcasting (DVB) devices or systems, multi-standard radio devices or systems, wired or wireless handheld devices (e.g., smart phones), wireless application protocol (WAP) devices, etc.

[0158] Some examples may be associated with one or more types of wireless communication signals and / or systems (e.g., radio frequency (RF), infrared (IR), frequency division multiplexing (FDM), orthogonal FDM (OFDM), orthogonal frequency division multiple access (OFDMA), FDM time division multiplexing (TDM), time division multiple access (TDMA), multi-user MIMO (MU-MIMO), space division multiple access (SDMA), extended TDMA (E-TDMA), general packet radio service (GPRS), extended GPRS, code division multiple access (CDMA), wideband CDMA (WCDMA), CDMA 2000, single carrier CDMA, multi-carrier CDMA, multi-carrier modulation (MDM), discrete multi-tone (DMT), Bluetooth, global positioning system (GPS), Wi-Fi, Wi-Max, ZigBee TM, Ultra Wideband (UWB), Global System for Mobile Communications (GSM), 2G, 2.5G, 3G, 3.5G, 4G, fifth generation (5G) or sixth generation (6G) mobile networks, 3GPP, Long Term Evolution (LTE), LTE Advanced, Enhanced Data Rates for GSM Evolution (EDGE), etc.) are used in combination. Other examples can be used for various other devices, systems and / or networks.

[0159] Some illustrative examples may be used in conjunction with a WLAN (wireless local area network) such as a Wi-Fi network. Other examples may be used in conjunction with any other suitable wireless communication network such as a wireless local area network, a "piconet", a WPAN, a WVAN, etc.

[0160] Some examples may be used in conjunction with a wireless communication network that communicates on frequency bands of 2.4 GHz, 5 GHz, and / or 60 GHz. However, other examples may be implemented using any other suitable wireless communication frequency band(s), such as an extremely high frequency (EHF) band (millimeter wave (mmWave) band), such as a band between 20 GHz and 300 GHz, a WLAN band, a WPAN band, a band according to a WGA specification, etc.

[0161] Although only some simple examples of various device configurations are provided above, it should be appreciated that many variations and permutations are possible. Furthermore, the techniques are not limited to any particular channel, but are generally applicable to any frequency range(s) / channel(s). Furthermore, and as discussed, the techniques may be useful in unlicensed spectrum.

[0162] Although examples are not limited in this regard, discussions utilizing terms such as, for example, "processing," "computing," "operating," "determining," "creating," "analyzing," "examining," etc., may refer to operations (or operations) and / or processes (or processes) of a computer, computing platform, computing system, communication system or subsystem, or other electronic computing device that manipulates data of physical (e.g., electronic) quantities within a computer's registers and / or memory and / or transforms that data into other data similarly represented as physical quantities within a computer's registers and / or memory or other information storage media that may store instructions to perform the operations and / or processes.

[0163] Although examples are not limited in this regard, the terms "plurality" and "a plurality" as used herein may include, for example, "plurality" or "two or more". The terms "plurality" and "a plurality" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, circuits, etc. For example, "a plurality of stations" may include two or more stations.

[0164] It may be helpful to clarify the definitions of certain words and phrases used throughout this document: the terms "include" and "comprise" and their derivatives mean including but not limited to; the term "or" is inclusive, meaning and / or; the phrases "associated with" and "associated with" and their derivatives may mean include, be included within, interconnect with, be interconnected with, contain, be contained within, connect to or be connected with, be coupled to or be coupled with, be compatible with, cooperate with, interleave, juxtapose, be proximate to, be bound to or be bound with, have, possess, and the like; and the term "controller" means any device, system, or portion thereof that controls at least one operation, such device may be implemented in hardware, circuitry, firmware, or software, or some combination of at least two thereof. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this document, and those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0165] Examples have been described with respect to communication systems and protocols, techniques, means and methods for performing communications such as in a wireless network or generally any communication network operating using any communication protocol(s). Examples of this are home or access networks, wireless home networks, wireless corporate networks, etc. However, it should be appreciated that, in general, the systems, methods and techniques disclosed herein will be equally applicable to other types of communication environments, networks and / or protocols.

[0166] For the purpose of explanation, many details are set forth in order to provide a thorough understanding of the present technology. However, it should be appreciated that the present disclosure can be put into practice in a variety of ways beyond the specific details set forth herein. In addition, although the examples illustrated herein show the various components of the system in parallel, it should be appreciated that the various components of the system can be located at the remote part of a distributed network (such as a communication network, a node), located in a domain host and / or the Internet or in a dedicated security, unsafe and / or encryption system and / or located in a network operation or management device inside or outside the network. As an example, a domain host can also be used to refer to any one or more aspects of management and / or configuration network or communication environment and / or (multiple) transceivers and / or sites and / or (multiple) access points described herein or any device, system or module communicating therewith.

[0167] Therefore, it should be understood that the components of the system can be combined into one or more devices, or split between devices (such as transceivers, access points, stations, domain hosts, network operation or management equipment, nodes), or are listed in parallel on a specific node of a distributed network (such as a communication network). As should be understood from the following description, and for reasons of computing efficiency, the components of the system can be arranged at any location in the distributed network without affecting its operation. For example, various components can be located at a domain host, a node, a domain management device (such as a MIB), a network operation or management equipment, (multiple) transceivers, stations, (multiple) access points or some combination thereof. Similarly, one or more functional parts of the system can be distributed between a transceiver and an associated computing device / system.

[0168] In addition, it should be understood that various links (including (multiple) any communication channels / elements / lines connecting elements) can be wired or wireless links or any combination thereof, or (multiple) any other known or later developed elements capable of providing and supplying and / or communicating data to and from connected elements. The term module as used herein can refer to any known or later developed hardware, circuitry, software, firmware or a combination thereof capable of performing the functionality associated with the element. As used herein, the terms determination, operation and calculation and variants thereof are used interchangeably and include any type of methodology, process, technology, mathematical operation or protocol.

[0169] Furthermore, while some examples described herein relate to a transmitter portion of a transceiver that performs certain functions or a receiver portion of a transceiver that performs certain functions, the present disclosure is intended to include corresponding and complementary transmitter-side or receiver-side functionality in both the same transceiver and / or another transceiver(s), respectively, and vice versa.

[0170] Examples are described with respect to enhanced communications. However, it should be appreciated that, in general, the systems and methods herein will be equally applicable to any type of communication system utilizing any one or more protocols (including wired communications, wireless communications, power line communications, coaxial cable communications, fiber optic communications, etc.) in any environment.

[0171] Example systems and methods for IEEE 802.11 and / or and / or A low energy transceiver and associated communication hardware, software, and communication channels are described.However, to avoid unnecessarily obscuring the present disclosure, the following description omits well-known structures and devices that may be shown in block diagram form or otherwise summarized.

[0172] Although the flowcharts described above have been discussed with respect to a particular sequence of events, it should be appreciated that changes to that sequence may occur without materially affecting the operation of the example(s). Additionally, the example techniques illustrated herein are not limited to the specific illustrated examples, but may also be utilized with other examples, and each described feature may be individually and separately claimable.

[0173] The system described above may be implemented on (multiple) wireless telecommunication devices / systems, such as IEEE 802.11 transceivers, etc. Examples of wireless protocols that may be used with the technology include IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, IEEE 802.11ac, IEEE 802.11ad, IEEE 802.11af, IEEE 802.11ah, IEEE 802.11ai, IEEE 802.11aj, IEEE 802.11aq, IEEE 802.11ax, Wi-Fi, LTE, 4G, WirelessHD, WiGig, WiGi, 3GPP, Wireless LAN, WiMAX, DensiFi SIG, UnifiSIG, 3GPP LAA (Licensed Assisted Access), etc.

[0174] Additionally, the systems, methods, and protocols may be implemented to improve one or more of a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit component(s), an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit (such as a discrete component circuit), a programmable logic device (such as a PLD, PLA, FPGA, PAL, modem, transmitter / receiver, any comparable device, etc. In general, any device capable of implementing a state machine, which in turn is capable of implementing the methodology illustrated herein, may benefit from the various communication methods, protocols, and techniques according to the disclosure provided herein.

[0175] Examples of a processor as described herein may include, but are not limited to, at least one of the following: 800 and 801, with 4G LTE integration and 64-bit computing 610 and 615, with 64-bit architecture A7 processor, M7 motion coprocessor, series, Core TM Processor series, Intel Processor series, i5-4670K and i7-4770K 22nm Haswell, i5-3570K 22nm Ivy Bridge, FX TM Processor series, FX-4300, FX-6300 and FX-8350 32nm Vishera, Kaveri processor, Texas Jacinto C6000 TM Automotive infotainment processors, Texas OMAP TM Automotive-grade mobile processors, Cortex TM -M processor, Cortex-A and ARM926EJ-S TM processor, AirForce BCM4704 / BCM4703 wireless networking processors, AR7100 wireless network processing units, other industry equivalent processors, and may use any known or future developed standards, instruction sets, libraries and / or architectures to perform computing functions.

[0176] In addition, the disclosed method can be easily implemented in software using an object or object-oriented software development environment, which provides portable source code that can be used on various computer or workstation platforms. Alternatively, the disclosed system can be implemented in hardware in part or in full using standard logic circuits or VLSI designs. Whether software or hardware is used to implement the system according to the example depends on the speed and / or efficiency requirements of the system, the specific functions, and the specific software or hardware system or microprocessor or microcomputer system utilized. The communication system, method, and protocol illustrated herein can be easily implemented in hardware and / or software by a person of ordinary skill in the applicable field according to a functional description using any known or later developed system or structure, device, and / or software, which is provided herein and has a general basic knowledge of computer and telecommunication fields.

[0177] In addition, the disclosed technology can be readily implemented in software and / or firmware that can be stored on a storage medium to improve the performance of a programmed general-purpose computer in cooperation with a controller and memory, a dedicated computer, a microprocessor, etc. In these cases, the systems and methods can be implemented as programs embedded on a personal computer, such as an applet JAVA.RTM. or CGI script, as a resource resident on a server or computer workstation, as a routine embedded in a dedicated communication system or system component, etc. The system can also be implemented by physically incorporating the system and / or method into a software and / or hardware system (such as the hardware and software system of a communication transceiver).

[0178] It is therefore apparent that at least systems and methods for enhancing and improving conversational user interfaces have been provided. Many alternatives, modifications, and variations will be or will be apparent to those of ordinary skill in the applicable art. Therefore, the present disclosure is intended to include all such alternatives, modifications, equivalents, and variations within the spirit and scope of the present disclosure.

Claims

1. A system comprising: Multiple access point AP devices in a wireless network; as well as Network management system, including: a memory storing network event data received from the AP device, wherein the network event data indicates operational behavior of the wireless network, and wherein the network event data defines a series of network events of one or more event types over a plurality of observation time periods; and One or more processors coupled to the memory and configured to: An unsupervised machine learning model is applied to the network event data to dynamically determine, for a most recent one of the observation time periods: (i) a predicted occurrence count of the network event for each of the one or more event types, and (ii) a minimum MIN threshold and a maximum MAX threshold for each of the one or more event types, wherein the MIN threshold and the MAX threshold define an expected range of occurrences of the network event for the one or more event types; and identifying one or more of the network events as indicating abnormal network behavior based on the MIN threshold and the MAX threshold and actual network event data for the most recent one of the observation time periods, The one or more processors are configured to, for each of the one or more event types: determining a prediction error for a corresponding event type, the prediction error indicating a difference between the predicted occurrence count of the network event generated by the unsupervised machine learning model and a count of actual network events of the actual network event data; and The abnormal network behavior is detected when the prediction error exceeds a limit between the MIN threshold and the MAX threshold for the corresponding event type.

2. The system of claim 1 , wherein the unsupervised machine learning model has been trained using training data, the training data comprising network event data, the network event data being augmented using the MIN threshold and the MAX threshold, the MIN threshold and the MAX threshold being dynamically determined for the one or more event types from the network event data for multiple overlapping time windows having different time origins.

3. The system of claim 1 , wherein the processor is further configured to: In response to identifying the one or more of the network events as indicating the abnormal network behavior, triggering, by a machine learning-based virtual network assistant, a root cause analysis of the network event data; and Remedial actions are triggered based on one or more identified root causes for the abnormal network behavior.

4. The system of claim 1 , wherein the processor is further configured to adaptively calculate the MIN threshold and the MAX threshold based on the network event data by: For the series of network events, determining a plurality of time windows, each of the time windows having a different time origin; for each of the time windows, counting a number of events in a given observation time period within the time window for each event type; For each event type and based on the count, determining the MAX threshold as a maximum number of occurrences of a network event of the corresponding event type for any time window in the plurality of time windows; and For each event type and based on the count, the MIN threshold is determined as a minimum number of occurrences of a network event of the corresponding event type for any time window in the plurality of time windows.

5. A network management system for managing one or more access point AP devices in a wireless network, comprising: a memory storing network event data received from the AP device, wherein the network event data indicates operational behavior of the wireless network, and wherein the network event data defines a series of network events of one or more event types varying over a plurality of observation time periods; as well as One or more processors coupled to the memory and configured to: Applying an unsupervised machine learning model to the network event data to determine, for a most recent one of the observation time periods: (i) a predicted occurrence count of the network event for each of the one or more event types, and (ii) a minimum MIN threshold and a maximum MAX threshold for each of the one or more event types, wherein the MIN threshold and the MAX threshold define an expected range of occurrences of the network event for the one or more event types; as well as Based on the MIN threshold and the MAX threshold and actual network event data for the most recent one of the observation time periods, identifying one or more of the network events as indicating abnormal network behavior, wherein the one or more processors are configured to, for each of the one or more event types: determining a prediction error for a corresponding event type, the prediction error indicating a difference between the predicted occurrence count of the network event generated by the unsupervised machine learning model and a count of actual network events of the actual network event data; as well as The abnormal network behavior is detected when the prediction error exceeds a limit between the MIN threshold and the MAX threshold for the corresponding event type.

6. A method for network management, comprising: storing, by one or more processors of the network management system, network event data received from a plurality of access point (AP) devices in a wireless network in a memory, wherein the network event data indicates operational behavior of the wireless network, and wherein the network event data defines a series of network events of one or more event types in a plurality of observation time periods; Determine, for a most recent one of the observation time periods and based on applying an unsupervised machine learning model to the network event data: (i) a predicted occurrence count of the network event for each of the one or more event types, and (ii) a minimum MIN threshold and a maximum MAX threshold for each of the one or more event types, wherein the MIN threshold and the MAX threshold define an expected range of occurrences of the network event for the one or more event types; as well as Based on the MIN threshold and the MAX threshold and actual network event data for the most recent observation time period in the observation time period, one or more network events among the network events are identified as indicating abnormal network behavior, wherein identifying the one or more network events among the network events as indicating abnormal network behavior comprises, for each of the one or more event types: determining a prediction error for a corresponding event type, the prediction error indicating a difference between the predicted occurrence count of the network event generated by the unsupervised machine learning model and a count of actual network events of the actual network event data; as well as The abnormal network behavior is detected when the prediction error exceeds a limit between the MIN threshold and the MAX threshold for the corresponding event type.

7. The method of claim 6, wherein the unsupervised machine learning model has been trained using training data, the training data comprising network event data, the network event data being enhanced using the MIN threshold and the MAX threshold, the MIN threshold and the MAX threshold being dynamically determined for the one or more event types from the network event data for multiple overlapping time windows having different time origins.

8. The method according to claim 6, further comprising: In response to identifying the one or more of the network events as indicating the abnormal network behavior, triggering, by a machine learning-based virtual network assistant, a root cause analysis of the network event data; as well as Remedial actions are triggered based on one or more identified root causes for the abnormal network behavior.

9. The method of claim 6, further comprising adaptively calculating the MIN threshold and the MAX threshold based on the network event data, wherein the adaptively calculating comprises: For the series of network events, determining a plurality of time windows, each of the time windows having a different time origin; for each of the time windows, counting a number of events in a given observation time period within the time window for each event type; For each event type and based on the count, determining the MAX threshold as a maximum number of occurrences of a network event of the corresponding network type for any time window in the plurality of time windows; as well as For each event type and based on the count, the MIN threshold is determined as a minimum number of occurrences of a network event of the corresponding network type for any time window in the plurality of time windows.

Images