Network system fault resolution via machine learning model
Through machine learning models, the operating parameters of wireless network systems are monitored and analyzed, the root cause of failure is identified and appropriate actions are selected for diagnosis and correction, which solves the problem of low fault handling efficiency of complex network systems and improves system performance and user experience.
Patent Information
- Application Number
- CN202411371458.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2020-03-31
- Filing Date
- 2020-06-12
- Publication Date
- 2024-12-03
AI Technical Summary
When complex wireless network systems encounter failures, it is difficult to quickly identify the root cause and take effective measures, resulting in system performance degradation and affecting user experience.
The machine learning model is used to monitor the operating parameters of the network system, and the fault and its potential root cause are identified by training the model, and appropriate actions are selected based on probability and cost for diagnosis and correction, and iterate until the system returns to normal.
It quickly identifies the root cause of network failure and takes effective measures, improves system performance, reduces user impact, and improves fault handling efficiency.
Smart Images

Figure CN119071833A_ABST
Abstract
Description
[0001] Description of divisional application
[0002] This application is a divisional application of the invention patent application with application date of June 12, 2020, application number 202010535033.1, and invention name “Network System Fault Solving via Machine Learning Model”. Technical Field
[0003] The present disclosure generally relates to diagnosis of network systems. In particular, the disclosed embodiments describe the use of machine learning models to automatically resolve faults in network systems. Background Art
[0004] Users of complex wireless networks, such as Wi-Fi networks, may experience degradation in system-level experience (SLE) parameters, which may be caused by a variety of complex factors. In order to ensure that complex wireless networks meet the needs of their user population, it is important to quickly resolve any problems that may arise with the operation of the system. Problem solving may include identifying one or more root causes of system-level experience problems and initiating corrective measures. However, when the network is composed of a large number of devices, including devices of different types and functionalities, identifying the root causes can take a significant amount of time. If the system does not function during this time period or operates at a reduced capacity, users of the system may be affected, and in some cases, users of the system may be severely affected. Therefore, there is a need for improved methods of isolating the root causes of problems associated with complex network systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] The embodiments of the present invention may be better understood by referring to the following description in conjunction with the accompanying drawings, in which the same reference numerals indicate the same or functionally similar elements. The drawings include the following:
[0006] Figure 1 is an overview diagram of an example system that may be implemented in one or more of the disclosed embodiments.
[0007] Figure 2 Example message portions implemented in one or more of the disclosed embodiments are shown.
[0008] Figure 3 Example data structures maintained by one or more of the disclosed embodiments are shown.
[0009] Figure 4A Examples of actions to correct the underlying root cause are shown.
[0010] Figure 4B Example actions are shown for unremediating the underlying root cause.
[0011] Figure 4C An example of an action that does not remediate the underlying root cause is shown.
[0012] Figure 5 is a flow chart of an example process for detecting and resolving problems with a network system.
[0013] Figure 6 is a flow diagram of an example process for selecting an action to be invoked on a monitored system.
[0014] Figure 7 An example machine learning module 700 is shown according to some examples of the present disclosure.
[0015] Figure 8 The diagram illustrates a data flow implemented in one or more of the disclosed embodiments.
[0016] Fig. 9 Data flow associated with a machine learning model implemented in one or more of the disclosed embodiments is shown.
[0017] Fig.10 is a flow chart of an example method for determining a category of problem experienced by a monitored system.
[0018] Fig.11A is a flow chart of an example process for iteratively applying diagnostic actions as needed until a root cause is sufficiently identified (eg, probability is greater than a threshold) or no diagnostic actions are available for injection.
[0019] Fig. 11B is a flow chart of an example process for determining which diagnostic action should be performed.
[0020] Fig.12 is a flow chart of an example process for determining whether to perform a corrective action or a diagnostic action performed in one or more of the disclosed embodiments.
[0021] Fig.13A is a flow chart of an example process for determining whether to inject a diagnostic action.
[0022] Fig. 13B is a flow chart of an example process for determining whether to inject a diagnostic action.
[0023] Fig.14A is a diagram illustrating the operation of one or more of the disclosed embodiments.
[0024] Fig. 14BAn embodiment is illustrated which applies the more expensive action if the cost is less than a predetermined threshold, and similarly, applies the less expensive action if the cost of the more expensive action is above a predetermined threshold.
[0025] Fig. 14C Illustrated are embodiments of determining the action to be applied based on a confidence level or probability that a particular root cause has caused a problem in the system being monitored.
[0026] Fig.14D An embodiment is illustrated where an action is selected based on a predetermined threshold and its relationship to the confidence level or probability that a root cause is causing a problem identified by a system monitored by the embodiment.
[0027] Fig.15 Illustrated is a block diagram of an example machine that can perform any one or more of the techniques (eg, methodologies) discussed herein. DETAILED DESCRIPTION
[0028] Example embodiments are disclosed for determining and executing corrective actions for complex network systems (e.g., wireless network systems) to improve system performance. The performance of the complex system is evaluated based on service level experience parameters or, more generally, operational parameters. These parameters may include parameters such as data transmission delay measurements, percentage of successful connection attempts, percentage of access points (APs) available for association, error statistics (such as errors generated via dropped connections, packet collisions, or other sources of errors), system throughput measurements, or other SLE parameters.
[0029] Some embodiments also monitor messages exchanged within a complex network system. The message information is also provided to a machine learning model that is trained to identify faults and potential root causes of the faults. In various embodiments, a fault may include any deviation from nominal system operation that the machine learning model is trained to detect. For example, in some embodiments, a fault includes any one or more of latency, throughput, jitter, error count, or other operating parameters that meet a standard. Standards are defined to detect undesirable system conditions. For example, an example standard evaluates the latency of a device (such as an access point) to determine whether the latency is below a predetermined latency threshold. In some embodiments, a fault may be defined as including two or more operating parameters that meet one or more corresponding standards. For example, in some embodiments, a fault may be defined as including a latency of a device that meets a first standard and a throughput of a device that meets a second standard (satisfying both conditions simultaneously, wherein latency and throughput are measured within a predetermined elapsed time of each other). The root cause of the fault is related to the condition that caused the fault. For example, the root cause may include a software problem and / or firmware problem of a specific device, an inoperative network connection between two devices, or other root causes.
[0030] Along with the root cause identification, the disclosed embodiments identify possible actions to be taken to resolve the system problem or obtain additional diagnostic information, which can then be applied to increase the confidence of the root cause identification. These actions include one or more of the following: initializing a specific beacon radio; restarting the radio; restarting the device; restarting the software component; restarting the computer; changing the operating parameters of the software component or hardware component; querying the system component for status information; requesting the system component to perform a task or other action.
[0031] Each of these actions is associated with a probability, indicating the probability that the action will solve the problem. Actions are also associated with costs. For example, a first action that causes a large number of user sessions to be closed will typically have a higher cost than a second action that is transparent to the user population.
[0032] Then, the disclosed embodiments select a course of action based on the identified probabilities and associated costs. Some embodiments of the disclosed embodiments operate in an iterative manner, because a first action is applied to the system, and then, the system is monitored to collect additional data. For example, if the first action is designed to solve a problem, the disclosed embodiments monitor the system to determine whether the problem has been solved (e.g., the monitored system has returned to nominal operation). If the first action is designed to provide additional diagnostic information, the system is monitored after applying the second action to collect additional diagnostic information. In some cases, additional actions are identified based on system behavior after applying the first action. The process can be iterated until the system achieves nominal performance, at which point the diagnostic process is considered complete.
[0033] Some embodiments utilize a cost function as defined below in Equation 1:
[0034] Cost Action i = c1*(number of affected users)*(Impact Action i ) Equation 1
[0035] in:
[0036] Cost Action i – Inject costs for specific actions
[0037] C1 – Predetermined coefficient
[0038] Impact Action i – Action specific parameters, such as:
[0039] 0.1 is used to initialize the radio beacon.
[0040] 0.2 is used to reset the radio,
[0041] 0.3 for rebooting the device, and
[0042] 0.4 is used to reset the power of the device.
[0043] Some embodiments provide a user interface configured to accept input defining the root cause of a particular problem. For example, in some cases, a person (e.g., an IT technician) diagnoses a system problem and identifies the root cause. The user interface is configured to allow the person to identify the time period in which the problem occurred and also enter information about the root cause and corrective action. In some aspects, the user interface also provides the operator with the ability to associate a distribution list or a warning list with the identified root cause and / or corrective action. Based on the input provided by the user interface, training data indicating symptom information, diagnostic information, and corrective information is generated.
[0044] In some embodiments, the machine learning model is trained at least in part via the assistance of human support personnel. In this mode of operation, a technician (e.g., a field support engineer) can analyze a failure of a network system and identify the root cause. The technician can then enter information defining the failure and the root cause and possible actions to be taken in response to the failure into a training database. The training database is then used to further train the machine learning model, which benefits from the input provided by the technician.
[0045] Some embodiments are configured to automate defect reporting. For example, some embodiments interface with a defect reporting system (e.g., Jira) via a service-oriented interface or other API available through a provider of the defect reporting system. Some embodiments perform an automatic search of the defect reporting system to obtain existing defects that define parameters similar to those identified during the automatic diagnosis as described above. If similar defect reports are identified, some embodiments update the report to indicate additional, defect incidences based on the most recent diagnosis. If similar defects are not identified within the defect database, a new defect report will be generated. The new defect report is populated with information from the measured operating parameters and information derived from the diagnostic process as described above.
[0046] Figure 1 is an overview diagram of an example system that may be implemented in one or more of the disclosed embodiments. Figure 1 Three APs 102a to 102c are shown in communication with wireless terminals 104a, 104b, 104c, and 104d. AP 102a is in communication with switch 106. AP 102b and switch 106 are in communication with router 108. Router 108 is in communication with network 110 (such as the Internet). A network management system 112 is also connected to network 110 and is configured to have network connectivity with at least AP 102a to AP 102c and router 108.
[0047] The network management system 112 is configured to monitor the activity of the system 100. The network management system 112 monitors the activity of the system 100 via messages 114a, 114b, 114c, 114d, 114e, 114f, 114g, 114h, 114i, and 114j that include information related to the operation of the system 100. For example, in various embodiments, the messages 114a to 114i indicate operating parameter values of various devices included in the system 100, message activity of messages exchanged between network components of the system 100, or other information. For example, the network management system 112 collects information related to operating parameters of one or more of the devices, such as any of the APs 102a to 102d, the wireless terminals 104a to 104d, the switches 106, or the routers 108. The information may include statistical information maintained by the respective devices. For example, in some embodiments, one or more of AP 102a to AP 102d maintains statistical information that describes, for example, the number of wireless terminals associated with the respective APs, communication latency or throughput, delays in establishing connections or associations with wireless terminals, detected communication errors, packet collisions, packet errors, CPU utilization, memory utilization, I / O capacity, and other metrics that characterize communication conditions at the APs. In some embodiments, the network management system 112 is also configured to monitor individual messages between network components of the system 100. For example, in some embodiments, the network management system is configured to monitor network messages transmitted between AP 102a and switch 106 or between AP 102b and router 108. In some aspects, such monitoring is accomplished via message summary information provided to the network management system by a device (e.g., AP 102a or AP 102b). Examples of message summary information are provided below.
[0048] Based on the monitored activities and operating parameters, the network management system is configured to perform one or more actions on one or more of the components of the system 100 at least when a particular condition is detected. For example, by monitoring operating parameters and / or separate messages passed between network components, the network management system 112 identifies that the system 100 is operating at a reduced level (relative to a nominal level). Further based on the monitoring of operating parameters and messages, the network management system 112 identifies possible root causes of the reduced performance of the system 100 and determines one or more actions to be taken. In some cases, the action(s) are designed to correct the problem identified by the network management system. In other cases, the actions provide additional diagnostic information that allows the network management system to determine the root cause of the problem. These concepts are further elaborated below:
[0049] Figure 21 shows an example message portion implemented in one or more of the disclosed embodiments. Figure 2 The message portion 200, message portion 220 and message portion 230 discussed above are included in Figure 1 In one or more of the messages 114a to 114j discussed. In some of the disclosed embodiments, Figure 2 One or more fields of the example message parts shown in are used to transmit message content information exchanged between network component devices of a network system (eg, 100) to a network management system (eg, 110) for processing.
[0050] Figure 2 Message portion 200, message portion 220, and message portion 230 are shown. Message portion 200 includes a timestamp field 204, a source device field 206, a destination device field 208, a type field 210, a length field 212, and a parameters of interest field 214. Timestamp field 204 indicates the time when the message information described by the remaining fields of message portion 200 was generated. Source device field 206 identifies the source device of the message. Destination device field 208 indicates the destination device of the message. Type field 210 indicates the type of message. For example, in some embodiments, type field 210 indicates whether the message is a data message, a connection request message, a connection establishment message, a connection reset message, or some other message type. The length field indicates the length of the message. Parameters of interest field 214 indicates any other characteristics that may be of interest in the message. In some embodiments, parameter of interest field 214 includes a tag value used to assist the device in decoding message portion 200 when interpreting the contents of parameter of interest field 214. The message portion 200 is used in those embodiments that send information about individual messages passed between components of the system 100 to the network management system 112. The message portion 200 generally does not aggregate data related to multiple messages, but rather represents a single message. For example, while the message portion 200 provides a detailed level of detail about the messages passed between components of the system 100, it may generate more overhead to the system 100 than other messages discussed below.
[0051] The example message portion 220 includes a timestamp field 222, a source device field 224, a destination device field 226, a type field 228, and a count field 229. The timestamp field 222 defines the time period during which the message information conveyed by the message portion 220 was generated. In some embodiments, the machine learning model employed by one or more of the disclosed embodiments relies on the value stored in the timestamp field 222 to establish a time sequence of message exchanges, based on which a diagnosis of a complex network system is derived. The source device field 224 identifies the source device of one or more messages. The destination device field 226 identifies the destination device of one or more messages represented by the message portion 220. The type field 228 indicates the type of one or more messages represented by the message portion 220. The count field 229 identifies the number of messages represented by the message portion 220. Thus, while message portion 200 represents a single message, and therefore, may represent the message in more detail (e.g., via parameter of interest field 214 and length field 212), message portion 220 summarizes multiple messages of a particular type that are exchanged between a common source (e.g., source device field 224) and a destination (e.g., destination device field 226). Some embodiments are configured to utilize both message portion 200 and message portion 220. For example, some embodiments utilize message portion 220 to summarize messages that meet a first criterion, and utilize message portion 200 to convey information about messages that meet a second criterion. For example, certain types of messages (e.g., error messages) are represented via message portion 200, where more detailed information is provided to network management system 112, while message portion 220 is used to represent other message types (e.g., data messages or other messages indicating nominal operation).
[0052] The example message portion 230 includes a timestamp field 232, a CPU utilization field 234, a memory utilization field 236, a latency field 238, a packet error count field 240, a collision count field 242, a number of connections field 244, and other operational parameter value fields 246. While message portions 200 and 220 summarize or provide information about messages passed between components of a monitored system (e.g., 100), message portion 230 is designed to convey parameter values from network components of a monitored system (e.g., AP 102a to AP 102d) to the network management system 112. Timestamp field 232 defines the time period to which the operational parameter values defined by message portion 230 are related. Source device field 233 identifies the device whose parameters are described by message portion 230. CPU utilization field 234 defines the CPU utilization of the device that generated message portion 230. Memory utilization field 236 defines the memory utilization of the device that generated message portion 230. The latency field 238 defines the latency given by or experienced by the device on the network. The packet error field 240 defines the number of packet errors detected by the device. The collision count field 242 defines the number of packet collisions experienced by the device. The number of connections field 244 defines the number of connections maintained by the device. The other operating parameter value field 246 defines one or more other operating parameter values of the device. For example, the other operating parameter values indicated by the message portion 230 may include, but are not limited to: access point name, basic service set identifier (BSSID), communication channel, communication frequency band, media access control (MAC) information, the number of associated wireless terminals (e.g., at the AP) of the network component device, or the service set name.
[0053] Figure 3 1 shows an example data structure maintained by one or more of the disclosed embodiments. Figure 3 The data structure is described as a relational database table, but other embodiments utilize other data organization methods. For example, some embodiments utilize traditional internal memory structures, such as arrays or linked lists, trees, queues, graphs, or other data structures. In other embodiments, unstructured data storage techniques are relied upon.
[0054] Figure 3Model output table 300, root cause table 310, action table 320, warning list table 330, category table 340, and diagnostic action table 350 are shown. Model output table 300 includes probability field 304, cause identifier field 306, and component identifier field 308. Probability field 304 defines the probability that the root cause identified via cause identifier field 306 is the root cause of the problem identified by the model as employed in the present disclosure. Cause identifier field 306 uniquely identifies the root cause and can be cross-referenced with field 312 in root cause table 310 discussed below. Component identifier field 308 identifies the component associated with the cause (identified via cause identifier field 306). For example, component identifier field 308 identifies a software component or process, a hardware component or process, or a device. Root cause table 310 maps the cause (identified via cause identifier field 312) to one or more actions (identified via field 314). Root cause table 310 also includes warning list identifier field 316. The alert list identifier field 316 identifies a list of addresses that issue alerts when a particular cause is identified (the cause identified by the cause identifier field 312). Thus, the root cause table 310 indicates that multiple different actions (or a single action) may be appropriate for a single routing cause (identified via the cause identifier field 312).
[0055] The action table 320 includes an action identifier field 322, an action type field 324, an action function field 325, a cost function field 326, a confidence value (e.g., a probability of resolution if the action is taken) 328, and an action allowed field 329. The action identifier field 322 uniquely identifies a specific action performed in one or more of the disclosed embodiments. The action type field 324 indicates whether the action is designed to correct a problem or provide additional diagnostic information about the root cause of the problem. The action function field 325 stores information that allows an implementation to perform the identified action. For example, in some embodiments, the action function field 325 can store an entry point of an API that implements the action. Examples of actions include: restarting a specific radio in an access point; restarting beacons in an access point; restarting only radios with specific frequencies (e.g., 2.4Ghz and / or 5Ghz) in an access point; restarting a device (such as an AP). Other examples of possible actions include: upgrading software running on a device; upgrading driver software; upgrading application software; upgrading software for a specific module.
[0056] The cost function field 326 defines a cost function for an action. At least some of the disclosed embodiments utilize the cost function defined by field 326 to determine the cost of invoking an action. This cost information is used in some embodiments to select between multiple actions. The confidence value field 328 indicates the probability that the action will solve the root cause problem for the corrective action. When determining whether to call an action, some embodiments may relate the cost of the action to the probability or confidence that the action solves the root cause. For example, some embodiments determine the cost of performing the action based on the impact of the action divided by the probability or confidence that the impact solves the identified problem. In other words, some embodiments determine that the cost of the action is inversely related to the probability or confidence that the action solves the underlying problem. The action permission field 329 defines whether the action can be automatically executed in a specific implementation. For example, some embodiments provide a user interface that allows a system administrator or other individual to define which corrective actions the disclosed embodiments can automatically execute. In various embodiments, the user interface is a graphical user interface or even a simple thing, such as a text configuration file that defines allowed actions or disallowed actions. Therefore, some embodiments consult the permission field 329 before executing the action to confirm that such an action is allowed. Alternatively, as described above and below with respect to alert list identifier field 316 and alert list table 330, if the action is not marked as allowed, one or more alerts may still be generated for the appropriate distribution list.
[0057] The warning list table 330 includes a warning list identifier field 332 and a warning address field 334. The warning list identifier field 332 uniquely identifies the warning distribution list. The warning address field 334 identifies one address included in the warning distribution address (the warning distribution address identified via the warning list identifier field 332). When the warning distribution list includes multiple addresses, multiple rows for a single warning list identifier value are included in the warning list table 330.
[0058] The category table 340 includes a category identifier field 342 and a warning list identifier field 344. The category identifier field 342 may be cross-referenced with the category id field 315 discussed above with respect to the root cause table 310. The category table 340 or a similar data structure is implemented in embodiments that prefer to associate a distribution list or warning list with a category of causes (e.g., software, hardware, driver, etc.) rather than with each individual cause (e.g., divide by zero, out of memory, etc.). Thus, some embodiments associate a distribution with a category of root causes rather than with each root cause itself.
[0059] Diagnostic action table 350 includes component type identifier field 352 and action identifier field 354. Diagnostic action table 350 maps from component types (via field 352) to possible diagnostic actions (e.g., via field 354) to be taken when a component of the indicated type is experiencing a problem (or may be experiencing a problem).
[0060] Injection history table 360 includes action identifier field 362, injection time field 364, component identifier field 366 and probability improvement field 368. Action identifier field 362 uniquely identifies the diagnostic action. Action identifier field 362 can be cross-referenced with action identifier field 362 or action identifier field 322 or action identifier field 354. Injection time field 364 identifies the time of injecting the diagnostic action. Component identifier field 366 identifies the component to which the injection is performed. For example, if the action is restart, component identifier field 366 identifies the component to be restarted. In various embodiments, the component identifier is composed of multiple parts. For example, in some aspects, the first part identifies the physical device (e.g., station address or other unique identifier), and the second part identifies the component of the physical device (e.g., wireless chip, CPU, software component or other hardware component). According to an example embodiment, when the diagnostic action is not injected into the same component that exhibits a higher probability of being the root cause of the performance degradation, the table 360 includes a first component ID identifying the component into which the diagnostic action is injected, a second component ID (not shown in the figure) identifying the component that exhibits the highest probability of being the root cause of the underlying problem. When the same diagnostic action is injected more than once, the table 360 also includes a probability improvement field 368 indicating the improvement achieved when the root cause is identified by reapplying the diagnostic action.
[0061] Component table 370 maps from component identifiers via field 372 to component types via field 374. Some embodiments utilize component table 370 to determine the type of a component from a component identifier. For example, some embodiments of machine learning models discussed below provide possible root causes and component identifiers of components that potentially caused the problem. Component table 370 is used in some embodiments to determine the type of a component identifier via a machine learning model.
[0062] Figure 4A 400A is a graph of data demonstrating an example of an action to correct an underlying root cause. In this case, the measured SLE parameter is a counter of Ethernet errors on a particular link (Ethernet link). Before an action (in this case, a restart of the communication link) was injected into the system, the system experienced a high link error rate. At time 410, a restart action 405 was invoked. The injected action proved to be a corrective action that reduced the error rate to zero. No further action was required.
[0063] Figure 4B is a graph 400B of data illustrating an example action for an unremediated underlying root cause. Figure 4B The measured SLE parameter in the example data of is a counter of Ethernet errors on the Ethernet link. Prior to the injection action (in this case, a restart of the communication link), the system experienced a high error rate. At times 420a to 420j, restart action 415a, restart action 415b, restart action 415c, restart action 415d, restart action 415e, restart action 415f, restart action 415g, restart action 415h, restart action 415i, and restart action 415j are called. Figure 4B It is shown that the action of injection did not correct the underlying problem, and Ethernet errors continued to occur at the same rate and, therefore, were not affected by the restart action. Figure 4B The error counts shown at different times in are recorded and stored for later addition to the historical information 730 discussed further below.
[0064] Some of the disclosed embodiments measure SLE and system parameter values after performing an action. Figure 4B In the example of , the Ethernet error rate is monitored after restarting the link. If the error rate does not decrease as a result of the link restart, a new root cause is identified. For example, in some embodiments, the new root cause indicates that the problem is caused by a loose Ethernet cable or a HW problem. Some embodiments then generate an alert via any known messaging technology that is used to notify a human support technician to correct the problem. In this case, the alert can indicate that the physical connection of the Ethernet link should be verified, and if the physical connection is all good, the Ethernet hardware should be swapped out for service.
[0065] Figure 4CIt is a chart 400C that demonstrates the data of the action of not remedying the basic root cause. In this case, the measured SLE parameter is a counter of the Ethernet error on a specific Ethernet link. Before executing the action (e.g., the restart of the communication link), the monitored system experienced a higher error rate. At each time in time 430a, time 430b, time 430c, time 430d and time 430e, restart action 425a, restart action 425b, restart action 425c, restart action 425d and restart action 425e are performed. As shown by chart 400C, the action does not correct the basic problem, and the Ethernet error continues to occur at the same rate that is not affected by (multiple) restart actions. At each time in time 430a, time 430b, time 430c, time 430d and time 430e, this situation may occur. In some embodiments, the error count is recorded and stored, and is included in the historical SLE measurement. As discussed further below, these error counts can be used as training for machine learning models.
[0066] In this particular example, the disclosed embodiments monitor SLE measurements and system parameters (e.g., CPU utilization, memory consumption, etc.) after an action (e.g., Ethernet error rate after link restart) is performed, and determine that since the action does not solve the problem, the problem is most likely caused by a defect in the software or firmware of the monitored system. Then, some disclosed embodiments generate warnings via any known messaging technology to warn humans of the problem. Some embodiments automatically initiate an update to the software and / or firmware installed on the monitored system. For example, if the embodiment determines that the underlying problem is caused by software (rather than by some other component (e.g., hardware)) and these existing software and / or firmware versions are below a threshold version level, an upgrade is performed. In some embodiments, an analysis is performed between the known defects of the existing software and / or firmware versions and the problems exhibited by the monitored system. If the problem exhibited is similar to the problem described for the existing software / firmware version, the disclosed embodiments initiate an upgrade of the software and / or firmware to a newer version (which may solve the problem).
[0067] Figure 5 is a flow chart of an example process for detecting and resolving problems with a network system. Figure 5 One or more of the functions discussed are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524) stored in electronic memory (e.g., 1504 and / or 1506) configure the hardware processing circuitry (e.g., 1502) to perform the following instructions regarding Figure 5and one or more of the functions discussed in process 500. In some embodiments, network management system 112 performs the following Figure 5 One or more of the functions in question.
[0068] After start operation 502, process 500 moves to operation 505, which monitors the operating parameter values and / or message exchanges of the network system. Figure 1 and Figure 2 As discussed, operating parameter values of network component devices (such as one or more of APs 102a to 102c, routers 108, wireless terminals 104a to 104d, or switches 106) are provided to a network management system (e.g., 112). In some embodiments, each of the network component devices maintains statistical information indicative of operating parameters of the devices. In other embodiments, network monitoring devices are deployed at strategic locations within the network system to collect this information with or without direct intervention by the network component devices.
[0069] The statistical information includes one or more of: CPU utilization, memory utilization, number of established connections, latency measurements, throughput measurements, dropped connection counts, roaming information, packet error information, collision information, media access control (MAC) information, access point identification information (such as, basic service set identifier, association identifier), or other indicators of component health and / or network performance. In some embodiments, operation 505 also includes: obtaining information about messages exchanged between network component devices of the monitored network system. For example, as discussed above, in some aspects, a message including one or more fields of example message portion 200, message portion 220, or message portion 230 is provided to a network management system (e.g., 112). The one or more fields convey information related to the number and type of messages exchanged between components of the monitored network system. The network management system (e.g., a device performing process 500) receives operating parameter values and / or message exchange information from one or more component devices of the network system. For example, one or more of APs 102a-102c may send a message (eg, any of message portion 200, message portion 220, or message portion 230) to a network management system (eg, 112).
[0070] The statistical information related to the operation of each network component device can be described as a time series. Therefore, in some embodiments, operation 505 includes: receiving a time series of corresponding device operating parameter values from a plurality of devices included in the network system. In some embodiments, each of these time series is provided to a machine learning model, as further discussed below.
[0071] Decision operation 510 determines whether a fault is detected based on the monitored operating parameter values. In some aspects, the detection of the fault is detected via a machine learning model. For example, as discussed above, in some embodiments, the machine learning model is trained to detect a system operating under suboptimal or unsatisfactory conditions. In other embodiments, the detection is based on evaluating one or more operating parameter values of the monitored system relative to one or more criteria. In some embodiments, a fault is detected based on a probability or confidence level provided by the machine learning model being above a threshold. For example, as described below with respect to Fig. 9 As discussed, some embodiments of the machine learning model provide multiple probabilities or confidence indications that the corresponding multiple root causes are the cause of the fault. If all of these probabilities or confidence indications are below a predetermined threshold, some embodiments interpret the operation of the monitored system as being considered normal or nominal. (For example, no fault was detected). If any of these indications are above a predetermined threshold, decision operation 510 determines that a fault is detected (note that in some embodiments, each root cause may have its own predetermined threshold for detecting a fault). If a fault is detected, process 500 moves from decision operation 510 to operation 515. Alternatively, if a fault is not detected, process 500 moves from decision operation 510 back to operation 505.
[0072] In operation 515, a root cause of the problematic operating condition is predicted. As discussed above, in some embodiments, the machine learning model is trained to indicate the probability of multiple different root causes occurring in the monitored system. Figure 3 As discussed, in some embodiments, the machine learning model generates a plurality of probabilities (e.g., 304), where each probability or confidence is associated with a root cause (e.g., via field 306).
[0073] In operation 520, an action is selected based on the root cause. As discussed above, a root cause can be associated with multiple possible actions. Operation 520 evaluates the possible actions for their corresponding costs and the probability or confidence of resolving the problem. Figure 6 This was discussed further.
[0074] Operation 525 performs the selected action. The selected action may include one or more of the following: restarting a software process or component of a network device included in the monitored network system; resetting the entire network device (e.g., power cycling); adjusting one or more configuration parameters of the network device or a software component of the network device; resetting a specific hardware component of the network device (e.g., resetting a network card or chip of the network device while maintaining the operation of the GPU of the device). In some embodiments, performing the action includes determining a category of the cause, for example, whether the cause is caused by hardware, software, a driver, or other technical component. In some embodiments, performing the action includes forwarding a notification to a specific distribution list based on the cause. For example, as described above with respect to Figure 3 As discussed, some embodiments associate a distribution list (e.g., via the alert list identifier field 316) with a cause. In at least some embodiments, the distribution list is then notified when a cause is identified. Note that in some cases, the selected action may be empty or no action is performed. This may cause an alert to be generated to the specified distribution list without performing any corrective action.
[0075] Operation 530 monitors the system in response to the actions performed. FIG. 4A to FIG. 4C As discussed, in some cases, the system behavior after the action is performed is analyzed to determine whether the system has returned to normal operation. This is the case when the selected action is designed to solve the problem. In some cases, the selected action is designed to elicit additional information for determining the root cause. For example, in some embodiments, the selected action queries the network component for status information, or requests the network component to perform a function. The result of the request can be used to determine whether the network component is operating correctly or experiencing a failure.
[0076] In some embodiments, the monitoring of the system in operation 530 is performed by a machine learning model. The machine learning model generates an indicator of whether the system has returned to normal operation. In some embodiments, the monitored time series of operating parameter values and / or message exchanges between network component devices are processed by one or more heuristics, wherein the output of the heuristic (processed time series) is provided to the machine learning model. For example, in some embodiments, the heuristic determines the rate of change of the link error rate over time, rather than providing a specific link error to the machine learning model. For example, in some embodiments, the rate of change is classified as constant over time, slowly increasing over time, or increasing faster over time. Some embodiments classify the time frame of the change in the link error rate. For example, the time frame is classified as the link error starting to grow n seconds after restart, starting to grow immediately after restart, or other classifications. In these embodiments, the heuristic maps each of these different classifications into a different error growth type. The error growth type is then provided to the machine learning model.
[0077] Decision operation 535 evaluates whether the system has returned to normal operation or nominal operation. If the system has returned to normal operation, process 500 returns from decision operation 535 to operation 505 and continues to monitor the system for new indications of the problem. If the system has not returned to normal operation, process 500 moves from decision operation 535 to operation 515, where a second root cause has been identified. Typically, the second root cause identified in the second iteration of operation 515 is more specific than the root cause identified during the first iteration of operation 515.
[0078] Figure 6 is a flow chart of an example process for selecting an action to be invoked on a monitored system. Figure 6 One or more of the functions discussed are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524) stored in electronic memory (e.g., 1504 and / or 1506) configure the hardware processing circuitry (e.g., 1502) to perform the following instructions regarding Figure 6 and one or more of the functions discussed in process 600. In some embodiments, network management system 112 performs the following Figure 6 One or more of the functions in question.
[0079] In some embodiments, the network management system 112 performs the following Fig.10 One or more of the functions in question.
[0080] In some embodiments, the process 600 discussed below is utilized when the root cause of the problem has been identified.The root cause is associated with one or more actions that can be performed in response to the root cause.These actions have various costs associated with them.For example, in some embodiments, the first action is transparent to the user and does not have a negative impact (querying status information to the network component).The second action causes the user to lose connectivity or experience reduced functionality (for example, slower data transmission, higher network jitter, etc.) in some other way.Therefore, in some embodiments, the first action is selected based on cost.The process 600 discussed below also considers the probability or confidence that each action will solve the root cause problem.Therefore, when some actions may bring higher costs to the monitored system, if these actions also provide higher probability of resolution or confidence of resolution relative to other less expensive actions, then in some cases, they may be reasonable.
[0081] After starting operation 602, an action is identified in operation 604. In at least some embodiments, the action is associated with a root cause (e.g., via root cause table 310). In operation 605, a cost associated with the action is determined. For example, as described above with respect to Figure 3 As discussed, some embodiments maintain an action table (e.g., 320) or other data structures that provide cost information for a specific action. In some embodiments, a specific action is identified based on a determined root cause (e.g., via the root cause table 310 discussed above). In some embodiments, the cost of an action is a function of one or more parameters of a monitored system. For example, in a system that experiences severe degradation, relatively speaking, the cost of some actions (e.g., restarting a computer or other network component) may be smaller than when performing an action on a system that only experiences minor problems. Therefore, some cost functions for an action may receive input parameters for determining appropriate costs. In various embodiments, the input parameters may include any one or more of the operating parameters discussed above. In some embodiments, the cost of an action is based on the number of users affected by the action. In some embodiments, the cost is dynamically determined before utilizing the cost to determine the action to be performed.
[0082] In operation 610, a probability or confidence that the action solves the underlying problem is determined. For example, as discussed above, some embodiments associate a probability of resolution with an action via an action table (eg, 320).
[0083] In operation 615, a score for the action is determined based on the cost and the probability or confidence. In some embodiments, the score is determined by dividing the cost by the probability or confidence. In some other embodiments, one or more weights may be applied to the cost and / or probability or confidence before performing the multiplication.
[0084] Decision operation 620 determines whether there are additional actions available for comparison (e.g., multiple actions associated with the root cause). If there are additional actions, process 600 moves from decision operation 620 to operation 605. In at least some embodiments, as process 600 iterates, a second action and a second cost and a second probability or a second confidence are identified, thereby obtaining a second score. Additional iterations can result in a third action, a third cost, and a third probability / confidence, and then, a third score can be determined. If no other actions remain, process 600 moves from decision operation 620 to operation 625, which compares the determined (multiple) scores to select an action. In some embodiments, the action with the highest or lowest score is determined or selected. Then, the action is applied to the monitored network system.
[0085] Figure 7 An example machine learning module 700 according to some examples of the present disclosure is shown. The example machine learning module 700 utilizes a training module 710 and a prediction module 720. The training module 710 uses historical information 730 as an input to a feature determination module 750a. The historical information 730 can be labeled. The example historical information can include historical operating parameter values, such as any of the operating parameter values discussed above, such as but not limited to: CPU utilization, memory utilization, latency measurements, error counts, conflict metrics, throughput measurements. In some example embodiments, as explained above, the input includes historical data or operating parameter data processed by heuristics. In some embodiments, the historical information 730 also includes one or more indications of messages passed between network components of the monitored system. For example, in some embodiments, one or more of the fields described above for message portion 200, message portion 220, or message portion 230 are included in the historical information. In some embodiments, the historical information 730 also includes actions performed by the disclosed embodiments and operating parameter values and / or message transmission activities of the monitored system after the actions are performed. Thus, in at least some embodiments, the historical information includes the response of the network system to the selected action. For example, if the action selected by the disclosed embodiments queries the state of a network component or requests a task to be performed by a network component, then in some embodiments, the action and result are included in the historical information. In some embodiments, these indications are stored in a training repository of network data (e.g., such as historical information 730).
[0086] The label / annotation information 735 included in the training library indicates, for example, whether the time-related network data is associated with nominal or acceptable system performance. The label also indicates whether the time-related network data is associated with unacceptable or problematic system performance or whether it indicates unacceptable or problematic system performance. In some embodiments, the label / annotation training data also indicates the root cause of the network data representing problematic system performance. In some embodiments, labels are also provided for the system's response to actions performed by the disclosed embodiments. For example, in some cases, the machine learning model generates a suggested action that includes: generating a status request for the monitored system. Alternatively, the machine learning model generates a suggested action that generates a request for the system to perform a specific task (pass data; open a connection; restart a server, etc.). The generated action is included in the historical training data. Then, a label indicating whether the response of the monitored system to the generated action indicates that the generated action solves the problem, or a label indicating a second or different root cause of the problem is indicated by the response to the generated action.
[0087] The feature determination module 750a determines one or more features from the historical information 730. Generally speaking, a feature is a set of information inputs and is information that is determined to predict a specific outcome. In some examples, the feature may be all of the historical activity data, but in other examples, the feature may be a subset of the historical activity data. In some embodiments, the features are encoded into a feature vector 760. In some embodiments, the feature determination module 750a utilizes one or more heuristics when processing the historical information 730 to determine the features in the feature vector 760. The machine learning algorithm 770 generates a model 718 based on the feature vector 760 and the labels.
[0088] In the prediction module 720, the current information 790 can be used as an input to the feature determination module 750b. The current information 790 in the disclosed embodiment includes similar indications as described above for the historical information 730. However, the current information 790 provides these indications for the simultaneous messaging activity or operating parameter values of the monitored system. For example, in some embodiments, the simultaneous activity of the monitored system is provided to the feature determination module 750b to determine whether the monitored system is experiencing an operational problem, and if so, what the most likely root cause is.
[0089] Feature determination module 750b can determine the same set of features or a different set of features from current information 790 as feature determination module 750a determined from historical information 730. In some examples, feature determination modules 750a and 750b are the same module. Feature determination module 750b generates feature vector 715. In some embodiments, feature determination module 750b utilizes one or more heuristics when processing current information 790 to determine the features in feature vector 715. Feature vector 715 is then provided as input to model 718 to generate output 795. Fig. 9 An example of output 795 is discussed. The training module 710 may operate in an offline manner to train the model 718. However, the prediction module 720 may be designed to operate in an online manner. It should be noted that the model 718 may be periodically updated via additional training and / or user feedback.
[0090] The machine learning algorithm 770 can be selected from many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, decision trees (e.g., iterative bisection 3, C4.5, classification and regression trees (CART), chi-square automatic interaction detector (CHAID), etc.), random forests, linear classifiers, quadratic classifiers, k nearest neighbors, linear regression, logistic regression, hidden Markov models, models based on artificial life, simulated annealing and / or virology. Examples of unsupervised learning algorithms include: expectation maximization algorithm, vector quantization and information bottleneck method. The unsupervised model may not have a training module 710. In an example embodiment, a regression model is used, and the model 718 is a coefficient vector corresponding to the learning importance of each feature in the feature vector 760 and the feature vector 715. In some embodiments, in order to calculate the score, the dot product of the coefficient vector of the feature vector 715 and the model 718 is used.
[0091] Figure 8 The diagram illustrates a data flow implemented in one or more of the disclosed embodiments. Figure 8 The diagram above shows Figure 7The historical information 730 discussed above is used by at least some of the disclosed embodiments to train a machine learning model. The historical information may include a time series 810a of operating parameter values. In at least some embodiments, a time series for one or more of the example operating parameters discussed above is provided as historical information. For example, in some embodiments, a time series related to packet errors, CPU utilization, memory utilization, latency, throughput, or other operating parameters is provided as historical information. Note that one or more of the operating parameter time series is provided for one or more network devices or components of the monitored system. Thus, for example, in some aspects, the above description of the operating parameter time series is provided. Figure 1 The operating parameters of each of the APs 102a through 102c in question (eg, the latency experienced at the AP 102a and the latency experienced at the AP 102b).
[0092] Figure 8 Also shown is a second operating parameter time series 810b that is processed by the heuristic 815 before being included in the historical information 730. Figure 8 The heuristic 815 is shown being applied before the second operating parameter time series 810b is stored or included in the historical information 730, but in some embodiments, the heuristic 815 is applied to the second operating parameter time series 810b after it is read from the historical information 730 but before it is provided to the machine learning model.
[0093] In various embodiments, the historical information 730 also includes message exchange information 820. The message exchange information represents information about messages exchanged between components of the monitored system. Figure 1 , message exchange information 820 indicates messages exchanged between, for example, AP 102c and wireless terminal 104c, AP 102b and router 108, switch 106 and router 108, or AP 102c and wireless terminal 104d. Message exchange information 820 is of course not limited to these examples. In some embodiments, message exchange information 820 includes one or more of the fields discussed above for any one or more of message portion 200, message portion 220, or message portion 230. In some embodiments, message exchange information 820 is a time sequence of message exchange information. For example, if wireless terminal 104c sends an association request message to AP 102a and AP 102a responds to the association request message with an association response message, the message exchange information 820 indicates that the association request message precedes the association response message in time.
[0094] Figure 8The above also shows Figure 7 The label / annotation information 735 in question. In some embodiments, a machine learning model is not relied upon to determine whether a monitored system is functioning in an acceptable manner. For example, some embodiments monitor one or more operating parameter values and evaluate these monitored values against corresponding criteria for determining the acceptability of the monitored parameter values. If the monitored parameter value does not meet the criteria, the monitored operating parameter value is deemed unacceptable.
[0095] In some embodiments, the tag / annotation information 735 includes a root cause indicator 860. The root cause indicator 860 is temporally correlated with the historical information 730. In some embodiments, the root cause indicator 860 is network device or component specific. For example, in some embodiments, the root cause indicator indicates one or more of the device and / or component of the device (e.g., the network interface chip of the access point) that caused the problem. In at least some embodiments, the root cause indicator 860 is similar to the root cause indicator 860 described above with respect to the historical information 730. Figure 3 The root cause table 310 discussed above is consistent. Some embodiments also associate one or more corrective actions with each root cause indicator included in the root cause indicators 860 .
[0096] Fig. 9 Data flow associated with a machine learning model implemented in one or more of the disclosed embodiments is shown. Fig. 9 The above shows Figure 7 The model 718 discussed and the output 795 generated by the model. The output 795 includes one or more root cause indicators 935. Each of the one or more root cause indicators 935 indicates a root cause (e.g., such as Figure 3 300 and / or the root cause table 310). Note that by providing an indicator of the root cause, the machine learning model provides possible actions to be taken based on the root cause. Figure 3 As discussed, some embodiments map causes to actions via a data structure similar to the root cause table 310. In some other embodiments, the machine learning model 718 provides both causes and actions, or only causes.
[0097] Each of the one or more root cause indicators 935 also includes a component id field 940b. The component id 940b indicates a specific component that is identified as causing the problem. In various embodiments, the component id 940b identifies one of a physical device, a software or firmware component of a device, a specific hardware component of a device (e.g., a chip, an interface, a power supply, or other device component).
[0098] Each of the one or more root cause indicators 935 also includes a probability or confidence indicator 940c. The probability or confidence indicator 940c indicates the probability that the monitored system is experiencing a problem caused by the root cause identified by the corresponding individual root cause indicator 940a.
[0099] Fig.10 is a flow chart of an example method for determining a category of problems experienced by a monitored system. Fig.10 One or more of the functions discussed are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524 below) stored in memory (e.g., 1504, 1506) configure hardware processing circuitry (e.g., 1502) to perform one or more of the functions discussed below. In some embodiments, network management system 112 performs the following instructions regarding Fig.10 One or more of the functions in question.
[0100] After start operation 1005, process 1000 moves to operation 1010, which monitors the value of the operating parameter. For example, as described above with respect to Figure 1 and Figure 2 As discussed, operating parameter values of network component devices (such as one or more of APs 102a to 102c, routers 108, wireless terminals 104a to 104d, or switches 106) are provided to a network management system (e.g., 112). In some embodiments, each of the network component devices maintains statistical information indicative of operating parameters of the devices. In other embodiments, network monitoring devices are deployed at strategic locations within the network system to collect this information with or without direct intervention by the network component devices.
[0101] Decision operation 1015 determines whether any degradation is detected in the monitored operating parameters (e.g., monitored during operation 1010). If no degradation is detected, process 1000 moves from decision operation 1015 to operation 1010, where the operating parameter values continue to be monitored as described above. Alternatively, when performance degradation is detected, process 1000 moves from decision operation 1015 to operation 1020, which injects diagnostic actions.
[0102] In some embodiments, operation 1020 is performed via a machine learning model (such as the one described above with respect to Figures 7 to 9 The machine learning model 718 discussed above is used to obtain possible components that cause the problem. For example, as described above Fig. 9As discussed, some embodiments of the machine learning model 718 generate a root cause indication (e.g., 940a) and a component identifier associated with the root cause (e.g., 940b). Based on the component identifier, operation 1020 then identifies one or more diagnostic actions that can be taken to obtain additional information about the possible problem (e.g., via the diagnostic action table 350). When multiple diagnostic actions are possible for a given component, various embodiments use various techniques to select which action to take. Some embodiments select the diagnostic action with the lowest cost. As discussed above, in some embodiments, the cost of the diagnostic action is determined dynamically. For example, in some embodiments, the cost is determined based on specific components and / or devices identified as possible sources of the problem and the number of users currently communicating through the device or component. In some embodiments, the cost is proportional to the number of users affected by the diagnostic action. Some embodiments use more complex techniques to determine which diagnostic action to select among the diagnostic actions, such as the following with respect to Fig.13A and those described in process 1300. Some embodiments are based on process 1350 and Fig. 13B to determine which diagnostic action among the selected diagnostic actions.
[0103] After injecting the diagnostic action in operation 1020, process 1000 moves to operation 1025, which again monitors the operating parameter value after the injection of the action has been performed. In at least some embodiments, operation 1025 operates in a similar manner as operation 1010.
[0104] In operation 1030, the second monitored operating parameter value is provided to a classifier or machine learning model (e.g., 718).
[0105] In operation 1035, a category associated with the cause of the underlying problem is obtained. In some embodiments, operation 1035 obtains the category from a machine learning model (e.g., 718). Fig. 9 As discussed, the machine learning model provides one or more possible causes for the underlying problem (e.g., cause identifiers 312 in root cause table 310). Associated with each cause is a category identifier field 315. Thus, in some embodiments, operation 1035 determines the most likely cause of the underlying problem and the category associated with the most likely cause.
[0106] Decision operation 1040 determines whether the probability or confidence of the most likely cause is above a predetermined threshold or meets a criterion. If the probability or confidence is above the threshold, process 1000 moves to operation 1045, which performs an action associated with the cause. For example, as described above with respect to Figure 3 As discussed, actions may be associated with causes via the root cause table 310 .
[0107] After performing decision operation 1040 or operation 1045, process 1000 moves to operation 1050, which sends an alert indicating the identified cause and the category of the problem identified via operation 1035. In some embodiments, the alert is sent to an address associated with the cause. For example, as described above with respect to Figure 3 As discussed, each cause has a warning distribution list associated with it. In some other embodiments, the warning is sent to an address associated with the category of the problem. For example, if the root cause of the problem has been identified as a specific software module or a specific hardware module, the system selects the correct distribution list and automatically sends a notification to the team that can quickly resolve the problem. This is an example of how operation 1050 obtains an address to send a warning. After operation 1050 is completed, process 1000 moves to end operation 1055.
[0108] Fig.11A 1504 and / or 1506). The example process 1100 is performed in one or more of the disclosed embodiments. In some embodiments, one or more of the functions discussed below are performed by a hardware processing circuit device. For example, in some embodiments, instructions (e.g., 1524) stored in an electronic hardware memory (e.g., 1504 and / or 1506) configure the hardware processing circuit device (e.g., 1502) to perform one or more of the functions discussed below. In some embodiments, the network management system 112 performs the following description regarding Fig. 11B One or more of the functions in question.
[0109] After start operation 1105, process 1100 moves to operation 1110 where a cost factor is initialized to an initial value. In some aspects, the initial value is one (1). As described below, the cost factor is used to adjust a cost tolerance (typically lower in some embodiments) for the diagnostic action when performing multiple iterations of applying the diagnostic action.
[0110] In operation 1115, a possible root cause of the fault or problem is determined. As discussed above, in at least some embodiments, the possible root cause is identified based on a machine learning model that analyzes the monitored operating parameter values of the monitored system. Operation 1115 also includes identifying the possible root cause as an accurate determination or a probability or confidence that the cause of the problem is correct. For example, as discussed above with respect to Fig. 9As described, in some embodiments, the machine learning model 718 provides one or more root cause indicators 935 that indicate both individual root cause indicators 940a and associated probability or confidence indicators 940c. Some embodiments of operation 1115 also identify corrective actions based on the possible root causes. For example, as described above with respect to Figure 3 As discussed, some embodiments maintain an association between root causes and corrective actions. For example, Figure 3 This association is illustrated via a cause table 310, which includes a cause identifier field 312 and an action identifier field 314. Some embodiments identify component identifiers associated with possible root causes. For example, as described above with respect to Fig. 9 As discussed, in some embodiments, the machine learning model provides an output (e.g., 940b) that indicates components that may contribute to the root cause.
[0111] Decision operation 1120 determines whether the probability or confidence associated with the root cause is above a predetermined threshold. If the probability is above the predetermined threshold, process 1100 moves from decision operation 1120 to operation 1125, where a corrective action is performed. In some embodiments, the corrective action is associated with the root cause. For example, as described above with respect to Figure 3 As discussed, some embodiments implement a cause table 310 that associates a root cause with an action (eg, via a cause identifier field 312 and an action identifier field 314). The process then ends in operation 1149.
[0112] If the probability or confidence is not above the threshold, the process 1100 moves from decision operation 1120 to operation 1128, where the process determines the highest cost of the diagnostic action that the process is willing to accept. In some embodiments, the cost is determined based on a cost factor and the probability or confidence indicator 940c and the root cause provided by the machine learning model. The process 1100 then moves to operation 1130, which selects a diagnostic action based at least in part on the cost factor. Fig. 11B One embodiment of operation 1130 is discussed in detail in conjunction with process 1150. Fig.13A Another embodiment of operation 1130 is discussed in and process 1300. Some embodiments select a diagnostic action associated with a component that is less expensive. In at least some embodiments, the component is identified as described above via output from a machine learning model. Fig. 13B Another embodiment of operation 1130 is discussed in conjunction with process 1350 .
[0113] Decision operation 1135 determines whether operation 1130 has selected a diagnostic action. For example, if it is determined that there is a diagnostic action with an associated cost less than a particular threshold, operation 1130 is able to select a diagnostic action. Similarly, if it is determined that the costs associated with all possible diagnostic actions are greater than the threshold, operation 1130 may not be able to select a diagnostic action. If not, process 1100 moves from decision operation 1135 to end operation 1149. Alternatively, if an action is selected, process 1100 moves from decision operation 1135 to operation 1138, which injects the selected action. Then, process 1100 moves from operation 1138 to operation 1140. In operation 1140, the cost factor is adjusted. As described above, some embodiments iteratively inject diagnostic actions that attempt to increase the probability that a root cause has been identified. For each iteration, some embodiments reduce the cost tolerance of each subsequently injected action. As discussed further below in reviewing processes 1150 and Fig. 11B As will become apparent later, in at least some example embodiments, reducing the cost factor implements the method in operation 1140. After the cost factor has been reduced in operation 1140, process 1100 returns to 1115 where the root cause is re-determined and processing continues.
[0114] Fig. 11B is a flow chart of an example process for determining which diagnostic action should be performed. In one or more of the disclosed embodiments, Fig. 11B In some embodiments, one or more of the functions discussed below are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524) stored in electronic hardware memory (e.g., 1504 and / or 1506) configure the hardware processing circuitry (e.g., 1502) to perform one or more of the functions discussed below. In some embodiments, network management system 112 performs the following description of Fig. 11B Some embodiments of process 1150 are similar to those described above with respect to Fig.11A The process 1100 discussed above integrates. For example, in some embodiments, the process 1150 implements Fig.11A Operation 1130 is shown. Thus, in these embodiments, process 1150 inherits one or more parameters, states, and / or variables utilized by process 1100.
[0115] After starting operation 1155, process 1150 moves to operation 1165, which sets a cost tolerance based on a cost factor. At least in some embodiments, the cost factor is inherited from process 1100. Some embodiments implement a function that dynamically determines a cost tolerance based on a probability or confidence indicator 940c that the correct root cause has been identified and a cost factor. As an example, if the cost factor is a first value, process 1150 sets the cost tolerance threshold to a first tolerance, otherwise, process 1150 sets the cost tolerance threshold to a second tolerance. At least in some embodiments, the first tolerance is higher than the second tolerance. In some embodiments, the cost factor decreases with subsequent iterations. For example, in some embodiments, the cost factor has an initial value of one (1), wherein in some embodiments, subsequent values are reduced by one-tenth (0.1). The amount by which the cost factor is reduced for each iteration varies depending on the embodiment. In some embodiments, the cost tolerance for the diagnostic action is proportional to the confidence in the root cause determination (e.g., as determined by a machine learning model (such as, for example, the above description of Figure 7 and Fig. 9 The model 718) discussed provides an inverse relationship between .
[0116] After operation 1165 is completed, process 1150 moves to operation 1170 of identifying multiple diagnostic actions. The manner of identifying diagnostic actions may vary depending on the embodiment. As discussed above, in some embodiments, one or more diagnostic actions are first identified based on components or component types (e.g., 940b) identified by the machine learning model as being associated with the root cause of the underlying problem. These embodiments maintain the association between components and / or component types and (multiple) diagnostic actions (e.g., via diagnostic action table 350). When the machine learning model identifies a particular type of component as being associated with a possible problem, the (multiple) diagnostic actions associated with the component type are considered for injection into the monitored system. In at least some embodiments, the component identifier and / or component type information is inherited from process 1100.
[0117] In operation 1175, a cost of each of the plurality of diagnostic actions is determined. As discussed above, some embodiments dynamically determine the cost of an action. In some embodiments, the cost of an action is based at least in part on the number of users affected by the performance of the action. Thus, for example, if the action includes restarting a wireless device, then in some embodiments, the number of users currently transmitting via the device is used to determine the cost of performing the action.
[0118] Operation 1180 selects a diagnostic action from a plurality of diagnostic actions. Operation 1180 ensures that the cost of the selected action is less than the cost indicated by the cost tolerance. When a plurality of diagnostic actions are available, embodiments may vary in the manner of selecting a single diagnostic action in operation 1180. Some embodiments of operation 1180 rank available candidate diagnostic actions based on the cost of available candidate diagnostic actions. Actions whose cost exceeds the cost tolerance are eliminated from the ranking. Then, these embodiments select diagnostic actions according to the ranking. For example, the first iteration of process 1150 selects the diagnostic action with the highest ranking (lowest cost), wherein subsequent iterations select diagnostic actions with gradually lower rankings. Some embodiments may adjust the ranking of possible actions not only based on cost but also based on previous injections of those actions. For example, some embodiments track any improvement in the confidence level of the root cause caused by the injection of diagnostic actions. Then, the ranking is based on both cost and previous relative improvement. Other embodiments may use the alternative techniques of the examples provided herein to select from a plurality of candidate diagnostic actions or possible diagnostic actions. Figure 13 provides another example of the manner of selecting diagnostic actions in multiple iterations. After operation 1180 is completed, process 1100 moves to end operation 1190 .
[0119] Fig.12 is a flow chart of an example process for determining whether to perform a corrective action or a diagnostic action to be performed in one or more of the disclosed embodiments. In some embodiments, one or more of the functions discussed below are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524) stored in electronic hardware memory (e.g., 1504 and / or 1506) configure the hardware processing circuitry (e.g., 1502) to perform one or more of the functions discussed below. In some embodiments, the network management system 112 performs the following with respect to Fig.12 One or more of the functions in question.
[0120] After start operation 1205, process 1200 moves to operation 1210, which evaluates the possible causes and corrective actions and probabilities received from the machine learning model. For example, in embodiments utilizing a machine learning model that provides one or more possible causes of an underlying problem and probabilities associated with each of those causes (e.g., as described above for one or more root cause indicators 935 (including individual root cause indicators 940a and probability or confidence indicators 940c)), operation 1210 compares the probabilities to determine the cause of the underlying problem with the highest probability.
[0121] Decision operation 1215 evaluates whether the cause with the highest probability or confidence identified in operation 1210 meets a first criterion. In some embodiments, the first criterion evaluates whether the probability or confidence associated with the cause is above a first predetermined threshold. If the first criterion is met (e.g., the probability or confidence is above the first predetermined threshold), process 1200 moves from decision operation 1215 to operation 1248 discussed below. If the first criterion is not met, process 1200 moves from decision operation 1215 to operation 1220.
[0122] Operation 1220 sets a diagnostic action tolerance threshold based on the highest probability. In some embodiments, if the highest probability is within a first range, the diagnostic action threshold is set to a first value, and if the highest probability falls within a second range, the diagnostic action threshold is set to a second value. The disclosed embodiments contemplate any number of ranges and values. Therefore, determination operations 1215 and 1220 describe example implementations of thresholds for taking diagnostic actions based at least in part on a confidence level or probability associated with one or more corrective actions that is higher than a threshold. Therefore, if there is a high confidence solution for correcting a problem, the need for further diagnostic actions is reduced. By reducing the cost of the tolerance threshold for diagnostic actions, these embodiments suppress more expensive diagnostic actions when a relatively high confidence level has been obtained in the solution.
[0123] In operation 1230, a diagnostic action is selected. In some embodiments, a diagnostic action is obtained based on the component associated with the most likely cause identified by operation 1210. For example, as described above with respect to Figure 3 As discussed, some embodiments associate component types with diagnostic actions (e.g., via diagnostic action table 350). In some cases, multiple diagnostic operations are associated with components or component types. Various embodiments use various techniques to select diagnostic actions. Some embodiments select the diagnostic action with the lowest cost among multiple diagnostic actions. Other embodiments determine a score associated with each diagnostic action injected into multiple diagnostic actions, and select a diagnostic action based on the score. Figure 13 describes an embodiment of selecting a diagnostic action. Fig. 11B Another embodiment of selecting a diagnostic action is also described.
[0124] Determine operation 1240 whether the cost of the diagnostic action selected by operation 1230 is less than a diagnostic action threshold. As discussed above, some embodiments dynamically evaluate or determine the costs associated with the diagnostic action based on the number of users that will be affected by the execution of the diagnostic action. In some embodiments, the number of users is the number of users communicating via the device on which the diagnostic action is performed.
[0125] If the cost of performing the diagnostic action is less than the diagnostic action threshold, the process 1200 moves from decision operation 1240 to operation 1245, which performs the diagnostic action. In at least some embodiments, performing the diagnostic action includes: restarting a specific radio of the AP; restarting a specific module; restarting all radios of the AP; powering off the AP, etc. and collecting operating parameters immediately after the injected diagnostic action. In some embodiments, performing the diagnostic action includes: notifying an address included in the warning list associated with the diagnostic action (e.g., via the warning list identifier field 316 of the root cause table 310). If the cost of the diagnostic action exceeds the diagnostic action threshold (the perceived benefit of injecting the diagnostic action), the process 1200 moves from decision operation 1240 to operation 1248, which performs the corrective action associated with the cause with the highest probability or confidence. The cause with the highest probability is identified at least in operation 1210 as discussed above. In some embodiments, if the probability that the corrective action solves the problem is below a predetermined lower probability threshold, the corrective action is not performed. In some embodiments, operation 1248 also includes addressing the generated alert to one or more messages associated with the root cause (eg, via cause table 310 ). After operation 1245 or operation 1248 is completed, process 1200 moves to end operation 1249 .
[0126] Fig.12 A single assessment of the probability that a root cause of an underlying problem has been identified is illustrated. If the probability that a root cause has been identified is below a certain threshold, one of a plurality of actions is invoked to facilitate the collection of additional debug information. Each injected action (e.g., restart beacon; restart radio; restart a particular software module; restart a particular hardware module; restart the AP; power cycle the AP, etc.) has a cost associated with it. In some embodiments, the rule defines that the impact (cost) of an injected action that the system is willing to accept is inversely proportional to the confidence that a root cause of an underlying problem has been identified.
[0127] After injecting an action to facilitate the collection of additional debug data, the additional debug data is provided to the machine learning model. Based on the additional debug data, the machine learning model outputs new estimates of root causes and their corresponding probabilities. These new probabilities are compared to thresholds, and if the new probabilities are still below a certain threshold, a second new tolerance threshold is determined and used to decide which debug (data collection) action should be injected into the system. The new action is injected into the system, and new data is collected and used as input to a machine learning model that attempts to identify offensive components.
[0128] In some embodiments, the process continues to iterate until the specific component is determined to be the root cause of the underlying problem or failure, or until it is determined that the cost of injecting another action to facilitate the collection of additional data is too expensive compared to the benefit of collecting additional data. In some embodiments, each time a specific action is injected, the tolerance threshold for the cost of accepting the action injection is lowered. In this way, the number of times a specific action is injected is limited because each time after the specific action is injected, the acceptable cost threshold for the action is lowered.
[0129] Fig.13A is a flow chart of an example process for determining an action to inject based on a cost-benefit analysis of the injecting action. Fig.13A In some embodiments, one or more of the functions discussed below are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524) stored in electronic hardware memory (e.g., 1504 and / or 1506) configure the hardware processing circuitry (e.g., 1502) to perform the following instructions regarding Fig.13A and one or more of the functions discussed in process 1300. In some embodiments, network management system 112 performs the following Fig.13A and one or more of the functions discussed in process 1300. Fig.11A Some embodiments of operation 1130 discussed include the following: Fig.13A and one or more of the functions discussed in process 1300. Fig.10 Some embodiments of operation 1020 discussed below implement the following Fig.13A and one or more of the functions discussed in process 1300.
[0130] After start operation 1305, process 1300 moves to operation 1310, in which multiple candidate diagnostic actions or possible diagnostic actions are identified. As discussed above, some embodiments identify candidate diagnostic actions or possible diagnostic actions via a mapping between component types and diagnostic actions (e.g., via diagnostic action table 350 and / or component table 370). In some embodiments, the component type is obtained based on an output from a machine learning model indicating a possible root cause of the problem (e.g., component identifier 940b output by machine learning model 718 identifies a component that may be the source of the problem).
[0131] Operation 1315 selects a single action from a plurality of possible diagnostic actions. As described below, operation 1315 is designed to iteratively select different diagnostic actions from a plurality of diagnostic actions as process 1300 iterates.
[0132] After operation 1315, decision operation 1320 determines whether the selected diagnostic action has been previously injected. In some embodiments, the determination of whether the diagnostic action has been previously injected evaluates whether the action has been previously injected within a predetermined elapsed time of the current time. In some embodiments, the determination of whether the action has been previously injected is related to a specific determination of a possible root cause. For example, some embodiments of process 1300 are as described above with respect to Fig.11A The process 1100 discussed inherits the root cause determination (e.g., as determined by operation 1115). Some embodiments maintain a history of the actions injected. (e.g., via injection history table 360). These embodiments track the history of the injected actions and any improvement in probability in the root cause determination that occurred after the injection of the actions. In some embodiments, the process 1300 determines the benefit of the injected action from this information. The benefit can be specific to a particular component or component type (e.g., via component id field 366 and / or component table 370).
[0133] If the action has not been previously injected, the process 1300 moves from the decision operation 1320 to an operation 1325, which determines the cost of the action. For example, as discussed above, some embodiments maintain an association between an action and the cost of applying the action. (For example, the action table 320 stores a cost function in the cost function field 326 to calculate the cost of the action, which provides a dynamic determination of the cost of the action). In at least some embodiments, the dynamic determination of the cost of the action is based on the number of users transmitted via the identified component. Then, operation 1325 determines a score for the action based on the cost. In some embodiments, the score is inversely proportional to the cost.
[0134] If the action was previously injected, process 1300 moves from decision operation 1320 to operation 1330, which determines the history of injected actions. Determining the history can include determining the number of times a particular action has been injected previously and the specific circumstances under which the particular action was injected (e.g., to correct for components, injection time, etc.).
[0135] Operation 1335 evaluates any previous changes in probability in the root cause determination based on previous injections of the action. For example, if an action was injected between a first root cause determination and a second root cause determination, operation 1335 determines the difference between the probabilities associated with the first root cause determination and the second root cause determination. Some embodiments of operation 1335 determine multiple differences in confidence levels or probabilities between multiple pairs of root cause determinations. Some embodiments predict differences in probability determinations based on differences in previous probability determinations resulting from previous injections of the action. For example, some embodiments examine the injection history of the action and predict the next probability increase in the injected action based on the history of probability differences.
[0136] Decision operation 1336 evaluates whether the determined probability change meets the standard. In some embodiments, the standard evaluates whether the rate of improvement of the probability or confidence in the root cause determination exceeds a threshold rate. In some embodiments, the standard evaluates the change of the probability or confidence value of the injection action over time. If the amount of change or the rate of change is lower than the threshold, some embodiments move from decision operation 1336 to decision operation 1345. This causes no score to be generated for the current diagnostic action, and actually removes the current diagnostic action from consideration. Otherwise, process 1300 moves from decision operation 1336 to operation 1340.
[0137] Operation 1340 determines a score for the action based on the determined probability change and the cost of the action in operation 1335. For example, some embodiments of operation 1340 relate the probability difference or probability improvement between injected actions to the action cost, where a greater improvement in probability relative to the cost provides a relatively better score.
[0138] Decision operation 1345 determines whether there are additional actions to be evaluated in multiple diagnostic actions. If additional diagnostic actions can be used to determine cost / benefit measurements, process 1300 moves from decision operation 1345 to operation 1315. Operation 1315 selects additional actions, and processing continues as described above. Alternatively, if all actions in multiple actions have been processed, process 1300 moves from decision operation 1345 to operation 1346, which selects the action to be injected from multiple actions with scores. The selection is based at least in part on the score determined by process 1300. For example, in some embodiments, the action with the highest score is selected. After operation 1346 is completed, process 1300 moves to end operation 1348.
[0139] Fig. 13B is a flow chart of an example process for determining an action to inject based on a cost-benefit analysis of the injecting action. Fig. 13BIn some embodiments, one or more of the functions discussed below are performed by hardware processing circuitry. For example, in some embodiments, instructions (e.g., 1524) stored in electronic hardware memory (e.g., 1504 and / or 1506) configure the hardware processing circuitry (e.g., 1502) to perform the following instructions regarding Fig. 13B and process 1350. In some embodiments, the network management system 112 performs the following Fig. 13B and one or more of the functions discussed in process 1350. Fig.11A Some embodiments of operation 1130 discussed include the following: Fig. 13B and one or more of the functions discussed in process 1350. Fig.10 Some embodiments of operation 1020 discussed below implement the following Fig. 13B and one or more of the functions discussed in process 1350.
[0140] After start operation 1355, process 1350 moves to operation 1360, in which multiple candidate diagnostic actions or possible diagnostic actions are identified. As discussed above, some embodiments identify candidate diagnostic actions or possible diagnostic actions via a mapping between component types and diagnostic actions (e.g., via diagnostic action table 350 and / or component table 370). In some embodiments, the component type is obtained based on an output from a machine learning model indicating a possible root cause of the problem (e.g., component identifier 940b output by machine learning model 718 identifies a component that may be the source of the problem).
[0141] Decision operation 1365 determines whether there are any diagnostic operations that meet the tolerable cost threshold. If there are no diagnostic operations that meet this criterion, the process ends in operation 1399. However, if decision operation 1365 identifies one or more diagnostic actions whose cost is below the tolerable cost threshold, the process 1350 moves to operation 1370, which selects a single action from a plurality of possible diagnostic actions. As described below, operation 1370 is designed to iteratively select different diagnostic actions from a plurality of diagnostic actions as process 1350 iterates.
[0142] After operation 1370, operation 1375 injects or invokes diagnostic actions, and the system collects the resulting operational data and uses it as input to a machine learning process that determines the root cause with a new certainty probability. As previously described, the new probabilistic certainty is used to determine a new tolerance cost for additional diagnostic actions.
[0143] After operation 1375 is completed, the process 1350 moves to a decision operation 1380 to determine whether the selected diagnostic action has been used previously. If this is the first time the diagnostic action has been used, the process moves to operation 1396.
[0144] Operation 1396 checks the tolerance costs derived in operation 1375 and if it finds diagnostic actions with costs above the new cost tolerance, it removes those actions from the list of possible diagnostic actions.
[0145] After operation 1396 is completed, process 1350 loops back to operation 1360.
[0146] Returning to the discussion of decision operation 1380, if the same action has been previously invoked, process 1350 moves from decision operation 1380 to operation 1385, where the impact history of the diagnostic action is examined. More specifically, operation 1390 determines the change between the confidences in the root cause determinations that have been achieved by successive invocations of the diagnostic action.
[0147] Decision operation 1392 determines whether the change or improvement in the determination of the root cause is greater than a predetermined threshold. If decision operation 1392 determines that reuse of the diagnostic action has improved the determination of the root cause by more than the threshold, process 1350 moves to operation 1396 and then returns to operation 1310 as described above. On this path, the diagnostic action may be attempted again, as there is hope of improving the probability of identifying the root cause.
[0148] However, if operation 1392 determines that reusing the diagnostic action does not improve the ability to collect new information that can help machine learning determine the root cause, the process moves to operation 1394, in which the diagnostic action is removed from the list of possible diagnostic actions.
[0149] The process 1350 then moves to operation 1396, which functions as described above. After operation 1396 is completed, the process 1350 returns to operation 1310, where at least one less diagnostic action is in the list of possible diagnostic actions.
[0150] Fig.14A is a diagram illustrating the operation of one or more of the disclosed embodiments. Fig.14A The following rule is illustrated: When the probability or confidence that the identified root cause causes the problem is below a threshold 1405, an action (diagnostic action) having a greater cost than the second diagnostic action may be applied to the monitored system. Similarly, Fig.14A The embodiment operates as follows Fig.14AA predetermined rule is applied in the process that indicates that when the confidence of the root cause generated by the ML model is below a threshold 1405, network management can inject actions with higher (escalating) costs, and similarly, when the confidence or probability that the root cause causes the identified problem is above a predetermined threshold, a lower cost action is applied. In other words, in some embodiments, if the confidence that the underlying problem can be automatically resolved is low, a relatively higher cost diagnostic action can be injected to collect additional diagnostic information. However, when the root cause is determined with high confidence, only relatively lower cost diagnostic actions are allowed to be injected because the need for additional diagnostic information is relatively small.
[0151] Fig. 14B Illustrated are example preconfigured rules implemented in some of the disclosed embodiments. Fig. 14B The rules in guide network management: if the cost associated with the injected action used to determine the root cause is less than a predetermined cost tolerance threshold 1410, then a more expensive action is allowed to be injected into the device, and similarly, if the cost of the injected action used to determine the root cause is above a predetermined threshold, then the next more expensive action is not allowed to be injected into the device. As explained above, each time after an action is injected into the system, new current information is obtained and used as input to the machine learning process. Machine learning produces output 795 as candidates for being the root cause of the underlying problem and the probabilities associated with each of these root cause candidates. As a result, the system calculates a new cost threshold for allowing additional actions to be injected into the communication system. Therefore, in each iteration, the predetermined cost tolerance threshold 1410 is dynamically calculated.
[0152] Fig. 14C Another example preconfigured rule is illustrated in which network management is guided according to a bending cost tolerance threshold 1415, which is a function of the confidence of the machine learning model in determining the root cause of the underlying problem. The bending cost tolerance threshold 1415 illustrates that when the confidence that the machine learning has identified the root cause of the underlying problem is low, the rule allows for the injection or invocation of a more costly diagnostic action, such as restarting the AP (rather than just restarting a specific radio in the AP). Thus, for example, when the confidence in the ability of the corrective action to resolve the fault or underlying problem is below a threshold, some embodiments determine that a relatively more costly diagnostic action is acceptable for injection or invocation. This contrasts with a relatively lower cost tolerance for the injected diagnostic action when the confidence in identifying the root cause and invoking the corrective action to resolve the problem is relatively higher.
[0153] Fig.14DAnother example of a preconfigured rule is illustrated in which network management is guided according to a threshold 1420 that is a function of the difference between the confidences of the machine learning model in determining the root cause of the underlying problem based on two consecutive calls to the same action (injecting an action that facilitates the collection of additional debugging information). Specifically, as explained in more detail in FIG. 13, some embodiments allow diagnostic actions to be upgraded to more expensive diagnostic actions when the incremental benefits (cost delta) of repeatedly injecting (the same) actions, collecting current information (e.g., information 790) and determining the root cause produce a lower cost benefit. A more costly action can have a relatively wider range of impact than a less costly action.
[0154] Fig.15 A block diagram of an example machine 1500 that can perform any one or more of the techniques (e.g., methods) discussed herein is illustrated. The machine 1500 (e.g., a computer system) can include a hardware processor 1502 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1504, and a static memory 1506, some or all of which can communicate with each other via an interconnection link 1508 (e.g., a bus).
[0155] Specific examples of main memory 1504 include random access memory (RAM) and semiconductor memory devices, which in some embodiments may include storage locations (such as registers) in a semiconductor. Specific examples of static memory 1506 include non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable magnetic disks; magneto-optical disks; RAM; and CD-ROM disks and DVD-ROM disks.
[0156] The machine 1500 may further include a display device 1510, an input device 1512 (e.g., a keyboard), and a user interface (UI) navigation device 1514 (e.g., a mouse). In an example, the display device 1510, the input device 1512, and the UI navigation device 1514 may be a touch screen display. The machine 1500 may additionally include a mass storage device (e.g., a drive unit) 1516, a signal generating device 1518 (e.g., a speaker), a network interface device 1520, and one or more sensors 1521, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 1500 may include an output controller 1528, such as a serial connection (e.g., a universal serial bus (USB), a parallel connection, or other wired connection or a wireless connection (e.g., infrared (IR), near field communication (NFC), etc.) for transmitting or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.). In some embodiments, the hardware processor 1502 and / or the instruction 1524 may include a processing circuit device and / or a transceiver circuit device.
[0157] The mass storage device 1516 may include a machine-readable storage medium 1522 having stored thereon one or more sets of data structures or instructions 1524 (e.g., software) implemented or utilized by any one or more of the techniques or functionality described herein. The instructions 1524 may also reside, completely or at least partially, within the main memory 1504, within the static memory 1506, or within the hardware processor 1502 during execution of the instructions 1524 by the machine 1500. In an example, one or any combination of the hardware processor 1502, the main memory 1504, the static memory 1506, or the mass storage device 1516 may constitute a machine-readable medium.
[0158] Specific examples of machine-readable media may include: nonvolatile memory, such as semiconductor memory devices (e.g., EPROM or EEPROM) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; RAM; and CD-ROM disks and DVD-ROM disks.
[0159] Although the machine-readable medium 1522 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (eg, a centralized or distributed database and / or associated caches and servers) configured to store the instructions 1524 .
[0160] The apparatus of the machine 1500 may be one or more of the following: a hardware processor 1502 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1504 and a static memory 1506, one or more sensors 1521, a network interface device 1520, one or more antennas 1560, a display device 1510, an input device 1512, a UI navigation device 1514, a mass storage device 1516, instructions 1524, a signal generation device 1518, and an output controller 1528. The apparatus may be configured to perform one or more of the methods and / or operations disclosed herein. The apparatus may be intended to be a component of the machine 1500 to perform one or more of the methods and / or operations disclosed herein, and / or to perform one or more of the methods and / or operations disclosed herein. In some embodiments, the apparatus may include a pin or other means for receiving power. In some embodiments, the apparatus may include power regulation hardware.
[0161] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 1500 and causing the machine 1500 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), and flash memory devices); disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); and CD-ROM disks and DVD-ROM disks. In some examples, the machine-readable medium may include non-transient machine-readable media. In some examples, the machine-readable medium may include machine-readable media that is not a transient propagating signal.
[0162] The instructions 1524 may be further transmitted or received over a communication network 1526 using a transmission medium via the network interface device 152, which utilizes any of a number of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include: a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network known as a wireless network). The Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as IEEE802.16 series standards), IEEE 802.15.4 series standards, Long Term Evolution (LTE) series standards, Universal Mobile Telecommunications System (UMTS) series standards, peer-to-peer (P2P) networks, etc.
[0163] In an example, the network interface device 1520 may include one or more physical jacks (e.g., Ethernet jacks, coaxial jacks, or telephone jacks) or one or more antennas for connecting to the communication network 1526. In an example, the network interface device 1520 may include one or more antennas 1560 for wireless communication using at least one of single-input multiple-output (SIMO) technology, multiple-input multiple-output (MIMO) technology, or multiple-input single-output (MISO) technology. In some examples, the network interface device 1520 may use multi-user MIMO technology for wireless communication. The term "transmission medium" should be taken to include any intangible medium that can store, encode, or carry instructions for execution by the machine 1500, and includes digital communication signals or analog communication signals or other intangible media for facilitating the communication of such software.
[0164] Examples as described herein may include logic or multiple components, modules or mechanisms, or may operate on logic or multiple components, modules or mechanisms. A module is a tangible entity (e.g., hardware) that can perform a specified operation, and a module may be configured or arranged in a particular manner. In an example, a circuit may be arranged as a module (e.g., internally or relative to an external entity (such as, other circuits)) in a specified manner. In an example, all or part of one or more computer systems (e.g., independent client or server computer systems) or one or more hardware processors may be configured to operate as a module that performs a specified operation by firmware or software (e.g., instructions, application parts or applications). In an example, software may reside on a machine-readable medium. In an example, the software causes the hardware to perform a specified operation when executed by the underlying hardware of the module.
[0165] Although the above flow charts have been discussed with respect to a specific sequence of events, it should be appreciated that changes may be made to the sequence without materially affecting the operation of the embodiment(s). Additionally, the example techniques described herein are not limited to the specifically described embodiments, but rather may also be utilized with other example embodiments, and each described feature may be individually and separately claimed.
[0166] Those skilled in the art should recognize that although the above discussion focuses on the measurement of SLE parameters, the terms (Key Performance Indicator) KPI parameters and SLE parameters should be considered interchangeable, and as such, the disclosed embodiments cover scenarios in which KPI parameters are used in conjunction with or in place of SLE parameters.
[0167] The above system may be implemented on (multiple) wireless communication devices / systems, such as IEEE 802.11 transceivers, etc. Examples of wireless protocols that may be used with the technology include: IEEE 802.11a, IEEE 802.11b, IEEE802.11g, IEEE 802.11n, IEEE 802.11ac, IEEE 802.11ad, IEEE 802.11af, IEEE 802.11ah, IEEE 802.11ai, IEEE 802.11aj, IEEE 802.11aq, IEEE 802.11ax, Wi-Fi, LTE, 4G, WirelessHD, WiGig, WiGi, 3GPP, Wireless LAN, WiMAX, DensiFi SIG, Unifi SIG, 3GPP LAA (License Assisted Access), etc. Similarly, the above embodiments can be implemented on (multiple) wired communication devices / systems and / or (multiple) optical communication devices / systems.
[0168] Additionally, these systems, methods, and protocols may be implemented to improve one or more of the following: a special purpose computer, a programmed microprocessor or microcontroller and (multiple) peripheral integrated circuit components, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic device or logic circuit (such as a discrete component circuit), a programmable logic device (such as a PLD, PLA, FPGA, PAL), a modem, a transmitter / receiver, any comparable device, etc. In general, any device capable of implementing a state machine, and thus capable of implementing the methods described herein, may benefit from the various communication methods, protocols, and techniques according to the disclosure provided herein.
[0169] Examples of a processor as described herein may include, but are not limited to, at least one of the following: 800 and 801, with 4G LTE integration and 64-bit computing 610 and 615, with 64-bit architecture A7 processor, M7 motion coprocessor, series, Core TM Series processors, Series processors, Atom TM Series processors, Intel Series processors, i5-4670K and i7-4770K 22nm Haswell, i5-3570K22nm Ivy Bridge, FX TM Series processors, FX-4300, FX-6300 and FX-8350 32nmVishera, Kaveri processor, Texas Jacinto C6000 TM Automotive infotainment processors, Texas OMAP TM Automotive-grade mobile processors, Cortex TM -M processor, Cortex-A and ARM926EJ-S TM processor, AirForce BCM4704 / BCM4703 wireless networking processors, AR7100 wireless network processing units, other industry equivalent processors, and may use any known or future developed standards, instruction sets, libraries and / or architectures to perform computing functions.
[0170] In addition, the disclosed method can be easily implemented in software using an object or object-oriented software development environment that provides portable source code that can be used on various computer or workstation platforms. Alternatively, the disclosed system can be partially or completely implemented in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement the system according to the embodiment depends on the speed requirements and / or efficiency requirements of the system, the specific functions, and the specific software or hardware system or microprocessor or microcomputer system utilized. Through the functional description provided herein and having general basic knowledge of the computer and telecommunication fields, a person of ordinary skill in the applicable field can easily use any known or later developed system or structure, device and / or software to implement the communication system, method and protocol described herein in hardware and / or software.
[0171] Furthermore, the disclosed methods may be readily implemented in software and / or firmware that may be stored on a storage medium to enhance the performance of a programmed general-purpose computer in cooperation with a controller and memory, a special-purpose computer, a microprocessor, etc. In these instances, the systems and methods may be implemented as programs embedded on a personal computer (such as applet, JAVA.RTM or CGI script), resources resident on a server or computer workstation, routines embedded in a special-purpose communication system or system component, etc. The system may also be implemented by physically incorporating the system and / or method into a software system and / or a hardware system (such as a hardware system and software system of a communication transceiver).
[0172] Therefore, it is apparent that at least systems and methods for enhancing and improving communication reliability have been provided. Although embodiments have been described in conjunction with a number of embodiments, it is apparent that many alternatives, modifications, and variations will be apparent or obvious to those of ordinary skill in the applicable fields. Therefore, the present disclosure is intended to encompass all such alternatives, modifications, equivalents, and variations within the spirit and scope of the present disclosure.
[0173] Example 1 is a method comprising: receiving a time series of operating parameter values from one or more devices of a network system; providing the time series of operating parameter values to a machine learning model; receiving an indication of a cause of a failure in the operation of the network system from the machine learning model; selecting a first action to be performed on the network system based on the cause; performing the first action; and notifying the machine learning model of the first action performed.
[0174] In Example 2, the subject matter of Example 1 optionally includes: receiving a second time series of operating parameter values from the network system after performing the action; determining whether the fault is resolved based on the second time series; and conditionally applying a second action to the network system based on whether the fault is resolved.
[0175] In Example 3, the subject matter of any one or more of Examples 1 to 2 may optionally include: identifying a first distribution list associated with a first type of root causes; and identifying a second distribution list associated with a second type of root causes; and generating a warning based on the first distribution list and the second distribution list.
[0176] In Example 4, the subject matter of any one or more of Examples 1 to 3 optionally includes: wherein receiving a time series of operating parameter values includes: receiving a time series of corresponding device operating parameter values from a plurality of devices included in the network system; and providing each time series in the time series to a machine learning model.
[0177] In Example 5, the subject matter of any one or more of Examples 1 to 4 optionally includes: wherein the operating parameter value indicates one or more of the following: CPU utilization of the network component, memory utilization of the network component, latency at the network component, throughput of the network component, the number of connections maintained by the network component, a packet error count at the network component, or the number of associated wireless terminals at the network component.
[0178] In Example 6, the subject matter of any one or more of Examples 1 to 5 may optionally include: wherein the operating parameter value indicates one or more of: an access point name, a service set identifier, a channel, a frequency band, media access control (MAC) information, or a basic service set identifier.
[0179] In Example 7, the subject matter of any one or more of Examples 1 to 6 optionally includes: receiving information indicating message content exchanged between devices of the network system from one or more devices of the network system; and providing the information indicating the message content to a machine learning model.
[0180] In Example 8, the subject matter of any one or more of Examples 1 to 7 may optionally include: wherein selecting the first action includes: determining a first cost of the first action and a second cost of a second action associated with the cause; and selecting the first action or the second action based on the first cost and the second cost.
[0181] In Example 9, the subject matter of Example 8 optionally includes: wherein the first action or the second action is one of: resetting a device included in the network system; generating a status request for a component of the network system; resetting a hardware component of a device included in the network system; resetting a software component or a firmware component of a device included in the network system; or requesting a component of the network system to perform a task.
[0182] In Example 10, the subject matter of any one or more of Examples 1 to 9 optionally includes: performing a first evaluation of the confidence that the selected action will resolve the fault; setting a diagnostic action cost threshold based on the first evaluation; performing a second evaluation of the diagnostic action based on the diagnostic action cost threshold; and conditionally performing the diagnostic action based on the second evaluation.
[0183] In Example 11, the subject matter of Example 10 may optionally include: if the confidence level is higher than a predetermined threshold, setting the diagnostic action cost threshold to a first value, otherwise, setting the diagnostic action cost threshold to a second value, wherein the first value is lower than the second value.
[0184] In Example 12, the subject matter of any one or more of Examples 10 to 11 optionally includes injecting a first diagnostic action having a first cost instead of a second diagnostic action having a second cost, the second cost being lower than the first cost, the injecting being responsive to the confidence being lower than an upgrade threshold.
[0185] In Example 13, the subject matter of any one or more of Examples 10 to 12 optionally includes: injecting a first diagnostic action for a first time, the first diagnostic action having a first cost; after the injection of the first diagnostic action, monitoring an operating parameter value; determining a first root cause and an associated first probability based on the monitored operating parameter; injecting the first diagnostic action a second time based on a determination that the first probability is lower than a predetermined threshold; after the second injection of the first diagnostic action, monitoring the operating parameter value a second time; determining a second probability associated with the first root cause; adjusting a diagnostic cost threshold based on the first probability and the second probability; and determining whether to inject an additional diagnostic action based on the adjusted diagnostic cost threshold.
[0186] In Example 14, the subject matter of Example 13 can optionally include determining a difference between the first probability and the second probability, wherein determining whether to inject an additional diagnostic action is based on the difference.
[0187] Example 15 is a non-transitory computer-readable storage medium comprising instructions that, when executed, configure a hardware processing circuit device to perform operations, the operations comprising: receiving a time series of operating parameter values from one or more devices of a network system; providing the time series of operating parameter values to a machine learning model; receiving an indication of a cause of a failure in the operation of the network system from the machine learning model; selecting a first action to be performed on the network system based on the cause; performing the first action; and notifying the machine learning model of the first action performed.
[0188] In Example 16, the subject matter of Example 15 optionally includes: receiving a second time sequence of operating parameter values from the network system after performing the action; determining whether the fault is resolved based on the second time sequence; and conditionally applying a second action to the network system based on whether the fault is resolved.
[0189] In Example 17, the subject matter of any one or more of Examples 15 to 16 may optionally include: identifying a first distribution list associated with a first type of root causes; and identifying a second distribution list associated with a second type of root causes; and generating a warning based on the first distribution list and the second distribution list.
[0190] In Example 18, the subject matter of any one or more of Examples 15 to 17 optionally includes: wherein receiving a time series of operating parameter values includes: receiving a time series of corresponding device operating parameter values from a plurality of devices included in the network system; and providing each time series in the time series to a machine learning model.
[0191] In Example 19, the subject matter of any one or more of Examples 15 to 18 optionally includes: wherein the operating parameter value indicates one or more of: CPU utilization of the network component, memory utilization of the network component, latency at the network component, throughput of the network component, the number of connections maintained by the network component, a packet error count at the network component, or the number of associated wireless terminals at the network component.
[0192] In Example 20, the subject matter of any one or more of Examples 15 to 19 optionally includes: wherein the operating parameter value indicates one or more of the following: an access point name, a service set identifier, a channel, a frequency band, a media access control (MAC) information, or a basic service set identifier.
[0193] In Example 21, the subject matter of any one or more of Examples 15 to 20 optionally includes: receiving information indicating the content of messages exchanged between devices of the network system from one or more devices of the network system; and providing the information indicating the message content to a machine learning model.
[0194] In Example 22, the subject matter of any one or more of Examples 15 to 21 may optionally include: wherein selecting the first action includes: determining a first cost of the first action and a second cost of a second action associated with the cause; and selecting the first action or the second action based on the first cost and the second cost.
[0195] In Example 23, the subject matter of Example 22 optionally includes: wherein the first action or the second action is one of: resetting a device included in the network system; generating a status request for a component of the network system; resetting a hardware component of a device included in the network system; resetting a software component or a firmware component of a device included in the network system; or requesting a component of the network system to perform a task.
[0196] In Example 24, the subject matter of any one or more of Examples 15 to 23 optionally includes: performing a first evaluation of the confidence that the selected action will resolve the fault; setting a diagnostic action cost threshold based on the first evaluation; performing a second evaluation of the diagnostic action based on the diagnostic action cost threshold; and conditionally performing the diagnostic action based on the second evaluation.
[0197] In Example 25, the subject matter of Example 24 may optionally include: if the confidence level is higher than a predetermined threshold, setting the diagnostic action cost threshold to a first value, otherwise, setting the diagnostic action cost threshold to a second value, wherein the first value is lower than the second value.
[0198] In Example 26, the subject matter of any one or more of Examples 24 to 25 optionally includes: injecting a first diagnostic action having a first cost instead of a second diagnostic action having a second cost, the second cost being lower than the first cost, the injecting being responsive to a confidence level being lower than an upgrade threshold.
[0199] In Example 27, the subject matter of any one or more of Examples 24 to 26 optionally includes: injecting a first diagnostic action for a first time, the first diagnostic action having a first cost; after the injection of the first diagnostic action, monitoring an operating parameter value; determining a first root cause and an associated first probability based on the monitored operating parameter; injecting the first diagnostic action a second time based on a determination that the first probability is lower than a predetermined threshold; after the second injection of the first diagnostic action, monitoring the operating parameter value a second time; determining a second probability associated with the first root cause; adjusting a diagnostic cost threshold based on the first probability and the second probability; and determining whether to inject an additional diagnostic action based on the adjusted diagnostic cost threshold.
[0200] In Example 28, the subject matter of Example 27 can optionally include determining a difference between the first probability and the second probability, wherein determining whether to inject an additional diagnostic action is based on the difference.
[0201] Example 29 is a system comprising: a hardware processing circuit device; one or more hardware memories storing instructions that, when executed, configure the hardware processing circuit device to perform operations, the operations comprising: receiving a time series of operating parameter values from one or more devices of a network system; providing the time series of operating parameter values to a machine learning model; receiving an indication of a cause of a failure in the operation of the network system from the machine learning model; selecting a first action to be performed on the network system based on the cause; performing the first action; and notifying the machine learning model of the first action performed.
[0202] In Example 30, the example of Example 29 optionally includes operations, which further include: receiving a second time series of operating parameter values from the network system after performing the action; determining whether the fault is resolved based on the second time series; and conditionally applying a second action to the network system based on whether the fault is resolved.
[0203] In Example 31, the subject matter of any one or more of Examples 29 to 30 optionally includes operations further comprising: identifying a first distribution list associated with a first class of root causes; and identifying a second distribution list associated with a second class of root causes; and generating a warning based on the first distribution list and the second distribution list.
[0204] In Example 32, the subject matter of any one or more of Examples 29 to 31 optionally includes: wherein receiving a time series of operating parameter values includes: receiving a time series of corresponding device operating parameter values from a plurality of devices included in the network system; and providing each time series in the time series to a machine learning model.
[0205] In Example 33, the subject matter of any one or more of Examples 29 to 32 optionally includes: wherein the operating parameter value indicates one or more of: CPU utilization of the network component, memory utilization of the network component, latency at the network component, throughput of the network component, the number of connections maintained by the network component, a packet error count at the network component, or the number of associated wireless terminals at the network component.
[0206] In Example 34, the subject matter of any one or more of Examples 29 to 33 optionally includes: wherein the operating parameter value indicates one or more of: an access point name, a service set identifier, a channel, a frequency band, media access control (MAC) information, or a basic service set identifier.
[0207] In Example 35, the subject matter of any one or more of Examples 29 to 34 optionally includes operations further comprising: receiving information indicating the content of messages exchanged between devices of the network system from one or more devices of the network system; and providing the information indicating the content of the messages to the machine learning model.
[0208] In Example 36, the subject matter of any one or more of Examples 29 to 35 may optionally include: wherein selecting the first action includes: determining a first cost of the first action and a second cost of a second action associated with the cause; and selecting the first action or the second action based on the first cost and the second cost.
[0209] In Example 37, the subject matter of Example 36 optionally includes: wherein the first action or the second action is one of: resetting a device included in the network system; generating a status request for a component of the network system; resetting a hardware component of a device included in the network system; resetting a software component or a firmware component of a device included in the network system; or requesting a component of the network system to perform a task.
[0210] In Example 38, the subject matter of any one or more of Examples 29 to 37 optionally includes operations that further include: performing a first assessment of the confidence that the selected action will resolve the fault; setting a diagnostic action cost threshold based on the first assessment; performing a second assessment of the diagnostic action based on the diagnostic action cost threshold; and conditionally performing the diagnostic action based on the second assessment.
[0211] In Example 39, the subject matter of Example 38 optionally includes operations further comprising: if the confidence level is higher than a predetermined threshold, setting the diagnostic action cost threshold to a first value, otherwise, setting the diagnostic action cost threshold to a second value, wherein the first value is lower than the second value.
[0212] In Example 40, the subject matter of any one or more of Examples 38 to 39 optionally includes operations further comprising: injecting a first diagnostic action having a first cost instead of a second diagnostic action having a second cost, the second cost being lower than the first cost, the injecting being responsive to a confidence level being lower than an upgrade threshold.
[0213] In Example 41, the subject matter of any one or more of Examples 38 to 40 optionally includes operations, which further include: injecting a first diagnostic action for a first time, the first diagnostic action having a first cost; after injecting the first diagnostic action, monitoring an operating parameter value; determining a first root cause and an associated first probability based on the monitored operating parameter; injecting the first diagnostic action a second time based on a determination that the first probability is lower than a predetermined threshold; after injecting the first diagnostic action for the second time, monitoring the operating parameter value a second time; determining a second probability associated with the first root cause; adjusting a diagnostic cost threshold based on the first probability and the second probability; and determining whether to inject an additional diagnostic action based on the adjusted diagnostic cost threshold.
[0214] In Example 42, the subject matter of Example 41 can optionally include operations further comprising: determining a difference between the first probability and the second probability, wherein determining whether to inject an additional diagnostic action is based on the difference.
Claims
1. A network management system, comprising a processing circuit, wherein the processing circuit is configured to: using a machine learning model provided with values of operating parameters of one or more network devices of a network system to obtain an indication of a cause of a fault in the operation of the network system and a confidence level associated with the cause of the fault; Based on the confidence associated with the cause of the fault not satisfying a fault confidence threshold, determining a diagnostic action cost threshold by: setting the diagnostic action cost threshold to a first value based on a confidence level associated with the cause of the fault being within a first range; and setting the diagnostic action cost threshold to a second value based on being within a second range associated with the cause of the fault; selecting a diagnostic action from a plurality of diagnostic actions; and Based on the selected diagnostic action not satisfying the selected diagnostic action cost threshold, a corrective action from a plurality of corrective actions is performed to correct the cause of the fault.
2. A network management system, comprising a processing circuit, wherein the processing circuit is configured to: Based on a plurality of candidate causes of a fault in the operation of the network system and a corresponding plurality of confidences among the plurality of candidate causes of the fault, determining, from among the plurality of candidate causes of the fault, a candidate cause of the fault having a highest confidence among the corresponding plurality of confidences; identifying a diagnostic action for diagnosing the fault; performing the diagnostic action to diagnose the fault to obtain a root cause of the fault based on a determination that the identified cost of the diagnostic action satisfies a diagnostic threshold, the diagnostic threshold being configured at least in part based on the determined candidate cause; as well as Based on a determination that the cost of the identified diagnostic action does not satisfy the diagnostic threshold, a corrective action is performed to correct the determined candidate cause, the corrective action being selected from a plurality of corrective actions and having a highest confidence in correcting the determined candidate cause.
