Determining cause of dependency of computer system event

Abstract

Embodiments of the present disclosure relate to determining a cause of dependency of a computer system event. Methods and systems are disclosed for determining a combination of system parameters indicating a root cause of system level experience degradation (SLED). Some disclosed embodiments generate a decision tree from a first type of operating parameter data set. The rules are derived from the decision tree. A set of filtering rules for feature parameters included in the system parameters is then determined. In at least some embodiments, pairs of features in a particular data set, each satisfying its respective filtered set of rules, indicate a root cause of reduction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with application number 202011052168.9, application date September 29, 2020, and invention name “Determining Dependent Causes of Computer System Events”; this application claims priority to U.S. Provisional Application No. 62 / 907,896 filed on September 30, 2019, invention name “Automatically Discovering a Combination of Base-Causes of anEvent”. The contents of the prior application are considered part of this application, and the entire contents are incorporated herein by reference. Technical Field

[0003] The present application relates to monitoring wireless communication systems and, more particularly, to methods and / or apparatus for determining the cause of a network event when the cause is a result of multiple operating parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The embodiments of the present invention may be better understood by referring to the following description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements, wherein:

[0005] Figure 1A is a diagram of data flow implemented in one or more disclosed embodiments.

[0006] Figure 1B A graph is shown that relates target parameter values ​​to feature parameter values.

[0007] Figure 1C A graph is shown that relates target parameter values ​​to feature parameter values.

[0008] Figure 1D A graph is shown that relates target parameter values ​​to feature parameter values.

[0009] Figure 1E A graph is shown that relates target parameter values ​​to feature parameter values.

[0010] Figure 2 is a block diagram illustrating an embodiment of a network environment.

[0011] Figure 3A is a block diagram illustrating an embodiment of a wireless access point.

[0012] Figure 3B is a block diagram illustrating an embodiment of a network management system that determines which system level experience (SLE) degradation will require manual intervention.

[0013] Figure 4 is a block diagram illustrating an embodiment of a network node server.

[0014] Figure 5A is a block diagram illustrating an embodiment of a communication device such as a UE.

[0015] Figure 5B Example multiple data sets are shown.

[0016] Figure 5C An example decision tree is shown.

[0017] Figure 6 is a flow chart of an example process for classifying SLE parameters.

[0018] Figure 7 is a flow chart of an example process for classifying SLE parameters.

[0019] Figure 8 is a flow chart of an example process for classifying a data set.

[0020] Fig. 9 is a flow chart of an example process for generating a filtered sub-rule set.

[0021] Fig.10 is a flow chart of an example process for identifying a rule having a greatest number of matching filter sub-rules.

[0022] Fig.11 is a flowchart of an example process for selecting one or more filtered sub-rules satisfied by an outlier value

[0023] Fig.12 is a flow chart of an example process for marking a feature parameter value as "high" or "low".

[0024] Fig.13 is a flow chart of an example process for determining a plurality of operating parameters that cause a service level experience degradation (SLED).

[0025] Fig.14 is a flow chart of an example process for invoking corrective action when a combination of underlying causes is identified.

[0026] Fig.15 is a flowchart of a process implemented in one or more disclosed embodiments.

[0027] Fig.16 is a flowchart of a process implemented in one or more disclosed embodiments.

[0028] Fig.17 is a flowchart of a process implemented in one or more disclosed embodiments.

[0029] Fig.18 is a flowchart of a process implemented in one or more disclosed embodiments. DETAILED DESCRIPTION

[0030] Users of wireless networks, such as Wi-Fi networks, may experience degradation of the system level experience (SLE) of the network. This degradation can be caused by a variety of problems. When a degradation is identified, it is important to quickly identify the root cause. Automatic or manual corrective actions can then be initiated to restore system performance. Some degradations are the result of a combination of problems that manifest themselves. In some cases, the combination of problems spans multiple system components. For example, in some embodiments, a particular degradation will not occur unless at least two system components enter two specific states. Identifying the root cause of such problems is more complex because it requires determining the combination of root causes that led to the degradation, which is inherently more complex than determining a single root cause.

[0031] Some actions taken to address system degradation may have adverse effects on the user community. Therefore, these actions should not be invoked unless the root cause can be determined with a high level of confidence. Obtaining this confidence level may be more difficult in the specific case discussed above, where the cause of the system degradation is related to two different system components or at least two different issues within a system component.

[0032] At least some of the disclosed example embodiments monitor operating parameters of a computer or network system. Operating parameters include independent parameters and dependent parameters. Independent or feature parameters include parameters that can be configured or selected by an operator of the system. These independent parameters include, for example, the amount of time since the system was restarted, the amount of memory configured, the value of a configuration parameter, the software and / or firmware version, and other parameters. Dependent parameters reflect the results of system operation and at least partially reflect the value of independent parameters. Dependent parameters include parameters such as CPU utilization, latency, throughput, memory utilization, or other parameters. In some embodiments, tools such as Ookla's SpeedTest can be used to evaluate the performance of the monitored computer or network system. These tools inject packets into the network and measure dependent or target parameters such as download speed (we call this parameter "download"), upload speed (we call this parameter "upload"), round-trip time (we call this parameter Ping), and jitter (we call this parameter jitter). We call these dependent parameters target parameters. Other example target parameters include connection time, throughput, roaming, successful connection, coverage, capacity, jitter, round-trip time, upload / download speed, or AP uptime. When a particular target parameter value is determined to represent a service level experience degradation (SLED), the disclosed embodiments determine a probability that the particular one or more characteristic parameter values ​​cause the SLED.

[0033] Some embodiments monitor independent and / or dependent parameters related to access points (APs), Wi-Fi devices (e.g., mobile clients, user equipment (UEs), wireless terminals (WTs), etc.), and / or WAN indicators. We refer to independent parameters as features.

[0034] Examples of access point characteristics include, but are not limited to:

[0035] a) Average number of different clients attached to the AP. The frequency of collection of this number of clients may vary depending on the embodiment, but for example in some embodiments it is collected every two seconds and averaged over a one-minute period.

[0036] b) Average utilization (e.g., CPU, memory, I / O bandwidth, etc.) Depending on the embodiment, these operating parameters may be monitored at varying frequencies, but in some embodiments, these operating parameters are monitored at two second intervals and a moving average is generated over a one minute period.

[0037] c) Average non-Wi-Fi utilization of devices such as TV remote monitors, where

[0038] The utilization is measured every, for example, 2 seconds and averaged over, for example, 1 minute.

[0039] d) Average utilization in a particular SSID, where the utilization is measured every, for example, 2 seconds and averaged over, for example, 1 minute.

[0040] Examples of Wi-Fi device characteristics include, but are not limited to:

[0041] a) Received noise measured by hardware,

[0042] b) Received Signal Strength Indicator (RSSI),

[0043] c) the calculated signal-to-noise ratio (SNR),

[0044] d) Specific radio channels or frequency bands (e.g. 2.4G / 5G)

[0045] e) Transmission bandwidth,

[0046] f) the calculated number of packet retransmissions divided by the number of packets requested for transmission (e.g., retransmission rate),

[0047] g) The number of multiple-input / multiple-output (MIMO) streams.

[0048] Examples of WAN characteristics include, but are not limited to:

[0049] a) Network latency, e.g. for transmitting AP statistics to a network management server,

[0050] b) The ratio of current network jitter to long-term average jitter as a factor of maximum WAN capacity, where WAN capacity is defined as the maximum Ethernet port rate measured within a given time window (e.g., a 72-hour window).

[0051] The measurement of the characteristic and target parameters can be performed by any device attached to the Wi-Fi network. The measured parameters will be forwarded to a network-attached server for storage and further analysis, as described below. Table 1 below includes an example description of these stored parameters.

[0052] Some disclosed embodiments periodically collect j characteristic parameters and k target parameters. In various aspects, the frequency of collection of each of these parameters can be set independently, or can be synchronized.

[0053] Table 1 below represents multiple time-synchronized measurements of characteristic parameters and target parameters (also individually referred to herein as data sets). Each row stores data representing a data set. The rows of Table 1 store measurements collected simultaneously with each other. The length of the time window comprising each measurement of the rows of Table 1 can vary according to the embodiment, from a very short and precise window (e.g., 10 seconds) to a less precise window (e.g., 1 minute). In the embodiment shown in Table 1 below, two access point characteristic parameters, three device characteristic parameters, and (j-5) WAN characteristic parameters are collected. Table 1 also shows k target parameters.

[0054] Table 1 indicates that at a time associated with index i and feature parameter j, an embodiment measures and stores feature parameter F i，j Similarly, at a time associated with index i and target parameter k, an embodiment measures and stores the target parameter T i，k .

[0055] Table 1

[0056]

[0057] Some embodiments apply predetermined criteria to one or more target parameter values. The application of criteria is used to classify a time-dependent monitored parameter set (e.g., a data set or row in a table) including parameter values ​​into a first category or a second category. In some embodiments, predetermined criteria are defined as describing a data set including a nominal value of the target parameter and other data sets including an abnormal or problematic value of the target parameter. In some embodiments, criteria apply a threshold to the target parameter value. In some embodiments, if the criteria are met, such as if the target parameter value is less than, less than or equal to, greater than, or greater than or equal to one of the predetermined thresholds, the data set including the target parameter value is classified as the first category, and if the criteria are not met, the data set is classified as the second category.

[0058] For example, in some embodiments, the system is configured to have an upload throughput threshold of 2 Mb / sec. In this example, some embodiments consider a data set including an upload speed target parameter below 2 Mb per second as a data set indicating a SLED condition, while a data set including an upload speed target parameter indicating an upload speed above 2 Mb per second is considered to indicate a good SLE. Similarly, if the predetermined threshold for the target Ping response time parameter is five (5) milliseconds, a data set including a round-trip ping response time longer than five (5) milliseconds is considered to indicate a SLED condition, while a data set including a round-trip ping response time shorter than 5 milliseconds is considered to indicate a normal or good SLE.

[0059] Thus, by applying the criteria to the target parameter values, the disclosed embodiments divide multiple time-dependent data sets (e.g., data sets) of monitored operating parameters into two different groups (based on the target parameter value of each data set and the value relationship to the criteria). The two different groups of data sets are sometimes referred to as class one (1) data sets or class zero data sets.

[0060] In some embodiments, a data set representing poor SLE is labeled as data set class zero (0), and a data set representing nominal SLE is labeled as data set class one (1). An equation demonstrating this concept is shown below as Equation 1:

[0061] Dataset classification = threshold (dataset) equation (1)

[0062] in:

[0063] Data set classification is the classification of data sets.

[0064] Threshold() a threshold function or criterion that relates a threshold to a target parameter of the data set, and

[0065] The data set is a set of system parameters related to time, usually including characteristic parameters and target parameters.

[0066] In various embodiments, these classification identifiers (eg, zero or one) may of course be reversed. Table 2 below illustrates this labeling, where the class is determined based on whether a particular target parameter satisfies a criterion (eg, in some embodiments, it associates the target parameter with a threshold).

[0067] Table 2

[0068]

[0069] In some embodiments, multiple classifications of data sets can be performed by applying multiple criteria to the values ​​of the target parameter across multiple data sets. For example, a first criterion can associate the target parameter value with a first predetermined threshold, thereby resulting in a first classification of the data set including the target parameter value. In these embodiments, a second criterion that associates the target parameter value with a second predetermined threshold is also evaluated, thereby resulting in a second classification of the data set.

[0070] In some embodiments, additional classifications are performed by applying additional criteria to one or more other target parameter values. Thus, for a single target parameter value (e.g., a column in Table 1), each data set is classified once or multiple times, and for each of the second, third, fourth, fifth, etc. target parameter values, zero or multiple times.

[0071] For each classification, the data set is then grouped based on its classification. The grouping of parameter values ​​of Table 2 is visually illustrated in Table 3 below.

[0072] Table 3

[0073]

[0074]

[0075] The data sets are grouped for each criterion applied to them (one or more criteria for one or more corresponding target parameters). For example, in some embodiments, if four (4) target parameters are evaluated, each target parameter is evaluated according to five different criteria (each specific to the corresponding target parameter), generating twenty different data set groupings, which are similar in structure to Table 3, as discussed below. Note that although the present disclosure conveys certain features via the attached tables, it is not required that any disclosed embodiment itself generate a tabular data structure. Those skilled in the art will appreciate that a variety of data structure types may be used to organize data to convey the relationships and groupings discussed.

[0076] In some embodiments, the classified data set is used to generate a binary decision tree. Some embodiments utilize a Gini decision tree (GDT). Other embodiments utilize a tree-based classifier that uses an optimization metric other than the Gini score. Some embodiments of a decision tree or classification tree include non-leaf decision nodes labeled with input features. The edge of each non-leaf node is labeled with each possible value of the input feature. The edge leads to a subordinate decision node of different input features. Each leaf node of the tree is labeled with a probability distribution on a class or class. Therefore, the data set applied to the tree is classified by traversing the tree from the root of the tree to the leaf nodes of the tree. The data set is classified, and it is consistent with the leaf nodes of the result obtained by applying the data set to the tree.

[0077] Some embodiments construct a decision tree for each data set grouping. Thus, in the above example, twenty (2) decision trees are constructed from twenty different groups (five groups for each of the four target parameters). When a decision tree is generated based on a grouped data set, the nodes of the decision tree indicate the criteria for a specific feature parameter for segmenting the data at the node. In some embodiments, the node also indicates the determined Gini score (or weight), the number of affected data sets of the node (e.g., the number of rows in Table 1), and an indication of how the data set represented by the node is divided between a first classification (e.g., class zero) and a second classification (e.g., class one). In some embodiments, a single class is also assigned to each node based on whether a majority of the evaluations of the multiple data sets traverse the node when evaluating the multiple data sets for the decision tree.

[0078] In some embodiments, the probability Pi of selecting an item with label i is multiplied by the probability ∑ of incorrectly classifying the item k！＝i Pk=1-Pi is used to calculate the Gini score. When all parameter data sets represented below the node in the tree are classified into a single category, the Gini score reaches its minimum or optimal value (zero). In the case of a decision tree with unlimited depth, the leaf nodes of the tree represent data sets with a single classification type (e.g., zero or one). Some embodiments limit the depth of the decision tree to save resources (e.g., memory and CPU). In these embodiments, one or more leaf nodes of the decision tree may represent parameter data sets classified into more than a single classification.

[0079]

[0080] GS - Gini score

[0081] Pi - the probability of selecting an item with label i

[0082] Some disclosed embodiments traverse a decision tree and obtain a list of all nodes corresponding to one of two classes (e.g., class zero, representing an abnormal parameter value, in some embodiments). In some embodiments, the assignment of the first classification or the second classification is based on whether the majority of data sets represented by the node (e.g., based on the number of data sets that traverse the node when evaluating the decision tree) are classified as class zero (0) or class one (1). In some embodiments, nodes of the decision tree are assigned a node class vector (NCV), such as

[0083] NCV = [n0n1] Equation 3

[0084] in:

[0085] NCV - Node Classification Vector,

[0086] n0 - the number of data sets (e.g. rows in Table 3) that are classified as class zero (0) represented by the node,

[0087] n1 - the number of data sets (eg, rows in Table 3) represented by the node that are classified as class one (1).

[0088] Some commercially available decision tree tools refer to NCV as a "value". Each node can also be labeled with the total number of data sets represented by that particular node (e.g., below the node in the tree). This number is called a "sample".

[0089] Samples = n0 + n1 Equation 4

[0090] in:

[0091] Sample is the total number of data rows represented (or evaluated) by a particular decision tree node,

[0092] n0 is the number of data sets classified by the first classification (e.g., class zero (0)) represented by the decision tree node,

[0093] n1 is the number of data sets classified by the second classification (eg, class one (1)) represented by the decision tree node.

[0094] The NCV or "value" vector can then be used to calculate classification probabilities.

[0095]

[0096] and similarly

[0097]

[0098] When the number of data sets represented by the node having a first classification (e.g., class zero (0)) is greater than the number of data sets having a second classification (e.g., class one (1)), the node is assigned the first classification (e.g., class zero (0)). Similarly, when the number of data sets represented by the node having a first classification (e.g., class zero (0)) is less than the number of data sets having a second classification (e.g., class one (1)), the node is assigned the first classification (e.g., class zero (0)). When there is an equal partition of the data sets between the first classification and the second classification, various embodiments may classify the node into the first classification or the second classification.

[0099] In an embodiment where the depth of the decision tree is not limited, the leaf nodes only include members of a single class (the first classification or the second classification). The Gini scores of these leaf nodes are zero. As described above, when the depth of the tree is limited, the leaf nodes may represent a mixture of data set classifications, resulting in Gini scores of those leaf nodes being greater than zero.

[0100] Expanding the discussion of Equation 1, since the choice of a particular value for the threshold of a particular target parameter determines whether a data set (a row in Table 1) is classified as belonging to data set class zero (0) or data set class one (1), and since the membership of a data set in the first or second classification determines the value of a node in the decision tree, we obtain

[0101] Threshold -> Data set divided into two classifications -> Decision tree Equation 7

[0102] Where:

[0103] Threshold The threshold of a particular target parameter,

[0104] -> Selection operator.

[0105] Some embodiments use criteria for evaluating the values of target parameters against randomly determined thresholds. Each of these thresholds is chosen to fall within a range between the minimum possible value of the target parameter and the maximum possible value of the target parameter.

[0106] Minimum target parameter (k) < Threshold (k, n) < Maximum target parameter (k) Equation 8

[0107] Where:

[0108] Minimum target parameter (k) is the minimum value of the kth target parameter,

[0109] Maximum target parameter (k) is the maximum value of the kth target parameter,

[0110] Threshold (k, n) The nth threshold of the kth target parameter.

[0111] And

[0112] 0 < n < N Equation 9

[0113] Where:

[0114] N The number of random thresholds for each target parameter

[0115] By selecting random thresholds and applying them to each target parameter value, these embodiments generate one or more separate classifications of the data set, such as the example in Table 3. Then, these embodiments generate a decision tree based on each separate classification.

[0116] Identifying Feature Outliers

[0117] Some embodiments select a data set that includes the values of a target parameter for a single class (e.g., class one). An example of this selection is shown in Table 4 below.

[0118] Table 4

[0119]

[0120] In the specific example shown in Table 4, the specific F belonging to class (1) j The measured values ​​of the characteristic parameters are F(1,j), F(2,j), F(i-1,j), and F(i+1,j).

[0121] If the Fj feature parameter is a measured RSSI value, and there are five RSSI values ​​that fall into the feature set of class one (1) measurements, then an example value for the Fj parameter for the class 1 value above may be:

[0122] RSSI Class 1 = { -41, -48, -50, -60, -62} Equation 10

[0123] Then, some embodiments calculate the median of the feature parameter and the low quartile value of the feature parameter. In some embodiments, the low quartile value Q1 is a first boundary value drawn between the first quartile and the second quartile of the values ​​of the feature parameter in multiple data sets. In some embodiments, the high quartile value Q3 is a second boundary value drawn between the third quartile and the fourth quartile. In some embodiments, these low and high quartile values ​​are determined for each feature parameter based on the values ​​of the individual feature parameters in the data set of a single class.

[0124] In the example of equation 10 above, the median is -50. In some embodiments, the first quartile (Q1) boundary is defined as the middle number between the minimum value and the median value. In the example of equation 10, the Q1 boundary value is -48. Similarly, the upper limit of the third quartile (Q3) is defined as the middle number between the maximum value and the median value. In the example of equation 10, the Q3 boundary between the third quartile value and the fourth quartile value is -60.

[0125] In some embodiments, feature parameter values ​​below the Q1 boundary or above the Q3 boundary are classified as outliers. Other embodiments may determine outliers in other ways.

[0126] Some embodiments utilize a relaxation factor R, such as R=0.2, to reduce the percentage of values ​​that fall within the first and fourth quartiles (reducing the number of outliers). In some embodiments, the relaxation factor is used to set the relaxation value of the first quartile boundary using the following equation 11:

[0127] RQ1 = (1-R) ​​* median [minimum, median] Equation 11 where:

[0128]

[0129] An example data set for Equation 10 is RQ1 = -48*0.8 = -.38.4. Similarly, in some embodiments, a relaxation factor is used to set the relaxation value of the third quartile boundary to:

[0130] RQ3 = (1 + R) * median [maximum number, median] Equation 12 Where:

[0131]

[0132] Equations 11 and 12 describe the relaxation operation for feature parameters with positive values, such as delay time. For feature parameters with negative values, such as RSSI, equation 11 determines Q3, and equation 12 determines Q1. For example, using the data set of equation 10 and the expression of equation 12 yields RQ3 = -60*1.2 = -.72. According to this embodiment, features with values ​​less than RQ1 or greater than RQ3 are considered outliers.

[0133] Determine the list of rules based on decision trees

[0134] Returning to the processing of the decision tree described above, some embodiments identify nodes of the decision tree that have been classified as a single class (e.g., each of the identified nodes represents a data set of more single classes than other data sets of different classifications). For each of these identified nodes, a traversal of the decision tree from the identified node to the root node of the tree is performed. All traversed nodes that also have a single class are identified, including the classification rule used by the node.

[0135] Some other embodiments identify nodes during the traversal that have a probability of being labeled as a single class according to Equation 5 above that is greater than a predetermined first threshold. Therefore, nodes that are not labeled as a single class are excluded from the node forest.

[0136] Some other embodiments identify nodes of a single class whose Gini scores satisfy a predetermined criterion during the traversal. Thus, nodes that do not satisfy the predetermined criterion are excluded. For example, in some embodiments, consistent with Equation 2 above, some embodiments identify nodes whose Gini scores are less than a predetermined second threshold value during the traversal. Those skilled in the art will recognize that various embodiments may utilize any combination of conditions (single class, probability of class, Gini score criterion, etc.) that may implement the above three options.

[0137] For each embodiment, a list of nodes is identified; each node has a decision rule associated with it. The list of nodes is described below as an ancestor list. Rules are then derived from the ancestor list. The derived rules include sub-rules from each identified node included in the ancestor list. Equation 13 below provides two examples of how to derive rules from identified nodes of a decision tree:

[0138]

[0139]

[0140] in:

[0141]

[0142] For example, the first line of Equation 13 can take the following form:

[0143] Rule_79 = {('46', RSSI, 'GT', -62)} Equation 14a

[0144] It is illustrated above that rule 79 is based on node 46 and indicates that the RSSI parameter is greater than -62. As shown in equation 13, rule 79 may include many other conditions not shown herein.

[0145] In some embodiments, the node numbers in equation 13 are not particularly relevant. For purposes of illustration, rule 84 may take the following form:

[0146]

[0147] Each line of Equation 14b is considered as a sub-rule.

[0148] As described above, in some embodiments, each target parameter is classified via multiple (e.g., five) thresholds. For example, each of the five thresholds will result in a specific decision tree and a corresponding rule list or ancestor list. For each target parameter, the system combines the rule lists (e.g., five lists) into a single rule list for the specific target parameter.

[0149] In some embodiments, this process is performed only for specific target parameters that exhibit SLE degradation. According to another exemplary embodiment, this process is repeated for multiple target parameters. The rules derived from each traversal of the decision tree are combined into a combined rule list. Therefore, some embodiments create a combined rule list for each target parameter.

[0150] Rule filtering

[0151] Next, for each list in the combined list of rules (corresponding to a specific target parameter), some disclosed embodiments perform rule filtering. For each of the feature parameters, the disclosed embodiments scan the ancestor list of equation 13 and collect the inequalities and conditional values ​​associated with the feature parameter.

[0152] The filtering rule information is in the form of

[0153]

[0154] in:

[0155]

[0156] An example of a filter rule for average RSSI is provided below.

[0157]

[0158]

[0159] in:

[0160]

[0161] After determining the filtering rules for each feature, at least some disclosed embodiments eliminate redundant rules. In some embodiments, if another rule for the same feature parameter is included in the same ancestor list that already includes the constraint, the rule is redundant. For example, assume that the following filtering rule applies to multiple streams.

[0162]

[0163]

[0164] The two rules of equation 17 are derived from the ancestor rule 63, namely:

[0165] ('GT',2.5,'Rule_63') Rule 18a, and

[0166] ('GT',1.5,'Rule_63') Rule 18b.

[0167] When the number of flows is greater than 2.5, the number of flows is also greater than 1.5. Therefore, the above rule 18b is considered redundant and can be removed. Eliminating this rule provides a new filtering rule for the number of flows:

[0168]

[0169] As an example, if the filtered rules include a first sub-rule that tests whether a parameter is greater than 2.5, and a second filtering sub-rule that tests whether the parameter is greater than 1.5, the first rule is redundant because the parameter values ​​that satisfy the second rule also satisfy the first rule. Therefore, the first rule can be deleted from the filtered rule set. Similarly, if the third rule tests whether the second feature parameter is less than 1.5, and the fourth rule tests whether the second feature parameter is less than 2.5, then for similar reasons, the fourth rule is also redundant and can be discarded.

[0170] In an example embodiment with four target parameters and five criteria (e.g., for five different threshold evaluation parameter values), five decision trees are generated for each target parameter. Each of the five decision trees results in a rule list. In an example implementation, the five rule lists are combined into a single combined rule list, which is filtered, resulting in a single filtered rule list for each feature parameter (for each target parameter).

[0171] In some embodiments, a separate rule list is generated from each of the plurality of decision trees. These separate rule lists are then processed independently as described above. This results in the generation of five separate rule histograms. Based on these separate histograms, feature parameters can be identified, as described below.

[0172] Once a list of filter rules for each feature is obtained, the disclosed embodiments use the list of filter rules to examine parameter data sets such as those shown in Table 3, and more specifically examine data sets classified into a particular classification (eg, Class (0)).

[0173] Identify features corresponding to outliers

[0174] Some embodiments evaluate each data set classified as a first classification (e.g., class zero (0)) relative to a filtered set of sub-rules for a feature. The evaluation identifies values ​​for each feature parameter that may qualify as an outlier. If the feature parameter values ​​for the data set satisfy the filtered sub-rules in the set, it is determined whether the feature parameter values ​​are outliers. If the filtered rule is satisfied by parameter values ​​included in the data set that qualify as an outlier, an indication that the sub-rule has been satisfied by the outlier is stored. Some embodiments store these indications in a rule occurrence list. Also stored in the rule occurrence list in some embodiments are identifications of specific feature parameters and identifications of specific data sets that satisfy the filtered sub-rules (e.g., row identifiers in Table 3).

[0175] In some embodiments, the process is repeated for each filter rule list for each feature parameter. Each data set is evaluated for each filtered rule list. Any data set that meets the filter rule is evaluated to determine whether the specific feature parameter value that meets the filter rule is an outlier. If so, an indication of this is stored. Therefore, when this process is completed, the outliers mark certain inversion rule subsets for each feature as being met. The specific data set identifier (e.g., row index) that meets the rule and the specific feature parameter that meets the rule are also recorded.

[0176] Then, some disclosed embodiments determine via outliers which data sets include feature parameter values ​​that satisfy at least one or more of the inversion rules. Various embodiments determine a data set whose feature parameter values ​​satisfy one or more inversion rules (in some embodiments, via outliers). When such a condition is found, some embodiments determine that the feature parameters that satisfy the sub-rule are responsible for the data set with a first classification (e.g., representing abnormal behavior in some embodiments). Some embodiments modify the sub-rules extracted from the decision tree so that they are evaluated to true values ​​only when the feature parameter values ​​that are outliers are evaluated.

[0177] The following is an example of a sub-rule satisfaction list after evaluating the data set of Table 3 classified as the first classification (eg, class zero (0)) against the filtered rule list:

[0178] Rule satisfaction list = {R17, R8, R89, R178 & R782, ..} Equation 20

[0179] in:

[0180] R# is the index number into the ancestor list.

[0181] When a data set satisfies two or more sub-rules (or two or more sub-rules also have corresponding outliers), some embodiments generate additional composite rules. A composite rule includes conditions that satisfy all satisfied sub-rules. For example, equation 20 above shows a list of sub-rules that includes a composite rule, which includes two sub-rules (sub-rule 178 and sub-rule 782), both of which satisfy a specific data set (row) of class 0 in Table 3 above.

[0182] Rule Selection

[0183] Some disclosed embodiments rank the list of rule satisfaction by the frequency of satisfaction of sub-rules and / or compound rules. An example of sub-rules and compound rules ranked by satisfaction frequency is provided in Table 5 below.

[0184] Table 5

[0185] rule# frequency 184 83,456 56 78,340 2,078 38,765 1,367 12,976 745 7,098 922 5,028 27 3,097 374 1,237 --- ---

[0186] Some disclosed embodiments select the top "n" sub-rules and / or compound rules with the highest number of satisfaction ratings. Using the example Table 5 above, these embodiments would select sub-rules 184, 56, 2078, 1367, and 745 when selecting the top five (5) rules based on the number of satisfaction ratings.

[0187] Identify features and feature combinations that contribute to poor performance

[0188] Some embodiments identify the rule with the highest number of sub-rule and / or compound rule matches.If the feature parameter satisfies the sub-rule condition, and in some embodiments, if the feature value is an outlier, these embodiments increase the match count of the sub-rule.

[0189] Some embodiments determine the correlation between the feature parameter and the target parameter. If the matching sub-rule includes an inequality greater than (GT), and the feature parameter is classified as a positively correlated feature, and the value of the feature parameter is greater than a factor of the third quartile, then some embodiments classify the feature value as a high value. If the feature value is less than a factor of the first quartile, then the feature value is classified as a low value.

[0190] Similarly, if the matching sub-rule includes an LE inequality, the feature parameter is considered a positively correlated feature, and the feature parameter value is greater than a factor of the third quartile, then the feature value is classified as high. If the feature value is less than a factor of the first quartile, then the feature value is classified as low. If there are multiple features that are classified as high or low, in some embodiments, these commonly classified features are combined with an AND clause.

[0191] This process is repeated for all data sets in the plurality of data sets (e.g., rows of Table 3). This results in the identification of frequently satisfied inversion sub-rules that reference specific features and / or feature combinations. The higher the number of satisfactions of sub-rules that reference specific features or feature combinations, the higher the likelihood that the feature or feature combination is the cause of performance degradation.

[0192] The system estimates the likelihood that a particular feature (or combination of features) is the root cause of performance degradation by dividing the number of times the feature (or combination of features) appears in a sub-rule by the total number of sub-rules.

[0193] Some embodiments determine that when we observe a failure C of a particular client F The specific characteristic parameter f i The probability of being responsible is estimated to be

[0194]

[0195] in:

[0196]

[0197] In some cases, multiple features (or combinations of features) may be identified as the root cause of the SLED. For example, the probabilities determined according to Equation 22 above for multiple features (or combinations of features) may be within a threshold distance of each other, and therefore, the first feature of the multiple features (or combination of features) or the second feature of the multiple features (or combination of features) cannot be unambiguously determined as the root cause of the SLED. In this case, some embodiments determine the difference between each feature value and its corresponding interquartile boundary. Some embodiments select the feature with the largest outlier, or in other words, the feature with the largest normalized difference between its feature value and the corresponding interquartile boundary of the feature. Then, the system selects f2 relative to f1 if the normalized absolute difference satisfies the condition:

[0198] Nabs(value of f2–(Q1 or Q3)of f2)>

[0199] Nabs(value of f1–(Q3 or Q1)of f1)Equation 23Where:

[0200] Nabs – Normalized absolute function where the values ​​are normalized to an unequal scale, e.g. between 1-100.

[0201] Thus, some embodiments identify the features that deviate most from the quartile boundaries as the most likely cause of the SLED.Some embodiments identify multiple contributing features.

[0202] Remedial Action

[0203] Some embodiments generate one or more indications of monitored system performance. These indications may be collectively referred to as service level metrics (SLMs), which indicate a service level experience (SLE). The SLMs indicate the extent to which a target parameter of the monitored system matches a value indicating nominal performance. For example, an SLM may provide an indicator of download speed as a percentage of the nominal download speed, e.g., 2 Mb / s. In addition, for each target characteristic SLM, some embodiments present a list of probabilities that any characteristic parameter (or combination of characteristic parameters) measured by a particular target parameter SLM contributes to performance degradation.

[0204] Each feature is mapped to a high-level networking concept that can be acted upon. The mapping is performed by a technician with domain knowledge and is stored in a lookup table in the system. For example, "low RSSI" is mapped to "signal_strength," which is in turn mapped to a concept called "coverage." When some embodiments detect low coverage, they issue an automatic command to the radio resource manager to increase the transmit power of the associated radio transmitter. Similarly, in some embodiments, "Chanutil_inf" is mapped to "Wi-Fi interference," which is in turn mapped to a concept called "capacity." When these embodiments detect Wi-Fi interference, in some embodiments, the radio resource manager issues a command to the adjacent interfering radio transmitter to reduce the power of its transmissions.

[0205] Some embodiments display this information (e.g., to an IT technician). The displayed information includes data for one or more of the following: an identifier for each device, each AP, and each station. This displayed information facilitates understanding of the status of the Wi-Fi network.

[0206] The following are examples of remedial actions implemented in one or more disclosed embodiments. In a first example, the RSSI / signal strength value is determined to be the cause of the SLED. In response to determining that the RSSI caused the SLED, some embodiments obtain the current value of the RSSI of the device and determine the path loss (e.g., Equation 24 below) to calculate the estimated distance of the device to the nearest AP.

[0207] D = exp((RSSI – TP) / Alpha) Equation 24

[0208] in:

[0209]

[0210] Some embodiments may repeat this calculation using the same RSSI value for the first quartile (Q1) for the RSSI range. Let this distance be D2. If D1>D2, some embodiments display instructions (e.g., to the user of the affected device) to move D2-D1 meters toward the nearest AP in order to improve the "coverage metric" or "improve RSSI" or "improve signal strength." This remedial action may guide the end user to improve their SLE by moving in a specific direction.

[0211] In a second example, an embodiment determines that the client density of an access point is causing SLEDs for wireless terminals (or other devices) associated with the access point. In response to the determination, the embodiment takes one or more of the following actions:

[0212] • Determining the average number of clients associated with the access point associated with the wireless terminal.

[0213] Determine the location of the AP.

[0214] Select other APs near the AP.

[0215] • Identify APs from among the identified APs that are not associated with the client that is experiencing SLED.

[0216] • Selecting a subset of the identified APs that have an average number of associated clients that is below the Quartile 4 (Q3) value of the average number of associated clients.

[0217] Calculate the distance between the wireless terminal subject to SLED and each identified AP,

[0218] A second AP closest to the wireless terminal is identified based on the distance.

[0219] • Instruct the client's user to move closer to the second AP.

[0220] This second example remedial action may result in guiding mobile terminal end users to improve their SLE by instructing them to move in a particular direction. This movement will result in the mobile terminal disassociating from the overloaded AP and forming a new association with a more lightly loaded AP.

[0221] In a third example, an embodiment determines that Wi-Fi interference is causing a SLED of a particular wireless terminal or other device. In response to this determination, the embodiment performs one or more of the following operations:

[0222] Calculate the channel interference measurement of the wireless device and the quartile 3 (Q3) of the channel interference measurement

[0223] The difference between the thresholds.

[0224] If the difference is greater than a predetermined threshold, then cause the transmit power of the interfering radio to be reduced.

[0225] Alternatively, if this difference is greater than a predetermined threshold, have the interfering radio switch to another channel. Clients associated with the AP that includes the interfering radio will automatically follow and switch their channels accordingly.

[0226] An embodiment determines that the SLED is caused by a combination of two characteristic parameter values. To address this issue, some embodiments compare the number of users that would be affected if remedial measures were taken to improve the first characteristic parameter value with the number of users that would be affected if remedial measures were taken to improve the second characteristic parameter value. These embodiments then select the remedial measures that affect the smaller number of users. Another example embodiment takes into account other factors, such as the priority of various users or data flows, or the severity of the impact of the corrective measures on the users. Some embodiments maintain a mapping between the root causes of the SLED and the remedial actions to be performed when those root causes are detected.

[0227] Some embodiments maintain separate sets of SLMs for multiple devices participating in a network. For example, in at least some of these embodiments, a set of service level metrics is maintained for each wireless terminal communicating with an access point managed by the disclosed embodiments. Some embodiments maintain SLMs for one or more servers attached to the network, such as an AA server, a DHCP server, a DNS server, or other servers. By maintaining separate SLMs for various devices communicating on the network, the disclosed embodiments are provided with data that can be used to identify a SLED related to the performance of any of those devices.

[0228] Other remedial actions performed in one or more disclosed embodiments include: restarting a device that encountered a SLED, restarting a DHCP server, restarting an AAA server, restarting a specific router, adding an SSID to an AP, restarting a specific radio on a specific AP, restarting an AP, or downloading a new version of software or firmware to a network component.

[0229] FIG. 1 is a diagram of a data flow implemented in one or more disclosed embodiments. FIG. 1 shows a data set 11502. The data set includes multiple time-related sets or groups of parameter values. The parameter values ​​include values ​​of feature parameters 11532a and target parameters 11532b. In some embodiments, the data set 11502 is organized into rows, where each row represents a single set of time-related parameter values, and the data set 11502 represents multiple time-related parameter values, each set or group or row of parameter values ​​being measured or captured at a different time period.

[0230] 1 shows that data sets 11502 have each been classified into a first classification 11504 or a second classification 11506. Data flow 11500 shows the generation of a decision tree 11508 based on data set 11502. Data flow 11500 also shows the generation of a rule set 11510 from decision tree 11508. Rule set 11510 is then inverted to generate a filtered rule set 11512.

[0231] The filtered rule set 11512 is then evaluated for each data set 11502, wherein the evaluation results are stored in the evaluation data structure 11514. When a specific data set (e.g., a row) in the data set 11502 satisfies two or more sub-rules, a composite rule will be generated. Then, the row for the generated composite rule is included in the evaluation data structure 11514. The evaluation data structure 11514 indicates which data set 11502 satisfies which filtered sub-rule (represented by the cells of the evaluation data structure 11514). For example, the first row of the evaluation data structure 11514 corresponds to the evaluation results of the data set in the first row of the data set 11502. Each column of the evaluation data structure 11514 corresponds to a corresponding filtered sub-rule of a specific feature parameter or filtered rule set 11512. Based on the results of the evaluation data structure 11514, the counts of the satisfied filtered sub-rules and composite rules are obtained, wherein the results are stored in the rule histogram 11516. Some embodiments select one or more highest ranked filtering sub-rules (or composite sub-rules) from the rule histogram 11516 to identify features 11518. The identified features (or feature combinations) are those features referenced by the highest ranked filtering sub-rules. Based on the identified features, one or more remedial actions 11520 are identified. Then, at least in some embodiments, these one or more remedial actions 11520 are performed. In some embodiments, the execution of one or more remedial actions 11520 is based on an assessment of the impact of the remedial action. Some embodiments compare the impact of each of a plurality of remedial actions, and at least in some embodiments, select a remedial action to perform the comparison. For example, in some embodiments, the action with the smallest impact is selected to be performed.

[0232] Figure 1B A graph 174 is shown relating target parameter values ​​and feature parameter values. Figure 1B A y-axis 174A and an x-axis 174B are shown. The y-axis 174A relates to the value of the target parameter. The target parameter value depends in part on the feature parameter value shown on the x-axis 174B. Based on a threshold value 175C, the value of the target parameter is classified into a first category 175A or a second category 175B. Figure 1B In the illustrated embodiment, the target parameter values ​​for the first category 175A are considered to represent poor SLE, while the target parameter values ​​for the second category 175B are considered to represent nominal or acceptable SLE. Correlation line 176 shows that the target parameter is positively correlated with the characteristic parameter. As shown, as the characteristic parameter value increases to the right of the graph 174, the target parameter tends to move toward the second category 175B, which, as previously described, represents nominal or good SLE.

[0233] Figure 1BAlso shown are two filtering sub-rules, a first filtering sub-rule 177A and a second filtering sub-rule 177B. The filtering sub-rule 177A compares the feature parameter F1 with the first constant K1 and tests whether the feature parameter F1 is less than the constant K1. The value of K1 is shown on the x-axis 174B. The second filtering sub-rule 177B tests whether the feature parameter value F1 is greater than a second constant K2. Some embodiments delete filtering sub-rules that are positioned in a similar manner to the second filtering sub-rule 177B. In some embodiments, the second filtering sub-rule 177B is displayed with a bold border to indicate that it is deleted. As described above, deletion occurs before the filtered sub-rules are evaluated against the data set. This avoids including these sub-rules in the same data set as the second filtering sub-rule 177B. Figure 1B In the list / summary of consistent satisfied filtered sub-rules, the second filtering sub-rule 177B tests for a condition that the feature parameter F1 tends to be associated with good SLE (e.g., the second classification 175B gives a positive correlation, shown by the correlation line 176). Because some embodiments attempt to not consider these rules when determining the root cause of the bad SLE condition, these rules are removed before evaluating the filtered sub-rules against the data set.

[0234] Figure 1C A graph 178 is shown relating target parameter values ​​and feature parameter values. Figure 1C and Figure 1B The difference is that the target parameter value associated with a relatively good SLE is numerically lower than the target parameter value associated with a poor or sub-nominal SLE (eg, the second category value of the target parameter is lower than the first category value).

[0235] Figure 1C A y-axis 178A and an x-axis 178B are shown. The y-axis 178A relates to the value of the target parameter. The target parameter value depends in part on the feature parameter value shown on the x-axis 178B. Based on a threshold value 178C, the value of the target parameter is classified into a first category 179A or a second category 179B. Figure 1C In the illustrated embodiment, the target parameter values ​​of the first category 179A are considered to represent poor SLE, while the target parameter values ​​of the second category 179B are considered to represent nominal or acceptable SLE. Correlation line 179 shows that the target parameter is positively correlated with the characteristic parameter. As shown, as the characteristic parameter F2 value increases to the right side of the graph 178, the target parameter tends to move toward the first category 179A of the target parameter value, which, as previously described, represents poor SLE. Figure 1CAlso shown are two filtering sub-rules, a third filtering sub-rule 180A and a fourth filtering sub-rule 180B. The third filtering sub-rule 180A compares the feature parameter F2 with a third constant K3 and tests whether the feature parameter F2 is less than the constant K3. The value of K3 is displayed on the x-axis 178B. The fourth filtering sub-rule 180B tests whether the feature parameter value F2 is greater than a fourth constant K4. Some embodiments delete filtering sub-rules that are positioned in a similar manner to the third filtering sub-rule 180A. In some embodiments, the third filtering sub-rule 180A is displayed with a bold border to indicate that it is deleted. As described above, deletion occurs before the filtered sub-rules are evaluated against the data set. This avoids including these sub-rules in the same filter as the data set. Figure 1C In the consistent list / summary of satisfied filtered sub-rules, the third filtering sub-rule 180A tests the condition of the characteristic parameter F2 associated with good SLE (e.g., the second classification 179B is given a positive correlation, shown by the correlation line 179). Because some embodiments attempt to not consider these rules when determining the root cause of the bad SLE condition, these rules are removed before evaluating the filtered sub-rules against the data set.

[0236] Figure 1D A graph 181 is shown relating target parameter values ​​and feature parameter values. Figure 1D A y-axis 182A and an x-axis 182B are shown. The y-axis 182A relates to the value of the target parameter. The target parameter value depends in part on the feature parameter value shown on the x-axis 182B. Based on a threshold value 182C, the value of the target parameter is classified into a first category 183A or a second category 183B. Figure 1D In the illustrated embodiment of FIG. 180 , the target parameter values ​​for the first category 183A are considered to represent a poor SLE, while the target parameter values ​​for the second category 183B are considered to represent a nominal or acceptable SLE. Correlation line 186 shows that the target parameter is negatively correlated with the characteristic parameter. As shown, as the characteristic parameter value increases on the right side of the graph 180 , the target parameter tends to decrease and enter the first category 183A, which, as previously described, represents a poor or sub-nominal SLE.

[0237] exist Figure 1DAlso shown are two filtering sub-rules, a fifth filtering sub-rule 188A and a sixth filtering sub-rule 188B. The fifth filtering sub-rule 188A compares the feature parameter F3 with the constant K5 and tests whether the feature parameter F3 is less than the constant K5. The value of K5 is displayed on the x-axis 182B. The sixth filtering sub-rule 188B tests whether the feature parameter value F3 is greater than the constant K6. Some embodiments delete filtering sub-rules positioned in a similar manner to the fifth filtering sub-rule 188A. In some embodiments, the fifth filtering sub-rule 188A is displayed with a bold border to indicate that it is deleted. As described above, deletion occurs before the evaluation of the filtering sub-rules for the data set. This avoids including these sub-rules in the evaluation of the filtering sub-rules with respect to the data set. Figure 1D In the list / summary of consistent satisfactory filtered sub-rules, the fifth filtered sub-rule 188A tests the condition of feature parameter F3 that tends to be associated with good SLE (e.g., the second classification 186 is given a negative correlation, as shown by correlation line 186). Since some embodiments attempt to not consider such rules when determining the root cause of a poor SLE condition, these rules are removed prior to evaluation of the filtered sub-rules against the data set.

[0238] Figure 1E A graph 190 is shown relating target parameter values ​​to feature parameter values. Figure 1E and Figure 1D The difference is that the target parameter value associated with a relatively good SLE is lower than the target parameter value associated with a poor or sub-nominal SLE (eg, the second category value of the target parameter is lower than the first category value).

[0239] Figure 1E A y-axis 192A and an x-axis 192B are shown. The y-axis 192A relates to the value of the target parameter. The target parameter value depends in part on the feature parameter value shown on the x-axis 192B. Based on a threshold value 192C, the value of the target parameter is classified into a first category 193A or a second category 193B. Figure 1E In the illustrated embodiment, the target parameter values ​​for the first category 193A are considered to represent a poor SLE, while the target parameter values ​​for the second category 193B are considered to represent a nominal or acceptable SLE. Correlation line 196 shows that the target parameter is negatively correlated with the feature parameter. As shown, as the feature parameter F4 value increases to the right side of the graph 190, the target parameter tends to become smaller and moves toward the second category 193B, which, as previously described, represents a nominal or good SLE. Figure 1EAlso shown are two filtering sub-rules, a seventh filtering sub-rule 198A and an eighth filtering sub-rule 198B. ​​The seventh filtering sub-rule 198A compares the characteristic parameter F4 with the third constant K7 and tests whether the characteristic parameter F4 is less than the constant K7. The value of K7 is displayed on the x-axis 192B. The eighth filtering sub-rule 198B tests whether the characteristic parameter value F4 is greater than the fourth constant K8. Some embodiments delete filtering sub-rules positioned in a similar manner to the eighth filtering sub-rule 198B. ​​In some embodiments, the eighth filtering sub-rule 198B is displayed with a bold border to indicate that it is deleted. As described above, the deletion occurs before the evaluation of the filtering sub-rules for the data set. This deletion avoids including these sub-rules in the list / summary of the filtering sub-rules that meet the conditions. Figure 1E Consistently, the eighth filter sub-rule 198B test for conditions that tend to be associated with the characteristic parameter F4 of good SLE (e.g., the second class 193B) is given a negative correlation, shown by the correlation line 196. Because some embodiments attempt to not consider such rules when determining the root cause of the bad SLE condition, such rules are removed prior to the evaluation of the filtered sub-rules against the data set. Figure 1B -E shows some embodiments storing with each target parameter information whether the target parameter is positively correlated with a good SLE (e.g. Figure 1B -C) or negative correlation (e.g. Figure 1D -E) indicator.

[0240] Figure 21 is a diagram of an example system 100 implemented in accordance with an example embodiment. The example system 100 includes a plurality of access points (AP1 142, ..., AP X 144, AP 1' 150, ..., AP X' 152, AP 1" 160, ..., AP X" 162, AP 1"' 170, ..., AP X" 172), a plurality of authentication, authorization, and accounting (AAA) servers (only one AA server 110 is shown), a plurality of dynamic host configuration protocol (DHCP) servers (only one DHCP server 116 is shown), a plurality of domain name system (DNS) servers (only one DNS server 122 is shown), a plurality of web servers (only one web server 128 is shown), a plurality of resource managers (only one resource manager 132 is shown), a plurality of network test servers (only one network test server 124 is shown), and a network management system (NMS) 136, such as an access point management system, coupled together via a network 134, such as the Internet and / or an intranet. 14. Network communication links 143, 145, 151, 153, 161, 163, 171, and 173 couple access points (AP 1142, AP X 144, AP 1' 150, AP X' 152, AP 1" 160, AP X" 162, AP 1"" 170, AP X"" 172), respectively, to network 134. Network communication link 111 couples AA servers (only AA server 110 is shown) to network 134. Network communication link 117 couples DHCP servers (only one DHCP server 116 is shown) to network 134. Network communication link 123 couples DNS servers (only one DNS server 122 is shown) to network 134. Network communication link 125 couples network test servers (only one network test server 124 is shown) to network 134. Network communication link 129 couples Web servers (only one Web server 128 is shown) to network 134. A network communication link 133 couples a resource manager (only a single resource manager, resource manager 132, is shown) to a network 134. The example system 100 also includes a plurality of client or user equipment devices (UE 1 138, ..., UE Z 140, UE 1' 146, ..., UE Z' 148, UE 1" 154, ..., UE Z" 156, UE 1"' 164, UE Z"' 166). At least some of the UEs (138, 140, 146, 148, 154, 156, 164, 166) are wireless devices that can move throughout the example system 100. A network communication link 137 couples the NMS 136 to the network 134.

[0241] In the example system 100, a collection of access points are located at different customer premises sites. Customer premises site 1 102, such as a shopping mall, includes access points (AP 1 142, ..., AP X 144). Customer premises site 2 104, such as a sports stadium, includes access points (AP 1' 150, ..., AP X' 152). Customer premises site 3 106, such as an office, includes access points (AP 1" 160, ..., AP X" 162). Customer premises site N 108 includes access points (AP 1"' 170, ..., AP X"' 172). As shown in FIG. 1, UEs (UE 1 138, ..., UE Z 140) are currently located at customer premises site 1 102. UEs (UE 1' 146, ..., UE Z' 148) are currently located at customer site 2 104; UEs (UE 1" 154, ..., UE Z" 156) are currently located at customer premises site 3 106; and UEs (UE 1"' 164, ..., UE Z"' 166) are currently located at customer premises site N 108.

[0242] UEs such as UE 1 138, ..., UE Z 140, UE 1'146, ..., UE Z'148, UE 1"154, ..., UE Z"156, UE 1"'164, ...UE Z"'166 integrate modules that can interact with a network test server. When triggered to collect network performance, at least in some embodiments, the client module sends a test packet to the network test server. In some embodiments, the network test server returns the packet directly to the sending client. Alternatively, according to another embodiment, when triggered, the network test server sends the test packet to the client. The client can then return the packet to the originating network test server. In either case, the receiving module and the associated AP measure parameters associated with network performance, such as target parameters, Wi-Fi device characteristics, AP characteristics, and WAN functionality, and send them to the network management server.

[0243] In various embodiments, testing of the network can be invoked by a client end user (from the client), an IT technician (from the network test server), or automatically in response to a SLED detected by the system or as part of a recurring process (e.g., once an hour).

[0244] The NMS 136 receives SLE statistics, such as characteristic parameters and / or target parameters, from the network test server and stores them in a memory, as shown in the example of Table 1, related to the performance experienced by all the clients or UEs described above. In addition, in some embodiments, the network management system receives AP and WAN characteristic and / or target parameters from other network components of the example system 100. In some embodiments, the NMS 136 then classifies the data into at least two different classes, such as class zero (0) and class one (1), using predefined thresholds, establishes an associated decision tree, and identifies the rules via equation 13. Then, the inverse sub-rule for each characteristic parameter is determined (via equations 15 and / or 16 in some embodiments). These sub-rules are then used to determine the potential root cause or root cause combination that caused / resulted in the SLED. Once the root cause (or combination of root causes) of the SLED is understood, corrective action is taken. In some embodiments, information indicating the root cause of the SLED is presented to the system administrator via a display. Some disclosed embodiments accept inputs commanding corrective action to restore appropriate SLE.

[0245] Figure 3A Access points 200 (eg, access points AP 1142, ..., APX 144, AP 1'150, ..., APX'152, AP 1"1 160, ..., APX"162, AP 1'"170, ..., APX'"172) are shown in accordance with one or more of the disclosed embodiments.

[0246] The access point 200 includes a wired interface 230, a wireless interface 236, a wireless interface 242, a processor 206 such as a CPU, a memory 212, and components of a module 208, such as components of a hardware module, such as circuit components coupled together via a bus 209, on which the various components can exchange data and information. The wired interface 230 includes a receiver 232 and a transmitter 234. The wired interface 230 couples the access point 200 to the network 134 of FIG. 1 (e.g., the Internet). The wireless interface 236 supports a Wi-Fi interface, such as an 802.11 interface, including: a receiver 238 coupled to a receiving antenna 239, via which the access point can receive wireless signals from a communication device such as a wireless terminal; and a transmitter 240 coupled to a transmitting antenna 241, via which the access point can transmit wireless signals to a communication device such as a wireless terminal. The wireless interface 242 can support a Bluetooth interface, which includes: a receiver 244 coupled to a receiving antenna 245, via which the access point can receive wireless signals from a communication device such as a wireless terminal; and a transmitter 246 coupled to a transmitting antenna 247, via which the access point can transmit wireless signals to a communication device such as a wireless terminal.

[0247] Memory 212 includes routines 214 and data / information 216. Routine 214 includes components of module 218, such as components of software modules, and application programming interface (API) 220. Data / information 216 includes a collection of configuration information 222, message event flow capture 224, and remedial actions to be taken in the event of discovery of abnormal message flow 226. In some embodiments, the components of module 218 include software capable of collecting parameters related to the SLE and forwarding them to a network test server or directly to a network management system.

[0248] Figure 3B An exemplary network management system 300 according to one or more of the disclosed embodiments is shown, such as a wireless system monitoring server, an access point management node. In some embodiments, the network management system 300 of FIG. 3 is equivalent to Figure 2 The NMS 136 of the present invention. The network management system 300 includes a communication interface 330 such as an Ethernet interface, a processor 306, an output device 308 such as a display, a printer, etc., an input device 310 such as a keyboard, a keypad, a touch screen, a mouse, etc., a memory 312, and a module component 340 such as a hardware module component such as a circuit component, which are coupled together via a bus 309, and the various components can exchange data and information via the bus 309. The communication interface 330 couples the network management system 300 to a network and / or the Internet. The communication interface 330 includes: a receiver 332, via which the network monitoring system can receive data and information from the access point, such as service-related information including, for example, message streams, connection time information, and information related to success / failure, such as for radio connection establishment, AA service, DHCP request, simple notification service (SNS) search, and web page request; and a transmitter 334, via which the network management system 300 can send data and information, such as configuration information and instructions, such as instructions for access point restart, change of transmission power, addition of SSID, etc.

[0249] Memory 312 includes routines 314 and data / information 317. Routine 314 includes components of modules 318, such as components of software modules.

[0250] Memory 312 includes routines 314 and data / information 317. Routine 314 includes components of modules 318, such as components of software modules, and application programming interfaces (APIs) 320. Data / information 317 includes configuration information 322, captured message event streams 324, and a collection of remedial actions to be taken in the event of discovery of abnormal message streams 326.

[0251] Remedial actions may be configured by a system administrator based on past experience. According to some example embodiments, remedial actions may be downloaded to a specific AP. According to other embodiments, remedial actions (not shown) are automatically invoked once an abnormal message flow is detected by a message delivery model that may also reside in memory 312.

[0252] Figure 4 An example node server 400 is shown, such as an AA server, a DHCP server, a DNS server, a Web server, a network test server, etc. In some embodiments, Figure 4 The node server 400 is Figure 2 The server 110, 116, 122, 128. The node server 400 includes a communication interface 402 such as an Ethernet interface, a processor 406, an output device 408 such as a display, a printer, etc., an input device 410 such as a keyboard, a keypad, a touch screen, a mouse, etc., a memory 412 and a module component 416, such as a component of a hardware module, such as a circuit component, which is coupled together via a bus 409, and each component can exchange data and information on the bus 409.

[0253] The communication interface 402 couples the node server 400 to a network and / or the Internet. The communication interface 402 includes a receiver 420, through which the node server can receive data and information, such as information related to operations, such as registration requests, AA services, DHCP requests, simple notification service (SNS) lookups, and web page requests; and a transmitter 422, through which the node server 400 can send data and information, such as configuration information, authentication information, web page data, etc.

[0254] The memory 412 includes routines 428 and data / information 430. The routines 428 include components 432 of modules, such as software modules and components 430 of data / information. The components 432 of modules include modules capable of interacting with a network test server, interacting with network devices, initiating the sending of test packets to the network test server, sending test packets back to the network test server, measuring parameters related to the SLE, and forwarding them to the network test server or directly to a network management system.

[0255] Figure 5A An example client such as UE 500 (eg, user equipment devices UE 1 138, ..., UE Z 140, UE 1' 146, ..., UE Z' 148, UE 1" 154, ..., UE Z" 156, UE 1'" 164, ..., UE Z'" 166) is shown in accordance with one or more of the disclosed embodiments.

[0256] UE 500 includes a wired interface 502, a wireless interface 504, a processor 506 such as a CPU, a display 508, an input device 510, a memory 512, and a component of a module 516, such as a component of a hardware module, such as a circuit component, which are coupled together via a bus 509, and the various components can exchange data and information on the bus 509. The wired interface 502 includes a receiver 520 and a transmitter 522. The wired interface couples UE 500 to Figure 2 network 134 (eg, the Internet).

[0257] The wireless interface 504 includes a cellular interface 524, a first wireless interface 526, such as an 802.11 Wi-Fi interface, and a second wireless interface 528, such as a Bluetooth interface. The cellular interface 524 includes: a receiver 532 coupled to a receiver antenna 533, through which the access point can receive wireless signals from access points, such as AP 1 142, ..., APX 144, AP 1'150, ..., APX'152, AP 1"1 160, ..., APX"162, AP 1'"170, ..., APX'"172; and a transmitter 534 coupled to a transmitting antenna 535, through which the access point can send wireless signals to APs, such as AP 1 142, ..., APX144, AP 1'150, ..., APX'152, AP 1"1 160, ..., APX"162, AP 1'"170, ..., APX'"172. The first wireless interface 526 may support a Wi-Fi interface, such as an 802.11 interface, including: a receiver 536 coupled to a receiving antenna 537, via which the UE can receive wireless signals from a communication device such as an AP; and a transmitter 538 coupled to a transmitting antenna 539, via which the UE can transmit wireless signals to a communication device such as an AP. The second wireless interface 528 may support a Bluetooth interface, including: a receiver 540 coupled to a receiving antenna 541, via which the UE can receive wireless signals from a communication device such as an AP; and a transmitter 542 coupled to a transmitting antenna 543, via which the UE can send wireless signals to a communication device such as an AP.

[0258] The memory 512 includes routines 529 and data / information 517. The routines 529 include components 515 of modules, such as components of software modules. The data / information 517 may include configuration information and any other information required for the normal operation of the UE 500.

[0259] Figure 5B A plurality of data sets 550 are shown. The plurality of data sets show a plurality of feature parameters 552 and target parameters 554. Figure 5BAlso shown is a classification 556 for each data set. The classification 556 is based on a specific value of the target parameter 554.

[0260] Figure 5C An example decision tree 560 is shown. Decision tree 560 is based on the above with respect to Figure 5A The decision tree 560 includes a plurality of nodes, which are labeled 562a-i. Each non-leaf node of the decision tree 560 includes a rule. For example, node 562a includes rule (or decision rule) 564. Most nodes also indicate that the node represents the above Figure 5A Which data set in the multiple data sets 550 discussed. For example, node 562b includes an indicator 566, which indicates that node 562b represents a data set with indexes 4, 6, and 8. Most nodes of decision tree 560 also indicate the score assigned to the node. The score describes the ratio of the data set represented by the node. The ratio is the ratio between the data set classified as the first classification and the data set classified as the second classification. For example, the score 568 indicating 0 / 3 of node 562b indicates that node 562b represents a data set that is not classified as the first class and three data sets classified as the second class (defined by classification 556).

[0261] Figure 6 is a flow chart of an example process for classifying SLE parameters. Figure 6 The process 600 discussed is performed by a client or user equipment UE, such as UE (UE 1 138, ..., UE Z 140), (UE 1' 146, ..., UE Z' 148), (UE 1" 154, ..., UE Z" 156, and (UE 1"' 164, ..., UE Z"' 166), or by a network test server 124. The device performing the process 600 collects parameters related to SLE, such as the characteristics and / or target parameters described in Table 1. In some embodiments, the following description of Figure 6 One or more functions discussed with process 600 are performed by hardware processing circuitry. For example, in some embodiments, any one or more of processors 206, 306, 406, or 506 are configured by instructions stored in one or more memories (e.g., 212, 312, 412, or 512) to perform one or more functions discussed below.

[0262] After start operation 605, process 600 moves to operation 606 where a trigger is received to start collecting SLE parameters. The trigger may be activated by a user of the client UE, activated by an IT technician, automatically invoked when an automatic monitoring process detects bad SLEs, or invoked periodically.

[0263] Operation 607 injects the test packets into the network. In some embodiments, the test packets are injected by the client and sent to the network test server 124. Alternatively, the test packets are sent by the network test server to the client that is experiencing SLED. In some embodiments, the packets are both sent from the client (UE) to the network test server and sent by the network test server to the client (UE) that is experiencing SLED.

[0264] In some embodiments, the device receiving the test packet is configured to send the test packet back to the original sending device.

[0265] In operation 608, the original sending device receives the looped back packet and calculates network parameters such as jitter, throughput, RSSI, round trip time, etc. from the received test packet. Decision operation 610 determines the value of the target parameter. For example, in some embodiments, the UE or client measures SLE parameters (characteristics and / or target parameters), such as received noise measured by hardware, received RSSI, calculated SNR, specific frequency band (e.g., 2.4G / 5G), transmission bandwidth, etc.

[0266] In some embodiments, one or more of the characteristics and / or target parameters are measured by each client and sent to the network management server via an associated access point (AP). According to one embodiment, the UE compares the RSSI to a predetermined threshold and determines whether the RSSI reading corresponds to a good or faulty SLE. According to another embodiment, the UE sends the raw RSSI information to the NMS 136 and the comparison to the threshold is performed by the network management server. In either case, the network management server maintains information whether the RSSI (coverage) is correct or faulty.

[0267] Decision operation 615 determines whether the SLE measurement satisfies a criterion. The criterion evaluates whether a particular SLE measurement value indicates a good or faulty SLE (e.g., class one (1) or class zero (0)). If the operation determines that the measurement corresponds to a good SLE, process 600 proceeds to operation 620, which marks the event accordingly as corresponding to a good SLE (pass). However, if decision operation 615 determines that the measurement of the target parameter value corresponds to a faulty SLE, the process proceeds to operation 625, which marks the event accordingly as corresponding to a poor SLE (failure). As described above, it is optional for the client to mark each measured target parameter as a failure or a pass; in some embodiments, at least in some embodiments, the UE sends the original measurement parameters to the NMS 136, where marking of the measured target parameters occurs.

[0268] In operation 630, the SLE event measurements are forwarded to NMS 136. Then, a decision operation 635 determines whether additional SLE parameters should be collected. For example, some embodiments collect a limited number of data points and then stop. Alternatively, in some embodiments, process 600 terminates when a command is received (e.g., from a user interface) to stop SLE parameter collection. If the operation determines that SLE parameter data collection should continue, the process returns to operation 607, which continues to measure and evaluate its SLE parameters and forwards these evaluations to the network management server via the AP. However, if there are no additional SLE parameters to be measured, process 600 moves to end operation 640.

[0269] Figure 7 is a flow chart of an example process for classifying SLE parameters. Figure 7 The process 700 discussed is performed by any one or more of the access points AP 1 142, ..., APX 144, AP 1'150, ..., APX'152, AP 1"1 160, ..., APX"162, AP 1'"170, ..., APX'"172. In some embodiments, the device performing the process 700 passes on SLE parameters collected by the associated UE and / or collects SLE-related parameters itself and passes these parameters to a network management system (e.g., NMS 136). In some embodiments, the following description of the process 700 is provided below. Figure 7 One or more functions discussed with process 700 are performed by hardware processing circuitry. For example, in some embodiments, any one or more of processors 206, 306, 406, or 506 are configured by instructions (e.g., 214, 314, 428, 529) stored in one or more memories (e.g., 212, 312, 412, or 512) to perform one or more functions discussed below.

[0270] After start operation 705, process 700 moves to operation 707. In operation 707, the AP obtains a message associated with the SLE from a UE associated with the AP. In operation 730 described below, the SLE information derived from the message received in operation 707 is forwarded to the NMS 136.

[0271] In operation 710, the AP observes and measures SLE parameters (characteristics and / or targets), such as, in particular, one or more of the average number of different clients attached to the AP, average utilization, average non-Wi-Fi utilization, average value, or utilization. Those skilled in the art will recognize that certain parameters may be determined by Figure 6 The UE shown or Figure 7However, some specific parameters, such as the number of failed attempts by a UE to connect via a specific AP, can only be observed and reported by the AP (since the UE / client may not be able to send messages over the wireless network).

[0272] Decision operation 715 determines whether the SLE measurement corresponds to a good or faulty SLE (above or below a corresponding threshold). For example, the AP may compare the average uplink bandwidth of a particular client to a predetermined threshold. If the operation determines that the measurement is above the threshold, the process proceeds to operation 720, where the row is marked as a Class 1 or good SLE, e.g., an uplink bandwidth greater than 300 KB / sec.

[0273] However, if it is determined in decision operation 715 that the measured value corresponds to a faulty SLE, such as the uplink bandwidth is equal to or less than 300KB / sec, the process proceeds to operation 725, which marks the event accordingly as corresponding to a bad (faulty) or class zero (0) SLE.

[0274] In either case, the process continues to operation 730 where the statistics collected from the particular event are forwarded to the NMS 136. As described above, marking each measured target parameter as failed or passed by the AP is optional; in some embodiments, at least in some embodiments, the AP sends the raw measured parameters to the NMS 136 where marking of the measured target parameters occurs.

[0275] Process 700 then returns to operation 707. Note that although Figure 6 and Figure 7 Both describe that the UE and AP classify the SLE parameters into one of two classes, but in some embodiments, the classification is performed by a network management server rather than by the UE and / or AP individually. Therefore, in these embodiments, process 600 omits decision operation 615 and operations 620 and 625. Similarly, in these embodiments, process 700 omits decision operation 715 and operations 720 and 725.

[0276] Figure 8 is a flow chart of an example process for classifying a data set, such as shown above with respect to Tables 2 and 3. In some embodiments, the following Figure 8 The method 800a discussed is performed by a network management system (e.g., NMS 136). Figure 8One or more of the functions discussed and method 800a are performed by hardware processing circuitry. For example, in some embodiments, any one or more of processors 206, 306, 406, or 506 are configured by instructions (e.g., 214, 314, 428, 529) stored in one or more memories (e.g., 212, 312, 412, or 512) to perform one or more functions discussed below.

[0277] After start operation 805, method 800a moves to operation 810, which receives one or more SLE characteristics and target parameters, such as those described above. In some embodiments, these parameters are obtained from, for example, Figure 2 The time-related parameter value groups received by the client and the AP shown form a data set as described above.

[0278] In operation 815, the received features and target parameter values ​​are stored as time-dependent parameter values ​​or data sets. For example, some embodiments represent the received features and target parameter values ​​in a data structure similar to the example of Table 1 above.

[0279] Operation 817 selects a specific target parameter (e.g., Tj of Table 1). In operation 820, a plurality of thresholds (e.g., 5) are determined for the selected target parameter. In some embodiments, the thresholds are randomly selected within a range between the minimum and maximum values ​​of the selected target parameter (e.g., Tj of Table 1).

[0280] Operation 825 selects a threshold value from a plurality of threshold values. Operation 827 selects a data set. As described above, a data set is a plurality of time-dependent parameter values, such as values ​​included in a single row of any of Tables 1, 2, or 3 discussed above.

[0281] Decision operation 830 determines whether the target parameter value included in the selected data set meets the classification criteria. In some embodiments, the criteria evaluate the target parameter value against a selected threshold. Depending on whether the target parameter value meets the criteria, each of operations 835 and 840 classifies the data set into a first category (e.g., operation 835) or a second category (e.g., operation 840).

[0282] After the data set is classified, decision operation 855 determines whether there is an additional data set to classify for the selected threshold for the target parameter (operation 825). If classification of additional data sets is indeed required, method 800a moves to operation 827, which selects a new data set.

[0283] The process then returns to decision operation 830. If all data sets have been classified according to the selected threshold value, the method 800a moves from decision operation 855 to decision operation 860, which determines whether another threshold value in the plurality of threshold values ​​can be selected and used for another classification of the data set. If more threshold values ​​are available, the method 800a moves from decision operation 860 to operation 825, which selects a threshold value in the plurality of threshold values ​​that is different from any threshold value used in the previous iteration of the method 800a. Then, as described above, processing continues from operation 825. If the data set has been classified using all threshold values ​​in the plurality of threshold values, the method 800a moves from decision operation 860 to end operation 870.

[0284] Fig. 9 is a flowchart of an example process for generating a set of filtered sub-rules. Figure 8 The data set discussed is classified according to method 800a to generate a set of filtered sub-rules.

[0285] In some embodiments, the following Fig. 9 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, process 900 is performed by NMS 136.

[0286] After start operation 905, process 900 moves to operation 908. Operation 908 selects a first set of data sets having a single classification. For example, as discussed above with respect to Tables 1-3, a set of data sets is classified according to a single target parameter and a criterion applied to the target parameter. This classification divides the data sets into two groups, each with a different classification. Operation 908 selects one of these groups of data sets, where each data set in the group has the same classification. In some embodiments, operation 908 relies on the above reference Figure 8 The method 800a discussed performs classification of a data set.

[0287] Process 900 continues to operation 920 where a decision tree is constructed from all data sets based on the selected group of data sets.

[0288] Operation 925 calculates the probability that any other node below the node in the decision tree has a single classification for each non-leaf node in the decision tree. Some embodiments calculate the probability using equations 5 and / or 6. In embodiments where the depth of the decision tree is not limited, the probability associated with each leaf node is zero or one, depending on whether the leaf node represents a data set with a first single classification or a second classification.

[0289] Operation 930 determines the Gini score for each non-leaf node of the decision tree (eg, via Equation 2).

[0290] Operations 935, 940, and 945 describe an alternative method of selecting a node list (ancestor list, as described above) from a decision tree.

[0291] Some embodiments perform operation 935, which traverses the decision tree and identifies nodes that are individually classified. As described above, some embodiments classify the subject node of the decision tree as a particular class if the majority of the nodes in the subject node are classified as a particular class. Some embodiments perform operation 940, which identifies nodes of the decision tree that are classified as a single class but also have a probability of having a single class that meets a criterion (e.g., above a predetermined threshold) (e.g., as determined by operation 925) (e.g., a probability of belonging to class zero (0) greater than 66%).

[0292] Some embodiments perform operation 945, which traverses the decision tree and identifies nodes that have a single class and also have a Gini score below a predetermined Gini score threshold. For example, some embodiments identify nodes with a Gini score less than 0.25. Therefore, these embodiments compare the Gini score of each node to a predetermined threshold and classify the nodes based on the corresponding comparison of the nodes.

[0293] Process 900 then moves to operation 950 where rules are identified based on the identified nodes (identified via traversal of the decision tree). Some embodiments of operation 950 are consistent with one or more of Equations 13 or 14 when generating rules.

[0294] Decision operation 952 determines whether there are additional grouped data sets (e.g., grouped based on different thresholds for a given target SLE parameter). If there are additional grouped data sets available, process 900 returns to operation 908, where a different group of data sets with a single classification is selected. The process then continues from operation 908 as described above. Process 900 continues to iterate until all data set groups (each group corresponding to a specific threshold value of the target parameter) have been processed and rules extracted from their corresponding decision trees.

[0295] If decision operation 952 determines that there are no additional data set groups to process, the process continues to operation 955 where rule filtering is performed on the generated rules. Some embodiments perform rule filtering consistent with the explanations provided above, and in some embodiments, include any one or more of equations 15 to 19 above.

[0296] Fig.10 is a flow chart of an example process for identifying a rule with a maximum number of matched filtered sub-rules. In some embodiments, method 1000 operates on a list of filtering rules. In some embodiments, by referring to the above Fig. 9 The process 900 discussed above is used to generate a list of filter rules. In some embodiments, the following Fig.10 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more of the functions discussed below. In some embodiments, method 1000 is performed by NMS 136.

[0297] After start operation 1002, method 1000 moves to operation 1005, which selects a filtered sub-rule from the set of filtered sub-rules. In some embodiments, operation 1005 selects a filtered sub-rule that has not been processed by method 1000. As discussed further below, method 1000 is designed to iterate through the filtered sub-rules and evaluate each filtered sub-rule against a data set to determine which rule has the best matching filtered sub-rule.

[0298] Operation 1007 selects a data set having a first classification. For example, as discussed above, some embodiments store multiple data sets and then classify the data sets according to the criteria. Some embodiments perform multiple classifications on multiple data sets using different criteria (e.g., as described above, in some embodiments, each criterion compares the target parameter to a different predetermined threshold to perform the classification).

[0299] In operation 1010, the selected data set is evaluated for the selected filtered sub-rule. Since the filtered sub-rule typically only includes references to a single feature or target parameter, evaluating the data set for the filtered sub-rule includes evaluating feature values ​​in the data set that match the feature referenced in the filtered sub-rule. The evaluation of operation 1010 ignores parameter values ​​included in the data set that do not match the feature referenced by the selected filtered sub-rule.

[0300] Decision operation 1015 determines whether the currently selected filtering sub-rule evaluates to true for the selected data set. If the filtered sub-rule does not evaluate to true, the method 1000 moves from decision operation 1015 to decision operation 1025, as described below. If the filtered sub-rule does evaluate to a true value, the method 1000 moves from decision operation 1015 to operation 1020. Some embodiments of decision operation 1015 also evaluate whether the value evaluated by the rule is an outlier. In these embodiments, the evaluation of the rule is considered true only if the value is an outlier.

[0301] In operation 1020, a count is incremented indicating the number of times the currently selected filtering sub-rule has been evaluated to "true" for one of the data sets, or the number of times the composite sub-rule has been evaluated to a "true" value when evaluated for one of the data sets. Then, method 1000 moves from operation 1020 to decision operation 1025.

[0302] Decision operation 1025 determines whether to evaluate additional data sets for the filtered sub-rule. If there are data sets that remain to be evaluated for the currently selected filtered sub-rule, the method 1000 moves from decision operation 1025 to operation 1007, which selects a data set that has not yet been evaluated for the currently selected filtered sub-rule. Otherwise, if all data sets have been evaluated for the currently selected sub-rule, the method 1000 moves from decision operation 1025 to decision operation 1030.

[0303] Decision operation 1030 determines whether there are additional filtered sub-rules to evaluate for the data set. If there is another filtered sub-rule to be evaluated, the method 1000 moves from decision operation 1030 to operation 1005, which selects additional sub-rules that have not been evaluated by the method 1000. Otherwise, if all sub-rules have been evaluated, the method 1000 moves from decision operation 1030 to operation 1040, which identifies the rule with the most sub-rule matches. If two or more feature parameters of a particular data set satisfy the filtered sub-rules, some embodiments generate a composite rule that includes both filtered sub-rules and then joins using a logical "and" operator. These embodiments then add the newly generated composite rule and the filtered sub-rules it includes to the records that satisfy the sub-rules and composite sub-rules. As described above, some embodiments generate multiple decision trees, which can be generated by applying multiple criteria to a single target parameter and / or applying criteria to multiple target parameters. In these embodiments, the filtered sub-rule satisfaction count includes the filtered sub-rules derived from each of these decision trees.

[0304] After operation 1040 , method 1000 moves to end operation 1045 .

[0305] Fig.11 is a flowchart of an example process for selecting one or more filtering sub-rules satisfied by an outlier. Fig.11 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, method 1100 is performed by NMS 136.

[0306] After start operation 1105, method 1100 moves to operation 1110, which selects a sub-rule from the identified rule. In some embodiments, the identified rule is identified by method 1000 discussed above. In some embodiments, the identified rule is extracted from a decision tree. In some embodiments, the identified rule is the rule with the largest number of sub-rule matches when compared to a data set having a first classification (e.g., a class 0 data set as discussed above with respect to Tables 1-3).

[0307] Operation 1112 selects a data set from the plurality of data sets. The identified data set is classified according to a first classification (e.g., evaluated according to a criterion of comparing a target parameter value to a predetermined threshold to determine a classification). Operation 1112 is designed to select a different data set from the plurality of data sets for each sub-rule evaluated by the iteration of method 1100.

[0308] Operation 1114 evaluates the selected sub-rule against the selected data set. When each data set defines values ​​for multiple features and target parameters, and the sub-rule references a single parameter, evaluating the selected sub-rule against the selected data set includes evaluating the specific parameter value defined by the data set against the criteria defined for the specific parameter defined by the sub-rule.

[0309] Decision operation 1115 determines whether the sub-rule evaluates to true. If the sub-rule evaluates to true, method 1100 moves from decision operation 1115 to decision operation 1120. If the sub-rule does not evaluate to true, method 1100 moves from decision operation 1125 to decision operation 1145, discussed further below.

[0310] Decision operation 1120 and decision operation 1125 determine whether the value specified by the data set (evaluated by the sub-rule) is an outlier. As described above, some embodiments determine the outlier by determining whether the value falls in the first quartile or the fourth quartile of the value. As described above, at least in some embodiments, the quartile boundaries are determined by associating the median of the value with the minimum value (for the lowest quartile) and associating the median with the maximum value (for the highest quartile). As discussed further above, some embodiments define thresholds different from Q1 and Q3 thresholds for outliers. As described above, some embodiments of decision operation 1120 and / or decision operation 1125 utilize relaxation factors, which reduce the number of values ​​that are considered to be outliers compared to the values ​​that fall into the Q1 and Q3 quartiles. As described above, in some embodiments, thresholds Q1 and Q3 are determined as relaxed quartile boundaries.

[0311] If any one of decision operation 1120 or decision operation 1125 determines that the value is an outlier, then method 1100 moves to operation 1140, which adds the sub-rule to the list of outlier sub-rules. Otherwise, if the value is not an outlier, then method 1100 moves from decision operation 1120 or 1125 to decision operation 1142.

[0312] Decision operation 1142 determines whether there are additional data sets to be evaluated for the selected sub-rule. If there are additional data sets, the method 1100 moves to operation 1112, which selects a different data set for evaluation. Otherwise, the method 1100 moves from decision operation 1142 to decision operation 1145. Decision operation 1145 determines whether there are additional sub-rules to be evaluated. If there are additional sub-rules to be evaluated, processing moves from decision operation 1145 to operation 1110, which selects a different sub-rule from the rule for processing by the method 1100. Otherwise, if there are no additional sub-rules that need to be processed, the method 1100 moves from decision operation 1145 to reference E 1180.

[0313] Fig.12 is a flow chart of an example process for marking a feature parameter value as "high" or "low". Fig.12 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, method 1200 is performed by NMS 136.

[0314] The method 1200 continues from reference E 1180. After leaving the page reference 1180, the method 1200 moves to operation 1205. Operation 1205 selects a data set from a plurality of data sets. The method 1200 continues to operation 1210, which uses the same selected sub-rule used in operation 1110. The method 1200 then proceeds to decision operation 1215, which determines whether the sub-rule satisfies the "greater than" rule. If it is determined that the "greater than" rule is satisfied, the process proceeds to decision operation 1220, which determines whether the correlation between the feature and the target parameter is positive. If the decision operation 1220 determines that the correlation is positive, the process continues to decision operation 1225, which determines whether the value of the feature parameter is greater than the third quartile (e.g., falls within the fourth quartile). (According to another embodiment, this operation checks whether the value of the feature is greater than the relaxed third quartile). If the value of the feature parameter is greater than the boundary between the third quartile and the fourth quartile, method 1200 moves from decision operation 1225 to operation 1230, which marks the feature parameter value as a "high" value.

[0315] Returning to decision operation 1215, if the operation determines that the sub-rule does not satisfy the "greater than" rule, the process proceeds to decision operation 1250, which determines whether the correlation between the feature and the target parameter is negative. If the operation determines that the correlation is negative, the process proceeds to decision operation 1225.

[0316] Decision operation 1225 determines whether the value of the feature parameter is greater than the third quartile. (According to another embodiment, this operation checks whether the value of the feature is greater than the relaxed third quartile.) If decision operation 1225 determines that the value of the feature parameter is greater than the third quartile boundary (or falls within the fourth quartile formed by the associated boundary between the third and fourth quartiles), then method 1200 moves from decision operation 1235 to operation 1240, which marks the sub-rule as "high". The process then continues to decision operation 1245.

[0317] However, if the decision operation 1225 determines that the feature parameter value is not within the fourth quartile (or within a smaller fourth quartile based on the relaxed third quartile boundary), the method 1200 proceeds to a decision operation 1235, which determines whether the value of the feature parameter falls within the first quartile. (According to another embodiment, this operation checks whether the value of the feature falls within the reduced size first quartile, which relies on the relaxed boundary between the first quartile and the second quartile.) If the operation determines that the value falls within the first quartile (or the relaxed first quartile), the process continues to a decision operation 1140, which marks the sub-rule as "low".

[0318] The high and low labels can also be viewed as natural language friendly expressive values, i.e. good (does not contribute to SLED) or bad (does not contribute to SLED). For a given feature value, we can express the deviation from the (loose) quartiles as low or high which IT staff can understand or good or bad which non-technical staff can understand. The mapping between {good, bad} and {high, low} depends on the specific feature. The label low is associated with parameters that indicate a high probability of bad SLE as their numerical values ​​become smaller. Similarly, the label high is associated with parameters that indicate a higher probability of bad SLE and therefore the parameter numerical values ​​become larger.

[0319] For example, an interference of 0.9 is considered "high", which maps to a poor or bad SLE, and as the interference gets larger, the probability of a poor SLE increases. A -90dBm measurement of RSSI is low and is poor or bad, and as RSSI gets smaller, higher probability values ​​of a poor SLE are indicated. Therefore, we see that for some target features, values ​​above a certain threshold indicate a good SLE, and for other features, values ​​above a certain threshold indicate a poor SLE, so depending on the specific feature, a feature can be mapped to good or very poor.

[0320] For the case where a larger target feature indicates better performance (e.g., a larger throughput indicates a better SLE), and for a positive correlation between the feature parameter and the target parameter, only the LT and LE rules apply, since only a lower feature parameter translates to a lower target parameter (compared to the threshold) and, therefore, to a poor SLE (class 0). Similarly, for a negative correlation between the feature parameter and the target parameter, only the GT and GE rules apply, since only a higher feature parameter translates to a lower target parameter (compared to the threshold) and, therefore, to a poor SLE.

[0321] The opposite is true for a smaller target feature indicating better performance (e.g., a smaller connection delay indicating better SLE), and for a positive correlation between the feature parameter and the target parameter, only the GE and GE rules are applicable, since only a larger feature parameter translates to a larger target parameter (compared to the threshold), and thus to a poor SLE (classification 0). Similarly, for a negative correlation between the feature parameter and the target parameter, only the LT and LE rules are applicable, since only a lower feature parameter translates to a higher target parameter (compared to the threshold), and thus to a poor SLE.

[0322] Similarly, if the decision operation 1220 determines that the correlation between the feature and the target parameter is not positive, or the decision operation 1250 determines that the correlation between the feature and the target parameter is not negative, or if the decision operation 1235 determines that the value of the feature is not less than the first quartile. (According to another embodiment, the relaxed first quartile), the process proceeds to decision operation 1245.

[0323] Decision operation 1245 checks whether there are more sub-rules that need to be processed. If decision operation 1245 determines that there are additional sub-rules that need to be processed, method 1200 returns to operation 1210, which selects a different sub-rule.

[0324] However, if the decision operation 1245 determines that there are no additional sub-rules to be processed, then the method 1200 proceeds to the method 1300 via the connector operation H 1290, which will be referred to below. Fig.13 Have a discussion.

[0325] Fig.13 is a flow chart of an example process for determining multiple operating parameters that cause SLED. In some embodiments, method 1300 takes corrective action based on multiple operating parameters. In some embodiments, the following Fig.13 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, method 1300 is performed by NMS 136.

[0326] Method 1300 is performed at connector operation H 1290 (also refer to method 1200 and Fig.12 ) and proceeds to decision operation 1310, which determines whether there are multiple outlier features (classified as high or low) associated with the selected data set. If decision operation 1310 determines that there are multiple outlier features, method 1300 proceeds from decision operation 1310 to operation 1315, which combines the multiple features with a logical "AND" operation. Decision operation 1320 evaluates whether there are additional data sets to evaluate. If there are additional data sets, method 1300 proceeds to operation 1322, where the next data point is selected. Method 1300 then returns to method 1200 and 1322 discussed above via connector 1390. Fig.12 Operation 1205. Operation 1205 is designed to select a different data set each time the process 1200 / 1300 is iterated.

[0327] If the decision operation 1320 determines that there are no more data sets to evaluate, then the method 1300 moves to operation 1325 where frequently satisfied sub-rules and / or composite sub-rules that contribute to the poor SLE are identified.

[0328] Operation 1330 identifies the first and third quartiles associated with low SLE. (As described above, some embodiments utilize relaxed first and third quartiles, according to another embodiment).

[0329] In operation 1335, a corrective action is determined based on parameters associated with the first quartile and third quartile classifications (e.g., prefixed with low and / or high). In some embodiments, the corrective action is determined from a list of stored and / or preconfigured corrective measures. Such stored and / or preconfigured actions include an increase (or decrease) in radio power to turn on or off a specific radio channel on a specific AP, etc. According to another embodiment, the system uses the first and third quartiles (Q1 and Q3) and the current value of the parameter to modify the parameter to facilitate the removal of the reduced SLE root cause. For example, if the RSSI measured by the device is -70 and the Q1 threshold is -65, operation 1335 determines that the coverage parameter of the specific device should be increased by 5db by moving the device closer to the AP or increasing the transmit power of the AP.

[0330] Some embodiments of method 1300 generate a message describing how a user of an access point (identified as problematic) can change his or her location to improve the signal strength experienced by the user's respective device. Alternatively, a message is generated to instruct the user to associate with a different access point, which is expected to provide improved SLE relative to the user's existing access point connection.

[0331] Operation 1340 performs a remedial action. In various embodiments, the remedial action includes presenting a suggested corrective action on a display (e.g., to an IT professional) or automatically (without human intervention) taking the remedial action and generating a notification that the action has been performed via, for example, a message sent to an email, a text message, or other messaging technology. In some embodiments, the NMS 136 issues a command to the resource manager to increase the transmit power of a particular radio in a particular AP. Continuing with the previous example, where the RSSI detected by the device is -70 dB, and the first quartile is determined to be -65 dB, the resource manager commands the radio in the associated AP to increase its transmit power by 5 dBs. Possible remedial actions also include changing the operating channel of the access point, or generating a manual instruction to move toward the access point.

[0332] The process continues to operation 1340 where corrective action is invoked. The process ends at operation 1350.

[0333] Those skilled in the art will recognize that although Fig.12 The flowchart of / 13 ends at operation 1350, but some embodiments of method 1200 / 1300 are performed iteratively to continuously monitor SLE parameters and parameters. By continuously monitoring system performance, corrective action can be taken immediately after SLE degradation is identified.

[0334] Fig.14 is a flow chart of an example process for invoking corrective action when a combination of underlying causes is identified. Fig.14 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, method 1400 is performed by NMS 136.

[0335] Method 1400 begins at start operation 1405 and proceeds to operation 1410. Operation 1410 identifies a plurality of operating parameters that cause SLED. In some embodiments, operation 1410 is based on the above description of Figure 6-13 In some embodiments, operation 1410 includes the following: Fig.15 The feature or features being discussed.

[0336] Operation 1420 identifies the expected action based on the plurality of operating parameters. For example, if the plurality of operating parameters indicates a root cause of the SLED based on the evaluation of the first sub-rule and the second sub-rule, the remedial action may include a first action (e.g., restarting the first server) and / or a second action (e.g., restarting the second server).

[0337] Operation 1430 determines the impact of each identified action. The impact may include multiple dimensions or components, including multiple devices affected by the action and an estimated size of each effect. In some embodiments, operation 1440 determines the action with the least adverse impact. In some embodiments, operation 1430 determines the impact based on the priority of the devices affected by the action and / or the priority of network traffic between the devices and the network. In some embodiments, operation 1430 associates the determined impact with a predetermined threshold to determine whether to perform the action. In some embodiments, the determined impact is related to the severity of the SLE downgrade, where a high severity downgrade justifies a larger impact action.

[0338] In operation 1450, the selected action is performed. As described above, the action may include one or more of restarting a particular server or sending an instruction message (e.g., to resource manager 132) to change one or more operating parameters (e.g., such as the transmit power of a particular radio, etc.). The disclosed embodiments also contemplate other remedial actions. Method 1400 ends at operation 1490.

[0339] Fig.15 is a flow chart of a method implemented in one or more disclosed embodiments. Fig.15 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, method 1500 is performed by NMS 136.

[0340] After starting operation 1505, method 1500 moves to operation 1510, where a plurality of data sets are obtained. Each data set includes a plurality of time-related operating parameter values. The operating parameter values ​​include feature parameters (e.g., independent variables in some embodiments) and target parameters (e.g., dependent variables in some embodiments).

[0341] In operation 1520, the data set is classified according to the value of the first target parameter. In some embodiments, the data set including the value of the first target parameter that meets the criteria (e.g., greater than a defined threshold or less than a defined threshold) is classified as the first category, while the data set including the value of the first target parameter that does not meet the criteria is classified as the second category. Some embodiments of operation 1520 perform multiple classifications of the data set. For example, after the data set is classified into the first category or the second category, they can be further classified into the third category or the fourth category based on the values ​​of different second target parameters (e.g., based on different second target parameter values ​​that meet the second criterion). In some embodiments, the second criterion evaluates the value of the second target parameter relative to the second threshold. In some embodiments, the data set is classified based on the third criterion applied to the first target parameter. Therefore, based on whether the first target parameter meets the third criterion (e.g., it evaluates the value of the first target parameter relative to the third threshold in various embodiments), the data set is divided into the fifth category or the sixth category. Therefore, various embodiments perform multiple classifications of the data set according to any number of target parameters and / or any number of multiple criteria applied to the common target parameter.

[0342] In some embodiments, to the extent that a particular embodiment implements these thresholds, one or more of the first threshold, the second threshold, and the third threshold are randomly generated. Alternatively, the threshold is generated based on the minimum and maximum values ​​of the relevant target parameter in the data set. For example, in some embodiments, the threshold is determined to fall between the minimum and maximum values. In some embodiments, the threshold is selected as the median or average value of the target parameter value.

[0343] In operation 1530, a decision tree is generated based on the data set and the classification of the data set. Therefore, in some embodiments, a decision tree is generated so as to generate a result consistent with the classification. In some embodiments, each node of the decision tree is assigned a classification. The assignment of the classification indicates a plurality of data sets each of the first classification and the second classification represented by the node. Therefore, for example, if a particular node of the decision tree represents x data sets of the first class and y data sets of the second class, then if x>y, the node is assigned to the first classification, and if y>x, the node is assigned to the second classification. If x=y, the embodiment can assign the node to the first classification or the second classification. If multiple classifications are performed by operation 1520, multiple corresponding decision trees are generated in operation 1530, one decision tree for each classification.

[0344] Some embodiments of operation 1530 classify each node of a decision tree (or multiple decision trees). In some embodiments, the classification of a particular node is based on a ratio between a data set classified as a first class and a data set classified as a second class represented by the particular node. The data set represented by the particular node indicates that the evaluation of the data set for the decision tree will traverse the particular node.

[0345] Some embodiments of method 1500 assign a Gini score to each node of a decision tree. Some embodiments assign Gini scores in a manner consistent with known methods of decision tree construction.

[0346] In operation 1540, a set of rules is generated based on the decision tree. In some embodiments, the set of rules is generated based only on those nodes of the decision tree that have a particular class of the two assigned classes. In some embodiments, the set of rules is generated based only on those nodes of the decision tree that have a Gini score that satisfies a defined criterion (e.g., comparing the Gini score to a predetermined threshold). In some embodiments, the rules are generated based only on those nodes of the decision tree that have at least a defined number of ancestors of a particular classification.

[0347] A rule set is generated by extracting rules from each node. The rule of a node determines whether a path through the tree passes to the first child node of the node or the second child node of the node. In an embodiment where multiple decision trees are generated, different rule sets are generated based on the multiple decision trees. Nodes from each of the multiple decision trees that are classified into one of the two classifications are used to determine the rules (e.g., similar to the first class of the first decision tree).

[0348] In operation 1550, a filtered rule set is generated based on the rule set of operation 1540. As described above, rule filtering includes one or more processing steps that reduce or modify the rules extracted from one or more decision trees. As described above, in some embodiments, rule filtering includes eliminating sub-rules extracted from the decision tree that are redundant with other sub-rules extracted from the decision tree. In some embodiments, rule filtering includes eliminating sub-rules that do not test for characteristic parameter conditions associated with poor or undesirable SLE (e.g., according to the above reference Figure 1B -E.) In some embodiments, one or more rules derived from the decision tree are modified to identify only feature parameter values ​​that are classified as outliers (or relaxed outliers as described above).

[0349] In operation 1560, a first classified data set is identified that satisfies at least a sub-rule in a filtered rule of a filtered rule set. For example, as discussed above with respect to FIG. 1, the filtering rules generated in operation 1550 (e.g., similar to filtering rule set 11512) are evaluated against a data set (e.g., data set 11502 of FIG. 1). In some embodiments, composite rules that include filtering rules (e.g., following the same structure as the rules of the rule set generated in operation 1540) are evaluated to identify a composite rule that has a defined number of sub-rules (or greater) that are satisfied by the data set.

[0350] In some embodiments, the identification of the data set is also based on the sub-rules satisfied by the outliers. As described above, outliers are considered to be values ​​that fall within the first quartile Q1 (less than the boundary between the first and second quartiles) or the third quartile Q3 (greater than the boundary between the third and fourth quartiles) of the feature parameter values. Some embodiments implement relaxed quartile boundaries that move the boundary between the first and second quartiles away from the mean of the value and also move the boundary between the third and fourth quartiles away from the mean of the value.

[0351] The sub-rules or composite sub-rules are then ranked according to how often they are satisfied by the data set. The sub-rules or composite sub-rules that are most frequently satisfied are selected. The feature parameters referenced by these sub-rules are identified in operation 1560.

[0352] In operation 1570 , feature parameters referenced by the identified operation sub-rule 1560 are identified.

[0353] In operation 1575, a remedial action is identified based on the identified feature parameter of operation 1570. As described above, the remedial action can vary depending on the feature parameter, but can include, for example, resetting a radio in a device, resetting the entire device, adjusting a configuration parameter of a specific device, or other actions. Other remedial actions include reducing the transmit power of a radio in an access point, increasing the transmit power of a radio in an access point, changing an operating channel in an access point, and / or generating an instruction to move to an access point. Certain remedial actions result in the display of an indicator of the identified function. For example, notifying support personnel that a specific feature parameter value is abnormal (determined by method 1500) can improve the efficiency of support personnel in trying to resolve a problem.

[0354] In some aspects, remedial actions are identified based on their potential impact on the operation of the monitored system. For example, some embodiments of operation 1575 determine the impact of a remedial action based on one or more of the number of users affected by the remedial action, the priority of one or more devices affected by the remedial action, or the priority of network communications affected by the remedial action. In some embodiments, the impacts of multiple possible remedial actions are determined and then compared to determine which remedial actions have the least impact. Some embodiments select the remedial action with the least impact.

[0355] In operation 1580, the one or more remedial actions identified in operation 1575 are performed. After operation 1580, the method 1500 moves to end operation 1585.

[0356] Fig.16 is a flow chart of a method implemented in one or more disclosed embodiments. In some embodiments, the following is performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506) Fig.16 For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure hardware processing circuitry to perform one or more functions discussed below. In some embodiments, method 1600 is performed by NMS 136. In some embodiments, method 1600 implements the above-mentioned Fig.15 Operations 1540 and 1550 of method 1500 are discussed.

[0357] After start operation 1605, method 1600 moves to operation 1610. In operation 1610, a rule is generated that defines a unique path to a leaf node of the decision tree.

[0358] In operation 1620, feature-specific sub-rules are generated from the rules generated in operation 1610. In some embodiments, operation 1620 is consistent with the above discussion of equation 15.

[0359] In operation 1630, the sub-rules are filtered.

[0360] In some embodiments, the filtering of the sub-rules includes the elimination of sub-rules according to the examples above Figure 1B-1E In some embodiments, the filtering of the sub-rules further includes modifying the sub-rules to be satisfied only by eigenvalue classified as outliers (or related outliers). Thus, for example, if a sub-rule associates a feature parameter with a constant value Kx, then Kx is replaced by depicting quartile boundaries (or loose quartile boundaries). If the relationship implemented by the sub-rule is greater than or greater than or equal to operator, it is replaced by the upper quartile boundary, and if the relationship is less than or less than or equal to operator, it is replaced by the lower quartile boundary (or lower quartile relaxation boundary).

[0361] In some embodiments, the filtering of the sub-rules includes eliminating less restrictive sub-rules in favor of maintaining more restrictive sub-rules. Thus, for example, two sub-rules apply the greater than operator to feature F1. In this example, the first sub-rule is F1>K1, and the second sub-rule is F1>K2. In this example, if K1>K2, then the second sub-rule is less restrictive and can be eliminated in operation 1630. Similarly, if two sub-rules apply the less than operator to feature F1, such as the third sub-rule is F1<K3, and the fourth sub-rule is F1<K4, if K3<K4, then the fourth sub-rule is less restrictive and is eliminated from the set of filtered sub-rules. The elimination of less restrictive sub-rules can also operate on sub-rules including less than or equal to operator and / or greater than or equal to operator.

[0362] After operation 1630, method 1600 moves to the end operation 1650.

[0363] Fig.17 is a flowchart of a method implemented in one or more disclosed embodiments. In some embodiments, one or more of the functions discussed below are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more of the functions discussed below. In some embodiments, method 1700 is executed by NMS 136. In some embodiments, method 1700 implements the above regarding Fig.17 discussed one or more functions. For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more of the functions discussed below. In some embodiments, method 1700 is executed by NMS 136. In some embodiments, method 1700 implements the above regarding Fig.15Operations 1560 and 1570 of method 1500 are discussed.

[0364] After start operation 1705, method 1700 moves to operation 1710. In operation 1710, the filtered sub-rules are evaluated for a data set having a first classification, for example, as discussed above with respect to FIG. 1, which data set 11502 is evaluated relative to the filtered rule set 11512.

[0365] In operation 1720, a composite rule is generated from a data set that matches at least a predefined number of filtered sub-rules. In some embodiments, the predetermined number is one, two, three, four, five, six, seven, eight, nine, or ten sub-rules. The disclosed embodiments consider any number of sub-rule matches to be a composite rule.

[0366] In operation 1730, the sub-rules and compound rules are ranked based on how often they are satisfied by the data set.

[0367] In operation 1740, features referenced by the highest ranked sub-rule or composite rule set are identified. Some embodiments then determine a remedial action based on the referenced features. After operation 1740, method 1700 moves to end operation 1750.

[0368] Fig.18 is a flowchart 1800 of a method implemented in one or more disclosed embodiments. In some embodiments, the following Fig.18 One or more functions discussed are performed by a hardware processing circuit (e.g., any one of 206, 306, 406, 506). For example, in some embodiments, instructions (e.g., 214, 314, 428, 529) stored in an electronic memory (e.g., any one of 208, 340, 416, 516) configure the hardware processing circuit to perform one or more functions discussed below. In some embodiments, method 1800 is performed by NMS 136.

[0369] After starting operation 1805, method 1800 moves to operation 1810, which selects a target parameter from a data set and / or selects a classification criterion for the target parameter. As described above, some disclosed embodiments classify a data set based on a target parameter value. The classification applies a criterion to the target parameter value and divides the data set into one of two classes based on whether the criterion evaluates a true value or a false value. In some embodiments, the criterion compares the target parameter value with a predetermined threshold. In some embodiments, the predetermined threshold is randomly generated. In some embodiments, the predetermined threshold is selected from a range of values ​​of the target parameter. Operation 1810 selects a single target parameter and a single criterion. As described below, as method 1800 iterates, different target parameters and / or different criteria are selected to perform additional classifications on the data set. Therefore, some embodiments classify a data set based on the value of a single target parameter and a single criterion applied to those target parameter values. Other embodiments classify a data set based on the value of a single parameter and apply multiple different criteria to those parameter values. Each criterion results in a separate binary classification of the data set. Other embodiments classify a data set based on the values ​​of multiple target parameters, wherein the value of each target parameter is evaluated for a single criterion. Other embodiments classify a data set based on values ​​of multiple target parameters, where the values ​​of one or more target parameters are evaluated against multiple corresponding criteria.

[0370] In operation 1820, a decision tree is generated based on the classification of the data set according to the selected target parameters and criteria of operation 1810. In some embodiments, operation 1820 first classifies the data set based on the selected target parameters and criteria, and then generates a decision tree based on the classification.

[0371] In operation 1830, a filtered set of sub-rules is generated based on the decision tree. As described above, generating the filtered set of sub-rules includes extracting sub-rules from the decision tree, grouping the sub-rules according to feature parameters referenced by the sub-rules, eliminating less restrictive sub-rules, possibly modifying the sub-rules to evaluate to true only when evaluating outlier feature parameter values ​​(or related outliers), and Figure 1B-1E Use any one or more of the techniques described in to delete a sub-rule.

[0372] In operation 1850, when the data set is evaluated (e.g., as shown in FIG. 1, relative to the evaluation data structure 11514 and the rule histogram 11516), the filtered sub-rule set is ranked according to the satisfaction frequency of the filtered sub-rule set. Consistent with the above discussion, if a particular data set (e.g., a row of the data set 11502) satisfies two or more sub-rules, then operation 1850 generates a composite rule including two or more sub-rules. These composite sub-rules are then ranked together with the single sub-rule according to the satisfaction frequency of the data set (e.g., the data set 11502). Note that if a particular sub-rule is satisfied, and the same data set (e.g., the row 11502 of the data set) also satisfies at least a second sub-rule, then the satisfaction of the particular sub-rule is counted for the composite sub-rule, and is not counted for the particular sub-rule independently of the composite sub-rule. If the second data set (e.g., the second row of the data set 11502) satisfies the particular sub-rule without the other sub-rules, then the satisfaction is counted separately. Thus, the ranking tracks the satisfaction of an individual sub-rule and the satisfaction of the sub-rule together with one or more other sub-rules, respectively.

[0373] Decision operation 1870 evaluates whether the embodiment represented by method 1800 evaluates additional criteria and / or additional target parameters. If additional target parameters and / or criteria are to be evaluated, method 1800 returns to operation 1810, where new target parameters and / or criteria are selected. As described above, in some embodiments, some data sets are classified based on the values ​​of two or more target parameters. In some embodiments, data sets are classified by applying multiple different criteria to a single target parameter. If further evaluation of target parameters or criteria is not required, method 1800 moves to operation 1880.

[0374] Operation 1880 identifies the highest ranked set of sub-rules and / or composite sub-rules based on one or more rankings of operation 1850. Some embodiments of operation 1880 identify identical sub-rules and / or composite sub-rules referenced by multiple rankings (generated via multiple calls to operation 1850 when method 1800 iterates) and summarize their respective satisfaction frequencies. The highest ranked set of sub-rules and / or composite sub-rules in the ranking is then selected. As described above, feature parameters referenced based on the selected sub-rules and / or composite sub-rules are then identified, and actions are selected based on the identified feature parameters. In some embodiments, operation 1880 includes the functionality discussed above with respect to operations 1575 and 1580. After operation 1880 is completed, method 1800 moves to end operation 1890.

[0375] The techniques of various embodiments may be implemented using software, hardware, and / or a combination of software and hardware. Various embodiments are directed to devices such as management entities, such as location / area tracking servers, network monitoring nodes, routers, gateways, access points, DHCP servers, DNS servers, AAA servers, resource managers, user equipment devices such as wireless nodes such as mobile wireless terminals, base stations, communication networks, and communication systems. Various embodiments are also directed to methods, such as methods of controlling and / or operating one or more communication devices, such as location / area tracking servers, network management nodes, access points, wireless terminals (UEs), base stations, control nodes, DHCP nodes, DNS servers, AAA nodes, mobility management entities (MMEs), networks, and / or communication systems. Various embodiments are also directed to non-transient machines, such as computers, readable media, such as ROMs, RAMs, CDs, hard disks, etc., which include machine-readable instructions for controlling the machine to implement one or more steps of the method.

[0376] It should be understood that the specific order or hierarchy of steps in the disclosed processes are examples of approaches. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the processes can be rearranged while remaining within the scope of the present disclosure. The accompanying method claims present elements of the various steps in an example order and are not meant to be limited to the specific order or hierarchy presented.

[0377] In various embodiments, one or more modules are used to perform steps corresponding to one or more methods, such as signal generation, transmission, processing, analysis and / or reception steps, to implement the devices and nodes described herein. Therefore, in some embodiments, modules are used to implement various features. Such modules can be implemented using software, hardware, or a combination of software and hardware. In some embodiments, each module is implemented as a separate circuit with a device or system, which includes a separate circuit for implementing the functions corresponding to each described module. Many of the above methods or method steps can be implemented using machine executable instructions such as software included in a machine-readable medium, such as a storage device such as RAM, a floppy disk, etc., to control a general-purpose computer with or without additional hardware, such as to implement all or part of the above methods in one or more nodes. Therefore, among other things, various embodiments are directed to a machine-readable medium, such as a non-transitory computer-readable medium, including machine executable instructions for causing a machine such as a processor and associated hardware to perform one or more steps of the above methods. Some embodiments are directed to a device including a processor configured to implement one, more or all steps of one or more methods of the disclosed embodiments.

[0378] In some embodiments, one or more processors, such as CPUs, of one or more devices of a communication device, such as a network management node, a wireless terminal (UE) and / or an access node, are configured to perform the steps of the method described as being performed by the device. The configuration of the processor can be achieved by using one or more modules, such as software modules, to control the processor configuration and / or by including hardware, such as hardware modules, in the processor to perform the steps and / or control the processor configuration. Therefore, some but not all embodiments are directed to a communication device with a processor, such as a network management device such as a server, such as a location / area tracking server, an access point, a user device, etc., the processor includes a module corresponding to each step of the various described methods performed by the device in which the processor is included. In some but not all embodiments, the communication device includes a module corresponding to each step of the various described methods performed by the device in which the processor is included. These modules can be implemented purely in hardware, such as as a circuit, or can be implemented using software and / or hardware or a combination of software and hardware.

[0379] Some embodiments are directed to a computer program product, which includes a computer-readable medium, which includes codes for causing one or more computers to implement various functions, steps, actions and / or operations, such as one or more steps described above. According to an embodiment, the computer program product can and sometimes does include different codes for each step to be performed. Therefore, the computer program product can and sometimes does include codes for each individual step of the method, such as a method of operating a communication device, such as a network management node such as a server, such as a location / area tracking server, an access point, a base station, a wireless terminal or a node. The code can be in the form of a machine such as a computer, an executable instruction stored on a computer-readable medium such as a RAM (random access memory), a ROM (read-only memory) or other type of storage device. In addition to being directed to a computer program product, some embodiments are also directed to a processor configured to implement one or more of the various functions, steps, actions and / or operations of one or more of the above-mentioned methods. Therefore, some embodiments are directed to a processor configured to implement some or all steps of the method described herein, such as a CPU. The processor can be used in, for example, a communication device or other device described in this application.

[0380] Although described in the context of communication systems including cellular, Wi-Fi, Bluetooth, and BLE, at least some methods and apparatus of various embodiments are applicable to a wider range of communication systems including many non-OFDM and / or non-cellular systems.

[0381] In view of the above description, many other changes of the methods and devices of the various embodiments described above will be apparent to those skilled in the art. Such changes should be considered to be within the scope. The method and apparatus can and in various embodiments can be used together with CDMA, orthogonal frequency division multiplexing (OFDM), Wi-Fi, Bluetooth, BLE and / or various other types of communication technologies that can be used to provide a wireless communication link between an access node and a wireless node. In some embodiments, the access node is implemented as a base station, which uses Wi-Fi, Bluetooth, BLE, OFDM and / or CDMA to establish a communication link with a user device such as a mobile node. In various embodiments, the wireless node is implemented as a notebook computer, a personal data assistant (PDA) or other portable devices including a receiver / transmitter circuit and logic and / or routines to implement the method.

[0382] Example 1 is a method, comprising: obtaining multiple data sets, each data set including time-related operation parameter values, the operation parameter values ​​including the value of a characteristic parameter and the value of a target parameter; first classifying a portion of the data sets in the data sets into a first category of data sets or a second category of data sets based on the target parameter value indicated by each data set in the data sets; generating a decision tree based on the first classification of the multiple data sets; generating a rule set from the decision tree; filtering the rule set; identifying the data set of the first category that at least satisfies the sub-rules included in the filtered rule set; identifying a first characteristic parameter based on the sub-rule; identifying a remedial action based on the first characteristic parameter; and performing the remedial action.

[0383] In Example 2, the technical solution of Example 1 may optionally include: identifying the value of the first characteristic parameter as an outlier, wherein the identification of the first characteristic parameter is based on the identification of the value.

[0384] In Example 3, the technical solution of Example 2 may optionally include: determining quartile boundaries of the values ​​of the characteristic parameters, wherein the identification of the first characteristic parameter is based on the quartile boundaries.

[0385] In Example 4, the technical solutions of any one or more of Examples 1-3 may optionally include: determining that at least two sub-rules are satisfied by the data set, wherein the identification of the first characteristic parameter is based on the first sub-rule of the at least two sub-rules; and identifying the second characteristic parameter based on the second sub-rule of the at least two sub-rules.

[0386] In Example 5, the technical solution of Example 4 may optionally include: identifying a second remedial action based on the second characteristic parameter; determining a first impact of the first remedial action; determining a second impact of the second remedial action; and conditionally executing one or more remedial actions of the first remedial action or the second remedial action based on the first impact and the second impact.

[0387] In Example 6, the technical solution of Example 5 may optionally include: wherein determining the first impact is based on one or more of the following: the number of users affected by the first remedial action, the priority of devices affected by the first remedial action, or the priority of network communications affected by the first remedial action.

[0388] In Example 7, the technical solutions of any one or more of Examples 1-6 optionally include: wherein the part is classified as the first action class to evaluate the target parameter against a first predetermined threshold, the method further includes: classifying the multiple data sets into a third category of data sets or a fourth category of data sets based on a second threshold; and generating a second decision tree based on the multiple data sets and the second classification, wherein the rule set is further generated based on the second decision tree.

[0389] In Example 8, the technical solutions of any one or more of Examples 1-7 may optionally include, wherein generating the decision tree includes generating nodes of the decision tree, and assigning each node of the decision tree to the first category or the second category.

[0390] In Example 9, the technical solution of Example 8 optionally includes that the allocation of each node is based on a ratio of the first type of data set to the second type of data set represented by the node.

[0391] In Example 10, the technical solutions of any one or more of Examples 8-9 may optionally include: wherein nodes of the decision tree assigned a classification different from the first category are excluded from the decision tree generation rule.

[0392] In Example 11, the technical solutions of any one or more of Examples 8-10 may optionally include: wherein the generation of the decision tree includes assigning a Gini score to each node, and wherein nodes having Gini scores that satisfy a criterion are excluded from the decision tree generation rule set.

[0393] In Example 12, the technical solutions of one or more of Examples 8-11 may optionally include: wherein generating a rule set from a decision tree is based only on nodes of the decision tree, wherein the nodes of the decision tree have at least a predefined number of ancestor nodes assigned to the first class.

[0394] In Example 13, the technical solution of Example 12 optionally includes: wherein generating the rule set from the decision tree further excludes nodes having a number of ancestor nodes assigned to the first class that satisfy the second criterion from consideration.

[0395] In Example 14, the technical solutions of any one or more of Examples 1-13 may optionally include: ranking the filtered sub-rules and composite rules in the filtered rule set according to satisfaction of multiple data sets, wherein the identification of the data set is based on each ranking in the ranking.

[0396] In Example 15, the technical solution of Example 14 may optionally include: selecting the filtered rule based on the ranking; determining the feature parameter referenced by the selected filtering rule; and displaying an indicator of the determined feature parameter on the electronic display.

[0397] In Example 16, the technical solutions of any one or more of Examples 14-15 may optionally include: selecting a filtering rule based on a ranking; and determining a feature parameter referenced by the selected filtering rule, wherein identifying the remedial action is based on the determined feature parameter.

[0398] In Example 17, the technical solutions of any one or more of Examples 10-16 may optionally include, wherein the remedial action includes one or more of the following: reducing the transmission power of the radio in the access point, increasing the transmission power of the radio in the access point, changing the operating channel in the access point, and generating instructions to move toward the access point.

[0399] In Example 18, the technical solutions of any one or more of Examples 1-17 may optionally include: classifying a second part of the data set into a first category of data set based on a second target parameter value indicated by each of the data sets; generating a second decision tree based on multiple data sets and a second classification; and generating a rule set based on the second decision tree.

[0400] In Example 19, the technical solutions of any one or more of Examples 1-18 may optionally include: wherein the classification of a part of a first category data set is based on a first criterion, the method further includes: classifying a second part of the data set as the first category data set for a second time based on a second criterion; generating a second decision tree based on the second classification; and generating a rule set based on the second decision tree.

[0401] In Example 20, the technical solution of Example 19 optionally includes that the first criterion and the second criterion refer to a first predetermined threshold and a second predetermined threshold respectively, and the method also includes randomly determining the first predetermined threshold and randomly determining the second predetermined threshold.

[0402] In Example 21, the technical solutions of any one or more of Examples 19-20 may optionally include, wherein the first criterion and the second criterion respectively reference a first predetermined threshold and a second predetermined threshold, and the method further includes selecting the first predetermined threshold and the second predetermined threshold based on the minimum value and the maximum value of the target parameter in the data set.

[0403] In Example 22, the technical solutions of any one or more of Examples 1-21 optionally include determining the correlation between each feature parameter and the target parameter value of the first category; and selecting a sub-rule set based on the determined correlation, wherein at least an identifier of a data set of the first category that satisfies the sub-rule identifies a data set that satisfies the sub-rules included in the sub-rule set.

[0404] Example 23 is a system comprising: a hardware processing circuit; one or more memories storing instructions that, when executed, configure the hardware processing circuit to perform operations including the following: obtaining multiple data sets, each data set including time-related operating parameter values, the operating parameter values ​​including the value of a characteristic parameter and the value of a target parameter; first classifying a portion of the data sets in the data sets into a first category of data sets or a second category of data sets based on the target parameter value indicated by each data set in the data sets; generating a decision tree based on the multiple data sets and the first classification; generating a rule set from the decision tree; filtering the rule set; identifying the first classified data set that at least satisfies the sub-rule included in the filtered rule set; identifying a first characteristic parameter based on the sub-rule; identifying a remedial action based on the first characteristic parameter; and executing the remedial action.

[0405] In Example 24, the technical solution of Example 23 optionally includes an operation, which further includes: identifying that the value of the first feature parameter is an outlier, wherein the identification of the first feature parameter is based on the identification of the value.

[0406] In Example 25, the technical solution of Example 24 optionally includes an operation, which further includes: determining the quartile boundaries of the values ​​of the characteristic parameters, wherein the identification of the first characteristic parameter is based on the quartile boundaries.

[0407] In Example 26, the technical solution of one or more examples in Examples 23-25 ​​optionally includes operations, which further include: determining that at least two sub-rules are satisfied by the data set, wherein the identification of the first characteristic parameter is based on the first sub-rule of the at least two sub-rules; and identifying the second characteristic parameter based on the second sub-rule of the at least two sub-rules.

[0408] In Example 27, the technical solution of Example 26 optionally includes an operation, which further includes: identifying a second remedial action based on a second characteristic parameter; determining a first impact of the first remedial action; determining a second impact of the second remedial action; and conditionally executing one or more of the first remedial action or the second remedial action based on the first impact and the second impact.

[0409] In Example 28, the technical solution of Example 27 may optionally include: wherein determining the first impact is based on one or more of: the number of users affected by the first remedial action, the priority of devices affected by the first remedial action, or the priority of network communications affected by the first remedial action.

[0410] In Example 29, the technical solution of one or more examples in Examples 23-28 optionally includes: wherein the target parameter is evaluated against a first predetermined threshold value by classifying part of the data sets as a first category, and the operation also includes: classifying the multiple data sets into a third category data set or a fourth category data set based on a second threshold value; and generating a second decision tree based on the multiple data sets and the second classification, wherein the rule set is further generated based on the second decision tree.

[0411] In Example 30, the technical solution of one or more of Examples 23-29 may optionally include: wherein the generation of the decision tree includes generating nodes of the decision tree, and assigning each node of the decision tree to the first category or the second category.

[0412] In Example 31, the technical solution of Example 30 optionally includes that the allocation of each node is based on a ratio of the first type of data set to the second type of data set represented by the node.

[0413] In Example 32, the technical solutions of any one or more of Examples 30-31 may optionally include: wherein nodes of the decision tree assigned a classification different from the first category are excluded from the decision tree generation rule.

[0414] In Example 33, the technical solution of one or more of Examples 30-32 optionally includes: wherein the generation of the decision tree includes assigning a Gini score to each node, and wherein nodes having Gini scores that satisfy a criterion are excluded from the decision tree generation rule set.

[0415] In Example 34, the technical solutions of one or more of Examples 30-33 may optionally include: wherein generating a rule set from a decision tree is based only on nodes of the decision tree, wherein the nodes of the decision tree have at least a predetermined number of ancestor nodes assigned to the first class.

[0416] In Example 35, the technical solution of Example 34 optionally includes that generating the rule set from the decision tree further excludes consideration of nodes having a number of ancestor nodes assigned to the first class that satisfy the second criterion.

[0417] In Example 36, the technical solution of one or more of Examples 23-35 optionally includes an operation, which further includes: ranking the filtered sub-rules and synthetic rules in the filtered rule set according to the frequency of satisfaction of multiple data sets, wherein the identification of the data set is based on each ranking in the ranking.

[0418] In Example 37, the technical solution of Example 36 optionally includes an operation, which further includes: selecting a filtering rule based on the ranking; determining a feature parameter referenced by the selected filtering rule; and displaying an indicator of the determined feature parameter on an electronic display.

[0419] In Example 38, the technical solution of one or more of Examples 36-37 optionally includes an operation, which further includes: selecting a filtering rule based on a ranking; and determining a feature parameter referenced by the selected filtering rule, wherein the identification of the remedial action is based on the determined feature parameter.

[0420] In Example 39, the technical solutions of any one or more of Examples 33-38 may optionally include, wherein the remedial action includes one or more of the following: reducing the transmission power of the radio in the access point, increasing the transmission power of the radio in the access point, changing the operating channel in the access point, and generating instructions to move toward the access point.

[0421] In Example 40, the technical solutions of any one or more of Examples 23-39 optionally include operations, which further include: classifying a second part of the data set into a first category of data set for a second time based on a second target parameter value indicated by each data set in the data set; generating a second decision tree based on multiple data sets and the second classification; and generating a rule set based on the second decision tree.

[0422] In Example 41, the technical solution of one or more examples in Examples 23-40 optionally includes: wherein the classification of a part of a data set of the first category is based on a first criterion, the method further includes: classifying a second part of the data set as a data set of the first category for a second time based on a second criterion; generating a second decision tree based on the second classification; and generating a rule set based on the second decision tree.

[0423] In Example 42, the technical solution of Example 41 optionally includes that the first criterion and the second criterion refer to a first predetermined threshold and a second predetermined threshold respectively, and the operation also includes randomly determining the first predetermined threshold and randomly determining the second predetermined threshold.

[0424] In Example 43, the technical solution of one or more examples in Examples 41-42 optionally includes, wherein the first criterion and the second criterion respectively refer to a first predetermined threshold and a second predetermined threshold, and the operation also includes selecting the first predetermined threshold and the second predefined threshold based on the minimum value and the maximum value of the target parameter in the data set.

[0425] In Example 44, the technical solutions of any one or more of Examples 23-43 optionally include an operation, which further includes: determining the correlation between each characteristic parameter and the target parameter value of the first category; and selecting a set of sub-rules based on the determined correlation, wherein at least the identifier of the data set of the first category that satisfies the sub-rule identifies the data set that satisfies the sub-rule included in the set of sub-rules.

[0426] Example 45 is a non-transitory computer-readable medium comprising instructions that, when executed, configure a hardware processing circuit to perform operations including the following: obtaining multiple data sets, each data set comprising time-related operating parameter values, the operating parameter values ​​comprising a characteristic parameter value and a target parameter value; first classifying a portion of the data sets into a first category of data sets or a second category of data sets based on the target parameter value indicated by each data set in the data sets; generating a decision tree based on the multiple data sets and the first classification; generating a rule set from the decision tree; filtering the rule set; identifying a data set of the first category that at least satisfies a sub-rule included in the filtered rule set; identifying a first characteristic parameter based on the sub-rule; identifying a remedial action based on the first characteristic parameter; and performing the remedial action.

[0427] In Example 46, the technical solution of Example 45 optionally includes an operation, which further includes: identifying that the value of the first feature parameter is an outlier, wherein the identification of the first feature parameter is based on the identification of the value.

[0428] In Example 47, the technical solution of Example 46 optionally includes an operation, which further includes: determining the quartile boundaries of the values ​​of the characteristic parameters, wherein the identification of the first characteristic parameter is based on the quartile boundaries.

[0429] In Example 48, the technical solution of one or more of Examples 45-47 optionally includes the following operations: further comprising: determining that at least two sub-rules are satisfied by the data set, wherein identification of the first characteristic parameter is based on the th sub-rule of the at least two sub-rules; and identifying the second characteristic parameter based on the second sub-rule of the at least two sub-rules.

[0430] In Example 49, the technical solution of Example 48 optionally includes an operation, which further includes: identifying a second remedial action based on a second characteristic parameter; determining a first impact of the first remedial action; determining a second impact of the second remedial action; and conditionally executing one or more of the first remedial action or the second remedial action based on the first impact and the second impact.

[0431] In Example 50, the technical solution of Example 49 may optionally include: wherein determining the first impact is based on one or more of: the number of users affected by the first remedial action, the priority of devices affected by the first remedial action, or the priority of network communications affected by the first remedial action.

[0432] In Example 51, the technical solutions of any one or more of Examples 45-50 optionally include: wherein a target parameter is evaluated against a first predetermined threshold value by classifying part of the data sets as a first classification, and the operation further includes: classifying the multiple data sets into a third category data set or a fourth category data set based on a second threshold value; and generating a second decision tree based on the multiple data sets and the second classification, wherein the rule set is further generated based on the second decision tree.

[0433] In Example 52, the technical solution of one or more of Examples 45-51 may optionally include: wherein the generation of the decision tree includes generating nodes of the decision tree, and assigning each node of the decision tree to the first category or the second category.

[0434] In Example 53, the technical solution of Example 52 may optionally include that the allocation of each node is based on a ratio of the first type of data set to the second type of data set represented by the node.

[0435] In Example 54, the technical solutions of any one or more of Examples 52-53 may optionally include: wherein nodes of the decision tree assigned a classification different from the first classification are excluded from the decision tree generation rule.

[0436] In Example 55, the technical solution of one or more of Examples 52-54 optionally includes: wherein generating a decision tree includes assigning a Gini score to each node, and wherein nodes with Gini scores that meet a criterion are excluded from the decision tree generation rule set.

[0437] In Example 56, the technical solution of Example 55 may optionally include that generating the rule set from the decision tree is based only on nodes of the decision tree, wherein the nodes of the decision tree have at least a predefined number of ancestor nodes assigned to the first class.

[0438] In Example 57, the technical solution of Example 56 optionally includes that generating the rule set from the decision tree further excludes nodes having a number of ancestor nodes that satisfy the second criterion from consideration.

[0439] In Example 58, the technical solution of one or more of Examples 45-57 optionally includes an operation, which further includes: ranking the filtered sub-rules and composite rules in the filtered rule set according to the frequency of satisfaction of multiple data sets, wherein the identification of the data set is based on each ranking in the ranking.

[0440] In Example 59, the technical solutions of one or more of Examples 56-58 may optionally include operations, which further include: selecting filtering rules based on rankings; determining feature parameters referenced by the selected filtering rules; and displaying an indicator of the determined feature parameters on an electronic display.

[0441] In Example 60, the technical solutions of any one or more of Examples 56-59 may optionally include the following operations: further comprising: selecting filtering rules based on rankings; and determining characteristic parameters referenced by the selected filtering rules, wherein determining the identification of the remedial action is based on the determined characteristic parameters.

[0442] In Example 61, the technical solutions of any one or more of Examples 45-60 may optionally include, wherein the remedial action includes one or more of the following: reducing the transmission power of the radio in the access point, increasing the transmission power of the radio in the access point, changing the operating channel in the access point, and generating instructions to move toward the access point.

[0443] In Example 62, the technical solution of one or more of Examples 45-61 optionally includes an operation, which further includes: based on a second target parameter value indicated by each of the data sets, classifying a second part of the data set into a first category of data set for a second time; generating a second decision tree based on multiple data sets and the second classification; and generating a rule set based on the second decision tree.

[0444] In Example 63, the technical solution of one or more examples in Examples 45-62 optionally includes: wherein the classification of a part of a first classified data set is based on a first criterion, the method further includes: classifying a second part of the data set as a first category data set for a second time based on a second criterion; generating a second decision tree based on the second classification; and generating a rule set based on the second decision tree.

[0445] In Example 64, the technical solution of Example 63 optionally includes that the first criterion and the second criterion refer to a first predetermined threshold and a second predetermined threshold respectively, and the operation also includes randomly determining the first predetermined threshold and randomly determining the second predetermined threshold.

[0446] In Example 65, the technical solution of any one or more of Examples 63-64 optionally includes, wherein the first criterion and the second criterion respectively reference a first predetermined threshold and a second predetermined threshold, and the operation also includes selecting the first predetermined threshold and the second predetermined threshold based on the minimum value and the maximum value of the target parameter in the data set.

[0447] In Example 66, the technical solution of Example 65 optionally includes the following operations: further including: determining the correlation between each feature parameter and the target parameter value of the first category; and selecting a sub-rule set based on the determined correlation, wherein the identifier of at least the data set of the first category that satisfies the sub-rule identifies the data set that satisfies the sub-rule included in the sub-rule set.

Claims

1. A system for determining a cause of a network event, comprising: Processing circuit; one or more memories storing instructions that, when executed, cause the processing circuit to: Obtaining a plurality of data sets, each of the plurality of data sets comprising one or more operating parameter values, each of the one or more operating parameter values ​​comprising a characteristic parameter and a target parameter; classifying each data set in the plurality of data sets based on a corresponding target parameter value in the corresponding data set; generating a rule set including one or more sub-rules based on the category of each data set in the plurality of data sets; Determining one or more characteristic parameters in the one or more data sets that satisfy at least one of the one or more sub-rules based on identifying one or more data sets in the multiple data sets; as well as Based on the one or more characteristic parameters in the one or more data sets that satisfy the one or more sub-rules, a remedial action is performed.

Images