Monitoring network variability

Abstract

The invention relates to monitoring network variability. Novel solutions are used for monitoring and analyzing networks in terms of variability of various devices. Some solutions consider a set of weighted metrics in determining this variability. Evaluating devices and peers in view of these factors may create insights about network conditions.

Description

Technical Field

[0001] The present disclosure relates generally to the field of computer networks, and more particularly, to network monitoring tools. Background Art

[0002] Variability in measured performance statistics, such as burstiness in an interface, is difficult to predict and current monitoring tools have difficulty assessing this variability. Lack of consistency means that network architects and planners cannot accurately assess, predict, and plan for resource availability and performance. This results in unpredictable performance and user experience for network transactions involving volatile devices. For example, in a situation where different devices experience very different data patterns, but the averages for each are similar and do not breach operational thresholds, this is often not visible to network monitoring tools and analysts.

[0003] Instead, current tools and techniques focus on identifying metrics that exceed operational thresholds using over-threshold time windows, identifying abnormal performance caused by baseline deviations, and identifying long-term metric impacts (which are displayed as TopN performance outliers on the dashboard). On the other hand, these tools and techniques have difficulty identifying short spikes / bursts that do not exceed the over-threshold time and do not generate events, normalizing deviations (widening the "normal" baseline window, preventing future deviations from being identified and eliminating consistent deviations of events), and short bursts that do not significantly affect overall TopN visibility, especially over longer reporting periods.

[0004] Therefore, new tools and techniques are needed that can provide more robust network monitoring and analysis. Summary of the invention

[0005] One aspect of the present disclosure relates to a system comprising: one or more processors; and a non-transitory computer-readable medium having encoded thereon a set of instructions executable by the one or more processors, the set of instructions comprising: instructions for receiving a first plurality of data points, the plurality of data points indicating a measurement of a first metric in a computer network and a second metric in the computer network, the plurality of data points comprising data points of a first device in the computer network and data points of a second device in the computer network; instructions for calculating a variability measure of the first metric of the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the first metric of the second device from at least some of the plurality of data points; and instructions for calculating the second metric of the first device from at least some of the plurality of data points. instructions for calculating a variability measure of the second metric of the second device from at least some of the plurality of data points; instructions for applying a first weighting factor to weight the variability measure of the first metric of the first device and the second device; instructions for applying a second weighting factor to weight the variability measure of the second metric of the first device and the second device; instructions for sorting the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device; and instructions for displaying a list of the sorted first and second devices to a user, the list showing the variability measures of the first metric and the second metric of each of the first device and the second device.

[0006] Another aspect of the present disclosure relates to an apparatus comprising: a non-transitory computer-readable medium having encoded thereon a set of instructions executable by one or more processors, the set of instructions comprising: instructions for receiving a first plurality of data points, the plurality of data points indicating a measurement of a first metric in a computer network and a second metric in the computer network, the plurality of data points comprising data points of a first device in the computer network and data points of a second device in the computer network; instructions for calculating a variability measure of the first metric for the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the first metric for the second device from at least some of the plurality of data points; instructions for calculating a variability measure of the second metric for the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the second metric of the second device from at least some of the plurality of data points; instructions for applying a first weighting factor to weight the variability measure of the first metric of the first device and the second device; instructions for applying a second weighting factor to weight the variability measure of the second metric of the first device and the second device; instructions for sorting the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device; and instructions for displaying a list of the sorted first and second devices to a user, the list showing the variability measures of the first metric and the second metric of each of the first device and the second device.

[0007] Another aspect of the present disclosure relates to a method comprising: receiving at a computer system a first plurality of data points, the plurality of data points indicating a measurement of a first metric in a computer network and a second metric in the computer network, the plurality of data points including data points of a first device in the computer network and data points of a second device in the computer network; using the computer system and calculating from at least some of the plurality of data points a measure of variability of the first metric for the first device; calculating from at least some of the plurality of data points a measure of variability of the first metric for the second device; using the computer system and calculating from at least some of the plurality of data points a measure of variability of the second metric for the first device; using the computer system and calculating from at least some of the plurality of data points a measure of variability of the second metric for the first device; at least some of which calculate a variability measure of the second metric of the second device; using the computer system to apply a first weighting factor to weight the variability measure of the first metric of the first device and the second device; using the computer system to apply a second weighting factor to weight the variability measure of the second metric of the first device and the second device; using the computer system to sort the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device; and displaying a list of the sorted first and second devices to a user, the list showing the variability measures of the first metric and the second metric for each of the first device and the second device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating a network according to various embodiments.

[0009] Figure 2A is a block diagram illustrating an edge router according to various embodiments.

[0010] Figure 2B is a block diagram illustrating a core router according to various embodiments.

[0011] Figure 3 is a flow chart illustrating a network monitoring method according to some embodiments.

[0012] Figure 4 is a flow chart illustrating a method of identifying a peer device according to some embodiments.

[0013] Figure 5 is a block diagram illustrating example components of a computer system according to some embodiments. DETAILED DESCRIPTION

[0014] Some embodiments may utilize a combination of standard deviation and variance to identify variability in a specific device. Some embodiments may provide comparisons across multiple devices in a group across multiple metrics of different unit types, allowing cross-comparison of metrics, such as utilization as a percentage value and error as a count or rate (count over time). According to various embodiments, measures of variability may be used to provide dashboards and / or workflows to help identify variability items and their causes / effects. In some embodiments, these measures may provide unconventional (beyond the norm) use cases with different measures and weightings that can be applied to both the technology and its specific implementation in a given network. Additionally and / or alternatively, some embodiments may provide a list and / or ranking of the most volatile devices in a specific group within a specified time period (one day, two days, etc.). Other embodiments may provide trendable variability measures to help understand and predict changes in the network.

[0015] One set of embodiments provides a system. A system according to one set of embodiments may include one or more processors and logic, such as a non-transitory computer-readable medium having encoded thereon a set of instructions executable by the processor. In some cases, the logic may be circuitry such that the instructions are encoded into and / or directly executed by hardware / firmware circuitry.

[0016] The set of instructions may include instructions for receiving a first plurality of data points. In one aspect, the plurality of data points may indicate a measurement of a first metric in a computer network and a second metric in the computer network. In another aspect, the plurality of data points may include data points of a first device in the computer network and data points of a second device in the computer network. The metric may be measured at any layer of the Open Systems Interconnection (OSI) model, and in a particular embodiment, the metric is measured at a layer below layer 4 of the OSI model. In some cases, the first metric is named in a first unit of measurement, the second metric is named in a second unit of measurement, and the first and second units of measurement are different.

[0017] In some embodiments, the device may be an interface, such as a LAN or WAN interface of a router; other examples of devices are described below. Thus, the metric may include, but is not limited to, measurements of: incoming packet rate at the interface, outgoing packet rate at the interface, incoming utilization at the interface, outgoing utilization at the interface, incoming discards at the interface, outgoing discards at the interface, and / or errors at the interface. In some embodiments, the system may perform the monitoring function even if the plurality of data points do not exceed an over-threshold time condition for the first metric or the second metric and / or the plurality of data points indicate a normalized deviation for the first metric and / or the second metric.

[0018] In some embodiments, the set of instructions includes instructions to calculate a variability measure of a first metric for a first device from at least some of the plurality of data points and / or instructions to calculate a variability measure of a first metric for a second device from at least some of the plurality of data points. There may also be instructions to calculate a variability measure of a second metric for the first device from at least some of the plurality of data points and / or instructions to calculate a variability measure of a second metric for the second device from at least some of the plurality of data points.

[0019] Particular embodiments may include instructions to determine that a first device and a second device are peer devices from among a plurality of devices in a peer-to-peer network. This determination may include monitoring the first device and the second device, identifying one or more peer characteristics between the first and second devices, and / or classifying the first device and the second device as peer devices. By way of example only, these peer characteristics may include things such as similar application traffic flows, similar network arrangements with entities external to the computer network, one or more similar characteristics of a control plane of a software defined network (SDN), and / or the like.

[0020] The set of instructions may also include instructions to apply a first weighting factor to weight the variability measures of the first metric of the first device and the second device and / or to apply a second weighting factor to weight the variability measures of the second metric of the first device and the second device. The instructions may then include instructions to rank the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device.

[0021] In some embodiments, the set of instructions may include instructions to display to a user a list of ordered first devices and second devices, the list showing a first metric and a measure of variability of the second metric for each of the first device and the second device. In a particular set of embodiments, this may include displaying a dashboard display in a user interface. The dashboard display may include a group display including a plurality of network devices (including the first device and the second device) that are determined to be peer devices. The group display may display the plurality of devices in an ordered manner and may include a measure of variability of the first metric and the second metric for each of the plurality of devices.

[0022] In one aspect, embodiments may include instructions for ranking a plurality of additional devices other than the first device and the second device according to the weighted volatility measurements of the first metric and the second metric of each of the plurality of additional devices. Embodiments may also include instructions for filtering a particular device from the ranked device list, for example, based on one or more characteristics that make the particular device not representative of the ranked device list.

[0023] Particular embodiments include instructions for identifying a network condition based on a weighted variability measurement of the first metric and the second metric of the first device and the second device. In some cases, the network condition may include a misconfiguration of an application at layer 7 of the OSI model. Some embodiments may include instructions for receiving feedback regarding the identified network condition and / or instructions for updating the first weighting factor and the second weighting factor based on the feedback.

[0024] Particular embodiments may include instructions for determining one or more actions to correct the identified network condition. Such embodiments may also include instructions for displaying to a user a recommendation of the one or more actions determined to correct the identified network condition and / or instructions for implementing the one or more actions determined to correct the identified network condition.

[0025] Another set of embodiments provides an apparatus. One such apparatus may include a non-transitory computer-readable medium having encoded thereon an instruction set executable by one or more processors. The instruction set may include one or more instructions, such as the instructions described above.

[0026] A set of other embodiments provide methods, including but not limited to methods comprising operations performed by a computer system in accordance with the above limitations. By way of example only, a method according to one set of embodiments may include receiving a first plurality of data points indicating a first metric in a computer network and a measurement of a second metric in the computer network, the plurality of data points including data points for a first device in the computer network and data points for a second device in the computer network.

[0027] In some embodiments, the method further includes calculating a variability measure of a first metric of a first device from at least some of the plurality of data points and / or calculating a variability measure of a first metric of a second device from at least some of the plurality of data points. The method may include calculating a variability measure of a second metric of a first device from at least some of the plurality of data points and / or calculating a variability measure of a second metric of a second device from at least some of the plurality of data points. On the one hand, the method may include applying a first weighting factor to weight the variability measure of the first metric of the first device and the second device and / or applying a second weighting factor to weight the variability measure of the second metric of the first device and the second device. On the other hand, the method may include sorting the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device. In some embodiments, the method includes displaying a list of sorted first devices and second devices to a user, the list showing the variability measures of the first metric and the second metric of each of the first device and the second device.

[0028] Exemplary Embodiments

[0029] Specific exemplary embodiments are described below. As those skilled in the art will appreciate, each of the described examples can be implemented alone or in any combination. Therefore, a single embodiment or combination of embodiments should not be considered limiting.

[0030] Figure 1 1 is a block diagram illustrating a computer network 100 (also referred to simply as a "network") according to various embodiments. The network 100 may include one or more monitoring computers 105, which may individually and / or collectively perform some or all of the operations described herein. In some embodiments, the monitoring computers 105 may include multiple computers that are centralized or distributed throughout the network. By way of example only, in some embodiments, the monitoring computers 105 may be operable to collect data points, analyze the data points, transmit the data points to a central monitoring area in the computers 105, and / or perform the methods and analyses described herein.

[0031] The network 100 may also include one or more network devices, including, but not limited to, routers, switches, hubs, gateways, access points, and / or the like. All of these network devices are examples of devices that can be monitored and from which data points can be collected. Alternatively and / or in addition, data points can be collected from various components of such devices, such as those described below. Figure 2A and 2B The components described in the context of FIG. 1 are collected. The illustrated network 100 includes, by way of example only and without limitation, a plurality of edge routers 110 and a plurality of core routers 115. In some embodiments, the edge routers 110 may be connected to other devices within the network 100, such as core routers 115c. Additionally, the edge routers 110 may be connected to one or more external resources 120. Such resources may include Internet points of presence 120a, 120b, Internet service providers 120c, 120d, cloud service providers 120e, 120f, and any other type of external resource 120.

[0032] The core router 115 may provide a connection between the edge router 110 and the rest of the network 100. In addition, the core routers 115 may provide connections between each other and resources within the network 100, and indeed, for example, the core routers 115a, 115b may provide connections to application servers 125a, 125b, respectively, which may be used for the same application, such as a business application, a web conferencing application, a VoIP application, and / or the like. Similarly, the core routers 115d, 115e may provide connections to application servers 125c, 125d, respectively, which may be used for another type of application, such as any of the applications described above. The core routers 115 may also provide communications with other internal resources, such as the subnetwork 130 and other types of internal resources.

[0033] Within the scope of various embodiments, network 100 may be any type of network or facility that provides communications between multiple devices (which may include network devices, servers, telecommunication devices, computers, etc.). In particular embodiments, network 100 may be an enterprise network, a content delivery network, a cloud service provider network, an Internet service provider network, a private virtual network, a virtual local area network (LAN), and / or the like, etc. Thus, Figure 1 The description of network 100 in provides a simplified overview for illustrative purposes only, and various embodiments may include many other types of network devices and / or terminal devices, such as end-user computers, servers, cell phones and tablets, VoIP phones, and / or the like. As described in further detail below, the monitoring computer 105 can monitor the operation of various network devices and other devices within the network, collect metric data about the devices, and / or analyze the variability of several different metrics based on the data. The monitoring computer 105 can detect various network conditions based on this analysis, display the analyzed data to the operator, and / or perform other functions described below. As used herein, the term "device" means any network device or other machine operating on a network whose metrics can be measured and / or various components of such a device whose metrics can be measured, such as processors, interfaces, and / or the like.

[0034] For example, Figure 2A is a block diagram illustrating an edge router 110 according to various embodiments, and Figure 2A A core router 115 is illustrated in accordance with various embodiments. Figure 2A The edge router 110 includes multiple interfaces 205, 210 (all of which need not be active at any given time). The router 110 itself may be a device in certain embodiments, and each individual interface 205, 210 on the router 110 may be a device. These interfaces may include one or more wide area network (WAN) interfaces 205 and one or more LAN interfaces 210. The term "interface" as used herein encompasses any component that provides a connection between a device (e.g., a router, a switch, a computer, etc.) and the network 100 (or any network, such as a WAN). By way of example only, the interfaces 205, 210 may include fiber optic interfaces, Ethernet interfaces, wireless (e.g., 5G, Wi-Fi) interfaces, and / or the like. A WAN interface may provide connectivity to a network 100 (or any network, such as a WAN). Figure 1 The LAN interface may provide connection to resources 120 external to the network 100, while the LAN interface may provide connection to resources within the network 100 (e.g., application servers 125, other routers 110, 115, subnets 130, monitoring computers 105, switches, bridges, and / or the like). Figure 1As illustrated, core router 110 provides connectivity only within network 100, so it includes LAN interface 210 (although it is possible that network 100 itself includes a WAN, in which case core router 110 may have a WAN interface (not shown) to enable intra-network wide area communications).

[0035] As discussed in more detail below, some embodiments of the present invention employ the concept of "peer" devices to analyze metric variability. In general, this concept recognizes that comparing two (or more) devices as peers can yield more actionable insights than comparing devices that are not peers. As discussed below, various embodiments may employ strategies to determine which devices should be considered peers. But at a very high level, peer devices may share one or more common characteristics that non-peer devices do not share. One example is that the role difference between two devices may provide some indication that the devices should not be considered peers.

[0036] By way of example only, comparing the variability of dropped packet metrics for a router and a switch (all other things being equal) may not provide deep insights simply because the roles these two devices play in a network are significantly different. A router operating at layer 3 of the OSI stack generally uses routing tables of IP addresses to route between different networks or subnetworks, while a (traditional) switch operates at layer 2 and forwards packets by port based on MAC addresses.

[0037] Similarly, although all routers 110 and 115 are routers, edge routers 110 and core routers 115 may play different roles in the network. Figure 1 As shown, edge router 110 may handle traffic entering and leaving network 100, while core router 115 may handle traffic only within network 100. Given the different roles, the difference between the volatility of the metrics of edge router 110 and core router 115 does not provide as much insight as the difference in the volatility of the two edge routers 110. Thus, in some embodiments, at any level of granularity, the role of a router (or other device) may be considered a characteristic of that device.

[0038] As another example, at a higher level of granularity, an edge router 110a connected to a POP 120a may have more in common with another edge router 110b connected to another POP 120b than with an edge router 110c connected to an ISP 120c or an edge router 110d connected to a CSP 120d (all other things being equal). Similarly, a WAN interface 205 on a given router 110a is more likely to be a peer of a WAN interface 205 on another router 110b than a peer of a LAN interface 210 on the same router 110a. Thus, the properties of the respective connections maintained by a device (and in particular, a router and its interfaces) may be considered characteristics of that device.

[0039] Thus, two edge routers 110 connected to the same CSP may have the common characteristic and may therefore be considered peers (given this characteristic alone or in combination with other characteristics). Likewise, core routers 115a, 115b providing connections to two application servers 125a, 125b for the same application (or, in some cases, the same type of application) may have the common characteristic and may therefore be more likely to be peers of each other than core router 115d providing connections to server 125c for a different application. And router 115d may be more likely to be a peer of router 115e than routers 115a, 115b, both of which are connected to servers 125c, 125d for the same application. Furthermore, all of these core routers may be more likely to be peers of each other than of core router 115f, which provides connectivity to subnetwork 120a, because all of these core routers handle application traffic directly to and from application server 125, while core router 115f handles traffic between subnetwork 130a and the remainder of network 100; this traffic may have different characteristics (e.g., a lower proportion of application traffic) than traffic typically carried by routers 115a, 115b, 115d, 115e. It may be appreciated from this example that the role of a device may have an impact on its characteristics and therefore which devices may be its peers. Furthermore, characteristics may be similar in degree (some embodiments do not require a binary decision on the similarity of characteristics), as demonstrated by the above example: routers 115a and 115b are most similar (in terms of the types of application traffic they handle), while 115a and 115c may share fewer common characteristics, and 125a and 125f may share yet fewer common characteristics.

[0040] But the role of a device is far from the only characteristic that can be considered. Various embodiments may consider any number of device characteristics. In general, the term "characteristic" as used herein may include any variable related to a device that may affect the function or performance of the device, all other things being equal.

[0041] For example, in some embodiments, the data plane 215 of each router 110, 115 (which handles the forwarding of packets and the mechanics of operating the network) is separated from the control plane 220 and / or the control plane 220 may use a software defined network configuration. In a particular embodiment, two routers (similar or different in role) with similarly configured control planes (which may specify, among other things, how the router makes traffic optimization decisions) may share enough characteristics to be considered peers. On the other hand, two similar routers with different traffic optimization settings may behave quite differently under normal conditions and therefore the settings may be considered characteristics of the routers. As another example, two devices controlled by the same brand of SDN software may share more characteristics than two devices controlled by different brands of SDN software.

[0042] In other cases, the various layers of the OSI stack may expose many different characteristics, including PHY (layer 1) characteristics (e.g., fiber optic vs. 1000Base-T vs. 100Base-T vs. Wi-Fi, etc.), layer 2 characteristics (e.g., virtual LAN (VLAN) configuration, spanning tree protocol configuration, etc.), layer 3 characteristics (e.g., routing protocols and configuration, packet inspection configuration, etc.), and the like. For example, a router's routing table may provide insight into whether a particular interface is a WAN interface, and the depth of the table may also be considered a characteristic for comparison. Other characteristics may include such things as: packet flow through the device, application traffic through the device, device brand and model, connection type, geographic and / or logical location, business services delivered, applicable service level agreements (SLAs), and the like. It should be understood from these examples that many other characteristics may also be considered within the scope of various embodiments when determining whether any two devices may be peers.

[0043] Figure 3 A method of measuring network variability according to some embodiments is described. Method 300 includes receiving a plurality of data points (block 305). The data points may be any data described herein that provides information (e.g., measurements) about one or more metrics to be analyzed. In some embodiments, the data points may include information identifying the metric measured, the device from which the metric was measured, and the like, but this is not always required. These data points may collectively reflect the measurement of several different metrics across many different devices. As used herein, the term "metric" means any statistical or operating parameter that can be measured and / or can indicate the performance level or operability of a device (particularly with respect to network performance or operability). There are many examples of metrics, which may include (but are not limited to) data about: throughput (incoming and / or outgoing packets together or individually), dropped packets, discarded packets, interface utilization (including incoming packets and / or outgoing packets together or individually), errors (including incoming packets and / or outgoing packets together or individually), delays, congestion, processor utilization, temperature, and the like.

[0044] Metrics can be measured at any appropriate layer of the OSI model. In many cases, metrics are measured at layers 1 to 3 of the model. Metrics can also be named using many different units of measurement. As mentioned below, evaluating the variability of the metric rather than just the raw data itself can provide a normalization that eliminates differences in measurement units. This can allow comparison and analysis of different metrics with different measurement units.

[0045] At block 310, the method includes associating a metric with a device based on the data point. By way of example only, each data point may include not only metric information (e.g., the number of packets discarded over a period of time), but also additional information, such as a time window within which the measurement was made, an identifier of the device from which the measurement was made (e.g., an IP address, a MAC address, etc.), and / or the like. From this information, the metric and the device from which the metric was taken may be identified and associated. The associated metric may then be stored (block 315) in, for example, one or more database tables or stored using any other suitable data structure. An exemplary table may include the following columns: a timestamp of the metric, an identifier of the metric (e.g., packets discarded), the metric itself (e.g., 10 discarded packets / second), and an identifier of the device (which may be an IP / MAC address or may be a name assigned to the device in a monitoring system, etc.).

[0046] The method 300 may further include calculating a variability measure of one or more metrics of the one or more devices (block 320). A variability measure may be any value that indicates a relative level of variability of a metric over time. By way of example only, one variability measure of a metric is the coefficient of variation, which is defined as follows:

[0047]

[0048] where c v is the coefficient of variation of the metric, σ is the standard deviation of the metric, and μ is the mean of the metric. In some cases, if the variability measure is a dimensionless number value (such as the coefficient of variation), it is very useful in this context. This means that it can be used to compare metrics with different dimensions (or units of measurement). Just by way of example, latency is typically measured as a unit of time, while incoming packets are measured as a rate (such as packets / second), which makes it difficult to directly compare the values. However, using the coefficient of variation (or another dimensionless measure of variability) for each metric allows for direct comparison of the metrics. In addition, as mentioned above, analyzing the variability of a metric can reveal insights that may not be obvious from the raw metric data itself. In some cases, the variability measures for each metric and / or device can be stored. In addition, as additional data points are received, the operations of blocks 305 to 320 can be repeated, with the variability measures for each metric-device pair being updated as more data is received.

[0049] At some point in time, older data points may be excluded from variability calculations due to age, either through the function of automated rules (e.g., no longer consider data points older than a month when calculating variability) or through triggers (e.g., input from an operator, detection of a configuration change to a device or some other network component that would affect the metric, etc.) Conversely, variability may be calculated using different time windows, which allows comparison of variability of metrics for the same device over different time periods, which may also reveal insights into network operations.

[0050] At block 325, method 300 includes applying a weighting factor to each of the metrics. The weighting factor may include any value that expresses the relative weight or emphasis that will be given to different metrics for analysis purposes. By way of example only, one weighting scheme will apply a value to each metric where the total of the values ​​is 1.0. Thus, for example, in a particular scenario, packet discards may be weighted by 0.25, packet drops may be weighted by 0.15, errors may be weighted by 0.10, incoming utilization may be measured by 0.25, and outgoing utilization may be weighted by 0.25, totaling 1.0. Other collected metrics may be weighted by 0.0 (i.e., ignored) and / or omitted from analysis altogether. The selection of the weighting factor to be applied to each metric may be based on many factors. For example, domain expertise may be used to set an initial weighting factor, which may be updated over time as more data points are collected and analyzed (e.g., as described below). On the one hand, the weighting factor may reflect how the relative predictive or diagnostic value of each particular metric is with respect to a specific network condition. More generally, analysis of different network conditions may weight the same metric differently and / or may use different metrics.

[0051] Method 300 may further include determining that two or more devices are peer devices (block 330). As mentioned above, a comparison of peer devices may provide greater insight than a comparison of non-similar devices. There are many ways to determine that a device is a peer device. For example, Figure 4 One method 400 of determining whether a device is a peer device is described in accordance with some embodiments.

[0052] Method 400 may include monitoring one or more devices (block 405). In some embodiments, monitoring may include interrogating the device to identify various characteristics of the device, including but not limited to characteristics such as those described above.

[0053] Method 400 may also include identifying peer characteristics between or among devices (block 410). As used herein, the term "peer characteristic" means any characteristic selected for evaluation to determine whether different devices should be classified as peers. At a high level, the nature of a device (bridge, switch, router, interface, etc.) and its physical and / or logical location in a network are characteristics of the device. Many other examples of characteristics are discussed above. In some embodiments, the selection of a peer characteristic may depend, at least in part, on the type of analysis to be performed.

[0054] In some cases, there may be no filtering for peer characteristics, and identifying peer characteristics may simply include identifying all characteristics for comparison on a set of devices that will generate monitoring and / or interrogation. In other cases, particular characteristics may be of greater interest than other characteristics, and the more interesting characteristics may be identified as peer characteristics. By way of example only, the type of device may be of greatest interest, based on the fact that comparing the metrics of two completely different devices (e.g., an unmanaged switch and an interface of an edge router) would not result in a very actionable analysis. In other cases, a particular portion of a network may be of interest, and thus the location (logical and / or physical) of the device may be of interest to exclude devices outside of the portion of the network being analyzed.

[0055] More generally, monitoring for different network conditions may change the relative importance of different characteristics, and the different characteristics may be identified as peer characteristics (i.e., characteristics of interest when determining whether a device is a peer for the purposes of the monitoring). For example, if there is a concern about network conditions primarily involving WAN traffic, then whether the router has a WAN interface and, if so, the external resources to which the interface is connected may be of greatest concern. In that case, the two characteristics may be identified as peer characteristics, and all other characteristics may be ignored. As another example, if an application appears to be causing network problems, then the peer characteristic may be the device communicating with the application server for that application.

[0056] At block 415, method 400 may include classifying two or more devices as peer devices. Classifying two or more devices as peers may involve any operation that determines that two or more devices are similar enough to provide a more executable comparison than if the devices were not peers. This classification may involve evaluating the identified peer characteristics to determine the degree of similarity and difference and dividing the device group into two or more subgroups, wherein all devices in the subgroups are classified as peers. Several techniques may be used to perform the classification. For example, if there is only one peer characteristic and it is binary (e.g., the device is or is not a WAN interface), then the classification may be a binary operation (if the peer characteristics match, then the device is a peer). In more complex situations, the technology may involve more. For example, if there are three identified peer characteristics, then the two devices may need to share all three common characteristics (i.e., the two devices are WAN interfaces; the two devices have connections to the same cloud service provider, and the two devices are managed by the same brand of SDN). In some cases, only two of the three characteristics may need to match. In particular embodiments, characteristics may be weighted in classifying peer devices, similar to how metrics may be weighted, for example, as described above.

[0057] Return to Figure 3 , method 300 may include ranking the devices. Ranking the devices may include any operation that arranging, prioritizing, or otherwise distinguishing between multiple devices on a numerical basis. In some cases, the devices may be ranked based on measured variability (e.g., according to the sum of weighted variability measures of their metrics). Returning to the weighted example above, the ranking may include the following operations: (1) for each device, add (variability measure of packet drops 5 0.25) + (variability measure of packet drops 5 0.15) + (variability measure of errors 5 0.10) + (variability measure of incoming utilization 5 0.25) + (variability measure of outgoing utilization 5 0.25) to arrive at a total weighted variability measure; (2) rank the devices by the total weighted variability measure of each device (e.g., from highest to lowest). If the devices have been classified into peer groups, then the ranking may be performed for all peer groups, for only some peer groups, or even for a single peer group (e.g., to compare the variability of a particular device with the variability of its peers). Each peer group may have its own ordered list of devices.

[0058] Furthermore, in some embodiments, the sorted list may be filtered, for example, to remove devices from the list that may not be suitable for inclusion in the list (block 340). Filtering may include any operation that distinguishes reliable data from unreliable data or, more generally, data that is to be used for analysis from data that is not to be used for analysis in a particular situation. For example, a particular device may be filtered from the list based on one or more characteristics that make the particular device not representative of the sorted list of devices (e.g., in embodiments without peer classification). In other cases, a device's measure of variability of one or more metrics may be too different from other devices to make the device suitable for sorting (e.g., if a device has too few packet drops such that the average packet drops approach zero, thereby driving the coefficient of variation too high to be of practical value). Different embodiments may employ various reasons for filtering devices from the sorted list.

[0059] At box 345, the illustrated method 300 includes displaying data, such as a sorted device list, to a user. In a particular set of embodiments, displaying the sorted list may include displaying a dashboard display in a user interface. In some embodiments, the dashboard display may include any user interface that includes multiple panels showing different information. Various panels may display various different analysis tools. For example, one panel of the dashboard display may include a sorted device list. Other panels may show more specific information about one or more of the sorted devices (e.g., top 1, 2, 3, etc.). Still other panels may show a list of network conditions, and the operator may select one of the network conditions from the list to change the display of the first panel to a sorted device list related to the network condition. Likewise, the sorted list of the first panel may display a variability measure of the metric for each of the analyzed metrics, and the operator may have the ability to select a column header to re-sort the list according to the particular metric. Likewise, the operator may have the ability to drill down into a particular device to view more metrics (e.g., expressed as raw data for each metric and / or variability measures for each metric, but with such details displayed in a different panel).

[0060] In some cases, method 300 includes identifying a network condition (block 350), for example, based on a weighted variability measurement of one or more metrics of various monitored devices (and, in a particular example, peer devices). In some embodiments, this identification may include reporting, displaying, notifying, or otherwise providing an indication of the network condition to another entity. A network condition may be any current, past, and / or expected future situation, configuration, performance, or service level, etc., involving a network, any portion of a network, or any device in a network. In some cases, a network condition may relate to and / or indicate suboptimal, unexpected, and / or unacceptable performance of a network, a portion thereof, or one or more devices therein. In some embodiments, a network condition may occur at or relate to one or more layers of the OSI stack. In some cases, a routing failure (layer 3) may constitute a network condition. In other cases, an underperforming or misconfigured application (layer 7) may constitute a network condition.

[0061] In many cases, network conditions cannot be easily detected from the metrics themselves. By way of example only, many monitoring systems will not flag a metric as problematic unless the metric reaches or exceeds an over-threshold time condition. (An over-threshold time condition is any condition in which a data point indicates a value of a metric that does not meet a threshold established for the metric within an established period of time.) In this case, the variability of the metric can provide an indication of a problem that cannot be seen by analyzing the raw metric alone. Similarly, some networks can cause normalization deviations in the metric. This condition also avoids detection by analyzing the raw metric data.

[0062] In still other cases, a single variability metric by itself may not indicate an alarming network condition, but the variability of multiple metrics (which may be weighted for analysis, as mentioned above) may indicate a problem, especially when compared to the weighted values ​​of peer devices. In some cases, a view of multiple devices ranked by weighted variability may indicate problems that cannot even be attributed to the network devices themselves. For example, a misconfiguration of an application (e.g., at layer 7) (e.g., a network application, a VoIP application, or any other application that produces an impact on network traffic) may result in abnormal variability in the metrics of a nearby device, which itself would not indicate such a misconfiguration when directly examined; such a condition may be diagnosed by comparing the device to other peer devices; if a nearby device ranks significantly ahead of peer devices in overall variability and no other metrics indicate a potential metric condition, then the configuration of the application may be investigated.

[0063] In some cases, method 300 includes receiving feedback about the identified network condition at block 355. Feedback may include, but is not limited to, any input to a process (and / or a machine executing the process, etc.) that conveys any information about the network condition and / or any actions taken to address the network condition. This feedback may be received from a human operator and / or from an automated process. By way of example only, after having investigated the identified network condition, the monitoring system may receive feedback indicating whether the selected metric and / or weighting factor has predictive and / or diagnostic value, such as, for example, whether the diagnosed condition actually exists (or existed when diagnosed). The system may adjust different parameters in its algorithm based on the feedback (block 360), such as the weighting factor of the selected metric, the selection of the metric itself, the device characteristics selected for peer classification, and / or the like.

[0064] In a particular embodiment, the method 300 may include determining one or more actions that may correct or improve the identified network condition (block 365). Such actions may include, but are not limited to, any configuration, rearrangement, adjustment, addition, subtraction, and the like of devices, applications, or other entities in the network and may include actions performed by computers and / or humans. For example, the system may infer that an application is misconfigured based on an evaluation of the weighted variability of the analyzed metrics, a ranked device list, and / or the like. The system may recommend one or more actions based on the inference, such as further investigation of the application, specific parameters for reconfiguring the application, and / or the like. In other cases, the system may recommend changing the configuration of one or more layer 1-3 parameters, such as changing routing optimization, control plane parameters, a brand of SDN software, and / or the like.

[0065] In some embodiments, these actions may be displayed to the user (block 370), such as recommending that one or more actions be performed to correct or improve the identified network condition and / or guiding how such actions should be performed. Depending on the circumstances, the system itself may be able to implement such actions (block 375), such as by accessing SDN software, control plane functionality, and / or the like using an API.

[0066] Figure 5 is a block diagram illustrating an example of a device 500 that can function as described herein, including but not limited to functioning as a network device or other device, a monitoring system, and / or the like according to various embodiments and / or performing some or all of the operations of the methods described herein. It should be noted that Figure 5 The components shown in the should not be considered necessary or required by every embodiment. For example, a device (such as a switch) in many embodiments may not include a processor and / or may be fully implemented in hardware or firmware circuitry. Similarly, many embodiments may not include input devices, output devices, and / or network interfaces.

[0067] Combined with the preamble, Figure 5 5, device 500 may include a bus 505. Bus 505 includes one or more components that enable wired and / or wireless communications among the components of device 500. Bus 505 may, for example, connect components of device 500 via operative coupling, communicative coupling, electronic coupling, and / or electrical coupling. Figure 5 Two or more components are coupled together.

[0068] The device 500 may also include a processor 510, which may include a central processing unit, a graphics processing unit, a microprocessor, a controller, a microcontroller, a digital signal processor (DSP), programmable logic (e.g., a field programmable gate array (FPGA), an erasable programmable logic device (EPLD) or the like), an application specific integrated circuit (ASIC), a system on a chip (SoC), and / or another type of processing component. The processor 510 may be implemented as hardware, firmware, or a combination of hardware, firmware, and / or software. In some embodiments, the processor 510 includes one or more processors that can be programmed to perform one or more operations or processes described elsewhere herein. For example, the device 500 may include a non-volatile storage device 515 for instructions (e.g., an operating system, an application, an object code, an assembly, a binary, etc.), which are used to program the processor 510 in this manner. In some cases, the storage device may be a hard disk drive according to various embodiments. This storage device 515 may be a non-volatile memory, a hard disk drive, etc. that stores instructions. In a particular set of embodiments, the instructions are stored as hardware instructions, such as encoded in hardware and / or firmware (such as hardware circuitry 550 discussed below).

[0069] In some embodiments, device 500 may include input components 520 and / or output components 525. Input components 520 may enable device 500 to receive input, such as user input and / or sensory input. For example, input components 520 may include a touch screen, a keyboard, a keypad, a mouse, a button, a microphone, a switch, a sensor, a global positioning system sensor, an accelerometer, a gyroscope, and / or an actuator. Output components 525 may enable device 500 to provide output, such as via a display, a speaker, and / or a light emitting diode.

[0070] Device 500 may also include a communication interface 560, which may enable device 500 to communicate with other devices via a wired (electrical and / or optical) connection and / or a wireless connection. For example, communication interface 560 may include a receiver, a transmitter, a transceiver, a modem, a network interface card, and / or an antenna. In some cases, communication interface 560 may include a plurality of I / O ports, each of which may be any facility that provides communication between device 500 and other devices, in particular embodiments, such ports may be network ports, such as Ethernet ports, fiber optic ports, and the like. Other embodiments may include different types of I / O ports, such as serial ports, pin-outs, and / or the like.

[0071] In some embodiments, the device 500 may include logic. This logic may be any kind of code, instructions, circuitry, or the like that may cause the device 500 to operate in accordance with the embodiments herein (e.g., to perform some or all of the processes and / or operations described herein). By way of example only, the device 500 may include a working memory 530 into which an operating system 535 and / or other instructions 540 may be loaded (e.g., from a storage device 515). (e.g., to perform the operations and methods described herein). In some implementations, the instructions 540 are executed by the one or more processors 510 to cause the one or more processors 510 and / or the device 500 to perform such operations.

[0072] Alternatively and / or in addition, logic may include circuitry 550 (e.g., hardware or firmware) that may operate independently of or in cooperation with any processor 510 that device 500 may or may not have. (As mentioned above, in some cases, circuitry 550 itself may be considered a processor 510.) In such cases, some or all of the logic that implements or causes the performance of some or all of the operations described herein may be encoded in hardware or firmware circuitry (e.g., circuitry 550) and executed directly by that circuitry rather than as software loaded into working memory 530. Thus, unless the context indicates otherwise, the embodiments described herein are not limited to any specific combination of hardware, firmware, and / or software.

[0073] in conclusion

[0074] In the above description, for the sake of explanation, many details are set forth to provide a thorough understanding of the described embodiments. However, it will be appreciated by those skilled in the art that other embodiments may be practiced without some of these details. In other examples, structures and devices are shown in block diagram form without complete details for the sake of clarity. Several embodiments are described herein, and although various features are attributed to different embodiments, it will be appreciated that the features described with respect to one embodiment may also be combined with other embodiments. However, for the same reason, a single feature or several features of any described embodiment should not be considered as necessary for each embodiment of the present invention, because other embodiments of the present invention may omit such features.

[0075] Thus, the above description provides an illustration and description of some features and aspects of various embodiments, but it is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Those skilled in the art will recognize that modifications may be made in light of the above disclosure or may be acquired from practice of the implementation, all of which may fall within the scope of the various embodiments. For example, as mentioned above, the methods and processes described herein may be implemented using software components, firmware, and / or hardware components (including but not limited to processors, other hardware circuitry, custom integrated circuits (ICs), programmable logic, etc.), and / or any combination thereof.

[0076] Furthermore, although the various methods and processes described herein may be described with respect to specific structural and / or functional components for ease of description, the methods provided by the various embodiments are not limited to any specific structural and / or functional architecture, but may instead be implemented in any suitable hardware configuration. Similarly, although some functionality is attributed to one or more system components, this functionality may be distributed throughout various other system components according to several embodiments unless the context dictates otherwise.

[0077] Likewise, although the procedures of the methods and processes described herein are described in a particular order for ease of description, various procedures may be rearranged, added, and / or omitted according to various embodiments unless the context dictates otherwise. In addition, procedures described with respect to one method or process may be incorporated into other described methods or processes; similarly, system components described with respect to a particular structural architecture and / or with respect to one system may be organized into alternative structural architectures and / or incorporated into other described systems. Therefore, although various embodiments are described as having or not having certain features for ease of description and for purposes of illustrating aspects of the embodiments, various components and / or features described herein with respect to specific embodiments may be substituted, added, and / or subtracted from other described embodiments unless the context dictates otherwise.

[0078] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software. It should be understood that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, and / or a combination of hardware and software. Unless expressly stated in the claims below, the actual specialized control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, when the operation and behavior of the system and / or method are described herein without reference to specific software code, it should be understood by those skilled in the art that software and hardware can be used to implement the system and / or method based on the description herein.

[0079] In the present disclosure, when an element is referred to herein as being "connected" or "coupled" to another element, it is understood that one element may be directly connected to another element or have an intervening element present between the elements. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, it is understood that there are no intervening elements in the "direct" connection between the elements. However, the presence of a direct connection does not exclude other connections in which there may be intervening elements. Similarly, although the methods and processes described herein may be described in a particular order for ease of description, it is understood that, unless the context dictates otherwise, intervening processes may occur before and / or after any portion of the described process, and as mentioned above, the described procedures may be rearranged, added, and / or omitted according to various embodiments.

[0080] In this application, unless otherwise specifically stated, the use of the singular includes the plural, and unless otherwise indicated, the use of the term "and" means "and / or". In addition, as used herein, unless otherwise explicitly stated (for example, if used in combination with "any one" or "only one"), the term "or" when used in a series is intended to be inclusive and may also be used interchangeably with "and / or". In addition, the use of the term "including" as well as other forms (such as "includes" and "included") should be considered non-exclusive. In addition, unless otherwise specifically stated, terms such as "element" or "component" cover both elements and components that include one unit and elements and components that include more than one unit. As used herein, the phrase "at least one of..." preceding a series of items using the terms "and" or "or" to separate any items modifies the entire list, rather than each member (i.e., each item) in the list. The phrase "at least one of..." does not require selection of at least one of each of the listed items; rather, the phrase allows for the meaning to include at least one of any of the items and / or at least one of any combination of items. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; and / or any combination of A, B, and C. In instances where selection of "at least one of each of A, B, and C" or, alternatively, "at least one of A, at least one of B, and at least one of C" is intended, it is expressly described as such.

[0081] Unless otherwise indicated, all numbers used to express quantity, size, etc. in this article should be understood to be modified by the term "about" in all examples. As used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more". Similarly, as used herein, the article "said" is intended to include one or more items referenced in conjunction with the article "said" and can be used interchangeably with "one or more". As used herein, the term "set" is intended to include one or more items (such as related items, unrelated items, combinations of related and unrelated items and / or the like) and can be used interchangeably with "one or more". When only one item is desired, the phrase "only one" or similar terms are used. In addition, as used herein, the term "has", "have", "having" or the like is intended to be an open term. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part". In the above description, depending on the context, satisfying a threshold may refer to a value being greater than a threshold, greater than or equal to a threshold, less than a threshold, less than or equal to a threshold, equal to a threshold, and / or the like, depending on the context.

[0082] Although specific combinations of features are set forth in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of the various embodiments. In fact, many of these features may be combined in ways not specifically cited in the claims and / or disclosed in the specification. Therefore, although each dependent claim listed below may directly depend on only one claim, the disclosure of the various embodiments includes each dependent claim in combination with every other claim in the claim set. Any element, action, or instruction used herein should not be interpreted as critical or essential unless explicitly described as such.

Claims

1. A system comprising: one or more processors; and A non-transitory computer-readable medium having encoded thereon a set of instructions executable by the one or more processors, the set of instructions comprising: receiving an instruction for a first plurality of data points indicating a measurement of a first metric in a computer network and a second metric in the computer network, the plurality of data points comprising data points for a first device in the computer network and data points for a second device in the computer network; instructions for calculating a variability measure of the first metric of the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the first metric of the second device from at least some of the plurality of data points; instructions for calculating a variability measure of the second metric of the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the second metric of the second device from at least some of the plurality of data points; instructions to apply a first weighting factor to weight the variability measures of the first metric of the first device and the second device; instructions to apply a second weighting factor to weight the variability measures of the second metric of the first device and the second device; instructions to rank the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device; and Instructions are provided to display to a user a list of the ordered first and second devices, the list showing the variability measures of the first and second metrics for each of the first and second devices.

2. The system of claim 1, wherein the variability measure is the coefficient of variation.

3. The system of claim 1 , wherein the instruction set further comprises: Instructions for identifying a network condition based on the weighted variability measurements of the first metric and the second metric of the first device and the second device.

4. The system of claim 3, wherein the metric is measured at a layer below layer 4 of an Open Systems Interconnection (OSI) model, and wherein the identified network condition occurs above layer 5 of the OSI model.

5. The system of claim 4, wherein the identified network condition is a misconfiguration of an application of layer 7 of the OSI model.

6. The system of claim 3, wherein the instruction set further comprises: determining one or more actions to correct the identified network condition; and Displaying to a user recommended instructions for one or more actions determined to correct the identified network condition; or Instructions are implemented to implement one or more actions determined to correct the identified network condition.

7. The system of claim 1, wherein the first device is a first network interface and the second device is a second network interface.

8. The system of claim 1, wherein the instruction set further comprises: Determining the first device and the second device from among a plurality of devices in a peer-to-peer network is an instruction of a peer device.

9. The system of claim 8, wherein determining that the first device and the second device are peer devices comprises: monitoring the first device and the second device; identifying one or more peer characteristics between the first and second devices; and The first device and the second device are classified as peer devices.

10. The system of claim 9, wherein the one or more similar peer characteristics include similar application traffic flows.

11. The system of claim 9, wherein the one or more peer-to-peer characteristics include a similar network arrangement with an entity external to the computer network.

12. The system of claim 9, wherein the one or more peer characteristics include one or more similar characteristics of a control plane of a software defined network (SDN).

13. The system of claim 8, wherein displaying the ordered list of first and second devices comprises displaying a dashboard display in a user interface, the dashboard display comprising a group display including a plurality of network devices determined to be peer devices, the plurality of network devices including the first device and the second device, the group display listing the plurality of devices in an ordered order, the listing including a variability measure of the first metric and the second metric for each of the plurality of devices.

14. The system of claim 1, wherein: The first metric is named after a first unit of measurement; The second metric is denominated in a second unit of measurement; and The first and second measurement units are different.

15. The system of claim 1, wherein the first metric comprises a measurement of one or more of: an incoming packet rate at an interface, an outgoing packet rate at an interface, an incoming utilization at an interface, an outgoing utilization at an interface, an incoming discard at an interface, an outgoing discard at an interface, or an error at an interface.

16. The system of claim 1, wherein none of the plurality of data points exceeds an over-threshold time condition of the first metric or the second metric, or the plurality of data points indicate a normalized deviation of the first metric or the second metric.

17. The system of claim 1, wherein the instruction set further comprises: receiving an instruction for feedback regarding the identified network condition; Instructions for updating the first weighting factor and the second weighting factor based on the feedback.

18. The system of claim 1, wherein the instruction set further comprises: instructions to rank a plurality of additional devices other than the first device and the second device according to the weighted variability measures of the first metric and the second metric for each of the plurality of additional devices; and Instructions to filter a particular device from the ordered list of devices based on one or more characteristics that cause the particular device to not be represented in the ordered list of devices.

19. An apparatus comprising: A non-transitory computer-readable medium having encoded thereon a set of instructions executable by one or more processors, the set of instructions comprising: receiving an instruction for a first plurality of data points indicating a measurement of a first metric in a computer network and a second metric in the computer network, the plurality of data points comprising data points for a first device in the computer network and data points for a second device in the computer network; instructions for calculating a variability measure of the first metric of the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the first metric of the second device from at least some of the plurality of data points; instructions for calculating a variability measure of the second metric of the first device from at least some of the plurality of data points; instructions for calculating a variability measure of the second metric of the second device from at least some of the plurality of data points; instructions to apply a first weighting factor to weight the variability measures of the first metric of the first device and the second device; instructions to apply a second weighting factor to weight the variability measures of the second metric of the first device and the second device; instructions to rank the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device; and Instructions are provided to display to a user a list of the ordered first and second devices, the list showing the variability measures of the first and second metrics for each of the first and second devices.

20. A method comprising: receiving, at a computer system, a first plurality of data points indicating measurements of a first metric in a computer network and a second metric in the computer network, the plurality of data points including data points of a first device in the computer network and data points of a second device in the computer network; calculating, using the computer system and from at least some of the plurality of data points, a measure of variability of the first metric of the first device; calculating a variability measure of the first metric of the second device from at least some of the plurality of data points; calculating, using the computer system and from at least some of the plurality of data points, a measure of variability of the second metric of the first device; calculating, using the computer system and from at least some of the plurality of data points, a measure of variability of the second metric of the second device; applying, using the computer system, a first weighting factor to weight the variability measures of the first metric of the first device and the second device; applying, using the computer system, a second weighting factor to weight the variability measures of the second metric of the first device and the second device; ranking the first device and the second device according to the weighted variability measures of the first metric and the second metric of the first device and the second device using the computer system; and A list of the ordered first and second devices is displayed to a user, the list showing the variability measures of the first and second metrics for each of the first and second devices.

Images