CONDUCTING NETWORK PERFORMANCE ANALYSES

Abstract

Method for performing a network performance analysis on a computer system (100) comprising a plurality of nodes (110) interconnected via a plurality of switches (124), the method comprising: through one of the nodes of the computer system, measuring the latencies of a plurality of packets transmitted by the node over a network (118), wherein the plurality of nodes and switches is divided into several groups (190), each group comprising one or more nodes and one or more switches; determining latency representations for a plurality of levels (126, 128, 130) of the network and three different types of communication routes, wherein the latency representations comprise the latency measurements, statistical representations of the latency measurements, and latency metrics derived from the latency measurements, wherein the plurality of levels of the network is an application level (126),comprising a library layer (128) and a hardware layer (130), wherein the three different types of communication routes comprise a first type of communication route between two nodes in the same group coupled to the same switch, a second type of communication route between two nodes in the same group coupled to different switches, and a third type of communication route between two nodes in different groups coupled to different switches, and wherein determining the latency metrics comprises determining a first average latency for the first type of communication routes and a second average latency for the second type of communication routes; comparing the determined latency representations with expected latency representations, wherein the expected latency representations are expected latencies,Expected statistical representations of latencies and / or expected latency metrics include; based on the comparison, determining a deviation between one of the expected latency representations and one of the determined latency representations; and based on the determined deviation, identifying a part of the network from the majority of communication routes as causing excessive latency.

Description

BACKGROUND

[0001] With the recent proliferation of high-performance computing (HPC) systems (e.g., exascale computing systems), it has become more important than ever to extract maximum performance from each system component (e.g., compute, memory, network, and storage) to ensure, for example, that real-world problems can be solved effectively. Maximizing system performance and collaboratively evolving software to extract maximum performance from all system components can be critical to the appropriate functioning of HPC systems. Ensuring that the performance commitments of HPC applications are met is also essential, which is an important part of exascale system deployment and adoption. Developing a detailed understanding of the performance achieved by each component of an HPC system is a first step toward maximizing system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various examples are described below with reference to the following figures. Fig. 1A is a block diagram of an example computer system in accordance with some implementations of the present disclosure. Fig. 1B is a block diagram of an example computer system in accordance with some implementations of the present disclosure, showing a first example communication. Fig. 1C is a block diagram of an example computer system in accordance with some implementations of the present disclosure and showing a second communication example. Fig. 1D is a block diagram of an example computer system according to some implementations of the present disclosure, showing a third example communication. Fig.2 is a schematic diagram showing example histograms that may be used to perform network performance analysis according to an example implementation of the present disclosure. Fig. 3 is a flowchart showing an illustrative method for performing a network performance analysis in accordance with an example implementation of the present disclosure. Fig. 4 is a block diagram illustrating an example computer system in accordance with some implementations of the present disclosure.

[0003] In the drawings, identical reference numbers indicate similar, but not necessarily identical, elements. Furthermore, the drawings contain examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. DETAILED DESCRIPTION

[0004] A multi-node computing system, e.g., a high-performance computing (HPC) system, comprises multiple computing nodes connected by a network. In such systems, the overall performance of the system depends in part on the performance of the network connecting the nodes. As the computing nodes in such systems become faster with each generation and the amount of data shared by the nodes increases, it is even more important that the network connecting the nodes is very powerful to enable maximum system performance and prevent the network from becoming a bottleneck in the system. However, a high-performance network of such an HPC system can consist of many different elements, both hardware and software, which can interact in different ways. For example, each of the nodes in the system can have a network interface element (e.g.,a network interface card (NIC)), applications that communicate over the network, and an interface library layer that facilitates communication between applications over the network. These may be referred to as different "layers" of the network (e.g., application layer, library layer, and hardware layer). In addition, the network may include various network devices external to the nodes (e.g., switches) that connect the nodes in a variety of topologies. For example, in some HPC systems, the nodes may be arranged in multiple groups (e.g., 32 nodes per group in some example systems), with each group including multiple switches (e.g., four switches in some example systems) to interconnect the nodes of the group. The switches in one group may also be connected to the switches in other groups to interconnect the groups.These external network devices are also part of the hardware layer of the network mentioned above. Because all of these elements are involved in communication across the network at different levels, it can be difficult to fully understand how each contributes to the overall performance of the network. In particular, it can be difficult for the system, or for a system administrator or user, to determine where in the network problems exist. For example, if the overall performance of the system is less than desired, it can be difficult to determine whether this is due to a problem in the network and, if so, whether there is a fault at the application layer, the library layer, or the hardware level.

[0005] Therefore, the present disclosure provides for performing an end-to-end network performance analysis, which may enable the identification of problem areas of the network by breaking down performance latency measurements based on various hardware and software components and comparing (e.g., by a processor) the latency measurements, statistical representations of the latency measurements, and / or metrics derived from the latency measurements or the statistical representations with expected latencies / statistical representations / metrics. Based on the comparison, it can be determined where (e.g., at which layer or layers of a network, for which communication paths, and / or for which communication types) higher than expected latencies occur. In some implementations, latency measurements at different network layers (e.g.,Application level, library level, and hardware level), for different types of communication routes and / or for different types of communication. Examples of these different communication routes, which can be used in dragonfly networks, for example, are routes between two nodes in the same group connected to the same switch (SGSS), routes between two nodes in the same group connected to different switches (SGDS), and routes between two nodes in different groups connected to different switches (DGDS), or other types of routes within the network. Examples of the different communication types are GET / read communication, PUT / write communication, or other types of communication. The latency measurements, their statistical representations (e.g., average, mode, median, histogram, etc.) and / or metrics derived from them can be compared with expected values. These expected values ​​can be specified, configurable, and / or determined dynamically (e.g., based on the measured latencies or the metrics derived from them).

[0006] Based on the comparisons, a deviation between the expected value and the measured latency (or statistical representation or derived metric) for at least one of the network levels, communication paths and / or communication types can be determined. Deviations can be detected, for example, if a difference between two latency measurements / statistical representations / metrics being compared exceeds a threshold or a certain percentage, if the latency measurements / statistical representations / metrics differ from each other by more than a certain number of standard deviations, or if there is another predetermined or dynamically determined deviation between them. In response to the detection of a deviation at a network level, communication path and / or communication type, that network level, path and / or communication type (e.g.by a processor) can be identified as potentially problematic, causing the excessive latency, and thus further troubleshooting can be focused on this potentially problematic part of the system. Thus, the examples disclosed here can enable improved performance in computing systems, such as HPC systems, by facilitating the identification (and subsequent correction) of network problems that may affect system performance. Such identification and correction can be performed by a processor of the system and / or output for analysis by a user.

[0007] For example, in one analysis mode, an average latency can be determined for a plurality of combinations of network levels, communication paths, and communication types. Intermediate metrics can then be determined from these average latencies. For example, an average latency value at one network level, communication path, and communication type can be subtracted from another average latency value at the same network level and the same communication type, but a different communication path. The resulting value can be a metric that identifies a difference in latency attributable to the difference in the communication path.The same type of metric can be determined for several different communication types at the same network layer, and the metrics can be compared with each other, using one of the derived metric values ​​as the "expected" value relative to the other derived metric, and any deviations between them can be determined. From the measured latencies, further metrics can be derived (in addition to or instead of the above example) and compared with each other, using some of the metrics as the expected value relative to others.

[0008] As another example, in a second analysis mode, a statistical distribution of the latency measurements for a particular network layer, communication route, and / or communication type may be determined, and this statistical distribution may be compared to an expected distribution to identify deviations. In some implementations, the statistical distribution comprises, for example, a histogram formed from the determined latency measurements for a particular network layer and communication type grouped into a plurality of bins. The histogram of the determined latency measurements may be compared to an expected histogram. Based on the comparison, a deviation between the expected distribution and the determined distribution may be determined.For example, the number and / or location of peaks in the distribution can be compared with the expected number and / or location of peaks in the expected distribution, and any deviations can be identified. Depending on the location of the deviation (e.g., which network layer, communication path, and / or communication type the deviating distribution corresponds to), one of the network layers, communication paths, and / or communication types can be identified as problematic (i.e., as a part of the network causing excessive latency).

[0009] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. However, it is expressly understood that the drawings are for purposes of illustration and description only. Although several examples are described in this document, modifications, adaptations, and other embodiments are possible. Accordingly, the following detailed description does not limit the disclosed examples. Instead, the proper scope of the disclosed examples may be defined by the appended claims.

[0010] Fig. 1A-1D are block diagrams of a computing system 100 configured to perform end-to-end network performance analysis according to some implementations of the present disclosure. The computing system 100 includes multiple nodes 110. In Fig. 1 shows two nodes, node A 110 and node B 110. In Fig. 1B-1D show twelve nodes 110. Those with normal technical knowledge will understand that the number of nodes in the Fig. 1A-1D is not intended to be limiting, and computing systems having any number of nodes may fall within the scope of the present disclosure. By way of example, and not as a limitation, each of the nodes 110 may include computing devices. As shown in Fig.1A, each node 110 includes a network interface element (e.g., a network interface card (NIC)) 114, 116 configured to connect the respective node 110 to the network 118. Node A 110 also includes an application layer 120 configured to include applications that communicate over the network 118 and a library layer 122 configured to facilitate communication of the applications over the network 118. (It should be understood that, although not shown with respect to Node B 110, Node B may also include an application layer and a library layer within implementations of the present disclosure.)

[0011] Computer system 100 also includes at least one switch 124 (in most systems, multiple switches 124) configured to connect Node A 110 and Node B 110. Network 118 includes the switch(es) 124, as well as the cables or other connections that run between the nodes and the switches 124. Network 118 may also be considered to include application layer 120, library layer 122, and NICs 114 and 116 for the purposes of network performance analysis, as these layers are all involved in the delivery of packets through network 118 and can have an impact on the performance of network 118.For example, in one implementation, system 100 is an HPC system (such as an HPE Cray EX system, an HPE Cray supercomputer, or an HPC cluster), switch 124 includes an HPC switch (such as an HPE Slingshot switch), NIC 114 includes an HPC NIC configured for use with switch 124 (such as a Cassini NIC), library layer 122 includes an interface configured for use with NIC 114 (such as a Cassini eXascale Interface (CXI)), and application layer 120 includes one or more HPC applications (such as a Symmetric Hierarchical Memory (SHMEM) application).

[0012] In some examples, the nodes may be arranged in multiple groups 190, where each group 190 includes multiple switches 124, as shown in Fig.1B-1D. For example, in one implementation, there are 32 nodes 110 and four switches 124 per group 190. System 100 may include any number of groups 190, any number of switches 124 per group 190, and any number of nodes 110 per switch 124 (this depends solely on the capabilities of the hardware involved and the desired topology and performance of system 100).

[0013] Each node 110 is connected to at least one of the switches 124 of its group 190, and the switches 124 of one group 190 are connected to the switches 124 of another group 190. Thus, if node A 110 and node B 110 belong to the same group and are connected to the same switch 124, their communication with each other can be classified here as SGSS. Fig.For example, Figure 1 shows such SGSS communication between node A:1 100 and node A:2 100, both connected to the same switch A:1 124 of the same group A 190. Such communications do not require switch hops (i.e., transmissions from one switch 124 to another switch 124) and are therefore expected to have relatively low latency.

[0014] If Node A 110 and Node B 110 belong to the same group and are connected to different switches 124 of this group, their communication with each other can be classified here as SGDS. Fig. For example, Figure 1 shows such SGDS communication between nodes A:1 100 and A:6 100, which are connected to switches A:1 124 and A:2 124, respectively, of the same group A 190. Such communications require a local switch hop (i.e., transmissions from one switch 124 to another switch 124 in the same group) and may therefore be associated with a somewhat higher latency.

[0015] If Node A 110 and Node B 110 belong to different groups and are connected to different switches 124 of their respective groups, their communication with each other can be classified here as DGDS. Fig.For example, Figure 1 shows such DGDS communication between nodes A:1 100 and A:6 100, which are connected to switches A:1 124 and B:1 124 of groups A 190 and B 190, respectively. Such communications require at least one global switch hop (i.e., transfers from one switch 124 to another switch 124 in another group 190) and are therefore expected to have the highest latency of the three routes described above. In particular, in some implementations, the different groups 190 may correspond to physically (and in some cases logically) separate locations, such as different chassis, different racks and / or rows of racks, or in some cases even different geographical locations (e.g., different data centers). As a result, the global switch hop, in some implementations, is expected to have a slightly higher latency than the intra-group switch hop.

[0016] The present disclosure provides a framework configured to measure performance based on packet latencies at different levels of the network, allowing the contribution of each network level to the overall achieved latency to be examined. Thus, in the computer system 100 of the Fig.1A-1D, the application layer 120 of Node A 110 represents the application layer 126 of the network 118, the library layer 122 of Node A 110 represents the library layer 128 of the network 118, and the combination of the NIC 114 of Node A 110, the switch 124, and the NIC 116 of Node B 110 represents the hardware layer 130 of the network 118. Each layer 126, 128, and 130 of the network 118 may include one or more tools or utilities associated with it to measure latency at that layer. For example, the application layer 126 may have a tool for measuring latency. Examples of such tools include microbenchmarks. The library layer 128 may have a tool for measuring latency at that layer. One example is the Cassini eXascale Interface (CXI) utility for use in the CXI library. The NIC can have hardware counters to measure precise latency at the hardware level 130.It is understood that some tools can measure latency directly, while others make measurements that require the application of a mathematical function to convert them into a latency measurement. All such measurements and / or conversions, and any combination thereof, are within the scope of this disclosure.

[0017] In some examples, after measuring the latencies, statistical representations of the latency measurements may be determined and / or other metrics may be derived from the latency measurements. The statistical representations may include averages, variances, distributions (e.g., histograms), or other ways of statistically summarizing or presenting the measured latency values. Metrics derived from the latency measurements may include values ​​derived from the latency measurements that quantitatively represent some aspect of system performance. For example, a metric may represent the application layer's contribution to the overall latency and may be derived from the measured latencies by subtracting the latency measured at the library layer from the latency measured at the application layer.A similar metric could be derived to represent the library-level contribution to the overall latency by subtracting the measured hardware-level latency from the measured library-level latency. These are just two examples of metrics, and any number of other metrics can be derived from the measured values, e.g., by addition or subtraction, or other mathematical manipulation of the measured values.

[0018] Once the latency measurements have been obtained and / or derived by the tools at the various network layers 126, 128, and 130, the measurements, statistical representations of the measurements, and / or metrics derived from the measurements or statistical representations (e.g., by a processor of the computer system 100) may be compared to expected latency measurements / statistical representations / metrics. By way of example and without limitation, expected latency measurements / statistical representations / metrics may include predetermined thresholds, ranges, or representations stored in a memory of the computer system 100 and may be based on specifications associated with the corresponding network layer. In some examples, the expected values ​​may be dynamically determined in addition to or instead of predetermined expected values, e.g.,by deriving the expected values ​​from the measurements themselves—for example, some measured values ​​or metrics derived therefrom may be used as expected values ​​for comparison with other measured values ​​or metrics, as explained in more detail below. Based on the comparison of the latency measurements / statistical representations / metrics with the expected latency measurements / representations / metrics, it may be determined (e.g., by a processor of computer system 100) where (e.g., at which layer(s) of a network, for which communication routes, and / or for which communication types) higher-than-expected latencies occur.

[0019] In one example, an average latency (i.e., an average latency for a plurality of packets routed over a particular network layer over a particular time interval or for a particular number of packets) can be determined for a plurality of combinations of network layers, communication paths, and communication types. Table 1 shows examples of such average latencies in the "Software Latency," "Library Latency," and "Hardware Latency" rows. Table 1 Measuring latency GET / Read (µs) PUT / Write (µs) SGSS SGDS SGSS SGDS Software latency 2.79 3.46 2.53 3.73 Library latency 2.62 3.33 2.02 2.63 Hardware latency 1.6 2.25 1.18 1.78 Change in software latency compared to SGSS 0.67 1.2 Change in library latency compared to SGSS 0.71 0.61 Change in hardware latency compared to SGSS 0.65 0.6 Additional software costs 0.17 0.13 0.51 1.1 Library overhead costs 1.02 1.08 0.84 0.85

[0020] Looking at the software latency line in the example shown, we can see that the average application layer latency for an SGSS communication route for a GET / Read communication is 2.79 µs, the average application layer latency for an SGDS communication route for a GET / Read communication is 3.46 µs, the average application layer latency for an SGSS communication route for a PUT / Write communication is 2.53 µs, and the average application layer latency for an SGDS communication route for a PUT / Write communication is 3.73 µs.Looking now at the Library Latency row, we see that the average library-level latency for an SGSS communication route for a GET / Read communication is 2.62 µs, the average library-level latency for an SGDS communication route for a GET / Read communication is 3.33 µs, the average library-level latency for an SGSS communication route for a PUT / Write communication is 2.02 µs, and the average library-level latency for an SGDS communication route for a PUT / Write communication is 2.63 µs. Finally, looking at the Hardware Latency line, we see that the average latency at the hardware level for an SGSS communication route for a GET / Read communication is 1.6 µs, while the average latency at the hardware level for an SGDS communication route for a GET / Read communication is 2.25 µs, the average hardware layer latency for an SGSS communication route for a PUT / Write communication is 1.18 µs, and the average hardware layer latency for an SGDS communication route for a PUT / Write communication is 1.78 µs.

[0021] Intermediate values ​​can then be determined from these average latencies. Such intermediate metrics are shown in the remaining rows of Table 1. For example, for a given network layer and communication type, an average latency value for one communication route can be subtracted from an average latency value for another communication route. The resulting value can be a metric indicating a difference in latency that is due to the difference in the communication route. This is shown in the shaded rows of Table 1. For example, the row Change in Software Latency w / r / t SGSS shows that the difference in application-level latency for a GET / read communication between an SGSS communication route and an SGDS communication route is 0.67 µs (i.e.3.46 µs - 2.79 µs) and that the difference in application-level latency for a PUT / write communication between an SGSS communication route and an SGDS communication route is 1.2 µs (i.e., 3.73 µs - 2.53 µs). Furthermore, the line "Change in Library Latency with SGSS" shows that the difference in library-level latency for a GET / read communication between an SGSS communication route and an SGDS communication route is 0.71 µs (i.e., 3.33 µs - 2.62 µs), and the difference in library-level latency for a PUT / write communication between an SGSS communication route and an SGDS communication route is 0.61 µs (i.e., 2.63 µs - 2.02 µs). Finally, the line "Change in Hardware Latency w / r / t SGSS" indicates that the difference in hardware-level latency for a GET / read communication between an SGSS communication route and an SGDS communication route is 0.65 µs (i.e.,2.25 µs - 1.6 µs) and the difference in hardware latency for a PUT / write communication between an SGSS communication route and an SGDS communication route is 0.60 µs (i.e., 1.78 µs - 1.18 µs). These metrics may be referred to here as SGSS-to-SGDS metrics for short, with the corresponding layer and / or communication type appended to identify a specific metric, e.g., a library SGSS-to-SGDS GET metric.

[0022] These derived intermediate metrics can be compared with each other, using one of the derived metric values ​​as the "expected" value relative to the other derived metrics, and identifying any discrepancies between them. In some implementations, the derived metric used as the "expected" value may be the one with the lowest latency. For example, the SGSS-to-SGDS metrics related to the difference between the SGDS and SGSS configurations at different measurement levels can be compared, yielding several conclusions. First, the SGSS-to-SGDS metrics for GET / Read communication are approximately the same (i.e., there is approximately the same latency difference (~0.65 µs) between the SGSS communication route and the SGDS communication route for GET / Read measurements at all three levels).Second, the SGSS-to-SGDS metrics for PUT / write communication are not all the same: There is an inconsistency between the SGSS-to-SGDS PUT metrics at the application layer and the SGSS-to-SGDS metrics at the other two layers (1.2 µs vs. 0.6 µs and 0.61). This suggests that the application layer may be a part of the network causing excessive latency, and that PUT / write communication, in particular, may be a potential cause of the application layer issue.

[0023] In some examples, system 100 may determine that a deviation exists between two measurements or metrics when they are more than a predetermined distance apart, as measured by actual values, percentages, number of standard deviations, or the like. Specifically, in some examples, a deviation may be defined as any difference of 20% or more between a value and the expected value. In other examples, a deviation may be defined as any difference of 40% or more between a value and the expected value. In other examples, a deviation may be defined as any difference of 50% or more between a value and the expected value. In other examples, a deviation may be defined as any difference that exceeds one standard deviation.In other examples, a deviation may be defined as any difference of 0.1 µs or more between a value and the expected value. In other examples, a deviation may be defined as any difference of 0.2 µs or more between a value and the expected value. In some examples, the above definitions of deviation may be combined, so that a deviation is determined when any of the definitions is met.

[0024] More specifically, in the example above, the SGSS-to-SGDS PUT metrics for the library and / or hardware layer are used as the "expected" value compared to the SGSS-to-SGDS PUT metric for the application layer, and a discrepancy is observed when comparing. Since this discrepancy occurs at the application layer, it indicates that there may be an issue at the application layer. In other words, additional latency is expected when switching from SGSS to SGDS due to the addition of an additional switch hop. This change is expected to result in a roughly uniform increase in latency across all layers (since adding a new switch hop should affect all layers equally), and this is precisely what is observed for the GET communications and two of the PUT communications: an increase of approximately 0.6 µs is observed for these metrics.However, for application-layer PUT communications, switching between SGSS and SGDS results in almost twice the additional latency (1.2 µs) expected. This doubling does not occur at the other layers, suggesting that this is likely a problem specific to the application layer. The fact that the same problem occurs not with application-layer GET communications but with application-layer PUT communications suggests that this problem is not specific to the application layer as a whole, but rather a problem specific to application-layer PUT communications (e.g., there could be a problem with the application layer's handling of PUT communications).

[0025] In some examples, system 100 is preconfigured to use certain metrics as "expected" values ​​compared to others. In other examples, system 100 (or in some cases, a user) may spontaneously determine which values ​​to use as "expected" values. For example, certain metrics may be grouped together on the grounds that they are expected to be similar (i.e., within a predetermined threshold). Such groupings may be derived from knowledge of the system, its network topology, the components used in the system, and other relevant data. For example, two communications expected to travel the same distance on the network may be expected to have similar latency, so latency measurements or metrics derived therefrom may be grouped together.Then, an “expected” value, used in comparison to each of the measurements / metrics in the group, can be determined as one of the values ​​(or a range of values) in the group, such as the lowest value in the group or the value that is most common or consistent across the group of measurements / metrics (e.g., t). In the example scenario above, the SGSS-to-SGDS metrics form a group of metrics that are expected to have similar values, and the most consistent value for the SGSS-to-SGDS metrics was around 0.6 µs, so this value could be determined as the “expected” value for comparison with all other SGSS-to-SGDS metrics. Another example: If two metrics are expected to be similar (e.g.,less than a predetermined distance, measured by actual values, percentages, number of standard deviations, or similar), but this is not the case, the lower value can be treated as the "expected" value for both metrics (e.g., because it represents the lowest latency, which is generally preferred). For example, in the example scenarios, the SGSS-to-SGDS metrics for application- and library-level PUT communications can be assumed to be similar, so the lower value—in this case, the SGSS-to-SGDS PUT metric for the library level—can be used as the "expected" value for comparison. In other examples, a user can specify which values ​​should be used as "expected" values ​​relative to the others.

[0026] It should be noted that the metrics and comparisons described with respect to the example scenario above are not the only comparisons that could be made in this scenario, and that the same insights and / or additional insights could be gained by comparing other metrics. For example, in some examples, additional metrics can be derived and compared. For example, the system can determine metrics that indicate the overhead attributable to the different network layers. These metrics are shown in the last two rows of Table 1 for the example scenario. Because the network layers build on each other, a measurement at each layer is an indicator of latency not only at that layer but also at the lower network layers. For example, a measurement at the application layer includes the latency for the application layer in addition to the latency at the library and hardware layers, as in Fig.1A. To obtain a latency measurement attributable only to a specific layer of the network, the latency measurement for the next lower layer in the communication stack can be subtracted from it (since the next lower layer includes not only the latency attributable to that layer, but also all other lower layers of the network). Thus, the application-level overhead for an SGSS communication route for a GET / read communication is 0.17 µs (i.e., 2.79 µs - 2.62 µs), the application-level overhead for an SGDS communication route for a GET / read communication is 0.13 µs (i.e., 3.46 µs - 3.33 µs), the application-level overhead for an SGSS communication route for a PUT / write communication is 0.51 µs (i.e., 2.53 µs - 2.02 µs), and the application-level overhead for an SGDS communication route for a PUT / write communication is 1.1 µs (3.73 µs - 2.63 µs).The library-level overhead for an SGSS communication route for a GET / read communication is 1.02 µs (i.e., 2.62 µs - 1.6 µs), the library-level overhead for an SGDS communication route for a GET / read communication is 1.08 µs (i.e., 3.33 µs - 2.25 µs), the library-level overhead for an SGSS communication route for a PUT / write communication is 0.84 µs (i.e., 2.02 µs - 1.18 µs), and the library-level overhead for an SGDS communication route for a PUT / write communication is 0.85 µs (i.e., 2.63 µs - 1.78 µs). The measured latency values ​​for the hardware layer also represent the overhead at the hardware layer, since the hardware layer is the lowest level of the network communication stack.

[0027] These overhead metrics can also be compared. In the example scenario above, comparing the overhead metrics confirms the observation that the part of the network causing excessive latency is likely the application layer for PUT / Write communication, since the overhead at this layer for SGDS communication routes and PUT / Write communication (i.e., 1.1 µs) is significantly higher (i.e., almost twice as high, which may indicate an unacceptable delay since all packets generally take the same path and thus have almost identical latencies) than the application-layer overhead for SGSS (i.e., 0.51 µs). Significantly higher here means that the amount exceeds the above-mentioned thresholds for detecting a deviation. In particular, the application overhead is expected to differ between GET and PUT communication, as they have different handling requirements.However, the application overhead is expected to remain unchanged when switching between SGSS and SGDS communication routes, as the route the communication takes should not affect the application overhead. Therefore, the application overhead metric for an SGSS communication can be used as an "expected" value compared to SGDS communication at the same level. When this is performed for GET communication, no deviation is observed (roughly the same overheads of 0.13 and 0.17 µs are found). For PUT communication, however, the application overhead for SGDS (1.1 µs) is much higher than the application overhead for SGSS (0.51 µs), and this deviation suggests that something is wrong with the handling of SGDS communication.Since this discrepancy between SGDS and SGSS overhead occurs only at the application layer and only for PUT communications, it suggests that the problem may lie in how the application layer handles PUT communications that traverse SGDS routes.

[0028] The above example was described with respect to a specific connection topology, where communication can have zero switch hops (SGSS), one intra-group switch hop (SGDS), or one global (inter-group) switch hop. However, in other examples, the connection topologies may differ. Another system topology, for example, is a so-called "fat-tree" topology. In such a fat-tree topology, communication may require 0, 2, 4, 6, or more switch hops, depending on the size of the system. In other examples, other connection topologies may also be used. Regardless of the connection topology used, a similar type of analysis to the one described above can be performed. For example, for any given connection topology, it can be determined what types of communication routes are to be expected, including, for example,their relative distances, the number and / or type of switch hops involved, and other similar factors, and based on this understanding, the latency changes between routes and / or the overhead attributable to each network layer can be determined.

[0029] The above example illustrates one way to identify problem areas of the network by performing an end-to-end network performance analysis by breaking down performance latency measurements based on various hardware and software components, comparing the latency measurements and / or metrics derived from the latency measurements with expected latency measurements / metrics, and based on the comparison, identifying where higher than expected latencies are occurring.

[0030] As another example, a statistical distribution of the latency measurements for a particular network level, communication route, and / or communication type may be determined, and this statistical distribution may be compared to an expected distribution to identify deviations. In some implementations, the statistical distribution comprises, for example, a histogram formed from the determined latency measurements for a particular network level, communication type, and communication path type (e.g., SGSS, SGDS, or DGDS), with the latencies grouped into a plurality of bins. This is Fig.2 for the hardware level (i.e., NIC latency) for GET / READ communications, using values ​​from a hypothetical scenario. Chart 206 shows histogram 210 for an SGSS communication path, chart 207 shows histogram 212 for an SGDS communication path, and chart 208 shows histogram 214 for a DGDS communication path.

[0031] The histogram of the determined latency measurements can be compared with an expected histogram. The expected histogram can be an actual statistical representation (not shown) or an expectation based on known data about the network in question. For example, in the illustrated implementation, it may be known at the hardware level that packet latencies are within a certain range (e.g., around 1600 nanoseconds) and that packets should take the shortest possible communication path. The expected latency can be known from hardware specifications or derived from measurements. If the expected latency and the number of peaks are known, expected histograms can be created that have peaks at the expected values.Note that this analysis compares the number and / or position of peaks, and therefore, the size of the peaks in the expected histograms can be arbitrarily set. A peak can be any bin in the histogram that has a count greater than a threshold, which can be a fixed value or a percentage (e.g., 10% or more of the total observed counts), both of which can be configurable or predefined. Fig. Figure 2 shows some example expected histograms 215, 216 and 217 in dashed lines for comparison with histograms 210, 212 and 214 respectively.

[0032] Based on the comparison, a deviation between the expected distribution and the determined distribution can be identified, and depending on where the deviation is located (e.g., which network level, communication route, and / or communication type the deviating distribution corresponds to), one of the network levels, communication routes, and / or communication types can be identified as problematic (i.e., as a part of the network that causes excessive latency). In the Fig.For example, in the scenario shown, the expected single-peak behavior can be seen for histograms 210 and 212, so that these histograms 210 and 212 match the expected histograms 215 and 216 (in terms of the number and location of the peaks). However, for histogram 214 (DGDS communication path), there are three peaks compared to the single peak in the expected histogram 217, and thus a deviation is observed. This deviation indicates that the packets take different routes with different lengths during DGDS GET communication. This contradicts the expected behavior and suggests that the hardware may be misconfigured or another problem is present. Accordingly, the hardware layer can be identified as a part of the network that causes excessive latency for DGDS communication routes. Accordingly, Fig.2 another way in which conducting a network performance analysis can help identify problem areas of the network.

[0033] The above example was described with respect to a specific interconnect topology, where communication can have zero switch hops (SGSS), one intra-group switch hop (SGDS), or one global (inter-group) switch hop. However, in other examples, interconnect topologies may vary. For example, in a fat-tree topology, communication may require 0, 2, 4, 6, or more switch hops, depending on the size of the system. Other interconnect topologies may also be used. Regardless of the interconnect topology used, a similar type of analysis to the one described above can be performed.For example, for any given connection topology, it is possible to determine what types of communication paths are to be expected, including, for example, their relative distances, the number and / or type of switch hops involved, and other similar factors, and based on this understanding, the number and / or location of peaks in the histogram can be predicted.

[0034] Fig. 3 is a flowchart illustrating an example method 300 for performing a network performance analysis according to an implementation of the present disclosure. The method 300 may be performed, for example, by the computer system 100 of Fig.1. In step 310, the latencies of a plurality of packets transmitted over a network may be measured. For example only, packet latencies may be measured using tools and / or utilities specific to an application, library, or hardware device of the computer system.

[0035] In step 312, latency representations may be determined for a plurality of layers of the network (e.g., an application layer, a library layer, and / or a hardware layer), for a plurality of communication routes (e.g., SGSS, SGDS, and / or DGDS), and / or for a plurality of communication types (e.g., GET / read and / or PUT / write communication). The latency representations may include the latency measurements themselves, statistical representations of the latency measurements (e.g., histograms), and / or latency metrics derived from the latency measurements (e.g., measurement differences that result in latencies attributable to a particular network layer, communication route, and / or communication type).

[0036] In step 314, the determined latency representations can be compared with expected latency representations. The expected latency representations can include expected latencies, expected statistical representations of latencies, and / or expected latency metrics. These expected latency representations can be predetermined, configurable, and / or dynamically determined. Based on the comparison, a deviation between one of the expected latency representations and one of the determined latency representations can be determined in step 316.

[0037] In step 318, based on the detected deviation, one of the multiple layers of the network, one of the plurality of communication routes, and / or one of the plurality of communication types may be determined as a portion of the network causing excessive latency. In some implementations, remedial actions may be taken to address and / or correct the deviation.

[0038] Fig. 4 is a block diagram illustrating an example of a computing device 400 according to some implementations of the present disclosure. In the Fig.4, the computing device 400 includes a processing resource 410 coupled to a non-transitory, computer-readable storage medium 412 encoded with computer-executable instructions to perform a system settings memory reset. The processing resource 410 may include a microcontroller, a microprocessor, central processing unit core(s), graphics processing unit core(s), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or other hardware device suitable for retrieving and / or executing instructions from the computer-readable storage medium 412 to perform the functions related to various examples described herein. Additionally or alternatively, the processing resource 410 may include electronic circuitry for implementing the functionality of the instructions described herein.

[0039] The computer-readable storage medium 412 may be any medium suitable for storing executable instructions. Non-limiting examples of a computer-readable storage medium 412 include RAM, ROM, EEPROM, flash memory, a hard disk drive, an optical disk, or the like. The computer-readable storage medium 412 may be disposed within the computing device 400, as shown in Fig. 4, in which case the executable instructions may be considered "installed" or "embedded" in the computing device 400. Alternatively, the computer-readable storage medium 412 may be a portable (e.g., external) storage medium and part of an "installation package." The instructions stored on the computer-readable storage medium 412 may be useful for implementing at least the methods described herein (e.g., the method described in Fig. 3 procedures).

[0040] In the context of the present example, computer-readable storage medium 412 is encoded with a series of executable instructions 414-424. It should be understood that some or all of the executable instructions and / or electronic circuitry included in one block may, in alternative implementations, be included in a different block illustrated in the figures or in a different block not illustrated.

[0041] Instructions 414, when executed, may cause processing resource 410 to measure (or otherwise obtain) the latencies of a plurality of packets transmitted over a network. For example only, packet latencies may be measured / obtained using tools and / or utilities specific to an application, library, or hardware device of the computer system.

[0042] Instructions 416, when executed, may cause processing resource 410 to determine latency representations for a plurality of network levels, for a plurality of communication paths, and / or for a plurality of communication types. The latency representations may include the latency measurements themselves, statistical representations of the latency measurements (e.g., histograms), and / or latency metrics derived from the latency measurements (e.g., measurement differences that result in latencies attributable to a particular network level, communication route, and / or communication type).

[0043] When executed, instructions 418 may cause processing resource 410 to compare the determined latency representations with expected latency representations. The expected latency representations may include expected latencies, expected statistical representations of latencies, and / or expected latency metrics.

[0044] The instructions 420, when executed, may cause the processing resource 410 to determine a deviation between one of the expected latency representations and one of the determined latency representations based on the comparison.

[0045] The instructions 422, when executed, may cause the processing resource 410 to identify one of the plurality of layers of the network, one of the plurality of communication routes, and / or one of the plurality of communication types as a portion of the network causing excessive latency based on the determined deviation.

[0046] Once executed, instructions 424 may cause processing resource 410 to perform troubleshooting or take remedial action focused on the identified portion causing excessive latency. For example, processing resource 410 may cause a notification to be sent instructing a user to change one or more configuration settings for the identified portion of the network causing excessive latency.

[0047] The information provided here with reference to the Fig.The processing described in Figures 1 to 4 may be in the form of executable instructions stored on a machine-readable medium and executed by a processing resource (e.g., a microcontroller, a microprocessor, one or more cores of a central processing unit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and the like) and / or in the form of other types of electronic circuits. This processing may be performed, for example, by one or more computing systems or nodes of various forms, such as the system described above with reference to Figures 1A-1D.

[0048] The technology described herein includes various steps, examples of which were described above. As described above, these steps may be performed by hardware components or embodied in machine-executable instructions that can be used to cause a processor programmed with the instructions to perform the steps. Alternatively, at least some steps may be performed by a combination of hardware, software, and / or firmware.

[0049] The technology described herein may be provided as a computer program product, which may comprise a tangible, machine-readable storage medium embodying instructions that can be used to program a computer (or other electronic device) to perform a process. The machine-readable medium may include, but is not limited to, hard disk drives, magnetic tapes, floppy disks, optical disks, compact disc read-only memories (CD-ROMs), magneto-optical disks, semiconductor memories such as ROMs, PROMs, random access memories (RAMs), programmable read-only memories (PROMs), erasable PROMs (EPROMs), electrically erasable PROMs (EE-PROMs), flash memory, magnetic or optical cards, or other types of media / machine-readable media suitable for storing electronic instructions (e.g., computer programming code, such as software or firmware).

[0050] In this technical description, numerous specific details are set forth in order to provide a thorough understanding of the example implementations. However, one skilled in the art will appreciate that the implementations described herein may be practiced without some of these specific details. In other instances, well-known structures and devices are depicted in block diagram form.

[0051] The terminology used herein is for the purpose of describing examples only and should not be considered limiting. The singular forms "a," "an," and "the" include the plural forms unless the context clearly indicates otherwise. As used herein, the term "plural" is defined as two or more than two. The term "and / or," as used herein, refers to and includes all possible combinations of one or more of the listed elements. As used herein, the term "includes" means "including, but not limited to," and the term "including" means "including, but not limited to." The term "based on" means based at least in part on.When the specification states that a component or feature "may," "can," "could," or "might," that particular component or feature need not be included or have that feature. As used in this specification, the meaning of "in" includes "in" and "on," unless the context clearly indicates otherwise.

[0052] Various methods described herein may be performed by combining one or more machine-readable storage media containing the code according to the example embodiments described herein with suitable standard computer hardware for executing the code contained therein. An apparatus for carrying out various embodiments described herein may include one or more computing elements or computers (or one or more processors within a single computer) and storage systems containing or network-accessible computer programs encoded in accordance with various methods described herein, and the method steps of various embodiments described herein may be performed by modules, routines, subprograms, or subparts of a computer program product.

[0053] In the foregoing description, numerous details are set forth to facilitate understanding of the subject matter disclosed herein. However, implementations may be made without some or all of these details. Other embodiments may include modifications and variations from the details described above. The following claims are intended to cover such modifications and variations.

Claims

[1] A method for performing a network performance analysis on a computer system comprising a plurality of interconnected nodes, the method comprising: by a node of the system, measuring the latencies of a plurality of packets transmitted by the node over a network; Determining latency representations for a plurality of layers of the network, for a plurality of communication routes, and / or for a plurality of communication types, wherein the latency representations comprise the latency measurements, statistical representations of the latency measurements, and / or latency metrics derived from the latency measurements; Comparing the determined latency representations with expected latency representations, wherein the expected latency representations comprise expected latencies, expected statistical representations of latencies, and / or expected latency metrics; based on the comparison, determining a deviation between one of the expected latency representations and one of the determined latency representations, and Based on the determined deviation, identifying one of the plurality of network layers, one of the plurality of communication routes, and / or one of the plurality of communication types as a part of the network causing excessive latency. [2] The method of claim 1, wherein the plurality of layers of the network comprise an application layer, a library layer, and a hardware layer. [3] The method of claim 1, wherein the latency representations comprise the latency metrics derived from the latency values. [4] The method of claim 3, wherein the latency metrics comprise: an average latency for a particular communication route and a particular communication type, a difference between an average latency for a first particular communication route and a first particular communication type and an average latency for a second particular communication route and the first particular communication type, and a latency attributable to a particular one of the plurality of layers of the network. [5] The method of claim 1, wherein the latency representations comprise statistical distributions of latency values. [6] The method of claim 5, wherein the statistical distributions of the latency values ​​comprise histograms. [7] The method of claim 5, wherein the statistical distributions correspond to respective communication routes from the plurality of communication routes. [8] The method of claim 7, wherein at least a portion of the plurality of communication routes comprises a same group and same switch route (SGSS), a same group and different switch route (SGDS), or a different group and different switch route (DGDS). [9] The method of claim 1, further comprising: troubleshooting or initiating remedial actions focused on the identified portion of the network causing excessive latency. [10] A non-transitory, computer-readable storage medium on which is stored a set of computer-executable instructions for causing one or more processors to: from a node of a computer system comprising a plurality of interconnected nodes, obtaining latency measurements of a plurality of packets transmitted by the node over a network; Determining latency representations for a plurality of layers of the network, for a plurality of communication routes, and / or for a plurality of communication types, wherein the latency representations comprise the latency measurements, statistical representations of the latency measurements, and / or latency metrics derived from the latency measurements; Comparing the determined latency representations with expected latency representations, wherein the expected latency representations comprise expected latencies, expected statistical representations of latencies, and / or expected latency metrics; based on the comparison, determining a deviation between one of the expected latency representations and one of the determined latency representations, and Based on the determined deviation, identifying one of the plurality of network layers, one of the plurality of communication routes, and / or one of the plurality of communication types as a part of the network causing excessive latency. [11] Computer-readable storage medium according to claim 10, where the latency representations include the latency metrics derived from the latency values, and wherein the latency metrics comprise: an average latency for a particular communication route and a particular communication type, a difference between an average latency for a first particular communication route and a first particular communication type and an average latency for a second particular communication route and the first particular communication type, and a latency attributable to a particular layer of the plurality of layers of the network. [12] The computer-readable storage medium of claim 10, wherein the latency representations comprise statistical distributions of latency values, and wherein the statistical distributions of latency values ​​comprise histograms. [13] The computer-readable storage medium according to claim 12, wherein the statistical distributions correspond to the respective communication routes from the plurality of communication routes. [14] The computer-readable storage medium of claim 13, wherein at least a portion of the plurality of communication routes comprises a same group and same switch (SGSS) route, a same group and different switch (SGDS) route, or a different group and different switch (DGDS) route. [15] The computer-readable storage medium of claim 13, wherein the set of computer-executable instructions further causes the one or more processors to: Troubleshoot or initiate remedial actions that focus on the identified part of the network causing excessive latency. [16] A system for performing a network performance analysis of a computer system comprising a plurality of interconnected nodes, the system comprising: one or more processors configured to: from a node of the computer system, obtaining latency measurements of a plurality of packets transmitted by the node over a network; Determining latency representations for a plurality of layers of the network, for a plurality of communication routes, and / or for a plurality of communication types, wherein the latency representations comprise the latency measurements, statistical representations of the latency measurements, and / or latency metrics derived from the latency measurements; Comparing the determined latency representations with expected latency representations, wherein the expected latency representations comprise expected latencies, expected statistical representations of latencies, and / or expected latency metrics; based on the comparison, determining a deviation between one of the expected latency representations and one of the determined latency representations, and Based on the determined deviation, identifying one of the plurality of network layers, one of the plurality of communication routes, and / or one of the plurality of communication types as a part of the network causing excessive latency. [17] System according to claim 16, where the latency representations include the latency metrics derived from the latency values, and wherein the latency metrics comprise: an average latency for a particular communication route and a particular communication type, a difference between an average latency for a first particular communication route and a first particular communication type and an average latency for a second particular communication route and the first particular communication type, and a latency attributable to a particular layer of the plurality of layers of the network. [18] The system of claim 16, wherein the latency representations comprise statistical distributions of latency values, and wherein the statistical distributions of latency values ​​comprise histograms. [19] The system of claim 18, wherein the statistical distributions correspond to the respective communication routes from the plurality of communication routes. [20] The system of claim 19, wherein at least a portion of the plurality of communication routes comprises a same group and same switch route (SGSS), a same group and different switch route (SGDS), or a different group and different switch route (DGDS).

Images