Intelligent performance analysis and anomaly detection for HPC and AI systems

Abstract

The present disclosure relates to intelligent performance analysis and anomaly detection for HPC and AI systems. Provided herein are techniques for performance analysis and anomaly detection for large complex systems. More specifically, the present disclosure uses tagging and correlation of metrics from different entities (switch, NIC, CPU, GPU, PCIe, memory) to provide the ability to identify events of interest in counter samples. The correlated events of the current techniques enable enhanced visual representations compared to previous visual representations that focus on identified events and timeline correlation of different metrics to understand performance changes of a job.

Description

Background Technology

[0001] This disclosure generally relates to techniques for performance analysis and anomaly detection in large, complex systems. More specifically, this disclosure provides the ability to identify events of interest in a sample of performance counters by using metrics that are labeled and correlated from different entities, such as network switches, network interface cards / controllers (NICs), central processing units (CPUs), graphics processing units (GPUs), high-speed peripheral interconnect buses (PCIe), and / or memory.

[0002] This section aims to introduce the reader to various aspects of the technology that may relate to the aspects of the present technology described and / or claimed below. It is believed that this discussion will help provide the reader with background information to better understand the various aspects of this disclosure. Therefore, it should be understood that these statements should be interpreted in this context, rather than as an admission of prior art.

[0003] A typical high-performance computing (HPC) and / or artificial intelligence (AI) system can have thousands of nodes consisting of CPUs and GPUs, along with high-speed network interfaces (HSN NICs). These nodes are connected via a high-performance network fabric using thousands of switches in network topologies such as fat trees or dragonflies. Large HPC jobs and their constituent applications can be quite complex and may require long periods of time (e.g., hours) to run on numerous such hosts.

[0004] Performance counters can be used to monitor and report on system components. Performance counters are hardware and / or software elements that monitor, count, and / or measure events within the hardware and / or software, thereby enabling performance analysis of the system's hardware and / or software. Using these performance counters, valuable information about the system and / or HPC and / or AI operations within the system can be collected.

[0005] Counter sampling on the host and the fabric is typically an independent activity, with host counter samples stored separately from fabric counter samples. Furthermore, these different counter samples often utilize different schemas and / or different persistent databases. Host counter sampling is initiated on the node executing a specific job. Fabric counter sampling (e.g., from a switch connecting the node) is typically generic for all jobs using the node. Attached Figure Description

[0006] These and other features, aspects, and advantages of this disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings, in which similar reference numerals denote similar parts, as shown in the drawings: Figure 1AIt is a diagram illustrating a system having multiple sets of different counters for performance analysis and anomaly detection according to various aspects of this disclosure; Figure 1B It is a diagram illustrating a system having a preprocessing engine and a postprocessing engine according to various aspects of this disclosure; Figure 2 It is a flowchart illustrating a process for efficiently analyzing counter samples according to various aspects of this disclosure; Figure 3A This is a diagram illustrating an example of marker insertion for a host indicator counter sample according to various aspects of this disclosure; Figure 3B It is a diagram illustrating an example of the insertion of markers in a sample of structural index counters according to various aspects of this disclosure; Figure 4 It is a flowchart illustrating a process for associating and visually presenting counter samples according to various aspects of this disclosure; Figure 5 It is a diagram illustrating example event data obtained by extracting markers according to various aspects of this disclosure; Figure 6A This is a diagram illustrating a system for associating counter samples across structural domains, host NIC domains, and host system domains according to various aspects of this disclosure; Figure 6B It is a diagram illustrating a cross-correlated counter sample generated by cross-correlation of counter samples according to various aspects of this disclosure; Figure 7 A graphical user interface (GUI) visualization of a visualization tool according to various aspects of this disclosure is described; Figure 8 A GUI visualization of a visualization tool according to various aspects of this disclosure is described, which provides a visual representation of associated events. Detailed Implementation

[0007] One or more specific aspects of this disclosure will be described below. To provide a concise description of these aspects, not all features of an actual implementation may be described in the specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions can be made to achieve the developer's specific objectives, such as compliance with system-related constraints and business-related constraints, which may vary from implementation to implementation. Furthermore, it should be understood that such development work may be complex and time-consuming, but remains a routine task of design, fabrication, and manufacturing for those skilled in the art who benefit from this disclosure.

[0008] When introducing elements of various aspects of this disclosure, the articles “a / an,” “the,” and “said” are intended to mean the presence of one or more of the elements. The terms “comprising / including” and “having” are intended to be inclusive and to mean that additional elements may exist in addition to the listed elements.

[0009] Additionally, as used herein, the terms “real time” or “substantially real time” are used interchangeably and are intended to describe operations performed without any human-perceptible interruptions between operations (e.g., computational operations). Furthermore, as used herein, the terms “automatic,” “automated,” “autonomous,” etc., are intended to describe operations performed and / or caused to be performed by a computing system (i.e., by a computing system without human intervention).

[0010] Performance analysis and anomaly detection for large systems (e.g., high-performance computing (HPC) and / or artificial intelligence (AI) systems) can be complex and challenging tasks. First, proactive monitoring of large systems is a challenge. For example, a system with over 5,000 switches may have up to 80,000 host endpoints and approximately 220,000 infrastructure endpoints requiring proactive monitoring for a given job. Second, it involves different types of metrics, such as infrastructure counters (e.g., monitoring network components like switches), a different set of NIC counters on a host, and a set of host system-level counters on a host. Raw data alone may be insufficient to determine the health of the infrastructure because the metrics provided by the various counters of the system are correlated and should be evaluated in combination. Therefore, performance analysis involves analyzing counters from all entities, as a failure or event in any one of them can lead to anomalies and changes. Real-time analysis of such large jobs is desirable so that any corrective actions can be taken to avoid wasting valuable time and resources.

[0011] However, providing effective analysis and scalability for large and complex environments can be challenging. For example, visualization tools may require significant processing and computational resources to load and analyze a large number (e.g., 80,000) of High Speed ​​Network (HSN) endpoints and establish timeline correlations. In fact, as system complexity increases, this scaling problem can worsen and face additional resource constraints (memory and CPU resources). Nevertheless, the increasing complexity of these systems underscores the need for enhanced analysis and visualization. Indeed, network errors can cause performance impacts, leading to application "stagnation." Enhanced capabilities to efficiently identify the root causes of stagnation and their impact can aid in prioritizing and mitigating system problems.

[0012] The present approach offers this efficiency improvement in system performance assessment and analysis. To this end, anomalies within each of the counter samples of individual system components are identified and (e.g., by inserting markers within and / or in association with the counter data) "marked." Time-based correlations of the marked portions are then provided, indicating regions of interest within each counter sample. This correlation of counter data enables efficient analysis and visualization, supporting efficient problem mitigation within the system. For example, in a large high-performance computing (HPC) facility, a high-performance LINPACK (HPL) job executing on 9000 nodes took several hours and exhibited "stagnation" ranging from seconds to minutes at different stages of execution. Leveraging the analytical and visualization efficiencies provided in this paper, the precise location of events of interest (i.e., the occurrence of counter sample events indicated in the counter samples and / or changes in the performance metrics of the counter samples exceeding associated event definition thresholds) significantly improves problem prioritization for mitigation.

[0013] Host and infrastructure counter sampling are typically independent activities, with host counter samples stored separately from infrastructure counter samples and using different schemas and / or different persistent databases. Host counter sampling (for both NICs and systems) is initiated on the node executing a specific job. Infrastructure counter sampling (from switches) is common to all jobs. Switch-side infrastructure counter sampling for edge ports may be related to the host that initiated the job. The techniques presented in this paper facilitate comprehensive analysis based on both job-related host and infrastructure counters. In this way, events indicated in infrastructure counter samples during a job (e.g., link failures, missing buffers in ports, invalid switch configurations, switch failures, correctable / uncorrectable errors) can be correlated with host counter samples to identify the correlation between performance variations or job failures and specific events. Therefore, the techniques presented in this paper can be used to analyze host system-level metrics (cache misses, interrupts, PCIe counters, correctable and uncorrectable errors) to understand (e.g., infrastructure-level) system-level impacts on jobs.

[0014] This disclosure generally relates to analyzing the performance of any system of any size and / or complexity and detecting anomalies. More specifically, this disclosure relates to providing the ability to identify events of interest in counter samples using markers introduced during the preprocessing stage. The markers are used to correlate metrics from different entities (switches, NICs, CPUs, GPUs, PCIe, memory) (e.g., a post-processing engine (PPE) downstream of counter sample acquisition). Correlated events from various counter samples can facilitate prioritization analysis, mitigation, and / or visualization by providing indications of cross-counter correlated events and / or regions of interest within the counter samples, which can provide priority areas where correlated events occur within the counter data. This proposed system is applicable to any large system with any network topology (e.g., dragonfly and / or fat tree).

[0015] Current technology provides a method for calculating metrics derived from raw counters. This enables visual representations to be achieved with a subset of desired metrics, even for structures with a large number (e.g., 300,000) of endpoints. The current technology achieves enhanced visual representations through correlated events, focusing on the timeline correlation between identified events of interest and different metrics to understand job performance variations.

[0016] With this in mind, Figure 1AThis diagram illustrates a system 100 that provides tag-based event associations across multiple different types of counter samples. HPC jobs can utilize multiple hosts connected via a structure. Therefore, a combination of performance information about these hosts, their underlying system performance, and the performance of the structure (e.g., network) coupling these hosts can be helpful in HPC job analysis. These jobs can run over extended periods (e.g., hours), and performance variations within the job should include analysis of structure (e.g., network) counters, as well as host NIC counters and host system counters, to understand the impact of each of these entities on the job.

[0017] To this end, the performance analysis and anomaly detection system 102 of system 100 is tasked with obtaining counter samples from multiple different counters, identifying and labeling anomalies within the counter samples, and cross-correlating the labeled anomalies across multiple samples. The cross-correlated anomalies can be used to provide enhanced analysis tools, such as a label-based graphical user interface (GUI) 104 that provides indications of labeled anomalies across multiple counter samples and / or indications of specific "regions of interest" that include portions of the counter samples that contain the cross-correlated labeled anomalies.

[0018] In the depicted system 100, the performance analysis and anomaly detection system 102 receives counter samples from three sets of performance counters: architecture counter 106, host-side network integration controller (NIC) counter 108, and host system counter 110, each of which is used in conjunction with HPC operations.

[0019] Structure counter 106 is a counter that tracks and provides performance metrics about network components such as switches. For example, these counters can provide performance metrics about each port of the switches in system 100.

[0020] Host-side NIC counters 108 provide host-level (e.g., per NIC for a host) performance metrics. These host-side NIC counters 108 can provide indications of host-specific high-speed network performance data.

[0021] System counter 110 provides host-level performance metrics specific to the host. For example, system counter 110 can provide performance data such as cache misses, interrupts, PCIe counters, correctable and uncorrectable memory errors, and other CPU and GPU events and counters.

[0022] The performance analysis and anomaly detection system 102 can identify, label, and cross-correlate anomalies from the counter samples received from the architecture counter 106, the host-side NIC counter 108, and the system counter 110. The labeled and cross-correlated anomalies are used to dynamically update the label-based GUI 104 to indicate the labeled anomalies and / or regions of interest, as will be discussed in more detail below.

[0023] To associate host-side counters 108 and 110 with structure counter 106, the preprocessing engine 122 of the performance analysis and anomaly detection system 102 provides preprocessing techniques introduced during metric acquisition. These preprocessing techniques process samples and use “tags” during the recording of these counters in the database. These “tags” signify priority events or periods where a counter exceeds a known good threshold. In some cases, a known good threshold (also referred to as an event threshold) can provide a value indicating a taggable event when one of the counter values ​​is breached. In some cases, these thresholds can indicate statistical outliers that, when breached by a certain counter value, indicate that the counter value is anomalous. In some cases, events in a counter sample can be identified without comparison to a specific threshold. For example, the presence of specific indicators in a counter sample (such as indications of timeouts and / or uncorrectable errors within the counter sample) can indicate events, making it possible to generate tags for these events as well. Events can be indicated in structure counter 106, host-side NIC counter 108, and / or host system counter 110. These tags are recorded in the database along with the sample data. These apply to both raw counters and counter differences (differences from previous samples).

[0024] A framework is provided to specify counters and corresponding thresholds representing events of interest used for preprocessing. This framework includes a template that specifies the counters and corresponding thresholds that should be used for record marking, and specifies the counters / metrics that signify priority events. In some cases, the threshold for a particular counter can be set by the user in a graphical user interface (GUI). In other cases, a specific sample of priority counters can be defined (e.g., in the GUI) to indicate the priority events to be highlighted when the indicated sample of counters exceeds its threshold.

[0025] The post-processing engine (PPE) 124 of the performance analysis and anomaly detection system 102 analyzes counter samples from hosts and components (e.g., switches) in the operation. PPE 124 is a general framework implemented as a dynamically pluggable interface to provide relevant analysis logic. PPE 124 analyzes host counters 108 and 110 and structure counter 106. The engine extracts markers from various timelines, which are used in conjunction with counters on the switch or host side.

[0026] PPE 124 uses these "tags" to correlate infrastructure-side metrics and host-side metrics, where the tag does not necessarily exist in both the host and infrastructure. PPE 124 uses both infrastructure-side and host-side metrics to further analyze anomaly detection and performance changes. For example, a timeout tagged on the host-side NIC counter 114 can lead to the following actions: processing the switch-side performance counters for the corresponding timeline to look for any infrastructure events near that time; processing NIC-specific events that occurred on other hosts involved in the job during that time period and could have caused the event; processing host-side performance metrics for all hosts in the job during that time period to generate an analysis of the performance impact; and processing infrastructure-side performance metrics to find the impact.

[0027] For example, PPE 124 can use flags to extract host timeout metric 114. Timeouts can be instances of packet loss and impacted job performance. Lost packets may be caused by architecture-level events, remote NIC problems, or system events (PCIe, DIMM, CPU, GPU).

[0028] For example, when a flag on a switch-side counter is associated with packet loss due to link latency failure or congestion (lack of processing buffer), PPE 124 will analyze any performance changes or timeouts on the host side during that time period. Switch-level congestion might be due to a lack of buffers on the switch's edge ports for holding frames. This situation can lead to packet loss, thus impacting application performance. Therefore, PPE 124 can also analyze host system counter 110 to obtain host events (PCIe, memory, GPU / CPU errors) 116 and related errors that have occurred during that time period.

[0029] Switch-level metric 112, indicating packet loss on different port architectures, may be caused by invalid routing states. This impacts job performance because lost packets need to be retransmitted. PPE 124 can be correlated with host-side metrics during this timeline to provide the desired impact analysis.

[0030] Correlating events and / or metrics across different counters facilitates root cause analysis for a variety of different scenarios. For example, PPE 124 can also detect high latency on the host side (e.g., for memory read / write, PCIe errors, or other errors), indicated by metrics derived from other indicators. Post-processing engine 124 can also interface with the structure manager (control plane) to obtain health events during that time period for analysis, such as... Figure 1BThis is discussed in more detail below. An additional example of analyzing timeouts or any performance changes is using tags on the host side to associate specific remote endpoints, thereby isolating the individual "rank" or "set of ranks" that are affected at that time.

[0031] For example, to associate host-side counters 108 and 110 with infrastructure-side counter 106 for events on both the host and infrastructure sides during the relevant time period, timeouts on the host side 114 are marked with tags recorded in the sample when recorded at times t1, t2, ..., tn. The post-processing engine 124 matches the time periods (t1, t2, and tn) of all the tags on the host side. This time period is used to analyze infrastructure-side counter 106 within the same time period. In another example, packet loss on the infrastructure side is marked with tags recorded when the event occurs (t1, t2, and tn). The post-processing engine 124 uses this time period and looks for performance anomalies on host-side counters 108 and 110. In yet another example, host-side system events (PCIe, memory, GPU / CPU errors) 116 are recorded as tags, and the corresponding host-side NIC counter 108 in the sample is analyzed to detect performance anomalies.

[0032] When visualization tools are unable to load large amounts of data to provide a visual representation due to a lack of resources, PPE 124 can help reduce the amount of data. For example, when visualization tools are unable to load large amounts of data due to a lack of resources, the post-processing engine 124 analyzes counter samples and provides smaller, more targeted derived metrics associated with events to understand job performance variations and / or failures.

[0033] PPE 124 can also use execution analysis to determine when host system-level metrics are associated with host NIC counter 108 and infrastructure counter 106. For example, sampling of host system-level events is performed in parallel with sampling of host NIC counter 108. These events are recorded in separate databases along with “tags.” PPE 124 extracts all time periods with “tags” and uses these time periods to analyze the counters on the relevant host and the jobs executed on that host.

[0034] The performance analysis and anomaly detection system 102 also provides real-time analysis for large HPC operations to implement corrective measures. For example, severely affected operations need to be corrected to avoid wasting valuable system time. Preprocessing using tags for high-impact events and thresholds (e.g., events of interest and / or thresholds of interest selected by the system or operator), along with correlations between switch-side metrics, host NIC metrics, and host system metrics, provides the ability to monitor high-impact anomalies in real time (e.g., via a tag-based GUI 104).

[0035] As can be understood, the performance analysis and anomaly detection system 102 can provide: automated network performance analysis capabilities for large systems; performance anomaly detection and operational failure analysis capabilities for network and related hardware; the ability to correlate host-side and infrastructure-side metrics and events in a highly automated manner; the ability to correlate system events (PCIe, memory problems) that affect operational performance; the ability to simplify visual representations using derived metrics for a large number of endpoint counts (e.g., via a tag-based GUI 104); and the ability to provide real-time analysis for large systems (e.g., via a tag-based GUI 104).

[0036] Figure 1B This is a diagram illustrating a system 120 having a preprocessing engine 122 and a postprocessing engine 124 according to various aspects of this disclosure.

[0037] The task of the preprocessing engine 122 is to receive metrics from (multiple) hosts / (multiple) nodes 126 and / or structural components such as (multiple) switches 128.

[0038] The (multiple) hosts / (multiple) nodes 126 may include a host NIC agent 130, which provides events and / or metrics 132 from (multiple) host system counters 134 and events and / or metrics 136 from (multiple) host NIC counters 138 to a preprocessing engine 122. These events and / or metrics may be accumulated by a host system metric collector 140 and a host NIC metric collector 142, and stored in a host system metric database 144 and a host NIC metric database 146, respectively. The host NIC counters 148 may generate events and / or metrics 146 associated with the host NIC agent 130, and the host system counters may generate events and / or metrics 132 associated with the hardware 148 of the (multiple) hosts / (multiple) nodes 126.

[0039] The (multiple) hosts / (multiple) nodes 126 also include an operating system (OS) 150, and any platform services / software development kits (SDKs) and / or drivers 152 associated with the OS 150.

[0040] Switch 128 includes a switch agent 154 that interacts with a fabric manager 156. Switch 128 also includes an operating system (OS) 158, and any drivers associated with the OS 158. Switch 128 further includes hardware 160, such as physical ports (e.g., edge ports and fabric ports), a set of processors, a set of memory devices, a set of persistent storage devices, a set of input / output (I / O) devices, etc. Switch agent 154 and OS 158 (and any associated drivers) are implemented using machine-readable instructions that can be executed on the set of processors in hardware 160.

[0041] Switch agent 154 includes telemetry agent 162, which is used to monitor the health status of switch 128. Telemetry agent 162 may include machine-readable instructions executable on hardware processing circuitry. The hardware processing circuitry may include any or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuitry.

[0042] Telemetry agent 162 collects events and / or metrics 164 of switch 128, which may be stored in data storage 166. Examples of metrics of switch 128 that may be monitored by telemetry agent 162 may include any one or a combination of the following: port or link error rate, port status (e.g., active, inactive, etc.), indication of congestion in switch 128 (e.g., queues in switch 128 have exceeded a threshold amount), port bandwidth utilization, number of data units dropped or lost by switch 128, etc.

[0043] In some examples, telemetry agent 162 may store the collected metrics in data storage 166 of switch 128 for local use by switch 128. Data storage 166 may be implemented using a collection of storage devices, such as disk-based storage devices, solid-state drives, memory devices, etc. Telemetry agent 162 may also transmit events and / or metrics 168 to preprocessing engine 122, more specifically to structured metrics collector 170, which may store events and / or metrics 168 in structured metrics database 172.

[0044] In some cases, telemetry agent 162 can generate events based on metrics sampled by telemetry agent 162. Events can also be based on hardware events, such as cable resets or other hardware events. Events are sent asynchronously when an event occurs. Telemetry agent 162 can store events and / or metrics 164 in data storage 166. An "event" refers to a notification generated when (multiple) specified conditions occur. An event indicating a problem with switch 128 or a link connected to switch 128 will generate an alarm 186. Therefore, alarm 186 is a special type of event with some indication that a problem has occurred (e.g., a flag in the alarm). For example, telemetry agent 162 can generate an event if the error rate exceeds a first threshold.

[0045] When a received event and / or metric exceeds an associated threshold defined in preprocessing template 174, preprocessing engine 122 can generate a tag, thus creating a taggable event. Therefore, when events and / or metrics 132, 136, and / or 168 are received by their respective collectors (host system metric collector 140, host NIC metric collector 142, and structural metric collector 170), these collectors can generate tags for the received events and can also compare them with the corresponding thresholds provided by preprocessing template 174, generating tags when a metric exceeds its corresponding threshold. The generated tags can be stored in the corresponding databases (e.g., host system metric database 144, host NIC metric database 146, and / or structural metric database 172).

[0046] The post-processing engine 124 may include a host system processing service 176 responsible for identifying tags in the host system metrics database 144, a host NIC processing service 178 responsible for identifying tags in the host NIC metrics database 146, and / or a structure processing service 180 responsible for identifying tags in the structure metrics database 172. Upon identification of a tag, the post-processing engine 124 may associate the tag with other metrics and / or the identified tag. For example, other metrics and / or tags occurring within a time window associated with the time corresponding to the tag may be associated with that tag. The associations performed by the post-processing engine 124 may be used to facilitate health monitoring and / or control of the system 120. For example, in some cases, the associations may be used to provide information on events and / or metrics occurring during a region of interest.

[0047] The post-processing engine 124 can also interface with the structure manager 156. The structure manager 156 provides a control plane 182 and structure management functions. The health service 184 of the structure manager 156 can provide additional events and / or metrics to the structure processing service 180 (e.g., based on alarms 186 provided to the health service 184 from the switch 128). When a root cause associated with one or more events and / or problematic metrics is identified, the control plane 182 can be used to adjust one or more characteristics of the structure, such as adjusting the configuration of structure components (e.g., the switch 128) (e.g., changing the routing table).

[0048] Figure 2 This is a flowchart illustrating a process 200 for providing efficient analysis of counter samples according to various aspects of this disclosure. Process 200 begins by receiving multiple counter samples, each providing a performance metric for a corresponding entity (box 202). Overall job performance analysis involves analyzing different counters from all entities, such as counters on switches, counters on NICs on hosts, and a set of system-level counters on hosts with different types of metrics. Since host-side counters and infrastructure-side counters represent different components, the patterns used for these counters often differ. In some cases, to accumulate host-side and infrastructure-side counters, counter samples from different counters can be accumulated into a common data store with a common pattern during metric acquisition.

[0049] For each of the received counter samples, an event is identified (box 204). For example, an event can be identified when a performance metric exceeds an event threshold (e.g., exceeding a maximum associated threshold and / or falling below a minimum associated threshold). The event threshold indicating when a counter sample value indicates an event may vary from counter to counter. For example, a first event threshold may be used to indicate when a counter sample value associated with a first specific measurement counter value indicates a first event, and a second event threshold may be used to indicate when a counter sample value associated with a second specific measurement counter value indicates a second event. Event thresholds may include, for example, performance metric value thresholds, performance metric rate of change thresholds (e.g., the rate at which a performance metric changes and / or the magnitude of a performance metric change over a given time period), and / or occurrence thresholds (e.g., the specific number of times a monitored performance activity (e.g., timeout or packet loss) occurs (e.g., 1 time or 10 times)). Templates may specify the counters and corresponding thresholds that should be used to identify events.

[0050] In some cases, events can be identified from the counter sample event indications observed within the counter sample. For example, timeout events and / or events associated with uncorrectable errors can be indicated as events within the counter sample.

[0051] For each of the received counter samples, within the corresponding counter sample, a tag is associated with each of one or more events (box 206). These "tags" signify a priority event or period in which the counter exceeds an event threshold (e.g., a known good threshold). The counter sample value can be compared to the associated threshold to identify the event, and / or the event in the counter sample can be indicated without comparing the counter sample value to the threshold (e.g., when a timeout occurs and / or when the counter sample indicates another event (e.g., an uncorrectable error exists)). In some cases, these tags are recorded along with the sample data in a data store storing the counter samples. Tags can be applied to the original counter sample as well as the counter sample difference (difference from previous samples). Events can be positive and / or negative. For example, a positive event may occur when a performance metric exceeds a specific performance metric indicated by a positive baseline threshold. A negative event may occur when a performance metric falls below a specific performance metric indicated by a negative baseline threshold.

[0052] Regions within all counter samples can be identified based on markers (box 208). For example, "healthy regions" can be identified within time windows where no negative events occur and therefore no markers are found / associated with them, and / or where positive events occur and therefore positive markers are present. "Unhealthy regions" can be identified within time windows where negative events do occur and therefore markers associated with negative events are found / associated with them. Because markers are associated with counter sample data, regions of interest can be efficiently identified within counter samples by identifying "regions of interest" based on time windows surrounding associated / inserted markers.

[0053] A graphical user interface (GUI) is generated and provided, offering visual representations of multiple counter samples, indications of one or more events using markers associated with those events, and indications of regions of interest (e.g., "unhealthy regions" and / or "healthy regions") (box 210). Visual representations of performance metrics, their corresponding events, and corresponding regions with other counter performance metrics aid in troubleshooting the causes of specific events (e.g., stagnation). Therefore, process 200 enables efficient visual representation and timeline filtering, allowing for precise targeting of specific time windows of interest during the visualization of counter samples. Enhanced visualizations can thus be provided, highlighting specific portions of the massive counter sample data that may require focused attention. Process 200 can further improve processing efficiency by reducing the amount of counter sample data loaded by the visualization tool, specifically focusing on loading portions within the counter samples where markers and / or regions of interest appear, while avoiding loading other, less relevant portions of the counter sample data. This can result in a significant reduction in processing resource utilization, freeing up those resources for other tasks.

[0054] Figure 3A Figure 300 illustrates an example of flag insertion for host metric counter samples (e.g., host NIC and / or host system counter samples) according to various aspects of this disclosure. Specifically, the illustrated example illustrates flag insertion when a timeout has been detected and one of the metrics has exceeded an acceptable threshold, according to various aspects of this disclosure. At time 1 (illustrated by box 302), multiple counter samples 304-310 are captured. Counter sample 308 is a timeout indicator indicating that no timeout has occurred.

[0055] At time 3 (illustrated by box 312), updated counter samples 314-320 are captured, where counter sample 318 provides an update to counter sample 308, indicating that a timeout has occurred. The event threshold of the timeout counter sample (e.g., counter sample 318) can indicate the event whenever a timeout occurs. Based on the timeout indication in counter sample 318, a tag 322 indicating an anomaly in counter sample 318 can be associated with counter sample 318 because the timeout indicated by counter sample 318 exceeded the threshold. This tag association can persist in the accumulated counter sample data until no more anomalies exist. In this way, when an anomaly exists within a counter sample, the associated tag can provide an indication of a specific time interval.

[0056] At time m (illustrated by box 324), additional updated counter samples 326-332 are collected. As illustrated by counter sample 328, an anomaly (e.g., the metric exceeds a threshold) has occurred. Based on the identification of this anomaly, marker 334 is associated with counter sample 328 indicating the anomaly. Furthermore, counter data 330 no longer indicates a timeout. Accordingly, the association between marker 322 and counter data 330 is removed.

[0057] Markers 322 and 334 provide indications of specific counter samples of interest in the performance analysis. A graphical user interface can provide enhanced counter sample visualization, highlighting specific counter sample 318 at time 3 and counter sample 328 at time m. Furthermore, markers 322 and 334 can be used to precisely locate additional portions of other counter samples to analyze anomalies and / or perform cross-correlation, thereby enabling the identification of regions of interest.

[0058] Figure 3BFigure 350 illustrates an example of marker insertion for a structural indicator counter sample according to various aspects of this disclosure. Specifically, the illustrated example illustrates marker insertion according to various aspects of this disclosure when a first event and a second event have been detected (e.g., when one of the indicators has exceeded an acceptable threshold). At time 1 (illustrated by box 302), multiple counter samples 352-358 are captured.

[0059] At time 2 (illustrated by box 360), updated counter samples 362-368 are captured, where counter sample 366 provides an update to counter sample 356, indicating an event (e.g., exceeding a preset threshold for counter sample 366). Based on this identified event, a tag 370 indicating an event / anomaly in counter sample 366 can be associated with counter sample 366 because the timeout indicated by counter sample 318 exceeded the threshold. This tag association can persist in the accumulated counter sample data until no more anomalies are found. In this way, when an anomaly exists within a counter sample, the associated tag can provide an indication of a specific time interval.

[0060] At time m (illustrated by box 324), additional updated counter samples 372-378 are captured. As illustrated by counter sample 374, an event / anomaly (e.g., an indicator exceeding a threshold) has occurred. Based on the identification of this anomaly, tag 380 is associated with counter sample 374 indicating the event and / or anomaly.

[0061] Markers 370 and 380 provide indications of specific regions of interest (e.g., time windows) in performance analysis. A graphical user interface can provide enhanced counter sample visualizations that highlight specific counter sample 366 at time 2 and counter sample 374 at time m. Furthermore, these markers 370 and 380 can be used to precisely locate additional portions of other counter samples to analyze anomalies and / or perform cross-correlation, thereby enabling the identification of regions of interest.

[0062] Figure 4 This is a flowchart illustrating a process 400 for associating and visually presenting a counter sample according to various aspects of this disclosure. Process 400 begins by receiving event markers (e.g., both host-side and structure-side markers), as shown in... Figure 3A and Figure 3BThe example provided shows the generated event markers (box 402). Event markers indicate events of interest within a counter sample (e.g., a counter sample value exceeding a threshold associated with the counter sample type). When markers are directly inserted into the counter sample data, they can be received by extracting them from the counter sample data. When markers are stored elsewhere (e.g., in a database), they can be retrieved from their storage location. This is described in more detail below. Figure 5 The diagram shows from Figure 3A and Figure 3B The example counter sample provided in the sample contains examples of tags extracted from it.

[0063] At box 404, counter samples with associated event tags are cross-correlated with other counter samples. Thus, for example, the received multiple counter samples include infrastructure counters and host counters. Infrastructure counter acquisition at the switch-side edge port is associated with the host that initiated the job. Host counter samples with associated tags can be cross-correlated with infrastructure counter samples, and infrastructure counter samples with associated event tags can be cross-correlated with host samples. Therefore, events occurring in the infrastructure counter samples during the job (e.g., events indicated in the infrastructure counter samples and / or identified based on metric values ​​exceeding thresholds) (e.g., link failure, missing buffer in a port, invalid switch configuration, switch failure, correctable / uncorrectable error) can be correlated with host counter samples to identify the association between performance changes or job failures and specific events. Correlation of all metrics during this period helps to understand the cause of timeouts.

[0064] Cross-correlation between counter samples can involve identifying metrics of various counter samples with respect to a specific common time. For example, when associating host counter sample events with structure counter samples, cross-correlation may include identifying structure counter sample values ​​that occur at the common time in which the host counter sample events occur (e.g., as indicated by the associated markers in the host counter samples). In some cases, cross-correlation may include identifying events that are cross-correlated between counter samples, which could include identifying a subset of other counter samples that occur at the common time and / or time window (e.g., as indicated by markers in the subsets of other counter samples). Therefore, in this case, cross-correlation between host counter sample events and structure counter samples would include identifying a subset of structure counter samples that includes events that occur at the common time and / or time window in which the host counter sample events occur (e.g., as indicated by the associated markers). This is discussed in more detail below. Figure 6A and Figure 6B A diagram of the associated counter samples is provided.

[0065] At box 406, a visual representation can be presented based on multiple cross-linked counter samples (e.g., via a graphical user interface (GUI)). For example, a region of interest around an event / marker can be used to filter portions of the counter samples for processing and / or presentation. Therefore, the visualization of counter samples can be filtered to prioritize regions of interest and linked counter samples. This can result in presenting less but more meaningful data via counter sample visualization tools, thus reducing the overload problem that arises when processing and / or visualizing massive amounts of available counter sample data.

[0066] Visual representations can be derived metric data representations that display metrics related to the job due to performance variations or job failures. A graphical user interface (GUI) indicates a simplified visual representation of marked anomalies from multiple counter samples associated with different entities. The simplified visual representation includes derived metrics and provides real-time analysis of job performance. Visual representations of relevant priority data facilitate the implementation of any corrective actions without wasting valuable time and resources.

[0067] Figure 5 The diagram illustrates the process of extracting from... Figure 3A and Figure 3B Example event data 500 is obtained by using the tags provided in the example. As shown in the figure, tag extraction provides event data 500 for the time when the tags are generated / associated with the counter sample data. Return to reference Figure 3A Host-side markers 322 and 334 are generated at time 3 (illustrated by box 312) and time m (illustrated by box 324), respectively, and / or associated with counter samples 318 and 328. Return to Reference Figure 3B Structural markers 370 and 380 are generated / associated with counter samples 366 and 374 at time 2 (illustrated by box 360) and time m (illustrated by box 324), respectively. Therefore, marker extraction includes event data 500 associated with each of these times. As shown, event data 500 may include entity data 504, which may specify the system (e.g., Figure 1A Specific components in system 100. Entity data 504 can specify, for example, a specific switch, NIC, CPU, GPU, PCIe, or system memory that the corresponding counter is tracking.

[0068] Event data 500 may also include the time 506 of the event data, which stores the time when the event occurred. This time 506 of the event data can represent, for example, a specific time when the event has occurred (such as a timeout or one of the metrics exceeding an event threshold).

[0069] Event data 500 may include event category data 508 for grouping events into broader categories. For example, event category data 508 may indicate whether an event belongs to the host side (e.g., system events and / or host events) and / or the structure side (e.g., structure events). Events may have different durations. Some events may be relatively transient (e.g., on the order of milliseconds), while others may have longer durations. Event categories may be used to determine the length of the time window used to define the corresponding region and / or associate other counter samples.

[0070] Event type data 510 describes the type of event that occurred. For example, event type data 510 can indicate a timeout event or other types of events. Event IDs, such as numbers specifying event types, can be used. For example, one event ID can indicate the timeout type of the event, while a second event ID can indicate that a specific type of metric exceeded a specific corresponding threshold.

[0071] Event information 512 is data associated with an entity during an event. For example, event information 512 may include recorded information (such as observed performance metric values ​​that actually led to the event).

[0072] The event data 500, which indicates the presence of events in the counter sample, can be used to correlate the counter sample. (About...) Figure 6A and Figure 6B This is illustrated in more detail.

[0073] Figure 6A This diagram illustrates a system 600 that correlates counter samples across architecture domains, host NIC domains, and host system domains; specifically, it correlates events across domains. As shown, system 600 includes a performance analysis and anomaly detection system 602 (e.g., performance analysis and anomaly detection system 102 of Figure 1) by correlating “tags” from host-side counters and architecture-side counters. In performance analysis and anomaly detection system 602, data from host counter sample 604, architecture counter sample 606, and host NIC counter sample 608 are analyzed to identify tags based on breach values ​​relative to thresholds set for specific counter types.

[0074] In the current example, at time 3 (illustrated by box 610), since host counter sample 614 indicates a threshold breach (e.g., a timeout), flag 612 indicating an anomaly in counter sample 614 can be associated with host counter sample 614. Furthermore, at time 3 (illustrated by box 610), flag 616 indicating an anomaly in counter sample 618 can be associated with host NIC counter sample 608.

[0075] For structure counter sample 606, at time 3 + / - zone tolerance (za) (illustrated by box 620), a marker 622 indicating anomalies in counter sample 614 can be associated with counter sample 614. Structure counter sample 606 includes a zone tolerance (za) applied to time 3 at the time indicated by box 620. The zone tolerance specifies a specific time window where correlations may exist between events. Therefore, structure marker 622 will be associated with host system indicator marker 612 and host NIC marker 616. However, if marker 622 is outside the zone tolerance, then marker 622 will not be associated with other markers 612 and 616.

[0076] A zone tolerance (ZA) can be a set duration specifically designated for a particular job or a particular type of event, and / or it can be a default predefined duration. Providing an adjustable ZA allows for flexibility in terms of event relevance.

[0077] Here, because markers 612, 616, and 622 all occur at time 3 or at least within the regional tolerance of time 3, each of these markers can be correlated with each other. As shown, the performance analysis and anomaly detection system 602 can store correlation 624 in correlation data storage 626. Correlation 624 can indicate the correlation between each of markers 612, 616, and 622, as shown in the figure.

[0078] The correlation 624 based on correlated events provides an indication of all correlated events within a specific region of interest (e.g., defined by the time of event occurrence plus / minus a region tolerance). Therefore, this time-based correlation of the markers in the correlation data store 626 can be used to highlight correlated events by providing specific visualizations that remove samples of non-correlated counters. In this way, efficient analysis and visualization are provided, thereby improving the efficiency of operational performance within the analysis system.

[0079] Not all cross-associations link events to other events. In fact, as mentioned above, cross-associations can associate counter samples based on markers observed in one of the counter samples at a specific time. In this case, the absence of an event within the region of interest can still be of interest, because such an absence can indicate that an event observed within the region of interest appears to be unrelated to an event at another entity, as indicated by the absence of an event in a counter sample associated with that entity during the region of interest.

[0080] Figure 6BThe illustration shows a cross-correlated counter sample 650 generated by cross-correlating the counter samples. Here, a marker (e.g., associated with at least one counter sample) is observed at time T1 indicated by node 652. Based on this observation, counter sample 654 on the infrastructure side 656 at time T1 and counter sample 658 on the host NIC side 670 occurring at time T1 are correlated, enabling correlation visualization and analysis. For example, counter 654 may include counters for a specific switch (e.g., S1-Sn) and its corresponding port (e.g., P1). Counter 658 may include specific host NIC counters for a specific host NIC (e.g., NIC1, NIC7) and / or a specific host / node (N1-Nm). Using current cross-correlation techniques, correlation analysis between the infrastructure side 656 and the host NIC side 670 can be facilitated.

[0081] Figure 7 A graphical user interface (GUI) visualization 700 depicts a visualization tool, providing examples of correlated counter samples and their benefits. Specifically, according to various aspects of this disclosure, the examples provided by the GUI visualization 700 include correlated network interface controller (NIC) counter samples 702A-702D, which have been correlated based on correlated events 704A-704D. The x-axis represents time, while the y-axis represents bit rate. The correlation analysis indicates the situation where the system has detected an event in each of the counter samples 702A-702D (e.g., because the bit rate has dropped below threshold 706), as shown in the figure. Events 704A-704D show that each of the correlated NIC counters is idle. The automatic correlation of events 704A-704D (e.g., via the visualization tool) can be quite useful in providing real-time analysis of job performance changes. For example, here, because each of the correlated NICs is idle, this may indicate a problem elsewhere in the system (e.g., a problem with the data arriving at the NIC), rather than a problem with one of these NICs. Therefore, by efficiently and effectively providing correlated events, current technology can provide a rapid understanding of important performance variables in a system.

[0082] As can be understood, any number of correlated events and / or metrics can be observed using current technology. For example, in one scenario, an event (e.g., indicated by a marker) where the number of cycles per packet exceeds an associated event threshold can be correlated with other counter samples in the visualization 700, thereby identifying metrics and / or events associated with each packet cycle event. In this way, events and / or metrics correlated with other counter samples can be viewed simultaneously, enabling diagnosis of the causes and impacts of events within a larger system. For example, based on the correlation between visualized PCIe metrics and each packet cycle event, a higher per-threshold cycle count can be attributed to host-level PCIe metrics.

[0083] Figure 8 A GUI visualization 800 is depicted, which provides a visual representation of correlated events across the host and infrastructure sides. For example, the GUI visualization 800 visualizes three distinct counter samples: a packet loss counter sample 802, a infrastructure link oscillation event counter sample 804, and a routing event update counter sample 806. The visualization tool can correlate these counter samples 802-806 based on markers appearing in one of these counter samples (when associating events with other counter samples) and / or markers appearing in each of these counter samples (when associating events across counter samples). Here, the visualization tool provides both types of correlation in the GUI visualization 800. For example, other related counter samples are introduced into the visualization 800 and correlated based on events occurring within one of the counter samples. Thus, a correlated view illustrating counter sample values ​​over a common time interval is provided (e.g., a stacked view where each of the counter samples is stacked relative to time). Furthermore, the visualization tool identifies correlated events within the counter samples. Here, for example, correlated events across regions of interest are identified in the visualized counter samples 802-806 (e.g., indicated by symbols 808A-808C visualized in GUI visualization 800). Visual symbols 810 are rendered to highlight this correlation of events across counter samples 802-806. Therefore, it can be understood that the marked events and / or correlations can be highlighted in GUI visualization 800, thereby providing efficient performance analysis.

[0084] Understandably, current technologies offer significant value. The proposed solution possesses the capability for automated network performance analysis of large systems, the ability to perform performance anomaly detection and operational failure analysis of network and related hardware, the ability to correlate host-side and infrastructure-side metrics and events in a highly automated manner, the ability to correlate system events (PCIe, memory issues) that affect operational performance, the ability to simplify visual representation using derived metrics for a large number of endpoint counts, and the ability to provide real-time analysis for large systems.

[0085] While certain features of this disclosure have been illustrated and described herein, many modifications and alterations will occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and alterations falling within the true spirit of this disclosure.

Claims

1. A computer-implemented method, comprising: Receive multiple counter samples, each of which provides a performance metric for the corresponding entity; For each of the multiple counter samples received: One or more events in the corresponding counter sample are identified based on the corresponding indicator breaking through the threshold, the indication of an event within the corresponding counter sample, or both. Associate the tag with each of the one or more events; These markers are used to identify one or more regions within the plurality of counter samples; as well as A graphical user interface (GUI) is generated and provided, which provides a visual representation of the plurality of counter samples and one or more identified regions.

2. The computer-implemented method as described in claim 1, comprising: Filter out at least a portion of the plurality of counter samples that are not associated with the one or more events from the visual representation of the plurality of counter samples in the GUI.

3. The computer-implemented method as described in claim 2, comprising: The client computer that provides the GUI is identified to have computer processing resource limitations; as well as The visual representation is filtered based on the client computer's processing resource limitations.

4. The computer-implemented method as described in claim 1, comprising: A power indication is generated and provided via the GUI, the power indication being used to specify a threshold corresponding to the plurality of counter samples, the threshold indicating an event when it is exceeded.

5. The computer-implemented method of claim 1, comprising: A power indication is generated and provided via the GUI, the power indication being used to specify one or more counter samples, the counter samples signifying a priority event when they exceed a priority event threshold; as well as The GUI generates and provides a visual representation of the priority event.

6. The computer-implemented method of claim 1, wherein: The plurality of counter samples include at least one structural-side counter and at least one host-side counter; and The computer-implemented method includes: Associate the events of the at least one structure-side counter and the at least one host-side counter.

7. The computer-implemented method as described in claim 6, wherein, The at least one host-side counter includes at least one network interface controller (NIC) counter and at least one host system counter.

8. The computer-implemented method as described in claim 6, wherein, The at least one structure-side counter and the at least one host-side counter are independently captured and stored in their respective first and second permanent databases via their respective first and second modes.

9. The computer-implemented method of claim 6, comprising: Associate the events of the switch-side performance counters with at least one of the following events: Network Interface Controller (NIC) performance counter, High-Speed ​​Peripheral Component Interconnect Bus (PCI-e) performance counter, Graphics Processing Unit (GPU) performance counter, Central Processing Unit (CPU) performance counter, or Dual In-line Memory Module (DIMM) performance counter.

10. The computer-implemented method as claimed in claim 1, wherein, The threshold includes at least one of the following: the rate of change threshold of the performance indicator or the performance value threshold of the performance indicator.

11. The computer-implemented method of claim 1, comprising: Based on the association of the multiple counter samples with the specific job being evaluated, the multiple counter samples are identified from a larger number of counter samples.

12. The computer-implemented method of claim 1, comprising: Based on at least one of the identified regions, at least a portion of the plurality of counter samples is cross-associated with one or more other portions of the plurality of counter samples; as well as Based on the cross-associations, a visual representation is generated and provided in the GUI.

13. The computer-implemented method of claim 12, wherein, The cross-associations include: Associate a first event of a first counter sample with a second event of a second counter sample that is different from the first counter sample.

14. A system comprising: A performance analysis and anomaly detection system, wherein the performance analysis and anomaly detection system is configured as follows: Receive multiple counter samples, each of which provides a performance metric for the corresponding entity; For each of the multiple counter samples received: One or more events in the corresponding counter sample are identified based on the corresponding indicator breaking through the threshold, the indication of an event within the corresponding counter sample, or both. Associate the tag with each of the one or more events; These markers are used to identify one or more regions within the plurality of counter samples; as well as A graphical user interface (GUI) is generated and provided, which provides a visual representation of the plurality of counter samples and one or more identified regions.

15. The system of claim 14, wherein, The performance analysis and anomaly detection system is configured as follows: The visual representation of the plurality of counter samples in the GUI is filtered such that at least a portion of the plurality of counter samples that is not associated with the one or more events is not presented.

16. The system of claim 14, wherein, The performance analysis and anomaly detection system is configured as follows: Cross-associate events from at least one structure-side counter with events from at least one host-side counter; and The at least one host-side counter includes at least one network interface controller (NIC) counter and at least one host system counter.

17. The system of claim 14, wherein, The performance analysis and anomaly detection system is configured as follows: Cross-associate the events of the switch-side performance counters with at least one of the following events: Network Interface Controller (NIC) performance counter, High-Speed ​​Peripheral Component Interconnect Bus (PCI-e) performance counter, Graphics Processing Unit (GPU) performance counter, Central Processing Unit (CPU) performance counter, or Dual In-line Memory Module (DIMM) performance counter.

18. The system of claim 14, wherein, The performance analysis and anomaly detection system is configured as follows: Cross-associate at least one of the following: one or more identified events or one or more identified regions with other counter samples in the plurality of counter samples; and Based on the cross-associations, a visual representation is generated and provided in the GUI.

19. A non-transitory computer-readable medium comprising computer-readable instructions that, when executed by one or more processors of one or more computers, cause the one or more computers to: Receive multiple counter samples, each of which provides a performance metric for the corresponding entity; For each of the multiple counter samples received: One or more events in the corresponding counter sample are identified based on the corresponding indicator breaking through the threshold, the indication of an event within the corresponding counter sample, or both. Associate the tag with each of the one or more events; These markers are used to identify one or more regions within the plurality of counter samples; as well as A graphical user interface (GUI) is generated and provided, which provides a visual representation of the plurality of counter samples and one or more identified regions.

20. The non-transitory computer-readable medium of claim 19, comprising computer-readable instructions that, when executed by the one or more processors of the one or more computers, cause the one or more computers to: Cross-associate at least one of the following: one or more identified events or one or more identified regions with other counter samples in the plurality of counter samples; and Based on the cross-associations, a visual representation is generated and provided in the GUI.

Images