Smart log analysis for large scale high performance computing and artificial intelligence systems

Abstract

The invention relates to intelligent log analysis for large-scale high-performance computing and artificial intelligence systems. A system obtains event information from a component operating jointly in the system, the event information indicating a first set of events interpreted from log entries associated with the component and a second set of events returned from queries for standard events. The system classifies events interpreted from log entries based on the hierarchy of the components. The system associates two or more events based on respective event classifications and predetermined time windows covering event times associated with the respective events. The event time is obtained from the log entry. The system generates a visual representation indicative of the associated event. In response to the visual representation indicating an anomaly, the system allows a corrective action to resolve the indicated anomaly.

Description

BACKGROUND

[0001] Large scale systems such as high performance computing (HPC) and artificial intelligence (AI) systems can include many subsystems, such as storage infrastructure, network fabric, host interfaces, centralized fabric managers (FMs), switches, and other controllers. Workloads in HPC and AI systems can have sensitivities to events in the subsystems and can impact performance of jobs. Anomaly detection and root cause analysis often involves extracting and analyzing event information from the subsystems. However, such event information can be distributed in many formats (e.g., host level log, FM console log, external system log, etc.) across many subsystems. Furthermore, there can be relationships between multiple subsystems that can make tracking for root cause analysis complex. BRIEF DESCRIPTION OF DRAWINGS

[0002] FIG. 1A FIG. illustrates a system overview of an environment that facilitates intelligent log analysis for large scale HPC and AI systems, including subsystems and logs, in accordance with an aspect of the present application.

[0003] FIG. 1B FIG. illustrates an example component topology that facilitates intelligent log analysis for large scale HPC and AI systems, in accordance with an aspect of the present application.

[0004] FIG. 2 FIG. illustrates a high level flow that facilitates intelligent log analysis for large scale HPC and AI systems, in accordance with an aspect of the present application.

[0005] FIG. 3A FIG. illustrates a diagram of log entry to standard format transformation, in accordance with an aspect of the present application.

[0006] FIG. 3B FIG. illustrates a diagram of log entry to related event transformation, in accordance with an aspect of the present application.

[0007] FIG. 3C FIG. illustrates a decision tree for event classification for log entries, in accordance with an aspect of the present application.

[0008] FIG. 4 FIG. illustrates an environment that facilitates intelligent log analysis for large scale HPC and AI systems, including a log analysis system in communication with multiple entities, in accordance with an aspect of the present application.

[0009] FIG. 5A FIG. illustrates an example display screen depicting a visualization that includes anomalies above a certain threshold for applications running on hosts, network interface controllers (NICs), and hardware, in accordance with an aspect of the present application.

[0010] FIG. 5BAn example display screen depicting a visualization including relevant time periods that consider the association of events based on changes in power consumption, according to an aspect of the application.

[0011] FIG. 5C An example display screen depicting a visualization including events associated with exceptions of applications running on hosts and hardware, according to an aspect of the application.

[0012] FIG. 5D An example display screen depicting a visualization including extraction from logs of fabric managers and fabric controller agents, according to an aspect of the application.

[0013] FIG. 5E An example display screen depicting a visualization including exceptions of applications running on hosts and events related to fabric links, according to an aspect of the application.

[0014] FIG. 5F An example display screen depicting a visualization including network drop events and link events, according to an aspect of the application.

[0015] FIG. 6A And FIG. 6B A flow diagram illustrating a method that facilitates intelligent log analysis for large-scale HPC and AI systems, according to an aspect of the application, is presented.

[0016] FIG. 7 A computer system that facilitates intelligent log analysis for large-scale HPC and AI systems, according to an aspect of the application, is illustrated.

[0017] FIG. 8 A computer-readable medium that facilitates intelligent log analysis for large-scale HPC and AI systems, according to an aspect of the application, is illustrated.

[0018] In these drawings, like reference numbers refer to like elements throughout. DETAILED DESCRIPTION

[0019] Aspects of the application provide an intelligent analysis automation engine that: defines a hierarchy of relationships between subsystems of an overall system; interprets log information from the subsystems as event information; and classifies the events to derive association information between them. The described aspects can also generate reports or visual representations of the associations, which can allow corrective actions to be taken to address indicated exceptions.

[0020] Large scale systems (e.g., HPC and AI systems) can include many subsystems (e.g., storage infrastructure, network fabric, host interfaces, centralized fabric manager, switches, and other controllers). Workloads in such large scale systems can be sensitive to events in the subsystems, which can impact performance of jobs running on the subsystems. Identifying related events and anomalies on many subsystems and components can require extracting and analyzing event information distributed in many formats across many subsystems, such as host-level record logs, fabric manager console logs, fabric controller agent console logs, external system logs, and the like. Moreover, there can be relationships between multiple subsystems, which can make tracking for performing root cause analysis complex.

[0021] Extracting and analyzing event information distributed in many formats across many subsystems can be performed by individually customized programs. However, such a solution can be inefficient in terms of time and computational cost. Additionally, analyzing relationships between subsystems can involve complex tasks. For example, a reliability service for a high-speed NIC can be recording events that are symptoms of a problem rather than the problem itself. Reported timeouts can impact performance of jobs, but their root cause can be other factors, such as a network interface malfunctioning in a different host or a link error in a fabric link. Thus, analyzing relationships between subsystems in the context of complex tasks can be a limitation for efficiently identifying root causes of various observed abnormal behaviors.

[0022] The described aspects address these limitations by providing a system that extracts, filters, and formats logs from multiple subsystems and subsequently transforms the logs into events. The system can also classify the events based on a hierarchy of relationships (e.g., as described below with respect to a decision tree) and can further correlate two or more events based on the classification and a certain time window associated with the respective events. The described aspects can also generate a report or visual representation of the correlations, which can produce interactive user feedback, e.g., allowing a user to perform corrective actions to address indicated anomalies. FIG. 3C

[0023] FIG. 1A ​An environment 100 that facilitates intelligent log analysis for large-scale HPC and AI systems is illustrated, in accordance with an aspect of the present application, including subsystems and logs. The environment 100 can be a large-scale HPC or AI system with multiple subsystems, where each subsystem records events in its own log during operation. For example, an application 110 can record events in an application log 112. A NIC controller agent 114 can record hardware events 116 in a console log and software events 118 in a host log. Host hardware 120 can include central processing units (CPUs), general processing units (GPUs), peripheral component interconnect express (PCIe) units, high-bandwidth memory (HBM) processors, and dual in-line memory modules (DIMMs). The host hardware 120 can record hardware events 122 in a console log and software events 124 in a job controller log. A fabric manager (FM) 126 can record hardware events 128 in a console log of a fabric manager host and software events 130 in a host log of the fabric manager host. A domain name server (DNS) service 132 can record hardware events 134 in a console log and software events 136 in a host log. A chassis manager (CM) 138 can record events in a chassis manager log 140. A fabric controller agent (FCA) 142 can record hardware events 144 in a console log of a switch and software events 146 in a switch log. A storage / cluster controller agent 148 can record events in a storage / cluster log 150. A rack manager 152 can record events in a rack manager log 154. The subsystems and logs depicted in the environment 100 are non-limiting and provided for illustrative purposes only. Other subsystems, components, units, and modules can create other logs based on hardware, firmware, software, or a combination.

[0024] FIG. 1BAn example component topology 160 that facilitates intelligent log analysis for large-scale HPC and AI systems is illustrated in accordance with an aspect of the present application. In topology 160, a rack 162 can include storage devices (or cluster) 164, hosts 166, and a chassis manager (CM) 184. Hosts 166 can include NICs 168, CPUs 174, DIMMs 176, HBMs 178, GPUs 180, and resource allocation (and application launcher service) 182. NICs 168 can interact based on NIC controller software 170 and PCIe 172. CPUs 174 can also interact based on PCIe 172. CM 184 can control or provide management services for switch 186. A fabric manager (FM) 192 can also provide management services for and interact with switch 186. FM 192 can also interact with a fabric controller agent (FCA) 188 and a domain name server / network time protocol (DNS / NTP) 194. FCA 188 can also interact with a protocol agent 190. The organization of elements (i.e., subsystems) in topology 160 is non-limiting and provided for illustrative purposes only. Other topologies, elements (subsystems), and relationships between elements can be part of a network topology.

[0025] FIG. 2 A high-level flow 200 that facilitates intelligent log analysis for large-scale HPC and AI systems is illustrated in accordance with an aspect of the present application. During operation, the operations of modules 210, 212, and 214 can be performed by log agents running in particular components or subsystems (e.g., log agents 412, 432, 452, and 472 depicted below in FIG. 4 FIG. 4), while the operations of modules 216, 218, 220, and 222 can be performed by a central orchestrator (e.g., log analysis orchestrator 401 depicted below in FIG. 4 FIG. 4). Log extraction module 210 can include log agents of a subsystem that extract various logs from components of the subsystem, such as host logs and console logs. Log filtering module 212 can include log agents that eliminate noise in the extracted logs. Log transformation module 214 can include log agents that transform the extracted and filtered log entries into event entries, as described below with respect to FIG. 3A and FIG. 3B Event classifier module 216 can include a central orchestrator that classifies events indicated in the transformed event entries, as described below with respect to FIG. 6A and FIG. 6B Event correlation module 218 can include a central orchestrator that correlates the classified events based on a hierarchy of components, as described below with respect to FIG. 3CThe reporting module 220 can include a central orchestrator that generates reports based on the associated events, as described below with respect to FIG. 4 and FIG. 5A through FIG. 5F The visual transformation module 222 can include a central orchestrator that generates visual representations indicative of the associated events, as described below with respect to FIG. 5A through FIG. 5F Additionally, a user interaction module (not depicted) can include user interaction with the information generated by the reporting module 220 or the visual transformation module 222, as described below with respect to FIG. 5A through FIG. 5F

[0026] FIG. 3A A diagram 300 illustrating the transformation of log entries to a standard format is shown, in accordance with an aspect of the present application. The diagram 300 depicts log entries 310, 320, and 330, all of which have the same standard format. For example, the log entry 310 can include information related to an event, such as: an entity 311 corresponding to the event or associated with the event; an event time 312 indicative of a time at which the event occurred, such as a start time, an end time, or a time window; an event category 313 indicative of, for example, a severity of the event; an event type 314 indicative of, for example, a software event, a hardware event, a processor event, a configuration event, or an error event; and event information 315 and other related information indicative of a description of the event. Similarly, the log entry 320 can include: an entity 321; an event time 322; an event category 323; an event type 324; and event information 325. Additionally, the log entry 330 can include: an entity 331; an event time 332; an event category 333; an event type 334; and event information 335.

[0027] A log agent running on a subsystem can create the formatted log entries of the diagram 300 based on raw logs extracted from various components of the subsystem. The log agent can further transform these formatted log entries, as described below with respect to FIG. 3B

[0028] FIG. 3B ​​A diagram 338 illustrating transformation of log entries to related events according to an aspect of the application is shown. Diagram 338 illustrates that log entries 360 can be transformed (as indicated by 364) to event entries 362 based on event type (e.g., host, software, hardware, etc.) and in a time format (e.g., single time or time window). For example, log entries occurring at time 340.A (or within a time window defined by time 340.A) can include entries 342.1, 342.2, and 342.N. Similarly: log entries occurring at time 344.A (or within a time window defined by time 344.A) can include entries 346.1, 346.2, and 346.N; and log entries occurring at time 348.A (or within a time window defined by time 348.A) can include entries 350.1, 350.2, and 350.N.

[0029] The log agent running on the subsystem can transform log entries 360 to event entries 362 such that the event entries are clustered or grouped in similar corresponding times. For example, log entries 342.1-N grouped to time 340.A can be transformed to events 352.1, 352.2, and 352.M grouped to time 340.B. Log entries 346.1-N grouped to time 344.A can be transformed to events 354.1, 354.2, and 354.M grouped to time 344.B. Log entries 350.1-N grouped to time 348.A can be transformed to events 356.1, 356.2, and 356.M grouped to time 348.B. The log agent can perform the transformation of log entries to events based on event type information (e.g., as described above with respect to event type 314 of log entries 310 in FIG. 3). FIG. 3A

[0030] FIG. 3C A decision tree 368 for event classification of log entries according to an aspect of the application is shown. As described above with respect to FIG. 3, the log agent running on the subsystem can transform log entries 360 to event entries 362 based on event type (e.g., host, software, hardware, etc.) and in a time format (e.g., single time or time window). For example, log entries 342.1-N grouped to time 340.A can be transformed to events 352.1, 352.2, and 352.M grouped to time 340.B. Log entries 346.1-N grouped to time 344.A can be transformed to events 354.1, 354.2, and 354.M grouped to time 344.B. Log entries 350.1-N grouped to time 348.A can be transformed to events 356.1, 356.2, and 356.M grouped to time 348.B. The log agent can perform the transformation of log entries to events based on event type information (e.g., as described above with respect to event type 314 of log entries 310 in FIG. 3). FIG. 2 ​As described above with respect to the event classifier module 216 of FIG. 3, the central orchestrator can perform event classification after obtaining transformed event entries from various log agents of the subsystems. Event classification 370 can be related to storage 371, host 374, or fabric 383. If the event is a storage 371 event, the classification can be hardware 372 or software 373. If the event is a host 374 event, the classification can be: hardware 375, which can be further classified as processor events 376, PCIe events 377, or DIMM / HBM events 378; or software 380, which can be further classified as related to memory leaks 381 or software libraries 382. If the event is a fabric 383 event, the classification can be hardware 384 or software 391. Fabric event hardware 384 can be further classified as related to: NIC 385, which can be further classified as related to hardware errors 386 or reliability services 387; or switches 388, which can be further classified as related to hardware / application specific integrated circuit (ASIC) errors 389 or fabric port errors 390. Fabric event software 391 can be further classified as related to fabric manager (FM) 392 or fabric controller agent (FCA) 394. FM 392 can be further classified as related to resources 393 or invalid switch configuration 396. FCA 394 can be further classified as related to resources 393, protocol agent 395, or invalid switch configuration 396.

[0031] FIG. 3C The organization and elements depicted in decision tree 368 are non-limiting and provided for illustrative purposes only. Other decision tree topologies and element relationships can be used.

[0032] FIG. 4 FIG. 3 illustrates an environment 300 that facilitates intelligent log analysis for large-scale HPC and AI systems, according to an aspect of the present application. Environment 300 includes a log analysis orchestrator 301 in communication with a plurality of entities. In environment 300, log analysis orchestrator 301 (also referred to as “orchestrator 301”) can be in communication with a plurality of entities, including a fabric manager (FM) 310, a switch 330, and hosts 350 and 370. Each entity can include its own log agent that performs log extraction / collection of various logs generated and stored by the respective entity, and also performs log transformation of event entries. For example, FM 310 can include a log agent 312 that includes a log extraction / collection module 314 that obtains raw logs, e.g., from fabric DB 327 or host logs 326. Log agent 312 can also include a log transformation module 316 that formats the raw logs into log entries and event entries, as described above with respect to FIG. 2. FIG. 3A through FIG. 3BAs described. Log agent 412 can store transformed event entries in, for example, host log 426. The operation of log extraction / collection module 414 can correspond to... FIG. 2 The operation of module 210, and the operation of log transformation module 416 can correspond to FIG. 2 Module 214. FM 410 may also include: a management plane 420 with a health engine 421; a control plane 422 with a routing engine 423; an operating system 424; and hardware 425.

[0033] Switch 430 may include a log agent 432, which includes a log extraction / collection module 434 that obtains raw logs, for example, from an agent database 446 or a host log 445. Log agent 432 may also include a log transformation module 436 that formats the raw logs into log entries and event entries, as described above. FIG. 3A through FIG. 3B As described. Log agent 432 can store extracted / collected logs in log event DB 444, and can further store transformed event entries in, for example, host log 445. Switch 430 may also include: switch agent 440; platform services / software development kit (SDK) / driver 441; operating system 442; and hardware 443.

[0034] Host 450 may include a log agent 452, which includes a log extraction / collection module 454 that obtains raw logs from, for example, host logs 465. Log agent 452 may also include a log transformation module 456 that formats the raw logs into log entries and event entries, as described above. FIG. 3A through FIG. 3B As described above, log agent 452 can store extracted / collected logs in log event DB 464, and can further store transformed event entries in, for example, host log 465. Host 450 may also include: host NIC agent 460; platform services / SDK / drivers 461; operating system 462; and hardware 463. Similarly, host 470 may include log agent 472, which includes a log extraction / collection module 474 that obtains raw logs from, for example, host log 485. Log agent 472 may also include a log transformation module 476 that formats the raw logs into log entries and event entries, as described above. FIG. 3A through FIG. 3B As described. Log agent 472 can store extracted / collected logs in log event DB 484, and can further store transformed event entries in, for example, host log 485. Host 470 may also include: host NIC agent 480; platform services / SDK / drivers 481; operating system 482; and hardware 483.

[0035] The orchestrator 401 can include an event extraction / collection module 404 and a log event extraction module 405. The event extraction / collection module 404 can query a plurality of entities for logs that can be related to standard events tracked by the respective entities (e.g., via communications 490 from the module 404 to the FMs 410). While only communications 490 to the FMs 410 are depicted, the module 404 can also query standard events from other entities. The log event extraction module 405 can communicate with log agents of the plurality of entities (e.g., via communications 491, 492, 493, and 494 with the log agents 412, 432, 452, and 472 of the FMs 410, the switch 430, the host 450, and the host 470, respectively) to obtain transformed event entries.

[0036] In obtaining both events returned from queries for standard events (e.g., via 490) and events interpreted from log entries associated with entities or components (e.g., via 491-494), the orchestrator 401 can store the extracted data in one or more of a relational database 406, a time series database 407, or a temporary database 408. In some aspects, the temporary DB 408 can include filtered, extracted, formatted, transformed event entries output by the operation of the module 214 in FIG. 2 The time series DB 407 can include log entries and event entries grouped or clustered based on event type and time format (e.g., by a certain time or time window). The relational DB 406 can include information correlating two or more events based on respective event classifications and event times of the two or more events. The system can also obtain power utilization of the components from the management software of each component over a period of time. The module 403 (or another module, not shown) can convert metrics related to application runtimes and transaction results into time series data and store the data in the DB 407 along with the obtained power utilization. The system can use the data stored in any of the relational DB 406, the time series 407, and the temporary DB 408 to determine correlations and identify relevant time periods of abnormal measurements or activities.

[0037] After classifying and correlating events, the visualization and reporting module 402 of the orchestrator 401 can generate reports and visualizations. The following example visualizations are provided for a display screen. The operation of the log event classification / correlation module 403 can correspond to the modules 216 and 218 of FIG. 5A through FIG. 5F The following example visualizations are provided for a display screen. The operation of the log event classification / correlation module 403 can correspond to the modules 216 and 218 of FIG. 2 The following example visualizations are provided for a display screen. The operation of the log event classification / correlation module 403 can correspond to the modules 216 and 218 of FIG. 2 The following example visualizations are provided for a display screen. The operation of the log event classification / correlation module 403 can correspond to the modules 216 and 218 of

[0038] InFIG. 4 The entities, components, and subsystems depicted in environment 400 are non-limiting and provided for illustrative purposes only. Other entities and relationships may be used. For example, the functionality of orchestrator 401 may reside in a single computing device, be accessible via a cloud computing environment, or be distributed across multiple virtual or physical network devices or nodes in a networked environment. As another example, for each of the depicted entities (FM 410, switch 430, and hosts 450 and 470), there may be more or fewer elements or components.

[0039] FIG. 5A An exemplary display screen depicting a visualization according to one aspect of this application is illustrated, the visualization including anomalies exceeding a certain threshold in applications running on a host, NIC, and hardware. FIG. 5A Schematic diagrams 500, 510, 520, and 530 illustrate representations of application measurements used to correlate anomalous events from logs and standard events. Schematic 500 indicates measurements associated with host events (e.g., host transaction A 502). Schematic 510 indicates measurements associated with host events (e.g., host transaction B 512). Schematic 520 indicates measurements associated with NIC events (e.g., NIC event 522). Schematic 530 indicates measurements associated with hardware events (e.g., hardware event 532). In schematic diagrams 500, 510, 520, and 530, the x-axis indicates time increments in ten-minute increments from 17:30 to 20:00. In schematic diagram 500, the y-axis indicates the amount of time in seconds and minutes. In schematic diagram 510, the y-axis indicates the amount of time in milliseconds. In diagrams 520 and 530, the y-axis indicates the number of errors (e.g., the error count at a given time).

[0040] Users can view visualizations of event measurements from various entities based on transformed log entries from the orchestrator. Visual inspection of the displayed information allows users to quickly identify and remedy related problems.

[0041] In diagram 500, partially shaded points correspond to the measurement results of transaction A (504) occurring at a given time. Most measurements appear on the 0 millisecond line, indicating that most measurements are below a certain expected threshold. However, diagram 500 also indicates the occurrence of transaction A that takes much longer than the threshold between 18:00 and 18:45 and 18:50.

[0042] In graph 510, the partially shaded dots correspond to measurements of transaction B (514) that occurred at a given time. For the indicated time period, most of the measurements occurred in a fairly distributed manner in the range between 1000 milliseconds and 1800 milliseconds. No immediately discernible unusual or abnormal activity appears in graph 510.

[0043] In graph 520, the dots correspond to counts of various NIC-related events (522). The partially shaded dots correspond to power-on events (524), while the thick-line dots correspond to flapping events (526). Graph 520 indicates three occurrences of NIC flapping between 18:45 and 18:50, which is also the same time period in which abnormal host transaction A measurements occurred (as depicted by graph 500). Thus, the user can determine the abnormality in the events of host transaction A and the NIC's flapping, and thus the correlation between them. The user can perform a corrective action to address the abnormality, e.g., rebooting or replacing the NIC.

[0044] In graph 530, the dots correspond to counts of various hardware-related events (532). The partially shaded dots correspond to core error events (534), the solid-color dots correspond to DIMM error events (536), and the thick-line dots correspond to machine check exception (MCE) error events (538). Graph 530 indicates that two MCE errors occurred between 17:45 and 17:50, and graph 510 indicates that several abnormal occurrences of host transaction B measurements also occurred in the same time window. Thus, the user can determine the abnormality in the events of host transaction B and the MCE errors detected in the hardware, and thus the correlation between them. The user can perform a corrective action to address the abnormality, e.g., isolating or removing the node in which the MCE errors were detected.

[0045] The system can also generate a report (not depicted) that can indicate the detected abnormality or correlation and suggest a corrective action for the user to take in order to address the abnormality. The report and visualizations can include one or more interactive elements that facilitate viewing or manipulating the displayed information (whether in the report or the visualizations). The interactive elements can relate to, for example, the detected abnormality; a recommended action that indicates a remedy for the detected abnormality; or a configurable option that indicates that the system will automatically perform the recommended action. In some aspects, the system can provide a configurable or selectable default option at startup that relates to when to take the recommended option, the type of automatic action that is approved by the user, the duration for which approval of the automatic action can be given, and the like.

[0046] FIG. 5BAn exemplary display screen depicting a visualization according to one aspect of this application is illustrated, the visualization including a relevant time period considering the association of events based on power consumption variations. Graph 550 indicates the change in power measurement results (y-axis, in megawatts (MW)) over time (x-axis, time increments of ten minutes from 17:30 to 20:00). Graph 550 indicates a significant power spike near 18:06. Used in conjunction with other visualizations, such as those of anomalies in applications running on a host, NIC, or hardware, a user can determine that one or more events occurring around the same time can be related to the power spike indicated in graph 550. The user can perform corrective actions based on the visual representation generated by the system to investigate or resolve the cause of the power spike. For example, the user can observe… FIG. 5A The simplified diagrams (i.e., 500, 510, 520, and 530) and FIG. 5B The pattern between the simplified diagrams 550. Drops or spikes in the power curves that occur during time periods similar to the time periods of anomalies in the application can identify relevant time periods for further analysis. In simplified diagram 510, several anomalous measurements related to host transaction B 512 occur between 18:00 and 18:10. During the same time period, the power curves in simplified diagram 550 indicate power spikes (between 18:00 and 18:10). Based on the visual representation, the user (or system) can correlate events and perform corrective actions to further investigate the correlation between the anomalous measurements of host transaction B 512 and the power curves in simplified diagram 550 that appear during that relevant time period (e.g., the time period between 18:00 and 18:10). The user can also further investigate other actions that may have occurred during the identified relevant time period.

[0047] FIG. 5C An exemplary display screen depicting a visualization according to one aspect of this application is illustrated, the visualization including events associated with anomalies of an application running on a host and hardware. Schematic 560 indicates measurements associated with host events (e.g., host transaction 562). In schematics 560 and 565, the x-axis indicates time in five-minute increments from 18:40 to 19:55. In schematic 560, the y-axis indicates the amount of time in milliseconds. In schematic 565, the y-axis indicates the number of errors (e.g., error count for a given time).

[0048] In graph 560, partially shaded dots correspond to measurements of host transactions (564) that occurred at a given time. In graph 560, transaction measurements greater than 1000 milliseconds can be considered abnormal. For example, several abnormal measurements occur between 19:10 and 19:53. In graph 565, solid color dots correspond to DIMM error events (567). The same number of DIMM errors repeatedly occur throughout the measured time period, including in groups of occurrences that align with the abnormalities of host transactions 562, for example, near 19:10 and 19:14, 19:45 and 19:26, 19:36 and 19:39, and 19:50 and 19:52. DIMM errors (567) that consistently occur from a particular node can be correlated with corresponding abnormal measurements of host transactions (564). Thus, a user can perform corrective actions to address the abnormality, for example, abort a job associated with the host transaction and take additional actions.

[0049] FIG. 5D FIGURE 8 illustrates an exemplary display screen depicting a visualization that includes extracting logs from fabric managers and fabric controller agents, according to an aspect of the application. Graph 570 indicates measurements associated with fabric link events 571, and graph 574 indicates route updates 575. In graphs 570 and 574, the x-axis indicates time from 18:40 to 19:55 in five minute increments. In graph 570, the y-axis indicates the number of fabric link events, and solid color dots represent link oscillations or changes for a particular link (572). In graph 574, the y-axis indicates route updates, and partially shaded dots indicate route updates (576) at the indicated times. Based on the correlation of route updates with fabric link changes, a user can identify abnormal activity or events that can be associated with a particular application or job. FIG. 5D Correlations between route updates and fabric link changes can be made during time periods near 19:08 and 19:51. It can be observed that route updates are a result of fabric link changes, i.e., correlated events. Thus, times or time windows near these time periods can be associated with detecting abnormal or abnormal activity.

[0050] FIG. 5E FIGURE 9 illustrates an exemplary display screen depicting a visualization that includes abnormalities of applications running on hosts and events related to fabric links, according to an aspect of the application. FIG. 5E An example of log analysis from a fabric controller agent is depicted that is used in conjunction with application logs in order to correlate behavior.

[0051] In graphs 578 and 582, the x-axis indicates time in ten minute increments from 17:30 to 20:00. The data in graphs 578 and 582 can be based on a sample high performance benchmark running on thousands of nodes. Graph 578 indicates measurements associated with transactions 579, where: the y-axis indicates the amount of time in seconds; partially shaded dots indicate measurements of exchange transactions 580; and solid color dots indicate measurements of broadcast transactions 581. Graph 582 indicates measurements associated with fabric link events 583, where: the y-axis indicates the number of fabric link events; and solid color dots represent link flapping or changes (584) for a particular link. Based on FIG. 5E Correlations can be made between certain transaction times and fabric events at time periods around 18:41. At 18:41, a fabric event (link flapping or changes 584) can cause high exchange transactions (580) and broadcast transactions (581) times in a high performance application.

[0052] FIG. 5F FIGS. 6-9 illustrate example displays depicting visualizations including network drop events and link events, in accordance with an aspect of the present application. In graphs 586, 592, and 596, the x-axis indicates time in 15 minute increments from 06:45 to 10:30. Local Link_A and Local Link_B can represent local links in, for example, a dragonfly topology, while Global Link_A can represent a global fabric link in, for example, a dragonfly topology. FIG. 5F The links described in the examples below are for illustrative purposes only. Other links and network topologies can be used. FIG. 5F Examples are depicted of extracting standard health events in addition to extracting logs to determine relevant time periods (i.e., predetermined time windows) for detecting abnormal behavior.

[0053] Graph 586 indicates measurements associated with network drop events 587, where: the y-axis indicates the number of drop events (e.g., the number of dropped packets); partially shaded dots indicate drop events for a local fabric link (Local Link_A 588); thick line dots indicate drop events for a local fabric link (Local Link_B 589); solid color dots indicate drop events for a global fabric link (Global Link_A 590); and other dots indicate drop events for other links (other links 591). Note that the other dots depicted as other links can represent separate local or global fabric links, and are depicted with the same label in graph 586 for illustrative purposes. Various colors, labels, formatting, or other identifiers can be used to indicate each of the other separate local or global fabric links.

[0054] The graph 592 indicates measurement results associated with global link oscillation events 593, where: the y-axis indicates the number of link oscillations (e.g., at a given time); the solid color points represent link oscillations of global link_A 594; and the other points represent link oscillations of other global links 595. The graph 596 indicates measurement results associated with local link oscillation events 597, where: the y-axis indicates the number of link oscillations (e.g., at a given time); the solid color points represent link oscillations of local link_A 598; and the thick line points represent link oscillations of local link_B 599.

[0055] Based on FIG. 5F The packets that can be dropped in the fabric can be correlated with link oscillations of different levels (i.e., local link_A, local link_B, and global link_A). For example, around 07:00, a link oscillation of local link_A (as depicted in the graph 596) can cause the network packet drops depicted for local link_A at the same time (as depicted in the graph 586). Similarly, around 08:17, a link oscillation of local link_B (as depicted in the graph 596) can cause the network packet drops depicted for local link_B at the same time (as depicted in the graph 586). In addition, around 09:38, a link oscillation of global link_A (as depicted in the graph 592) can cause the network packet drops depicted for global link_A at the same time (as depicted in the graph 586).

[0056] FIG. 6A And FIG. 6B Flowcharts 600 and 630 are presented that illustrate methods that facilitate intelligent log analysis for large-scale HPC and AI systems, in accordance with an aspect of the present application. During operation, the system obtains event information from components that are jointly operating in the system, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events (operation 602). Log agents running on various components can generate the log entries indicating the first set of events by extracting logs from one or more components in the system, as described above with respect to the log extraction / collection modules 414, 434, 454, and 474 of the log agents 412, 432, 452, and 472 of FIG. 4 , and the log extraction module 210 of FIG. 2 . The log agents can remove noise in the extracted logs by filtering the extracted logs, and can also obtain reformatted log entries by reformating the filtered logs. For example, the log entries 310, 320, and 330 of FIG. 3A may be obtained after the above filtering and reformating (as also described above with respect to the log extraction / collection modules 414, 434, 454, and 474 of the log agents 412, 432, 452, and 472 of FIG. 2described above with respect to the log filtering module 212 of FIG. 2. The log broker can generate event information based on characteristics of the reformatted log entries, as described above with respect to the event information 315, 325, and 335 of FIG. 3. FIG. 3A

[0057] The system classifies events interpreted from log entries based on a hierarchy of components (operation 604). For example, the log analysis orchestrator 401 can use a decision tree, as described above with respect to the decision tree of FIG. 4. The decision tree can be based on the hierarchy of components, as described above with respect to the hierarchy of components of FIG. 5. FIG. 4 FIG. 3C

[0058] The system correlates two or more events based on respective event classifications and a predetermined time window covering event times associated with the respective events, the event times derived from log entries (operation 606). The predetermined time window can be determined from measurements related to power consumption, application runtime, and transaction results associated with components. For example, for a given time window, two or more events can be correlated that have respective classifications and occur during the same time window, as described above with respect to the NIC oscillation error (526) in the NIC event 522 and the anomalous measurement (504) of the host transaction A 502 in the visual representation of FIG. 5 and the example described above in FIG. 5A FIG. 5B through FIG. 5F

[0059] The system stores information associated with the first and second groups of events in entries in a data structure, where respective entries indicate the determined event classifications and any correlations with other events (operation 608). The system can store the information before classifying events or correlating events (as in operations 604 and 606, respectively). The information can be stored in a format similar to the format described above with respect to the log entries 310, 320, and 330 in FIG. 3. FIG. 3A FIG. 4 The system can store the entries in a time series database, as described above with respect to the time series database 407 of the log analysis orchestrator 401 in FIG. 4.

[0060] ​​​​​​The system determines whether to directly query the data structure or extract additional information (Decision 610). This determination can be based on a previously configured setting indicating whether additional information, such as that related to power metrics, will be used to determine a first predetermined time period or to identify relevant time periods. If the system determines to directly query the data structure (Decision 610), it queries the data structure for events associated with the first predetermined time period (Operation 612). The first predetermined time period can be based on measurements related to power consumption, application runtime, and transaction results associated with components. The system associates the queried events by tagging the corresponding entries of the queried events with the same association identifier label (Operation 614). The system can also associate the queried events by linking the entries together using pointers or other related operations. FIG. 6B Continue at label A.

[0061] If the system determines to extract additional information (Decision 610), then the system extracts power metrics and application metrics within a time window (Operation 616), such as power utilization and application metrics associated with components in the system during a certain time window that can identify relevant time periods with abnormal measurement results, as mentioned above. FIG. 4 This is described in module 403 of the log analysis orchestrator 401. The system identifies relevant time periods within a time window based, for example, on the following: power decrease; application runtime increase; or slow measurement in the application (e.g., slower than a predetermined threshold) (operation 618). The factors listed herein as the basis for identifying relevant time periods are provided for illustrative purposes only. Other factors may be used. The system may use the identified relevant time periods as a first predetermined time period, and the operation continues at operation 610.

[0062] FIG. 6B Depicting FIG. 6A The operation in this section is a continuation of operation 614. The system generates a visual representation indicating the associated event (operation 632). The visual representation can indicate the associated event and the associated queried event (from operation 612). The visual representation can include a simplified diagram indicating measurement results (such as time quantity or number of errors) over a period of time, for example, such as... FIG. 5A through FIG. 5FThe system generates a report based on the correlated events (operation 634). The system can display the report, and the report can include one or more interactive elements that facilitate viewing or manipulating the displayed information, including but not limited to, for example: detected anomalies; recommended actions indicating remediation of the detected anomalies; or configurable options indicating that the computer will automatically perform the recommended actions. The system can further perform a first action based on the displayed report. The first action can be a corrective action performed by a user associated with the system, or the first action can be an action automatically performed by the system based on a previously configured option to automatically accept or perform the recommended action.

[0063] If the visual representation does not indicate an anomaly (decision 636), the operation returns. If the visual representation indicates an anomaly (decision 636), the system allows a corrective action to resolve the indicated anomaly (operation 638). For example, in response to the diagrams 500 and 520 indicating an anomaly based on the displayed measurements and related events, a user can perform a corrective action to resolve the indicated anomaly, for example, by rebooting a NIC, removing a job or suspending a host transaction, removing or replacing a node or other hardware component, etc. In some aspects, the operations 616 and 618 can be performed by a user in response to viewing the generated visual representation or report. That is, by viewing the visual representation or report, the user can identify a relevant time period in a certain time window based on the extracted and displayed power metrics and application metrics. The user (or system) can query events in the identified relevant time period in a data structure and correlate the queried events (as described above with respect to operations 612 and 614). Additionally, the user can perform a corrective action based on the displayed report (generated in operation 634 as described above), for example, based on a recommended action indicating remediation of a detected anomaly. For example, the user can replace a NIC identified as being related to anomalous activity in a host transaction. The system can also perform other corrective actions, including inputting information associated with the related events into an external system for training a machine learning model. Anomalous activity or anomalies can be delineated in the visual representation when measurements for the corresponding events are greater than a predetermined benchmark or other threshold. FIG. 6A

[0064] FIG. 7 ​The illustration depicts a computer system 700, according to one aspect of this application, that facilitates intelligent log analysis for large-scale HPC and AI systems. The computer system 700 includes a processor 702, a memory 704, and a storage device 706. The memory 704 may include volatile memory (e.g., random access memory (RAM)) that serves as managed memory and can be used to store one or more memory pools. Furthermore, the computer system 700 may be coupled to peripheral I / O user devices 710 (e.g., a display device 711, a keyboard 712, and a pointing device 713). The storage device 706 includes a non-transitory computer-readable storage medium and stores an operating system 716, instructions 718, and data 730. The computer system 700 may include a processor 702, a memory 704, and a storage device 706. FIG. 7 The entities or instructions shown are fewer or more entities or instructions.

[0065] Instruction 718 may include instructions that, when executed by computer system 700, cause computer system 700 to perform the methods and / or processes described in this disclosure. Specifically, instruction 718 may include instruction 720, which is used to obtain event information from components operating jointly in a network environment, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries against standard events, as described above. FIG. 6A Operation 602 and FIG. 3A As described in log entries 310, 320 and 330.

[0066] Instruction 718 may include instruction 722, which is used to classify events interpreted from log entries based on the topology of components in the network environment, as described above. FIG. 2 Event classifier module 216 FIG. 3C Decision tree 368 and FIG. 6A Operation 604 is described.

[0067] Instruction 718 may include instruction 724, which is used to associate two or more events based on corresponding event classifications and a predetermined time window covering the event times associated with the corresponding events, wherein the event times are obtained from log entries, and wherein the predetermined time window is determined from measurements related to power consumption, application runtime, and transaction results associated with the component, as described above. FIG. 6A Operation 604 and FIG. 5A through FIG. 5F The simplified diagram describes it.

[0068] Instruction 718 may include instruction 726, which is used to generate a visual representation (and report) indicating the associated event, as described above. FIG. 6B Operations 632 / 634 and decision 636 and FIG. 5A through FIG. 5F The simplified diagram describes it.

[0069] The instructions 718 can include instructions 728 to allow a corrective action to address an indicated anomaly in response to a visual representation indicating the anomaly, as described above with respect to the operations of FIG. 6B .

[0070] The instructions 718 can include more instructions than those shown in FIG. 7 . For example, the instructions 718 can include instructions to perform the operations described above with respect to: FIG. 2 the high-level flow of FIG. 3A through FIG. 3C the log entry collection, formatting, transformation, and classification of FIG. 4 the environment and communications of FIG. 5A through FIG. 5F the diagram of FIG. 6A the operations depicted in the flowcharts of FIG. 6B ; and FIG. 8 the instructions of the CRM 800 in .

[0071] The data 730 can include any data required as input or generated as output by the methods, operations, communications, and / or processes described in this disclosure. In particular, the data 730 can store at least: event information; entries; a first set of events interpreted from log entries; a second set of events returned from a query for a standard event; a classification; an event classification; an association between two or more events; a time window; an event time; a visual representation; a report; an anomaly indicator; an indicator or identifier of a hardware, software, or other component in or associated with a storage component, host component, or structural component in a system; raw logs or log data; extracted logs; noise; filtered logs; reformatted log entries; a characteristic of a log entry; an identity of an entity or component; a time; an event category; an event type; an event description; a data structure; information; a related event or a related query event; a report; an indicator or recommendation of an action or corrective action; and an interactive element that facilitates viewing or manipulating displayed information, including detected anomalies, recommended actions, and configurable options.

[0072] FIG. 8 FIG. illustrates a computer-readable medium (CRM) 800 that facilitates intelligent log analysis for large-scale HPC and AI systems, according to an aspect of the present application. The CRM 800 can be a non-transitory computer-readable medium or device that stores instructions that, when executed by a computer or processor, cause the computer or processor to perform a method. The CRM 800 can store instructions 810 to obtain event information from components jointly operating in a system, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from a query for a standard event, as described above with respect to the operations 602 of FIG. 6A andFIG. 3A As described in log entries 310, 320 and 330.

[0073] CRM 800 can store instruction 812, which is used to classify events interpreted from log entries based on the component hierarchy, as mentioned above. FIG. 2 Event classifier module 216 and FIG. 6A Operation 604 is described.

[0074] The CRM 800 can store instruction 814, which is used to associate two or more events based on corresponding event categories and a predetermined time window covering the event time associated with the corresponding events. The event time is obtained from log entries, and the predetermined time window is determined from measurements related to power consumption, application runtime, and transaction results associated with the component, as described above with respect to operation 604 in Figure 6. FIG. 5A through FIG. 5C The simplified diagram illustrates this. The CRM 800 can pull or extract power consumption and application host transaction metrics to identify changes and anomalies, such as those occurring within specific timeframes, as described above. FIG. 6A Operations 616 and 618 are described.

[0075] CRM 800 can store instructions 816, which are used to generate visual representations or reports indicating associated events, as mentioned above. FIG. 6B Operations 632 and 634 and FIG. 5A through FIG. 5F The simplified diagram describes it.

[0076] The CRM 800 can store instructions 818, which are used to respond to visual representations or reports indicating abnormalities and allow corrective actions to resolve the indicated abnormalities, as mentioned above. FIG. 6B The operation described.

[0077] CRM 800 can include more FIG. 8 The instructions shown are more than just instructions. For example, the CRM 800 can store instructions for performing the operations described above regarding the following: FIG. 2 High-level processes; FIG. 3A through FIG. 3C Log entries are collected, formatted, transformed, and categorized; FIG. 4 Environment and communication; FIG. 5A through FIG. 5F A simplified diagram; FIG. 6A and FIG. 6B The operations depicted in the flowchart; and FIG. 7 Instruction 718 of computer system 700.

[0078] Accordingly, the described aspects can provide improved anomaly detection across complex systems and enhanced root cause analysis capabilities. The described aspects can also provide more efficient identification of relationships between events in different subsystems and more efficient processing of different log formats and event types. Additionally, the described aspects can provide interactive user feedback for system optimization.

[0079] Generally, the disclosed aspects provide a method, computer system, and computer readable medium that facilitate intelligent log analysis for large-scale HPC and AI systems. During operation, the system obtains event information from components jointly operating in the system, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events. The system classifies the events interpreted from the log entries based on a hierarchy of the components. The system correlates two or more events based on respective event classifications and a predetermined time window covering event times associated with the respective events, the event times being derived from the log entries and the predetermined time window being determined from measurements related to power consumption, application runtime, and transaction results associated with the components. The predetermined time window can also be obtained based on detection of errors and events on components of the system, and the obtained time window can be used to search for application performance changes and power anomalies. The system generates a visual representation indicating the correlated events. In response to the visual representation indicating an anomaly, the system allows a corrective action to address the indicated anomaly.

[0080] In variations of this aspect, the components include at least one of: hardware or software associated with a storage component in the system; hardware or software associated with a host component in the system, where the host component includes one or more of: a graphics processor unit (GPU), a high-bandwidth memory (HBM), a central processing unit (CPU) or core, a CPU memory, and a peripheral component interconnect express (PCIe) component; or hardware or software associated with a fabric component of the system, where the fabric component includes one or more of: a network device, a switch, a switch agent, a centralized fabric manager, a fabric agent, and a network interface.

[0081] In further variations of this aspect, the system generates log entries indicating the first set of events by: extracting logs from one or more of the components in the system; removing noise in the extracted logs by filtering the extracted logs; obtaining reformatted log entries by reformating the filtered logs; and generating the event information based on characteristics of the reformatted log entries.

[0082] In further variants, the characteristics of the reformatted log entries include at least one of: an identity of an entity or component associated with the log entry; a time associated with an event that generated the log entry; an event category; an event type; or a description of the event.

[0083] In further variants, the system stores information associated with the first set of events and the second set of events in entries in a data structure and in a time series database, where respective entries indicate the determined event classifications and any associations with other events.

[0084] In further variants, the system queries events in the data structure associated with a first predetermined time period, where the first predetermined time period is based on at least one of: measurements related to power consumption, application runtime, and transaction results associated with the component; or detection of errors and events on components of the system. The system associates the queried events by tagging respective entries for the queried events with the same association identification tag. The system includes the associated queried events in the generated visual representation.

[0085] In further variants, the system generates and displays a report based on the associated events. The system performs a first action based on the displayed report, where the first action includes a respective corrective action to address the indicated anomaly.

[0086] In further variants, the displayed report includes one or more interactive elements that facilitate viewing or manipulating displayed information including at least one of: a detected anomaly; a recommended action indicating a remedy for the detected anomaly; or a configurable option indicating that the computer will automatically perform the recommended action.

[0087] In another aspect, a computer system includes a processor and a storage device storing instructions. The instructions are for obtaining event information from components jointly operating in a network environment, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events. The instructions are further for classifying events interpreted from the log entries based on a topology of the components in the network environment. The instructions are further for storing the log entries in a time series database. The instructions are further for correlating two or more events based on respective event classifications and a predetermined time window covering an event time associated with the respective events, where the event time is derived from the log entries and where the predetermined time window is determined from measurements related to power consumption, application runtimes, and transaction results associated with the components. The instructions are further for generating a visual representation indicating the correlated events. The instructions are further for allowing a corrective action to address an indicated anomaly in response to the visual representation indicating the anomaly. The computer system can include other instructions for performing the operations described herein, including operations depicted in the high-level flow of FIG. 2 the log entry collection, formatting, transformation, and classification of FIG. 3A through FIG. 3C the environment and communications of FIG. 4 the diagram of FIG. 5A through FIG. 5F the flowchart of FIG. 6A and FIG. 6B the operations depicted in the flowchart of FIG. 8 the instructions of the CRM 800 in

[0088] In another aspect, a non-transitory computer-readable storage medium (or CRM) stores instructions for obtaining event information from components jointly operating in a system, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events. The instructions are further for classifying events interpreted from the log entries based on a hierarchy of the components. The instructions are further for correlating two or more events based on respective event classifications and a predetermined time window covering an event time associated with the respective events, the event time derived from the log entries and the predetermined time window determined from measurements related to power consumption, application runtimes, and transaction results associated with the components. The instructions are further for generating a visual representation or report indicating the correlated events. The instructions are further for allowing a corrective action to address an indicated anomaly in response to the visual representation or report indicating the anomaly. The CRM can also store instructions for performing the operations described above with respect to: FIG. 2 the high-level flow of FIG. 3A through FIG. 3C the log entry collection, formatting, transformation, and classification of FIG. 4 the environment and communications of FIG. 5A through FIG. 5F the diagram of FIG. 6Aand FIG. 6B The operations described in the flowchart; and FIG. 7 Instruction 718 of computer system 700.

[0089] The foregoing description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are intended to be consistent with the maximum scope of the principles and features disclosed herein.

[0090] Furthermore, the foregoing descriptions of the various aspects have been presented solely for illustrative and descriptive purposes. These descriptions are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A method comprising: Event information is obtained from components that operate jointly in the system, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events; The events interpreted from the log entries are categorized based on the hierarchical structure of the components; Two or more events are associated based on the corresponding event classification and a predetermined time window that covers the event time associated with the corresponding event. The event time is obtained from the log entry, and the predetermined time window is determined from measurements related to power consumption, application runtime, and transaction results associated with the component. Generate a visual representation indicating the associated event; as well as In response to the visual indication of an anomaly, corrective actions are allowed to resolve the indicated anomaly.

2. The method as described in claim 1, wherein, The component includes at least one of the following: Hardware or software associated with the storage components in the system; Hardware or software associated with a host component in the system, wherein the host component includes one or more of the following: a graphics processing unit (GPU), high-bandwidth memory (HBM), a central processing unit (CPU) or core, CPU memory, and peripheral component fast interconnect (PCIe) components; or Hardware or software associated with structural components of the system, wherein the structural components include one or more of the following: network devices, switches, switch agents, centralized structure managers, structure agents, and network interfaces.

3. The method of claim 1, further comprising generating log entries indicating the first group of events by: Extract logs from one or more of the components in the system; Noise in the extracted logs is removed by filtering them. Obtain reformatted log entries by reformatting the filtered logs; as well as Event information is generated based on the characteristics of the reformulated log entries.

4. The method of claim 3, wherein, The characteristics of the reformulated log entries include at least one of the following: The identity of the entity or component associated with the log entry; The time associated with the event that generated the log entry; Event categories; Event type; or Description of the event.

5. The method of claim 1, further comprising: Information associated with the first group of events and the second group of events is stored in entries in a data structure and in a time series database, wherein the corresponding entries indicate the determined event classification and any association with other events.

6. The method of claim 5, further comprising: Query the events in the data structure that are associated with a first predetermined time period, wherein the first predetermined time period is based on at least one of the following: And measurements related to power consumption, application runtime, and transaction results associated with the component; or Detection of errors and events on components of the system; The queried event is associated by tagging the corresponding entries of the queried event with the same association identifier label; and Include the associated queried events in the generated visual representation.

7. The method of claim 1, further comprising: Generate reports based on associated events; Display the report; as well as Perform the first action based on the displayed report. The first action includes a corresponding corrective action to resolve the indicated anomaly.

8. The method as described in claim 7, in, The displayed report includes one or more interactive elements that facilitate viewing or manipulating the displayed information, which includes at least one of the following: Detected anomalies; Indicates recommended remedial actions for the detected anomaly; or A configurable option to instruct the computer to automatically perform the recommended action.

9. A computer system, comprising: processor; as well as A storage device that stores instructions, which, when executed by the processor, include instructions for performing the following operations: Event information is obtained from components operating jointly in a network environment, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events; The events interpreted from the log entries are classified based on the topology of the components in the network environment; Two or more events are associated based on the corresponding event classification and a predetermined time window that covers the event time associated with the corresponding event. The event time is obtained from the log entry, and the predetermined time window is determined from measurements related to power consumption, application runtime, and transaction results associated with the component. Generate visual representations indicating the associated events; and In response to the visual indication of an anomaly, corrective actions are allowed to resolve the indicated anomaly.

10. The computer system of claim 9, wherein, The component includes at least one of the following: Hardware or software associated with storage components in the network environment; Hardware or software associated with host components in the network environment, wherein the host components include one or more of the following: a graphics processing unit (GPU), high-bandwidth memory (HBM), a central processing unit (CPU) or core, CPU memory, and peripheral component rapid interconnect (PCIe) components; or Hardware or software associated with structural components of the network environment, wherein the structural components include one or more of the following: network devices, switches, switch agents, centralized structure managers that manage switches in the structure, structure agents that operate on the switches, and network interfaces, wherein the structure agents program the switches and interact with network protocol agents.

11. The computer system of claim 9, wherein the instructions are further configured to perform the following operations: Extract logs from one or more of the components in the network environment; Noise in the extracted logs is removed by filtering them. Obtain reformatted log entries by reformatting the filtered logs; as well as Event information is generated based on the characteristics of the reformulated log entries.

12. The computer system of claim 11, wherein, The characteristics of the reformulated log entries include at least one of the following: The identity of the entity or component associated with the log entry; The time associated with the event that generated the log entry; Event categories; Event type; or Description of the event.

13. The computer system of claim 9, wherein the instructions are further configured to perform the following operations: Information associated with the first group of events and the second group of events is stored in entries within a data structure and in a time-series database, wherein... The corresponding entry indicates the identified event category and any association with other events.

14. The computer system of claim 13, wherein the instructions are further configured to perform the following operations: Query the events in the data structure that are associated with a first predetermined time period, wherein, The first predetermined time period is based on measurements related to power consumption, application runtime, and transaction results associated with the component; The queried event is associated by tagging the corresponding entries of the queried event with matching association tags; as well as Include the associated queried events in the generated visual representation.

15. The computer system of claim 9, wherein the instructions are further configured to perform the following operations: Generate reports based on associated events; Display the report; and Perform the first action based on the displayed report. in, The first action includes a corresponding corrective action to resolve the indicated anomaly.

16. The computer system as described in claim 15, in, The displayed report includes one or more interactive elements that facilitate viewing or manipulating the displayed information, and the interactive elements include at least one of the following: Detected anomalies; Indicates recommended remedial actions for the detected anomalies; or A configurable option to instruct the computer to automatically perform the recommended action.

17. The computer system of claim 15, wherein the instructions are further configured to perform the following operations: In response to allowing the corrective action to resolve the anomaly indicated in the visual representation or based on the displayed report, the first action is performed: Obtain updated event information from the aforementioned component; The updated events indicated in the updated event information are classified; Two or more events are associated based on the updated event, the corresponding event category, and the predetermined time window; Regenerate a visual representation of the event to which the event is associated; as well as In response to the regenerated visual representation indicating one or more other anomalies, additional corrective actions are allowed to resolve the one or more other anomalies.

18. A non-transitory computer-readable medium storing instructions for performing the following operations: Event information is obtained from components that operate jointly in the system, the event information indicating a first set of events interpreted from log entries associated with the components and a second set of events returned from queries for standard events; The events interpreted from the log entries are categorized based on the hierarchical structure of the components; Two or more events are associated based on corresponding event classifications and a predetermined time window covering the event time associated with the corresponding events, the event time being obtained from the log entries, and the predetermined time window being determined from measurements related to power consumption, application runtime, and transaction results associated with the component. Generate visual representations or reports indicating the associated events; as well as In response to the visual representation or the report indicating an anomaly, corrective actions are allowed to resolve the indicated anomaly.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions are further configured to generate the log entries indicating the first set of events in such a manner as follows: Extract logs from one or more of the components in the system; Noise in the extracted logs is removed by filtering them. Obtain reformatted log entries by reformatting the filtered logs; as well as Event information is generated based on the characteristics of the reformulated log entries.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions are further configured to perform the following operations: Display the visual representation or the report. in, The displayed visual representation or report includes one or more interactive elements that facilitate viewing or manipulating the displayed information. The information displayed includes at least one of the following: Detected anomalies; Indicates recommended remedial actions for the detected anomaly; or Configurable options that instruct the computer to automatically perform the recommended action; and In response to allowing the corrective action to resolve the indicated anomaly: Obtain updated event information from the aforementioned component; The updated events indicated in the updated event information are classified; Two or more events are associated based on the updated event, the corresponding event category, and the predetermined time window; Regenerate a visual representation indicating the associated event; and In response to the regenerated visual representation indicating one or more other anomalies, additional corrective actions are allowed to resolve the one or more other anomalies.

Images