An AI intelligence calculation cluster fault root cause positioning method and system and a storage medium

Abstract

The application provides an AI intelligence calculation cluster fault root cause positioning method and system and a storage medium. By acquiring multi-source heterogeneous time sequence monitoring data, a consensus time axis is generated based on NPU monitoring data, and memory monitoring data and communication optical module monitoring data are aligned to the consensus time axis to carry a relative time lag index and nonlinear distortion intensity of each dimension sequence relative to the consensus time axis, and multi-dimensional time sequence data is obtained. Based on the multi-dimensional time sequence data and the physical topology structure of the cluster, an initial causal graph is constructed, the directed edges of the graph carry a causal direction and a correlation strength determined based on the relative time lag index and the nonlinear distortion intensity. The initial causal graph is input into a preset deep heterogeneous spatio-temporal graph encoder for iterative optimization to obtain an optimized global causal graph model. When an abnormal alarm is detected, the global causal graph model is used to determine a fault root cause node along a fault propagation conditional probability in a reverse random walk, thereby improving the accuracy and efficiency of fault root cause positioning.

Description

Technical Field

[0001] This disclosure relates to the field of AI intelligent computing cluster technology, and in particular to a method, system and storage medium for locating the root cause of AI intelligent computing cluster faults. Background Technology

[0002] With the rapid growth in demand for large-scale artificial intelligence models and scientific computing, high-performance AI computing clusters, consisting of thousands or even tens of thousands of GPUs / NPUs, have become the core computing infrastructure driving technological innovation. Their operational continuity is directly related to R&D progress, commercial service quality, and huge economic costs. Therefore, accurate and efficient fault location of AI computing clusters is of paramount importance.

[0003] However, current technologies for analyzing the root causes of AI computing cluster failures often employ rule-based alarm systems and expert-experience-based log analysis. These methods rely on manually predefined fixed thresholds or static rules to determine dependencies, lacking the systematic ability to trace back to the source of cascading concurrent failures. Consequently, they cannot accurately pinpoint the root cause of the chain reaction from massive amounts of alarms. Therefore, current root cause analysis methods are ill-suited to complex scenarios involving dynamically changing cluster sizes, diverse load patterns, and intertwined concurrent failures, ultimately reducing the accuracy and efficiency of AI computing cluster failure root cause analysis. Summary of the Invention

[0004] The main objective of this disclosure is to propose a method, system, and storage medium for locating the root cause of AI computing cluster faults, which can improve the accuracy and efficiency of locating the root cause of AI computing cluster faults.

[0005] To achieve the above objectives, a first aspect of this disclosure proposes a method for locating the root cause of AI computing cluster faults, comprising: Acquire multi-source heterogeneous time-series monitoring data of the target AI computing cluster, wherein the multi-source heterogeneous time-series monitoring data includes at least NPU monitoring data, memory monitoring data, and communication optical module monitoring data; A consensus timeline is generated based on the NPU monitoring data, and the memory monitoring data and the communication optical module monitoring data are aligned to the consensus timeline to carry the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline, thereby obtaining time-synchronized multidimensional time series data. An initial causal graph is constructed based on the multidimensional time-series data and the physical topology of the target AI computing cluster, so that the directed edges in the initial causal graph carry the causal direction and correlation strength determined based on the relative time delay exponent and the nonlinear distortion intensity. The initial causal graph is input into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model, which includes the fault propagation conditional probability of directed edges. When an abnormal alarm is detected, a reverse random walk is performed along the fault propagation conditional probability based on the global causal graph model to determine the root cause node of the fault.

[0006] In some embodiments, generating a consensus timeline based on the NPU monitoring data and aligning the memory monitoring data and the communication optical module monitoring data to the consensus timeline includes: The consensus time axis is generated by performing soft dynamic time warping and dynamic warping centroid averaging iteration on multiple subsequences of the NPU monitoring data. Using the consensus timeline as a global reference, dynamic programming with window constraints is used to align the sequences of each dimension of the memory monitoring data and the communication optical module monitoring data to the consensus timeline, and the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline are calculated.

[0007] In some embodiments, constructing an initial causal graph based on the multidimensional time-series data and the physical topology of the target AI computing cluster includes: Based on the physical connection relationships between the components in the target AI computing cluster, a candidate edge set for the heterogeneous graph is constructed. Based on the cross-correlation function with a penalty term constrained by the relative time delay index, the causal direction and peak correlation coefficient of each candidate edge are determined within the preset time delay search range; Based on the tolerance parameter threshold generated by random rearrangement of the sequence, the candidate edge set is subjected to saliency filtering and causal orientation pruning to obtain the initial causal graph.

[0008] In some embodiments, the step of inputting the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model includes: Nonlinear mapping is performed on the node features and edge features in the initial causal graph to obtain node embedding vectors and edge embedding vectors, wherein the edge features include the relative time delay exponent, the nonlinear distortion intensity, and the cross-correlation peak value; Based on the edge embedding vector, the node embedding vector is aggregated and updated in multiple layers through a multi-head graph attention mechanism to obtain the top-level node embedding. Based on the top-level node embedding and the edge embedding vector, the fault propagation conditional probability of each directed edge is predicted by the bilinear gated probability inference network, and end-to-end training is performed based on historical fault samples to obtain a global causal graph model.

[0009] In some embodiments, the step of performing multi-level aggregation and updating of the node embedding vector based on the edge embedding vector using a multi-head graph attention mechanism includes: The edge embedding vector is injected into the attention score calculation of the multi-head graph attention mechanism. After applying attention weighting to the pseudo-propagation link with high nonlinear distortion intensity and long time delay, the node embedding vector is updated by multi-level aggregation.

[0010] In some embodiments, the end-to-end training based on historical fault samples to obtain a global causal graph model includes: A binary classification mask is constructed based on historical fault samples. The deep heterogeneous spatiotemporal graph encoder is trained end-to-end using a binary cross-entropy loss function combined with a sparse regularization term. Edge pruning is performed according to a pre-set threshold to obtain a global causal graph model.

[0011] In some embodiments, determining the root cause node of the fault by performing a reverse random walk along the fault propagation conditional probability based on the global causal graph model includes: The detected abnormal alarms are mapped to the abnormal impact vectors of the corresponding nodes in the global causal graph model, and an inverse state transition matrix is ​​constructed based on the fault propagation conditional probability and the nonlinear distortion intensity. Based on the inverse state transition matrix, a Markov random walk with restart is performed to iteratively solve for the steady-state distribution vector; The root cause node of the fault is determined by sorting the scores of each node in the steady-state distribution vector.

[0012] In some embodiments, determining the root cause node of the fault based on the score ranking of each node in the steady-state distribution vector includes: Prioritize examining nodes not included in the initial anomaly observation set. If there is a node that has not been alerted and has the highest steady-state score, then that node is identified as the root cause of the fault. After determining the root cause node of the fault, the corresponding fault propagation path is extracted by extending along the positive maximum probability path to the abnormal observation set corresponding to the abnormal alarm on the global cause-effect graph model, starting from the root cause node of the fault.

[0013] To achieve the above objectives, a second aspect of this disclosure provides an AI intelligent computing cluster fault root cause localization system, comprising: The data acquisition module is used to acquire multi-source heterogeneous time-series monitoring data of the target AI computing cluster. The multi-source heterogeneous time-series monitoring data includes at least NPU monitoring data, memory monitoring data and communication optical module monitoring data. The timing alignment module is used to generate a consensus timeline based on the NPU monitoring data, and to align the memory monitoring data and the communication optical module monitoring data to the consensus timeline, so as to carry the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline, and obtain time-synchronized multidimensional timing data; The causal graph construction module is used to construct an initial causal graph based on the multidimensional time-series data and the physical topology of the target AI computing cluster, so that the directed edges in the initial causal graph carry the causal direction and correlation strength determined based on the relative time delay exponent and the nonlinear distortion intensity. The graph optimization module is used to input the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model, wherein the global causal graph model contains the fault propagation conditional probability of directed edges. The root cause localization module is used to determine the root cause node of the fault by performing a reverse random walk along the fault propagation conditional probability based on the global causal graph model when an abnormal alarm is detected.

[0014] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the AI ​​intelligent computing cluster fault root cause localization method described in the first aspect embodiment.

[0015] To achieve the above objectives, a fourth aspect of the present disclosure provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the AI ​​intelligent computing cluster fault root cause localization method described in the first aspect embodiment.

[0016] The beneficial effects of the embodiments disclosed herein include: By acquiring multi-source heterogeneous time-series monitoring data and generating a consensus time axis based on NPU monitoring data, and aligning other dimensions of monitoring data to this time axis while carrying relative time delay exponents and nonlinear distortion strength, accurate time synchronization and feature processing of multi-source heterogeneous monitoring data are achieved. Secondly, based on the time-synchronized multi-dimensional time-series data and the cluster physical topology, an initial causal graph is constructed, enabling directed edges to carry causal directions and correlation strengths determined by time delay and distortion strength. This data-driven causal relationship modeling replaces manually predefined static rules and fixed thresholds, enabling adaptive adaptation to dynamic changes in cluster size and load patterns. Subsequently, a deep heterogeneous spatiotemporal graph encoder is used to process the initial... The causal graph is iteratively optimized to obtain a global causal graph model carrying the conditional probability of fault propagation. This further mines and quantifies the propagation path and probabilistic relationship of faults among nodes in the cluster, which can effectively deal with complex scenarios of concurrent faults. Finally, when an abnormal alarm is detected, a reverse random walk is performed based on the global causal graph model along the conditional probability of fault propagation. The system traces back from the alarm node to the fault source node, which has the ability to systematically trace back the concurrent cascading faults. It can accurately locate the root cause of the chain reaction from massive alarms without the need for manual inspection of massive logs, thereby improving the accuracy and efficiency of fault root cause location in AI intelligent computing clusters. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the AI ​​intelligent computing cluster fault root cause localization method provided in this embodiment of the disclosure; Figure 2 yes Figure 1 The process diagram further includes step S102; Figure 3 yes Figure 1 The process diagram further includes step S103; Figure 4 This is a schematic diagram of the heterogeneous graph structure provided in the embodiments of this disclosure; Figure 5 yes Figure 1 The flowchart further includes step S104; Figure 6 This is a schematic diagram of a graph neural network optimized causal graph model provided in an embodiment of this disclosure; Figure 7 yes Figure 1 The process diagram further includes step S105; Figure 8 yes Figure 7 The process diagram further includes step S503; Figure 9 This is a schematic diagram of the functional modules of the AI ​​intelligent computing cluster fault root cause localization system provided in this embodiment of the disclosure; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0018] The accompanying drawings in the embodiments clearly and completely describe the technical solutions in the embodiments of this disclosure. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] It is understood that in the specific embodiments of this disclosure, multi-source heterogeneous timing monitoring data such as NPU monitoring data, memory monitoring data and communication optical module monitoring data, system monitoring data and related data are involved. When the above embodiments of this disclosure are applied to specific products or technologies, permission or consent can be obtained from the target, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards.

[0020] Furthermore, when this embodiment of the disclosure needs to retrieve multi-source heterogeneous timing monitoring data such as NPU monitoring data, memory monitoring data, and communication optical module monitoring data, as well as system monitoring data and related data, separate permission or separate consent can be obtained through pop-up windows or redirection to a confirmation page. After explicitly obtaining separate permission or separate consent for the multi-source heterogeneous timing monitoring data such as NPU monitoring data, memory monitoring data, and communication optical module monitoring data, as well as system monitoring data and related data, the necessary multi-source heterogeneous timing monitoring data such as NPU monitoring data, memory monitoring data, and communication optical module monitoring data, as well as system monitoring data and related data for the normal operation of this embodiment of the disclosure, the necessary multi-source heterogeneous timing monitoring data such as NPU monitoring data, memory monitoring data, and communication optical module monitoring data, as well as system monitoring data and related data can then be obtained.

[0021] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0022] In recent years, with the rapid growth in demand for large-scale artificial intelligence models and scientific computing, high-performance artificial intelligence (AI) computing clusters, composed of thousands or even tens of thousands of Graphics Processing Units (GPUs) / Neural Processing Units (NPUs), have become the core computing infrastructure driving technological innovation. These clusters support the training and inference of models with hundreds of billions or even trillions of parameters, and their operational continuity directly affects R&D progress, commercial service quality, and enormous economic costs. Any single hardware failure, link jitter, or software anomaly within the cluster can trigger a chain reaction, causing serious consequences such as training task interruption and data loss. For example, in a large-scale pre-training cluster, a silent data error in a single GPU failed to be located in time, and the erroneous gradient contaminated the entire model through the parameter synchronization process, ultimately forcing a training rollback lasting several days and wasting hundreds of thousands of GPU hours. Therefore, how to build operational and maintenance methods that can quickly trace the source, block the impact, and ensure cluster resilience, based on the mechanisms of fault occurrence and propagation, has become a pressing challenge for the industry.

[0023] However, current technologies for analyzing the root causes of AI computing cluster failures often employ rule-based alarm systems and expert-experience-based log analysis. These methods rely on manually predefined fixed thresholds or static rules to determine dependencies, lacking the systematic ability to trace back to the source of cascading concurrent failures. Consequently, they cannot accurately pinpoint the root cause of the chain reaction from massive amounts of alarms. Therefore, current root cause analysis methods are ill-suited to complex scenarios involving dynamically changing cluster sizes, diverse load patterns, and intertwined concurrent failures, ultimately reducing the accuracy and efficiency of AI computing cluster failure root cause analysis.

[0024] Based on this, the present disclosure provides a method, system, and storage medium for locating the root cause of AI computing cluster faults, which can improve the accuracy and efficiency of locating the root cause of AI computing cluster faults.

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating the AI ​​computing cluster fault root cause localization method provided in this embodiment. This AI computing cluster fault root cause localization method can be applied in an AI computing cluster fault root cause localization system. The AI ​​computing cluster fault root cause localization method includes steps S101 to S105: Step S101: Obtain multi-source heterogeneous time-series monitoring data of the target AI computing cluster. The multi-source heterogeneous time-series monitoring data includes at least NPU monitoring data, memory monitoring data, and communication optical module monitoring data. Step S102: Generate a consensus timeline based on NPU monitoring data, and align memory monitoring data and communication optical module monitoring data to the consensus timeline to carry the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline, thereby obtaining time-synchronized multidimensional time series data; Step S103: Construct an initial causal graph based on multi-dimensional time series data and the physical topology of the target AI computing cluster, so that the directed edges in the initial causal graph carry the causal direction and correlation strength determined based on the relative time delay exponent and nonlinear distortion strength. Step S104: Input the initial causal graph into the preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain the optimized global causal graph model. The global causal graph model contains the fault propagation conditional probability of directed edges. Step S105: When an abnormal alarm is detected, a reverse random walk is performed along the fault propagation conditional probability based on the global causal graph model to determine the root cause node of the fault.

[0026] Regarding step S101 above, multi-source heterogeneous timing monitoring data refers to a set of time-attributed monitoring indicators from different data acquisition channels covering the same target AI computing cluster. Multi-source heterogeneous timing monitoring data includes at least NPU monitoring data, memory monitoring data, and communication optical module monitoring data. Specifically, NPU monitoring data are timing indicators captured at a high sampling frequency by the in-band management interface or exporter, reflecting the real-time operating status of the computing core. These indicators may include core utilization, memory bandwidth utilization, temperature, PCIe retransmission count, SM active percentage, clock frequency, etc., with a reference sampling frequency of approximately 10Hz. Memory monitoring data are storage module operating indicators collected by the node-side exporter or baseboard management controller, and may include bandwidth utilization, Error Correcting Code (ECC) count, chip temperature, memory access request latency, etc., with a reference sampling frequency of approximately 1Hz. The communication optical module monitoring data consists of optical link status indicators collected by the switch telemetry channel or network interface. These indicators may include RX receive / TX transmit optical power, real-time bit error rate, link packet loss count, transmitted and received bytes, link latency, etc., with a reference sampling frequency of approximately 0.1Hz to 1Hz. These data originate from various sources, including iBMC, eBPF, DCGM-Exporter, in-band management interface, and switch telemetry channel. Each record carries a timestamp generated by a local clock, resulting in uneven sampling frequency distribution and clock drift.

[0027] It should be noted that the embodiments disclosed herein provide a complete data foundation for subsequent full-process processing from time series alignment, causal mapping to root cause localization by comprehensively acquiring multi-source heterogeneous time series monitoring data covering the three key subsystems of computing, storage and network. This avoids the one-sidedness of traditional methods that rely solely on monitoring data from a single type of log or isolated components.

[0028] Regarding step S102 above, this embodiment generates a consensus timeline based on NPU monitoring data and aligns memory monitoring data and communication optical module monitoring data to the consensus timeline. This aligns the memory monitoring data and communication optical module monitoring data to the consensus timeline, carrying the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline, thus obtaining time-synchronized multidimensional time-series data. Specifically, since the clocks of each acquisition source drift and the sampling times are not aligned, the original sequence must undergo unified preprocessing before entering causal analysis. First, a suitable window value is selected, for example, 5. Then, a median filter with a window of 5 is applied to remove isolated spikes caused by acquisition jitter, avoiding their interference with subsequent alignment and correlation calculation. For short-term missing data of no more than 5 consecutive points, linear interpolation is used to fill in the gaps. Continuous missing segments exceeding this length are marked as invalid intervals and will not be matched during subsequent time-series alignment. Next, for each univariate sequence... Implement Z-score standardization:

[0029] in, Let be the observation value of the original sequence at time step t, μ be the mean of the sequence, and σ be the standard deviation of the sequence. This is the standardized value. This process eliminates the influence of differences in the dimensions of various indicators on the distance metric, ensuring that the Euclidean distance in dynamic time warping can fairly integrate different physical quantities.

[0030] Further, please refer to Figure 2 , Figure 2 yes Figure 1 The flowchart further includes step S102. In one embodiment, the process of generating a consensus timeline based on NPU monitoring data and aligning memory monitoring data and communication optical module monitoring data to the consensus timeline may further include steps S201 to S202: Step S201: Perform soft dynamic time warping and dynamic warping centroid averaging iteration on multiple subsequences of NPU monitoring data to generate a consensus timeline; In step S202, taking the consensus timeline as a global reference, dynamic programming with window constraints is used to align the sequences of each dimension of the memory monitoring data and the communication optical module monitoring data to the consensus timeline, and calculate the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline.

[0031] In the above steps, this embodiment of the disclosure uses the NPU sequence, which is closely affected by the System Condition Parameter (SCP) and has the highest sampling frequency, as the alignment anchor. However, the NPU itself contains multiple sub-sequences, each with different sampling frequencies and response characteristics, making it impossible to directly select a single sequence as a benchmark. Therefore, internal self-alignment within the NPU is performed first to generate a consensus timeline.

[0032] For example, let the p-th subsequence of the NPU be... p = 1, 2, ..., P, where p = 1, 2, ..., P, and P is the total number of NPU subsequences. Let p be the length of the p-th subsequence, and the goal is to generate a consensus timeline. This minimizes the soft alignment cost of all NPU subsequences to c. A Soft Dynamic Time Warping (Soft-DTW) strategy is employed.

[0033] Soft-DTW considers all possible alignment paths through the softmin operator:

[0034] Where A is the set of all aligned paths that satisfy the monotonicity and continuity constraints, and C is the cost matrix, whose elements are... This represents the Euclidean distance between the i-th element of sequence x and the j-th element of sequence y, where β>0 is a smoothing parameter. It degenerates into the classic DTW, and the finite β-time gradient can be backpropagated through the smoothness of softmin, supporting iterative optimization.

[0035] Subsequently, a Dynamically Regularized Barycenter Averaging (DBA) iterative update rule is adopted: for the k-th time point on the consensus axis, the points mapped to k in all subsequences are collected and averaged.

[0036] in, Let c(t) be the optimal normalized path from the p-th subsequence in the t-th iteration. The indicator function takes a value of 1 when the mapping target index j equals the consensus axis index k, and 0 otherwise. Iteration continues until convergence, yielding the NPU consensus time axis. .

[0037] Next, with As a global reference timeline, the sequences of other components such as memory and communication optical modules are independently aligned to... Let the d-th dimension sequence of component s be... ,in Given the sequence length, a first-order difference transformation is first performed on each dimension to eliminate baseline drift: Then we have an independent dynamic programming recursion for the d-th dimension:

[0038] in, This represents the cumulative minimum cost of aligning the first i differences of the source sequence with the first j differences of the consensus axis. To prevent excessive path distortion, a window constraint mechanism is employed.

[0039] in, The window width is set based on the physical latency characteristics between components s and the NPU, such as the latency window between memory and the NPU. It can reach approximately 30 seconds, while the delay window between the optical module and the NPU... The time limit is then relaxed to 60 seconds. The optimal normalized path for each module is obtained through backtracking:

[0040] in, This represents the total number of mapping point pairs in the optimal regularized path. For source sequence index, For consensus axis indexing. Based on the path. Linear interpolation resampling is performed on each sequence, and the sequences are uniformly mapped to the NPU consensus timeline. Multidimensional time-series data with time synchronization is formed on a uniformly spaced time grid. Based on this, regularized paths in each dimension are calculated and retained as attribution priors. The relative time delay exponent of the d-th dimension of component s is defined as follows:

[0041] in, This indicates that this dimension lags behind the NPU consensus axis. This indicates that the time lag is ahead of its time. In subsequent causal inferences based on time delays, As a priori time delay constraint, the space of association search is reduced to... The narrow band centered on the root cause significantly improves the efficiency of root cause localization.

[0042] Simultaneously, the nonlinear torsion intensity is defined as follows:

[0043] in, This reflects the degree of nonlinearity of time drift in this dimension. A larger value indicates a less consistent pattern of change in that dimension with the NPU consensus axis, potentially suggesting an independent fault propagation path or measurement anomalies. The steps described above ensure that each sampled value of the cross-component metrics corresponds one-to-one in time, while providing an accurate time reference, directed time-delay priors, and nonlinear distortion metrics for subsequent causal inference.

[0044] It should be noted that the embodiments of this disclosure use high-frequency NPU sequences as anchors and utilize soft dynamic time warping and independent dynamic programming recursion with window constraints to complete the unified time grid reconstruction. This not only completely solves the barrier to synchronization of multi-source heterogeneous data, but also creatively extracts the relative time delay exponent and nonlinear distortion intensity of cross-dimensional correlation, providing accurate and self-consistent physical and spatiotemporal priors for subsequent time synchronization, path denoising and causal inference.

[0045] Regarding step S103 above, this embodiment of the disclosure constructs an initial causal graph based on multi-dimensional time-series data and the physical topology of the target AI computing cluster, so that the directed edges in the initial causal graph carry causal directions and correlation strengths determined based on relative time delay exponents and nonlinear distortion strength. Specifically, after obtaining the NPU consensus timeline... The time synchronization dataset as a benchmark, and the relative time delay exponent of other components. and nonlinear torsional strength Subsequently, the aforementioned data-driven dynamic temporal characteristics and the cluster's physical topology need to be integrated and introduced as the spatial skeleton for fault propagation, initializing the causal direction and intensity of fault propagation, thereby constructing a heterogeneous graph. .

[0046] Further, please refer to Figure 3 , Figure 3 yes Figure 1 The flowchart further includes step S103. In one embodiment, the process of constructing the initial causal graph based on multi-dimensional time-series data and the physical topology of the target AI computing cluster may also include steps S301 to S303: Step S301: Based on the physical connection relationships between the components in the target AI computing cluster, construct a candidate edge set for the heterogeneous graph; Step S302: Based on the cross-correlation function with penalty term constrained by relative time delay exponential constraint, determine the causal direction and peak correlation coefficient of each candidate edge within the preset time delay search range; Step S303: Based on the tolerance parameter threshold generated by random rearrangement of the sequence, the candidate edge set is subjected to saliency filtering and causal orientation pruning to obtain the initial causal graph.

[0047] In the steps described above, the physical connections between clusters define the static dependencies between components, limiting the prior paths for fault propagation. First, define the node set. Nodes are categorized into different types based on their hardware functions, including computing core nodes, storage module nodes, and network module nodes, and a set of directed edges is defined. Candidate physical edges are established based on the hardware architecture. These edges are considered if core computing components within the same node are interconnected via internal high-speed interconnect channels (either fully interconnected or mesh-interconnected); if there is a memory access / read / write dependency between the storage component and its corresponding core computing component; and if there is an input / output data dependency between the computing node's network interface or optical module and the core computing component. (See diagram...) In this context, for any pair of nodes with direct physical connections... By default, a pair of bidirectional candidate edges are initialized. and .

[0048] Traditional data-driven methods typically operate within a huge global time delay space. Blindly searching for the order of anomalies within the system is highly susceptible to noise interference, leading to causal inversion. Here, s represents the number of component types, D represents the dimension of each component type, and T represents the sequence length. This embodiment utilizes the relative time delay exponent τ extracted through the aforementioned cross-component regularization as a strong spatial prior. Let the detection sequence of node u be... The detection sequence for node v is Within the sliding observation window, a master-slave reference relationship is established based on the physical hierarchy. Taking the time characteristics of abnormal fluctuations in node u as a reference, a cross-correlation function with a penalty term is defined. Used to evaluate the time dependence of a signal propagation from u to v:

[0049] in, Let E[] represent the mathematical expectation, representing the search delay. and Let be the sequence values ​​of nodes u and v at time t, respectively. and Let be the mean of the sequences of nodes u and v, respectively. and , where represents the standard deviation of the corresponding sequence, and η is the penalty coefficient for controlling the bandwidth. Penalty term It is the relative time lag index derived from the previous stage. The introduced narrowband center constraint forces the correlation extremum search to be performed within the expected physical conduction delay window. Within a given delay search range... Searching for the range [-T, T] such that Optimal delay at maximum absolute value and its peak correlation coefficient :

[0050] like Exceeding the system's set tolerance parameter (TP) threshold. Then determine the physical link There exists a valid temporal causal relationship, and based on The positive and negative values ​​of the candidate edge set Perform pruning and orientation, when When this occurs, it indicates that the abnormal state change of node u precedes that of node v, thus activating the directed edge. And cut off the reverse side. ;when Conversely, activate and retain the reverse edge. ;when =0 indicates that the changes occur synchronously, and the bidirectional edges are retained and handed over to the subsequent graph encoder for further judgment based on the network's elastic unwinding efficiency.

[0051] Among them, the tolerance parameter threshold set by the system It is not a static constant, but rather dynamically generated based on nonparametric statistical tests of random sequence rearrangements. Specifically, for the time series of nodes u and v, it maintains... The order remains unchanged, Time step Sub-independent random shuffling generates a set of random rearranged sequences. For each rearranged sequence, the maximum absolute value of the cross-correlation coefficient is calculated to form the set of extreme values ​​of the null hypothesis background noise distribution. Tolerance parameter threshold Take the first value as the background noise distribution. Quantiles:

[0052] Here, α represents the significance level. This dynamic thresholding mechanism effectively filters out pseudo-correlated edges caused by the overall periodic load of the cluster or the drift of the background environment.

[0053] In the deep encoding stage of graph neural networks, directed edges not only need to indicate the existence of propagation paths, but also need to quantify the initial association strength. This embodiment transforms the byproducts of dynamic regularization into initial edge attribute vectors. For each one retained in Directed edges in Its initial edge feature vector is defined as:

[0054] in, The peak value of the linear correlation indicating abnormal fluctuations; The lag time for the propagation of experience after alignment; This represents the nonlinear twist strength calculated in the previous stage. A larger value indicates poorer temporal synchronization between the two nodes, suggesting the propagation link may be subject to other concurrent interferences. This is achieved through negative exponential mapping. This is transformed into a reliability coefficient within the interval (0,1], where γ>0 is a scaling scalar, serving as a penalty characterizing the stability of the edge propagation path. Therefore, the heterogeneous graph structure can be obtained as follows: Figure 4 As shown.

[0055] Please see Figure 4 The heterogeneous graph includes server node 1, server node 2, and a cross-node interconnection switch. Server node 1 contains computing core nodes NPU1 and NPU2, storage module node Memory 1, and network module node Optical Module 1. Server node 2 contains storage module node Memory 2, network module node Optical Module 2, and computing core node NPU3. The graph initializes bidirectional candidate edges based on the physical connections between components, denoted as "communication." The graph also illustrates the causal directional pruning results after temporal causality analysis and significance filtering: the cross-node candidate edge between NPU2 in server node 1 and Memory 2 in server node 2 is pruned due to latency deviations from expectations or correlation not reaching the significance threshold, denoted as "pruning." The effective propagation link between Memory 2 and NPU3 is retained after directional pruning, denoted as "communication." This refines the set of fully connected candidate edges into a sparse directed initial causal graph that combines physical topological constraints and temporal causal priors.

[0056] The aforementioned physical topological dependencies and time-delay causal relationships are ultimately encapsulated into a structured matrix form, serving as the fundamental framework for graph pattern learning. Assuming the total number of nodes in the cluster graph is N, we first define the full set of physical candidate feature matrices. Its elements The initial attribute calculation values ​​for the corresponding physically reachable links are shown below. The diagonal elements have no physical transmission significance and are set to zero vectors.

[0057] After the aforementioned threshold based on tolerance parameters Saliency filtering and optimal delay-based After causal-oriented pruning, edges that do not reach the significance threshold or whose causal direction violates the time-delay rule will be removed, and the fully connected candidate matrix will be pruned into a sparse directed initial causal feature matrix. Its form evolved as follows:

[0058] Thus, the initial causal graph, which integrates the physical connection framework and temporally warped prior information, is now complete. Construction complete. (Image) Each node Bearing node state characteristics reconstructed based on consensus timeline And each edge The initial eigenvector representing the conduction delay and reliability. This heterogeneous spatiotemporal graph precisely defines the possibility space of fault propagation, providing an input structure with a complete physical causal basis for subsequent optimization of the effective distance and recovery probability of elastic propagation by heterogeneous graph neural networks.

[0059] It should be noted that the embodiments of this disclosure introduce the relative time delay exponent as a strong spatial center constraint into the cross-correlation function search with a penalty term, and use nonparametric statistical tests based on independent random rearrangement of sequences to dynamically generate the significance threshold of the tolerance parameter. This can automatically reconcile and prune insignificant and reverse edges that violate the time delay law, and efficiently prune the fully connected matrix into a highly reliable sparse directed initial causal feature matrix, effectively solving the problem of spurious correlation and causal inversion caused by normal load fluctuations.

[0060] Regarding step S104 above, this embodiment of the disclosure inputs the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization, obtaining an optimized global causal graph model. The global causal graph model includes the fault propagation conditional probabilities of directed edges. Specifically, since the initial correlation only reflects the pairwise linear coupling between variables, it cannot eliminate complex cascading confusion effects, such as the spurious correlation between A and C caused by A→B→C. Therefore, this embodiment of the disclosure introduces a customized deep heterogeneous spatiotemporal graph feature encoder, jointly mapping node features and edge prior features, and designing an edge probability predictor to quantify the resilient recovery efficiency parameter (RREP) of anomaly propagation, thereby refining the initial graph into a rigorous causal propagation model.

[0061] Further, please refer to Figure 5 , Figure 5 yes Figure 1 The flowchart further includes step S104. In one embodiment, the process of inputting the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain the optimized global causal graph model may further include steps S401 to S403: Step S401: Perform nonlinear mapping on the node features and edge features in the initial causal graph to obtain node embedding vectors and edge embedding vectors, wherein the edge features include relative time delay exponent, nonlinear distortion intensity and cross-correlation peak. Step S402: Based on the edge embedding vector, the node embedding vector is updated by multi-level aggregation through a multi-head graph attention mechanism to obtain the top-level node embedding. Step S403: Based on the top-level node embedding and edge embedding vectors, the fault propagation conditional probability of each directed edge is predicted by the bilinear gated probability inference network, and end-to-end training is performed based on historical fault samples to obtain a global causal graph model.

[0062] In the above steps, in order to eliminate the heterogeneity of different functional components in the feature dimension, an observation window ending with the current data is selected for the feature nodes. Application-specific hardware type projection matrix Dimensionality reduction mapping is performed to obtain a unified initial embedding of hidden space nodes:

[0063] in, Indicates the hardware type to which node v belongs. For type-specific projection matrices, For the corresponding bias vector, This is a linear unit activation function with leakage correction, used to introduce nonlinear expressive power. Simultaneously, the initial causal graph... Each directed edge in Carrying physical priors extracted during the time warping phase To enable the graph network to extract time delay and nonlinear distortion information, an independent fully connected nonlinear transformation is performed on the edge features to generate edge embedding vectors:

[0064] in, The edge feature transformation weight matrix, Let be the edge feature bias vector, and ReLU be the modified linear unit activation function.

[0065] This disclosure employs a stacked multi-head graph attention architecture to iteratively update node embeddings. Unlike traditional methods that rely solely on node features to calculate attention, this disclosure mandates edge embedding. Inject an attention generation mechanism to punish high distortion. and extremely long time delay The pseudo-propagation link. In the first In the feature aggregation of the layer, the association score between node v and its predecessor neighbor u under relation type r. The calculation is as follows:

[0066] in, For the first Layer node feature weight matrix, For relation type r in the th... The attention parameter vector of the layer, This represents a vector concatenation operation. and For nodes v and u respectively, at the th... The embedding vectors at layer -1 are then used for local normalization within the neighbor node set to obtain the dynamic attention coefficients. :

[0067] in, Let v represent the set of neighboring nodes of node v under relation type r. Finally, combining a multi-head mechanism with k attention heads, the neighbor node features and edge embeddings are collaboratively aggregated to update the state representation of node v:

[0068] in, For the first The value transformation matrix of the k-th head of the layer. Here, ⨀ represents the gated weight matrix, and ⨀ represents the Hadamard product, indicating element-wise multiplication. The gated weights are generated through edge embedding. Modulate the physical delay dimension of neighbor information transmission. This is the activation function for the exponential linear unit. After... After layer aggregation, the top layer outputs It deeply integrates network resilience efficiency with fault context information under physical space constraints.

[0069] It should be noted that, in this embodiment of the present disclosure, by injecting edge embedding vectors into the attention score calculation of the multi-head graph attention mechanism, attention weighting is applied to pseudo-propagation links with high nonlinear distortion intensity and long time delay, thereby achieving automatic stripping of redundant propagation paths.

[0070] After obtaining the top-level node embedding that incorporates the local propagation context, considering the high sensitivity of system-level fault propagation to gradient differences in software and hardware states and prior physical time delays, this embodiment designs a bilinear gated probabilistic inference network to quantize each directed edge. The probability of abnormal propagation. First, the interaction between source node u and target node v in the latent state space is explicitly captured using a bilinear mapping, and the interaction score of the structure tensor is calculated:

[0071] in, is a learnable bilinear cross kernel matrix, and d is the hidden layer dimension, used to extract the nonlinear coupling features of anomalous signals between source-sink node pairs; Let be the cascaded weight matrix, and b be the bias term. Secondly, to prevent the deep network from overfitting or losing initial physical constraints during feature aggregation, an initial prior feature vector extracted during the temporal reconstruction stage is introduced. Construct a nonlinear gated adjustment factor:

[0072] in, The weight matrix is ​​transformed from prior features. This gating factor generates a theoretical interaction score by evaluating the physical time delay deviation and nonlinear distortion strength of the link. The penalty or amplification mapping is applied. Finally, the bilinear interaction features are multiplied by the prior gated features using the Hadamard product, and the standardized probability is output through the Sigmoid function:

[0073] in, This is a multilayer perceptron scoring network, where sigmoid is the sigmoid activation function. This is the elastic recovery efficiency parameter (RREP), which accurately characterizes the probability of an anomaly occurring between two nodes with this type of physical topology and time difference characteristics.

[0074] At the model training level, the system uses historical fault samples to construct a binary classification mask. ,in =1 indicates that edge e is a real fault propagation edge. =0 indicates a non-propagating edge. The binary cross-entropy loss function is used for end-to-end optimization of the network.

[0075] in, Let be the number of edges in the initial causal graph. Let be the predicted propagation probability of edge e, and λ be a sparse control hyperparameter used to suppress the weights of non-causal edges, achieving graph sparsity reduction and noise reduction. After training convergence, hard edge pruning is performed according to a set threshold, finally outputting a refined global causal graph model. ,in The optimized set of edge propagation probabilities serves as the decision framework for the source tracing phase. The entire optimization process is as follows: Figure 6 As shown.

[0076] Please see Figure 6The encoder first performs a joint spatial mapping of nodes and edge features (edge ​​features) in the initial causal graph. Node features undergo dimensionality reduction mapping, and edge features undergo feature mapping, resulting in a unified hidden space mapping representation and edge weight vectors. Subsequently, the edge weight vectors are injected into the spatiotemporal attention feature aggregation layer. In the attention score calculation, pseudo-propagation links with high nonlinear distortion intensity and long time delay are weighted. The gating weights generated by the edge weight vectors modulate the physical delay dimension of neighbor information transmission. After multi-layer iterative aggregation, a top-level node embedding that integrates local propagation context is obtained. Finally, based on the top-level node embeddings and edge weight vectors, the fault propagation conditional probability of each directed edge is predicted by the bilinear gating probability inference network. End-to-end joint training and edge pruning are performed based on historical fault samples using binary cross-entropy loss combined with sparse regularization, outputting a refined global causal graph model.

[0077] It should be noted that the embodiments of this disclosure construct a customized deep heterogeneous graph spatiotemporal encoder, inject the initial nonlinear distortion intensity and physical prior as edge attributes into the attention aggregation architecture, and use bilinear gating probability inference network to predict the conditional probability of fault propagation. Combined with sparse regularization for end-to-end training, it can automatically converge from complex concurrent cascading faults and strip away redundant pseudo-concurrent paths to generate a denoised and optimized global causal graph structure.

[0078] Regarding step S105 above, in this embodiment of the present disclosure, when an abnormal alarm is detected, a reverse random walk is performed along the fault propagation conditional probability based on a global causal graph model to determine the root cause node of the fault. Specifically, when the online system generates multi-source heterogeneous alarms, it is difficult to accurately locate the root cause by relying solely on the forward propagation probability, because downstream affected nodes often trigger a large number of cascading alarms, resulting in a reversed cause-effect relationship.

[0079] Further, please refer to Figure 7 , Figure 7 yes Figure 1 The flowchart further includes step S105. In one embodiment, the process of determining the root cause node of the fault by performing a reverse random walk along the conditional probability of fault propagation based on a global causal graph model may further include steps S501 to S503: Step S501: The detected abnormal alarms are mapped to the abnormal impact vectors of the corresponding nodes in the global causal graph model, and an inverse state transition matrix is ​​constructed based on the fault propagation conditional probability and nonlinear distortion intensity. Step S502: Perform a Markov random walk with restart based on the inverse state transition matrix to iteratively solve the steady-state distribution vector; Step S503: Determine the root cause node of the fault based on the score ranking of each node in the steady-state distribution vector.

[0080] In the above steps, when a system performance collapses, the out-of-limit alarms of each functional component form an abnormal observation set O. For any... The normalized anomalous impact score is calculated based on its deviation from the healthy baseline of the System Condition Parameter (SCP). ,satisfy thereby constructing dimensions as The prior impact vector q is set to zero for non-alarm nodes. To trace the source backwards in the causal chain, the forward graph is... Transform it into a reverse reasoning graph and introduce a path stability penalty. For each directed path... Its reverse tracing transfer weight Calculated based on normalized forward conductivity and reliability coefficient:

[0081] in, This represents the conditional probability of forward fault propagation. Nonlinear torsional strength, >0 indicates a scaling scalar, and the denominator is... The normalization factor ensures that the sum of the transition probabilities from node v to all its upstream neighbors is 1. The above formula penalizes the reverse transition rate through a nonlinear distortion intensity σ. If the sequence synchronization in the original forward propagation is extremely poor, i.e., σ is maximum, then the system refuses to attribute a large number of anomalies to that channel in the reverse propagation. (This is based on the entire...) Construct a reverse state transition matrix with column sums of 1 .

[0082] In the transition matrix A random walk with restart (RWR) is performed. To simulate the tendency to focus on the first alarm source in real-world operations and maintenance, the restart probability is set to c∈(0,1), and the Markov chain iterative equation of the system state distribution vector r is:

[0083] Where r(t) is the state distribution vector of the t-th iteration, and the superscript T denotes matrix transpose. The physical meaning of this iterative process is that each step of the tracing deduction explores upstream along the stability-weighted reverse causal chain with probability c, and directly jumps back to the core alarm node set defined by q with probability 1-c, preventing it from wandering and spreading to meaningless edge nodes. Because... To normalize the Markov transition matrix, the system must converge to a unique steady-state distribution, and its analytical solution is:

[0084] in, Let V be the identity matrix of |V| × |V|. Vector Components in It measures the global stationary source tracing score of the flow eventually reaching node v from all abnormal end nodes along a stable and high-probability reverse link. The higher the node, the greater the likelihood that it is the root cause of system damage under the current topology.

[0085] Further, please refer to Figure 8 , Figure 8 yes Figure 7 The flowchart further includes step S503. In one embodiment, the process of determining the root cause node of the fault based on the score ranking of each node in the steady-state distribution vector may further include steps S601 to S602: Step S601: Prioritize examining nodes not included in the initial anomaly observation set. If there is a node that has not been alerted and has the highest steady-state score, then determine that node as the root cause node of the fault. Step S602: After determining the root cause node of the fault, on the global cause-effect graph model, starting from the root cause node of the fault, extend along the positive maximum probability path to the abnormal observation set corresponding to the abnormal alarm, and extract the corresponding fault propagation path.

[0086] In the above steps, to effectively filter high-scoring nodes of the Process Vulnerability Parameter (PVP) that are in the middle of the propagation chain and generate secondary alarms due to being affected by high traffic volume, the source point determination strategy performs a strict disaster point isolation check. Priority is given to examining node groups not included in the initial anomaly observation set O; if such nodes exist... And its steady-state score If the overall value is the highest, then the hidden, deep-seated vulnerability is confirmed. If all high-scoring candidates have been covered by the warning set O, then the node with the highest global score is selected and marked as a potential source of concurrency, triggering the degradation protection mechanism.

[0087] Furthermore, in one embodiment, after determining the root cause node of the fault, in the global cause-effect graph model... Taking the root cause node of the fault as the root, a directed maximum probability connected subgraph extraction is performed. The forward maximum probability path is extended to the abnormal observation set corresponding to the abnormal alarm to extract the corresponding fault propagation path. Specifically, a greedy strategy can be used to extend to the abnormal observation set O, locking the outgoing edge with the highest probability that is not shielded in the connection direction at each step, until the core alarm point is included, thereby extracting a complete causal topology chain that includes the underlying lesion, the hierarchical propagation cascade path, and the symptoms of the upper-level software.

[0088] It should be noted that the embodiments of this disclosure solve the stationary confidence distribution by flipping the probability transition matrix and applying time-varying penalties using nonlinear distortion intensity, performing a probability walk with a restart mechanism across the entire network, and combining it with the judgment rule of prioritizing the examination of the highest score point not covered by the initial alarm set. This can effectively eliminate interference from victim nodes, accurately locate the hidden source node that causes the disaster to spread, and automatically greedily extract the complete causal propagation path emanating from the source point.

[0089] In summary, this embodiment of the present disclosure, through the AI ​​intelligent computing cluster fault root cause localization method in steps S101 to S105, acquires multi-source heterogeneous time-series monitoring data, generates a consensus time axis based on NPU monitoring data, aligns the monitoring data of other dimensions to this time axis and carries the relative time delay exponent and nonlinear distortion strength, thereby achieving accurate time synchronization and feature processing of multi-source heterogeneous monitoring data; secondly, based on the time-synchronized multi-dimensional time-series data and the cluster physical topology, an initial causal graph is constructed, so that directed edges carry the causal direction and correlation strength determined based on time delay and distortion strength. This data-driven causal relationship modeling replaces manually predefined static rules and fixed thresholds, enabling adaptive adaptation to cluster size and load patterns. The dynamic changes are then addressed by iteratively optimizing the initial causal graph through a deep heterogeneous spatiotemporal graph encoder, resulting in a global causal graph model carrying the conditional probability of fault propagation. This further mines and quantifies the propagation path and probabilistic relationship of faults among the nodes of the cluster, effectively addressing complex scenarios of intertwined concurrent faults. Finally, when an abnormal alarm is detected, a reverse random walk is performed based on the global causal graph model along the conditional probability of fault propagation, systematically tracing back from the alarm node to the fault source node. This provides a systematic reverse tracing capability for concurrent cascading faults, accurately locating the root cause of the chain reaction from massive alarms without requiring manual inspection of massive logs, thereby improving the accuracy and efficiency of fault root cause location in AI intelligent computing clusters.

[0090] Based on this, the embodiments of this disclosure improve the accuracy and efficiency of root cause localization of AI intelligent computing cluster faults through a fault hierarchical chain building method. This method is generally divided into three stages: multi-source time-series data alignment and topological association modeling, fault chain optimization based on graph neural networks, and reverse tracing for root cause localization. These three stages constitute a complete processing pipeline, with the output of the previous stage serving as the input of the next, gradually transforming the original observation data into an interpretable fault propagation causal structure, ultimately serving online root cause diagnosis. Specifically: Phase 1: Alignment and topological association modeling of multi-source time-series data.

[0091] In AI computing clusters, monitoring metrics generated by key components such as memory, NPU, and communication optical modules originate from different acquisition channels. These channels exhibit significant differences in timestamp resolution, sampling frequency, and transmission latency. Clock drift and data loss further exacerbate the asynchrony of cross-source time-series data. If cross-layer fault correlation analysis is performed directly without strict alignment, the sequential relationships between different metrics will be disrupted, causing subsequent fault chain modeling to completely lose its temporal basis. Simultaneously, fault propagation is inherently constrained by the physical topology: memory errors only affect the node where the error originates and its directly communicating peers; NPU faults first manifest on the local card and its neighbors sharing a PCIe switch or NVSwitch; while optical module anomalies spread along physical network links. Therefore, relying solely on temporal correlations while ignoring physical connectivity will generate numerous spurious dependencies, rendering the fault chain unreliable. The core issue that must be addressed in this stage is to construct an initial correlation model with both spatiotemporal accuracy, using the physical topology as the spatial skeleton and aligned multi-source time-series data as a dynamic representation.

[0092] In terms of technical implementation, the embodiments of this disclosure first perform multi-level alignment and preprocessing on the timing index data of memory, NPU, and optical modules. Using a unified NTP / PTP clock synchronization benchmark across the cluster, combined with interpolation, resampling, and missing value imputation strategies, heterogeneous data is projected onto a unified time grid to obtain aligned timing data with a consistent time benchmark. Based on this, a heterogeneous graph is constructed according to the cluster's physical topology, including the affiliation between compute nodes and NPUs, GPU / NPU direct connection topology, memory channel mapping, and network link connections of the optical modules. In this graph, each physical component is used as a node, and physical connection relationships and communication dependencies are used as initial edges. The aligned timing indexes are loaded into the graph as dynamic feature vectors of the nodes, forming a spatiotemporal graph snapshot that evolves over time. The aligned spatiotemporal graph output in this stage provides an accurate spatiotemporal reference for the data-driven fault chain learning in the second stage, ensuring that subsequent feature extraction and dependency weight learning are based on physical reachability and time synchronization.

[0093] Phase 2: Fault chain optimization and modeling based on graph neural networks.

[0094] While the spatiotemporal graph obtained from the first stage integrates physical topology and observation time series, its edges only represent the possibility of physical connections and do not reflect the causal strength and direction of fault propagation, nor can they distinguish between normal cooperative fluctuations and abnormal propagation. Traditional methods often rely on manually predefined fixed thresholds or static rules to determine dependencies, making it difficult to adapt to complex scenarios with dynamically changing cluster sizes, diverse load patterns, and intertwined concurrent faults. Therefore, a method with representation learning capabilities is needed to automatically extract deep features representing fault propagation from historical fault data and normal system operation modes, and to quantify and optimize the weights and directions of propagation edges, thereby refining the original physical connection graph into a hierarchical fault chain that can accurately depict where it occurs, to whom it spreads, and the extent of its impact.

[0095] This stage introduces graph neural networks to meet the aforementioned requirements. Using the spatiotemporal graph generated in the first stage as input, a heterogeneous graph neural network incorporating attention mechanisms performs multi-layer transformations and aggregations on node features. This ensures that the embedding vector of each component node encodes both its own temporal pattern and incorporates the state information of its topological neighborhood. To distinguish between fault propagation paths and ordinary functional dependencies, the model is parameterized with propagation probability or causal strength as edge weights. Using historical fault cases with labeled root causes and propagation paths, a loss function is constructed to maximize causal likelihood or minimize propagation reconstruction error. The network parameters and edge weights are then jointly trained end-to-end. During training, regularization terms and sparsity constraints are used to suppress the weights of non-causal edges, achieving structural optimization and noise reduction of the fault chain. Finally, the graph neural network outputs a weighted directed fault propagation graph, which is the optimized hierarchical fault chain model. This fault chain not only records the dependencies between different levels, from memory and NPU to optical modules, but also quantifies the conditional probability of fault propagation along each path, thus providing structured causal prior knowledge for the reverse root cause localization in the third stage.

[0096] Phase 3: Reverse root cause localization based on the fault chain.

[0097] In actual operation and maintenance, when the cluster experiences anomalies such as service fluctuations or training task interruptions, the system will simultaneously generate a series of software and hardware error events and metric limit exceedance alarms. These signals are often scattered across multiple components and multiple layers. The root cause cannot be reliably determined solely by the time sequence of alarms, because alarms from downstream affected components often appear before alarms from the upstream true root cause, creating a false impression of reversed causality. Therefore, a reverse reasoning mechanism is needed that can integrate real-time anomaly signals with a fault propagation causal model. Starting from observed symptoms, it traces backward along the causal chain to ultimately locate the root cause of the chain reaction. The core constraint of this process is that it must be able to handle the uncertainty of multi-source anomalies in the fault chain graph and calculate the explanatory power or responsibility score of each candidate root cause.

[0098] Utilizing the hierarchical fault chain model constructed in the second phase, this phase of root cause localization first maps the currently collected hardware and software system error codes and key indicator deviations to the abnormal activation states of corresponding nodes on the fault chain graph. Subsequently, a root cause analysis algorithm is run on the fault chain graph: using abnormal nodes as observed evidence, probabilistic backpropagation is performed in the opposite direction of the fault propagation edge, or the attention interpretation mechanism embedded in the graph neural network is used to calculate the responsibility score of each upstream node for the downstream anomaly. By cross-validating multiple pieces of abnormal evidence, redundant paths and interference from indirect influences are eliminated, and finally, the root cause nodes and the complete fault propagation causal chain are output in sorted order of scores. This reverse tracing method fully leverages the time alignment accuracy guaranteed in the first phase and the propagation causal relationship in the second phase, making root cause localization no longer an isolated rule matching, but a systematic diagnosis supported by a global causal graph, significantly improving the accuracy and interpretability of localization in complex cascading fault scenarios.

[0099] Please see Figure 9 This disclosure also provides an AI computing cluster fault root cause localization system, which can implement the above-mentioned AI computing cluster fault root cause localization method. The AI ​​computing cluster fault root cause localization system includes: The data acquisition module 901 is used to acquire multi-source heterogeneous time-series monitoring data of the target AI intelligent computing cluster. The multi-source heterogeneous time-series monitoring data includes at least NPU monitoring data, memory monitoring data and communication optical module monitoring data. The timing alignment module 902 is used to generate a consensus timeline based on the NPU monitoring data, and to align the memory monitoring data and the communication optical module monitoring data to the consensus timeline, so as to carry the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus timeline, thereby obtaining time-synchronized multidimensional timing data; The causal graph construction module 903 is used to construct an initial causal graph based on the multidimensional time series data and the physical topology of the target AI intelligent computing cluster, so that the directed edges in the initial causal graph carry the causal direction and correlation strength determined based on the relative time delay exponent and the nonlinear distortion intensity. Graph optimization module 904 is used to input the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model, wherein the global causal graph model contains the fault propagation conditional probability of directed edges. The root cause localization module 905 is used to determine the root cause node of the fault by performing a reverse random walk along the fault propagation conditional probability based on the global causal graph model when an abnormal alarm is detected.

[0100] In summary, the AI-powered intelligent computing cluster fault root cause localization system, by executing the AI-powered intelligent computing cluster fault root cause localization method in the above embodiments, acquires multi-source heterogeneous time-series monitoring data, generates a consensus time axis based on NPU monitoring data, aligns the monitoring data of other dimensions to this time axis and carries the relative time delay exponent and nonlinear distortion strength, thus achieving accurate time synchronization and feature processing of multi-source heterogeneous monitoring data. Secondly, based on the time-synchronized multi-dimensional time-series data combined with the cluster's physical topology, an initial causal graph is constructed, enabling directed edges to carry causal directions and correlation strengths determined based on time delay and distortion strength. This data-driven causal relationship modeling replaces manually predefined static rules and fixed thresholds, enabling adaptive adaptation to cluster size and load patterns. The dynamic changes are then addressed by iteratively optimizing the initial causal graph through a deep heterogeneous spatiotemporal graph encoder, resulting in a global causal graph model carrying the conditional probability of fault propagation. This further mines and quantifies the propagation path and probabilistic relationship of faults among the nodes of the cluster, effectively addressing complex scenarios of intertwined concurrent faults. Finally, when an abnormal alarm is detected, a reverse random walk is performed based on the global causal graph model along the conditional probability of fault propagation, systematically tracing back from the alarm node to the fault source node. This provides a systematic reverse tracing capability for concurrent cascading faults, accurately locating the root cause of the chain reaction from massive alarms without requiring manual inspection of massive logs, thereby improving the accuracy and efficiency of fault root cause location in AI intelligent computing clusters.

[0101] The specific implementation of the AI ​​computing cluster fault root cause localization system is basically the same as the specific embodiment of the AI ​​computing cluster fault root cause localization method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this disclosure, the AI ​​computing cluster fault root cause localization system may also be equipped with other functional modules to implement the AI ​​computing cluster fault root cause localization method in the above embodiments.

[0102] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned AI intelligent computing cluster fault root cause localization method. This electronic device can include any smart terminal such as a tablet computer or in-vehicle computer.

[0103] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the AI ​​intelligent computing cluster fault root cause localization method of the embodiments of this disclosure. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0104] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described AI intelligent computing cluster fault root cause localization method.

[0105] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0106] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0107] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0108] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0110] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such orders can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0111] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0112] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of the units described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0113] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. A method for locating the root cause of faults in an AI intelligent computing cluster, characterized in that, include: Acquire multi-source heterogeneous time-series monitoring data of the target AI computing cluster, wherein the multi-source heterogeneous time-series monitoring data includes at least NPU monitoring data, memory monitoring data, and communication optical module monitoring data; The NPU monitoring data is subjected to soft dynamic time warping and dynamic warping centroid averaging iteration on multiple subsequences to generate a consensus time axis. Using the consensus time axis as a global reference, dynamic programming with window constraints is used to align the sequences of each dimension of the memory monitoring data and the communication optical module monitoring data to the consensus time axis. The relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus time axis are calculated to obtain time-synchronized multidimensional time series data carrying the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus time axis. An initial causal graph is constructed based on the multidimensional time-series data and the physical topology of the target AI computing cluster, so that the directed edges in the initial causal graph carry the causal direction and correlation strength determined based on the relative time delay exponent and the nonlinear distortion intensity. The initial causal graph is input into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model, which includes the fault propagation conditional probability of directed edges. When an abnormal alarm is detected, the detected abnormal alarm is mapped to the abnormal impact vector of the corresponding node in the global causal graph model, and an inverse state transition matrix is ​​constructed based on the fault propagation conditional probability and the nonlinear distortion intensity; a Markov random walk with restart is performed based on the inverse state transition matrix to iteratively solve the steady-state distribution vector; and the root cause node of the fault is determined according to the score ranking of each node in the steady-state distribution vector.

2. The AI ​​intelligent computing cluster fault root cause localization method according to claim 1, characterized in that, The construction of the initial causal graph based on the multidimensional time-series data and the physical topology of the target AI computing cluster includes: Based on the physical connection relationships between the components in the target AI computing cluster, a candidate edge set for the heterogeneous graph is constructed. Based on the cross-correlation function with a penalty term constrained by the relative time delay index, the causal direction and peak correlation coefficient of each candidate edge are determined within the preset time delay search range; Based on the tolerance parameter threshold generated by random rearrangement of the sequence, the candidate edge set is subjected to saliency filtering and causal orientation pruning to obtain the initial causal graph.

3. The AI ​​intelligent computing cluster fault root cause localization method according to claim 1, characterized in that, The step of inputting the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model includes: Nonlinear mapping is performed on the node features and edge features in the initial causal graph to obtain node embedding vectors and edge embedding vectors, wherein the edge features include the relative time delay exponent, the nonlinear distortion intensity, and the cross-correlation peak value; Based on the edge embedding vector, the node embedding vector is aggregated and updated in multiple layers through a multi-head graph attention mechanism to obtain the top-level node embedding. Based on the top-level node embedding and the edge embedding vector, the fault propagation conditional probability of each directed edge is predicted by the bilinear gated probability inference network, and end-to-end training is performed based on historical fault samples to obtain a global causal graph model.

4. The AI ​​intelligent computing cluster fault root cause localization method according to claim 3, characterized in that, The step of performing multi-level aggregation and updating of the node embedding vector based on the edge embedding vector using a multi-head graph attention mechanism includes: The edge embedding vector is injected into the attention score calculation of the multi-head graph attention mechanism. After applying attention weighting to the pseudo-propagation link with high nonlinear distortion intensity and long time delay, the node embedding vector is updated by multi-level aggregation.

5. The AI ​​intelligent computing cluster fault root cause localization method according to claim 3, characterized in that, The global causal graph model, obtained through end-to-end training based on historical fault samples, includes: A binary classification mask is constructed based on historical fault samples. The deep heterogeneous spatiotemporal graph encoder is trained end-to-end using a binary cross-entropy loss function combined with a sparse regularization term. Edge pruning is performed according to a pre-set threshold to obtain a global causal graph model.

6. The AI ​​intelligent computing cluster fault root cause localization method according to claim 1, characterized in that, The step of determining the root cause node of the fault based on the score sorting of each node in the steady-state distribution vector includes: Prioritize examining nodes not included in the initial anomaly observation set. If there is a node that has not been alerted and has the highest steady-state score, then that node is identified as the root cause of the fault. After determining the root cause node of the fault, the corresponding fault propagation path is extracted by extending along the positive maximum probability path to the abnormal observation set corresponding to the abnormal alarm on the global cause-effect graph model, starting from the root cause node of the fault.

7. A system for locating the root cause of AI intelligent computing cluster faults, characterized in that, include: The data acquisition module is used to acquire multi-source heterogeneous time-series monitoring data of the target AI computing cluster. The multi-source heterogeneous time-series monitoring data includes at least NPU monitoring data, memory monitoring data, and communication optical module monitoring data. The time-series alignment module is used to perform soft dynamic time warping and dynamic warping centroid averaging iteration on multiple sub-sequences of the NPU monitoring data to generate a consensus time axis. Using the consensus time axis as a global reference, dynamic programming with window constraints is used to align the sequences of each dimension of the memory monitoring data and the communication optical module monitoring data to the consensus time axis, and calculate the relative time delay exponent and nonlinear distortion intensity of each dimension sequence relative to the consensus time axis. This results in time-synchronized multi-dimensional time-series data. The causal graph construction module is used to construct an initial causal graph based on the multidimensional time series data and the physical topology of the target AI computing cluster, so that the directed edges in the initial causal graph carry the causal direction and correlation strength determined based on the relative time delay exponent and the nonlinear distortion intensity. The graph optimization module is used to input the initial causal graph into a preset deep heterogeneous spatiotemporal graph encoder for iterative optimization to obtain an optimized global causal graph model, wherein the global causal graph model contains the fault propagation conditional probability of directed edges. The root cause localization module is used to map the detected abnormal alarm to the abnormal impact vector of the corresponding node in the global causal graph model when an abnormal alarm is detected, and construct an inverse state transition matrix based on the fault propagation conditional probability and the nonlinear distortion intensity; perform a Markov random walk with restart based on the inverse state transition matrix to iteratively solve the steady-state distribution vector; and determine the root cause node of the fault according to the score ranking of each node in the steady-state distribution vector.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the AI ​​intelligent computing cluster fault root cause localization method according to any one of claims 1 to 6.

Images