A cluster fault management method and device, electronic equipment and medium

Abstract

The application discloses a cluster fault management method and device, electronic equipment and medium. The method comprises the following steps: acquiring performance index data of multi-field units in a cluster; constructing a dynamic fault propagation model of the cluster based on a graph model and the performance index data; constructing a four-dimensional fault visualization based on a time sequence according to the dynamic fault propagation model; and performing cluster fault analysis through the four-dimensional fault visualization. Thus, the performance index data of different field units is constructed into a dynamic fault propagation model based on a time sequence, cross-domain fault positioning, analysis, monitoring and deduction based on a time axis are realized, cross-domain fault spatiotemporal correlation is avoided to be split due to separate fault monitoring of each field unit, and fault positioning efficiency and accuracy are improved. In addition, the four-dimensional fault visualization expression based on the time sequence reduces the complexity of multi-screen switching monitoring of operation and maintenance personnel, and improves the cluster fault management efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence operation and maintenance technology, and in particular to a method, device, electronic device and medium for managing cluster faults. Background Technology

[0002] With the rapid development of artificial intelligence (AI), the scale of AI computing power clusters has exceeded 10,000 kilowatts. In order to ensure the efficient management and normal operation of AI computing power clusters, fault monitoring of AI computing power clusters is one of the most important research directions.

[0003] Current fault monitoring systems monitor hardware, logical topology, and network status independently and present them in a visual interface. This results in the fragmentation of the spatiotemporal correlation of cross-domain faults, making it impossible to verify the fault propagation process through convenient interactive operations. It is also difficult to support maintenance personnel in quickly locating the root cause of faults and has low efficiency in fault management.

[0004] Therefore, how to achieve rapid location and efficient management of cross-domain cluster faults is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, one aspect of this application provides a method for managing cluster failures, the method comprising: Obtain performance metrics data from multiple domain units within the cluster; Based on the graph model, a dynamic fault propagation model for the cluster is constructed using the performance metric data. Based on the dynamic fault propagation model, a four-dimensional fault visualization based on time series is constructed. Cluster fault analysis is performed using the aforementioned four-dimensional fault visualization.

[0006] Optionally, the step of constructing a dynamic fault propagation model for the cluster based on the graph model and the performance index data includes: Based on the graph model, the nodes and edges of the graph structure are defined using the performance index data; Determine the time-varying probability of fault propagation between node pairs with edges in the graph structure; A dynamic update mechanism for the graph structure is established to obtain the dynamic fault propagation model.

[0007] Optionally, the performance metric data is a multi-dimensional vector based on a time series; defining the nodes and edges of the graph structure based on the graph model and the performance metric data includes: The hardware entities in the cluster are defined as physical nodes; the multidimensional vector is split, and the various performance index data after splitting are defined as logical nodes to form the nodes of the graph structure. A relational database table is used to bind the correspondence between the physical nodes and the logical nodes; Based on the correspondence, the edges and edge types in the graph structure are defined; the edge types include intra-domain propagation edges and cross-domain propagation edges.

[0008] Optionally, determining the time-varying failure propagation probability between node pairs with edges in the graph structure includes: Obtain the time decay factor, the physical distance between target node pairs, and the target performance index data corresponding to the target node pairs; wherein, the target node pairs are node pairs with edges in the graph structure; Based on the physical distance, a physical propagation weight is determined between the target node pairs; the physical propagation weight is the probability of the target node pairs propagating faults through physical connections; the closer the physical distance, the greater the physical propagation weight, which represents a higher probability of the faults propagating through physical connections. Based on the target performance index data, determine the edge weights between the target node pairs; the edge weights represent the degree to which a failure of one node in a target node pair affects the operation of the other node. The fault propagation probability is determined based on the time decay factor, the physical propagation weight, and the edge weight.

[0009] Optionally, determining the edge weights between the target node pairs based on the target performance index data includes: Based on the target performance index data, determine the Pearson correlation coefficient between the target node pairs; The edge weights are determined based on the time decay factor and the Pearson correlation coefficient. The edge weights range from [-1, 1]. When the edge weight is greater than 0, the faults between the target node pairs are positively correlated, and the greater the edge weight, the higher the degree of influence. When the edge weight is equal to 0, the faults between the target node pairs are not correlated. When the edge weight is less than 0, the faults between the target node pairs are negatively correlated, and the smaller the edge weight, the higher the degree of influence.

[0010] Optionally, the step of constructing a time-series-based four-dimensional fault visualization based on the dynamic fault propagation model includes: Set alarm thresholds for various performance indicators of the multi-domain units; The chain-like faults in the dynamic fault propagation model are abstracted into visual elements; Based on the pre-set visualization specifications and visualization elements, a time-series-based four-dimensional fault visualization is constructed; wherein, the visualization specifications include node rendering rules, color encoding rules, and cluster topology rules.

[0011] Optionally, the multi-domain unit includes a computing domain, a storage domain, and a network domain; the cluster topology structure rule adopts a hierarchical leaf-spine architecture; The hierarchical leaf-spine architecture includes a first layer of spine switch layer, a second layer of leaf switch layer, a third layer of server node layer, and a fourth layer of compute node layer; The computing domain corresponds to the third and fourth layers; the storage domain corresponds to the third layer; the network domain corresponds to the first and second layers; and there is network communication between the layers.

[0012] Optionally, the performance indicators of the computing domain include video memory utilization, and the video memory alarm threshold of the video memory utilization is determined based on a basic alarm threshold and a task type coefficient; wherein, the task type coefficient is used to reflect the degree of influence of task type on video memory utilization, and the task type coefficient is determined based on a pre-constructed first mapping relationship, the first mapping relationship being the correspondence between the task type and the task type coefficient; The performance metrics of the network domain include packet loss rate, and the packet loss alarm threshold of the packet loss rate is determined according to a pre-constructed second mapping relationship; wherein, the second mapping relationship is the correspondence between the packet loss alarm threshold and the service sensitivity, and the service sensitivity is used to reflect the degree of sensitivity of the service type to the packet loss rate.

[0013] Optionally, the video memory alarm threshold is the product of the basic alarm threshold and the task type coefficient; wherein, the greater the video memory requirement of the task type, the greater the task type coefficient, and the greater the video memory alarm threshold. The higher the business sensitivity, the larger the packet loss alarm threshold.

[0014] Another aspect of this application provides a cluster failure management apparatus, the apparatus comprising: The data acquisition module is used to acquire performance index data of multiple domain units in the cluster; The model building module is used to construct a dynamic fault propagation model of the cluster based on the graph model and the performance index data. A visualization building module is used to construct a time-series-based four-dimensional fault visualization based on the dynamic fault propagation model. The fault analysis module is used to perform cluster fault analysis through the four-dimensional fault visualization.

[0015] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the program to implement the steps of the cluster fault management method.

[0016] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cluster fault management method.

[0017] The cluster fault management method, device, electronic equipment, and medium provided in this application have the following beneficial effects: Based on time series, performance index data of different domain units are constructed into a dynamic fault propagation model, enabling cross-domain fault location, analysis, monitoring, and simulation based on the time axis. This avoids the fragmentation of spatiotemporal correlation of cross-domain faults caused by individual fault monitoring of each domain unit, thus improving the efficiency and accuracy of fault location. Furthermore, the four-dimensional fault visualization based on time series reduces the complexity of multi-screen switching monitoring for maintenance personnel, improving the efficiency of cluster fault management. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a cluster fault management method provided in an embodiment of this application; Figure 2 A schematic diagram of a four-dimensional fault visualization interface provided in an embodiment of this application; Figure 3 A flowchart illustrating a cluster fault management method provided in another embodiment of this application; Figure 4 A flowchart illustrating a cluster fault management method provided in another embodiment of this application; Figure 5 This is a schematic diagram of a layered leaf-ridge architecture provided in an embodiment of this application; Figure 6 A schematic diagram of the structure of a cluster fault management device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0019] The attached diagram is labeled as follows: 60 is the data acquisition module, 61 is the model building module, 62 is the visualization building module, 63 is the fault analysis module, 70 is the memory, 71 is the processor, 72 is the display screen, 73 is the input / output interface, 74 is the communication interface, 75 is the power supply, 76 is the communication bus, 701 is the computer program, 702 is the operating system, and 703 is the data. Detailed Implementation

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] Figure 1 This is a flowchart illustrating a cluster fault management method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: S10: Obtain performance index data of multiple domain units in the cluster; In a specific embodiment, to achieve cross-domain fault monitoring, location, and simulation, the cluster fault management method provided in this application collects and acquires performance index data of multiple domain units in the cluster. These multiple domain units include high-fault-prone areas in the cluster system, such as storage domain, computing domain, network domain, power domain, temperature domain, and management domain.

[0023] Performance index data refers to the fault indicators and corresponding specific data used for monitoring faults in multiple domain units. In one optional embodiment, the complex nonlinear performance index data in each domain unit is converted into dynamic performance parameters based on time series.

[0024] For example, potential failures in the computing domain include GPU memory leaks, CPU overload, and task scheduling failures.

[0025] GPU memory leaks refer to the failure to release memory resources in a timely manner, leading to a gradual increase in memory usage and ultimately causing task failures or system crashes. Performance metrics that need to be monitored for GPU memory leaks include memory usage (t), GPU utilization (t), and task failure rate (t), where t represents time.

[0026] CPU overload is usually caused by improper task scheduling or too many computationally intensive tasks, resulting in high CPU utilization, slow system response, or even system crash. Performance metrics that need to be monitored for CPU overload include CPU utilization (t), task queue length (t), and system response time (t).

[0027] Task scheduling failures are typically caused by resource allocation conflicts or scheduling algorithm defects, preventing tasks from starting or executing normally. Performance metrics that need to be monitored for task scheduling failures include task start success rate (t), number of resource allocation conflicts (t), and scheduling latency (t).

[0028] Potential failures in the storage domain include input / output (I / O) throughput latency, disk corruption or node failure, and insufficient disk space. Storage performance bottlenecks typically manifest as decreased I / O throughput, leading to increased data read / write latency and impacting overall cluster performance. Performance metrics that need to be monitored for I / O throughput latency include I / O throughput (t), read / write latency (t), and queue depth (t).

[0029] Storage node failures, caused by disk damage or node downtime, can lead to data access failures or data loss. Correspondingly, the performance metrics that need to be monitored include node health status (t), disk error rate (t), and data access success rate (t).

[0030] Insufficient disk space, or exhaustion of storage resources, manifests as insufficient disk space, leading to data write failures or system alarms. Performance metrics that need to be monitored for insufficient disk space include disk utilization (t), remaining space (t), and number of write failures (t).

[0031] Potential faults in the network domain include network congestion, network packet loss, and network connection interruption. Network congestion occurs when network traffic exceeds its capacity, leading to increased data transmission latency and even data loss. Performance metrics that need to be monitored for network congestion include network bandwidth utilization (t), data transmission latency (t), and packet loss rate (t).

[0032] Network packet loss refers to the loss of data packets during transmission, resulting in compromised data integrity and impacting task execution. Performance metrics that need to be monitored for network packet loss include packet loss rate (t), retransmission rate (t), and data integrity (t). Network connection interruption refers to a failure in network equipment or links, leading to a complete interruption of data transmission. Performance metrics that need to be monitored for network connection interruption include connection status (t) and data traffic (t).

[0033] In a VARIABLES cluster system, besides the storage, compute, and network domains, other domains (such as power, temperature, and management domains) are also high-risk areas for failures and require monitoring. For example, failures that need to be monitored in other domains include unstable power supply, overheating, and configuration errors.

[0034] Unstable power supply can lead to sudden power outages or voltage fluctuations in cluster nodes, affecting the normal operation of the system. Correspondingly, the performance metrics that need to be monitored for unstable power supply include voltage fluctuation range (t), current stability (t), and number of power outages (t).

[0035] In specific embodiments, overheating is typically caused by poor heat dissipation or excessively high ambient temperatures, which may lead to hardware damage or degraded system performance. The performance metrics that need to be monitored for overheating include temperature (t), fan speed (t), and hardware health status (t).

[0036] Configuration errors can cause the system to fail to start or run properly, affecting the overall stability of the cluster. Performance metrics that need to be monitored for configuration errors include the number of configuration errors (t), the system startup success rate (t), and the number of log alerts (t).

[0037] It should be noted that, in one optional embodiment, to improve the accuracy of fault monitoring, after acquiring a large amount of performance indicator data, the performance indicator data is first cleaned to remove noise data and fill in missing values, ensuring the integrity and accuracy of the data. Furthermore, it should be noted that the cluster fault management method provided in this application embodiment can be applied to intelligent fault operation and maintenance management scenarios such as intelligent computing centers and supercomputing platforms, and this application does not limit its application in this regard.

[0038] S11: Based on the graph model, a dynamic fault propagation model for the cluster is constructed using performance metric data; After obtaining performance index data from multiple domain units, a dynamic fault propagation model is constructed based on a graph model. A graph model is a mathematical model used to represent probabilistic relationships between variables, graphically illustrating the dependencies and independence between them.

[0039] Therefore, in a specific embodiment, based on the graph model, the nodes and edges in the graph structure are defined and the directions between nodes are determined by the performance index data obtained in step S10. In addition, it is also necessary to calculate the probability of a fault occurring on an edge and the probability of a fault being transmitted from one node to another. That is, a time dimension is introduced into the graph model to represent the propagation of faults over time, thereby constructing a dynamic fault propagation model.

[0040] It should be noted that the dynamic fault propagation model refers to a model that simulates the changes and propagation of performance index data over time. This dynamic fault propagation model not only considers the state of the fault at the current point in time, but also how the fault affects other parts over time. Therefore, based on this dynamic fault propagation model, the future behavior of the fault can be predicted, and the impact of the fault on the overall system can be assessed.

[0041] S12: Construct a time-series-based four-dimensional fault visualization based on the dynamic fault propagation model; Furthermore, to achieve rapid fault location and improve the efficiency of fault management for operations and maintenance personnel, a time-series-based four-dimensional fault visualization is constructed based on a dynamic fault propagation model. The four dimensions refer to the spatial dimension (i.e., cluster architecture), the time dimension, the data dimension (i.e., performance indicator data), and the state dimension (i.e., whether a fault has occurred).

[0042] In one optional embodiment, the physical architecture and network topology of the cluster are displayed according to a dynamic fault propagation model, and time, performance indicators, or fault types are represented by means of color, size, shape, or animation effects, thereby achieving four-dimensional fault visualization. It should be noted that this application does not impose specific limitations on the cluster architecture, rendering rules, and color encoding.

[0043] S13: Perform cluster fault analysis through four-dimensional fault visualization.

[0044] After obtaining the four-dimensional fault visualization, operations and maintenance personnel can directly perform cluster fault root cause analysis and fault backtracking through this visualization. Specifically, based on the four-dimensional fault visualization results, and combining the propagation relationships of nodes and edges in the dynamic fault propagation model with time series data, the fault probability distribution can be analyzed, thereby deduce the root cause of the cluster fault.

[0045] Figure 2 This is a schematic diagram of a four-dimensional fault visualization interface provided in an embodiment of this application. For ease of understanding, it will be described below in conjunction with... Figure 2 Please provide an explanation. For example... Figure 2 The cluster four-dimensional fault visualization interface shown indicates a training task timeout, with intermittent NVLink errors occurring on the associated GPU node. The visualization interface displays GPU3 and its corresponding NVLink in red fault condition, and the corresponding Node also displays a red fault condition.

[0046] In response, when reasoning about the fault propagation path, for example, at 10:40:10, the GPU3 load exceeds the alarm threshold, and the GPU3 memory fragmentation rate reaches 88%, displaying a red GPU3 alarm. At 10:45:30, an NVLink error is highlighted, with retries occurring 120 times per second, displaying a red NVLink alarm. The fault propagation probability between the NVLink and GPU3 performance metrics is calculated based on a dynamic fault propagation model; that is, the fault propagation probability between the two nodes is calculated. If high significance is displayed, indicating that the fault has propagated, the propagation connection is displayed. Further, tracing upwards, faults in associated nodes are displayed, highlighted in red.

[0047] Therefore, the fault propagation path was determined to be GPU memory fragmentation—memory allocation delay—CUDA kernel waiting for NVLink transmission—multiple timeout retries—temperature rise. Furthermore, the fault analysis concluded that a timing conflict defect existed in the GPU memory controller firmware, leading to memory allocation fragmentation, which in turn triggered an NVLink protocol layer retry storm and signal degradation.

[0048] Therefore, the cluster fault management method provided in this application, based on time series, constructs a dynamic fault propagation model from performance index data of different domain units, realizing cross-domain fault location, analysis, monitoring, and inference based on the time axis. This avoids the fragmentation of spatiotemporal correlation of cross-domain faults caused by individual fault monitoring of each domain unit, thus improving the efficiency and accuracy of fault location. Furthermore, the four-dimensional fault visualization based on time series reduces the complexity of multi-screen switching monitoring for operations and maintenance personnel, improving the efficiency of cluster fault management.

[0049] In one optional embodiment, a dynamic fault propagation model of the cluster is constructed based on a graph model and performance metric data, including: Based on the graph model, the nodes and edges of the graph structure are defined using performance index data; Determine the time-varying probability of fault propagation between node pairs with edges in a graph structure; A dynamic update mechanism for the graph structure is established, resulting in a dynamic fault propagation model.

[0050] Yes, a graph model is a mathematical model used to represent probabilistic relationships between variables, visually illustrating the dependencies and independence between them. When constructing a dynamic fault propagation model, a graph model can clearly represent the relationships between fault data and how these relationships change over time. Therefore, in this embodiment, a dynamic fault propagation model in a cluster is constructed based on a graph model.

[0051] Specifically, after collecting a large amount of performance metric data, nodes and edges in the graph structure are defined based on a graph model. In a specific embodiment, nodes can be used to represent faulty devices in the cluster, and also to represent performance metrics in the cluster, such as memory utilization (t), GPU utilization (t), CPU utilization (t), I / O throughput (t), and read / write latency (t). Edges in the graph structure are used to represent fault propagation paths between nodes; if a fault in one node affects the normal operation of another node, then an edge exists between these two nodes.

[0052] Furthermore, as time changes, the probability of failure between nodes with edges also changes. Therefore, in a specific embodiment, the failure propagation probability between pairs of nodes with edges changes over time.

[0053] Understandably, a cluster generates a large amount of performance metrics data in real time during normal operation. To monitor cluster faults in real time, a dynamic update mechanism needs to be set for the graph structure to obtain a dynamic fault propagation model. Specifically, in one optional embodiment, performance metrics data of each domain unit are acquired in real time, and an incremental data update is triggered every preset period (e.g., every 5 seconds).

[0054] In one optional embodiment, the performance metric data is a multi-dimensional vector based on time series; based on a graph model, the nodes and edges of the graph structure are defined using the performance metric data, including: Define hardware entities in the cluster as physical nodes; split multidimensional vectors and define various performance index data after splitting as logical nodes to form nodes in a graph structure. Use relational database tables to bind the correspondence between physical nodes and logical nodes; Based on the correspondence, edges and edge types are defined in the graph structure; edge types include intra-domain propagation edges and cross-domain propagation edges.

[0055] In one optional embodiment, to facilitate the analysis of performance index data and the construction of a dynamic fault propagation model, the acquired performance index data is vectorized and modeled. Specifically, real-time performance index data from multiple domain units, such as the computing domain, storage domain, and network domain, are modeled as a multi-dimensional vector, with each dimension corresponding to a specific performance parameter, and their dynamic changes are reflected through time series analysis.

[0056] In an optional embodiment, the mathematical expression can be represented as V(t) = [FLOPS(t), Latency(t), Wear(t), Temp(t)], where FLOPS(t) is the performance index data of the computation domain, Latency(t) is the performance index data of the network domain, Wear(t) is the performance index data of the storage domain, and Temp(t) is the performance index data of the temperature domain.

[0057] It should be noted that the above expressions are illustrated using the computing domain, network domain, storage domain, and temperature domain as examples. In fact, in specific embodiments, performance index data of multiple domain units with high failure rates, such as the management domain and power domain, may also be included. This application does not limit this, and the corresponding domain unit can be selected for fault monitoring according to actual business needs.

[0058] Therefore, through vectorized modeling, the embodiments of this application can transform the complex nonlinear performance index data of various domain units into dynamic performance vectors, that is, into quantifiable and computable objects, supporting machine learning prediction and resource scheduling optimization.

[0059] Based on this, as an optional implementation, when defining nodes and edges in the graph structure, hardware entities in the cluster can be defined as physical nodes, and corresponding physical attributes can be assigned to them. For example, physical nodes may include GPU servers, switches, storage servers, etc., and the corresponding physical attributes may include data center location, rack, and U-position, etc. For example, the physical attribute can be represented as NM132-E01-U36 (data center location is NM132, rack number is E01, and U-position is U36). Here, U-position is used to represent the height unit of the device in the rack.

[0060] Furthermore, the multidimensional vectors obtained in the above embodiments are split, and the various performance index data after splitting are defined as logical nodes. It can be understood that logical nodes represent abstract indicators, such as memory usage and network connectivity. Thus, physical nodes and logical nodes constitute all nodes in the graph structure.

[0061] Furthermore, relational database tables are used to bind the correspondence between physical nodes and logical nodes. Table 1 is a schematic table of the correspondence between physical nodes and logical nodes provided in an embodiment of this application. For ease of understanding, the following explanation will be based on Table 1.

[0062] Table 1 is a schematic diagram of the correspondence between physical nodes and logical nodes. First, it should be noted that in other domains, physical entity D is used as an example for illustration, and Table 1 is an illustrative example. In actual cluster fault management, the corresponding physical nodes and logical nodes can be obtained according to actual business needs. As shown in Table 1, the hardware entities of different domain units are defined as physical nodes, and the performance index data corresponding to the physical nodes are defined as logical nodes, thereby constructing the correspondence between physical nodes and logical nodes.

[0063] Therefore, based on this correspondence, when a certain indicator of a logical node becomes abnormal, the corresponding physical node can be quickly located, that is, the equipment location where the fault occurred can be identified. Furthermore, the causal chain of the fault can be traced through the correlation between logical nodes.

[0064] Furthermore, based on the above-mentioned binding correspondence, the edges and edge types in the graph structure are defined. It can be understood that in specific embodiments, faults may occur within the same domain unit, and faults may occur across domain units. That is, the fault transmission path may be within the same domain or across domains.

[0065] Therefore, when defining edge types, we include intra-domain propagation edges and cross-domain propagation edges. Intra-domain propagation edges refer to edges where faults propagate and spread between nodes within the same domain. For example, a GPU memory leak leading to task failure occurs, where both the GPU memory leak and the task failure are logical nodes in the computing domain. Cross-domain propagation edges, on the other hand, refer to edges where the impact of faults propagates and spreads between nodes across different domains. For example, a decrease in storage I / O causing network congestion occurs, where the decrease in storage I / O is a logical node in the storage domain, while network congestion is a logical node in the network domain.

[0066] Figure 3 This is a flowchart illustrating a cluster fault management method according to another embodiment of this application. In an optional embodiment, such as... Figure 3 As shown, the time-varying probability of fault propagation between node pairs with edges in a graph structure is determined, including: S30: Obtain the time decay factor, the physical distance between target node pairs, and the target performance index data corresponding to the target node pairs; where the target node pairs are node pairs with edges in the graph structure; In specific embodiments, the propagation of faults between nodes is affected by factors such as physical distance, time, and the strength of correlation between nodes. It is understandable that when nodes are physically connected, the closer the physical distance, the easier it is for faults to propagate through these connections. Furthermore, the stronger the correlation between nodes, the greater the impact of a fault in one node on the normal operation of another node; that is, the higher the probability of fault propagation.

[0067] Therefore, in one optional embodiment, to accurately calculate the time-varying probability of fault propagation between node pairs with edges, it is necessary to comprehensively consider factors such as physical distance and the correlation strength between time-core nodes. Specifically, in a specific embodiment, obtaining the time decay factor, the physical distance between target node pairs, and the target performance index data corresponding to the target node pairs can be understood as follows: faults only propagate between nodes with edges, that is, a fault in one node can affect another node. Therefore, a target node pair refers to a node pair with edges in the graph structure.

[0068] S31: Determine the physical propagation weight between target node pairs based on physical distance; the physical propagation weight is the probability of a fault propagating between target node pairs through physical connections; the closer the physical distance, the greater the physical propagation weight, which represents the higher the probability of a fault propagating through physical connections. Furthermore, based on the physical distance, the physical propagation weight (physical_weight) of the target node pair is calculated using the formula: physical_weight = 0.8 × e^(-0.2 × physical distance). Here, physical_weight measures the basic strength of fault propagation between target node pairs, that is, the probability of fault propagation between target node pairs through physical connections.

[0069] In a specific embodiment, the closer the physical distance, the higher the base strength, and the larger the physical propagation weight (physical_weight), the higher the probability of fault propagation between target node pairs through physical connections. For example, if the target node pair is GPU1 and GPU2, when GPU1 overheats, it will quickly affect the nearby GPU2 node, causing it to overheat and thus leading to a failure of GPU2.

[0070] S32: Determine the edge weights between target node pairs based on the target performance index data; the edge weights represent the degree of impact of a failure in one node pair on the operation of the other node. While calculating the physical propagation weight (physical_weight), the edge weights between target node pairs are calculated based on the target performance metric data. Edge weights are used to describe fault propagation in non-physically connected but logically strongly correlated nodes. This can be understood as representing the probability of a fault occurring between one node (e.g., memory utilization) and another node (e.g., network bandwidth utilization) in a target node pair, or the degree to which a fault in one node affects the normal operation of another node. Alternatively, edge weights can be understood as measuring the correlation between two nodes in a target node pair.

[0071] In one optional embodiment, the edge weight ranges from [0, 1]. When the edge weight is equal to 0, it indicates that the two nodes have no impact on each other when they fail, that is, the two nodes are unrelated. When the edge weight is equal to 1, it indicates that the two nodes completely influence each other, that is, the failure of one node will inevitably cause the failure of the other node.

[0072] In fact, in specific embodiments, the correlation between two variables can be calculated in various ways, such as using the Pearson Correlation Coefficient or the Phi Coefficient, and this application does not limit the specific method used.

[0073] S33: Determine the fault propagation probability based on the time decay factor, physical propagation weight, and edge weight.

[0074] Furthermore, after calculating the physical propagation weights and edge weights, a time decay factor is introduced to calculate the fault propagation probability. In a specific embodiment, it is considered that as long as there is a strong dependency between the two nodes in the target node pair, the influence of physical distance will be covered.

[0075] Therefore, as an optional embodiment, the probability of fault propagation can be calculated according to the formula P=max(physical_weight, logical_weight)×decay, where P is the probability of fault propagation, physical_weight is the physical propagation weight, logical_weight is the edge weight, and decay is the time decay factor.

[0076] In one optional embodiment, the time decay factor decayay can be an exponential decay function, i.e., decayay = e^(-λΔt), where λ is the device decay coefficient, which can be determined according to the actual business requirements in specific embodiments. For example, in some optional embodiments, λ can be 0.1, and Δt is the fault propagation time. In fact, in specific embodiments, this application does not specifically limit the time decay factor.

[0077] Therefore, in a specific embodiment, the larger of the physical propagation weight (physical_weight) and the edge weight (logical_weight) is used to calculate the fault propagation probability P. In fact, the fault propagation probability P can be understood as the likelihood of fault propagation after considering factors such as physical distance, node relevance, and timeliness; it is the output result of the propagation probability.

[0078] In another optional embodiment, in order to comprehensively consider the impact of physical propagation weight and edge weight logical_weight on the fault propagation probability P, corresponding weights can be assigned to physical propagation weight and edge weight logical_weight, and the sum of the two weights is equal to 1. The weighted sum of physical propagation weight and edge weight logical_weight is calculated and multiplied by the time decay factor to obtain the fault propagation probability P.

[0079] As can be seen, the cluster fault management method provided in this application comprehensively considers multiple influencing factors such as physical distance between nodes, node correlation and timeliness in fault propagation, and performs comprehensive calculations to improve the calculation accuracy of fault propagation probability, thereby improving the accuracy of fault location.

[0080] In one optional embodiment, determining the edge weights between target node pairs based on target performance index data includes: Based on the target performance index data, determine the Pearson correlation coefficient between the target node pairs; The edge weights are determined based on the time decay factor and the Pearson correlation coefficient. The edge weights range from [-1, 1]. When the edge weight is greater than 0, the faults between the target node pairs are positively correlated, and the greater the edge weight, the greater the impact. When the edge weight is equal to 0, the faults between the target node pairs are not correlated. When the edge weight is less than 0, the faults between the target node pairs are negatively correlated, and the smaller the edge weight, the greater the impact.

[0081] In a specific embodiment, since the performance metric data in the cluster are continuous variables and exhibit a linear relationship, the edge weight logical_weight is calculated using the Pearson correlation coefficient. Specifically, the Pearson correlation coefficient is calculated according to formula (1): (1) in, The Pearson correlation coefficient is used. The first node of one of the target nodes Target performance metrics data, For the target node pair, the other node's first... Target performance metrics data, For one of the nodes in the target node pair The mean of the target performance index data, For the target node pair, another node The mean of the target performance index data, To determine the amount of performance data to be observed, in a specific embodiment, the underlying observation logic of the Vanka engineering system collects and records data on a per-second basis. It can take values ​​greater than 60.

[0082] Furthermore, based on the time decay factor and Pearson correlation coefficient... Calculate the edge weights logical_weight. Specifically, in an optional embodiment, the time decay factor decay and the Pearson correlation coefficient can be used. The product of these values ​​is used as the edge weight logical_weight, i.e., logical_weight = r × decay.

[0083] It should be noted that, in fact, in specific embodiments, the Pearson correlation coefficient can be used. It is directly used as the edge weight logical_weight. However, considering the timeliness, in this embodiment, the Pearson correlation coefficient is adjusted using a time decay factor decay. After dynamic correction, the edge weight logical_weight is obtained, so that the edge weight logical_weight exhibits decay characteristics over time.

[0084] In one optional embodiment, the edge weight `logical_weight` ranges from [-1, 1]. Specifically, when the edge weight is greater than 0, it indicates that the two nodes in the target node pair are positively correlated and completely influence each other. That is, when one node fails, it will inevitably cause the other node to fail. For example, when a core module (one of the nodes) fails, the sub-modules that depend on it (the other node) will inevitably fail to work. Furthermore, within the range of 0 to 1, the larger the edge weight `logical_weight`, the stronger the correlation between the two nodes.

[0085] When the edge weight `logical_weight` equals 0, it indicates that the two nodes in the target node pair are unrelated; that is, a failure in one node will not affect the normal operation of the other node. When the edge weight `logical_weight` equals -1, it indicates that the failures between the target node pairs are negatively correlated; that is, the failure will propagate in the back. Between -1 and 0, the smaller the edge weight, the greater the impact of the failure.

[0086] It should be noted that, based on this, the update mechanism for the graph structure can be formulated by refreshing the logical node attribute values ​​based on a sliding time window, that is, recalculating the physical propagation weights physical_weight, logical_weight, and fault propagation probability P, for example, updating once every 5 seconds.

[0087] Figure 4 This is a flowchart illustrating a cluster fault management method according to another embodiment of this application. In an optional embodiment, such as... Figure 4 As shown, based on the dynamic fault propagation model, a time-series-based four-dimensional fault visualization is constructed, including: S40: Set alarm thresholds for various performance indicators of multi-domain units; In a specific embodiment, to locate and display faults, corresponding alarm thresholds need to be set for various performance indicators of different domain units. These multi-domain units include several areas prone to faults, such as the computing domain, storage domain, network domain, power domain, and temperature domain.

[0088] For the computing domain, in one optional embodiment, the performance metrics of the computing domain include memory utilization, and the memory alarm threshold for memory utilization is determined based on a base alarm threshold and a task type coefficient. The task type coefficient reflects the degree of influence of task type on memory utilization, and is determined based on a pre-constructed first mapping relationship, which is the correspondence between task types and task type coefficients.

[0089] In one optional embodiment, the memory alarm threshold is the product of the base alarm threshold and the task type coefficient; wherein, the greater the memory requirement of the task type, the greater the task type coefficient, and the greater the memory alarm threshold.

[0090] Traditional monitoring of video memory utilization (t) often uses a fixed threshold (e.g., triggering an alarm when video memory utilization (t) exceeds 90%). However, different task types have different video memory requirements and stability. Therefore, to adapt to the needs of different tasks, this embodiment introduces a task type coefficient to dynamically adjust the video memory alarm threshold. For ease of understanding, examples will be provided below.

[0091] For example, if the basic alarm threshold is 90%, and the task type coefficient is 1.2 for training tasks, it is understandable that training tasks have high and fluctuating requirements for GPU memory. Therefore, the threshold can be increased to 108% (i.e., 90% * 1.2) to avoid frequent false alarms.

[0092] For reasoning tasks, the task type coefficient is 0.8. Reasoning tasks have high stability requirements. Therefore, the threshold can be reduced to 72% (i.e., 90% * 0.8), which can provide early warning of potential risks and solve the problem that static thresholds cannot adapt to diverse business scenarios.

[0093] Table 2 is a schematic table of threshold rules for computing domain performance indicators provided in the embodiments of this application. In specific embodiments, the computing domain also includes multiple performance indicators such as GPU utilization (t) and task failure rate (t). For the threshold settings of other performance indicators of the computing domain, please refer to Table 2.

[0094] Table 2 is a schematic table of threshold rules for computational domain performance indicators. For the network domain, in one optional embodiment, the performance metrics of the network domain include packet loss rate, and the packet loss alarm threshold for the packet loss rate is determined according to a pre-constructed second mapping relationship; wherein, the second mapping relationship is the correspondence between the packet loss alarm threshold and service sensitivity, and service sensitivity is used to reflect the degree to which the service type is sensitive to the packet loss rate. In one optional embodiment, the higher the service sensitivity, the larger the packet loss alarm threshold.

[0095] Similarly, when monitoring network domain faults, a fixed value is typically set for the packet loss alarm threshold, for example, 5%. This is clearly unsuitable for different business scenarios. Therefore, in this embodiment, business sensitivity is introduced to achieve dynamic adjustment of the packet loss alarm threshold. For ease of understanding, an example will be provided below.

[0096] For example, scenarios such as real-time video conferencing and financial transactions are more sensitive to packet loss rates, so a smaller packet loss alarm threshold can be set (e.g., 3%). File transfer services, on the other hand, are less sensitive to packet loss rates, so the threshold can be relaxed to 5%. Thus, by precisely matching service type with network quality, false alarms or excessive alarms caused by a "one-size-fits-all" threshold are avoided, overcoming the static nature of traditional monitoring.

[0097] It should be noted that this application does not specifically limit the first mapping relationship and the second mapping relationship, and they can be added, deleted, and modified according to actual business needs. Table 3 is a schematic table of threshold rules for network domain performance indicators provided in an embodiment of this application. Similarly, in specific embodiments, the network domain also includes multiple performance indicators such as network congestion, network packet loss, and network connection interruption. For the threshold settings of other performance indicators of the network domain, please refer to Table 3.

[0098] Table 3 is a schematic table of threshold rules for network domain performance indicators. Table 4 is a schematic table of threshold rules for storage domain performance indicators provided in the embodiments of this application. In specific embodiments, the storage domain includes performance indicators such as I / O throughput latency, read / write latency, and queue depth. For the threshold settings of storage domain performance indicators, please refer to Table 4.

[0099] Table 4 is a schematic diagram of the threshold rules for storage domain performance indicators. Table 5 is a schematic table of threshold rules for performance indicators of a specified domain provided in the embodiments of this application. The specified domain includes various areas where failures may occur, such as the power domain, management domain, and temperature domain. Table 5 mainly shows current stability and temperature; specific examples are not provided in this application, please refer to Table 5 for details.

[0100] Table 5 is a schematic table of threshold rules for specified domain performance indicators. Therefore, based on Tables 2 to 5, alarm thresholds are set for various performance indicators of each domain unit. It should be noted that the above tables are for illustrative purposes only. The alarm thresholds for different performance indicators can be set according to actual business needs. In addition, alarm thresholds for each domain unit can be added, deleted, and modified, which is not limited in this application.

[0101] S41: Abstract the chain of faults in the dynamic fault propagation model into visual elements; S42: Construct a time-series-based four-dimensional fault visualization based on pre-set visualization specifications and visualization elements; wherein, the visualization specifications include node rendering rules, color encoding rules, and cluster topology rules.

[0102] Furthermore, such as Figure 2 As shown, chain faults in the dynamic fault propagation model are abstracted into visual elements; that is, chain faults such as GPU memory leaks and network congestion are abstracted into visual elements in a spatiotemporal field. Therefore, based on pre-defined visualization specifications and elements, a time-series-based four-dimensional fault visualization can be constructed, integrating time series data with physical-logical topology. This is achieved through spatiotemporal field modeling, addressing the core pain points of "data silos" and "cognitive delays" in current cluster fault analysis, improving the readability of complex cluster fault analysis, and enhancing the work efficiency of operations and maintenance personnel.

[0103] In one optional embodiment, for node rendering rules, physical nodes are mapped to three-dimensional space according to the actual coordinates of the hardware at the computing center, and logical nodes are attached to physical nodes by clicking "floating labels". When the mouse hovers over a physical node, a floating window including the logical node is displayed.

[0104] In specific embodiments, the computing server, storage server, and switch can be represented by a cube, or by geometric shapes such as cylinders or spheres; this application does not limit this representation. Data transmission is represented by lines, while the time dimension, such as... Figure 2As shown, in one alternative embodiment, a sliding interaction can be performed via a timeline selector to support time retracing of the fault propagation process.

[0105] Regarding the color coding rules, in one optional embodiment, for server rendering, green represents a normal state, and red represents a fault alarm, highlighted when a fault occurs. For network transmission, red represents a transmission interruption, and green represents a smooth transmission. In fact, this application does not specifically limit the colors selected.

[0106] In one optional embodiment, the multi-domain unit includes a computing domain, a storage domain, and a network domain; the cluster topology adopts a hierarchical leaf-spine architecture. The hierarchical leaf-spine architecture includes a first layer of spine switches, a second layer of leaf switches, a third layer of server nodes, and a fourth layer of computing nodes. In a specific embodiment, the computing domain corresponds to the third and fourth layers, the storage domain corresponds to the third layer, the network domain corresponds to the first and second layers, and the network communication between each layer is defined.

[0107] It's worth noting that the physical topology graph employs a hierarchical leaf-spine architecture, enabling high bandwidth, low latency, and scalability. Furthermore, it utilizes physical hierarchy control, with a "drill-down" interactive approach allowing expansion on demand; clicking on a corresponding leaf / spine node expands the next level of connections. For example... Figure 2 As shown, the system automatically generates the minimum sufficient causal link for failures, highlights the critical path, and displays other normal devices with a secondary transparency of a preset percentage (e.g., 80%).

[0108] Figure 5 This is a schematic diagram of a layered leaf-ridge architecture provided in an embodiment of this application, combined with... Figure 2 and Figure 5 As can be seen, the first layer of the layered leaf-spine architecture is the Spine Layer, used for interconnecting leaf switches to build a cluster-level non-blocking network. The second layer is the Leaf Layer, used for directly connecting compute nodes and providing rack-level switching. The third layer is the Node Layer, specifically including compute server nodes and storage server nodes; clicking drill-down expands the internal hierarchical structure of the nodes. The fourth layer is the Internal Layer, in one optional embodiment, where a single compute node contains 8 GPU cards interconnected via NVLink. The internal connections of the nodes are hidden to ensure clear and unambiguous visualization.

[0109] In the above embodiments, the method for managing cluster failures has been described in detail. This application also provides an embodiment of a cluster failure management device.

[0110] Figure 6This is a schematic diagram of the structure of a cluster fault management device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes: Data acquisition module 60 is used to acquire performance index data of multiple domain units in the cluster; Model building module 61 is used to build a dynamic fault propagation model of the cluster based on graph model and performance index data; Visualization building module 62 is used to construct a time-series-based four-dimensional fault visualization based on the dynamic fault propagation model; The fault analysis module 63 is used to perform cluster fault analysis through four-dimensional fault visualization.

[0111] Furthermore, the cluster fault management device provided in this application embodiment also includes: The definition module is used to define the nodes and edges of a graph structure based on a graph model and performance metric data. The fault propagation probability determination module is used to determine the time-varying fault propagation probability between node pairs with edges in a graph structure. The update mechanism formulation module is used to formulate the dynamic update mechanism of the graph structure, thereby obtaining a dynamic fault propagation model.

[0112] The first definition submodule is used to define hardware entities in the cluster as physical nodes; to split multidimensional vectors and define various performance index data after splitting as logical nodes to form nodes in a graph structure. The relationship binding module is used to bind the correspondence between physical nodes and logical nodes using relational database tables. The second definition submodule is used to define the edges and edge types in the graph structure based on the correspondence; the edge types include intra-domain propagation edges and cross-domain propagation edges.

[0113] The acquisition module is used to acquire the time decay factor, the physical distance between target node pairs, and the target performance index data corresponding to the target node pairs; wherein, the target node pairs are node pairs with edges in the graph structure; The physical propagation weight determination module is used to determine the physical propagation weight between target node pairs based on physical distance. The physical propagation weight is the probability of a fault propagating between target node pairs through physical connections. The closer the physical distance, the greater the physical propagation weight, which represents the higher the probability of a fault propagating through physical connections. The edge weight determination module is used to determine the edge weights between target node pairs based on the target performance index data; the edge weight represents the degree of impact of the failure of one node in a target node pair on the operation of the other node. The first determination module is used to determine the fault propagation probability based on the time decay factor, physical propagation weight, and edge weight.

[0114] The correlation coefficient determination module is used to determine the Pearson correlation coefficient between target node pairs based on the target performance index data. The second determination module is used to determine the edge weights based on the time decay factor and the Pearson correlation coefficient. The edge weights range from -1 to 1. When the edge weight is greater than 0, the faults between the target node pairs are positively correlated, and the larger the edge weight, the greater the influence. When the edge weight is equal to 0, the faults between the target node pairs are not correlated. When the edge weight is less than 0, the faults between the target node pairs are negatively correlated, and the smaller the edge weight, the greater the influence.

[0115] The alarm threshold setting module is used to set alarm thresholds for various performance indicators of multi-domain units; The Visual Element Abstraction Module is used to abstract chain faults in dynamic fault propagation models into visual elements. The target building module is used to construct a time-series-based four-dimensional fault visualization based on pre-set visualization specifications and visualization elements; the visualization specifications include node rendering rules, color encoding rules, and cluster topology rules.

[0116] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device includes: a memory 70 for storing computer programs; The processor 71 is configured to execute a computer program to implement the steps of the cluster fault management method as described in the above embodiments.

[0117] The electronic devices provided in this embodiment may include, but are not limited to, tablet computers, laptop computers, or desktop computers.

[0118] The processor 71 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 71 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 71 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 71 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 71 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0119] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 70 is used to store at least the following computer program 701, which, after being loaded and executed by the processor 71, is capable of implementing the relevant steps of the cluster failure management method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 70 may also include an operating system 702 and data 703, etc., and the storage method may be temporary storage or permanent storage. The operating system 702 may include Windows, Unix, Linux, etc. The data 703 may include, but is not limited to, the relevant data involved in the cluster failure management method.

[0120] In some embodiments, the electronic device may further include a display screen 72, an input / output interface 73, a communication interface 74, a power supply 75, and a communication bus 76.

[0121] Those skilled in the art will understand that Figure 7 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0122] The electronic device provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the cluster fault management method described in the above embodiments.

[0123] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A method of managing cluster failures, characterized by, The method includes: Obtain performance index data of multi-domain units in the cluster; the multi-domain units include computing domain, storage domain and network domain; Based on the graph model, a dynamic fault propagation model for the cluster is constructed using the performance metric data. Based on the dynamic fault propagation model, a time-series-based four-dimensional fault visualization is constructed; the four-dimensional fault visualization includes spatial dimension, time dimension, data dimension and fault state dimension. Cluster fault analysis is performed using the aforementioned four-dimensional fault visualization; The graph-based model, using the performance metric data, constructs a dynamic fault propagation model for the cluster, including: Based on the graph model, the nodes and edges of the graph structure are defined using the performance index data; Determine the time-varying probability of fault propagation between node pairs with edges in the graph structure; A dynamic update mechanism for the graph structure is established to obtain the dynamic fault propagation model; The performance metric data is a multi-dimensional vector based on a time series; the definition of nodes and edges of the graph structure based on the graph model and the performance metric data includes: The hardware entities in the cluster are defined as physical nodes; the multidimensional vector is split, and the various performance index data after splitting are defined as logical nodes to form the nodes of the graph structure. A relational database table is used to bind the correspondence between the physical nodes and the logical nodes; Based on the correspondence, the edges and edge types in the graph structure are defined; the edge types include intra-domain propagation edges and cross-domain propagation edges.

2. The cluster fault management method as described in claim 1, characterized in that, Determining the time-varying failure propagation probability between node pairs with edges in the graph structure includes: Obtain the time decay factor, the physical distance between target node pairs, and the target performance index data corresponding to the target node pairs; wherein, the target node pairs are node pairs with edges in the graph structure; Based on the physical distance, a physical propagation weight is determined between the target node pairs; the physical propagation weight is the probability of the target node pairs propagating faults through physical connections; the closer the physical distance, the greater the physical propagation weight, which represents a higher probability of the faults propagating through physical connections. Based on the target performance index data, determine the edge weights between the target node pairs; the edge weights represent the degree to which a failure of one node in a target node pair affects the operation of the other node. The fault propagation probability is determined based on the time decay factor, the physical propagation weight, and the edge weight.

3. The cluster fault management method as described in claim 2, characterized in that, The step of determining the edge weights between the target node pairs based on the target performance index data includes: Based on the target performance index data, determine the Pearson correlation coefficient between the target node pairs; The edge weights are determined based on the time decay factor and the Pearson correlation coefficient. The edge weights range from [-1, 1]. When the edge weight is greater than 0, the faults between the target node pairs are positively correlated, and the greater the edge weight, the higher the degree of influence. When the edge weight is equal to 0, the faults between the target node pairs are not correlated. When the edge weight is less than 0, the faults between the target node pairs are negatively correlated, and the smaller the edge weight, the higher the degree of influence.

4. The cluster fault management method as described in claim 1, characterized in that, The step of constructing a time-series-based four-dimensional fault visualization based on the dynamic fault propagation model includes: Set alarm thresholds for various performance indicators of the multi-domain units; The chain-like faults in the dynamic fault propagation model are abstracted into visual elements; Based on the pre-set visualization specifications and visualization elements, a time-series-based four-dimensional fault visualization is constructed; wherein, the visualization specifications include node rendering rules, color encoding rules, and cluster topology rules.

5. The cluster fault management method as described in claim 4, characterized in that, The cluster topology structure rules adopt a hierarchical leaf-spine architecture; The hierarchical leaf-spine architecture includes a first layer of spine switch layer, a second layer of leaf switch layer, a third layer of server node layer, and a fourth layer of compute node layer; The computing domain corresponds to the third and fourth layers; the storage domain corresponds to the third layer; the network domain corresponds to the first and second layers; and there is network communication between the layers.

6. The cluster fault management method as described in claim 5, characterized in that, The performance metrics of the computing domain include video memory utilization, and the video memory alarm threshold of the video memory utilization is determined based on the basic alarm threshold and the task type coefficient; wherein, the task type coefficient is used to reflect the degree of influence of the task type on the video memory utilization, and the task type coefficient is determined based on a pre-constructed first mapping relationship, which is the correspondence between the task type and the task type coefficient; The performance metrics of the network domain include packet loss rate, and the packet loss alarm threshold of the packet loss rate is determined according to a pre-constructed second mapping relationship; wherein, the second mapping relationship is the correspondence between the packet loss alarm threshold and the service sensitivity, and the service sensitivity is used to reflect the degree of sensitivity of the service type to the packet loss rate.

7. The cluster fault management method as described in claim 6, characterized in that, The video memory alarm threshold is the product of the basic alarm threshold and the task type coefficient; wherein, the greater the video memory requirement of the task type, the greater the task type coefficient, and the greater the video memory alarm threshold. The higher the business sensitivity, the larger the packet loss alarm threshold.

8. A cluster fault management device, characterized in that, The device includes: The data acquisition module is used to acquire performance indicator data of multi-domain units in the cluster; the multi-domain units include computing domain, storage domain and network domain; the performance indicator data is a multi-dimensional vector based on time series. The model building module is used to construct a dynamic fault propagation model of the cluster based on the graph model and the performance index data. A visualization building module is used to construct a time-series-based four-dimensional fault visualization based on the dynamic fault propagation model; the four-dimensional fault visualization includes spatial dimension, time dimension, data dimension and fault state dimension; The fault analysis module is used to perform cluster fault analysis through the four-dimensional fault visualization. A definition module is used to define the nodes and edges of the graph structure based on the graph model and the performance index data; The fault propagation probability determination module is used to determine the fault propagation probability of node pairs with edges in the graph structure as a function of time. The update mechanism formulation module is used to formulate the dynamic update mechanism of the graph structure to obtain the dynamic fault propagation model. The first definition submodule is used to define hardware entities in the cluster as physical nodes; to split the multidimensional vector and define the various performance index data after splitting as logical nodes, so as to form the nodes of the graph structure; The relationship binding module is used to bind the correspondence between the physical nodes and the logical nodes using relational database tables; The second definition submodule is used to define the edges and edge types in the graph structure based on the correspondence; the edge types include intra-domain propagation edges and cross-domain propagation edges.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the cluster fault management method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the cluster fault management method according to any one of claims 1 to 7.

Images