Server fault debugging method and device, electronic equipment and storage medium

Abstract

The application relates to a server fault debugging method and device, electronic equipment and a storage medium, wherein the server fault debugging method comprises the following steps: in response to a triggering event of server fault, obtaining running state data of each server component in a plurality of hierarchical structures of a server; constructing a cross-layer fault propagation graph according to the running state data of the plurality of server components, wherein edges in the cross-layer fault propagation graph are used for connecting nodes with cross-layer fault correlation; determining a fault root cause node of cross-layer fault in a plurality of nodes of the cross-layer fault propagation graph according to the cross-layer fault propagation graph and the running state data of the plurality of server components; determining a fault repair action according to the fault root cause node, and executing the fault repair action. The application embodiment can realize cross-layer correlation analysis of a composite fault, locate a fault source, and automatically repair the fault, thereby effectively shortening fault locating and repairing time, and improving the efficiency and accuracy of server fault debugging.

Description

Technical Field

[0001] This application relates to the field of server fault handling, and in particular to a server fault debugging method, apparatus, electronic device and storage medium. Background Technology

[0002] In the field of server fault diagnosis technology, the current mainstream technical solution adopts a layered and independent data acquisition architecture, which divides the server's operating data into three independent dimensions: hardware layer, system layer and application layer. Data in the hardware layer is collected through BMC sensors, data in the system layer is extracted from OS logs, and data in the application layer is summarized based on business logs. The data acquisition links of each layer are isolated from each other, forming a layered and siloed technical status quo.

[0003] For complex faults triggered across multiple levels, such as cascading faults caused by abnormal BIOS voltage leading to CPU overheating and subsequent OS process blocking, existing technologies cannot achieve automatic correlation analysis of cross-level data. Instead, they rely on technicians to manually trace and locate faults layer by layer, resulting in incomplete functional coverage and low automation. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a server fault debugging method, apparatus, electronic device and storage medium.

[0005] Firstly, this application provides a server fault debugging method, including: In response to a server failure trigger event, the system acquires the operational status data of each server component in the multiple hierarchical structure of the server. A cross-layer fault propagation graph is constructed based on the operating status data of multiple server components. The cross-layer fault propagation graph includes multiple nodes and edges. Each node corresponds to a server component, and the edges are used to connect nodes that have cross-layer fault association relationships. Based on the cross-layer fault propagation diagram and the operating status data of multiple server components, the root cause node of the cross-layer fault is determined from multiple nodes in the cross-layer fault propagation diagram. The fault repair action is determined based on the root cause node of the fault, and the fault repair action is executed.

[0006] This application embodiment collects operational status data of the server's five-layer architecture in response to fault-triggered events, achieving comprehensive acquisition of heterogeneous data across the entire stack and solving the layered silo problem of traditional fault diagnosis. By constructing a cross-layer fault propagation graph, server components at different levels are linked through fault correlations. Based on the cross-layer fault propagation graph and the operational status data of multiple server components, the root cause node of the fault is determined across layers, realizing cross-layer correlation analysis of complex faults and locating the source of the fault. This eliminates the need for manual layer-by-layer tracing and location, significantly improving the accuracy of root cause location. By matching and executing repair actions through a policy library, automated fault repair is achieved, effectively shortening fault location and repair time and improving the efficiency and accuracy of server fault debugging.

[0007] Optionally, based on the cross-layer fault propagation graph and the operational status data of multiple server components, the root cause node of the cross-layer fault is determined from multiple nodes in the cross-layer fault propagation graph, including: For each node in the cross-layer fault propagation graph, an initial feature vector is determined based on the running status data of the server component corresponding to the node; Obtain the initial feature vectors of the cross-layer neighbor nodes of the node; The initial feature vector of the node is used to determine the cross-layer fused feature vector of the node by combining the initial feature vectors of the cross-layer neighbor nodes. Based on the cross-layer fusion feature vector of each node in the cross-layer fault propagation graph, the root cause node of the cross-layer fault is determined among multiple nodes in the cross-layer fault propagation graph.

[0008] This application embodiment determines the initial feature vector of a node based on the server component's operating status data, laying a data foundation for cross-layer feature analysis; it retrieves the features of cross-layer associated nodes by obtaining the initial feature vectors of cross-layer neighbor nodes; it obtains a cross-layer fused feature vector by fusing the initial feature vectors of the target node and cross-layer neighbor nodes, integrating the feature information of the node itself and cross-layer associated nodes, thus improving the comprehensiveness of feature representation; and it determines the root cause node of the fault based on the cross-layer fused feature vector, fully considering the impact of cross-layer association on the fault, and significantly improving the accuracy of cross-layer fault root cause localization.

[0009] Optionally, based on the cross-layer fusion feature vector of each node in the cross-layer fault propagation graph, the root cause node of the cross-layer fault is determined among multiple nodes in the cross-layer fault propagation graph, including: For each node in the cross-layer fault propagation graph, the attention score of the node is determined based on the cross-layer fusion feature vector of the node; The edge weights of the edges between the node and each of its cross-layer neighbor nodes are determined based on the cross-layer fusion feature vector and the attention score. Based on the multiple edge weights of each node in the cross-level fault propagation graph, the root cause node of the cross-level fault is determined.

[0010] This application embodiment determines node attention scores based on cross-layer fusion feature vectors, which highlights key features in fault analysis and improves the effectiveness of feature utilization. By determining edge weights through cross-layer fusion feature vectors and attention scores, the calculation of edge weights is made to better fit the actual needs of fault analysis and accurately characterize the strength of fault correlation between nodes. The root cause node of the fault is determined based on the edge weight of each node. The accurate location of the root cause node is achieved through quantified weight calculation, which effectively reduces the false alarm rate and false negative rate of cross-layer fault root cause location.

[0011] Optionally, the root cause node of the cross-level fault is determined based on the multiple edge weights of each node in the cross-level fault propagation graph, including: For each node in the cross-layer fault propagation graph, determine the sum of the edge weights of the outgoing edges of the node and the sum of the edge weights of the incoming edges of the node; Determine the difference between the sum of the weights of the outgoing edges of the node and the sum of the weights of the incoming edges of the node; The node corresponding to the largest difference is identified as the root cause node of the cross-level fault.

[0012] This application embodiment achieves quantitative statistics on the impact and degree of influence of node failures by determining the sum of outgoing edge weights and the sum of incoming edge weights for each node; by calculating the difference between the sum of outgoing edge weights and the sum of incoming edge weights, the likelihood of a node being the root cause of a failure can be intuitively quantified; the node corresponding to the largest difference is determined as the root cause node, and the accurate and rapid location of the root cause node is achieved through explicit numerical sorting, avoiding the one-sidedness of manual layer-by-layer investigation and significantly shortening the root cause location time for cross-level failures.

[0013] Optionally, determining and executing fault repair actions based on the root cause node includes: Obtain the first fault indicator parameter when the server fails; Obtain the hierarchical features of the hierarchical structure where the root cause node of the fault is located; The fault repair action is determined based on the first fault indicator parameter, the hierarchical characteristics, and the fault root cause node, and then the fault repair action is executed.

[0014] This application embodiment obtains the first fault indicator parameter when the server fails, obtains the hierarchical characteristics of the level where the root cause node of the fault is located, and determines the fault repair action based on the first fault indicator parameter, hierarchical characteristics and the root cause node of the fault. This achieves accurate matching between the repair action and the fault and level, improves the pertinence and effectiveness of the fault repair action, and avoids secondary faults caused by blind operation.

[0015] Optionally, a fault repair action is determined based on the first fault indicator parameter, the hierarchical characteristics, and the fault root cause node, and the fault repair action is executed, including: Based on the first fault indicator parameters, the hierarchical features, and the fault root cause node, candidate repair actions are matched in the strategy library, and the current first weight of each candidate repair action is obtained. The candidate repair action with the highest first weight is determined as the fault repair action; The fault repair action is performed via an out-of-band channel.

[0016] In this embodiment, by matching candidate repair actions in the policy library and obtaining the first weight, multiple repair schemes are selected, improving the flexibility of fault repair; the candidate repair action with the largest weight is determined as the fault repair action, and the optimal selection of the repair scheme is achieved based on the weight of reinforcement learning, improving the efficiency of fault repair; by executing the fault repair action through the out-of-band channel, the server can be debugged without downtime, ensuring the continuous operation of services and eliminating the business interruption problem of traditional repair methods.

[0017] Optionally, determining and executing fault repair actions based on the first fault indicator parameters, the hierarchical features, and the fault root cause node further includes: After performing the fault repair action through the out-of-band channel, the second fault index parameter at the end of the observation window and the fault location time used to determine the root cause node of the fault are obtained. If the second fault indicator parameter is abnormal or a secondary fault occurs, obtain the current false alarm rate; The reward value is determined based on the false alarm rate and the fault location time. The reinforcement learning model is input with the first fault indicator parameter, the hierarchical feature, the fault root cause node, the fault repair action, the reward value, and the second fault indicator parameter, so that the reinforcement learning model outputs the action value of the fault repair action; The first weight corresponding to the fault repair action is updated using the second weight corresponding to the value of the action.

[0018] This application embodiment achieves quantitative evaluation of fault repair effect and location efficiency by acquiring the second fault index parameter and fault location time of the observation window; by acquiring the false alarm rate and calculating the reward value, the repair effect and location efficiency are transformed into training indicators for the reinforcement learning model; by inputting multi-dimensional fault information into the action value of the reinforcement learning model, the effect of repair actions is accurately quantified; by using the action value to update the weight of the repair actions, the weight of the repair actions in the policy library is dynamically optimized, enabling the policy library to adapt to different fault scenarios, continuously improve the efficiency and accuracy of fault repair, and reduce the false alarm rate.

[0019] Secondly, this application provides a server fault debugging device, comprising: The data acquisition module is used to acquire the operating status data of each server component in the multiple hierarchical structure of the server in response to the server failure triggering event; The graph construction module is used to construct a cross-layer fault propagation graph based on the running status data of multiple server components. The cross-layer fault propagation graph includes multiple nodes and edges. The nodes correspond one-to-one with the server components, and the edges are used to connect nodes that have cross-layer fault association relationships. The root cause determination module is used to determine the root cause node of the cross-level fault in multiple nodes of the cross-level fault propagation graph based on the cross-level fault propagation graph and the operating status data of multiple server components. The action determination module is used to determine the fault repair action based on the fault root cause node and execute the fault repair action.

[0020] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the server fault debugging method described in any of the first aspects.

[0021] Fourthly, this application provides a computer-readable storage medium storing a server fault debugging method program, which, when executed by a processor, implements the steps of the server fault debugging method described in any of the first aspects.

[0022] The technical solutions provided in this application have the following advantages compared with the prior art: This application embodiment collects operational status data of the server's five-layer architecture in response to fault-triggered events, achieving comprehensive acquisition of heterogeneous data across the entire stack and solving the layered silo problem of traditional fault diagnosis. By constructing a cross-layer fault propagation graph, server components at different levels are linked through fault correlations. Based on the cross-layer fault propagation graph and the operational status data of multiple server components, the root cause node of the fault is determined across layers, realizing cross-layer correlation analysis of complex faults and locating the source of the fault. This eliminates the need for manual layer-by-layer tracing and location, significantly improving the accuracy of root cause location. By matching and executing repair actions through a policy library, automated fault repair is achieved, effectively shortening fault location and repair time and improving the efficiency and accuracy of server fault debugging. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a server fault debugging method provided in this application embodiment; Figure 2 A structural diagram of a server fault debugging device provided in an embodiment of this application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] For complex faults triggered across multiple levels, such as cascading faults where abnormal BIOS voltage causes CPU overheating, leading to OS process blocking, existing technologies cannot achieve automatic correlation analysis of cross-level data. They still rely on technicians to manually trace and locate faults layer by layer, resulting in incomplete functional coverage and low automation. Therefore, this application provides a server fault debugging method, apparatus, electronic device, and storage medium.

[0028] This application provides a server fault debugging method, such as... Figure 1 As shown, it includes: Step S101: In response to a server failure trigger event, obtain the operating status data of each server component in the multiple hierarchical structure of the server; In this embodiment, the triggering event refers to the fault-related triggering signal that occurs at each level of the server, which is the triggering condition for starting the fault repair process; the hierarchical structure refers to the server's central processing unit (CPU) layer, baseboard management controller (BMC) layer, basic input / output system (BIOS) layer, operating system (OS) layer, and application layer, etc.; server components refer to the hardware or software entities that participate in the operation of the server at each level; and the running status data refers to various indicators, logs, configurations, and other data generated by the server components during operation.

[0029] In practical applications, corresponding acquisition probes can be pre-deployed at the server's central processing unit (CPU) layer, baseboard management controller (BMC) layer, basic input / output system (BIS) layer, operating system layer, and application layer. Examples include: CPU layer probes: reading temperature, voltage, frequency, and hardware error counts via the Machine Status Register (MSR), Machine Check Architecture (MCA), and Platform Environment Control Interface (PECI) bus; BMC layer probes: pulling sensor (temperature, voltage, fan speed) and system event (SEL) logs via IPMI / Redfish; BIOS layer probes: mirroring the configuration area to BMC shared memory during startup via the BIOS Setup Protocol, which is then periodically read by the BMC during runtime; OS layer probes: collecting kernel scheduling, file system, network, and memory events using an extended Berkeley packet filter and Linux kernel dynamic probes (kprobe); and application layer probes: embedding a lightweight SDK within the Java / Python / Go runtime to collect business metrics, error codes, and call chains via JVM TI, LD_PRELOAD, USDT, and other mechanisms.

[0030] In this step, when the system detects a server failure trigger event, the acquisition probes at each layer simultaneously start data acquisition. At the central processing unit layer, the probes distributed at each layer collect the operating status data of each server component, and then push it to the central diagnostic node (which can be located on the management board in the same chassis or on an external diagnostic server) through a gRPC streaming pipeline.

[0031] For example: the application layer monitoring software development kit detected an order service response delay of 3.2 seconds, triggering a fault event; the CPU layer probe read a CPU temperature of 98°C; the baseboard management controller layer probe read a fan speed of 3600 RPM; the basic input / output system layer probe read a fan curve in quiet mode; the operating system layer probe collected a memory reclamation process delay of 15 milliseconds; and the application layer probe collected an order service error rate of 10%.

[0032] The raw data collected by each probe is first corrected by local timestamp (using BMC's RTC as the benchmark); the data is then serialized into a JSON format UDIM message by the local agent, with fields including: {ts, layer_id, node_id, metric_type, value, error_code, context_snapshot}. Where ts represents the timestamp, the data type is a high-precision integer (milliseconds / microseconds), and the value rule is 1735689600123 (milliseconds, corresponding to 2025-01-01 00:00:00.123). The rule is: all levels of data are uniformly converted to UTC timestamps with a precision of ≥ milliseconds to avoid cross-time zone / backup clock deviations. The core function is to serve as the core of time alignment and to associate fault data of different levels within the same time window. For example, when ts=1735689600123, abnormal CPU temperature and application timeout can be linked for analysis. `layer_id` represents the layer identifier. Its data type is an enumerated value / integer. The enumeration rules are: 1=CPU (hardware core), 2=BMC layer (out-of-band management), 3=BIOS layer (firmware configuration), 4=OS layer (operating system), 5=application layer (business). Its core function is to mark the technical layer to which the data belongs. It is the basic node attribute for GNN to build a cross-layer fault propagation graph. For example, the association between `layer_id=1` (CPU) and `layer_id=5` (application) can locate the fault link from hardware to business. `node_id` represents the node ID, and its data type is string (globally unique). The naming rule is: [layer abbreviation]-[device identifier]-[component name]. Example: CPU-CPU0 (CPU layer 0 CPU) BMC-00:1B:44:11:3A:B7 (BMC MAC address) OS-process-12345 (process with PID=12345 in the OS layer), APP-order-service-01 (order service instance 01 in the application layer). Its core function is to accurately locate the specific hardware / software node where the fault occurs. It is the unique identifier of the node in the GNN graph. For example, node_id=CPU-CPU0 can be directly associated with the CPU's temperature, voltage and other indicators. `metric_type` represents the metric type, with data types including enumerated values / strings. Examples of hierarchical classifications include: CPU layer: temperature, voltage, frequency; BMC layer: fan_speed, sel_error; BIOS layer: fan_curve, power_plan; OS layer: cpu_usage, mem_used; Application layer: latency, error_rate. Its core function is to define the business meaning of the data, distinguishing between metric values, logs, and configuration items. It is a core dimension for filtering / aggregating data; for example, filtering for `metric_type=temperature` will extract all hardware temperature data. The value represents the indicator value, and the data type is dynamic (numeric / string / boolean). For example: CPU temperature: 98.5 (floating point, unit °C), fan speed: 3600 (integer, unit rpm), BIOS fan curve: quiet_mode (string), application timeout: true (boolean). Its core function is to store the specific value of the indicator, which is the core basis for fault judgment. For example, if value > 95 (CPU temperature), it can trigger the overheat warning rule. `error_code` represents the error code. Its data type is string / integer (customizable), and the encoding rule is: [layer_id]-[error type]-[specific code]. Examples: 1-001 (CPU layer voltage anomaly), 4-005 (OS layer process blocking), 5-010 (application layer database connection failure). No error: null / 0. Its core function is to standardize fault identification, replacing the native, non-uniform error codes of each layer (such as BMC's 0x22 and OS's EAGAIN), facilitating cross-layer fault matching. For example, `error_code=1-001` can directly link the causal relationship between CPU voltage anomaly and application timeout. `context_snapshot` represents a context snapshot, with data type JSON string. Example (CPU layer exception snapshot): `{"core_count":8,"load_avg":7.8,"power_consumption":120,"last_` operation":"bios_power_plan_update"}, Example (application layer exception snapshot): {"request_id"} :"req-12345","sql":"select The core function of `from order where id=?","execution_time":5000}` is to store the surrounding context information when a fault occurs, supplementing the shortcomings of a single indicator. It is a contextual feature of GNN root cause analysis. For example, the `last_operation` in the snapshot can be used to locate the root cause of CPU overheating by modifying the BIOS power plan.

[0033] Step S102: Construct a cross-layer fault propagation graph based on the operating status data of multiple server components. The cross-layer fault propagation graph includes multiple nodes and edges. Each node corresponds to a server component, and the edges are used to connect nodes that have cross-layer fault association relationships. In this embodiment, the cross-layer fault propagation graph refers to a heterogeneous graph constructed with server components as nodes and cross-layer fault association relationships between components as edges. A node is a basic unit of the graph structure after abstracting server components and is the core entity of fault association. For example, node = {CPU, DIMM, BIOS_Setting, OS_Process, App_Request}. The definition of each node is shown in Table 1 below. Nodes represent key hardware / software components in the entire server stack that participate in fault propagation or association, covering five layers: CPU, BMC, BIOS, OS, and application. They are the core objects of GNN algorithm analysis. An edge is a graph structure connection unit that represents the cross-layer fault association relationship between server components. It is the fault association relationship between nodes. Edge = {trigger, dependency, cause, run}. An edge represents the causal, dependency, or run relationship between different nodes (cross-layer / same layer) and is the key to GNN algorithm identifying fault propagation paths. Cross-layer association analysis is achieved by defining clear relationship types. The specific definitions are shown in Table 2 below. Cross-layer fault association relationship refers to the trigger, dependency, cause, and run relationships between server components at different levels caused by faults.

[0034] Table 1

[0035] Table 2

[0036] In this step, the operational status data collected from each layer is standardized into Unified Debug Information Model (UDIM) messages. After receiving the messages, the central diagnostic node maps each server component to a node in the cross-layer fault propagation graph. Based on the fault associations between server components, such as abnormal configuration of the basic input / output system triggering a central processing unit failure, or application requests depending on operating system processes, edges are established between the corresponding nodes. The attributes of the edges are marked with the corresponding cross-layer fault association type.

[0037] For example: Map CPU 0, BIOS fan curve configuration, memory reclamation process, and order service request to nodes respectively. If CPU 0 is overheated due to an abnormal BIOS fan curve configuration, establish an edge marking the trigger relationship between the corresponding nodes. If the order service request depends on the memory reclamation process, establish an edge marking the dependency relationship between the corresponding nodes.

[0038] Step S103: Based on the cross-layer fault propagation diagram and the operating status data of multiple server components, determine the root cause node of the cross-layer fault among multiple nodes in the cross-layer fault propagation diagram. In this embodiment of the application, the root cause node refers to the node that serves as the source of the fault in the cross-layer fault propagation graph, which is the node corresponding to the core server component that caused the entire cross-layer fault.

[0039] In this step, a graph neural network (GNN) combined with a root cause scoring function can be used to determine the root cause node of the cross-level fault among multiple nodes in the cross-level fault propagation graph. Step S104: Determine the fault repair action based on the fault root cause node, and execute the fault repair action.

[0040] In this embodiment, the fault repair action refers to the debugging and repair operations performed on the server component corresponding to the fault root cause node; the out-of-band channel refers to the remote communication channel independent of the business network implemented through the intelligent platform management interface IPMI / Redfish of the baseboard management controller.

[0041] In this step, based on information such as the type of the root cause node, the corresponding fault repair action can be matched in the preset strategy library. The repair action command is then sent to the corresponding server level through the out-of-band channel to perform the corresponding debugging and repair operation. After execution, a 30-60 second observation window is set to verify the repair effect.

[0042] For example: The root cause node of the fault is the Basic Input / Output System (PIOS) fan curve configuration node, which belongs to the PIOS layer. The fault characteristic is that the fan curve is in quiet mode, causing the CPU to overheat. The policy library matches the repair action of modifying the fan curve to performance mode and adjusting the fan duty cycle to 80%. The PATCH request of the Redfish protocol is sent to the PIOS layer to execute the modification operation of fan curve and duty cycle.

[0043] This application embodiment collects operational status data of the server's five-layer architecture in response to fault-triggered events, achieving comprehensive acquisition of heterogeneous data across the entire stack and solving the layered silo problem of traditional fault diagnosis. By constructing a cross-layer fault propagation graph, server components at different levels are linked through fault correlations. Based on the cross-layer fault propagation graph and the operational status data of multiple server components, the root cause node of the fault is determined across layers, realizing cross-layer correlation analysis of complex faults and locating the source of the fault. This eliminates the need for manual layer-by-layer tracing and location, significantly improving the accuracy of root cause location. By matching and executing repair actions through a policy library, automated fault repair is achieved, effectively shortening fault location and repair time and improving the efficiency and accuracy of server fault debugging.

[0044] In another embodiment of this application, step S102, based on the cross-layer fault propagation graph and the operating status data of multiple server components, determines the root cause node of the cross-layer fault among multiple nodes in the cross-layer fault propagation graph, including: Step S201: For each node in the cross-layer fault propagation graph, determine an initial feature vector based on the running status data of the server component corresponding to the node; In this embodiment of the application, the initial feature vector refers to a vector derived from the running status data of the server component and used to characterize the initial features of the node.

[0045] In this step, the core features of the running status data of the server components corresponding to each node are extracted, including timestamps, level identifiers, indicator types, indicator values, error codes, etc. The various features are standardized and quantified, and arranged according to preset dimensions to form the initial feature vector of each node.

[0046] For example, an initial feature vector is assigned to each node in the cross-layer fault propagation graph. The feature dimensions are customized according to the server scenario. For example: hardware node (CPU): features include voltage, temperature, utilization, and runtime (dimension=4); software node (OS process): features include CPU usage, memory usage, and number of log error codes (dimension=3); unified dimension: all node features are mapped to the same dimension (such as 128 dimensions) through linear transformation to ensure uniformity in subsequent calculations.

[0047] Step S202: Obtain the initial feature vectors of the cross-layer neighbor nodes of the node; In this embodiment of the application, a cross-layer neighbor node refers to a node in the cross-layer fault propagation graph that is connected to the target node by an edge and belongs to a different server level. It is a fault-associated node of the target node.

[0048] In this step, the topology of the cross-layer fault propagation graph is traversed, and all cross-layer neighbor nodes of each target node are found according to the edge connection relationship between nodes. The initial feature vectors of each determined cross-layer neighbor node are retrieved and summarized.

[0049] Step S203: Determine the cross-layer fusion feature vector of the node by combining the initial feature vector of the node with the initial feature vector of the cross-layer neighbor node; In this embodiment of the application, the cross-layer fusion feature vector refers to the vector obtained by fusing the initial feature vector of the target node with the initial feature vector of the cross-layer neighboring nodes, which can comprehensively represent the features of the target node and the cross-layer associated nodes.

[0050] In this step, the GraphSAGE algorithm in Graph Neural Networks (GNN) is used to aggregate the initial feature vector of the target node and the initial feature vectors of each cross-layer neighbor node. Multiple sets of vectors are fused into one vector through mean aggregation, pooling aggregation, and other methods to obtain the cross-layer fused feature vector of the target node. GraphSAGE can solve the problem of information fusion of heterogeneous nodes and provide a unified feature basis for weight calculation.

[0051] GraphSAGE does not directly calculate edge weights. Instead, it first aggregates the neighbor information of each node to obtain the node's fused features, laying the foundation for subsequent attention mechanism calculations: For each node v, it samples its k-th order neighbors (e.g., 1st order neighbors = directly related components, 2nd order neighbors = indirectly related components); it uses a mean aggregator to aggregate the neighbor features, obtaining the node's cross-layer fused feature vector. The parameters of the following formula are explained in Table 3 below:

[0052] Table 3

[0053] The meaning of each term in the formula: Its function is feature transformation, which projects the features of each neighbor onto a unified space; Its function is to perform nonlinear activation, suppress negative values, and retain positive fault propagation signals; Its function is to sample and aggregate neighbors, taking the average of all neighbors to obtain aggregated features.

[0054] Scenario Example: When analyzing CPU-CPU0: Neighbors include: BIOS_Setting (fan curve), DIMM-0 (memory status), and OS_Process (scheduling load). The formula calculates the average impact of these three heterogeneous components on the CPU, generating... (e.g., [0.42, 0.85, ...]), the node's own features are concatenated with the aggregated neighbor features to obtain the node's fused features. : The core of this step is to make each node's features include its own state + the state of associated components, which aligns with the characteristics of cross-layer linkage in servers. The parameters in the calculation formula for this fused feature are shown in Table 4 below.

[0055] Table 4

[0056] This application employs a concatenation-based, rather than additive, identity preservation mechanism to ensure that CPU temperature characteristics are not overwhelmed by BIOS configuration information. and It can come from different levels (hardware / firmware / system) to achieve cross-layer heterogeneous integration.

[0057] Example scenario: For CPU-CPU0 (abnormal temperature): =[98.0,1.68,8,…] (self-temperature 98℃, voltage 1.68V, 8 cores). =[0.42,0.85,…] (BIOS fan configuration is low, OS load is high), after concatenation: =[98.0,1.68,8,0.42,0.85,…], the model can simultaneously show: I am very hot (98 degrees), and the fan configuration given by the neighbor's BIOS is insufficient (0.42).

[0058] Step S204: Based on the cross-layer fusion feature vector of each node in the cross-layer fault propagation graph, determine the root cause node of the cross-layer fault among multiple nodes in the cross-layer fault propagation graph.

[0059] In this step, the cross-layer fusion feature vectors of each node can be processed by combining the attention mechanism, the weight of the edge between each node and its cross-layer neighbor nodes can be calculated, the root cause score of each node can be calculated by the root cause scoring function, and the fault root cause node can be determined according to the ranking result of the root cause score.

[0060] This application embodiment determines the initial feature vector of a node based on the server component's operating status data, laying a data foundation for cross-layer feature analysis; it retrieves the features of cross-layer associated nodes by obtaining the initial feature vectors of cross-layer neighbor nodes; it obtains a cross-layer fused feature vector by fusing the initial feature vectors of the target node and cross-layer neighbor nodes, integrating the feature information of the node itself and cross-layer associated nodes, thus improving the comprehensiveness of feature representation; and it determines the root cause node of the fault based on the cross-layer fused feature vector, fully considering the impact of cross-layer association on the fault, and significantly improving the accuracy of cross-layer fault root cause localization.

[0061] In another embodiment of this application, step S204, based on the cross-layer fusion feature vector of each node in the cross-layer fault propagation graph, determines the root cause node of the cross-layer fault among multiple nodes in the cross-layer fault propagation graph, including: Step S301: For each node in the cross-layer fault propagation graph, determine the attention score of the node based on the cross-layer fusion feature vector of the node; In this embodiment of the application, the attention score refers to the score calculated based on the attention mechanism, which represents the importance of each feature dimension in the cross-layer fusion feature vector.

[0062] In this step, the attention mechanism of Graph Neural Network (GNN) is introduced. The cross-layer fused feature vector of each node is input into the attention calculation module. The feature vector is transformed through a fully connected layer and then normalized through a softmax function to obtain the attention score of the node. The higher the score, the higher the importance of the corresponding feature dimension to the fault analysis. Through the attention mechanism, the importance of each fault-related edge is quantified, allowing the model to focus on the key fault path.

[0063] The attention mechanism calculates an attention score (i.e., the initial weight of the edge) for each edge from node v to node u. The core logic is that the more relevant the neighbors, the higher the attention score. Define the attention coefficient calculation function: ( =Attention weight vector, =Activation function (to avoid gradient vanishing), the specific parameter definitions are shown in Table 5 below; Table 5

[0064] The larger the value, the higher the probability of fault propagation from component v to u. A prior bias of +0.2 is set for hardware layer edges (such as BMC→CPU), and additional weight is applied to error associations explicitly marked in the logs.

[0065] Scenario example: Calculating the CPU's attention in BIOS_Setting: =[0.4,98.0,1.68,…] (Fan speed at 40%, CPU temperature at 98℃). I learned that when the BIOS fan setting is low (0.4) and the CPU temperature is high (98), the correlation should be very high. =2.5 (high attention score), normalized using softmax to obtain the attention weight of each edge. (Range 0~1): (After normalization, the sum of the weights of all outgoing edges of node v is 1, ensuring that the weights are comparable.) The parameter definitions in the formula can be seen in Table 6 below. Table 6

[0066] The core function of Softmax is difference amplification: if the CPU has three neighbors (BIOS, DIMM, OS), its The values ​​are 2.0, 0.1, and 0.5 respectively, after Softmax: The result is [0.70, 0.13, 0.17]. The BIOS receives 70% of the attention, clearly indicating that the BIOS is the main influencing factor.

[0067] Scenario significance: In scenarios where CPU overheating is caused by BIOS fan curves: even if the CPU is connected to multiple neighbors, Softmax will highlight the weight of BIOS_Setting (e.g., =0.82), while suppressing other minor factors (such as =0.10 (irrelevant processes).

[0068] Since hardware failures are more likely to cause cross-layer problems, prior weights (e.g., base weight = 0.7) can be set for hardware layer edges (e.g., BMC → CPU); for error associations that are clearly marked in the logs (e.g., OS logs showing CPU temperature too high), the attention score of the corresponding edge can be increased (by adding a bias of 0.2).

[0069] Step S302: Determine the edge weights of the edges between the node and each cross-layer neighbor node based on the cross-layer fusion feature vector and the attention score; In this embodiment of the application, the edge weight refers to the numerical value that characterizes the strength of the fault association relationship between a node and its cross-layer neighbor nodes in the cross-layer fault propagation graph.

[0070] In this step, the cross-layer fusion feature vector of a node is multiplied by the corresponding attention score in a weighted manner to obtain a weighted cross-layer fusion feature vector. Then, using the graph neural network calculation model, the edge weights between the node and each cross-layer neighbor node are calculated based on the weighted feature vector. The edge weights range from 0 to 1. The larger the value, the stronger the fault correlation. The edge weights of 0 to 1 can directly reflect the fault correlation strength of the edge and are the core basis for subsequent root cause node screening. The higher the edge weight, the more likely its starting node is to be the root cause of the fault.

[0071] The node features aggregated by GraphSAGE are combined with the attention scores to obtain the edge weights of each edge: edge weights ; ( =Cosine similarity, which calculates the similarity of features between two nodes and further filters true related edges), where the parameter definitions can be seen in Table 7 below.

[0072] Table 7

[0073] Dual verification mechanism: (Attention) focuses on the structural importance learned by the model; Sim (Similarity): focuses on the geometric distance between actual feature vectors, and the product... The edge weight is high only when both are high, to prevent false alarms.

[0074] Scenario example: Weight calculation for the BIOS→CPU edge: =0.82 (the model assumes that the BIOS has a significant impact on the CPU). = 0.95 (The two characteristics are highly correlated: low fan configuration) (High temperature) =0.82×0.95=0.779 (high-weight edge, determined to be a core fault edge), compared with DIMM→CPU (irrelevant): =0.10, Sim=0.30, =0.03 (low weight, filtered).

[0075] Post-processing of weights: Filter edges with weights < 0.1 (considered as meaningless associations); mark edges with weights > 0.8 as core fault edges and prioritize their inclusion in root cause analysis.

[0076] Step S303: Determine the root cause node of the cross-level fault based on the multiple edge weights of each node in the cross-level fault propagation graph.

[0077] In this embodiment of the application, multiple edge weights refer to the set of outgoing edge weights and incoming edge weights for each node in the cross-layer fault propagation graph, which are the starting points and the ending points.

[0078] In this step, the weights of all outgoing edges and incoming edges of each node are counted, and the root cause score of each node is calculated using the root cause scoring function. The root cause scoring function is the sum of the outgoing edge weights of the node minus the sum of the incoming edge weights. The node with the highest root cause score is identified as the root cause node of the cross-level fault.

[0079] This application embodiment determines node attention scores based on cross-layer fusion feature vectors, which highlights key features in fault analysis and improves the effectiveness of feature utilization. By determining edge weights through cross-layer fusion feature vectors and attention scores, the calculation of edge weights is made to better fit the actual needs of fault analysis and accurately characterize the strength of fault correlation between nodes. The root cause node of the fault is determined based on the edge weight of each node. The accurate location of the root cause node is achieved through quantified weight calculation, which effectively reduces the false alarm rate and false negative rate of cross-layer fault root cause location.

[0080] In another embodiment of this application, step S303 determines the root cause node of the cross-level fault based on the multiple edge weights of each node in the cross-level fault propagation graph, including: Step S401: For each node in the cross-layer fault propagation graph, determine the sum of the edge weights of the outgoing edges of the node and the sum of the edge weights of the incoming edges of the node. In this embodiment, outgoing edges refer to edges in the cross-layer fault propagation graph that start from the target node and point to other nodes, representing the fault impact of the target node on other nodes; incoming edges refer to edges in the cross-layer fault propagation graph that start from other nodes and point to the target node, representing the fault impact of other nodes on the target node; the sum of the weights of outgoing edges refers to the sum of the weights of all outgoing edges of the target node; the sum of the weights of incoming edges refers to the sum of the weights of all incoming edges of the target node.

[0081] In this step, the edge connections of each node in the cross-layer fault propagation graph are traversed, outgoing edges and incoming edges are distinguished, the weight values ​​of each edge are retrieved, and the weights of all outgoing edges and incoming edges of each node are summed to obtain the sum of outgoing edge weights and the sum of incoming edge weights for each node.

[0082] Step S402: Determine the difference between the sum of the weights of the outgoing edges of the node and the sum of the weights of the incoming edges of the node. In this embodiment of the application, the difference refers to the value obtained by subtracting the sum of the weights of the incoming edges from the sum of the outgoing edge weights of each node, which is the core indicator for quantifying the possibility of a node being the root cause of a failure.

[0083] In this step, each node is used as a calculation unit. The sum of the outgoing edge weights is subtracted from the sum of the incoming edge weights to obtain the difference value corresponding to each node. A positive difference value indicates that the external impact of the fault on the node is greater than the internal impact of the fault. The larger the difference value, the higher the probability that the node is the root cause of the fault.

[0084] Step S403: The node corresponding to the largest difference is determined as the root cause node of the cross-level fault.

[0085] In this embodiment, the largest difference refers to the largest difference between the sum of the outgoing edge weights and the sum of the incoming edge weights of all nodes, and the node corresponding to it is the source of fault propagation.

[0086] In this step, the differences calculated from all nodes are sorted, the largest difference is selected, and the node corresponding to the largest difference is identified as the root cause node of the cross-level fault.

[0087] In practical applications, the root cause scoring function RootScore(v) = Σ_out w – Σ_in w can be used to find the k nodes with the largest RootScore. The root cause scoring function RootScore(v) is used to filter out root cause nodes of faults from the heterogeneous graph (such as abnormal BIOS fan curve configuration, CPU voltage deviation, etc.). The meaning of each parameter is explained in detail in conjunction with the patented technical solution in Table 8 below.

[0088] Table 8

[0089] The core logic behind the parameters: The core characteristic of the root cause node is that it actively affects other nodes, rather than being affected by other nodes. The root cause node will propagate the fault to downstream nodes through outgoing edges (such as abnormal BIOS configuration → CPU overheating → OS process blocking). Therefore, its outgoing edge weights will be significantly higher than its incoming edge weights.

[0090] Taking a scenario where a conservative BIOS fan curve leads to CPU overheating and application timeouts as an example: Key nodes and edge relationships in the heterogeneous graph: BIOS_Setting-FanCurve(v) → [Triggered, w=0.92] → CPU (overheating) → [Caused, w=0.85] → App_Request (response delay) Calculation of node v (BIOS_Setting-FanCurve): Σ_out w: v has only 1 outgoing edge (pointing to the CPU), and the sum of its weights is 0.92; Σ_in w: v has no incoming edges (no other nodes affect it, it is the starting point of the failure), and the sum of weights is 0; Root cause score: RootScore(v) = 0.92 - 0 = 0.92 (high score, identified as root cause node).

[0091] Let's take the affected CPU nodes as an example again: Edge relationships between CPU nodes: BIOS_Setting-FanCurve → [Trigger, w=0.92] → CPU(v) → [Cause, w=0.85] → App_Request calculate: Σ_out w: CPU outgoing edge weight = 0.85 (pointing to the application); Σ_in w: CPU's incoming edge weight = 0.92 (from BIOS); Root cause score: RootScore(v) = 0.85 - 0.92 = -0.07 (low score, excluded as root cause).

[0092] By calculating the RootScore(v) of all nodes and selecting the top k nodes with the highest scores, the root cause of the fault can be accurately located. This solves the one-sidedness of traditional single-layer analysis and realizes the automated root cause identification of cross-layer faults. It is the core of the GNN cross-layer root cause localization algorithm in this application.

[0093] This application embodiment achieves quantitative statistics on the impact and degree of influence of node failures by determining the sum of outgoing edge weights and the sum of incoming edge weights for each node; by calculating the difference between the sum of outgoing edge weights and the sum of incoming edge weights, the likelihood of a node being the root cause of a failure can be intuitively quantified; the node corresponding to the largest difference is determined as the root cause node, and the accurate and rapid location of the root cause node is achieved through explicit numerical sorting, avoiding the one-sidedness of manual layer-by-layer investigation and significantly shortening the root cause location time for cross-level failures.

[0094] In another embodiment of this application, step S104 determines a fault repair action based on the fault root cause node and executes the fault repair action, including: Step S501: Obtain the first fault indicator parameter when the server fails; In this embodiment of the application, the first fault indicator parameter refers to the core indicator parameter that characterizes the fault features in the operating status data of each level of server components when the server fails.

[0095] In this step, the system extracts directly related indicators from the server component operation status data collected from various levels when the server fails. These indicators include hardware temperature, voltage, software response latency, error rate, and configuration parameters. The extracted parameters are then summarized and standardized to obtain the first fault indicator parameters when the server fails.

[0096] Step S502: Obtain the hierarchical features of the hierarchical structure where the root cause node of the fault is located; In this embodiment of the application, the hierarchical feature refers to the server hierarchy to which the root cause node of the fault belongs.

[0097] In this step, the server level to which the root cause node belongs is determined based on its hierarchical identifier, which serves as the hierarchical feature.

[0098] Step S503: Determine the fault repair action based on the first fault indicator parameter, the hierarchical feature, and the fault root cause node, and execute the fault repair action.

[0099] In this embodiment of the application, the fault repair action refers to a targeted operation that can resolve the fault corresponding to the root cause node and eliminate cross-level faults of the server, and is matched with the fault characteristics and level characteristics.

[0100] In this step, the first fault indicator parameters, hierarchical features, and information on the root cause node of the fault are used as matching conditions. The system searches in a preset strategy library and matches at least one repair operation that matches all the conditions. This operation is then identified as the fault repair action for the current cross-level fault.

[0101] This application embodiment obtains the first fault indicator parameter when the server fails, obtains the hierarchical characteristics of the level where the root cause node of the fault is located, and determines the fault repair action based on the first fault indicator parameter, hierarchical characteristics and the root cause node of the fault. This achieves accurate matching between the repair action and the fault and level, improves the pertinence and effectiveness of the fault repair action, and avoids secondary faults caused by blind operation.

[0102] In another embodiment of this application, step S503 determines a fault repair action based on the first fault index parameter, the hierarchical features, and the fault root cause node, and executes the fault repair action, including: Step S601: Based on the first fault index parameters, the hierarchical features, and the fault root cause node, match candidate repair actions in the strategy library and obtain the current first weight of each candidate repair action; In this embodiment of the application, the strategy library refers to a database that pre-stores various fault matching conditions and corresponding repair actions, including information such as triggering conditions, repair actions, rollback schemes, and risk scores; the candidate repair action refers to all repair actions in the strategy library that match the matching conditions, and is an alternative to the fault repair action; the first weight refers to the current priority weight of each candidate repair action in the strategy library, which is obtained by training a reinforcement learning model, and the higher the weight, the higher the execution priority.

[0103] In this step, the first fault index parameters, hierarchical features, and information on the root cause nodes of the fault are combined into matching conditions. The system is then searched in the strategy library to select all repair actions that meet the matching conditions as candidate repair actions. At the same time, the system retrieves the current first weight that has been set for each candidate repair action in the strategy library.

[0104] In practical applications, root cause nodes are mapped to executable actions (such as BIOS voltage offset +20 mV, restarting service A). The mapping is essentially a multi-dimensional correspondence between root cause nodes and executable actions. The core is based on fault type + node attribute matching. A single node can correspond to multiple actions (dynamically selected according to priority / scenario). The mapping is the key step in transforming the root cause nodes (such as CPU, BIOS_Setting, OS_Process, etc.) output by the GNN algorithm into system-executable debugging instructions (such as adjusting parameters, restarting services). In essence, it is the correspondence rule between the root cause entity of the fault and the solution action.

[0105] The core components of the mapping relationship (3-layer correspondence logic): The mapping is not a direct node→action, but rather a three-dimensional matching of node type + fault characteristics + hierarchical attributes to ensure the accuracy of the action. The specific logic is shown in Table 9 below: Table 9

[0106] Key conclusion: Each node does not have only one corresponding action. The automated debugging execution engine combines the strategy library (YAML definition {condition, action, rollback_plan, risk_score}) to match multiple actions for the same root cause node, executing them according to priority or in combination, as shown in Table 10 below: Table 10

[0107] Constraints on action mapping (ensuring safety and effectiveness).

[0108] Risk scoring control: Each action in the strategy library is bound to a risk_score (0~1). High-risk actions (such as hardware parameter modification) must have a root cause confidence score ≥ 0.9 before they are executed. Combined action rules: Multiple actions on the same node must be conflict-free (e.g., CPU upclocking and CPU downclocking cannot be performed simultaneously), and the engine will automatically verify them; Rollback mechanism: Each action corresponds to a rollback_plan (e.g., if the voltage is modified but not restored, it will be restored to the original voltage) to avoid secondary failures caused by action execution failure.

[0109] The core value of mapping: This mapping mechanism solves the problems of traditional debugging actions being singular and lacking specificity. Through multi-dimensional matching and multiple action alternatives, it ensures that the actions corresponding to the root cause nodes are accurate, safe, and efficient. It is a key link in achieving automated repair and also the training basis for reinforcement learning optimization strategy library.

[0110] The automated debugging execution engine maintains a policy library (YAML description). Each policy contains: {condition, action, rollback_plan, risk_score}, corresponding to a complete closed loop of triggering condition, execution action, rollback plan, and risk score. This is used to achieve a precise mapping from root cause nodes to automated repair. The meanings of each field are explained below: The core meaning of "condition" (trigger condition) is to define when the strategy is executed, that is, the prerequisite for the strategy to take effect. It needs to be precisely matched with the attributes and fault characteristics of the root cause node. The logic is based on the combination of the core fields (layer_id, node_id, error_code, metric_type, value, etc.) of the Unified Debug Information Model (UDIM) and supports multi-condition logical operations (AND / OR / NOT).

[0111] For example, this policy is triggered when the root cause node is the BIOS layer fan curve configuration (layer_id=3 corresponds to the BIOS layer, node_id=BIOS_Setting-FanCurve), the fault code is BIOS layer fan curve abnormal (error_code=3-001), and the current configuration value is quiet mode (value=quiet_mode).

[0112] The core meaning of "action" (execution action): After a policy is triggered, the system automatically performs debugging / repair operations. These operations must match the level and type of the root cause node and be executed remotely via out-of-band management channels (IPMI / Redfish) or system interfaces. Action types are categorized as shown in Table 11 below: Table 11

[0113] For example, a PATCH request can be sent via the Redfish protocol to change the BIOS fan curve mode to performance mode and adjust the duty cycle to 80% (directly solving the root cause of CPU overheating caused by a conservative fan curve).

[0114] The core meaning of a rollback plan: To avoid unrecoverable failures or secondary problems after an action is executed, a pre-defined undo operation ensures the system can return to its state before the action was executed, which is crucial for ensuring business security. Design principles: Completely reversible with the action, and the execution conditions for the rollback operation are clearly defined (the metric within the observation window has not recovered).

[0115] For example, if the CPU temperature does not drop to a safe threshold (e.g., ≤80℃) within a 30-60 second observation window after performance mode adjustment, the system will automatically trigger a rollback, restoring the fan curve to its original configuration (quiet mode, duty cycle 40%), to avoid excessive heat dissipation leading to energy waste or hardware damage.

[0116] The core meaning of risk score is to quantify the potential risks to business and hardware after the strategy is executed. The value ranges from 0 to 1 (0 for no risk, 1 for extremely high risk), and it is used to prioritize strategy selection. Scoring criteria include: Business impact: Whether it may cause momentary lag or service interruption (e.g., restarting core processes carries a higher risk than adjusting fan speed); Hardware impact: Whether it involves modifying critical hardware parameters (e.g., voltage adjustment carries a higher risk than fan speed adjustment); Irreversibility: Whether the action is difficult to roll back (e.g., firmware upgrade carries a higher risk than parameter modification).

[0117] For example, adjusting the BIOS fan curve is a low-risk action (it only modifies the cooling configuration, does not interrupt business operations, does not change the core hardware parameters, and rollback is simple), with a risk score of 0.2; while actions such as CPU downclocking and restarting database services may have a risk score of 0.6 to 0.8, and will only be executed if the root cause confidence level is ≥0.9.

[0118] Step S602: The candidate repair action with the largest first weight is determined as the fault repair action; In this embodiment of the application, the candidate repair action with the highest first weight refers to the action with the highest first weight value among the candidate repair actions, which is the repair scheme with the highest priority in the current strategy library.

[0119] In this step, the first weight of all candidate repair actions is numerically sorted, and the candidate repair action with the largest first weight value is selected and determined as the final fault repair action for the current server cross-level fault.

[0120] Step S603: Perform the fault repair action through the out-of-band channel.

[0121] In this embodiment of the application, the candidate repair action with the highest first weight refers to the action with the highest first weight value among the candidate repair actions, which is the repair scheme with the highest priority in the current strategy library.

[0122] In this step, the first weight of all candidate repair actions is numerically sorted, and the candidate repair action with the largest first weight value is selected and determined as the final fault repair action for the current server cross-level fault.

[0123] In practical applications, the following actions can be issued through out-of-band management channels: BIOS settings: Redfish / bios / Settings PATCH; BMC sensor threshold: IPMI Set SensorThreshold; OS service: IPMI Chassis Control restart; Application example: Redfish / Systems / {id} / Actions / ComputerSystem.Reset. For detailed explanations, please see Table 12 below.

[0124] Table 12

[0125] The core features of these actions are precise matching of levels, out-of-band execution, and no business interruption: 1. Strong hierarchical binding of actions and root cause nodes: BIOS layer issues are addressed by modifying BIOS configurations using Redfish, BMC layer issues by adjusting sensors using IPMI, and OS / application layer issues by restarting corresponding services / resetting instances, ensuring targeted repairs; 2. All operations are performed through out-of-band management channels: They do not rely on the business network, do not require plugging or unplugging hardware probes or shutting down the system, and solve the pain point of traditional debugging interrupting services; 3. Each action corresponds to the action field in the strategy library. After execution, a 30-60 second observation window will be entered. If the metric does not recover, the rollback plan will be triggered to ensure system security.

[0126] After the action is executed, the system enters a 30-60 second observation window. If the indicators recover, the system marks the repair as successful; otherwise, it automatically rolls back.

[0127] In this embodiment, by matching candidate repair actions in the policy library and obtaining the first weight, multiple repair schemes are selected, improving the flexibility of fault repair; the candidate repair action with the largest weight is determined as the fault repair action, and the optimal selection of the repair scheme is achieved based on the weight of reinforcement learning, improving the efficiency of fault repair; by executing the fault repair action through the out-of-band channel, the server can be debugged without downtime, ensuring the continuous operation of services and eliminating the business interruption problem of traditional repair methods.

[0128] In another embodiment of this application, step S503, which determines a fault repair action based on the first fault index parameter, the hierarchical features, and the fault root cause node, and executes the fault repair action, further includes: Step S701: After performing the fault repair action through the out-of-band channel, obtain the second fault index parameter at the end of the observation window and the fault location time used to determine the root cause node of the fault. In this embodiment of the application, the observation window refers to the preset effect verification time window after the fault repair action is executed, which can be 30-60 seconds for example; the second fault indicator parameter refers to the fault-related core indicator parameters of each level of server components when the observation window ends; the fault location time refers to the time consumed from the occurrence of the server fault trigger event to the determination of the root cause node of the fault.

[0129] In this step, after the fault repair action is executed, an observation window of 30-60 seconds is started. When the observation window ends, the operating status data of the server components are re-collected by the acquisition probes at each level, and the fault-related core indicator parameters are extracted as the second fault indicator parameters. At the same time, the time interval from the occurrence of the fault trigger event to the determination of the root cause node of the fault is counted as the fault location time.

[0130] Step S702: If the second fault indicator parameter is abnormal or a secondary fault occurs, obtain the current false alarm rate; In this embodiment of the application, the second fault indicator parameter abnormality means that the second fault indicator parameter has not recovered to the threshold range for normal server operation when the observation window ends; secondary fault refers to a new server fault caused after the fault repair action is performed; false alarm rate refers to an indicator that characterizes the effectiveness of the fault repair action. When the repair fails, the false alarm rate is counted as 1. When it is partially effective, it is calculated according to the degree of indicator recovery. When the repair is successful, the false alarm rate is counted as 0.

[0131] In this step, the second fault indicator parameter is compared with the indicator threshold for normal server operation to determine whether there is an anomaly. At the same time, it is detected whether the server has generated a new fault. If there is an abnormal indicator or a secondary fault, the current false alarm rate is calculated based on the effect of fault repair. When the complete repair fails, the false alarm rate is 1. When the indicator is partially recovered, the false alarm rate is calculated according to a preset formula.

[0132] Step S703: Determine the reward value based on the false alarm rate and the fault location time; In this embodiment, the reward value refers to a value calculated based on the false alarm rate and fault location time. It is the basis for training the reinforcement learning model. The higher the reward value, the better the corresponding fault repair action.

[0133] In this step, a preset reward function is used to calculate the reward value. The reward function is R=α×(1-false alarm rate)+β×(baseline fault location time-actual fault location time) / baseline fault location time, where α and β are weighting coefficients, and the baseline fault location time is the average location time of the traditional fault diagnosis method. The false alarm rate and the actual fault location time are substituted into the formula to calculate the reward value.

[0134] Step S704: Input the reinforcement learning model with the first fault index parameter, the hierarchical feature, the fault root cause node, the fault repair action, the reward value and the second fault index parameter, so that the reinforcement learning model outputs the action value of the fault repair action; In this embodiment, the reinforcement learning model refers to a model constructed using the proximal policy optimization (PPO) algorithm, which is used to optimize the weights of fault repair actions; the action value refers to the numerical value output by the reinforcement learning model that characterizes the effect of the fault repair action in the current fault scenario, and is the basis for updating the weights of the repair actions.

[0135] In this step, the first fault indicator parameter, hierarchical features, fault root cause node, fault repair action, reward value and the second fault indicator parameter are used as input data and fed into the pre-built Proximal Policy Optimization (PPO) reinforcement learning model. The model learns and analyzes the input data and outputs the action value corresponding to the fault repair action.

[0136] Step S705: Update the first weight corresponding to the fault repair action using the second weight corresponding to the action value.

[0137] In this embodiment of the application, the second weight refers to the new weight of the fault repair action determined based on the action value output by the reinforcement learning model; the update refers to replacing the original first weight of the fault repair action with the second weight obtained based on the action value, thereby realizing the dynamic optimization of the repair action weight in the policy library.

[0138] In this step, based on the action value output by the reinforcement learning model, the second weight corresponding to the fault repair action is calculated using a preset mapping formula, which is w1=w0×(1+γ×(R-R_avg)), where w0 is the original first weight, γ is the learning rate coefficient, R is the current reward value, and R_avg is the historical average reward value. The first weight of the fault repair action in the policy library is then updated to the calculated second weight.

[0139] In practical applications, the core logic of using reinforcement learning (PPO) to update policy weights online is as follows: with the optimization goals of shortening the time to fault location (MTTD) and reducing the false alarm rate, the priority weights of each policy in the policy library are continuously adjusted through an interactive-feedback-update closed loop. The specific steps are as follows: Step 1: Initialize core components and parameters Policy library weight initialization: Assign an initial weight w0 (default is uniformly 0.5, range 0~1) to all policies in the policy library (each policy contains condition-action-rollback_plan-risk_score). The higher the weight, the greater the probability that the policy will be matched and executed first. PPO agent initialization: Load the pre-trained PPO model (pre-training data comes from the policy execution results of historical failure cases), and set the core parameters of the model: State space S: contains root cause node attributes (layer_id / node_id / error_code), fault metrics (metric_type / value), and current system load; Action Space A: All executable actions in the policy library (such as adjusting BIOS parameters, restarting services, etc.); Reward function R: Core correlation between MTTD and false alarm rate, the formula is R=α×(1-false alarm rate)+β×(baseline MTTD-actual MTTD) / baseline MTTD (α / β are weighting coefficients, both default to 0.5 to ensure balanced optimization of the two indicators). Observation window configuration: Set the observation period after the action is executed (30~60 seconds, default value) to collect data on the effect of strategy execution.

[0140] Step 2: Real-time interaction and data collection (online phase) Fault Triggering and Policy Matching: When the system detects a fault (such as application timeout or abnormal CPU temperature), after the GNN algorithm outputs the root cause node, the automated debugging engine matches all policies that meet the conditions in the condition matching policy library, sorts them by their current weight w, and selects the policy with the highest weight to execute. Action Execution and Status Recording: Execute the selected policy's action via IPMI / Redfish out-of-band channel, and record the system state before and after execution. Before execution: Fault indicator values ​​(e.g., CPU temperature 98℃, application latency 3.2s), current time T1; After execution: the index values ​​at the end of the observation window (e.g., CPU temperature 72℃, application latency 0.5s), and the fault closure time T2 (actual MTTD = T2 - T1). False alarm / repair result determination: Repair successful: If the indicators within the observation window have recovered to the safe threshold (e.g., CPU temperature ≤ 80℃), the strategy is deemed effective and the false alarm rate is 0. Repair failure: If the indicator is not restored or a secondary fault occurs (such as CPU frequency reduction after voltage adjustment), it is determined to be an invalid strategy, and the false alarm rate is counted as 1. Partially effective: The indicator improves but does not reach the threshold (e.g., the CPU temperature drops to 85℃). The false alarm rate is calculated as (current indicator - safety threshold) / (indicator at fault - safety threshold) (e.g., (85-80) / (98-80) ≈ 0.28).

[0141] Step 3: Reward Calculation and Strategy Weight Update Reward value R calculation: Based on the actual MTTD and false alarm rate recorded in step 2, substitute them into the reward function R to calculate the reward value of the current policy: Example: Baseline MTTD = 60 minutes (average of traditional method), actual MTTD = 4 minutes, false alarm rate = 0.05, then R = 0.5 × (1 - 0.05) + 0.5 × (60 - 4) / 60 ≈ 0.475 + 0.467 ≈ 0.942 (high reward indicates excellent strategy performance); If the actual MTTD is 30 minutes and the false alarm rate is 0.3, then R = 0.5 × (1 - 0.3) + 0.5 × (60 - 30) / 60 = 0.35 + 0.25 = 0.6 (medium reward). PPO model parameter update: The trajectory data from pre-execution state S1 → action A → reward R → post-execution state S2 is input into the PPO agent. The model parameters are updated by pruning the objective function (the core mechanism of PPO) to avoid system instability caused by excessive weight updates. Objective function: L(θ) = E[min(r(θ)×Â, clip(r(θ), 1-ε, 1+ε)×Â)] (r(θ) is the probability ratio of the new and old strategies, ε=0.2 is the clipping coefficient, and  is the dominance function, which quantifies the relative value of action A). Model training: Mini-batch gradient descent (batch size = 32, learning rate = 1e-4) is used to update the policy network parameters of the PPO model; Policy weight mapping: The action value (i.e. the expected reward of action A) output by the updated PPO model is mapped to the new weight w1 of the corresponding policy in the policy library: w1 = w0 × (1 + γ × (R - R_avg)) (γ is the learning rate coefficient, which is 0.1 by default, and R_avg is the historical average reward to avoid the influence of a single case being too large). The new weight is still limited to the range of 0 to 1 (if it is greater than 1, it is taken as 1, and if it is less than 0, it is taken as 0).

[0142] Step 4: Iterative Optimization and Convergence Guarantee Continuous iteration: Repeat steps 2-3 after each fault loop is closed, and update the strategy weights in real time. The better the strategy performs, the higher its weight will be, and the greater the probability that it will be selected first in the future. Weight smoothing mechanism: To avoid weight oscillation caused by high-frequency failures, a weight decay coefficient δ (default 0.99) is set. Every hour, w=w×δ is executed on all strategy weights, and then normalized again (ensuring that the sum of all strategy weights is 1), balancing the priority of new strategies and historically effective strategies. Convergence determination: When the average reward R_avg fluctuation of 100 consecutive failure cases is ≤5%, and the MTTD is stable at 3~5 minutes and the false alarm rate is stable at 5%~15% (the target range of the patent), the model is determined to have converged, the weight update frequency is reduced (once every 10 cases), and the system computing power consumption is reduced.

[0143] Through the above steps, the PPO algorithm achieves a strong binding between policy weights and actual debugging results: shortening MTTD: prioritizing policies with faster repair speeds (such as adjusting fan curves being faster than restarting services), continuously reducing the total time spent on fault location and repair; reducing false alarm rate: eliminating invalid / secondary fault policies (such as blindly restarting services), improving the accuracy of policy matching; adapting to scenarios: when faced with new fault types (such as BIOS configuration anomalies that have not been pre-trained), quickly adjusting weights through online updates, without requiring manual intervention in the policy library.

[0144] This application embodiment achieves quantitative evaluation of fault repair effect and location efficiency by acquiring the second fault index parameter and fault location time of the observation window; by acquiring the false alarm rate and calculating the reward value, the repair effect and location efficiency are transformed into training indicators for the reinforcement learning model; by inputting multi-dimensional fault information into the action value of the reinforcement learning model, the effect of repair actions is accurately quantified; by using the action value to update the weight of the repair actions, the weight of the repair actions in the policy library is dynamically optimized, enabling the policy library to adapt to different fault scenarios, continuously improve the efficiency and accuracy of fault repair, and reduce the false alarm rate.

[0145] For ease of understanding, this application also provides an embodiment in practical application, as follows: T0 fault trigger If the APP layer monitoring SDK detects that the service response time exceeds the SLA, it immediately sends a UDIM message to the DE: {ts:T0, layer:APP, metric:latency, value:3.2 s, error_code:SLO_BREACH} T0+20 ms DE initiates the cross-layer forensics process, first reading the temperature / voltage from the BMC via the out-of-band channel (IPMI Get Sensor Reading): DE → BMC: IPMI command: Read sensor (CPU_TEMP, VCCIN, DIMM_TEMP) BMC → DE: Returns temperature / voltage data {CPU_TEMP=98 °C, VCCIN=1.68 V, DIMM_TEMP=85 °C} T0+40 ms DE determined that the CPU temperature was too high and continued to read the current fan configuration curve from the BIOS via Redfish: DE → BIOS : Redfish GET / Systems / 1 / Bios / FanCurve BIOS → DE: Returns fan curve {curve_id=quiet, duty_cycle=40%} DE determined "fan curve is too conservative" as the root cause node based on the GNN root cause algorithm, with a confidence level of 0.92.

[0146] T0+60 ms DE generates a repair strategy: Change the fan curve from quiet to performance.

[0147] DE → BMC: IPMI command: Adjust fan speed (duty_cycle=80%) / / Effects immediately BMC → DE: IPMI completion response {status:ok} T0+80 ms Meanwhile, to prevent CPU overheating from causing OutOfMemory (OOM) errors, DE decided to trigger OS memory reclamation simultaneously: DE → OS : Redfish POST / Systems / 1 / Actions / Oem / MemoryReclaim OS → DE : Returns execution result {reclaimed:2 GiB, status:ok} T0+100 ms During OS memory reclamation, the CPU usage of the kernel kswapd spikes momentarily, and business operations still experience occasional timeouts.

[0148] OS → DE: Log hook: Kernel prints "kswapd: high latency 15 ms" After assessing the risks, DE decided to restart the affected business services to quickly restore the SLA: DE → APP: Log hook: Restart service "order-service" APP → DE: Returns exit code 0, then systemd restarts the service.

[0149] T0+5 s (End of observation window) DE pulls the sensor again: DE → BMC: IPMI command: Read sensor (CPU_TEMP) BMC → DE: Returns temperature / voltage data {CPU_TEMP=72 °C} DE determines that the temperature has dropped, the strategy takes effect, records "Repair successful", and updates the strategy weights.

[0150] Additional notes: All commands are transmitted out-of-band over the management network, ensuring uninterrupted service network operation.

[0151] Each step includes a UDIM message write-back for subsequent model training.

[0152] If the temperature does not drop within the observation window, DE will automatically roll back the fan curve and trigger the secondary strategy (CPU downclocking).

[0153] In another embodiment of this application, a server fault debugging device is also provided, such as... Figure 2 As shown, it includes: The data acquisition module 11 is used to acquire the operating status data of each server component in the multiple hierarchical structure of the server in response to the triggering event of server failure; Graph construction module 12 is used to construct a cross-layer fault propagation graph based on the running status data of multiple server components. The cross-layer fault propagation graph includes multiple nodes and edges. The nodes correspond one-to-one with the server components, and the edges are used to connect nodes that have cross-layer fault association relationships. The root cause determination module 13 is used to determine the root cause node of the cross-level fault in multiple nodes of the cross-level fault propagation graph based on the cross-level fault propagation graph and the operating status data of multiple server components. The action determination module 14 is used to determine the fault repair action based on the fault root cause node and execute the fault repair action.

[0154] In another embodiment of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When the processor executes a program stored in memory, it implements the server fault debugging method described in any of the foregoing method embodiments.

[0155] The electronic device provided in this invention enables the processor to respond to fault triggering events by executing programs stored in the memory, collect operational status data of the five-layer server architecture, and achieve comprehensive acquisition of heterogeneous data across the entire stack, thus solving the problem of layered silos in traditional fault diagnosis. By constructing a cross-layer fault propagation graph, server components at different levels are linked through fault correlations. Based on the cross-layer fault propagation graph and the operational status data of multiple server components, the root cause node of the fault is determined across layers, realizing cross-layer correlation analysis of complex faults and locating the source of the fault. This eliminates the need for manual layer-by-layer tracing and location, significantly improving the accuracy of root cause location. By matching and executing repair actions through a policy library, automated fault repair is achieved, effectively shortening the fault location and repair time and improving the efficiency and accuracy of server fault debugging.

[0156] The communication bus 1140 mentioned in the above-mentioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0157] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0158] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0159] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0160] In another embodiment of this application, a computer-readable storage medium is also provided, on which a server fault debugging method program is stored, wherein when the server fault debugging method program is executed by a processor, the server fault debugging method program implements the steps of the server fault debugging method described in any of the foregoing method embodiments.

[0161] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0162] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A server fault debugging method, characterized in that, include: In response to a server failure trigger event, the system acquires the operational status data of each server component in the multiple hierarchical structure of the server. A cross-layer fault propagation graph is constructed based on the operating status data of multiple server components. The cross-layer fault propagation graph includes multiple nodes and edges. Each node corresponds to a server component, and the edges are used to connect nodes that have cross-layer fault association relationships. Based on the cross-layer fault propagation diagram and the operating status data of multiple server components, the root cause node of the cross-layer fault is determined from multiple nodes in the cross-layer fault propagation diagram. The fault repair action is determined based on the root cause node of the fault, and the fault repair action is executed.

2. The server fault debugging method according to claim 1, characterized in that, Based on the cross-layer fault propagation graph and the operational status data of multiple server components, the root cause node of the cross-layer fault is determined from multiple nodes in the cross-layer fault propagation graph, including: For each node in the cross-layer fault propagation graph, an initial feature vector is determined based on the running status data of the server component corresponding to the node; Obtain the initial feature vectors of the cross-layer neighbor nodes of the node; The initial feature vector of the node is used to determine the cross-layer fused feature vector of the node by combining the initial feature vectors of the cross-layer neighbor nodes. Based on the cross-layer fusion feature vector of each node in the cross-layer fault propagation graph, the root cause node of the cross-layer fault is determined among multiple nodes in the cross-layer fault propagation graph.

3. The server fault debugging method according to claim 2, characterized in that, Based on the cross-layer fusion feature vector of each node in the cross-layer fault propagation graph, the root cause node of the cross-layer fault is determined among multiple nodes in the cross-layer fault propagation graph, including: For each node in the cross-layer fault propagation graph, the attention score of the node is determined based on the cross-layer fusion feature vector of the node; The edge weights of the edges between the node and each of its cross-layer neighbor nodes are determined based on the cross-layer fusion feature vector and the attention score. Based on the multiple edge weights of each node in the cross-level fault propagation graph, the root cause node of the cross-level fault is determined.

4. The server fault debugging method according to claim 3, characterized in that, Based on the multiple edge weights of each node in the cross-layer fault propagation graph, the root cause node of the cross-layer fault is determined, including: For each node in the cross-layer fault propagation graph, determine the sum of the edge weights of the outgoing edges of the node and the sum of the edge weights of the incoming edges of the node; Determine the difference between the sum of the weights of the outgoing edges of the node and the sum of the weights of the incoming edges of the node; The node corresponding to the largest difference is identified as the root cause node of the cross-level fault.

5. The server fault debugging method according to claim 1, characterized in that, Determine fault repair actions based on the root cause node of the fault, and execute the fault repair actions, including: Obtain the first fault indicator parameter when the server fails; Obtain the hierarchical features of the hierarchical structure where the root cause node of the fault is located; The fault repair action is determined based on the first fault indicator parameter, the hierarchical characteristics, and the fault root cause node, and then the fault repair action is executed.

6. The server fault debugging method according to claim 5, characterized in that, Based on the first fault indicator parameters, the hierarchical features, and the root cause node of the fault, a fault repair action is determined and executed, including: Based on the first fault indicator parameters, the hierarchical features, and the fault root cause node, candidate repair actions are matched in the strategy library, and the current first weight of each candidate repair action is obtained. The candidate repair action with the highest first weight is determined as the fault repair action; The fault repair action is performed via an out-of-band channel.

7. The server fault debugging method according to claim 5, characterized in that, Determining and executing fault repair actions based on the first fault indicator parameters, the hierarchical features, and the fault root cause nodes, further includes: After performing the fault repair action through the out-of-band channel, the second fault index parameter at the end of the observation window and the fault location time used to determine the root cause node of the fault are obtained. If the second fault indicator parameter is abnormal or a secondary fault occurs, obtain the current false alarm rate; The reward value is determined based on the false alarm rate and the fault location time. The reinforcement learning model is input with the first fault indicator parameter, the hierarchical feature, the fault root cause node, the fault repair action, the reward value, and the second fault indicator parameter, so that the reinforcement learning model outputs the action value of the fault repair action; The first weight corresponding to the fault repair action is updated using the second weight corresponding to the value of the action.

8. A server fault debugging device, characterized in that, include: The data acquisition module is used to acquire the operating status data of each server component in the multiple hierarchical structure of the server in response to the server failure triggering event; The graph construction module is used to construct a cross-layer fault propagation graph based on the running status data of multiple server components. The cross-layer fault propagation graph includes multiple nodes and edges. The nodes correspond one-to-one with the server components, and the edges are used to connect nodes that have cross-layer fault association relationships. The root cause determination module is used to determine the root cause node of the cross-level fault in multiple nodes of the cross-level fault propagation graph based on the cross-level fault propagation graph and the operating status data of multiple server components. The action determination module is used to determine the fault repair action based on the fault root cause node and execute the fault repair action.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the server fault debugging method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a server fault debugging method program, which, when executed by a processor, implements the steps of the server fault debugging method according to any one of claims 1-7.

Images