Fault diagnosis method, system and device, electronic equipment and storage medium

Abstract

The invention discloses a fault diagnosis method, system and device, electronic equipment and a storage medium, and relates to the technical field of computers, and the method comprises the steps: obtaining the operation state data of a target node; based on the operation state data, training a machine learning model corresponding to the target node, and uploading a training result to a federal aggregation server; receiving model updating parameters fed back by the federal aggregation server, and updating the machine learning model based on the model updating parameters; predicting a fault probability of a target node by using the updated machine learning model, and when the fault probability is greater than a preset threshold, broadcasting a fault message to an adjacent node; the fault message is used for triggering the adjacent node to determine the fault type of the target node by using a machine learning model corresponding to the adjacent node and report fault type information to the federal aggregation server. Through the technical scheme provided by the invention, the fault diagnosis efficiency can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault diagnosis method, system, device, electronic device, and storage medium. Background Art

[0002] The Baseboard Management Controller (BMC) is a key component on the server motherboard that can be used for hardware status monitoring, remote management, fault diagnosis, and recovery of server nodes.

[0003] Traditional BMCs typically focus on hardware monitoring of single server nodes and are unable to cope with global failures in cluster scenarios (such as network topology anomalies and concurrent multi-node failures), resulting in low fault diagnosis efficiency. Summary of the Invention

[0004] The present application provides a fault diagnosis method, system, device, electronic device and storage medium, which can at least solve the technical problems in related technologies such as difficulty for BMC to identify global faults in cluster scenarios and low fault diagnosis efficiency.

[0005] The present application provides a fault diagnosis method, which includes:

[0006] Get the running status data of the target node;

[0007] Based on the above operating status data, the machine learning model corresponding to the target node is trained and the training results are uploaded to the federated aggregation server;

[0008] Receive model update parameters fed back by the federated aggregation server, and update the machine learning model based on the model update parameters;

[0009] The updated machine learning model is used to predict the failure probability of the target node. When the failure probability is greater than the preset threshold, a fault message is broadcast to the adjacent nodes. The fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the failure type of the target node and report the failure type information to the federal aggregation server.

[0010] The present application also provides a fault diagnosis system, which includes a federated aggregation server and a plurality of nodes, wherein the nodes are provided with a baseboard management controller;

[0011] The baseboard management controller is used to: obtain the operating status data of the target node; based on the operating status data, train the machine learning model corresponding to the target node and upload the training results to the federated aggregation server; receive the model update parameters fed back by the federated aggregation server and update the machine learning model based on the model update parameters; use the updated machine learning model to predict the failure probability of the target node and broadcast a fault message to the adjacent nodes when the failure probability is greater than a preset threshold; the fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the fault type of the target node and report the fault type information to the federated aggregation server;

[0012] The federated aggregation server is used to: determine model update parameters based on the training results received from multiple nodes, and feed back the model update parameters to the multiple nodes; and determine the root cause of the fault based on the fault type information of the target node uploaded by at least one node.

[0013] The present application also provides a fault diagnosis device, comprising:

[0014] The acquisition module is used to obtain the running status data of the target node;

[0015] A training module is used to train the machine learning model corresponding to the target node based on the above-mentioned operating status data and upload the training results to the federated aggregation server;

[0016] An update module, configured to receive model update parameters fed back by the federated aggregation server and update the machine learning model based on the model update parameters;

[0017] The prediction module is used to use the updated machine learning model to predict the failure probability of the target node. When the failure probability is greater than the preset threshold, a fault message is broadcast to the adjacent nodes. The fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the failure type of the target node and report the failure type information to the federal aggregation server.

[0018] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned fault diagnosis methods when executing the computer program.

[0019] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned fault diagnosis methods are implemented.

[0020] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned fault diagnosis methods when executed by a processor.

[0021] The fault diagnosis method, system, device, electronic device, and storage medium provided in this application enable models at each node to share global knowledge through federated learning, continuously optimizing their own model performance without transmitting raw data, thus balancing data privacy and improving model performance. Furthermore, collaborative fault analysis across multiple nodes provides a more comprehensive understanding of the fault situation, avoiding misjudgments by individual nodes and improving fault diagnosis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 A schematic diagram of the architecture of a fault diagnosis system provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a flow chart of a fault diagnosis method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a sub-flow diagram of a fault diagnosis method provided in an embodiment of the present application;

[0026] Figure 4 A schematic structural diagram of a fault diagnosis device provided in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0030] The following is an explanation of some of the data involved in the examples of this application:

[0031] Baseboard Management Controller (BMC): A key component on the server motherboard, it is an independent system that operates independently of the server's operating system, hard drive, or power supply. The BMC is responsible for server hardware status monitoring, remote management, fault diagnosis, and recovery. Through the BMC, administrators can remotely monitor server hardware status, including temperature, voltage, and fan speed, remotely power on / off, and restart the server, and perform diagnostics and recovery when a server fails.

[0032] Management Component Transport Protocol (MCTP): A protocol used for communication between management controllers in computer systems. MCTP enables communication between management controllers (such as BMCs and power controllers).

[0033] Message Queuing Telemetry Transport (MQTT): MQTT operates in low-bandwidth and unstable network environments, making it suitable for resource-constrained environments such as the Internet of Things and mobile devices. Its message header is only 2 bytes, minimizing protocol exchanges and reducing overhead.

[0034] MQTT uses a publish / subscribe messaging model, providing one-to-many message publishing. Publishers and subscribers communicate through a messaging middleware, decoupling the two. Publishers don't need to know the details of subscribers, and subscribers don't need to know the publisher's information.

[0035] In some technical solutions, BMC usually focuses on hardware monitoring of a single node, for example, server fault detection is achieved through instruction interaction between the BMC and the central processing unit (CPU).

[0036] However, single-node monitoring solutions struggle to address global failures in cluster scenarios (such as network topology anomalies and concurrent multi-node failures), and lack cross-node data sharing mechanisms, resulting in inefficient fault diagnosis. Furthermore, existing redundancy designs (such as hot standby nodes) rely on manual or static rule configuration, resulting in slow response times, limited fault tolerance, and inability to dynamically adapt to complex failure scenarios.

[0037] In response to the above technical problems, a fault diagnosis method is provided in an embodiment of the present application. This method, through federated learning, enables the models of each node to share global knowledge and continuously optimize the performance of its own model without transmitting original data, thus taking into account both data privacy and model performance improvement. In addition, through collaborative fault analysis of multiple nodes, a more comprehensive understanding of the fault situation can be achieved, avoiding misjudgment of a single node and improving fault diagnosis efficiency. In addition, in this method, the optimal redundant node can be automatically triggered to take over based on real-time fault scoring, reducing manual intervention and adapting to complex fault scenarios.

[0038] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0039] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of a fault diagnosis system provided in an embodiment of the present application.

[0040] In some embodiments, the fault diagnosis system includes a hardware layer, a BMC function layer, and a coordination layer.

[0041] In some implementations, the hardware layer can support an inter-BMC communication protocol based on an extension of the Redfish standard. Redfish is a modern, open, and scalable standard for hardware management. By extending the inter-BMC communication protocol to the Redfish standard, interoperability between hardware devices from different vendors can be achieved, facilitating unified management and monitoring.

[0042] In some implementations, the hardware layer can support the Representational State Transfer (RESTful) Application Programming Interface (API) and lightweight message queues (such as MQTT). The RESTful API provides a simple and flexible interface for facilitating resource access and manipulation; the lightweight message queue MQTT is suitable for data transmission in low-bandwidth and unstable network environments, effectively increasing data exchange rates and reducing bandwidth usage.

[0043] In some implementations, the hardware layer may include:

[0044] N server nodes (nodes 1 to N): represent the working servers that actually run the business and are the business execution units of the entire system. The number can be expanded according to actual needs.

[0045] Network management (node interconnection) module: responsible for managing the network connections between server nodes, ensuring stable and efficient data transmission and communication between nodes, and is the basis for achieving collaborative work between BMCs.

[0046] Redundant node pool: Contains backup server nodes. When a working server node fails, the redundant node can take over the business in time to ensure the availability and reliability of the system.

[0047] In some embodiments, the BMC functional layer may include:

[0048] Federated learning module, which includes:

[0049] Local model training unit: This runs on the BMC of each worker server node and is responsible for training the model using local data. Local training reduces data transmission, protects data privacy, and improves the model's adaptability to the local environment.

[0050] Parameter Encryption Transmission Interface: This interface is used to encrypt model parameters obtained through local training and securely transmit them to the federated aggregation server at the central coordination layer. Encrypted transmission prevents parameter theft or tampering during transmission, ensuring data security.

[0051] Collaborative communication module, which includes:

[0052] MCTP Extended Protocol Stack: A protocol for managing communication between components that can add new functions and features to meet requirements such as federated learning and dynamic redundancy control, ensuring efficient communication between BMCs.

[0053] Priority scheduling engine: Assign priorities to different tasks and messages to ensure that important tasks (such as fault handling, model parameter transmission, etc.) are handled first, improving the system's response speed and efficiency.

[0054] Dynamic redundancy control module, which includes:

[0055] Health score calculator: It is used to regularly collect various operating indicators of server nodes (such as temperature, voltage, performance indicators, etc.) and calculate the health score of each node according to a preset algorithm to evaluate the health status of the node.

[0056] Redundant node elector: When a working server node fails, the optimal redundant node is selected from the redundant node pool based on the health score and other relevant factors (such as physical topology distance, load balancing weight, etc.) to take over the business of the failed node.

[0057] In some embodiments, the coordination layer includes:

[0058] Federated Aggregation Server: This server receives model parameters from each worker server node and executes aggregation algorithms, such as the Federated Averaging (FedAvg) algorithm. The FedAvg algorithm takes a weighted average of the model parameters from multiple nodes to create a global model that integrates the data characteristics of each node and has better generalization capabilities.

[0059] Global Model Repository: This is used to store the latest global model and distribute it to all worker server nodes. This allows each node to continue local training based on the latest global model, enabling continuous model optimization and updates.

[0060] For example, the above fault diagnosis system may have the following advantages:

[0061] Data privacy protection: The federated learning module performs model training locally and only transmits model parameters instead of original data, effectively protecting the node's sensitive data.

[0062] High availability: The dynamic redundancy control module can monitor node status in real time and perform redundancy switching in a timely manner to ensure the continuous operation of the system.

[0063] Efficient communication: The communication protocol and lightweight message queue based on the Redfish standard extension improve the data exchange rate and reduce bandwidth usage.

[0064] Optionally, the above fault diagnosis system can be applied to the following application scenarios:

[0065] Data center management: It can be used to manage server clusters in large-scale data centers, achieve fault prediction, automatic redundancy switching and model optimization, and improve the reliability and operational efficiency of data centers.

[0066] Accurate fault warning: Each working server node in the hardware layer continuously collects its own operating data, such as temperature, voltage, fan speed, hardware performance indicators, etc. The health score calculator of the BMC functional layer analyzes this data in real time and can promptly detect abnormal changes in hardware status through preset algorithms and thresholds. The local model training unit of the federated learning module can use historical data and real-time data to train models and predict possible hardware failures. For example, the remaining life and potential failure risk of a hard drive can be predicted based on data such as the number of read and write times and error rate of the hard drive. When the health score is lower than a certain threshold or the model predicts a potential failure, the system can promptly notify the administrator through a security alert mechanism. At the same time, due to the use of federated learning, the model can integrate data features from multiple nodes to improve the accuracy and precision of fault warnings and avoid false positives and missed reports.

[0067] Security alerts and proactive security protection: The collaborative communication module of the BMC functional layer can monitor the communication data between nodes and use the extended MCTP protocol stack and priority scheduling engine to identify abnormal communication behaviors. For example, abnormal network traffic, illegal access requests, etc. are detected. When training the model, the federated learning module can incorporate security-related data, such as the number of login attempts, abnormal instruction execution, etc., to train a model that can identify security threats. When abnormal behavior is detected, the system can trigger a security alert. When the dynamic redundancy control module detects a security threat, it can take proactive measures, such as isolating the attacked node and electing a healthy node from the redundant node pool to take over the business, to prevent the further spread of security threats and achieve proactive security protection.

[0068] Reference Figure 2 , Figure 2 This is a flow chart of a fault diagnosis method provided in an embodiment of the present application. In some embodiments, the fault diagnosis method includes:

[0069] S201: Obtain the operating status data of the target node.

[0070] In some implementations, the BMC may obtain the operating status data of its corresponding target node.

[0071] Optionally, target node operational status data can be obtained from various sensors and monitoring modules at the hardware layer. For example, temperature sensors on server nodes can provide temperature data for components like the CPU and memory. Performance monitoring modules can obtain performance metrics such as CPU utilization, memory usage, and disk input / output (I / O) rates. Furthermore, network interface traffic monitoring data is an important source of operational status data, reflecting the node's network communication status.

[0072] The aforementioned operational status data is both real-time and diverse. Real-time data is reflected in the fact that it is continuously updated as the node operates, promptly reflecting the node's current status. Diversity is reflected in the fact that the data covers multiple aspects, including hardware performance and network communication, providing rich feature information for subsequent model training.

[0073] Optionally, the BMC may periodically obtain the operating status data of the target node.

[0074] S202. Based on the operating status data, the machine learning model corresponding to the target node is trained, and the training results are uploaded to the federated aggregation server.

[0075] In some implementations, the BMC can use the acquired operational status data to train a machine learning model corresponding to its target node. This machine learning model can be any suitable model for fault prediction, such as a decision tree or neural network. During training, the machine learning model learns the potential relationship between operational status data and node faults, continuously adjusting its parameters to fit the data characteristics.

[0076] Optionally, training results can be presented as model parameters. Encrypted model parameters are uploaded to the federated aggregation server via the parameter encryption transmission interface in the BMC functional layer. Encrypted transmission ensures data security during transmission, preventing parameter theft or tampering.

[0077] Optionally, the machine learning model can adopt a lightweight federated learning model with a Tiny Machine Learning (TinyML) architecture to output a failure probability (0-100%) and a classification label (such as "power supply anomaly" or "memory failure") to generate parameter increments.

[0078] In some implementations, a lightweight neural network model based on the TinyML architecture is used to adapt to the limited hardware resources of the BMC (such as embedded processors and memory particles). Through model pruning and quantization techniques, the number of parameters can be compressed to less than 10% of the original, thereby ensuring that the above-mentioned machine learning model can run efficiently locally on the BMC.

[0079] In some implementations, during the model parameter aggregation phase, by adding Gaussian noise to the gradient, differential privacy requirements can be met, avoiding the inference of node sensitive data (such as operation and maintenance information in server logs) through model parameters, thereby reducing the risk of data leakage.

[0080] S203. Receive the model update parameters fed back by the federated aggregation server, and update the above-mentioned machine learning model based on the model update parameters.

[0081] In some implementations, after receiving model parameters from multiple nodes, the federated aggregation server executes an aggregation algorithm (e.g., FedAvg) to combine the model parameters from multiple nodes to obtain global model parameters. These global model parameters are then fed back to each node as model update parameters.

[0082] After receiving the model update parameters from the federated aggregation server, the target node applies them to the local machine learning model and updates the model parameters. This allows the local model to incorporate data features and training experience from other nodes, thereby improving the model's generalization and accuracy.

[0083] S204. Predict the failure probability of the target node using the updated machine learning model.

[0084] S205: Determine whether the above-mentioned failure probability is greater than a preset threshold. If so, execute S206.

[0085] S206. Broadcast a fault message to adjacent nodes; the fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the fault type of the target node and report the fault type information to the federated aggregation server.

[0086] In some embodiments, the updated machine learning model predicts the probability of failure of the target node based on its current operating status data.

[0087] When the predicted failure probability exceeds a preset threshold, the coordinated communication module in the BMC functional layer broadcasts the failure message to adjacent nodes. This broadcast method ensures that adjacent nodes receive the failure warning information in a timely manner.

[0088] After receiving a fault message, neighboring nodes can use their corresponding machine learning models to further determine the fault type of the target node. Based on their own data and models, each node analyzes the fault characteristics from different perspectives and then reports the determined fault type to the federated aggregation server. The federated aggregation server can integrate the information reported by multiple neighboring nodes to more accurately determine the root cause of the fault and take appropriate action.

[0089] For example, assume that a data center has 1,000 nodes deployed. When the BMC predicts that the probability of node A failing due to power anomaly is greater than a preset threshold, the fault message is broadcast to adjacent nodes through the extended MCTP protocol, triggering the 10 adjacent nodes to start collaborative diagnosis.

[0090] The above 10 adjacent nodes use their corresponding machine learning models to determine the fault type of node A and report the fault type information to the federated aggregation server.

[0091] The federated aggregation server (deployed on an independent management node) combines the fault type information reported by the 10 nodes and the parameters of node A to determine that the root cause of the fault is voltage fluctuation of the power module.

[0092] The fault diagnosis method provided in the embodiments of this application, through federated learning, enables the models of each node to share global knowledge and continuously optimize their own performance without transmitting original data, thus balancing data privacy and improving model performance. Furthermore, through collaborative analysis of multiple nodes, a more comprehensive understanding of the fault situation can be achieved, avoiding misjudgment by a single node and improving fault diagnosis efficiency.

[0093] In some embodiments, the training of the machine learning model corresponding to the target node based on the operating status data includes:

[0094] The above-mentioned operating status data is converted into data in a specified format; based on the converted data in the specified format, a time series feature matrix is generated; based on the time series feature matrix, a machine learning model corresponding to the target node is trained.

[0095] Optionally, the specified format is Redfish. Redfish is a standardized hardware management data format that is open and extensible, facilitating data exchange and integration between hardware devices and systems from different manufacturers. Converting raw operational status data to Redfish improves data compatibility and readability.

[0096] The time series feature matrix described above can capture patterns in data over time. For example, by analyzing temperature trends over a period of time, we can determine whether a device is experiencing overheating or poor heat dissipation. For example, a rapid temperature increase over a short period of time could indicate an impending device failure.

[0097] Through the above implementation, the original operating status data can be effectively converted into a standard format, and time series features can be extracted for model training to achieve accurate fault prediction and processing.

[0098] In some embodiments, broadcasting the fault message to adjacent nodes includes:

[0099] Based on the extended MCTP, a fault message is broadcast to adjacent nodes; wherein the fault message includes at least one of the following fields: a fault priority tag field, a fault type code field, a timestamp field, and a health score field.

[0100] The extended MCTP has a broadcast capability, allowing a node to send messages to all adjacent nodes simultaneously. In fault handling scenarios, when a target node detects a fault, it can leverage the extended MCTP to quickly broadcast the fault message to adjacent nodes, ensuring timely dissemination of the fault information.

[0101] Priority tags can be used to identify the urgency of a fault. Different faults have varying degrees of impact on the system. By setting priority tags, adjacent nodes can prioritize high-priority faults based on their own resources and tasks. For example, faults affecting core system functions can be marked as high priority, while minor hardware performance warnings can be marked as low priority.

[0102] Optionally, the fault priority mark field can be digitally encoded. For example, when 3 bits are used, 0 represents the highest priority, which is used for serious faults that affect the core functions of the system or may cause data loss; 7 represents the lowest priority, which can be used for some minor hardware performance warnings, etc.

[0103] The fault type code field can be used to identify the specific type of fault, allowing adjacent nodes to take targeted action. Fault types can include hardware faults (such as hard drive failures and memory failures), software faults (such as system crashes and driver errors), and network faults.

[0104] Alternatively, 4-bit encoding can be used to efficiently transmit fault type information within limited bandwidth. For example, a series of fault type codes can be predefined, such as 0001 for hard disk failure, 0010 for memory failure, and 0011 for network connection failure.

[0105] The timestamp field records the exact time a fault occurred, helping nodes understand the order in which faults occurred, analyze the correlations between faults, and perform troubleshooting and diagnosis. Neighboring nodes can further assess the fault's impact on overall system performance based on the health score.

[0106] In some implementations, upon receiving a fault message, the neighboring node first parses the various fields in the message. Based on the fault priority field, it determines whether the fault needs to be addressed immediately. Based on the fault type field, it determines the specific type of fault and invokes the appropriate fault handling logic.

[0107] In some implementations, neighboring nodes utilize their own machine learning models, combined with the target node's fault information and their own operational status data, to further analyze the target node's fault type and impact. Once the analysis is complete, the determined fault type and related analysis results are reported to the federated aggregation server for global fault coordination and resolution.

[0108] In some implementations, the extended MCTP protocol may support breakpoint resuming and data fragmentation.

[0109] The breakpoint resume function ensures that after the connection is restored, data transmission can be continued from the last interruption point without having to retransmit the entire data packet. Data fragmentation can divide data into smaller fragments for transmission, which can reduce the size of each packet and enable data to be transmitted more quickly across the network.

[0110] Through the above implementation, by broadcasting fault messages containing rich fields based on the extended MCTP protocol, the system can achieve rapid and accurate dissemination of fault information, enabling neighboring nodes and federated aggregation servers to promptly understand the fault situation and take effective measures, thereby improving system reliability and fault diagnosis efficiency.

[0111] In some embodiments, if the target node receives a fault message sent by an adjacent node, the target node determines the fault type of the adjacent node based on the updated machine learning model and the fault message, and reports the fault type information to the federated aggregation server.

[0112] In some implementations, the target node may also trigger a target redundant node to take over services of an adjacent node.

[0113] In some implementations, an optimal redundant node may be selected from the redundant node pool as the target redundant node based on the health score of each redundant node, the physical topological distance between the nodes, and the load balancing weight.

[0114] In some implementations, the health score S of the redundant node may be determined using the following formula:

[0115] S=w1×hardware status parameter+w2×historical failure rate+w3×federal model prediction value.

[0116] Among them, the above hardware status parameters can be dynamically calculated through sensor data (such as voltage values and temperature values) collected by the BMC (for example, the voltage value can be directly used as the hardware status parameter), and the weights w1, w2, and w3 can be adaptively adjusted according to the node type (for example, edge nodes focus more on network stability).

[0117] The above embodiment enables collaborative fault diagnosis between nodes. Adjacent nodes broadcast fault messages, allowing the target node to promptly learn about the fault conditions of surrounding nodes and use its own model to analyze and diagnose them. This collaborative mechanism improves the accuracy and comprehensiveness of fault detection and avoids misjudgment by individual nodes.

[0118] In some embodiments, reference Figure 3 , Figure 3 This is a sub-flow diagram of a fault diagnosis method provided in an embodiment of the present application. In some embodiments, after the target node receives a fault message sent by an adjacent node, the fault diagnosis method further includes:

[0119] S301: Determine storage devices associated with adjacent nodes and their configuration information, and obtain network configuration information related to the adjacent nodes.

[0120] The configuration information of the storage device includes the connection port and access rights of the storage device; the network configuration information includes at least one of an Internet Protocol address, a media access control address, and virtual local area network setting information.

[0121] S302: Based on the configuration information of the storage device, a configuration change request is sent to the storage device using an Intelligent Platform Management Interface (IPMI) instruction. The configuration change request is used to instruct the storage device to redirect the read and write operations of the service data from the adjacent node to the target redundant node.

[0122] IPMI commands are a set of standardized interfaces and protocols for managing server hardware. By sending IPMI commands, the BMC can remotely control and configure server hardware.

[0123] S303: Based on the network configuration information, update the network configuration using IPMI instructions to redirect the network traffic of the adjacent nodes to the target redundant node.

[0124] After the network configuration is updated, network traffic from adjacent nodes will be redirected to the target redundant node. This way, business data and network requests that were originally sent to the adjacent node are forwarded to the target redundant node for processing, ensuring business continuity.

[0125] In some implementations, a timer may be introduced. If a node triggers a fault alarm twice consecutively within a preset time period, redundant switching is suspended and a manual review process is started to avoid resource jitter caused by misjudgment.

[0126] In the above embodiment, by redirecting service data read and write operations and network traffic from adjacent nodes to the target node, service continuity can be ensured in the event of an adjacent node failure, reducing service interruption time and improving system availability. Furthermore, by utilizing intelligent platform management interface instructions to automatically modify storage device and network configurations, the efficiency and accuracy of fault diagnosis are improved, the need for manual intervention is reduced, and operational and maintenance costs are lowered.

[0127] In some embodiments, the above method further comprises:

[0128] When the target node is successfully repaired, the repair data collected during the repair process is uploaded to the federated aggregation server; optionally, the repair data includes the cause of the failure, the fault repair method, and the operating status information of the target node before and after the repair.

[0129] In some implementations, when any node fails and undergoes maintenance (such as firmware repair or hardware replacement), it can enter a preparation phase for rejoining the federated learning network.

[0130] Repaired nodes can also upload relevant data collected during the repair process to the federated learning network. This repair data includes information such as the cause of the failure, the repair method, and the node's operating status before and after the repair. For example, if a node is being repaired due to a hard drive failure, the repair data may include the hard drive's error log and performance test results after the hard drive is replaced.

[0131] In the above embodiment, by continuously utilizing new data uploaded by repaired nodes for incremental learning, the global model can learn a wider range of fault characteristics and repair experiences, thereby improving the accuracy of fault diagnosis. Furthermore, the self-healing feedback and model update mechanisms form a closed loop, enabling the system to continuously optimize and improve itself. When a node fails and is repaired, the system can automatically incorporate the repair experience into the global model, enabling other nodes to better diagnose and handle similar faults.

[0132] The following embodiment of the present application further provides a fault diagnosis system, which includes a federated aggregation server and multiple nodes, each of which is provided with a BMC.

[0133] BMC is used to: obtain the operating status data of the target node; based on the operating status data, train the machine learning model corresponding to the target node and upload the training results to the federated aggregation server; receive the model update parameters fed back by the federated aggregation server and update the machine learning model based on the model update parameters; use the updated machine learning model to predict the failure probability of the target node, and when the failure probability is greater than the preset threshold, broadcast a fault message to the adjacent nodes; the fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the fault type of the target node and report the fault type information to the federated aggregation server.

[0134] The federated aggregation server is used to: determine model update parameters based on the training results received from multiple nodes, and feed back the model update parameters to the multiple nodes; and determine the root cause of the fault based on the fault type information of the target node uploaded by at least one node.

[0135] It can be understood that the implementation process of the above-mentioned fault diagnosis system can refer to the content of the above-mentioned method embodiment, and will not be repeated here.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0137] The present application also provides a fault diagnosis device. Figure 4 As shown, Figure 4 Schematic diagram of the structure of a fault diagnosis device provided in an embodiment of the present application. The fault diagnosis device 40 includes:

[0138] The acquisition module 401 is used to acquire the operating status data of the target node.

[0139] The training module 402 is used to train the machine learning model corresponding to the target node based on the above-mentioned operating status data, and upload the training results to the federal aggregation server.

[0140] The updating module 403 is used to receive the model updating parameters fed back by the federated aggregation server and update the machine learning model based on the model updating parameters.

[0141] Prediction module 404 is used to use the updated machine learning model to predict the failure probability of the target node. When the failure probability is greater than a preset threshold, a fault message is broadcast to adjacent nodes. The fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the failure type of the target node and report the failure type information to the federal aggregation server.

[0142] In some embodiments, the training module 402 is configured to:

[0143] Convert the above running status data into data in a specified format;

[0144] Generate a time series feature matrix based on the converted data in the specified format;

[0145] Based on the above time series feature matrix, the machine learning model corresponding to the target node is trained.

[0146] In some implementations, broadcasting the fault message to adjacent nodes includes:

[0147] Broadcast fault messages to adjacent nodes based on the extended management component transmission protocol;

[0148] The above-mentioned fault message includes at least one of the following fields: a fault priority mark field, a fault type code field, a timestamp field, and a health score field.

[0149] In some embodiments, the apparatus further comprises a processing module configured to:

[0150] When receiving a fault message from a neighboring node, the fault type of the neighboring node is determined based on the updated machine learning model and the received fault message, and the fault type information is reported to the federated aggregation server;

[0151] The target redundant node is triggered to take over the services of the adjacent node.

[0152] In some embodiments, the processing module is further configured to:

[0153] Determining a storage device associated with the adjacent node and its configuration information, and obtaining network configuration information related to the adjacent node; the storage device configuration information includes a connection port and access permissions of the storage device; the network configuration information includes at least one of an Internet Protocol address, a media access control address, and virtual local area network configuration information;

[0154] Based on the configuration information of the storage device, a configuration change request is sent to the storage device using an intelligent platform management interface instruction. The configuration change request is used to instruct the storage device to redirect the read and write operations of the service data from the adjacent node to the target redundant node;

[0155] Based on the above network configuration information, the network configuration is updated using the intelligent platform management interface instructions to redirect the network traffic of the adjacent nodes to the target redundant node.

[0156] In some embodiments, the processing module is further configured to:

[0157] When the target node is successfully repaired, the repair data collected during the repair process is uploaded to the federated aggregation server; the above repair data includes the cause of the failure, the fault repair method, and the operating status information of the target node before and after the repair.

[0158] For the description of the features in the embodiment corresponding to the above-mentioned fault diagnosis device, reference can be made to the relevant description of the embodiment corresponding to the above-mentioned fault diagnosis method, and no further details will be given here.

[0159] The fault diagnosis device provided in the embodiments of this application, through federated learning, enables the models of each node to share global knowledge and continuously optimize their own performance without transmitting raw data, thus balancing data privacy and improving model performance. Furthermore, through collaborative analysis of multiple nodes, a more comprehensive understanding of the fault situation can be achieved, avoiding misjudgments by individual nodes and improving fault diagnosis efficiency.

[0160] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0161] During the specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502 , so that the at least one processor 501 executes the above-mentioned fault diagnosis method embodiment.

[0162] The specific implementation process of the processor 501 can be found in the above-mentioned fault diagnosis method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0163] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the fault diagnosis method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0164] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0165] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0166] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned fault diagnosis method embodiments when running.

[0167] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0168] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned fault diagnosis method embodiments are implemented.

[0169] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned fault diagnosis method embodiments are implemented.

[0170] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] The above is a detailed introduction to a fault diagnosis method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and such improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A fault diagnosis method, characterized in that: The method comprises: Get the running status data of the target node; Based on the operating status data, training the machine learning model corresponding to the target node and uploading the training results to the federated aggregation server; receiving the model update parameters fed back by the federated aggregation server, and updating the machine learning model based on the model update parameters; The updated machine learning model is used to predict the failure probability of the target node. When the failure probability is greater than a preset threshold, a fault message is broadcast to the adjacent nodes. The fault message is used to trigger the adjacent nodes to use their own corresponding machine learning models to determine the failure type of the target node and report the failure type information to the federal aggregation server.

2. The method according to claim 1, characterized in that The training of the machine learning model corresponding to the target node based on the operating status data includes: Converting the running status data into data in a specified format; Generate a time series feature matrix based on the converted data in the specified format; Based on the time series feature matrix, the machine learning model corresponding to the target node is trained.

3. The method according to claim 1, characterized in that The broadcasting of the fault message to the adjacent nodes includes: Broadcasting a fault message to the adjacent nodes based on an extended management component transmission protocol; The fault message includes at least one of the following fields: a fault priority mark field, a fault type code field, a timestamp field, and a health score field.

4. The method according to claim 1, wherein The method further comprises: When receiving a fault message sent by a neighboring node, determining the fault type of the neighboring node based on the updated machine learning model and the received fault message, and reporting the fault type information to the federated aggregation server; The target redundant node is triggered to take over the service of the adjacent node.

5. The method according to claim 4, characterized in that The triggering of the target redundant node to take over the service of the adjacent node includes: Determining a storage device associated with the adjacent node and its configuration information, and obtaining network configuration information related to the adjacent node; the storage device configuration information includes a connection port and access rights of the storage device; the network configuration information includes at least one of an Internet Protocol address, a media access control address, and virtual local area network setting information; Based on the configuration information of the storage device, a configuration change request is sent to the storage device using an intelligent platform management interface instruction, wherein the configuration change request is used to instruct the storage device to redirect the read and write operations of the service data from the adjacent node to the target redundant node; Based on the network configuration information, the network configuration is updated using intelligent platform management interface instructions to redirect the network traffic of the adjacent node to the target redundant node.

6. The method according to claim 1, characterized in that The method further comprises: When the target node is successfully repaired, the repair data collected during the repair process is uploaded to the federated aggregation server; the repair data includes the cause of the failure, the failure repair method, and the operating status information of the target node before and after the repair.

7. A fault diagnosis system, characterized in that: The system includes a federated aggregation server and a plurality of nodes, wherein the nodes are provided with a baseboard management controller; The baseboard management controller is used to: obtain operating status data of the target node; based on the operating status data, train the machine learning model corresponding to the target node, and upload the training results to the federated aggregation server; Receive the model update parameters fed back by the federated aggregation server, and update the machine learning model based on the model update parameters; use the updated machine learning model to predict the failure probability of the target node, and when the failure probability is greater than a preset threshold, broadcast a fault message to the adjacent nodes; the fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the failure type of the target node and report the failure type information to the federated aggregation server; The federation aggregation server is configured to: determine a model update parameter based on the received training results uploaded by the multiple nodes, and feed back the model update parameter to the multiple nodes; And determining the root cause of the fault based on the fault type information of the target node uploaded by at least one node.

8. A fault diagnosis device, characterized in that: include: The acquisition module is used to obtain the running status data of the target node; A training module, configured to train a machine learning model corresponding to the target node based on the operating status data, and upload the training results to a federated aggregation server; An updating module, configured to receive the model updating parameters fed back by the federated aggregation server and update the machine learning model based on the model updating parameters; A prediction module is used to use the updated machine learning model to predict the failure probability of the target node. When the failure probability is greater than a preset threshold, a fault message is broadcast to adjacent nodes. The fault message is used to trigger the adjacent nodes to use their corresponding machine learning models to determine the failure type of the target node and report the failure type information to the federal aggregation server.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the fault diagnosis method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault diagnosis method according to any one of claims 1 to 6.

Images