Device fault prediction method and device, electronic device, and storage medium

Abstract

The application discloses a device fault prediction method and device, an electronic device and a storage medium, and relates to the field of servers.The method comprises the following steps: acquiring a state data set, wherein the state data set comprises state data of M devices in a server at N continuous time points; performing fault prediction on each of the M devices by using a prediction model according to the state data set and a topological relationship of the M devices, and obtaining M pieces of fault prediction data corresponding to the M devices, wherein the prediction model uses state data of devices having a topological connection relationship with an mth device in the M devices in the process of performing fault prediction on the mth device, and the mth fault prediction data is used for predicting whether the mth device has a fault and a type of the fault. By using the technical scheme, the problem that the accuracy of fault prediction on the devices in the server is low is solved.

Description

Technical Field

[0001] This application relates to the field of servers, and more particularly to a method and apparatus for predicting equipment failures, electronic devices, and storage media. Background Technology

[0002] With the rapid expansion of data center server scale and the increase in device density, internal server devices (such as Graphics Processing Units (GPUs), Non-Volatile Memory Express Solid State Drives (NVMe SSDs), network interface cards, etc.) face more complex operating environments and higher failure rates. Traditional device failure prediction methods typically rely on the status data of individual devices, such as sensor readings of temperature and voltage, and set static thresholds for fault warning and handling. However, this method has significant limitations, mainly in the following aspects: 1) An abnormal state of one device may be indirectly caused by the failure of another device connected to it, but traditional methods struggle to identify such indirect correlations; 2) In high-density server environments, the mutual influence between devices is significant, and the analysis of single device status data cannot fully reflect its health status, leading to a decrease in the accuracy of fault prediction.

[0003] There is currently no effective solution to the problem of low accuracy in predicting faults in server devices in related technologies. Summary of the Invention

[0004] This application provides a method and apparatus for predicting equipment failures, an electronic device, and a storage medium, to at least solve the problem of low accuracy in predicting equipment failures in servers.

[0005] This application provides a device fault prediction method, comprising: acquiring a state data set, wherein the state data set includes state data of M devices in a server at N consecutive time points, where M and N are both positive integers greater than or equal to 2; and using a prediction model to predict the fault of each of the M devices based on the state data set and the topological relationship of the M devices, thereby obtaining M fault prediction data corresponding to the M devices. Specifically, in the process of predicting the fault of the m-th device among the M devices, the prediction model uses the state data of the devices among the M devices that have a topological connection with the m-th device, where m is an integer greater than or equal to 1 and less than or equal to M. The m-th fault prediction data is used to predict whether the m-th device has a fault and the type of fault if it does.

[0006] This application also provides a device for predicting device failures, comprising: an acquisition module for acquiring a state data set, wherein the state data set includes state data of M devices in a server at N consecutive time points, where M and N are both positive integers greater than or equal to 2; and a prediction module for predicting failures of each of the M devices based on the state data set and the topological relationship of the M devices using a prediction model, thereby obtaining M failure prediction data corresponding to the M devices. The prediction model uses the state data of devices with topological connections to the m-th device among the M devices during the failure prediction process, where m is an integer greater than or equal to 1 and less than or equal to M. The failure prediction data is used to predict whether a device has a failure and the type of failure.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described device fault prediction methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described device fault prediction methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described device fault prediction methods.

[0010] This application collects state data sets of M devices at multiple time points and combines this with the topological relationships among the M devices. The resulting prediction model can more comprehensively analyze device health status, capturing not only dynamic changes in device status but also considering physical connections and logical dependencies between devices, effectively identifying indirect fault impacts caused by topological associations. When predicting the fault of any device (the m-th device) in a server, the model simultaneously considers the status of other connected devices, achieving accurate fault type determination and avoiding inaccurate predictions due to a single perspective. This improves the overall accuracy and reliability of fault prediction, solving the problem of low accuracy in fault prediction for devices in servers. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a hardware structure block diagram of a device fault prediction method according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of a device failure prediction method according to an embodiment of this application;

[0014] Figure 3 This is a system architecture diagram of an equipment fault prediction system according to an embodiment of this application;

[0015] Figure 4 This is a flowchart of a prediction model according to an embodiment of this application;

[0016] Figure 5 This is a schematic diagram of the protocol flow of a protocol control layer according to an embodiment of this application;

[0017] Figure 6 This is a flowchart illustrating a multi-parameter monitoring and health scoring process for an optical link according to an embodiment of this application.

[0018] Figure 7 This is a flowchart illustrating another optical link multi-parameter monitoring and health scoring process according to an embodiment of this application.

[0019] Figure 8 This is a structural block diagram of a device for predicting equipment failure according to an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The specific application environment architecture or specific hardware architecture on which the execution of the equipment failure prediction method depends is described here.

[0024] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a device fault prediction method according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0025] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the operating system startup method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0027] To address the aforementioned problems, this embodiment provides a method for predicting equipment failures. Figure 2 This is a flowchart of a device fault prediction method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps S202-S204:

[0028] Step S202: Obtain a state data set, wherein the state data set includes the state data of M devices in the server at N consecutive time points, where M and N are both positive integers greater than or equal to 2;

[0029] Optionally, the device's status data at a given moment includes, but is not limited to: temperature, voltage, power consumption, computational load, error count, etc. Furthermore, for optical interconnect devices, the status data also includes: optical link health score.

[0030] Optionally, the M devices in this application are Peripheral Component Interconnect (ExpressPCIe) devices. It should be noted that a PCIe device is a computer hardware device based on the PCIe bus standard.

[0031] Step S204: Using the prediction model, based on the state data set and the topological relationship of the M devices, perform fault prediction on each of the M devices to obtain M fault prediction data corresponding to the M devices. In the process of predicting the fault of the m-th device among the M devices, the prediction model will use the state data of the devices that have a topological connection relationship with the m-th device among the M devices, where m is an integer greater than or equal to 1 and less than or equal to M. The m-th fault prediction data is used to predict whether the m-th device has a fault and the type of fault.

[0032] It should be noted that the topology of M devices is used to describe the physical connection between the M devices (such as which GPU is connected to which switch). It can be a graph structure that defines the edges between nodes (i.e., devices) (i.e., the topological connection between devices).

[0033] The above steps collect state data sets of M devices at multiple time points and, combined with the topological relationships of these M devices, enable the prediction model to more comprehensively analyze device health status. It not only captures dynamic changes in device status but also considers physical connections and logical dependencies between devices, effectively identifying indirect fault impacts caused by topological associations. When predicting the fault of any device (the m-th device) in the server, the model simultaneously considers the status of other connected devices, achieving accurate fault type judgment and avoiding inaccurate predictions caused by a single perspective. This improves the overall accuracy and reliability of fault prediction, solving the problem of low accuracy in fault prediction for devices in servers.

[0034] In an exemplary embodiment, the above-described method, using a prediction model to predict the faults of each of the M devices based on a set of state data and the topological relationships of the M devices, to obtain M fault prediction data corresponding to the M devices, includes: obtaining the fault prediction data corresponding to the m-th device among the M devices through the following steps S11-S12, to obtain M fault prediction data:

[0035] Step S11: Using the graph neural network of the prediction model, N target state data are obtained based on the state data of the m-th device at N consecutive time points and the state data of each of the P devices at N consecutive time points. Each of the P devices among the M devices has a topological connection relationship with the m-th device. The n-th target state data among the N target state data is determined based on the state data of the m-th device at the n-th time point and the state data of each of the P devices at the n-th time point.

[0036] It should be noted that the graph neural network layer of the model utilizes the state data of the devices at multiple time points, combined with their topological connections, to comprehensively analyze the state data of each device P that has a direct connection with the target device m, generating N target state data. Here, the target state data at each time point n integrates the state information of the m-th device itself and the state information of P related devices, thus constructing a dynamic picture that includes the mutual influence between devices.

[0037] Step S12: Obtain the target feature representation based on N target state data through the encoder of the prediction model;

[0038] Optionally, the encoder for the prediction model is a Transformer encoder, used for pattern capture in the time dimension.

[0039] It should be noted that by processing these N target state data through the Transformer encoder, the encoder can capture and analyze the complex patterns of device state changes over time, outputting a highly generalized target feature representation. This feature representation fully integrates the device's own temporal information and the dynamic characteristics of the topological neighborhood, laying a solid foundation for subsequent fault prediction.

[0040] Step S13: Obtain the fault prediction data corresponding to the m-th device through the fully connected layer of the prediction model based on the target feature representation.

[0041] It should be noted that the target feature representation is further processed through a fully connected layer. This layer acts as the "decision brain" of the model, leveraging the regression and classification capabilities of deep learning to output fault prediction data for the m-th device. The fault prediction data includes health score parameters and predicted fault probabilities for various fault types.

[0042] It should be noted that the above steps make full use of the topological relationships and timing data between devices, which significantly improves the reliability and timeliness of fault prediction.

[0043] In an exemplary embodiment, the above-described method of obtaining N target state data using a graph neural network of a prediction model, based on the state data of the m-th device at N consecutive time points and the state data of each of the P devices at N consecutive time points, includes: obtaining the n-th target state data using the graph neural network of the prediction model through the following steps S21-S22, to obtain N target state data:

[0044] Step S21: Determine the correlation coefficient between the m-th device and each of the P devices based on the state data of the m-th device at time n and the state data of each of the P devices at time n, thus obtaining P correlation coefficients. Among the P correlation coefficients, the p-th correlation coefficient is used to indicate the importance of the p-th device to the m-th device.

[0045] In an exemplary embodiment, step S21 above includes: calculating the p-th correlation coefficient to obtain P correlation coefficients by:

[0046] ;

[0047] in, Let p be the p-th correlation coefficient. This is the state data of the m-th device at time n. This represents the state data of the p-th device at time n. For P devices, This is the weight matrix. is a learnable attention vector; LeakyReLU is an activation function.

[0048] It should be noted that the above formula uses a shared weight matrix. and a learnable attention vector We calculate paired, unnormalized attention scores, then normalize them using the softmax function to obtain the final correlation coefficient.

[0049] Step S22: Determine the nth target state data based on the P correlation coefficients and the state data of each of the P devices at time n.

[0050] In an exemplary embodiment, step S22 above includes: calculating the nth target state data in the following manner:

[0051] ;

[0052] in, For the nth target state data, Let P be the state data of the p-th device at time n. Let be the correlation coefficient between the m-th device and the p-th device out of P correlation coefficients. For P devices, This is the activation function.

[0053] It should be noted that the above formula achieves spatial aggregation of information. The features of each neighboring node are arranged according to their importance. After weighting, the data is aggregated to the m-th device, and then activated by the activation function. By introducing a nonlinear transformation, the nth target state data of the mth device is finally generated.

[0054] It should be noted that the above formula not only enhances the model's understanding of the interactions between devices, but also ensures a more accurate and comprehensive prediction of the failure of the m-th device, effectively avoiding prediction bias that may arise from single-device status data. By dynamically adjusting the correlation coefficients between devices, the model can respond in real time to changes in the server topology environment, providing strong support for fault prevention.

[0055] It should be noted that in step S22, the GNN updates the state representation of the target device m based on these correlation coefficients and the state data of neighboring devices, generating the nth target state data. This process is similar to "information exchange" between devices, making the feature representation of the target device m no longer isolated, but incorporating the influence of all related devices in its topological environment, constructing a more global and interconnected view of device states. Through the above scheme, even when facing complex server topologies, the prediction model can meticulously capture the interdependencies between devices, significantly improving the accuracy and robustness of fault prediction.

[0056] In an exemplary embodiment, the encoder of the prediction model described above obtains the target feature representation based on N target state data, including the following steps S31-S32:

[0057] Step S31: Arrange the N target state data in chronological order to obtain the spatiotemporal feature sequence, where the spatiotemporal feature sequence is [ , ..., , ];

[0058] Step S32: The target feature representation is obtained from the spatiotemporal feature sequence through multiple coding layers of the encoder of the prediction model. Each coding layer uses a multi-head self-attention mechanism and a feedforward neural network to process the input feature representation of the coding layer to obtain the output feature representation. The input feature representation of the first coding layer of the multiple coding layers is the spatiotemporal feature sequence, and the output feature representation of the last coding layer of the multiple coding layers is the target feature representation.

[0059] It should be noted that the output feature representation of each coding layer in a multi-layer coding system becomes the input feature representation of the next coding layer.

[0060] It should be noted that step S31 above generates a spatiotemporal feature sequence by arranging N target state data in chronological order. This sequence not only contains snapshots of the device's state at multiple time points but also implicitly reveals the dynamic trend of the device's state evolution over time, providing a continuous data stream in the time dimension for subsequent time series analysis. Subsequently, in step S32, this spatiotemporal feature sequence is input into the encoder of the prediction model, which consists of a series of stacked encoding layers with identical structures. Each encoding layer employs a multi-head self-attention mechanism and a feedforward neural network to process the data, aiming to capture the complex dependencies between features at different time points in the sequence and the potential patterns of the device's state. The multi-head self-attention mechanism allows the model to focus on multiple key points in the sequence simultaneously, enhancing its ability to understand long sequences, while the nonlinear transformation introduced by the feedforward neural network further enhances the model's expressiveness, enabling it to fit more complex functional relationships.

[0061] It should be noted that the target feature representation obtained through the above method provides a highly comprehensive feature vector for subsequent fault prediction.

[0062] In an exemplary embodiment, the above-described method of obtaining a target feature representation based on a spatiotemporal feature sequence through multiple coding layers of the encoder of the prediction model includes: performing the following steps S41-S42 through the i-th coding layer of the multiple coding layers to obtain the target feature representation through the multiple coding layers of the encoder of the prediction model:

[0063] Step S41: Process the input feature representation of the i-th coding layer through the multi-head self-attention mechanism of the i-th coding layer to obtain the intermediate feature representation of the i-th coding layer;

[0064] In an exemplary embodiment, step S41 is implemented through the following steps S411-S412:

[0065] Step S411: Calculate the self-attention of the i-th coding layer using the following formula:

[0066] (Formula 1);

[0067] in, , This represents the input features of the i-th coding layer. For trainable weight matrix, Let k be the dimension of matrix k;

[0068] Step S412: Perform residual connections and layer normalization on the self-attention of the i-th coding layer using the following formula 2 to obtain the intermediate feature representation of the i-th coding layer:

[0069] (Formula 2);

[0070] in, This represents the intermediate feature representation of the i-th coding layer.

[0071] It should be noted that in this embodiment, the encoding layer first performs a linear transformation on the input feature representation using a shared trainable weight matrix, generating three matrices: query (Q), key (K), and value (V). Through the self-attention mechanism, the importance weights between positions in the input sequence can be calculated efficiently, accurately capturing even long-distance dependencies. Next, the calculated self-attention results are further optimized. This optimization process includes two key steps: residual connections and layer normalization. Residual connections, by adding and fusing the self-attention results with the original input feature representation, preserve the direct information of the input sequence while allowing the model to benefit from the complex dependencies learned from the self-attention mechanism, accelerating training convergence and preventing information loss. Layer normalization, through standardization operations, scales and translates the output features of each layer, ensuring a consistent distribution of output features across all layers, avoiding gradient vanishing or exploding problems, and improving the model's training stability and generalization ability.

[0072] Step S42: Process the intermediate feature representation of the i-th coding layer through the feedforward neural network of the i-th coding layer to obtain the output feature representation of the i-th coding layer.

[0073] In an exemplary embodiment, step S42 is implemented through the following steps S421-S422:

[0074] Step S421: Obtain the output of the feedforward neural network using the following formula three:

[0075] (Formula 3);

[0076] in, This represents the output of the feedforward neural network in the i-th coding layer. This represents the intermediate feature representation of the i-th coding layer. For trainable parameters, It is an activation function;

[0077] Step S422: Perform residual connections and layer normalization on the output of the feedforward neural network of the i-th coding layer using the following formula four to obtain the output feature representation of the i-th coding layer:

[0078] (Formula 4);

[0079] in, Let be the output feature representation of the i-th coding layer.

[0080] It should be noted that steps S421 and S422 perform nonlinear transformations on the intermediate feature representations using a feedforward neural network (FFN), followed by residual connections and layer normalization techniques to further optimize the output feature representations, ensuring the model's expressiveness and training stability. In step S421, the feedforward neural network performs a series of linear transformations and activation function calculations on the intermediate feature representations of the i-th encoding layer processed by the self-attention mechanism. This transformation process enables the model to learn more complex and abstract feature representations from the original intermediate feature representations, further capturing potential patterns in the device state sequence. Next, in step S422, the output of the feedforward neural network is optimized through residual connections and layer normalization. Residual connections, by directly adding the output of the feedforward neural network to the feature representation of the previous stage, ensure that important information of the input features is preserved, while allowing for deeper model structures and avoiding the gradient vanishing problem. Layer normalization, by calculating the mean and standard deviation of the output features of each layer, standardizes them, ensuring the consistency of the distribution of the output features of each layer, improving the model's generalization ability and training efficiency.

[0081] In an exemplary embodiment, obtaining the fault prediction data corresponding to the m-th device based on the target feature representation includes the following steps S51-S52:

[0082] Step S51: Through the first fully connected layer of the prediction model, obtain the health score parameter value of the m-th device based on the target feature representation, where the health score parameter value is used to reflect the health status of the m-th device;

[0083] It should be noted that in step S51, the first fully connected layer of the prediction model receives the target feature representation after deep processing by the encoder, and calculates and outputs the health score parameter value through a series of linear transformations and nonlinear activation functions. This score is a quantitative representation of the overall health status of the equipment, which can intuitively reflect whether the equipment is in good operating condition or has a risk of failure, and is an important basis for deciding whether to take preventive measures.

[0084] Optionally, the first fully connected layer obtains the health score parameter value of the m-th device in the following way:

[0085] ,in, , and Z represents the trainable parameters and Z represents the target feature.

[0086] Step S52: Through the second fully connected layer of the prediction model, obtain the predicted failure probability of the m-th device under multiple failure types based on the target feature representation;

[0087] The fault prediction data includes health score parameters and predicted fault probabilities for various fault types; the first fully connected layer and the second fully connected layer are two parallel fully connected layers of the prediction model.

[0088] It should be noted that in step S52, the second fully connected layer also takes the target feature representation as input, but it focuses on analyzing the probability of the device under various preset fault types. Through calculation, the model generates a predicted probability for each fault type, and this output provides an accurate probability assessment for identifying a specific fault type.

[0089] Optionally, the second fully connected layer obtains the predicted failure probability of the m-th device across multiple failure types in the following manner:

[0090] ;in, and Z represents the trainable parameters and the target feature representation. The predicted failure probabilities are for normal operation, overheating, link degradation, and hardware error, respectively.

[0091] It should be noted that the parallel fully connected layer design in steps S51-S52 not only reflects the ingenious layout of the model structure and ensures a comprehensive assessment of the device status, but also effectively distinguishes between the universality and particularity of faults, providing rich information for subsequent fault management and resource scheduling.

[0092] In an exemplary embodiment, after obtaining M fault prediction data corresponding to M devices, the method further includes: predicting whether each device has a fault and the type of fault based on the fault prediction data corresponding to each device and a preset decision rule.

[0093] Optionally, the preset decision rules can be as shown in Table 1 below.

[0094] In an exemplary embodiment, after obtaining M fault prediction data corresponding to M devices, the method further includes the following steps S61-S62:

[0095] Step S61: If it is determined that the m-th device needs to be isolated based on the fault prediction data corresponding to the m-th device, an isolation command is sent to the m-th device through the server's baseboard management controller;

[0096] Step S62: Upon receiving the determination instruction sent by the m-th device in response to the isolation instruction, an execution instruction is sent to the complex programmable logic device on the motherboard of the server via the baseboard management controller. The execution instruction is used to instruct the complex programmable logic device to perform a power-off operation on the m-th device.

[0097] It should be noted that in this embodiment, the server performs intelligent decision-making and security isolation operations on the m-th device based on fault prediction data (including health score parameters and predicted fault probabilities for various fault types) to prevent potential faults from affecting the overall operation of the data center. When the prediction model determines that the health status of the m-th device (such as a GPU or network adapter) is below a preset threshold, or the predicted probability of a specific fault type reaches the warning level, the isolation process will be automatically triggered to protect critical data and business continuity.

[0098] Optionally, in step S61, the server's Baseboard Management Controller (BMC) sends an isolation command to the m-th device via a security authentication protocol. This isolation command not only includes the device's specific power-off requirements but also includes a timestamp and digital signature to ensure the command's integrity and prevent tampering. Upon receiving the command, the device performs a validity verification to ensure the command's source is trustworthy and to avoid the risk of malicious interruption.

[0099] In step S62, after receiving the confirmation instruction, the BMC will send the confirmation instruction to the Complex Programmable Logic Device (CPLD) on the server motherboard. As a key component of the hardware isolation layer, the CPLD will perform the final power-off operation.

[0100] It should be noted that the above methods enable precise, safe, and non-destructive isolation of faulty equipment, effectively preventing equipment failure risks within the data center.

[0101] In an exemplary embodiment, the isolation instruction includes the certificate identifier of the baseboard management controller. The m-th device sends a verification request to the certificate authority based on the certificate identifier of the baseboard management controller to verify the legitimacy of the baseboard management controller, wherein the verification request carries the certificate identifier of the baseboard management controller. After sending the confirmation instruction, the m-th device performs an internal cache data synchronization operation. After the complex programmable logic device performs a power-off operation on the m-th device, the transient voltage suppression diode on the power input current of the m-th device is activated to absorb surges.

[0102] It should be noted that in this embodiment, when generating isolation commands, the BMC embeds its own certificate identifier (ID) into the command. This certificate identifier is pre-issued by an authoritative Certificate Authority (CA) as official proof of the BMC's identity. When the m-th device (such as a GPU or NVMe SSD) receives the isolation command, it sends a verification request to the CA based on the BMC certificate identifier in the command to ensure the legitimacy of the BMC and prevent malicious or forged commands from triggering unnecessary device isolation. After the verification request is successful, it is confirmed that the BMC's certificate is valid and the command source is trustworthy. The m-th device will then perform a series of internal data synchronization operations to ensure that all critical data is securely saved before physical power loss. This process involves synchronizing incomplete Transaction Layer Packets (TLPs) to the dual-port RAM cache or backing up important state information to persistent storage, providing data protection for the device's isolation operations.

[0103] Furthermore, when the Complex Programmable Logic Device (CPLD) receives the execution instruction from the BMC, it will perform a power-off operation on the m-th device. At the moment of power failure, the Transient Voltage Suppressor (TVS) diode at the device's power input is activated, effectively absorbing voltage spikes or surges caused by the power failure, protecting the device and server from power surge damage, and ensuring the hardware security of the isolation operation. This series of security designs and data protection measures enables the isolation mechanism of this application to effectively prevent data loss and hardware damage while ensuring server security.

[0104] In an exemplary embodiment, after obtaining M fault prediction data corresponding to M devices, the method further includes: when it is determined from the fault prediction data corresponding to the m-th device that the m-th device needs to perform a redundant link switching operation and the m-th device is a PCIe device, writing a target value to the link control register in the central processing unit of the server through the baseboard management controller of the server, so as to perform PCIe link training on the m-th device.

[0105] In this embodiment, when the server analyzes fault prediction data and determines that the m-th device (a PCIe device, such as a high-performance network card or GPU) faces a potential fault threat, and determines that redundant link switching can effectively prevent a cascading failure, the server's BMC first writes a preset target value to the link control register in the server's central processing unit (CPU). This target value contains the control signals and parameter settings required for link switching, for example, instructing the CPU to start the PCIe link training process for the m-th device. Link training is a complex process designed to rebuild or optimize the physical connection between the device and the motherboard, ensuring the correctness and high speed of data transmission. In actual operation, the CPU reads the target value, parses the corresponding link training instructions, and then sends a link training request to the m-th device through the PCI Express (PCIe) protocol stack. After receiving the request, the device will cooperate with the CPU to negotiate and retrain the link parameters, ultimately establishing a new, stable PCIe link.

[0106] It's important to note that by performing link switching and training in advance, the impact of faulty devices on the entire server architecture can be proactively avoided, while ensuring uninterrupted data transmission. Especially in the context of PCIe devices, link switching and training operations can smoothly transition to redundant links without affecting existing services, avoiding data interruptions and packet loss that could occur with direct power outages and isolation, thus significantly improving server reliability and user experience.

[0107] In an exemplary embodiment, the method further includes: if the m-th device is an optical interconnect device, sending a target instruction to the m-th device via a substrate management controller, wherein the target instruction instructs the m-th device to disable the problematic laser and enable a backup laser array.

[0108] In this embodiment, when the health score parameter of the m-th optical interconnect device is detected to be lower than a preset threshold, or when a certain laser is predicted to have a failure risk, the BMC will send a target instruction to the device, instructing the device to disable the current problematic laser and activate its built-in backup laser array.

[0109] It should be noted that the above method allows the device to seamlessly switch from one laser to a backup laser without interrupting network communication, effectively avoiding the risk of network service interruption and data loss. Furthermore, the target command sent by the BMC not only includes specific operational instructions for laser switching but can also include parameter configuration and performance adjustment suggestions for the laser array, ensuring that the backup laser can quickly stabilize after the switch and restore the device to its optimal transmission state.

[0110] In an exemplary embodiment, the method further includes the following steps S71-S73:

[0111] Step S71: If there are optical interconnect devices among the M devices, obtain the laser bias current, optical power attenuation, and temperature slope of the optical interconnect devices;

[0112] Step S72: When the laser bias current of the optical interconnect device is greater than or equal to the preset current value, perform an isolation operation on the optical interconnect device;

[0113] Step S73: When the laser bias current of the optical interconnect device is less than the preset current value, determine the optical link health score of the optical interconnect device based on the optical power attenuation and temperature slope, wherein the status data of the optical interconnect device at a certain moment includes the optical link health score.

[0114] It's important to note that three core metrics need to be obtained from the optical interconnect device: laser bias current, optical power attenuation, and temperature slope. These parameters reflect the health status and potential risks of the optical link. Furthermore, when the laser bias current reaches or exceeds a preset current threshold (e.g., 120mA), the optical interconnect device will be immediately isolated to prevent laser failure due to overload and protect the device from further damage. When the laser bias current has not yet reached the emergency threshold, a more detailed evaluation phase will begin. By integrating parameters such as optical power attenuation and temperature slope into an expert-rule health scoring algorithm, a comprehensive optical link health score is generated, providing a scientific basis for subsequent decision-making logic (such as gradual degradation and maintenance planning).

[0115] It should be noted that this application achieves comprehensive management and response to optical interconnect device faults through the organic combination of real-time monitoring, threshold control, and comprehensive scoring. This solution can not only respond to high-risk faults in a timely manner, but also provide early warning intervention in sub-optimal states, which is of great significance for ensuring the stability of data transmission and improving the service life of optical interconnect devices.

[0116] In one exemplary embodiment, determining the optical link health score of an optical interconnect device based on optical power attenuation and temperature slope includes: determining the optical link health score of the optical interconnect device by means of:

[0117] ;

[0118] in, Assess the health of the optical link. For optical power attenuation, This is the initial value of the optical power. This represents the temperature slope.

[0119] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above method, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0120] Figure 3 A system architecture diagram of an equipment failure prediction system is shown, which implements the aforementioned equipment failure prediction method. This system has a layered architecture and mainly includes:

[0121] Data Acquisition Layer: Collects multimodal time-series data (such as temperature, voltage, current, link load, optical power, etc.) from the device through the Intelligent Platform Management Interface (IPMI), PCIe configuration space, and built-in sensors in the optical module.

[0122] Model Inference Layer: At its core is a Transformer-GNN (Graph Neural Network) hybrid model. It receives raw data and topology information from the data acquisition layer, and through fusion analysis, outputs a health score and specific fault prediction for each PCIe device in the system (such as predicting that a GPU will overheat in X minutes, or that an optical module will degrade too quickly).

[0123] Protocol control layer: Responsible for executing security isolation commands. It employs a two-way authentication hot-plug protocol based on the Chinese national cryptographic algorithm SM2 to ensure that the source of the isolation commands is trustworthy and has not been tampered with.

[0124] Hardware isolation layer: The layer where physical operations are ultimately performed. It includes TVS diodes for surge suppression and circuit protection; and dual-port random access memory (RAM) to buffer incomplete transactions (TLPs) before isolation, ensuring zero data loss.

[0125] It should be noted that, in Figure 3 In this architecture, data flows from bottom to top, while control flows from top to bottom. The model inference layer outputs decisions to the protocol control layer.

[0126] To better understand, the following provides a detailed explanation of each of the above layers:

[0127] (1) Dynamic topology-aware prediction model (model inference layer workflow):

[0128] It should be noted that its basic process is as follows: Figure 4 As shown, specifically:

[0129] Step 1: Input data preparation and feature construction;

[0130] Input 1: Device Node Features. Each PCIe device (GPU, switch, SSD, etc.) is modeled as a node in the graph. The feature vector of each node (e.g., ...) , The data originates from the data acquisition layer and includes its own multimodal time-series data, such as: [temperature, voltage, load rate, optical power, bias current...].

[0131] Input 2: System topology: Describes the physical connections between PCIe devices (e.g., which GPU is connected to which PCIe switch). This is a graph structure that defines the edges between nodes.

[0132] Step 2: Graph Neural Network (GNN) processing - spatial dimension information aggregation;

[0133] Objective: To enable each node to perceive not only its own state but also the state of its neighboring nodes, achieving precise fault location. For example, an elevated GPU temperature could be due to a faulty fan or poor heat dissipation caused by an overheating switch it's connected to. GNNs are used to distinguish between these two scenarios.

[0134] Process: A graph attention network (GAT) mechanism is used. This process consists of two steps:

[0135] Calculate the attention coefficient: for the target node It calculates its relationship with each of its neighboring nodes. Attention coefficient This coefficient represents the neighbors. For nodes The importance of.

[0136] ;

[0137] Purpose: This formula is the core calculation of GAT. It uses a shared weight matrix. and a learnable attention vector The paired, unnormalized attention scores are calculated, and then normalized using the softmax function to obtain the final weight coefficients. . For the target node All neighboring nodes.

[0138] Weighted aggregation: Using the calculated attention coefficients, the features of neighboring nodes are summed in a weighted manner to update the features of the target node.

[0139] ;

[0140] Function: This formula achieves spatial aggregation of information. The characteristics of each neighboring node are ranked according to their importance. After weighting, the data is aggregated to the target node and then processed by the activation function. Introducing nonlinear transformations, nodes are ultimately generated. New feature representation .

[0141] Output: The output of the GNN layer is This is a set of enhanced node feature representations that incorporate topological neighbor information. These new feature vectors will be fed into subsequent Transformer layers for time series analysis.

[0142] Step 3: Transformer Encoding - Temporal Dimension Pattern Capture;

[0143] Objective: Building upon the spatial topology information already fused in the GNN layer, this study aims to deeply analyze the temporal sequence patterns of each device node's state (whose features already include the influence of its neighbors), ultimately achieving the fusion of spatiotemporal information. For example, it could analyze whether the historical temperature trend of a GPU, after considering switch load, exhibits short-term fluctuations or a continuous upward trend.

[0144] Input: The enhanced node feature representation sequence output by the GNN layer. For each monitored device node in the system (e.g., a GPU), its features over N consecutive time steps are... , ..., , Arranged in chronological order, they form a spatiotemporal feature sequence X (with shape [N, D], where N is the time step and D is the feature dimension), which serves as the input to the Transformer encoder.

[0145] Process: The Transformer encoder consists of multiple stacked encoder layers with identical structures. Each encoder layer contains two core sub-layers and employs residual connections and layer normalization to stabilize and accelerate the training process. Its data processing flow is as follows:

[0146] 1. Multi-Head Self-Attention Mechanism:

[0147] Calculation: For an input sequence X, using three different sets of trainable weight matrices Perform a linear transformation to generate a query, key, and value matrix:

[0148] ;

[0149] Then, self-attention is calculated:

[0150] ;

[0151] Function: This mechanism allows any point in time in the sequence to directly focus on and absorb information from all other points in time, thereby effectively capturing long-term dependencies and anomalous mutation patterns (such as sudden temperature increases).

[0152] Output and First Fusion: The output of self-attention is processed through residual connections and layer normalization to generate an intermediate representation of the initial fused spatiotemporal information. : ;

[0153] This addition operation is crucial: it directly fuses the spatial information (X) provided by the GNN with the temporal information (Attention(Q,K,V)) calculated by Attention for the first time.

[0154] 2. Position-wise Feed-Forward Network (FFN): Calculates: The feed-forward neural network pairs... Each location (feature vector at each time point) undergoes an independent, identical nonlinear transformation, typically consisting of two linear transformations and an activation function:

[0155] ;in These are trainable parameters.

[0156] Purpose: To introduce nonlinear transformations into the model, thereby enhancing its expressive power and ability to fit complex functions.

[0157] Output and Final Fusion: The output of FFN is again processed through residual connections and layer normalization to generate the final output Z of the encoder layer. ;

[0158] This step describes the characteristics of the initial fusion. It has been further enhanced and refined.

[0159] Output: Final feature representation Z. This feature matrix (sequence) simultaneously integrates key information from two dimensions: spatial topological association (derived from GNN input X) and temporal state evolution (derived from the self-attention mechanism). It is the ultimate, condensed digital representation of the device's health status in this system. This output Z will be fed into subsequent fully connected layers for final health score regression calculation and fault type prediction.

[0160] Step 4: Prediction and Output;

[0161] Objective: This step is the final stage of model inference. Its purpose is to transform the abstract feature representation Z, which incorporates spatiotemporal information and is output by the Transformer layer, into a concrete and actionable device health assessment.

[0162] Input: Final feature representation Z: The feature matrix output by the Transformer encoder. For each monitored device in the system, Z is a highly condensed and abstract representation of its state, which already includes its own multimodal time series data, the influence of neighboring devices, and the historical trend of its state changes.

[0163] Process: This process is accomplished by a prediction head, which typically consists of one or more fully connected layers that perform two tasks in parallel: regression and classification.

[0164] 1. Health score calculation (regression task);

[0165] 1) Represent the feature Z (usually by taking the feature vector corresponding to the latest time step t). (or the aggregated vector) is input into a fully connected layer.

[0166] 2) This fully connected layer performs a linear transformation: (in and (These are trainable parameters).

[0167] 3) The result of the linear transformation is passed through the Sigmoid activation function to compress the output range to the range (0, 1):

[0168] ;

[0169] Function: Generates a continuous health score. This score is a scalar value between 0.0 and 1.0, which intuitively quantifies the health status of the device (e.g., 0.95 indicates very healthy, and 0.18 indicates a serious risk of failure).

[0170] 2. Fault type prediction (classification task);

[0171] 1) Input the same feature representation Z into another parallel fully connected layer.

[0172] 2) The number of output nodes of the fully connected layer is equal to the number of preset fault types (e.g., 4 types: normal, overheating, link degradation, hardware error).

[0173] 3) Pass the output results through the Softmax function to obtain the predicted probability of each fault type:

[0174] ;

[0175] Function: To determine the most likely type of equipment failure. The model selects the category with the highest probability as the final failure type prediction.

[0176] 3. Establishment of the evaluation mechanism (decision-making logic);

[0177] The evaluation mechanism is not a direct output of the model, but rather is established based on a combination of the model output (score and type) and preset decision rules and thresholds. These thresholds are either predefined by the system or set by domain experts. It should be noted that the preset decision rules and thresholds are shown in Table 1 below.

[0178] Table 1

[0179]

[0180] Mechanism Establishment: These rules and thresholds (e.g., 0.3, 0.9) are pre-configured in the system based on historical data, experimental verification, and domain knowledge. They map the model's predictions to specific operational instructions.

[0181] Output:

[0182] 1. Health Score: A continuous value between 0.0 and 1.0 that directly reflects the overall health of the device.

[0183] 2. Fault Type Prediction: A discrete category label indicating the most likely type of fault the device is prone to.

[0184] 3. (Implicit) Decision Instructions: Based on the above outputs and preset rules, the system automatically generates control instructions (such as "degrade", "switch", "isolate", "no operation").

[0185] (2) Two-way authentication hot-swap protocol (protocol control layer workflow): Traditional hot-swap protocols (such as PCIe SHPC) rely only on one-way authentication (system verifies the legitimacy of the device), which poses risks of device forgery and man-in-the-middle attacks. For example, malicious devices may bypass system detection by forging vendor IDs.

[0186] Objective: To ensure that isolation commands issued from the BMC are legitimate and secure, and to prevent malicious commands from triggering false isolation.

[0187] The interaction flow is as follows:

[0188] 1) The protocol process is as follows Figure 5 As shown:

[0189] 2) Process:

[0190] 1. [BMC->Device]: When the model inference layer determines that a device needs to be isolated, the BMC generates an isolation command message. The message format is:

[0191] typedef struct{

[0192] uint8_t opcode; / / opcode, for example, 0xB2 represents a safety isolation instruction;

[0193] uint8_t slot_addr; / / Target device slot address;

[0194] uint32_t timestamp; / / Current timestamp (millisecond precision);

[0195] uint8_t sm2_sig

[32] ; / / Use the BMC's private key to perform an SM2 signature on the aforementioned fields (opcode, addr, timestamp);

[0196] PCIe_SafeCmd;

[0197] 2. [Device]: After receiving the instruction, the device first verifies the timestamp (to prevent replay attacks), and then uses the BMC public key pre-installed in the device firmware (or, better yet, requests verification from the Certificate Authority (CA) certificate store (see the steps below)) to verify the legitimacy of the SM2 signature.

[0198] 3. [Device -> CA Certificate Store (Optional)]: The device can send a certificate verification request to the central CA certificate store. The request carries the BMC's certificate identifier and the received signature information. The CA store returns the verification result.

[0199] 4. [Device -> BMC]: If the verification is successful, the device will return an "acknowledgment" message and begin performing preparatory operations such as internal cache data synchronization.

[0200] 5. [BMC -> Hardware Level]: After receiving the "confirmation", the BMC sends the final "execute" instruction.

[0201] 6. Once the equipment completes isolation preparation, it triggers the hardware isolation layer to perform a power-off operation.

[0202] (3) Multi-parameter monitoring and health scoring of optical links:

[0203] For detailed workflow information, please refer to [link / document / document / etc.]. Figure 6 Specifically:

[0204] 1) Purpose: This section presents a specialized monitoring and evaluation solution designed for the unique fault characteristics of optical interconnect devices (such as CPOs and optical modules). Integrated into the system's data acquisition and model inference layers, it aims to achieve real-time, accurate, and interpretable prediction of the health status of optical links through multi-parameter fusion analysis, providing crucial information for subsequent dynamic isolation or link switching decisions.

[0205] 2) Monitoring Parameters: The system collects the core parameters shown in Table 2 below in real time through onboard sensors and compares them with preset safety thresholds:

[0206] Table 2

[0207]

[0208] 3) Health scoring algorithm and expert rules:

[0209] To enable rapid and interpretable assessment of optical link status, the system incorporates an expert rule-based health scoring algorithm:

[0210] ;

[0211] Output: The formula outputs an expert score, Score_optical, between 0.0 and 1.0, which serves as a powerful feature input to subsequent AI models.

[0212] The parameters are explained in Table 3 below:

[0213] Table 3

[0214]

[0215] 4) Decision-making logic and workflow:

[0216] This section employs a dual-drive mechanism that combines "independent threshold alarms" and "expert scoring fusion," and its workflow is as follows: Figure 7As shown.

[0217] To better understand, the following examples will be used to illustrate the point:

[0218] Example 1:

[0219] Scenario: In a multi-GPU server, the temperature of a GPU core reaches 95℃ (threshold 85℃), and the adjacent switch is at 92% load;

[0220] step:

[0221] 1. Data Acquisition and Preprocessing:

[0222] Temperature acquisition: Real-time temperature is obtained via the GPU's built-in thermistor (±0.5℃ accuracy). =95℃;

[0223] Link load monitoring: Statistics on switch buffer occupancy are obtained via the PCIe egress interface. =92%;

[0224] Topology feature extraction: Constructing a system topology graph, with GPUs and switches as nodes. GPU node feature vectors. Include[ , =87%, =91%,...]. Switch node feature vector Include[ =92%,...].

[0225] 2. Model inference process:

[0226] Input layer: Input the above feature vectors and topology graph into the hybrid model.

[0227] GNN processing:

[0228] Calculate the attention weights of the GPU node to the switch node. Due to the extremely high load on the switch (92%), .

[0229] GNN output: The key information of "neighbor switch overload" is incorporated into the features of the GPU node.

[0230] Transformer processing:

[0231] Analysis of the GPU temperature timing revealed that the temperature rose sharply from 70°C to 95°C within a short period of time (e.g., 10 seconds). (Very high), capturing abnormal mutation patterns.

[0232] Output: The model combines the two factors of "its own temperature soaring" and "poor heat dissipation caused by overload of neighboring switches" to calculate that the GPU's health score drops sharply to 0.2 (below the system's preset threshold of 0.5) and predicts that it may crash within 2 minutes due to overheating.

[0233] 3. Implementation of security isolation protocols:

[0234] Because the health score is 0.2 < 0.3 (redundancy switching threshold), the system triggers the security isolation protocol.

[0235] BMC generates and sends isolation instructions signed with the national cryptographic standard SM2 to the GPU.

[0236] After the GPU verifies the instruction, the driver-level cache data is synchronized to the dual-port RAM.

[0237] 4. Hardware isolation and its effects:

[0238] Once data synchronization is complete, the BMC sends an execution command.

[0239] Hardware isolation layer action: Transient voltage suppression diodes are activated to absorb surges; then the power management IC is controlled to cut off the GPU's 12V power supply.

[0240] Outcome: The faulty GPU was safely isolated. Because its cached data was saved, there was no data loss. The system log recorded the complete isolation event.

[0241] Example 2:

[0242] Scenario: Performance degradation of 800G CPO (co-packaged optics) optical modules in servers;

[0243] step:

[0244] 1. Data Acquisition:

[0245] Laser bias current: Hall sensor sampling =122mA (threshold 120mA);

[0246] Optical power feedback: Integrated photodiode measurement = 5.2dBm (initial value) = (2.5dBm), the optical power attenuation was calculated. =|-5.2-(-2.5)|=2.7dB;

[0247] Temperature slope: The instantaneous rate of change is calculated using historical data from the thermistor. =5.8℃ / s.

[0248] Independent security sentinel mechanism activated: monitoring logic determination If the current exceeds 120mA (exceeding the safety threshold), a high-priority alarm will be immediately triggered, and a protective frequency reduction command will be initiated to force the optical module to reduce its transmission power, preventing damage to the laser from overload. This operation will proceed in parallel with the subsequent scoring process.

[0249] 2. Health Score and Model Inference:

[0250] Expert scoring rules: Substitute the parameters into the formula:

[0251] ;

[0252] Multimodal fusion inference: The original parameters such as Score_optical, I_bias, ΔP, dT / dt, and their time-series data are input into the Transformer-GNN hybrid model along with the system topology information. After in-depth model analysis, the optical link specificity and global state are fully integrated, and a comprehensive health score of 0.18 is output.

[0253] 3. Decision-making and lossless link switching:

[0254] Decision logic: The overall health score of 0.18 is less than the system's preset switching threshold (0.3), so the system eventually triggers the redundant link switching process.

[0255] Protocol Interaction and Hardware Actions: The BMC and optical module execute a two-way authentication security protocol based on the Chinese national cryptographic standard SM2. After successful authentication, the BMC sends a command, and the internal control circuit of the optical module disables the currently aging main laser and enables the backup laser array. Subsequently, the BMC retrains the link width from x16 to x4 mode by writing to the CPU's PCIe configuration space (offset 0x1400) to adapt to the degraded link quality.

[0256] Data protection mechanism: During link switching and retraining, the PCIe FLIT protocol's retransmission mechanism ensures that all unacknowledged Transaction Layer Packets (TLPs) are retransmitted (maximum 5 retries) to prevent data loss. Dual-port RAM in the hardware isolation layer caches incomplete transactions prior to isolation, ensuring zero data loss.

[0257] Performance verification: From alarm to link recovery, the average time for fault switching is 32ms.

[0258] Ultimately, the link bandwidth was reduced by 75% due to the downgrade to x4 mode, but there was no service interruption or data loss, successfully avoiding server downtime caused by optical module failure.

[0259] It should be noted that the technical effects of the technical solution in this application are shown in Table 4 below:

[0260] Table 4

[0261]

[0262] It should be noted that the following technical effects are achieved: 1) A Transformer-GNN hybrid model is adopted, combining graph attention mechanism (GAT) and self-attention mechanism to jointly model the spatial topology relationship and temporal behavior of PCIe devices to achieve accurate fault prediction; 2) Based on the national cryptographic SM2 algorithm and timestamp, a two-way authentication protocol from BMC to the device is constructed to ensure the security of isolation commands and prevent replay and man-in-the-middle attacks; 3) Through the hardware co-design of "driver layer data synchronization + dual-port RAM caching + TVS surge suppression", data preservation and hardware protection are achieved before physical power failure; and health assessment and prediction are performed by combining multiple parameters such as laser bias current, optical power attenuation and temperature slope through expert rules and AI models.

[0263] It should be noted that the fault prediction algorithm in this application can be migrated to the baseband processing unit (BBU) and remote radio unit (RRU) of 5G base stations to monitor the status of optical modules and radio frequency links. It can also be adapted to the joint servo drive of industrial robots to predict bearing faults using vibration sensors and current sensors. Adjustments to monitoring parameters (such as vibration spectrum and current harmonics) and adaptation of the protocol layer to industrial buses (such as CAN and EtherCAT) are required.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0265] Embodiments of this application also provide a device for predicting equipment failures. Figure 8 This is a structural block diagram of a device for predicting equipment failure according to an embodiment of this application, such as... Figure 8 As shown, the device includes:

[0266] The acquisition module 802 is used to acquire a status data set, wherein the status data set includes the status data of M devices in the server at N consecutive time points, where M and N are both positive integers greater than or equal to 2;

[0267] The prediction module 804 is used to predict the fault of each of the M devices based on the state data set and the topological relationship of the M devices through the prediction model, and obtain M fault prediction data corresponding to the M devices. In the process of predicting the fault of the m-th device of the M devices, the prediction model will use the state data of the devices that have a topological connection relationship with the m-th device. m is an integer greater than or equal to 1 and less than or equal to M. The fault prediction data is used to predict whether the device has a fault and the type of fault.

[0268] The aforementioned device collects state data sets of M devices at multiple time points and, combined with the topological relationships of these M devices, enables a more comprehensive analysis of device health status. It not only captures dynamic changes in device status but also considers physical connections and logical dependencies between devices, effectively identifying indirect fault impacts caused by topological associations. When predicting the fault of any device (the m-th device) in the server, the model simultaneously considers the status of other connected devices, achieving accurate fault type identification and avoiding inaccurate predictions due to a single perspective. This improves the overall accuracy and reliability of fault prediction, solving the problem of low accuracy in fault prediction for devices in servers.

[0269] In an exemplary embodiment, the prediction module 804 is further configured to obtain fault prediction data corresponding to the m-th device among the M devices in the following manner, so as to obtain M fault prediction data: using the graph neural network of the prediction model, N target state data are obtained based on the state data of the m-th device at N consecutive time points and the state data of each of the P devices at N consecutive time points, wherein each of the P devices among the M devices has a topological connection relationship with the m-th device, and the n-th target state data among the N target state data is determined based on the state data of the m-th device at the n-th time point and the state data of each of the P devices at the n-th time point; using the encoder of the prediction model, a target feature representation is obtained based on the N target state data; using the fully connected layer of the prediction model, the fault prediction data corresponding to the m-th device is obtained based on the target feature representation.

[0270] In an exemplary embodiment, the prediction module 804 is further configured to obtain the nth target state data using the graph neural network of the prediction model, so as to obtain N target state data: determine the correlation coefficient between the mth device and each of the P devices based on the state data of the mth device at time n and the state data of each of the P devices at time n, to obtain P correlation coefficients, wherein the pth correlation coefficient among the P correlation coefficients is used to indicate the importance of the pth device to the mth device; and determine the nth target state data based on the P correlation coefficients and the state data of each of the P devices at time n.

[0271] In an exemplary embodiment, the prediction module 804 is further configured to calculate the p-th correlation coefficient to obtain P correlation coefficients by:

[0272] ;

[0273] in, Let p be the p-th correlation coefficient. This is the state data of the m-th device at time n. This represents the state data of the p-th device at time n. For P devices, This is the weight matrix. is a learnable attention vector; LeakyReLU is an activation function.

[0274] In an exemplary embodiment, the prediction module 804 is further configured to calculate the nth target state data in the following manner:

[0275] ;

[0276] in, For the nth target state data, Let P be the state data of the p-th device at time n. Let be the correlation coefficient between the m-th device and the p-th device out of P correlation coefficients. There are P devices.

[0277] In an exemplary embodiment, the prediction module 804 is further configured to arrange the N target state data in chronological order to obtain a spatiotemporal feature sequence, wherein the spatiotemporal feature sequence is [ , ..., , The target feature representation is obtained by using multiple coding layers of the encoder of the prediction model based on the spatiotemporal feature sequence. Each coding layer uses a multi-head self-attention mechanism and a feedforward neural network to process the input feature representation of the coding layer to obtain the output feature representation. The input feature representation of the first coding layer of the multiple coding layers is the spatiotemporal feature sequence, and the output feature representation of the last coding layer of the multiple coding layers is the target feature representation.

[0278] In an exemplary embodiment, the prediction module 804 is further configured to perform the following operation through the i-th coding layer among multiple coding layers to obtain a target feature representation through multiple coding layers of the encoder of the prediction model: the input feature representation of the i-th coding layer is processed through the multi-head self-attention mechanism of the i-th coding layer to obtain the intermediate feature representation of the i-th coding layer;

[0279] The intermediate feature representation of the i-th coding layer is processed by the feedforward neural network of the i-th coding layer to obtain the output feature representation of the i-th coding layer.

[0280] In an exemplary embodiment, the prediction module 804 is further configured to calculate the self-attention of the i-th coding layer using the following formula:

[0281] (Formula 1);

[0282] in, , This represents the input features of the i-th coding layer. For trainable weight matrix, Let k be the dimension of matrix k;

[0283] The intermediate feature representation of the i-th coding layer is obtained by performing residual connection and layer normalization on the self-attention of the i-th coding layer using the following formula 2:

[0284] (Formula 2);

[0285] in, This represents the intermediate feature representation of the i-th coding layer.

[0286] In an exemplary embodiment, the prediction module 804 is further configured to obtain the output of the feedforward neural network using the following formula three:

[0287] (Formula 3);

[0288] in, This represents the output of the feedforward neural network in the i-th coding layer. This represents the intermediate feature representation of the i-th coding layer. For trainable parameters, It is an activation function;

[0289] The output feature representation of the i-th coding layer is obtained by performing residual connection and layer normalization on the output of the feedforward neural network of the i-th coding layer using the following formula 4:

[0290] (Formula 4);

[0291] in, Let be the output feature representation of the i-th coding layer.

[0292] In an exemplary embodiment, the prediction module 804 is further configured to obtain a health score parameter of the m-th device based on the target feature representation through the first fully connected layer of the prediction model, wherein the health score parameter is used to reflect the health status of the m-th device; and to obtain the predicted failure probability of the m-th device on multiple failure types based on the target feature representation through the second fully connected layer of the prediction model; wherein the failure prediction data includes the health score parameter and the predicted failure probability on multiple failure types; the first fully connected layer and the second fully connected layer are two parallel fully connected layers of the prediction model.

[0293] In an exemplary embodiment, the apparatus further includes: a processing module, configured to, after obtaining M fault prediction data corresponding to M devices, and if it is determined that the m-th device needs to perform an isolation operation based on the fault prediction data corresponding to the m-th device, send an isolation command to the m-th device through the baseboard management controller of the server; and, upon receiving a determination command sent by the m-th device in response to the isolation command, send an execution command to the complex programmable logic device on the motherboard of the server through the baseboard management controller, wherein the execution command is used to instruct the complex programmable logic device to perform a power-off operation on the m-th device.

[0294] In an exemplary embodiment, the isolation instruction includes the certificate identifier of the baseboard management controller. The m-th device sends a verification request to the certificate authority based on the certificate identifier of the baseboard management controller to verify the legitimacy of the baseboard management controller, wherein the verification request carries the certificate identifier of the baseboard management controller. After sending the confirmation instruction, the m-th device performs an internal cache data synchronization operation. After the complex programmable logic device performs a power-off operation on the m-th device, the transient voltage suppression diode on the power input current of the m-th device is activated to absorb surges.

[0295] In an exemplary embodiment, the processing module is further configured to, after obtaining M fault prediction data corresponding to M devices, and determining, based on the fault prediction data corresponding to the m-th device, that the m-th device needs to perform a redundant link switching operation and that the m-th device is a PCIe device, write a target value to the link control register in the central processing unit of the server through the baseboard management controller of the server, so as to perform PCIe link training on the m-th device.

[0296] In an exemplary embodiment, the processing module is further configured to send a target instruction to the m-th device via a substrate management controller when the m-th device is an optical interconnect device, wherein the target instruction instructs the m-th device to disable the problematic laser and enable a backup laser array.

[0297] In an exemplary embodiment, the processing module is further configured to: acquire the laser bias current, optical power attenuation, and temperature slope of the optical interconnect device when there is an optical interconnect device among the M devices; perform isolation operation on the optical interconnect device when the laser bias current of the optical interconnect device is greater than or equal to a preset current value; and determine the optical link health score of the optical interconnect device based on the optical power attenuation and temperature slope when the laser bias current of the optical interconnect device is less than the preset current value, wherein the status data of the optical interconnect device at a certain moment includes the optical link health score.

[0298] In one exemplary embodiment, the processing module is further configured to determine the optical link health score of the optical interconnect device by:

[0299] ;

[0300] in, Assess the health of the optical link. For optical power attenuation, This is the initial value of the optical power. This represents the temperature slope.

[0301] For a description of the features in the embodiment corresponding to the equipment failure prediction device, please refer to the relevant description of the embodiment corresponding to the equipment failure prediction method, which will not be repeated here.

[0302] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the device failure prediction method.

[0303] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the device fault prediction method when it is run.

[0304] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0305] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the device fault prediction method.

[0306] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the device fault prediction method.

[0307] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0308] The foregoing has provided a detailed description of the equipment failure prediction method and apparatus, electronic device, storage medium, and computer program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for predicting equipment failure, characterized in that, include: Obtain a set of state data, wherein the set of state data includes state data of M devices in the server at N consecutive time points, where M and N are both positive integers greater than or equal to 2; The prediction model performs fault prediction on each of the M devices based on the state data set and the topological relationship of the M devices, resulting in M ​​fault prediction data for the M devices. In the process of predicting the fault of the m-th device among the M devices, the prediction model uses the state data of the devices that have a topological connection with the m-th device, where m is an integer greater than or equal to 1 and less than or equal to M. The m-th fault prediction data is used to predict whether the m-th device has a fault and the type of fault if it does. Specifically, a prediction model is used to predict the faults of each of the M devices based on the state data set and the topological relationships of the M devices, resulting in M ​​fault prediction data corresponding to the M devices, including: The fault prediction data corresponding to the m-th device among the M devices is obtained in the following way to obtain the M fault prediction data: The graph neural network of the prediction model generates N target state data based on the state data of the m-th device at N consecutive time points and the state data of each of the P devices at N consecutive time points. Each of the P devices in the M devices has a topological connection with the m-th device. The n-th target state data in the N target state data is determined based on the state data of the m-th device at the n-th time point and the state data of each of the P devices at the n-th time point. The encoder of the prediction model obtains the target feature representation based on the N target state data; The fault prediction data corresponding to the m-th device is obtained through the fully connected layer of the prediction model based on the target feature representation.

2. The equipment failure prediction method according to claim 1, characterized in that, Using the graph neural network of the prediction model, N target state data are obtained based on the state data of the m-th device at N consecutive time points and the state data of each of the P devices at N consecutive time points, including: The nth target state data is obtained through the graph neural network of the prediction model in the following manner, in order to obtain the N target state data: Based on the state data of the m-th device at time n and the state data of each of the P devices at time n, the correlation coefficient between the m-th device and each of the P devices is determined, resulting in P correlation coefficients. Among the P correlation coefficients, the p-th correlation coefficient is used to indicate the importance of the p-th device to the m-th device. The nth target state data is determined based on the P correlation coefficients and the state data of each of the P devices at the nth time.

3. The equipment failure prediction method according to claim 2, characterized in that, Based on the state data of the m-th device at time n and the state data of each of the P devices at time n, the correlation coefficient between the m-th device and each of the P devices is determined, resulting in P correlation coefficients, including: The p-th correlation coefficient is calculated in the following way to obtain the P correlation coefficients: ； in, Let p be the p-th correlation coefficient. This is the state data of the m-th device at time n. The state data of the p-th device at time n. For the P devices, This is the weight matrix. is a learnable attention vector; LeakyReLU is an activation function.

4. The equipment failure prediction method according to claim 2, characterized in that, The nth target state data is determined based on the P correlation coefficients and the state data of each of the P devices at time n, including: The nth target state data is calculated in the following manner: ； in, For the nth target state data, Let P be the state data of the p-th device at time n. Let be the correlation coefficient between the m-th device and the p-th device among the P correlation coefficients. For the P devices, This is the activation function.

5. The equipment failure prediction method according to claim 1, characterized in that, The encoder of the prediction model obtains a target feature representation based on the N target state data, including: Arrange the N target state data in chronological order to obtain a spatiotemporal feature sequence, wherein the spatiotemporal feature sequence is [ , ..., , ]; The target feature representation is obtained from the spatiotemporal feature sequence through multiple coding layers of the encoder of the prediction model. Each coding layer uses a multi-head self-attention mechanism and a feedforward neural network to process the input feature representation of the coding layer to obtain the output feature representation. The input feature representation of the first coding layer of the multiple coding layers is the spatiotemporal feature sequence, and the output feature representation of the last coding layer of the multiple coding layers is the target feature representation.

6. The equipment failure prediction method according to claim 5, characterized in that, The target feature representation is obtained from the spatiotemporal feature sequence through multiple encoding layers of the encoder of the prediction model, including: The following operations are performed through the i-th coding layer of the plurality of coding layers to obtain the target feature representation through the plurality of coding layers of the encoder of the prediction model: The input feature representation of the i-th coding layer is processed by the multi-head self-attention mechanism of the i-th coding layer to obtain the intermediate feature representation of the i-th coding layer; The intermediate feature representation of the i-th coding layer is processed by the feedforward neural network of the i-th coding layer to obtain the output feature representation of the i-th coding layer.

7. The equipment failure prediction method according to claim 6, characterized in that, The input feature representation of the i-th coding layer is processed through the multi-head self-attention mechanism of the i-th coding layer to obtain the intermediate feature representation of the i-th coding layer, including: The self-attention of the i-th coding layer is calculated using the following formula: (Formula 1); in, , This represents the input feature representation of the i-th coding layer. For trainable weight matrix, Let k be the dimension of matrix k; The intermediate feature representation of the i-th coding layer is obtained by performing residual connection and layer normalization on the self-attention of the i-th coding layer using the following formula 2: (Formula 2); in, This represents the intermediate feature representation of the i-th coding layer.

8. The equipment failure prediction method according to claim 6, characterized in that, The intermediate feature representation of the i-th coding layer is processed by the feedforward neural network of the i-th coding layer to obtain the output feature representation of the i-th coding layer, including: The output of the feedforward neural network is obtained using the following formula three: (Formula 3); in, This is the output of the feedforward neural network in the i-th coding layer. This represents the intermediate feature representation of the i-th coding layer. For trainable parameters, It is an activation function; The output feature representation of the i-th coding layer is obtained by performing residual connection and layer normalization on the output of the feedforward neural network of the i-th coding layer using the following formula: (Formula 4); in, This represents the output feature representation of the i-th coding layer.

9. The equipment failure prediction method according to claim 1, characterized in that, Through the fully connected layer of the prediction model, fault prediction data corresponding to the m-th device is obtained based on the target feature representation, including: The health score parameter of the m-th device is obtained through the first fully connected layer of the prediction model based on the target feature representation, wherein the health score parameter is used to reflect the health status of the m-th device; The predicted failure probability of the m-th device under multiple failure types is obtained through the second fully connected layer of the prediction model based on the target feature representation. The fault prediction data includes the health score parameters and the predicted fault probabilities for the various fault types; the first fully connected layer and the second fully connected layer are two parallel fully connected layers of the prediction model.

10. The equipment failure prediction method according to claim 1, characterized in that, After obtaining the M fault prediction data corresponding to the M devices, the method further includes: If it is determined that the m-th device needs to be isolated based on the fault prediction data corresponding to the m-th device, an isolation command is sent to the m-th device through the baseboard management controller of the server. Upon receiving a confirmation instruction sent by the m-th device in response to the isolation instruction, the baseboard management controller sends an execution instruction to the complex programmable logic device on the motherboard of the server. The execution instruction is used to instruct the complex programmable logic device to perform a power-off operation on the m-th device.

11. The equipment failure prediction method according to claim 10, characterized in that, The isolation command includes the certificate identifier of the baseboard management controller. The m-th device sends a verification request to the certificate authority based on the certificate identifier of the baseboard management controller to verify the legitimacy of the baseboard management controller. The verification request carries the certificate identifier of the baseboard management controller. After sending the determination instruction, the m-th device performs an internal cache data synchronization operation; After the complex programmable logic device performs a power-off operation on the m-th device, the transient voltage suppression diode on the power input current of the m-th device is activated to absorb the surge.

12. The equipment failure prediction method according to claim 1, characterized in that, After obtaining the M fault prediction data corresponding to the M devices, the method further includes: If, based on the fault prediction data corresponding to the m-th device, it is determined that the m-th device needs to perform a redundant link switching operation, and the m-th device is a PCIe device, the target value is written to the link control register in the central processing unit of the server through the baseboard management controller of the server, so as to perform PCIe link training on the m-th device.

13. The equipment failure prediction method according to claim 12, characterized in that, The method further includes: In the case that the m-th device is an optical interconnect device, a target instruction is sent to the m-th device through the substrate management controller, wherein the target instruction instructs the m-th device to disable the problematic laser and enable the backup laser array.

14. The equipment failure prediction method according to claim 1, characterized in that, The method further includes: If there are optical interconnect devices among the M devices, obtain the laser bias current, optical power attenuation, and temperature slope of the optical interconnect devices; When the laser bias current of the optical interconnect device is greater than or equal to a preset current value, an isolation operation is performed on the optical interconnect device. When the laser bias current of the optical interconnect device is less than the preset current value, the optical link health score of the optical interconnect device is determined based on the optical power attenuation and temperature slope, wherein the status data of the optical interconnect device at a certain moment includes the optical link health score.

15. The equipment failure prediction method according to claim 14, characterized in that, The optical link health score of the optical interconnect device is determined based on the optical power attenuation and temperature slope, including: The optical link health score of the optical interconnect device is determined by the following method: ； in, Assess the health of the optical link. For the optical power attenuation, This is the initial value of the optical power. The temperature slope is given.

16. A device for predicting equipment failures, characterized in that, include: The acquisition module is used to acquire a status data set, wherein the status data set includes status data of M devices in the server at N consecutive time points, where M and N are both positive integers greater than or equal to 2; The prediction module is used to predict the fault of each of the M devices based on the state data set and the topological relationship of the M devices using a prediction model, thereby obtaining M fault prediction data corresponding to the M devices. In the process of predicting the fault of the m-th device of the M devices, the prediction model uses the state data of the devices that have a topological connection with the m-th device, where m is an integer greater than or equal to 1 and less than or equal to M. The fault prediction data is used to predict whether the device has a fault and the type of fault if it exists. The prediction module is further configured to obtain fault prediction data for the m-th device among M devices in the following manner, to obtain M fault prediction data: Through the graph neural network of the prediction model, N target state data are obtained based on the state data of the m-th device at N consecutive time points and the state data of each of the P devices at N consecutive time points, wherein each of the P devices among the M devices has a topological connection relationship with the m-th device, and the n-th target state data among the N target state data is determined based on the state data of the m-th device at time n and the state data of each of the P devices at time n; through the encoder of the prediction model, a target feature representation is obtained based on the N target state data; through the fully connected layer of the prediction model, the fault prediction data corresponding to the m-th device is obtained based on the target feature representation.

17. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the device failure prediction method as described in any one of claims 1 to 15 when executing the computer program.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the device fault prediction method as claimed in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the equipment fault prediction method according to any one of claims 1 to 15.

Images