Fault prediction and self-repairing method and device, electronic equipment and storage medium

Abstract

The invention discloses a fault prediction and self-repairing method and device, electronic equipment and a storage medium, and relates to the technical field of computers. According to the method, the function of dynamically monitoring various data of processor hardware for the target node can be realized, the dynamically monitored hardware state data is input, the fault probability is predicted by a method of weighting and combining key indexes through a bidirectional long-short-term memory model and an attention mechanism, the possible faults are intelligently predicted, and the fault prediction efficiency is improved. And when the target node has a test fault in advance, the target node is repaired in a gradual load reduction mode, so that the effects of real-time monitoring, accurate prediction and rapid self-regulation are achieved. Manual intervention is reduced, the intelligent decision-making capability is achieved, operation and maintenance automation and intelligentization are achieved, resource self-adaptive repairing can be integrated, the self-adaptive capability is improved, and the complex scene fault sensing capability is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault prediction and self-repair method, device, electronic device, and storage medium. Background Art

[0002] In traditional GPU testing, fault detection relies on manual inspections or threshold alarms, which cannot provide early warnings in the early stages of hardware anomalies, resulting in delayed fault response. Traditional fault detection methods rely on static threshold judgments, but GPUs experience drastic temperature fluctuations under transient high loads, resulting in a high false alarm rate. Therefore, fault detection with traditional technologies has a certain delay. In the current manual-dominated environment, test interruption costs are high and recovery efficiency is low. After a stress test is interrupted due to a GPU failure, manual intervention is required to restart the task, which extends the test cycle. Existing technologies only predict the specific type of fault and do not perform self-repair on the predicted faults or adaptive repair methods. They only avoid faults through periodic node replacement, but do not combine real-time hardware status dynamic adjustment strategies. This leads to low resource utilization, lack of adaptive repair, and insufficient fault perception capabilities in complex scenarios. Summary of the Invention

[0003] The present application provides a fault prediction and self-repair method, device, electronic device and storage medium to at least solve the problems in related technologies such as the lag in fault warning, the failure to predict risks in advance, only the prediction of specific fault types, the lack of self-repair of predicted faults and the lack of adaptive repair methods, insufficient repair rigidity and cross-architecture adaptability, the lack of adaptive recovery mechanism capabilities, and the inability to integrate resources for adaptive repair.

[0004] This application provides a fault prediction and self-repair method, including:

[0005] In response to the target node testing the processor, collecting hardware status data of the processor;

[0006] The hardware status data is sampled according to sampling time steps in a sliding window manner to form a plurality of sampling samples, the plurality of sampling samples are constructed into a time series data unit, the sampling samples in the time series data unit are processed using a bidirectional long short-term memory model to capture the front-end and back-end dependencies to form complete sequence data, and the failure rate of each fault type is obtained based on the complete sequence data;

[0007] Based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type. The failure probability is predicted based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type.

[0008] In response to the failure probability being greater than or equal to a first threshold, the target node is determined to have a test failure, and the target node is repaired by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

[0009] The present application also provides a fault prediction and self-repair device, comprising:

[0010] A dynamic monitoring module, configured to collect hardware status data of the processor in response to a target node testing the processor;

[0011] a data processing module, configured to sample the hardware status data according to sampling time steps in a sliding window manner to form a plurality of sampling samples, construct the plurality of sampling samples into a time series data unit, process the sampling samples in the time series data unit using a bidirectional long short-term memory model to capture the previous and next dependencies, thereby forming complete sequence data, and obtain the failure rate of each fault type based on the complete sequence data;

[0012] The fault prediction module is used to identify the key indicator features corresponding to each fault type based on the failure rate of each fault type using an attention mechanism, and predict the failure probability based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type;

[0013] The self-repair control module is used to determine that a test failure has occurred in the target node in response to the failure probability being greater than or equal to a first threshold, and repair the target node by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

[0014] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned fault prediction and self-repair methods when executing the computer program:

[0015] In response to the target node testing the processor, collecting hardware status data of the processor;

[0016] The hardware status data is sampled according to sampling time steps in a sliding window manner to form a plurality of sampling samples, the plurality of sampling samples are constructed into a time series data unit, the sampling samples in the time series data unit are processed using a bidirectional long short-term memory model to capture the front-end and back-end dependencies to form complete sequence data, and the failure rate of each fault type is obtained based on the complete sequence data;

[0017] Based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type. The failure probability is predicted based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type.

[0018] In response to the failure probability being greater than or equal to a first threshold, the target node is determined to have a test failure, and the target node is repaired by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

[0019] The present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned fault prediction and self-repair methods are implemented:

[0020] In response to the target node testing the processor, collecting hardware status data of the processor;

[0021] The hardware status data is sampled according to sampling time steps in a sliding window manner to form a plurality of sampling samples, the plurality of sampling samples are constructed into a time series data unit, the sampling samples in the time series data unit are processed using a bidirectional long short-term memory model to capture the front-end and back-end dependencies to form complete sequence data, and the failure rate of each fault type is obtained based on the complete sequence data;

[0022] Based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type. The failure probability is predicted based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type.

[0023] In response to the failure probability being greater than or equal to a first threshold, the target node is determined to have a test failure, and the target node is repaired by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

[0024] This application provides a fault prediction and self-repair method, device, electronic device and storage medium, which can realize the function of dynamically monitoring various data of processor hardware of target nodes, input dynamically monitored hardware status data, identify key indicator features corresponding to each fault type through a bidirectional long short-term memory model and an attention mechanism, predict the fault probability by weighting the key indicator features, make intelligent predictions of possible faults, and repair the target node through a gradual load reduction method when a test fault occurs in the target node, so as to achieve the effects of real-time monitoring, accurate prediction and rapid self-regulation. Human intervention is reduced, with intelligent decision-making capabilities, operation and maintenance automation and intelligence are realized, and resources can be integrated for adaptive repair, improving adaptive capabilities and enhancing the ability to perceive complex scene faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 This is an application environment diagram of the fault prediction and self-repair method in one embodiment of the present application;

[0027] Figure 2 This is a flowchart of the dynamic monitoring module dynamically acquiring GPU information data integration in one embodiment of the present application;

[0028] Figure 3 This is a training flow chart of the data processing module and the fault prediction module in one embodiment of the present application;

[0029] Figure 4 This is a self-repair decision tree flow chart of the self-repair control module in one embodiment of the present application;

[0030] Figure 5 This is a flowchart of a fault prediction and self-repair method in one embodiment of the present application;

[0031] Figure 6 This is a structural block diagram of a fault prediction and self-repair device in one embodiment of the present application;

[0032] Figure 7 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0035] In traditional GPU testing, fault detection relies on manual inspections or threshold alarms, failing to provide early warnings of hardware anomalies, such as increased memory error rates or unusual temperature fluctuations. This results in delayed fault response. Furthermore, traditional methods rely on static thresholds (e.g., alarms for temperatures > 90°C). However, GPU temperatures fluctuate dramatically under transient high loads (e.g., GEMM calculations), resulting in high false alarm rates. Therefore, traditional fault detection techniques exhibit a certain delay. In the current manual-based environment, test interruption costs are high and recovery efficiency is low. After a stress test is interrupted by a GPU failure, manual intervention is required to restart the task, extending the test cycle. This is particularly true in multi-node distributed scenarios. In multi-node NCCL testing, a single GPU failure can terminate the entire task, requiring manual topology reconfiguration, with an average recovery time exceeding two hours. Existing technologies avoid failures through periodic node replacement, but fail to incorporate dynamic adjustment strategies for real-time hardware status, resulting in low resource utilization and a lack of adaptive repair. Furthermore, existing technologies only support static resource allocation and cannot dynamically adjust the memory and computing power ratio based on failure risk.

[0036] Existing technologies reduce fault handling time from hours to minutes through periodic detection and backup node scheduling. However, these technologies rely on fixed detection cycles (e.g., 10 minutes) and are unable to respond to sudden faults in real time. Multi-model fusion (LSTM+GBDT) is used to predict failure rates, achieving a 25% improvement in accuracy compared to traditional threshold methods. However, this technology only outputs the probability of failure and is not linked to repair actions, lacking an end-to-end self-healing closed loop. Integration with the Kubernetes ecosystem supports cluster-level fault isolation and automated recovery, completing the fault closed loop within 5 minutes. However, self-healing actions are limited to node restarts and offline, preventing pre-failure intervention.

[0037] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0038] The fault prediction and self-repair method provided in this application can be applied to Figure 1 In the application environment shown, a test device 102 is connected to a server to be tested 104, which is equipped with a processor to be tested, such as a GPU. The test device 102 is built around three core modules: dynamic monitoring, intelligent prediction, and elastic repair. The test device 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server to be tested 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0039] The testing device 102 includes: a dynamic monitoring module, a data processing module, a fault prediction module and a self-repair control module.

[0040] like Figure 2 As shown, the dynamic monitoring module collects multi-dimensional hardware status data in real time, covering explicit metrics such as temperature and power consumption, as well as implicit metrics such as memory ECC error rate and PCIe link quality. Dynamic monitoring of multi-dimensional hardware status data is primarily implemented at the hardware, driver, and container layers. Taking the GPU as an example, the hardware layer uses the NVIDIA DCGMAPI to collect GPU core temperature (grouped by SM), memory bank-level ECC error counts (statistics by bank, recording single- and double-bit errors), and PCIe link retransmission rate (calculating the proportion of retransmitted packets, with a threshold >3% triggering an alert). At the driver layer, an eBPF program is embedded to monitor NVIDIA kernel driver events, providing real-time monitoring of kernel driver function call exceptions and CUDA context switch failures. Using the Linux kernel's cgroups feature, real-time GPU memory usage and CUDA core utilization are obtained for processes within Docker containers. Finally, the dynamically monitored data is integrated.

[0041] like Figure 3 As shown in the figure, the dynamic monitoring module collects GPU metrics every second, including temperature (T), video memory ECC error rate (E), PCIe retransmission rate (P), driver latency (D), and power consumption (W), totaling five features. The data processing module uses a time series construction and annotation method, taking the GPU information detected by the dynamic detection module as input data and constructing 60-second time series data units using a sliding window approach (e.g., sampling once per second, each time series data unit contains 60 time steps). A bidirectional long short-term memory (LSTM) model is used to capture forward and backward dependencies: the time series data of each sample is flipped and reversed along the time axis to form a reverse input sequence, and the forward and reverse outputs are spliced ​​together to return the complete sequence.

[0042] like Figure 3 As shown in the figure, the fault prediction module combines the attention mechanism to weight key indicators (for example, the weight of temperature mutation is increased by 30%). The combination of the two predicts the failure probability P (0% to 100%) and the fault type classification (abnormal heat dissipation, memory damage, PCIe link failure).

[0043] like Figure 4 As shown, the self-repair control module dynamically implements a multi-level repair strategy based on the failure probability P to minimize the impact of test interruption. A progressive load reduction strategy is employed. If the failure probability exceeds a threshold, the progressive load reduction strategy is triggered, and a snapshot migration switch test is performed. The first threshold is preferably 70%, the second threshold is preferably 85%, and the third threshold is preferably 95%. The values ​​of the first, second, and third thresholds can be set based on actual circumstances.

[0044] Phase 1 (P ≥ 70%): Reduce the CUDA core utilization of the current test task, limit the CUDA core frequency to 80% of the nominal value, and reduce power consumption and temperature;

[0045] Phase 2 (P ≥ 85%): Migrate tasks to spare GPU nodes in the same cluster;

[0046] Stage 3 (P ≥ 95%): Isolate the faulty GPU and initiate hardware diagnostics.

[0047] The checkpoint / restore CRIU in user space saves the state of the containerized test process and continues execution from the snapshot point after failure recovery to avoid repeated testing.

[0048] This embodiment dynamically monitors various GPU data. By inputting this data, the system intelligently predicts potential failures and issues corresponding warnings through a combination of an LSTM model and an attention mechanism. When a failure occurs, a gradual load reduction strategy is implemented to self-repair the GPU. After the repair, training can resume from the point of interruption through snapshot migration. This achieves real-time monitoring, accurate prediction, and rapid self-regulation. This reduces manual intervention, provides intelligent decision-making capabilities, and achieves automated and intelligent operations and maintenance.

[0049] like Figure 5 As shown, an embodiment of the present application provides a fault prediction and self-repair method, comprising the following steps:

[0050] Step S1, in response to a target node testing a processor, collecting hardware status data of the processor and integrating the hardware status data;

[0051] Step S2: Using a sliding window, the hardware status data is sampled according to the sampling time step to form multiple sampling samples, and the multiple sampling samples are constructed into a time series data unit. The sampling samples in the time series data unit are processed using a bidirectional long short-term memory model to capture the previous and next dependencies to form complete sequence data. The failure rate of each fault type is obtained based on the complete sequence data.

[0052] Step S3: Based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type, and the failure probability is predicted based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type.

[0053] Step S4: In response to the failure probability being greater than or equal to the first threshold, the target node is determined to have a test failure, and the target node is repaired by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

[0054] This embodiment can dynamically monitor various processor hardware data for target nodes, input dynamically monitored hardware status data, and predict fault probabilities by combining a bidirectional long-short-term memory model and an attention mechanism to weight key indicators. This allows for intelligent prediction of possible faults, and when a test fault occurs at a target node, the target node is repaired through a progressive load reduction approach, achieving real-time monitoring, accurate prediction, and rapid self-regulation. This reduces manual intervention, provides intelligent decision-making capabilities, and achieves automated and intelligent operation and maintenance. It can integrate resources for adaptive repair, improve self-adaptation, and enhance fault perception capabilities in complex scenarios.

[0055] This application uses the weight of the key indicator feature corresponding to each fault type as the weight of each fault type. This may result in deviations, so the weight of the key indicator feature corresponding to each fault type needs to be weighted. For example, the weight ratio of the key indicator feature corresponding to each fault type needs to be adjusted. For example, when the key indicator is a temperature change, the weight ratio is increased by 30%.

[0056] like Figure 2 As shown, in this embodiment, in response to the target node testing the processor, collecting the hardware status data of the processor and integrating the hardware status data include:

[0057] The processor core temperature is collected once per second from the hardware layer through the application programming interface (DCGMAPI), and single-bit or double-bit error counts are recorded by block to form an interface log;

[0058] Real-time monitoring of kernel driver function call exceptions and UCA context switch failures at the driver layer to generate kernel driver events.

[0059] Obtain the processor memory occupancy rate and unified computing device architecture core occupancy rate of the container process in real time from the container layer as container performance indicators;

[0060] Interface logs, kernel driver events, and container performance indicators are stored in the data lake, and the data in the data lake is distributed. The preprocessing engine is used to integrate the hardware status data and save it in the feature database.

[0061] Among them, DCGMAPI and eBPF programs are used to realize real-time synchronous collection of data at the hardware layer (temperature, video memory ECC errors), driver layer (NVIDIA kernel timeout events), and container layer (cgroups resource usage), solving the data island problem in traditional solutions.

[0062] like Figure 3 As shown, in this embodiment, the hardware status data is sampled according to the sampling time step in a sliding window manner to form multiple sampling samples, and the multiple sampling samples are constructed into a time series data unit including:

[0063] Adopting the time series construction and annotation method, the detected hardware status data is used as input data, and the duration of the sliding window and the sampling time step are set;

[0064] The hardware status data is sampled one by one according to the sampling time step of the sliding window to form sampling samples;

[0065] The number of sample samples in the sliding window is obtained by dividing the duration by the sampling time step, and the sample samples are formed into time series data units in chronological order.

[0066] Among them, a 60-second time series data unit is constructed in a sliding window manner, such as sampling once per second, and each time series data unit contains 60 time steps.

[0067] like Figure 3 As shown, in this embodiment, a bidirectional long short-term memory model is used to process the sampled samples in the time series data unit to capture the previous and next dependencies, forming a complete sequence data including:

[0068] Obtain the time series data of each sample in the time series data unit;

[0069] Flip the time series data of each sample in the time series data unit along the time axis to form a reverse input sequence;

[0070] The time series data of the sampled samples is concatenated with the reverse input sequence for forward and reverse output, and the complete sequence data is returned.

[0071] The forward and backward dependencies are captured through a bidirectional long-short-term memory model, and complete sequence data is formed based on the forward and reverse output splicing method. The hardware status data features can be highlighted in the complete sequence data, thereby improving the accuracy of predicting the failure rate of each fault type, making intelligent predictions of possible faults, and issuing corresponding warnings.

[0072] The bidirectional long short-term memory model consists of a forward LSTM model and a reverse LSTM model. The forward LSTM model processes the original time series (Past → Future), capturing the causal chain from historical data to the current state. The reverse LSTM model processes the flipped time series (Future → Past), capturing the inverse relationship between potential future failures and the current state. Forward and reverse output concatenation combines the outputs of the forward and reverse LSTM models in the feature dimension to form enhanced time series features.

[0073] like Figure 3 As shown, in this embodiment, based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type. The failure probability is predicted based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type, including:

[0074] An attention mechanism is used to identify the failure rate of each fault type in the complete sequence data. Key indicator features corresponding to each fault type are identified, including temperature fluctuations in interface logs, memory error rates in kernel driver events, and link retransmission rates in container performance indicators.

[0075] The attention weights of key indicator features are calculated, and the weight increase ratio of the key indicators corresponding to each fault type is set. The temperature mutation weight, memory error rate weight, and link retransmission rate weight are obtained, and the fault probability is predicted using a weighted summation method.

[0076] The fault type classification includes abnormal heat dissipation, video memory damage, and PCIe link failure. Temperature mutation is the preferred key indicator, with an improvement of 30%. Subsequent weighted feature fusion can highlight the impact of temperature mutation. This embodiment does not rely entirely on temperature mutation for judgment, but instead makes a comprehensive judgment based on multiple factors such as temperature mutation, video memory error rate, and link retransmission rate, avoiding false alarms caused by drastic temperature fluctuations of the GPU under transient high load.

[0077] The method of calculating the attention weight of the complete sequence data is to use the attention mechanism to identify key indicators. This method is different from the weighted summation method. The key indicators of this application are identified and can be adjusted with different architectures to improve the adaptability of cross-architectures and complex scenarios. It can self-evolve to predict faults and intelligently integrate historical data for accurate prediction.

[0078] It is understandable that in other embodiments, based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type, and the failure probability is predicted according to the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type, including:

[0079] Obtain the processor hardware state factor in the complete sequence data, and use the attention mechanism to perform attention scoring on the processor hardware state factor;

[0080] According to the attention score of the processor hardware state factor, the processor hardware state factor is divided into key indicators, important indicators and related indicators, among which the attention scores of key indicators, important indicators and related indicators decrease in turn;

[0081] The weight values ​​of key indicators, important indicators and related indicators are set for the corresponding target nodes, where the weight values ​​of key indicators, important indicators and related indicators are positively correlated with their corresponding attention scores;

[0082] Calculate the key indicators, important indicators and related indicators of the hardware status data of the target node in a weighted summation manner to obtain a weighted summation result;

[0083] The failure probability is predicted based on the weighted summation result, and the failure type corresponding to the key indicator with the largest weight value is obtained.

[0084] Fault probability prediction is implemented using global feature aggregation, compressing the time series dimension through global average pooling. A probability output layer uses a Sigmoid activation function to output a fault probability between 0% and 100%. A multi-task classification layer uses a Softmax function to output the probability of a fault type (e.g., abnormal cooling, damaged video memory, PCIe link failure).

[0085] This embodiment predicts the fault probability by combining a bidirectional long-short-term memory model and an attention mechanism to weight key indicators, makes intelligent predictions of possible faults, and repairs the target node by progressively reducing load when a test fault occurs at the target node, thereby achieving real-time monitoring, accurate prediction, and rapid self-regulation.

[0086] In this embodiment, in response to the failure probability being greater than or equal to the first threshold, determining that a test failure has occurred in the target node, and repairing the target node by progressive load reduction includes:

[0087] comparing the probability of failure to a first threshold;

[0088] In response to the failure probability being greater than or equal to the first threshold, executing a progressive load reduction strategy on the target node to reduce the core occupancy of the unified computing device architecture of the current test task, limit the core frequency of the unified computing device architecture, and reduce power consumption and temperature;

[0089] In response to the target node executing the progressive load reduction strategy, it is determined whether the fault of the target node is successfully repaired. If so, a repair log is recorded; otherwise, hardware-level isolation is triggered.

[0090] The method also includes:

[0091] In response to the failure probability being greater than or equal to the first threshold, comparing the failure probability with a second threshold;

[0092] In response to the failure probability being greater than or equal to a second threshold, migrating the task of the target node to a standby node in the same cluster;

[0093] In response to the failure probability being greater than or equal to the second threshold, comparing the failure probability with a third threshold;

[0094] In response to the failure probability being greater than or equal to a third threshold, the processor is isolated and a hardware diagnostic program is started to repair the failure of the target node.

[0095] Among them, the first threshold is preferably 70%, the second threshold is preferably 85%, and the third threshold is preferably 95%. The values ​​of the first threshold, the second threshold, and the third threshold can be set based on the actual situation. This application implements different fault probabilities and selects corresponding control strategies to control the target nodes, implements different specific control methods for different situations, improves the fault perception capability in complex scenarios, and can integrate resources for adaptive repair. Through the dynamic threshold trigger mechanism (such as P ≥ 70% triggers frequency reduction, P ≥ 95% triggers isolation), the repair strategy is accurately matched with the fault risk.

[0096] When the failure probability is greater than or equal to the first threshold, the fault is determined to have occurred. A gradual load reduction strategy is then implemented to self-repair the GPU. After the repair, training can be resumed from the breakpoint through snapshot migration. This achieves real-time monitoring, accurate prediction, and rapid self-regulation.

[0097] In this embodiment, migrating the task of the target node to the standby node in the same cluster includes:

[0098] Generate a snapshot file for the container on the target node to save the state of the containerized test process, and migrate the snapshot file to the standby node in the same cluster;

[0099] The standby node loads the snapshot file and continues testing the processor from the breakpoint.

[0100] In this embodiment, isolating the processor and starting a hardware diagnostic program to repair the target node includes:

[0101] Generate a snapshot file for the container of the target node to save the status of the containerized test process, and start the hardware diagnostic program to repair the fault of the target node;

[0102] After the target node recovers, the target node continues testing the processor from the breakpoint.

[0103] By dynamically monitoring GPU data and inputting the prediction model, faults can be predicted. After the fault, the system can self-repair and return to the power-off state to continue training and testing. Multiple processes are combined to achieve automated and intelligent operation and maintenance.

[0104] The following example uses the failure recovery of GPU node A in NCCL multi-machine training as an example.

[0105] Monitoring: The memory error rate of GPU node A increased from 50 times / minute to 200 times / minute (lasting for 3 minutes); the PCIe retransmission rate exceeded 5% (threshold: 3%).

[0106] Prediction: The LSTM model outputs a failure probability P = 88% (video memory failure class); the system marks node A as "high risk."

[0107] Repair: The self-healing controller notified the NCCL master node to suspend the task; a CRIU snapshot was taken of the Docker container on node A and migrated to node B; node B loaded the snapshot and resumed training from the breakpoint. The total interruption time was 2 minutes and 15 seconds.

[0108] The above-mentioned fault prediction and self-repair method can dynamically monitor various processor hardware data at the target node. By inputting the dynamically monitored hardware status data, the fault probability is predicted by combining a bidirectional long-short-term memory model and an attention mechanism to weight key indicators. This method intelligently predicts possible faults and, when a test fault occurs at the target node, repairs the target node through a gradual load reduction, achieving real-time monitoring, accurate prediction, and rapid self-regulation. This method reduces manual intervention, possesses intelligent decision-making capabilities, and achieves automated and intelligent operation and maintenance. It can integrate resources for adaptive repair, improve self-adaptation, and enhance fault perception capabilities in complex scenarios.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0110] In one embodiment, Figure 6 As shown, a fault prediction and self-repairing device 10 is provided, comprising: a dynamic monitoring module 1, a data processing module 2, a fault prediction module 3, and a self-repairing control module 4.

[0111] The dynamic monitoring module 1 is used for collecting hardware status data of the processor in response to the target node testing the processor, and integrating the hardware status data.

[0112] The data processing module 2 is used to sample the hardware status data according to the sampling time step in a sliding window manner to form multiple sampling samples, construct the multiple sampling samples into a time series data unit, and use a bidirectional long short-term memory model to process the sampling samples in the time series data unit to capture the previous and next dependencies to form complete sequence data.

[0113] The fault prediction module 3 is used to identify the key indicator features corresponding to each fault type based on the failure rate of each fault type using the attention mechanism, and predict the failure probability according to the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type.

[0114] The self-repair control module 4 is used to determine that a test failure has occurred in the target node in response to the failure probability being greater than or equal to the first threshold, and repair the target node by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

[0115] In this embodiment, in response to the target node testing the processor, collecting the hardware status data of the processor and integrating the hardware status data includes:

[0116] The processor core temperature is collected once per second from the hardware layer through the application programming interface, and single-bit or double-bit error counts are recorded by block to form an interface log;

[0117] Real-time monitoring of kernel driver function call exceptions and UCA context switch failures at the driver layer to generate kernel driver events.

[0118] Obtain the processor memory occupancy rate and unified computing device architecture core occupancy rate of the container process in real time from the container layer as container performance indicators;

[0119] Interface logs, kernel driver events, and container performance indicators are stored in the data lake, and the data in the data lake is distributed. The preprocessing engine is used to integrate the hardware status data and save it in the feature database.

[0120] In this embodiment, the hardware status data is sampled according to the sampling time step in a sliding window manner to form a plurality of sample samples, and the plurality of sample samples are constructed into a time series data unit, including:

[0121] Adopting the time series construction and annotation method, the detected hardware status data is used as input data, and the duration of the sliding window and the sampling time step are set;

[0122] The hardware status data is sampled one by one according to the sampling time step of the sliding window to form sampling samples;

[0123] The number of sample samples in the sliding window is obtained by dividing the duration by the sampling time step, and the sample samples are formed into time series data units in chronological order.

[0124] In this embodiment, a bidirectional long short-term memory model is used to process the sampled samples in the time series data unit to capture the previous and next dependencies, forming a complete sequence data including:

[0125] Obtain the time series data of each sample in the time series data unit;

[0126] Flip the time series data of each sample in the time series data unit along the time axis to form a reverse input sequence;

[0127] The time series data of the sampled samples is concatenated with the reverse input sequence for forward and reverse output, and the complete sequence data is returned.

[0128] In this embodiment, the attention mechanism is used to identify the key indicator features corresponding to each fault type based on the failure rate of each fault type. The failure probability is predicted based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type, including:

[0129] An attention mechanism is used to identify the failure rate of each fault type in the complete sequence data. Key indicator features corresponding to each fault type are identified, including temperature fluctuations in interface logs, memory error rates in kernel driver events, and link retransmission rates in container performance indicators.

[0130] The attention weights of key indicator features are calculated, and the weight increase ratio of the key indicators corresponding to each fault type is set. The temperature mutation weight, memory error rate weight, and link retransmission rate weight are obtained, and the fault probability is predicted using a weighted summation method.

[0131] In this embodiment, in response to the failure probability being greater than or equal to the first threshold, determining that a test failure has occurred in the target node, and repairing the target node by progressive load reduction includes:

[0132] comparing the probability of failure to a first threshold;

[0133] In response to the failure probability being greater than or equal to the first threshold, executing a progressive load reduction strategy on the target node to reduce the core occupancy of the unified computing device architecture of the current test task, limit the core frequency of the unified computing device architecture, and reduce power consumption and temperature;

[0134] In response to the target node executing the progressive load reduction strategy, it is determined whether the fault of the target node is successfully repaired. If so, a repair log is recorded; otherwise, hardware-level isolation is triggered.

[0135] In this embodiment, the self-repair control module 4 is further configured to:

[0136] In response to the failure probability being greater than or equal to the first threshold, comparing the failure probability with a second threshold;

[0137] In response to the failure probability being greater than or equal to a second threshold, migrating the task of the target node to a standby node in the same cluster;

[0138] In response to the failure probability being greater than or equal to the second threshold, comparing the failure probability with a third threshold;

[0139] In response to the failure probability being greater than or equal to a third threshold, the processor is isolated and a hardware diagnostic program is started to repair the failure of the target node.

[0140] In this embodiment, migrating the task of the target node to the standby node in the same cluster includes:

[0141] Generate a snapshot file for the container on the target node to save the state of the containerized test process, and migrate the snapshot file to the standby node in the same cluster;

[0142] The standby node loads the snapshot file and continues testing the processor from the breakpoint.

[0143] In this embodiment, isolating the processor and starting a hardware diagnostic program to repair the target node includes:

[0144] Generate a snapshot file for the container of the target node to save the status of the containerized test process, and start the hardware diagnostic program to repair the fault of the target node;

[0145] After the target node recovers, the target node continues testing the processor from the breakpoint.

[0146] The aforementioned fault prediction and self-repair device can dynamically monitor various processor hardware data at the target node. It inputs dynamically monitored hardware status data and predicts the probability of failure by combining a bidirectional long-short-term memory model and an attention mechanism to weight key indicators. This intelligently predicts possible faults and, when a test fault occurs at the target node, repairs the target node through progressive load reduction, achieving real-time monitoring, accurate prediction, and rapid self-regulation. This reduces manual intervention, provides intelligent decision-making capabilities, and achieves automated and intelligent operation and maintenance. It can integrate resources for adaptive repair, improve self-adaptation, and enhance fault perception in complex scenarios.

[0147] For the description of the features in the embodiments corresponding to the fault prediction and self-repair device, please refer to the relevant description of the embodiments corresponding to the fault prediction and self-repair method, which will not be repeated here.

[0148] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned fault prediction and self-repair method embodiments.

[0149] In one embodiment, the electronic device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the electronic device is used to store fault prediction and self-repair data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a fault prediction and self-repair method is implemented.

[0150] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned fault prediction and self-repair method embodiments when running.

[0151] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0152] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned fault prediction and self-repair method embodiments are implemented.

[0153] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned fault prediction and self-repair method embodiments.

[0154] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] The above is a detailed introduction to a fault prediction and self-repair method, device, electronic device and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A fault prediction and self-repair method, characterized in that: include: In response to the target node testing the processor, collecting hardware status data of the processor; The hardware status data is sampled according to sampling time steps in a sliding window manner to form a plurality of sampling samples, the plurality of sampling samples are constructed into a time series data unit, the sampling samples in the time series data unit are processed using a bidirectional long short-term memory model to capture the front-end and back-end dependencies to form complete sequence data, and the failure rate of each fault type is obtained based on the complete sequence data; Based on the failure rate of each fault type, an attention mechanism is used to identify the key indicator features corresponding to each fault type, and the failure probability is predicted according to the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type; in response to the failure probability being greater than or equal to a first threshold, it is determined that the target node has a pre-test failure, and the target node is repaired by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

2. The fault prediction and self-repair method according to claim 1, characterized in that: When the processor is tested in response to the target node, collecting the hardware status data of the processor includes: The processor core temperature is collected once per second from the hardware layer through the application programming interface, and single-bit or double-bit error counts are recorded by block to form an interface log; Real-time monitoring of kernel driver function call exceptions and UCA context switch failures at the driver layer to generate kernel driver events. Obtain the processor memory occupancy rate and unified computing device architecture core occupancy rate of the container process in real time from the container layer as container performance indicators; The interface log, the kernel driver event, and the container performance indicator are stored in a data lake, the data in the data lake is distributedly stored, and a preprocessing engine is used to integrate the hardware status data and save it in a feature database.

3. The fault prediction and self-repair method according to claim 1, characterized in that: The step of sampling the hardware status data in a sliding window manner according to sampling time steps to form a plurality of sampling samples, and constructing the plurality of sampling samples into a time series data unit comprises: Adopting the time series construction and annotation method, the detected hardware status data is used as input data, and the duration of the sliding window and the sampling time step are set; Sampling the hardware status data one by one according to the sampling time step of the sliding window to form sampling samples; The number of sampled samples in the sliding window is obtained by dividing the duration by the sampling time step, and the sampled samples are formed into a time series data unit in chronological order.

4. The fault prediction and self-repair method according to claim 2, characterized in that: The process of using a bidirectional long short-term memory model to process the sampled samples in the time series data unit to capture the previous and next dependencies to form complete sequence data includes: obtaining the time series data of each sampled sample in the time series data unit; Flipping and reversing the time series data of each sample in the time series data unit along the time axis to form a reverse input sequence; The time series data of the sampled samples is spliced ​​with the reverse input sequence for forward and reverse output, and the complete sequence data is returned.

5. The fault prediction and self-repair method according to claim 2, characterized in that: The method of using an attention mechanism to identify key indicator features corresponding to each fault type based on the failure rate of each fault type and predicting the failure probability based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type includes: An attention mechanism is used to identify the failure rate of each fault type on the complete sequence data, and key indicator features corresponding to each fault type are identified. The key indicator features include temperature mutations in interface logs, memory error rates in kernel driver events, and link retransmission rates in container performance indicators. The attention weights of the key indicator features are calculated, the weight increase ratio of the key indicators corresponding to each fault type is set, the temperature mutation weight, memory error rate weight and link retransmission rate weight are obtained, and the fault probability is predicted using a weighted summation method.

6. The fault prediction and self-repair method according to claim 1, characterized in that: In response to the failure probability being greater than or equal to a first threshold, determining that a test failure has occurred on the target node, and repairing the target node by progressive load reduction includes: comparing the failure probability with a first threshold; In response to the failure probability being greater than or equal to the first threshold, executing a progressive load reduction strategy on the target node to reduce the core occupancy of the unified computing device architecture of the current test task, limit the core frequency of the unified computing device architecture, and reduce power consumption and temperature; In response to the target node executing the progressive load reduction strategy, it is determined whether the fault of the target node is repaired successfully. If so, a repair log is recorded; otherwise, hardware-level isolation is triggered.

7. The fault prediction and self-repair method according to claim 6, characterized in that: The method further comprises: In response to the failure probability being greater than or equal to the first threshold, comparing the failure probability with a second threshold; In response to the failure probability being greater than or equal to the second threshold, migrating the task of the target node to a standby node in the same cluster; In response to the failure probability being greater than or equal to the second threshold, comparing the failure probability with a third threshold; In response to the failure probability being greater than or equal to the third threshold, isolating the processor and starting a hardware diagnostic program to repair the failure of the target node; Migrating the target node's tasks to a standby node in the same cluster includes: Generate a snapshot file for the container of the target node to save the state of the containerized test process, and migrate the snapshot file to a standby node in the same cluster; The standby node loads the snapshot file, and the standby node continues to test the processor from the breakpoint; Wherein, isolating the processor and starting a hardware diagnostic program to repair the target node includes: Generating a snapshot file for the container of the target node to save the state of the containerized test process, and starting a hardware diagnostic program to repair the fault of the target node; After the target node recovers, the target node continues to test the processor from the breakpoint.

8. A fault prediction and self-repair device, characterized in that: include: A dynamic monitoring module, configured to collect hardware status data of the processor in response to a target node testing the processor; a data processing module, configured to sample the hardware status data according to sampling time steps in a sliding window manner to form a plurality of sampling samples, construct the plurality of sampling samples into a time series data unit, process the sampling samples in the time series data unit using a bidirectional long short-term memory model to capture the previous and next dependencies, thereby forming complete sequence data, and obtain the failure rate of each fault type based on the complete sequence data; The fault prediction module is used to identify the key indicator features corresponding to each fault type based on the failure rate of each fault type using an attention mechanism, and predict the failure probability based on the weight of the key indicator features corresponding to each fault type and the failure rate of each fault type; The self-repair control module is used to determine that a test failure has occurred in the target node in response to the failure probability being greater than or equal to a first threshold, and repair the target node by progressive load reduction; in response to the failure probability being less than the first threshold, the target node continues to test the processor.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the fault prediction and self-repair method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the fault prediction and self-repair method according to any one of claims 1 to 7 are implemented.

Images