A node failure prediction method for large-scale cluster systems

Abstract

The invention discloses a node failure prediction method for a large-scale cluster system, which collects resource occupancy data of each node and generates a data set, uses a long and short-term memory network to build a first data prediction model, and uses a random forest to build a second failure prediction Model, establish the first observation window, judge the size of the first observation window, if it does not meet the set value, return to rebuild; if it meets the set value, use the first fault prediction model to predict the data in the advance time window, The window and the data in the advance time window are combined to form the second observation window, and the size of the second observation window is judged. If the set value is not satisfied, return to rebuild the second observation window; if it is satisfied, use the second fault prediction model to predict the prediction window. failure. On the premise of ensuring sufficient advance time to deal with the node failure, the present invention makes the accuracy rate of the prediction model the highest.

Description

technical field [0001] The invention belongs to the technical field of computer system reliability and availability, and in particular relates to a node failure prediction method for a large-scale cluster system. Background technique [0002] Cluster systems are common platforms used in high-performance computing, cloud computing, and data centers. As these platforms continue to grow in size and complexity, system reliability becomes a major concern, as the system's mean time between failures (MTBF) decreases as the number of system components increases. Recent research results show that the reliability of existing data center and cloud computing systems is limited by a mean time between failures of 10-100 hours. A data center usually has a high failure rate because it has many servers and components. Additionally, long-running applications and intensive workloads are common in these facilities. The performance of the system depends on the availability of the machines, wh...

AI Technical Summary

Problems solved by technology

Although predicting the next failure of a machine seems to be a feasible and promising solution to improve the reliability of data centers, it brings two main challenges: the first challenge is that high accuracy is required when predicting, especially To reduce false positives

The second challenge is how to choose the appropriate lead time

If the lead time is too long, the salient features before the failure cannot be fully utilized, resulting in low model accuracy; if the lead time is too short, although the prediction accuracy will increase, it is not enough for the administrator to have enough time to correlate the nodes operation to avoid failure

Images