Large-scale mixed heterogeneous storage system-oriented node fault prediction system and method

Abstract

The invention provides a large-scale mixed heterogeneous storage system-oriented node fault prediction system and method. A time sequence-based association rule mining algorithm is adopted to construct a node fault prediction system architecture, and a main process of node fault prediction includes: collecting state data and log information of each storage node; carrying out data preprocessing, and generating sequence modes on the basis of a sliding window; using the sequence modes and fault sequences, which are extracted in a fault identification process, together as input of an association rule algorithm, and outputting output results as typical fault sequences; carrying out matching on the typical fault sequences and sequence modes generated in real time; and if a matching result meetsan established rule, issuing early warning to notify a system administrator, and giving feedback to a prediction result by the administrator according to a subjective interest degree. According to thesystem and method, real-time online fault prediction is carried out for nodes of a large-scale mixed heterogeneous storage system, and accuracy and recall which are better than those of existing fault prediction algorithms and better scalability can be obtained.

Description

technical field [0001] The invention relates to the field of storage system reliability and availability, in particular to the fault prediction of large-scale hybrid heterogeneous storage system nodes. Background technique [0002] Building a big data storage software and hardware system based on NVM can better meet the needs of big data storage in terms of capacity, performance and power consumption. However, even with relatively reliable individual components, the sheer number of components can lead to an increased system failure rate. The economic losses caused by storage system failures cannot be underestimated. Node failure prediction and elimination and data pre-migration are important technologies to ensure system reliability and availability. A good failure prediction algorithm can greatly reduce system maintenance costs. The performance bottleneck of traditional disk-based storage systems still lies in the storage medium, and fault prediction research only focuses ...

AI Technical Summary

Problems solved by technology

The former establishes the corresponding state transition diagram of the occurrence of the event and analyzes the occurrence of the fault, but the state transition requires high expert domain knowledge; the latter predicts the fault through the probability relationship between the event sequence and the occurrence of the fault, and the accuracy of the algorithm is relatively high and the time complexity is relatively high. Low and interpretable, it is favored by many researchers, but the disadvantage is that: due to the need for certain professional knowledge, high requirements for data sets, and limited application scenarios, the accuracy of prediction, query The full rate and scalability still need to be improved so that it can meet the needs of node failure prediction in large-scale hybrid heterogeneous storage systems

Images