Method, system and device for operating and maintaining large-scale cluster and storage medium

Abstract

The invention provides a method, a system and equipment for operating and maintaining a large-scale cluster and a storage medium. The method comprises the following steps: deploying a management module and a client module at a management node, and deploying the client module at a computing node; the management module obtains various monitoring values of all management nodes and computing nodes from the client module, and compares the monitoring values with a currently configured monitoring strategy threshold; in response to meeting the alarm condition, generating alarm data at the management node, and performing task processing strategy matching according to the alarm data; and in response to successful matching, generating a fault processing task at the management node, and transmitting the fault processing task to the corresponding client module based on active pulling of the client module. According to the invention, the operation execution pressure is shared by the management node to each computing node; and the management node is only responsible for fault judgment and task state management and does not need to undertake maintenance work of task distribution execution, so that operation and maintenance work of a super-large-scale cluster can be adapted.

Description

technical field [0001] The present invention relates to the field of large-scale clusters, and more specifically refers to a method, system, device and storage medium for operating and maintaining large-scale clusters. Background technique [0002] At present, artificial intelligence technology has developed rapidly, and various industries are rapidly undergoing intelligent transformation. As the deep learning technology represented by artificial intelligence, various fields have produced a large number of constantly changing and rapidly developing needs for deep learning training. Larger training scale and larger data set size will significantly improve the effect of deep learning training. However, large-scale clusters undoubtedly pose higher challenges to the original operation and maintenance system capabilities. [0003] The systems in the industry can generally realize the monitoring and alarm functions of small-scale clusters, but cannot support automatic fault hand...

AI Technical Summary

Problems solved by technology

The operation and maintenance system used by the deep learning platform in the industry is generally unable to adapt to the problem of large-scale clusters

Images