Decentralized HPC computing cluster management method and system based on paxos algorithm

Abstract

The invention discloses a decentralized HPC computing cluster management method and system based on a paxos algorithm. The method comprises the steps of deploying a main management node and a plurality of standby management nodes, and setting a cluster management election mechanism, wherein the cluster management election mechanism comprises the steps that a reply of heartbeat connection sent by amain management node exceeds a preset value, and a standby management node carries out election according to the paxos algorithm to generate a new main management node; and enabling the original mainmanagement node to be offline, and enabling the new main management node to perform heartbeat monitoring on the remaining standby management nodes. According to the invention, the HPC high-performance job scheduling cluster can be optimized from a single-master centralized cluster mode to a decentralized cluster mode; the change of the mode enables the availability of the cluster to be greatly improved; limit of a single-point fault of the single-master cluster centralization mode is avoided; the fault-tolerant capability of the cluster is improved by several orders of magnitudes; the faultis more suitable for an actual scene; automatic high availability is provided for the cluster; and high availability is completed without a third-party tool.

Description

technical field [0001] The present invention relates to the technical field of computer data processing, specifically, a decentralized HPC computing cluster management method and system based on paxos algorithm. Background technique [0002] With the country's vigorous promotion of informatization reform, China's supercomputer construction is also among the best in the world, and there are more and more national-level supercomputing centers. This has higher and higher requirements for software such as job scheduling systems and cluster monitoring systems running on supercomputers. The HPC software product architecture used when the scale was originally small cannot adapt to larger-scale scheduling and computing resource monitoring, resulting in hardware It does not match the software system, thus affecting the actual computing performance of the entire computing cluster at the software level. The current HPC cluster product software is basically a master-slave cluster archi...

AI Technical Summary

Problems solved by technology

In the working mode of a single-master cluster, all jobs can only be submitted and scheduled through the main management node. When the cluster size is small, multiple jobs are queued to ease the job scheduling pressure. When the supercomputer scale is large enough, computing power is no longer the bottleneck. , the scheduling and availability of the main management node will become a new bottleneck, especially when many small jobs are submitted with high concurrency, the job scheduling pressure will increase exponentially; the same is true for cluster computing resource monitoring, and the pressure of collecting data is transferred to the management Node processing, it is difficult to achieve ultra-large-scale high-concurrency monitoring business scenarios

Since the existing high-performance computing cluster is a job scheduling system with a single management node, the job scheduler cannot achieve load balancing; the scalability is poor, or it does not support expansion, and management nodes cannot be added at will

Images