Node Manager state management method and device in cluster and computing equipment

Abstract

The embodiment of the invention relates to the technical field of distributed resource management and scheduling systems, and discloses a NodeManager state management method and device in a cluster and computing equipment. The method comprises steps of the network load information of a cluster being collected, and a hardware state of the cluster being evaluated according to the network load information; determining the health state of the nodes in the cluster according to an evaluation result; and when the state of the node is unhealthy, carrying out offline operation on the NodeManager. The method is advantaged in that pre-judgment and automatic offline before a NodeManager fault are achieved, stable operation of the system is guaranteed, and meanwhile the situation that when a node hostis preempted by a plurality of application programs, Container distribution fails, and consequently tasks fail is avoided.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of distributed resource management and scheduling systems, and in particular to a method, device and computing device for managing NodeManager states in a cluster. Background technique [0002] With the development of computer technology, various data-intensive application computing frameworks are emerging, such as MpaReduce, Spark, S4, Storm, etc. When adopting computing frameworks, factors such as resource utilization, operation and maintenance costs, and data sharing are generally considered. Applicators generally hope to deploy all these computing frameworks in a common cluster so that they can share cluster resources and control resources for unified use. In this way, a unified resource management and scheduling platform was born, and its typical representative is YARN (Yet Another Resource Negotiator, another resource coordinator). [0003] YARN is divided into ResourceManager (...

AI Technical Summary

Problems solved by technology

After NodeManager accepts the task, it can allocate Application Master and Container. When the host resources are not exclusive to YARN, it may cause ResourceManager resource application failure.

[0004] In the existing technology, YARN resource allocation only uses CPU and memory as computing resources, which are divided in advance in the form of yarn-site. make judgments to allocate resources

In addition, the Impala of the MPP architecture will also be deployed on the host of the Hadoop cluster, but the resource allocation is not managed according to YARN. When the MPP aggregation query is executed, a large amount of data will be accumulated in the memory. If you apply with the CPU, the allocation of the Container will fail and the task will fail.

Instant query takes up a lot of memory, but the usage time is relatively short

If all reserved, it will cause waste of YARN

Therefore, this method cannot adapt to the situation where multiple applications preempt the node host

Images