YARN based resource management method, device and system

Abstract

The invention provides a YARN based resource management method, device and system; a work message carries an application program and application program configuration written through an expansion calculating framework, wherein the application program configuration comprises an application program label and each subtask needed resource container description, the resource container description comprises resource types and resource quantity, and in the resource container description of the subtask need speeding up, the resource types comprise a FPGA resource type; the method comprises the following steps: distributing calculating resource for AM according to all NM idle resource container description and resource labels in a cluster, and using resource labels to mark FPGA needed by each subtask for speeding in the application program; the expanded cluster resource represents that a model is only related to calculating framework types supported by the cluster; when the FPGA is added into the YARN, the method and system can solve the YARN resource dimension dynamic expansion problems caused by calculating resource specificity, thus enhancing YARN resource management flexibility.

Description

technical field [0001] Embodiments of the present invention relate to communication technologies, and in particular to a resource management method, device and system based on YARN. Background technique [0002] A Field Programmable Gate Array (Field Programmable Gate Array, referred to as: FPGA) is a highly parallel, low power consumption, programmable hardware device, and its hardware computing performance is better than that of a general-purpose central processing unit (Central Processing Unit, referred to as: CPU). Using CPU and FPGA to build heterogeneous multi-core systems, using CPU to provide general-purpose computing capabilities, and using FPGA chips as dedicated hardware accelerators have become an important development trend in the field of high-performance computing. [0003] Another resource coordinator (YetAnotherResourceNegotiator, referred to as: YARN), as Hadoop 2.0 native resource scheduling system, can manage and allocate resources for various computing f...

AI Technical Summary

Problems solved by technology

In addition, the cluster resource representation dimension is too high will also affect the scheduling efficiency of the resource scheduler

Images