Resource scheduling method under Hadoop-based multi-job environment

Abstract

The invention discloses a resource scheduling method under a Hadoop-based multi-job environment, which includes: (1) collecting the three-party monitoring information of cluster loads, a Hadoop platform and hardware in real time; (2) collecting the job execution monitoring information of a user on each computing node of a cluster in real time; (3) gathering the three-party monitoring data of the cluster, modeling to evaluate the computing capabilities of the nodes, and dividing the nodes of the cluster into superior computing nodes and inferior computing nodes; (4) if the nodes are the superior computing nodes, then starting a job task resource demand allocation policy based on similarity evaluation; (5) if the nodes are the inferior computing nodes, then returning to a default resource demand allocation policy of the Yarn. The resource scheduling method under the Hadoop-based multi-job environment solves the problem of resource fragments caused by oversize job resource demand division granularity in conventional resource schedulers of the Yarn, can comprehensively take the heterogeneity of cluster nodes and jobs into consideration, and increases the execution concurrency of the cluster by reasonably and effectively allocating the node resources, thus increasing the execution efficiency of the multiple jobs of the Hadoop cluster.

Description

technical field [0001] The invention belongs to the technical field of big data, and more specifically relates to a resource scheduling method based on Hadoop multi-job environment. Background technique [0002] With the advent of the era of big data and the Internet, big data technology has become a research hotspot in academia and industry, and Hadoop, as an open source big data processing platform, has been widely used in both enterprises and academic research fields. However, the first generation of Hadoop has practical problems such as single point of failure, low resource utilization, and inability to support multiple computing frameworks. In order to overcome the above shortcomings, Apache launched the second generation of Hadoop, and built the resource management module into an independent general resource management system Yarn, which is responsible for the resource allocation and task scheduling of the cluster. Yarn enables multiple computing frameworks (MapReduce...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the defects of the existing resource scheduling technology, the purpose of the present invention is to provide a resource scheduling method that can dynamically adjust resource requirements according to the heterogeneity of cluster nodes, job execution monitoring status and resource usage, aiming at solving the problems of existing resource scheduling technologies. Poor cluster resource utilization and poor system performance caused by policies

Images