Hadoop job scheduling method based on genetic algorithm

Abstract

The invention discloses a Hadoop job scheduling method based on a genetic algorithm. The Hadoop job scheduling method comprises the following steps: firstly, pre-processing work to generate an encoding and decoding table; secondly, generating initial scheduling tables of a plurality of executing work, and carrying out fitness detection sorting on the initial scheduling tables to obtain a scheduling table list; finally, carrying out genetic operation on the initial scheduling tables in the scheduling table list to form a final scheduling table list; taking the scheduling table ranked in the most front of the final scheduling table list as an optimal scheduling table; distributing tasks of different work to corresponding TaskTracker for execution according to the optimal scheduling table, so as to finish a Hadoop job scheduling task. According to the scheduling method, resources in a platform do not need to be pre-set before jobs are scheduled; dynamic acquisition, counting and distribution are carried out in a scheduling process and the burden of an administrator is alleviated; furthermore, the total finishing time of the work and the average finishing time of the work can be controlled by the scheduling method, so that the fairness of executing the work is guaranteed and the executing efficiency can also be ensured.

Description

technical field [0001] The invention belongs to the field of information technology and relates to a Hadoop job scheduling method based on a genetic algorithm. Background technique [0002] Apache Hadoop is an open source distributed platform, mainly composed of two core projects, MapReduce and HDFS. MapReduce is the core computing framework of Hadoop. It is a software framework with a master-slave structure, which is divided into two roles: JobTracker and TaskTracker. The JobTracker node forms task fragments (splits) through the preprocessing of the job data Job, and then distributes them to each TaskTracker node to ensure the parallelism of tasks, and then decomposes each fragment in the Map stage and summarizes in the Reduce stage, and finally outputs the processing The results are saved; HDFS is the storage cornerstone for Hadoop to realize distributed computing. It is a highly fault-tolerant system suitable for deployment on inexpensive machines. HDFS is also a frame...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a Hadoop job scheduling method based on a genetic algorithm, which solves the problems in the prior art that cluster resources need to be preconfigured before job scheduling, and that the fairness of the job and the execution efficiency of the job cannot be taken into account. technical problem

Images