Task scheduling method and related device

Abstract

The invention provides a task scheduling method and a related device. The method is applied to a task scheduling system, the task scheduling system comprises a task scheduling platform and a task distribution assembly, and the method comprises the following steps: obtaining a task flow submitted by a user; splitting the task flow through the task scheduling platform to obtain a plurality of tasks,determining submission triggering conditions of the plurality of tasks, and generating a submitted task queue according to the submission triggering conditions of the plurality of tasks; submitting the tasks in the submitted task queue to the task distribution component; and distributing the tasks in the submitted task queue to a target distributed cluster through the task distribution componentto obtain a task running result. According to the technical scheme of the embodiment of the invention, the task establishment process is simplified, and the task scheduling efficiency is improved.

Description

Technical field [0001] This application relates to the field of computer technology, and in particular to a task scheduling method and related devices. Background technique [0002] Oozie is an open source framework based on a workflow engine, closely integrated with the Hadoop ecosystem, and can provide task scheduling and coordination for Hadoop MapReduce and Pig Job. [0003] However, the workflow in Oozie needs to use xml files to define the execution order. The definition is too complicated. When users need to establish multiple associated tasks, they need to write the execution order of the tasks into the xml file, and then use Oozie to submit this task, Oozie This task flow will be scheduled. In this way, in the case of big data, a large number of xml file configurations are required when creating tasks. The configuration process is complicated. And since all configuration work is performed in Oozie, Oozie’s work pressure is relatively high and task scheduling is low effici...

AI Technical Summary

Problems solved by technology

[0003] However, the workflow in Oozie needs to use xml files to define the execution sequence, which is too complicated. When users need to create multiple associated tasks, they need to write the execution sequence of the tasks into the xml file, and then use Oozie to submit the task. Oozie This task flow will be scheduled. In this way, in the case of big data, a large number of xml file configurations are required when creating a task. The configuration process is complicated, and since all configuration work is performed in Oozie, Oozie's work pressure is high, and task scheduling is difficult. low efficiency

Images