A task scheduling management system and method for hadoop big data platform

Abstract

The invention discloses a task scheduling management system and method of a hadoop big data platform. The system includes an authority module, a task retrieval module, a task dependency module, a scheduling retrieval module, a task compliance inspection module, a scheduling module, an alarm module, and a task status retrieval module. module; the present invention provides a visual system for visual task scheduling dependency management, and task blood relationship viewing and management, which is convenient for users to trace problems and configure task dependencies; no manual configuration is required for users The running time of the task, the scheduled task will automatically start running the subsequent task according to the running end time of the pre-dependent task; when the task has an operation error, it will retry the task according to the number of retries configured by the user, and provide an alarm Mechanism to remind users who depend on the current running failure; task dependent ring automatic retrieval function can effectively avoid the problem of task running endless loop.

Description

technical field [0001] The invention relates to the technical field of management systems, in particular to a task scheduling management system and method of a hadoop big data platform. Background technique [0002] With the advent of the era of big data, Hadoop open source technology is widely used. In actual business scenarios, there are more and more computing tasks based on the Hadoop platform. Visual dependency management, task dependencies are not clear, and task dependencies rely on manual maintenance, the maintenance cost is high, and the task running time needs to be manually set, and the running start time of the current task can only be estimated according to the running time of the dependent tasks. , if the running time of the dependent task changes, the current task cannot be sensed, which may cause the dependent task to start computing without completing the current task, or the dependent task has already been completed, but the current task has to wait for the...

AI Technical Summary

Problems solved by technology

[0002] With the advent of the big data era, Hadoop open source technology is widely used. In actual business scenarios, there are more and more computing tasks based on the Hadoop platform. The increase in computing tasks makes the interdependence between tasks difficult to schedule and manage. Visual dependency management, task dependencies are not clear, and task dependencies rely on manual maintenance, maintenance costs are high, the running time of tasks needs to be manually set, and the running start time of the current task can only be estimated based on the running time of the dependent tasks , if the running time of the dependent task changes, the current task cannot be perceived, which may cause the dependent task to be calculated before the current task is completed, or the dependent task has already been completed, but the current task has to wait for the start time to arrive before running, wasting computing resources And there may be data errors in the task, and when the previous dependent task fails, the subsequent task cannot be effectively reminded, and the failed task cannot be automatically re-run, resulting in the failure of the task chain, and there may be a task dependency loop in the task dependency relationship that causes the task to die Circulation, lack of alarm mechanism, when dependent tasks fail to run, users cannot perceive, therefore, it is necessary to design a task scheduling management system and method for hadoop big data platform

Images