General batch job execution method and device in distributed environment

Abstract

The invention can be applied to the technical field of big data, and provides a general batch job execution method and device in a distributed environment, and the general batch job execution method in the distributed environment comprises the steps: receiving a scheduling parameter and a scheduling request sent by a scheduler; generating a scheduling list according to the scheduling parameters; and in response to the scheduling request, executing a general batch job according to the scheduling list. According to the general batch job execution method and device in the distributed environment provided by the invention, a unique scheduler and actuator separation design is adopted, scheduling arrangement is elaborately designed, the scheduler does not serve as a role of a scheduling command brain any more, the actuator can automatically generate a scheduling task at the operation moment. The complexity of batch system design can be effectively reduced, and the technical defects of an existing technical scheme are effectively overcome.

Description

Technical field [0001] The present invention belongs to the field of large data technology, and more particularly to distributed system service calls, specific to a general batch job implementation method and apparatus in a distributed environment. Background technique [0002] In the prior art, in an application system based on data processing, the stored procedure and task scheduling or third-party dispatching system related to the database system are implemented, and there is also a special ETL software to implement, the main The problem is that the scheduling actuator has exclusive, universality, and scalability. How to design a simple and easy-to-use, can be applied to most of the common framework for execution scenarios, which is of great significance for batch operations. Specifically, there are usually two technology routes for existing running batches: [0003] (1) running batch system composed of database system stored procedures and job scheduling chains [0004] In th...

AI Technical Summary

Problems solved by technology

[0024] The scheduler and executor belong to exclusive status, and the openness is poor: the two are not open, which means that the scope of adaptation of the scheduler is limited, and the two can only be used together, but cannot be used flexibly with third-party schedulers or executors ;

[0025] Complex job scheduling design: Since job scheduling is completely controlled by the scheduler, in a distributed environment, designers need to consider scenarios such as whether different jobs must run on the same machine

Designers must manually design the scheduling logic of the directed acyclic graph, which increases the difficulty of design;

[0026] The control of parallel execution is complex: the scheduler can only execute parallel logic according to the path of the directed acyclic graph, but cannot control the degree of concurrency on each execution path, which may cause the situation that computing resources cannot be fully utilized, and dynamic adjustment cannot be displayed The degree of parallelism for each path

[0027] Difficulty in business expansion: the scheduler can only execute the job types that the executor currently has. For complex and highly relevant businesses, the business needs to be split into existing job types, which is difficult to expand. The interface is also more difficult to develop;

[0028] It is difficult to realize the idempotency of job execution, and it is difficult to realize the continuation of the breakpoint when the task is interrupted: since the job scheduling plan is allocated by the scheduler, the job plan is generally a complex tree structure. When the job execution fails, the new execution It cannot be restored to run on the host at that time, and it is difficult to continue running from breakpoints

Images