Distributed data integration job scheduling method and device

Abstract

The invention relates to a distributed data integration job scheduling method and device. The method aims at a special scene possibly faced by data integration. A job scheduling device is responsiblefor issuing the data integration job to the job operation device, and the job operation device receives the scheduling task, starts job execution, feeds back job operation state information to the jobmanagement module, feeds back working node computing resources to the resource scheduling module, and feeds back lost or fault information to the job preloading module. The method has the following comprehensive characteristics: (1) high availability, fault tolerance and weak consistency; (2) a low delay characteristic for quasi-real-time job scheduling; (3) multi-tenant concurrency control oriented to the cloud service application; (4) computing resource isolation and multi-job parallel scheduling; and (5) a priority scheduling mechanism.

Description

technical field [0001] The invention relates to the field of big data basic technology, in particular to a distributed data integration job scheduling method and device. Background technique [0002] With the evolution of the digital economy, business digitization in many industries has been fully developed, and digital business has gradually become a new focus. However, due to the large number of data islands derived from business digitization, which has become a common pain point in the realization of digital business, various industries urgently need data integration, open up and avoid data islands, integrate and manage data resources, so as to effectively develop the associated value between data. [0003] Data integration often faces tens of thousands of job scheduling, including data exchange, data preprocessing and other job types. The design of the scheduling system needs to consider various complex scenarios. For example, some scenarios not only have a large number...

AI Technical Summary

Problems solved by technology

Chinese invention patent CN201610197298 discloses a task scheduling method, device and system, and proposes a multi-channel multi-task distributed scheduling method, which solves the problem of starvation of other jobs caused by too long a single task scheduling time, but does not consider the job Low latency of metadata access, how to ensure the consistency of job metadata, etc.

Chinese invention patent CN201410748604 discloses a distributed task scheduling system and method, which proposes a distributed task scheduling system and method that ensures the reliability of the system itself, supports independent or associated tasks, and supports task rollback, but is not suitable for data Integrate complex scenarios. In these scenarios, the rollback of tasks is not the focus, and there is no need to associate data integration tasks. It does not consider the high availability of high downtime costs under large-scale complex data integration tasks, and quasi-real-time Under the premise of the existence of tasks, how to reduce the delay as much as possible, how to ensure the consistency of job metadata, etc.

Images