Loosely coupled distributed workflow coordination system and method

Abstract

The invention discloses a loosely coupled distributed workflow coordination system and method, and the method comprises the steps: a user carries out definition, online, operation and maintenance of a workflow through calling an interface service API (Application Program Interface); a distributed workflow coordinator schedules the workflow in a timed manner by integrating a distributed timing engine Quartz, adds the workflow to a workflow distribution distributed message queue MQ, receives the workflow, processes the task dependency relationship of the workflow, and adds a business type task to be executed after coordination to the task distribution distributed message queue MQ; a distributed task actuator Worker receives each business type task from the task distribution distributed message queue MQ and executes the business type tasks, and the task execution result is called back to the distributed workflow coordinator through the task callback distributed message queue MQ; and finally, the Coordinator persistently stores a task execution result in a database for feeding back the result to a user. According to the method, Coordinator is focused on logic coordination processing, so that full decoupling of workflow coordination processing and task execution is ensured, and the throughput, expansibility and scalability of the system are improved.

Description

technical field [0001] The invention relates to the technical field of big data development and processing, in particular to a loosely coupled distributed workflow coordination system and method. Background technique [0002] In a data development project, it is usually necessary to formulate several workflows (DAG with tasks as nodes, satisfying dependencies), and each workflow is executed periodically according to the corresponding timing (Cron expression); this type of business usually passes Workflow scheduling system to achieve. [0003] The traditional workflow scheduling system usually consists of: API interface service, Manager workflow manager, Worker task executor, RDBMS data storage. [0004] API interface service: responsible for the creation of workflow and its tasks, the creation of timing, and the timing of workflow online; [0005] Manager manager: responsible for the scheduled execution of workflow, workflow DAG to task linear queue, task queue execution c...

AI Technical Summary

Problems solved by technology

[0009] Manager does not support priority sorting when processing workflows and their task queues. The linear order of workflows or tasks without a clear sequence is random, and the scheduling is not precise enough;

[0010] Manager and Worker are highly coupled, which is not conducive to system expansion and maintenance;

[0011] The Manager memory manages a large number of workflow instances and their task team contexts, which consumes a lot of resources and limits the system throughput. In addition, because the workflow context is cached inside the Manager instance, one workflow instance can only be completed inside one Manager instance. Manager's distributed processing granularity is limited to workflow instances

Images