Loosely coupled distributed workflow coordination system and method

A distributed and workflow technology, applied in resource allocation, program startup/switching, inter-program communication, etc., can solve problems such as limited system throughput, inaccurate scheduling, and large resource consumption, to improve business processing capabilities, Responsibility is single and clear, and the effect of improving throughput

Pending Publication Date: 2021-12-21
浙江数新网络有限公司
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] Manager does not support priority sorting when processing workflows and their task queues. The linear order of workflows or tasks without a clear sequence is random, and the scheduling is not precise enough;
[0010] Manager and Worker are highly coupled, which is not conducive to system expansion and maintenance;
[0011] The Manager memory manages a ...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Loosely coupled distributed workflow coordination system and method
  • Loosely coupled distributed workflow coordination system and method
  • Loosely coupled distributed workflow coordination system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0058] The example of the present invention includes: API, Coordinator, Worker three parts, and RDBMS, Cache, MQ and Zookeeper.

[0059] The API side is responsible for processing the user's HTTP request for WEB UI operations; for workflow and its tasks, the task must specify a Worker group (WorkerGroup), where tasks are divided into logical tasks (including sub-workflows, dependent workflows and conditions) and business Type task (such as shell, http, procedure, spark, etc.) definition creation, timing creation (based on cron expression), and data persistence to RDBMS; scheduled online is allowed after the workflow is online, and scheduled online will use the workflow as a scheduled The job is added to the Quartz engine and executed after waiting for a timing trigger;

[0060] The API side stores workflow task dependent resource files through HDFS;

[0061] Through the re-run operation on the API side, the workflow instance that failed to terminate can be added to the workfl...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a loosely coupled distributed workflow coordination system and method, and the method comprises the steps: a user carries out definition, online, operation and maintenance of a workflow through calling an interface service API (Application Program Interface); a distributed workflow coordinator schedules the workflow in a timed manner by integrating a distributed timing engine Quartz, adds the workflow to a workflow distribution distributed message queue MQ, receives the workflow, processes the task dependency relationship of the workflow, and adds a business type task to be executed after coordination to the task distribution distributed message queue MQ; a distributed task actuator Worker receives each business type task from the task distribution distributed message queue MQ and executes the business type tasks, and the task execution result is called back to the distributed workflow coordinator through the task callback distributed message queue MQ; and finally, the Coordinator persistently stores a task execution result in a database for feeding back the result to a user. According to the method, Coordinator is focused on logic coordination processing, so that full decoupling of workflow coordination processing and task execution is ensured, and the throughput, expansibility and scalability of the system are improved.

Description

technical field [0001] The invention relates to the technical field of big data development and processing, in particular to a loosely coupled distributed workflow coordination system and method. Background technique [0002] In a data development project, it is usually necessary to formulate several workflows (DAG with tasks as nodes, satisfying dependencies), and each workflow is executed periodically according to the corresponding timing (Cron expression); this type of business usually passes Workflow scheduling system to achieve. [0003] The traditional workflow scheduling system usually consists of: API interface service, Manager workflow manager, Worker task executor, RDBMS data storage. [0004] API interface service: responsible for the creation of workflow and its tasks, the creation of timing, and the timing of workflow online; [0005] Manager manager: responsible for the scheduled execution of workflow, workflow DAG to task linear queue, task queue execution c...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50G06F9/54
CPCG06F9/4881G06F9/5038G06F9/546G06F2209/548G06F2209/5021G06F2209/484G06F2209/5011
Inventor 刘媛媛陈廷梁
Owner 浙江数新网络有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap