Task orchestration method, orchestrator, equipment and readable storage medium

A task and processor technology, applied in the field of big data, can solve the problems of low task scheduling efficiency, workflow errors, workflow editing and testing taking a lot of time and resources, and achieve the effect of improving task scheduling efficiency.

Active Publication Date: 2019-11-19
MICROFUN INC
View PDF3 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a task arrangement method, arranger, device and readable storage medium, which are used to solve the problem that in the prior art, the technician manually defines the workflow. If some tasks in the middle change, the technician needs to manually define the workflow Re-editing or adjustment will inevitably lead to errors in the workflow due to human error, and the new workflow needs to be re-tested. The editing and testing of the workflow consumes a lot of time and resources, and the efficiency of task arrangement is low.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task orchestration method, orchestrator, equipment and readable storage medium
  • Task orchestration method, orchestrator, equipment and readable storage medium
  • Task orchestration method, orchestrator, equipment and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] figure 1 It is a flow chart of the task scheduling method provided by Embodiment 1 of the present invention. The embodiment of the present invention is aimed at manually defining the workflow by technicians in the prior art. If some tasks in the middle change, the technicians need to manually re-edit or adjust the workflow, which will inevitably lead to the existence of workflow due to manual operation errors. Errors require re-testing of the new workflow, editing and testing of the workflow consumes a lot of time and resources, and the problem of low efficiency of task arrangement provides a task arrangement method.

[0031] The method in this embodiment is applied to the orchestrator, which can be installed on the server in the big data platform. In other embodiments, the method can also be applied to other devices. This embodiment uses the orchestrator as an example to illustrate sexual description.

[0032] Such as figure 1 As shown, the specific steps of the met...

Embodiment 2

[0049] figure 2 It is a flow chart of the task arrangement method provided by Embodiment 2 of the present invention. On the basis of the first embodiment above, in this embodiment, according to the task script, the data dependency between the source table and the target table in the task is automatically generated, including: extracting the input as the corresponding task from the syntax tree of the task script The data partition of the source table of the data, and the data partition of the target table as the output data of the corresponding task, the data partition of the target table depends on the data partition of the source table; according to the syntax tree, the data partition of the source table based on the intermediate variable is generated and Data dependencies between data partitions of the target table. Such as figure 2 As shown, the specific steps of the method are as follows:

[0050] Step S201, acquiring a task script, which is a descriptive language scr...

Embodiment 3

[0123] Figure 5 It is a schematic structural diagram of the arranger provided by Embodiment 3 of the present invention. The orchestrator provided in the embodiment of the present invention can execute the processing flow provided in the embodiment of the task orchestration method. Such as Figure 5 As shown, the orchestrator 30 includes: a task script acquisition module 301 , a dependency generation module 302 and a task driver module 303 .

[0124] Specifically, the task script obtaining module 301 is used to obtain the task script, and the task script is a descriptive language script.

[0125] The dependency generating module 302 is configured to automatically generate the data dependency between the source table and the target table in the task corresponding to the task script according to the task script.

[0126] The task driving module 303 is used to drive the execution of tasks in a data-driven manner according to the data dependency relationship between the source ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a task orchestration method, an orchestrator, equipment and a readable storage medium. The method comprises the following steps: editing each task script into a descriptive language script, and automatically generating a data dependency relationship between a source table and a target table in a task corresponding to the task script according to the descriptive language script of the task; according to the data dependency relationship between the source table and the target table in each task, executing the drive task through the data-driven mode, technicians do not needto manually edit workflow files, when some tasks change, the data dependence relation between the source tables and the target tables in the tasks can be automatically updated according to the task scripts, the scheduling process between the tasks cannot be affected, task scheduling testing does not need to be conducted again, and the task scheduling efficiency is improved.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a task arrangement method, arrangement device, equipment and readable storage medium. Background technique [0002] On the big data platform, the most important link is to process the collected raw data, including cleaning, organizing, classifying, and summarizing. In most cases, these links cannot be completed in one step. For example, the raw data itself has various sources, and each source has a different format and requires different preprocessing. Therefore, the data processing method of the big data platform is generally called "pipeline". Generally, a program or service is used to arrange the execution of each link in the pipeline at regular intervals, or to execute sequentially according to a process. This program or service generally Called a scheduler (scheduler), or orchestrator (orchestrator), each link in the pipeline is called a task (task). [0003] At ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/30G06F16/21G06F16/25
CPCG06F8/31G06F16/21G06F16/25G06F9/4881G06F2209/484Y02D10/00G06F16/2246G06F16/2358G06F9/3838G06F9/5061
Inventor 高翅李一凡
Owner MICROFUN INC
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap