Cross-language task scheduling method and device, equipment and readable storage medium

A task scheduling and cross-language technology, which is applied in the field of equipment and readable storage media, devices, and cross-language task scheduling methods, can solve the problems of increasing system storage IO overhead, reduce system storage IO overhead, and speed up task execution efficiency. Effect

Pending Publication Date: 2021-01-08
HANGZHOU ANHENG INFORMATION TECH CO LTD
View PDF0 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, for cross-process data communication between tasks, the form of intermediate result storag

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cross-language task scheduling method and device, equipment and readable storage medium
  • Cross-language task scheduling method and device, equipment and readable storage medium
  • Cross-language task scheduling method and device, equipment and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0052] Please refer to figure 1 , figure 1 It is a flow chart of a cross-language task scheduling method in an embodiment of the present invention, and the method includes the following steps:

[0053] S101. Obtain upstream tasks and downstream tasks with task dependencies.

[0054] Among them, the task dependency means that there is an interdependent relationship in terms of time and resources. Specifical...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a cross-language task scheduling method and device, equipment and a readable storage medium. The method comprises the steps of obtaining an upstream task and a downstream taskhaving a task dependency relationship; allocating an upstream task to the upstream node, and allocating a downstream task to the downstream node; wherein the upstream node and the downstream node correspond to different programming languages; determining a data exchange rule by utilizing the task dependency relationship; establishing communication connection corresponding to remote procedure callbetween the upstream node and the downstream node; and performing cross-language real-time data transmission between the upstream node and the downstream node by utilizing communication connection andcombining a data exchange rule. According to the method, under the condition that scheduling and calculation of distributed tasks can be met, independent storage of intermediate result data is omitted, the IO storage overhead of the system is reduced, and the task execution efficiency can be improved.

Description

technical field [0001] The present invention relates to the field of computer application technology, in particular to a cross-language task scheduling method, device, equipment and readable storage medium. Background technique [0002] With the rapid development of big data technology, the storage and calculation process of massive data is becoming more and more complicated. Especially under the demand of data analysis and mining, a large number of data analysis and calculation tasks need to be carried out by task scheduling system. For each computing task, different development languages ​​will be used for development, such as the sql node to complete the data extraction task, and the python task node to perform algorithm analysis and calculation. Whether efficient data exchange can be performed across processes is one of the important guarantees for task computing efficiency. [0003] Azkaban (a batch workflow task scheduler open sourced by Linkedin), oozie (workflow sch...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50
CPCG06F9/4881G06F9/5027
Inventor 李德恩范渊刘博
Owner HANGZHOU ANHENG INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap