Task pipeline execution method and device and electronic equipment

Abstract

Embodiments of the invention provide a task pipeline execution method and device, and electronic equipment, and belong to data scheduling. The method comprises the steps of setting a plurality of sub-processes isolated from a main process in a task container, each sub-process being used for executing a pipeline task; after a to-be-executed task related to the main process is obtained, selecting atarget sub-processd from the multiple sub-processes, wherein the target sub-process is used for running the to-be-executed task; after the to-be-executed task in the target sub-process is executed, executing a task in an associated sub-process related to the target sub-process according to a preset strategy; and after the tasks in all the sub-processes in the container are executed, releasing system resources occupied by all the sub-processes in the process of executing the tasks. Through the processing scheme disclosed by the invention, the execution efficiency of the task pipeline can be improved.

Description

technical field [0001] The present disclosure relates to data scheduling, and in particular to a task pipeline execution method, device and electronic equipment. Background technique [0002] There are two different approaches to the implementation of task pipelines, one is loosely coupled and the other is tightly coupled. Loose coupling means that each task runs in an independent container. When a task in the task pipeline is executed, the container will be killed, and then another container will be started to run the next task. Tight coupling means that all tasks are scheduled by the same main process. The advantage of the tight coupling method is that the execution efficiency of the task pipeline is high, and all tasks run in the same container, without the overhead of frequently starting the container. The problem with the tightly coupled method is that there is a memory leak in the python script, and the python modules cannot be uninstalled cleanly. If two tasks have...

AI Technical Summary

Problems solved by technology

The problem with the tight coupling method is that there is a memory leak in the python script, and the python module cannot be uninstalled cleanly

If two tasks have modules with the same name but different functions (such as different versions of the same tool), the tasks will not be executed normally

Images