Task processing method, device and system, equipment and storage medium

Abstract

The embodiment of the invention discloses a task processing method, device and system, equipment and a storage medium. The method comprises the steps of obtaining all tasks corresponding to a target data processing process; generating at least one scheduling unit according to the dependency relationship of each task and the first definition rule; generating at least one task flow according to thedependency relationship of each scheduling unit and a second definition rule; and allocating the at least one task flow as a task group matched with the target data processing process to a hardware scheduling component. According to the embodiment of the invention, the task can be processed according to the dependency relationship and the definition rule of each task, and the processed task is distributed to the hardware scheduling component, so that task arbitration in the task distribution process of the hardware scheduling component can be reduced, the task processing efficiency is improved, and the performance cost of task synchronization is reduced.

Description

technical field [0001] Embodiments of the present invention relate to data processing technologies, and in particular, to a task processing method, device, system, equipment, and storage medium. Background technique [0002] In deep learning scenarios based on neural networks, the calculation process of training and reasoning is usually abstracted into a directed acyclic graph composed of operators by the framework layer, and then converted into a directed acyclic graph composed of hardware-executable tasks through the middleware layer. Acyclic graph, so as to obtain the executable tasks of each hardware. [0003] In the prior art, various tasks are usually delivered sequentially in the form of streams to the task scheduling module in the hardware through the ring buffer, and the task scheduling module distributes them to other task execution modules in the hardware. The task execution module in the hardware executes tasks concurrently to improve task execution efficiency. ...

AI Technical Summary

Problems solved by technology

[0004] In the process of realizing the present invention, the inventor found that the defect of the prior art is that, due to the dependencies between tasks, the task scheduling module needs to wait for the execution of the previous task with dependencies to be completed before assigning the next task to the task The execution of the execution module may cause some task execution modules in the hardware to be idle, and the overhead of task synchronization will also seriously affect system performance

Images