Task processing method, scheduler, scheduling system, device and medium

Abstract

The present disclosure provides a task processing method, a scheduler, a scheduling system, a device and a medium, and relates to the technical field of computers. The task includes multiple sub-tasks, and the multiple sub-tasks are executed in a pipeline parallel manner by multiple computing units. The method comprises: monitoring the device state of the multiple computing units; in response to determining that there is an idle unit in the multiple computing units according to the device state, determining a target task from all currently executing sub-tasks to be executed based on the next scheduled task of the idle unit; and assigning the target task to the idle unit for execution. The next scheduled task of the idle unit is determined based on a preset binding relationship of the pipeline parallel. The method can reduce the idle time of the idle unit and improve the task processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a task processing method, scheduler, scheduling system, device, and medium. Background Technology

[0002] Pipeline parallelism is one of the key technologies for solving the problem of ultra-large-scale tasks that cannot be accommodated by a single device. Pipeline parallelism is a parallel strategy that breaks down ultra-large-scale tasks into several stages in a logical / temporal sequence, and then executes the different stages simultaneously on different computing units, aiming to reduce device idle time by alternating execution. Summary of the Invention

[0003] This disclosure provides a task processing method, a scheduler, a scheduling system, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, embodiments of this disclosure propose a task processing method, comprising: a task including multiple subtasks, the multiple subtasks being executed in a pipelined parallel manner through multiple computing units, the method comprising: monitoring the device status of the multiple computing units; in response to determining, based on the device status, that an idle unit has appeared among the multiple computing units, determining a target task from the unexecuted subtasks that have no dependency on all currently executed subtasks based on the next predetermined task of the idle unit; and assigning the target task to the idle unit for execution; wherein the next predetermined task of the idle unit is determined based on a preset binding relationship of pipelined parallelism.

[0005] In some embodiments, determining a target task from subtasks to be executed that are independent of all currently executing subtasks based on the next predetermined task of the idle unit includes: determining an idle window of the idle unit based on the next predetermined task of the idle unit; and selecting the target task from the subtasks to be executed based on the estimated idle duration corresponding to the idle window.

[0006] In some embodiments, based on the estimated idle time corresponding to the idle window, the target task is selected from the subtasks to be executed, including: if there are subtasks in the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time, the subtasks whose estimated execution time is less than or equal to the estimated idle time are selected from the subtasks to be executed to obtain the task to be scheduled; and the target task is determined from the task to be scheduled.

[0007] In some embodiments, determining the target task from the tasks to be scheduled includes: determining the target task from the tasks to be scheduled according to a preset task priority strategy.

[0008] In some embodiments, determining the target task from the tasks to be scheduled includes: selecting one or more subtasks from the tasks to be scheduled based on the estimated execution time of each subtask in the tasks to be scheduled and the estimated idle time of the idle unit to obtain the target task; wherein the sum of the estimated execution time of the selected subtasks is less than the estimated idle time, and the difference between the sum of the estimated execution time and the estimated idle time satisfies a preset difference condition.

[0009] In some embodiments, based on the estimated idle time corresponding to the idle window, the target task is selected from the subtasks to be executed, including: when the estimated execution time of all subtasks to be executed is greater than the estimated idle time, the subtask with the shortest estimated execution time is selected from the subtasks to be executed and determined as the target task.

[0010] In some embodiments, the idle window includes an idle start time and an idle end time; determining the idle window of an idle unit based on the next predetermined task of the idle unit includes: monitoring the actual execution status of each subtask running on multiple computing units and setting the actual completion time of the current subtask of the idle unit as the start time of the idle window; determining the predecessor subtask that directly depends on the next predetermined task of the idle unit from the subtasks that are being executed; and using the estimated completion time of the predecessor subtask as the idle end time of the idle window.

[0011] In some embodiments, the idle window of an idle unit is used to identify the estimated idle duration of the idle unit; determining the idle window of an idle unit based on the next predetermined task of the idle unit includes: determining the actual execution duration of the current subtask of the idle unit based on the actual start time and actual end time of the current subtask of the idle unit; determining the estimated execution duration of the current subtask of the idle unit based on the next predetermined task of the idle unit, wherein the estimated execution duration is greater than the actual execution duration; and determining the estimated idle duration of the idle unit based on the estimated execution duration and the actual execution duration of the current subtask of the idle unit.

[0012] In some embodiments, the method further includes: when it is predicted that the video memory of the idle unit meets preset conditions, migrating redundant data unrelated to the target task stored in the video memory of the idle unit to a preset cache; wherein the preset conditions include: the video memory usage required by the target task is greater than the current available video memory capacity of the idle unit or the current video memory utilization rate of the idle unit is greater than a preset utilization rate threshold.

[0013] In some embodiments, the method further includes: when the required data of the target task has been migrated to a preset cache, reading the required data from the preset cache based on the storage address information of the required data of the target task, and loading the required data into the video memory of the free unit.

[0014] In some embodiments, the task includes a model training task, and the method further includes: dividing the model training task based on a forward task, an input gradient calculation task, and a parameter gradient calculation task to obtain multiple sub-tasks.

[0015] In some embodiments, the model training task is divided into multiple subtasks based on a forward task, an input gradient calculation task, and a parameter gradient calculation task, including: dividing the model training task into one or more computational stage tasks; wherein each computational stage task includes a forward task, an input gradient calculation task, and a parameter gradient calculation task; and dividing each computational stage task into one or more subtasks based on a preset partitioning strategy, thereby obtaining multiple subtasks.

[0016] In some embodiments, after assigning the target task to an idle unit for execution, the method further includes updating the device state of the plurality of computing units and the state information of the subtask to be executed.

[0017] Secondly, this disclosure also provides a task scheduler, wherein a task includes multiple subtasks, and the multiple subtasks are executed in a pipelined parallel manner through multiple computing units. The scheduler includes: a monitoring module for monitoring the device status of the multiple computing units; a determination module for determining, in response to determining that an idle unit appears among the multiple computing units based on the device status, a target task is determined from the unexecuted subtasks that have no dependency on any of the currently executing subtasks based on the next predetermined task of the idle unit; and a scheduling module for allocating the target task to the idle unit for execution; wherein the next predetermined task of the idle unit is determined based on a preset binding relationship of pipelined parallelism.

[0018] Thirdly, this disclosure also provides a task scheduling system, wherein the task includes multiple subtasks, and the scheduling system includes: multiple computing units for executing the multiple subtasks in a pipelined parallel manner; and a scheduler for scheduling the multiple subtasks to the multiple computing units according to the processing method as described in any one of the first aspects.

[0019] Fourthly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the task processing method described in any implementation of the first aspect.

[0020] Fifthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to perform a task processing method as described in any implementation of the first aspect.

[0021] The task processing method disclosed herein can, by monitoring the device status of each computing unit in real time, select subtasks that are not dependent on the subtasks being executed from the subtasks to be executed as target tasks when the computing unit is idle, and assign the target task to the idle device for execution, so as to fill the idle time of the idle unit and improve the utilization efficiency of computing resources and the execution efficiency of tasks. Attached Figure Description

[0022] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of a task processing method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating a task processing method provided by an embodiment of the present disclosure in conjunction with a specific application scenario; Figure 3 Here is a structural block diagram of a task scheduler provided in this disclosure; Figure 4 A structural block diagram of a task scheduling system provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0024] Pipeline parallelism (PP) is one of the key technologies for addressing the limitations of single-device-intensive deep learning models. Its basic idea is to divide the model into multiple stages, each deployed on a different computing device (such as a GPU). A training batch is further divided into multiple micro-batches. Data flows sequentially through each stage, much like in a pipeline. "One-Forward-One-Backward (1F1B)" is a classic pipeline scheduling strategy designed to reduce device idle time (i.e., "bubbles") and memory spikes by alternating forward and backward computations.

[0025] However, for a given model parallel configuration (pipeline depth, number of micro-batches), the execution order of all tasks during the entire model training process is completely determined before runtime and will not change during runtime. That is, the above model training process is a static scheduling mechanism and cannot handle dynamic changes during training. When the actual execution time does not match the expectation, idle time bubbles are generated, reducing the utilization of hardware resources.

[0026] Based on this, the present disclosure provides a task processing scheme that can reduce device idle time and improve task execution efficiency.

[0027] Figure 1 The flowchart illustrates a task processing method provided in this embodiment of the present disclosure, wherein the task includes multiple subtasks, and the multiple subtasks are executed in a pipelined parallel manner through multiple computing units.

[0028] like Figure 1 The task processing method 100 shown specifically includes the following steps: Step 101: Monitor the device status of multiple computing units.

[0029] Specifically, in pipelined parallelism, the task is first broken down into multiple interdependent, sequentially executed subtasks. Then, based on a preset strategy, these subtasks are pre-bound to multiple computing units. This allows subtasks to be scheduled to their respective computing units based on their bound subtasks, enabling each computing unit to process its subtasks in a pipelined manner. This reduces communication overhead between computing units and maximizes equipment utilization. The preset strategy can be based on the dependencies between subtasks, the estimated execution time of each subtask, etc., and this disclosure does not limit its scope.

[0030] However, during the actual execution of a subtask, various factors may cause fluctuations in its execution time, resulting in idle time for the corresponding computing unit. This disclosure addresses this by enabling real-time monitoring of each computing unit to determine its device status.

[0031] The device status can include whether the device is in an idle state or in a working state.

[0032] Step 102: In response to determining that an idle unit has appeared among multiple computing units based on the device status, based on the next predetermined task of the idle unit, a target task is determined from the subtasks to be executed that have no dependency on all currently executing subtasks, and the target task is assigned to the idle unit for execution.

[0033] Specifically, by monitoring each computing unit in real time, if a computing unit is found to be idle, in order to avoid wasting the computing resources of the idle unit, a target task can be selected from multiple unexecuted subtasks that are not dependent on all currently executing subtasks and assigned to the idle unit for execution.

[0034] In addition, to avoid wasting computing resources in idle units, a target task can be determined in real time each time an idle unit is detected, and the target task can be assigned to that idle unit.

[0035] The target task has no data or sequence dependencies with any of the subtasks being executed, ensuring that the idle unit can directly execute the target task without affecting the execution of other computing units.

[0036] In some embodiments, the idle unit may be in an idle state because its current task has been completed ahead of schedule, resulting in the next scheduled task corresponding to the idle unit not yet being ready. Therefore, the idle unit needs to wait for the next scheduled task to be ready before it can continue to execute the next scheduled task. During the waiting process, the idle unit is in an idle state.

[0037] The next scheduled task for an idle unit is determined based on the pre-defined binding relationship of pipeline parallelism.

[0038] The purpose of the idle unit executing the target task is to fill the idle time of the idle unit. Furthermore, after the target task is completed and the next scheduled task corresponding to the idle unit is ready, the next scheduled task corresponding to the idle unit can be assigned to the idle unit so that the idle device can continue to execute the corresponding sub-task in a pipelined parallel manner.

[0039] The task processing method provided in this disclosure, by monitoring the device status of each computing unit in real time, can, when a computing unit is idle, select a subtask that has no dependency on the subtask being executed from the subtasks to be executed as the target task based on the next predetermined task of the idle unit, and assign the target task to the idle unit for execution, so as to fill the idle time of the idle unit and improve the utilization efficiency of computing resources and the execution efficiency of tasks.

[0040] In response to the above Figure 1 In step 102, during the process of filtering target tasks, filtering can be based on the estimated idle time of idle devices, the priority of the subtasks to be executed, or other strategies. A specific implementation method is given below.

[0041] In some embodiments, the step of determining the target task from the subtasks to be executed that are independent of all currently executing subtasks based on the next predetermined task of the idle unit includes: determining the idle window of the idle unit based on the next predetermined task of the idle unit; and selecting the target task from the subtasks to be executed based on the estimated idle time corresponding to the idle window.

[0042] Specifically, first, the subtasks being executed by each computing unit are determined, and then subtasks to be executed that have no dependency on any of the currently executing subtasks are selected.

[0043] Then, subtasks that match the idle window of the idle unit can be selected from the pending subtasks that have no dependency on any of the currently executing subtasks, and these subtasks can be used as target tasks. The idle window corresponds to the estimated idle duration of the idle unit, allowing target tasks to be selected based on this estimated idle duration. In this way, the idle time of the idle unit can be filled without affecting the execution of the next scheduled task in that unit, improving task execution efficiency and accuracy.

[0044] In some embodiments, subtasks to be executed that have no dependency on any currently executing subtasks can form a set of ready tasks. Further, subtasks matching the idle window of the idle unit can be selected from the set of ready tasks as target tasks. Of course, all subtasks in the set of ready tasks can be assigned to the idle unit as target tasks for execution. In some embodiments, the highest-priority subtask can also be selected from the set of ready tasks as the target task. In other embodiments, the target task can be determined from the set of ready tasks based on the real-time video memory of the idle device to ensure accurate execution of the target task. Of course, the target task can also be determined by comprehensively considering one or more factors, including the estimated idle time of the idle unit, the priority of the subtasks to be executed, and the real-time video memory of the idle unit; this disclosure does not limit this approach.

[0045] Furthermore, when selecting target tasks from the pending subtasks based on the estimated idle time corresponding to the idle window, the selection can also be based on the estimated execution time of each subtask within the pending subtasks. A specific implementation method is given below.

[0046] In some embodiments, the step of selecting a target task from the subtasks to be executed based on the estimated idle time corresponding to the idle window includes: if there are subtasks in the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time, selecting subtasks in the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time to obtain the task to be scheduled; and determining the target task from the task to be scheduled.

[0047] Specifically, the task to be scheduled includes one or more subtasks whose estimated execution time is less than or equal to the estimated idle time. In this case, any subtask can be selected from the task to be scheduled as the target task, so that the execution of the target task by the idle unit has almost no impact on the execution of the next scheduled task by the idle unit.

[0048] In some embodiments, determining the target task from the tasks to be scheduled includes: determining the target task from the tasks to be scheduled based on a preset task priority strategy.

[0049] Specifically, when there are multiple tasks to be scheduled that meet the execution time requirements, a target task can be determined from the tasks to be scheduled based on a priority strategy to improve task execution efficiency. The priority strategy can be preset, for example, based on the importance of subtasks, the degree of matching between the estimated execution time of subtasks and the estimated idle time of idle units, the degree of matching between the data capacity required by subtasks and the current video memory of idle units, etc., thereby improving task execution efficiency.

[0050] In some embodiments, determining a target task from a list of tasks to be scheduled includes: selecting one or more subtasks from the list of tasks to be scheduled based on the estimated execution time of each subtask in the tasks to be scheduled and the estimated idle time of the idle units, to obtain the target task; wherein the sum of the estimated execution times of the selected subtasks is less than the estimated idle time, and the difference between the sum of the estimated execution times and the estimated idle time satisfies a preset difference condition. Specifically, the target task can be one or more. The sum of the estimated execution times is greater than 0 and less than the estimated idle time. Further, the preset difference condition can be that the difference between the estimated idle time and the sum of the estimated execution times is greater than or equal to 0. When the difference between the estimated idle time and the sum of the estimated execution times is equal to 0, the idle time of the idle units can be filled to the maximum extent, thereby minimizing the idle time of the idle devices, improving the utilization efficiency of computing resources, and improving the execution efficiency of the tasks. For example, let's take the matching degree between the estimated execution time of subtasks and the estimated idle time of idle units as an example to illustrate how to select target tasks from the subtasks to be executed. Suppose that the subtasks to be executed include subtasks A, B, and C, with corresponding estimated execution times of 5μs, 10μs, and 20μs, respectively, while the estimated idle time of the idle unit is 28μs. In this case, the sum of the estimated execution times of subtasks B and C is still less than the estimated idle time of the idle unit. Therefore, subtasks B and C can be selected as target tasks to fill the idle time of the idle unit, which can minimize the idle time of idle devices, improve the utilization efficiency of computing resources, and improve the execution efficiency of tasks.

[0051] Furthermore, in some embodiments, the step of selecting the target task from the subtasks to be executed based on the estimated idle time corresponding to the idle window includes: when the estimated execution time of all subtasks to be executed is greater than the estimated idle time, selecting the subtask with the shortest estimated execution time from the subtasks to be executed and determining it as the target task.

[0052] Specifically, if all subtasks in the pending subtasks have a duration longer than the estimated idle time, in order to avoid wasting the computing resources of the idle units and to ensure that the execution of the target task by the idle units has as little impact as possible on the execution of the next scheduled task by the idle units, the subtask with the shortest estimated execution time can be selected from the pending subtasks and used as the target task.

[0053] The task processing method provided in this disclosure, when there are subtasks among the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time, determines the tasks to be scheduled based on the estimated idle time of the idle unit, and selects target tasks from the scheduled tasks. This can maximize the filling of the idle time of the idle unit and reduce the waste of computing resources. Simultaneously, in the process of selecting target tasks from the subtasks to be executed, the priority of each subtask in the subtasks to be executed is combined to determine the target task, which can further improve the efficiency of task execution. Furthermore, when the estimated execution time of all subtasks in the subtasks to be executed is greater than the estimated idle time, the subtask with the shortest estimated execution time is selected as the target task. This can fill the idle time of the idle unit, reduce the waste of computing resources, and minimize the impact on the execution of the next scheduled task in the idle unit, thereby reducing the impact on the entire task execution process.

[0054] In some embodiments, the idle window includes an idle start time and an idle end time. The step of determining the idle window of an idle unit based on the next predetermined task of the idle unit includes: monitoring the actual execution status of each subtask running on multiple computing units, and setting the actual completion time of the current subtask of the idle unit as the start time of the idle window; determining, from the currently executing subtasks, a predecessor subtask that directly depends on the next predetermined task of the idle unit; and using the estimated completion time of the predecessor subtask as the idle end time of the idle window.

[0055] Specifically, the idle window refers to the time interval between the completion of the current subtask and the start of the next scheduled task in the idle unit.

[0056] By monitoring the actual execution status of each subtask running on multiple computing units in real time, the actual completion time of the current subtask on the idle unit can be determined. If the next scheduled task for the idle unit has not been scheduled to that idle unit after the completion of the current subtask, it indicates that the next scheduled task is not yet ready. In other words, the predecessor subtask directly dependent on the next scheduled task of the idle unit has not yet completed. After the predecessor subtask directly dependent on the next scheduled task of the idle unit completes, the idle unit can continue to execute its corresponding next scheduled task. Therefore, this disclosure can use the estimated completion time of the predecessor subtask as the idle end time of the idle window.

[0057] This disclosure enables the accurate determination of the idle window of an idle unit by identifying its idle start time and idle end time.

[0058] In addition, the estimated idle duration corresponding to the idle window of the idle unit can be determined based on the time fluctuation of the current subtask of the idle unit.

[0059] In some embodiments, the idle window of an idle unit is used to identify the estimated idle duration of the idle unit. The step of determining the idle window of an idle unit based on the next predetermined task of the idle unit includes: determining the actual execution duration of the current subtask of the idle unit according to the actual start time and actual end time of the current subtask of the idle unit; determining the estimated execution duration of the current subtask of the idle unit according to the next predetermined task of the idle unit, wherein the estimated execution duration is greater than the actual execution duration; and determining the estimated idle duration of the idle unit according to the estimated execution duration and the actual execution duration of the current subtask of the idle unit.

[0060] Specifically, in pipelined parallelism, each subtask can be pre-bound to its corresponding computing unit based on its dependencies and actual execution time, ensuring that no computing unit becomes idle. In other words, if the actual execution of each subtask matches expectations, the next pre-defined subtask will be ready after the computing unit finishes executing the current subtask, allowing for direct execution of the next pre-defined subtask. Idle units may experience idle windows due to fluctuations in execution time of the current subtask. Therefore, the actual execution status of each subtask running on multiple computing units can be monitored, and the actual start and end times of the current subtask in the idle unit can be determined. The actual execution duration of the current subtask can be determined based on its actual start and end times. Furthermore, the estimated execution duration of the current subtask can be determined based on the expected start time of the pre-bound current subtask in the idle unit and the expected start time of the next pre-defined task in the idle unit. Furthermore, based on the actual and estimated execution times of the current subtasks within an idle unit, the time fluctuations of the current subtasks within the idle unit can be determined, thereby determining the estimated idle time of that idle unit. In this way, the estimated idle time of an idle unit can be estimated simply by examining the execution status of its current subtasks; this method is simple and improves the efficiency of determining the idle window for an idle unit.

[0061] Furthermore, the real-time video memory of each computing unit also affects its execution of each subtask.

[0062] In some embodiments, in addition to steps 101-102 described above, the task processing method may further include: when it is predicted that the video memory of the idle unit meets the preset conditions, migrating the redundant data stored in the video memory of the idle unit that is unrelated to the target task to a preset cache.

[0063] Among them, redundant data unrelated to the target task can be data that is not currently used for the target task, but may be used in subsequent tasks. For example, for model training tasks, the redundant data can be some intermediate activation values, etc.; it can also be intermediate data generated in the aforementioned sub-tasks, etc. This disclosure does not limit it in this regard.

[0064] Before scheduling the target task to the idle unit, it can be determined whether the current available cache of the idle unit meets the preset conditions. If not, the temporarily unused data in the current video memory of the idle unit can be swapped out to main memory or other secondary storage to free up sufficient video memory space for the execution of the target task and ensure the accuracy of the target task execution.

[0065] In some embodiments, the preset conditions include: the required video memory usage of the target task is greater than the current available video memory capacity of the free unit or the current video memory utilization rate of the free unit is greater than a preset utilization rate threshold.

[0066] Specifically, the strategy for triggering the swapping of data from video memory to a preset cache can be varied. For example, it can be triggered based on the current available video memory capacity of the idle unit, i.e., when the current available video memory capacity of the idle unit is insufficient to accommodate the data required by the target task; it can be triggered based on a threshold of the video memory utilization rate of the idle unit, i.e., when the current video memory utilization rate of the idle unit is much greater than a preset utilization rate threshold; it can also be triggered predictively before allocating large tasks, i.e., before scheduling a task to the idle unit, it can be predicted in advance whether the available video memory of the idle unit meets the video memory requirements of the task to be scheduled to the idle unit, etc.

[0067] Furthermore, in some embodiments, the method further includes: when the required data of the target task has been migrated to a preset cache, reading the required data from the preset cache based on the storage address information of the required data of the target task, and loading the required data into the video memory of the idle unit.

[0068] Specifically, during the process of swapping data from video memory to main memory or other secondary storage, a mapping relationship between data and corresponding storage addresses can be generated. For example, a mapping table between data and corresponding storage addresses can be obtained so that the data can be accurately and quickly loaded into video memory from the corresponding location in the future, and the corresponding subtask can be executed based on the data, thereby improving task processing efficiency.

[0069] Before executing the target task, the mapping table can be used to determine whether the corresponding data has been migrated to the preset cache. Of course, other methods can also be used to determine whether the data required by the target task has been migrated to the preset cache, and this disclosure does not limit this method.

[0070] The situation where the data required by the target task has been moved to the preset cache may include: the data required by the target task was obtained in a previously executed subtask, but since the data was not used temporarily before, it has been swapped out to the preset cache in order to make room for other subtasks.

[0071] The embodiments provided in this disclosure, by real-time monitoring of the video memory of idle units, can migrate data in or out according to the storage status of the video memory, thereby achieving coordinated processing of scheduling and video memory management. This proactive video memory management strategy, in deep collaboration with dynamic scheduling, breaks through the limitation of static scheduling schemes that can only passively optimize video memory through a fixed execution order.

[0072] Furthermore, the task processing method provided in this disclosure can be applied to various large-scale tasks that can be broken down into multiple stages with unidirectional data flow between stages, such as large model tasks, video processing tasks, chip verification tasks, and so on.

[0073] In some embodiments, the task includes a model training task, and the method further includes: dividing the model training task based on a forward task, an input gradient calculation task, and a parameter gradient calculation task to obtain multiple sub-tasks.

[0074] Specifically, the model training task includes forward computation and backward computation tasks. The backward computation task can further include the task of calculating the input gradient and the task of calculating the parameter gradient.

[0075] This disclosure divides the model training task into multiple subtasks based on a forward task, an input gradient computation task, and a parameter gradient computation task. This allows the traditional atomic backpropagation function to be explicitly decomposed into two separable, schedulable task units: the input gradient computation task and the parameter gradient computation task. The execution of the parameter gradient computation task no longer immediately follows the input gradient computation task in the same micro-batch, gaining scheduling flexibility and improving the efficiency of model training task processing.

[0076] Furthermore, in some embodiments, the model training task is divided into multiple subtasks based on a forward task, an input gradient calculation task, and a parameter gradient calculation task. This includes: dividing the model training task into one or more computational stage tasks; wherein each computational stage task includes a forward task, an input gradient calculation task, and a parameter gradient calculation task; and dividing each computational stage task into one or more subtasks based on a preset partitioning strategy, thereby obtaining multiple subtasks.

[0077] Specifically, the model training task can first be divided into multiple stages according to the model's layer topology, with each stage task deployed on different computational units. Each stage task includes a complete forward task, a task to compute the input gradient, and a task to compute the parameter gradient.

[0078] Then, each stage task can be further broken down into finer-grained subtasks based on a preset strategy. This preset strategy could be splitting by micro-batch or by computation operator type, etc.

[0079] This embodiment of the disclosure divides the computation phase task into finer-grained subtasks, making the estimated execution time of each subtask relatively shorter. This provides greater flexibility in allocating target tasks to idle units, further reducing the idle time of the computation unit during task processing and improving task processing efficiency.

[0080] This disclosure enables real-time monitoring of the device status of multiple computing units and the various subtasks to be executed in the task queue, thereby enabling real-time monitoring of whether any computing units are idle, and, if an idle unit is available, scheduling a target task that meets the conditions from the currently pending subtasks to that idle unit.

[0081] Furthermore, in some embodiments, after assigning the target task to an idle unit for execution, the method further includes updating the device state of multiple computing units and the state information of the subtask to be executed.

[0082] The status information of the subtasks is used to identify the execution status of each subtask.

[0083] This disclosure can further improve the execution efficiency and accuracy of tasks by updating the device status of the computing unit and the status information of the subtasks to be executed.

[0084] The task processing method provided in this disclosure, by monitoring the device status of each computing unit in real time, can select subtasks that are independent of the currently executing subtasks from the list of subtasks to be executed as target tasks when a computing unit is idle. These target tasks are then assigned to the idle device for execution to fill the idle time of the idle unit, improving the utilization efficiency of computing resources and the execution efficiency of tasks. Furthermore, it can monitor the task queue containing subtasks to be executed in real time, enabling dynamic scheduling of these subtasks. Further, for model training tasks, in addition to dividing the task into forward and backward computation, it can decouple the backpropagation input gradient calculation task and the parameter gradient calculation task, and further decompose the subtasks into finer-grained subtasks, improving the matching degree between the subtasks to be executed and the idle window of the idle unit. In addition, during the scheduling of target tasks to idle units, this disclosure can comprehensively consider the estimated execution time of the quantum task, the idle window matching of the idle unit, and the real-time memory pressure to determine the target task. The task processing method disclosed herein integrates a complete system of dynamic monitoring, fine-grained decomposition, dynamic scheduling, and active memory management, which improves the efficiency of task processing and reduces the idle time of computing units during task processing.

[0085] The following section provides a detailed explanation of the above task processing methods using specific scenarios. Specifically, the advantages of dynamic scheduling in the MoE (Mixture of Experts) model, which leads to fluctuations in task latency, will be explained.

[0086] Specifically, in a pipelined parallel training environment, a static scheduling strategy is used to train a model containing MoE layers. Before training, the static scheduler generates a fixed task time sequence diagram based on an idealized assumption: that is, the time consumption of the three tasks of forward computation (represented by F), backpropagation to compute the input gradient (represented by B), and computation of the weight gradient (represented by W) in each microbatch is stable and in a fixed ratio (e.g., 1:1:1).

[0087] However, in real-world MoE model training, due to the dynamic nature of token routing, the number of experts activated and the computational load vary significantly across different micro-batches. This directly leads to drastic fluctuations in the execution time of the same type of task (such as different W tasks), deviating significantly from the ideal proportions based on static scheduling.

[0088] In a static scheduling strategy, the static scheduler allocates a fixed, preset time window for each task. For example, a 100μs execution window might be preset for a weight gradient calculation task (denoted as W_x) on computing unit GPU0. The following scenarios may occur: Scenario A (Task "Timeout"): Because the micro-batch corresponding to W_x activates the computationally intensive expert, its actual execution time may be 120μs. This will forcibly delay all tasks that follow immediately in the schedule, triggering a chain reaction in the pipeline, disrupting the entire rhythm and creating idle waits.

[0089] Scenario B (Task "Completed Ahead of Schedule"): Another task, W_y, with a lighter load, completes in just 80μs. However, the static scheduler does not immediately assign a new task, and GPU0 must wait for 20μs until the time window for the next preset task begins.

[0090] Therefore, it can be seen that the static scheduling strategy is rigid. It cannot perceive or adapt to the deviation between the actual execution time of a task and the preset window. This disconnect between planning and reality caused by the inherent characteristics of MoE will inevitably create tiny "time fragments" and "bubbles" in the strict static timing diagram, thus failing to achieve the theoretical goal of zero bubbles.

[0091] To address the task latency fluctuations caused by using a static scheduling strategy to process the MoE model, the following section discusses... Figure 2 The advantages of using the above-described task processing method to handle MoE tasks in this disclosure are explained.

[0092] Figure 2 This is a flowchart illustrating a task processing method provided by an embodiment of the present disclosure in conjunction with a specific application scenario.

[0093] See Figure 2 The task processing method specifically includes the following steps: Step 201: Runtime status monitoring.

[0094] Specifically, during model training, the idle status of each computing device (such as GPU), the fulfillment of dependencies of all scheduled tasks, and the real-time memory pressure of the system are continuously monitored in real time. Optionally, the actual start and end times of each task can also be monitored in real time, thereby detecting fluctuations in task duration (e.g., detecting that W_y completes ahead of schedule).

[0095] Step 202: Monitor device idle status and scan dynamic task sequences.

[0096] Specifically, when any computing device is detected to be idle, or when a sensing task is completed ahead of schedule and the computing device (such as GPU0) becomes idle, the scheduler immediately scans the dynamic task queue, scanning each task in the task sequence. The task sequence is dynamic and includes all tasks to be executed.

[0097] Step 203: Filter out all tasks whose dependencies are satisfied.

[0098] Specifically, a task that satisfies the dependency relationship is dynamically selected from the task sequence, that is, a task that has no dependency relationship with any of the tasks being executed, so as to ensure that the idle device can execute the task.

[0099] Step 204: Determine if there is a task whose estimated execution time is less than or equal to the current idle window of the device among all tasks whose dependencies have been satisfied.

[0100] If, among all the tasks whose dependencies have been satisfied, there is a task whose estimated execution time is less than or equal to the current idle window of the device, proceed to steps 205-206.

[0101] Step 205: Evaluate memory impact and communication overhead.

[0102] Step 206: Select the task with the highest overall priority.

[0103] Specifically, you can select the highest priority task whose estimated execution time matches the device's current idle window.

[0104] Returning to step 204, if there is no task with an estimated execution time less than or equal to the current idle window of the device among all tasks whose dependencies have been satisfied, continue to execute steps 207-208.

[0105] Step 207: Select the ready task with the shortest estimated execution time.

[0106] Among them, ready tasks refer to the tasks to be executed in step 203 whose dependencies are satisfied.

[0107] Step 208: Attempt to trigger main memory management (such as swapping out data) to free up space.

[0108] Specifically, the proactive memory management in conjunction with scheduling allows the dynamic scheduling decision-making provided in this disclosure to simultaneously consider the real-time memory pressure of the device. When memory insufficiency is predicted or detected, the scheduler can coordinate with the proactive memory management unit to swap temporarily unused data (such as certain intermediate activation values) from the computing device to the main memory or other secondary storage, thereby proactively freeing up memory space. This operation creates conditions for scheduling tasks that require a large amount of memory later. Conversely, when a task needs to be scheduled and its required data has been swapped out, a prefetch operation is triggered. This proactive memory management strategy, deeply integrated with dynamic scheduling, overcomes the limitation of static scheduling schemes that can only passively optimize memory through a fixed execution order.

[0109] After completing step 206 or step 208 above, continue with steps 209 to 210.

[0110] Step 209: Assign the task to an idle device for execution.

[0111] Specifically, the scheduler immediately assigns the appropriate task to an idle device for execution, making perfect use of the time fragments that would otherwise be wasted by static scheduling.

[0112] Step 210: Update device status and task queue.

[0113] The specific details of each step provided in the embodiments of this disclosure are similar to the relevant steps in the above text, and will not be repeated here.

[0114] The steps in the training process described above can be executed repeatedly in the model training task until the model training task is completed. In this embodiment, the above steps are triggered each time the device is idle. The tasks assigned to the idle device depend on the real-time computing device status and the status of each task, rather than any preset static task sequence.

[0115] The task processing method provided in this disclosure is a continuous dynamic scheduling loop that does not rely on a fixed timing diagram but makes decisions based on the real-time system state. By dynamically sensing and responding to task time fluctuations caused by factors such as MoE (Moment of Estimation), the method intelligently fills fine-grained tasks into irregular time fragments that are unavoidable in static scheduling. This ability to transform computational uncertainty into scheduling flexibility is key to achieving higher device utilization and more stable training efficiency.

[0116] The task processing method disclosed herein has stronger environmental adaptability, can dynamically respond to runtime fluctuations, and maintain high efficiency in environments with heterogeneous hardware or uneven loads. It can more thoroughly eliminate memory bubbles, filling fragmented idle time that static scheduling cannot handle through finer-grained tasks and dynamic scheduling. Simultaneously, it features superior memory management, with a proactive memory control mechanism providing memory optimization capabilities that surpass the static scheduling order. The memory management provided in this disclosure is more proactive and global, supporting the training of larger models. Compared to static scheduling's passive memory optimization relying on a fixed order, this disclosure, through "proactive memory management in coordination with scheduling," can proactively swap out data based on predictions during runtime, thereby more flexibly controlling memory peaks and supporting the training of larger models than static scheduling schemes can support.

[0117] Based on the same inventive concept as the above-described task processing method, this disclosure also provides a task scheduler.

[0118] Figure 3 This is a block diagram of a task scheduler provided in this disclosure. A task includes multiple subtasks, which are executed in a pipelined parallel manner through multiple computing units.

[0119] See Figure 3 The scheduler includes a monitoring module 301, a determination module 302, and a scheduling module 303. The monitoring module 301 monitors the device status of multiple computing units. The determination module 302, in response to determining that an idle unit exists among the multiple computing units based on the device status, determines a target task from the unexecuted subtasks that are independent of all currently executing subtasks, based on the next predetermined task of the idle unit. The scheduling module 303 assigns the target task to the idle unit for execution. The next predetermined task of the idle unit is determined based on a preset binding relationship for pipelined parallelism.

[0120] The specific processing of the monitoring module 301, the determination module 302, and the scheduling module 303 in the scheduler provided in this embodiment, and the resulting technical effects, can be found in the following references: Figure 1 The relevant descriptions of steps 101-102 in the corresponding embodiments will not be repeated here.

[0121] The task scheduler provided in this disclosure monitors the device status of each computing unit in real time through the monitoring module 301. This enables the determination module 302 to select subtasks unrelated to the currently executing subtasks from the pending subtasks as target tasks when a computing unit becomes idle. The scheduling module 303 then assigns these target tasks to the idle device for execution, filling the idle time of the unit and improving the utilization efficiency of computing resources and the execution efficiency of tasks.

[0122] In some embodiments, the determining module 302 is specifically used to: determine the idle window of the idle unit based on the next predetermined task of the idle unit; and select the target task from the subtasks to be executed based on the estimated idle time corresponding to the idle window.

[0123] In some embodiments, the determining module 302 is specifically used to: when there are subtasks in the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time, select subtasks in the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time to obtain the tasks to be scheduled; and determine the target task from the tasks to be scheduled.

[0124] In some embodiments, the determining module 302 is specifically used to: determine the target task from the tasks to be scheduled based on a preset task priority strategy.

[0125] In some embodiments, the determining module 302 is specifically used to: select one or more subtasks from the task to be scheduled based on the estimated execution time of each subtask in the task to be scheduled and the estimated idle time of the idle unit to obtain the target task; wherein, the sum of the estimated execution time of the selected subtasks is less than the estimated idle time, and the difference between the sum of the estimated execution time and the estimated idle time satisfies a preset difference condition.

[0126] In some embodiments, the determining module 302 is specifically used to: when the estimated execution time of all subtasks in the subtasks to be executed is greater than the estimated idle time, select the subtask with the shortest estimated execution time from the subtasks to be executed and determine it as the target task.

[0127] In some embodiments, the idle window includes an idle start time and an idle end time; the determining module 302 is specifically configured to: monitor the actual execution status of each subtask running on multiple computing units, and set the actual completion time of the current subtask of the idle unit as the start time of the idle window; determine the predecessor subtask that is directly dependent on the next predetermined task of the idle unit from the subtasks that are being executed; and use the estimated completion time of the predecessor subtask as the idle end time of the idle window.

[0128] In some embodiments, the idle window of the idle unit is used to identify the estimated idle duration of the idle unit; the determining module 302 is specifically used to: determine the actual execution duration of the current subtask of the idle unit based on the actual start time and actual end time of the current subtask of the idle unit; determine the estimated execution duration of the current subtask of the idle unit based on the next predetermined task of the idle unit, wherein the estimated execution duration is greater than the actual execution duration; and determine the estimated idle duration of the idle unit based on the estimated execution duration and the actual execution duration of the current subtask of the idle unit.

[0129] In some embodiments, the scheduler further includes a control module, which is used to migrate redundant data unrelated to the target task stored in the video memory of the idle unit to a preset cache when it is predicted that the video memory of the idle unit meets the preset conditions.

[0130] In some embodiments, the control module is further configured to, based on the storage address information of the required data of the target task, read the required data from the preset cache and load the required data into the video memory of the idle unit when the required data of the target task has been migrated to the preset cache.

[0131] In some embodiments, the preset conditions include: the required video memory usage of the target task is greater than the current available video memory capacity of the free unit or the current video memory utilization rate of the free unit is greater than a preset utilization rate threshold.

[0132] In some embodiments, the task includes a model training task, and the scheduler further includes a partitioning module. This partitioning module is used to divide the model training task based on a forward task, an input gradient computation task, and a parameter gradient computation task, resulting in multiple subtasks.

[0133] In some embodiments, the partitioning module is specifically used to: divide the model training task into one or more computational stage tasks; wherein each computational stage task includes a forward task, an input gradient calculation task, and a parameter gradient calculation task; and based on a preset partitioning strategy, divide each computational stage task into one or more subtasks to obtain multiple subtasks.

[0134] In some embodiments, after assigning the target task to an idle unit for execution, the scheduler further includes an update module for updating the device status of multiple computing units and the status information of the subtasks to be executed.

[0135] The specific implementation details and technical effects of the task scheduler embodiments provided in this disclosure are the same as the implementation details and technical effects of the task processing method embodiments described above, and will not be repeated here.

[0136] This embodiment exists as a device embodiment corresponding to the above method embodiment. The task scheduler provided in this embodiment, by monitoring the device status of each computing unit in real time, can select subtasks that are not dependent on the currently executing subtasks from the subtasks to be executed as target tasks when the computing unit is idle, and allocate the target tasks to the idle device for execution to fill the idle time of the idle unit, thereby improving the utilization efficiency of computing resources and the execution efficiency of tasks. Furthermore, it can also monitor the task queue containing subtasks to be executed in real time to realize the dynamic scheduling of subtasks to be executed. Furthermore, for model training tasks, based on dividing the task into forward computation and backward computation, it can also decouple the input gradient calculation task and the parameter gradient calculation task of backward computation, and further decompose the operator subtasks into finer-grained subtasks to improve the matching degree between the subtasks to be executed and the idle window of the idle unit. In addition, in the process of scheduling the target task to the idle unit, this disclosure can also comprehensively consider the estimated execution time of the quantum task, the idle window matching of the idle unit, and the real-time memory pressure to determine the target task. The task processing method disclosed herein integrates a complete system of dynamic monitoring, fine-grained decomposition, dynamic scheduling, and active memory management, which improves the efficiency of task processing and reduces the idle time of computing units during task processing.

[0137] According to embodiments of this disclosure, this disclosure also provides a task scheduling system.

[0138] Figure 4This is a structural block diagram of a task scheduling system provided in an embodiment of the present disclosure. A task includes multiple subtasks.

[0139] See Figure 4 The scheduling system 400 includes multiple computing units 401 and a scheduler 402. The multiple computing units 401 are used to execute multiple subtasks in a pipelined parallel manner; the scheduler 402 is used to schedule the multiple subtasks to the multiple computing units 401 for execution according to the task processing method of any of the above embodiments.

[0140] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the task processing method described in any of the above embodiments when executed.

[0141] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 5 As shown, the electronic device 500 of this embodiment includes a processor 501 and a memory 502; wherein, the memory 502 is used to store computer execution instructions; the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 500 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0142] In some embodiments, the memory 502 can be either standalone or integrated with the processor 501.

[0143] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0144] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0145] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0146] Bus 503 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0147] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of the task processing method in any of the above method embodiments.

[0148] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the task processing method according to any of the above embodiments.

[0149] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0150] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0151] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0152] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0153] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0154] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0155] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0156] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0157] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A task processing method, characterized in that, The task includes multiple subtasks, which are executed in a pipelined parallel manner through multiple computing units. The method includes: Monitor the device status of the multiple computing units; In response to determining that an idle unit has appeared among the plurality of computing units based on the device status, an idle window of the idle unit is determined based on the next predetermined task of the idle unit; based on the estimated idle duration corresponding to the idle window, a target task is determined from the unexecuted subtasks that have no dependency on all currently executed subtasks, and the target task is assigned to the idle unit for execution; The next predetermined task of the idle unit is determined based on the preset binding relationship of the pipeline parallelism.

2. The method according to claim 1, characterized in that, The step of determining the target task from the unexecuted subtasks that have no dependency on any currently executing subtask, based on the estimated idle time corresponding to the idle window, includes: If there is a subtask among the subtasks to be executed whose estimated execution time is less than or equal to the estimated idle time, then the subtasks whose estimated execution time is less than or equal to the estimated idle time are selected from the subtasks to be executed to obtain the tasks to be scheduled. The target task is determined from the tasks to be scheduled.

3. The method according to claim 2, characterized in that, Determining the target task from the tasks to be scheduled includes: The target task is determined from the tasks to be scheduled based on a preset task priority strategy.

4. The method according to claim 2, characterized in that, Determining the target task from the tasks to be scheduled includes: Based on the estimated execution time of each subtask in the task to be scheduled and the estimated idle time of the idle unit, one or more subtasks are selected from the task to be scheduled to obtain the target task; wherein, the sum of the estimated execution time of the selected subtasks is less than the estimated idle time, and the difference between the sum of the estimated execution time and the estimated idle time satisfies a preset difference condition.

5. The method according to claim 1, characterized in that, The step of determining the target task from the unexecuted subtasks that have no dependency on any currently executing subtask, based on the estimated idle time corresponding to the idle window, includes: If the estimated execution time of all subtasks in the pending subtasks is greater than the estimated idle time, the subtask with the shortest estimated execution time is selected from the pending subtasks and determined as the target task.

6. The method according to claim 1, characterized in that, The idle window includes an idle start time and an idle end time; The step of determining the idle window of the idle unit based on the next predetermined task of the idle unit includes: Monitor the actual execution status of each subtask running on the multiple computing units, and set the actual completion time of the current subtask in the idle unit as the start time of the idle window; From the subtasks currently being executed, determine the predecessor subtasks that directly depend on the next predetermined task of the idle unit; The estimated completion time of the preceding subtask is used as the idle end time of the idle window.

7. The method according to claim 1, characterized in that, The idle window of the idle unit is used to identify the estimated idle time of the idle unit; The step of determining the idle window of the idle unit based on the next predetermined task of the idle unit includes: The actual execution duration of the current subtask in the idle unit is determined based on the actual start time and actual end time of the current subtask in the idle unit. Based on the next predetermined task of the idle unit, the estimated execution time of the current subtask of the idle unit is determined, wherein the estimated execution time is greater than the actual execution time; The estimated idle time of the idle unit is determined based on the estimated execution time and actual execution time of the current subtask of the idle unit.

8. The method according to claim 1, characterized in that, The method further includes: If it is predicted that the video memory of the idle unit meets the preset conditions, the redundant data stored in the video memory of the idle unit that is unrelated to the target task will be migrated to the preset cache. The preset conditions include: the required video memory usage of the target task is greater than the current available video memory capacity of the idle unit or the current video memory utilization rate of the idle unit is greater than a preset utilization rate threshold.

9. The method according to claim 8, characterized in that, The method further includes: If the required data for the target task has been migrated to the preset cache, the required data is read from the preset cache based on the storage address information of the required data for the target task, and the required data is loaded into the video memory of the idle unit.

10. The method according to claim 1, characterized in that, The task includes a model training task, and the method further includes: The model training task is divided into multiple sub-tasks based on the forward task, the input gradient calculation task, and the parameter gradient calculation task.

11. The method according to claim 10, characterized in that, The model training task is divided into multiple sub-tasks based on a forward task, an input gradient calculation task, and a parameter gradient calculation task, including: The model training task is divided into one or more computational stage tasks; wherein each computational stage task includes the forward task, the input gradient calculation task, and the parameter gradient calculation task. Based on a preset partitioning strategy, each computational stage task is divided into one or more subtasks, resulting in the plurality of subtasks.

12. The method according to claim 1, characterized in that, After assigning the target task to the idle unit for execution, the method further includes: Update the device status of the multiple computing units and the status information of the subtasks to be executed.

13. A task scheduler, characterized in that, The task includes multiple subtasks, which are executed in a pipelined parallel manner through multiple computing units. The scheduler includes: The monitoring module is used to monitor the device status of the multiple computing units; A determination module is configured to, in response to determining that an idle unit has appeared among the plurality of computing units based on the device state, determine an idle window of the idle unit based on the next predetermined task of the idle unit; and determine a target task from the unexecuted subtasks that are independent of all currently executed subtasks based on the estimated idle duration corresponding to the idle window. The scheduling module is used to assign the target task to the idle unit for execution; wherein the next predetermined task of the idle unit is determined based on the preset binding relationship of the pipeline parallelism.

14. A task scheduling system, characterized in that, The task includes multiple sub-tasks, and the scheduling system includes: Multiple computing units are used to execute the multiple subtasks in a pipelined parallel manner; A scheduler is configured to schedule the plurality of subtasks to the plurality of computing units according to the processing method as described in any one of claims 1-12.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the processing method according to any one of claims 1-12.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the processing method according to any one of claims 1-12.

Images