Deep learning training job combination scheduling system based on staggered execution

Abstract

The invention provides a deep learning training job combination scheduling system based on staggered execution. Comprising three components including a deep learning training operation controller, an operation combination scheduler and an operation group interlaced execution controller, and an operation group interlaced execution actuator. For a deep learning training job submitted by a user, the system firstly creates resource use characteristic data during operation of a Profile task collection job through a training job controller and inputs the data into a job combination scheduler; the job combination scheduler outputs a combination result of job group configuration and a job group selection node type to the job group staggered execution controller through a self-defined combination scheduling algorithm according to resource use characteristics; and finally, an operation group interlaced execution controller creates an operation group training task according to a scheduling result to control an executor for operation group interlaced execution to perform operation execution. Compared with a distributed training mode that a single training job occupies resources exclusively, the operation mechanism that the shared resources are executed in a staggered mode can remarkably improve the execution efficiency of the job.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing and distributed systems, and more specifically, to a deep learning training job combination scheduling system based on interleaved execution. Background Art

[0002] With the continuous development and industrial implementation of smart manufacturing, autonomous driving, smart healthcare and other fields, the development and deployment of artificial intelligence applications has an increasing demand for computing power. General computing systems and cluster systems represented by CPUs cannot meet the computing power requirements of compute-intensive, highly parallelized deep learning training operations. With the rapid development of cloud-native technologies represented by containers, deploying and hosting applications in a public cloud-native environment has become the new normal for enterprise IT architecture and operations. However, the overall resource utilization rate of global data centers is currently close to 10-20%.

[0003] With the development of hardware technology, a single deep learning training job running in a cluster is usually unable to fully utilize exclusive hardware resources. Deep learning training jobs are usually executed in a multi-round iterative mode during operation. Each round of iteration is divided into multiple different stages. These stages have different resource bottlenecks. For example, the data loading stage mainly uses disk IO and CPU resources, the forward and reverse propagation calculation stage mainly uses GPU resources, and the gradient synchronization stage mainly uses network resources. This also leads to the periodic occupation and idleness of resources in multiple dimensions during deep learning training, which in turn leads to a large amount of idle resources in the computing cluster.

[0004] Currently, the industry and academia have proposed the following solutions to the GPU resource scheduling problem:

[0005] 1. Implement container orchestration and scheduling of GPU resources based on Kubernetes and Device-Plugin

[0006] Kubernetes is the current mainstream container orchestration and scheduling architecture and is used by major cloud vendors as the infrastructure for large-scale cluster container orchestration and resource scheduling. Nvidia has launched the Nvidia-Device-Plugin plug-in for Kubernetes, which supports scheduling GPU resources in Kubernetes clusters. The Kubernetes scheduling framework implements various scheduling strategies through plug-ins. The default scheduling strategy built into Kubernetes selects the appropriate node for the Pod and binds the corresponding resources based on task priority, affinity, stains, etc. At the same time, the Kubernetes community provides scheduling strategies such as topology awareness for GPU resource scheduling to further optimize GPU resource scheduling.

[0007] Based on Kubernetes and device-plugin, Huawei has open-sourced Volcano, a Kubernetes scheduler for batch workloads such as high-performance computing (HPC) and deep learning training, for deep learning jobs. It provides a set of advanced scheduling and resource management functions to meet the needs of large-scale computing and data processing tasks, and provides hybrid scheduling capabilities for heterogeneous devices including CPUs and GPUs based on various mainstream architectures. Volcano provides strategies such as gang scheduling based on the granularity of Job, and supports user-defined plugins and actions to implement more scheduling algorithms.

[0008] 2. GPU sharing scheduling solution based on GPU isolation technology

[0009] Nvidia-Device-Plugin only supports scheduling GPU resources on Kubernetes, but does not provide isolation when multiple Pods share the same GPU resource. It is only allocated to Pods in a whole-card binding manner. In order to better support GPU resource scheduling and improve resource utilization, Nvidia provides a variety of GPU isolation and sharing solutions. MPS (Multi-Process Service) and MIG (Multi-Instance GPU) are technologies provided by NVIDIA to improve GPU utilization. MPS merges the contexts of multiple jobs on the same GPU instance at the software level and converts them into a single application context to run on the GPU, avoiding the overhead of scheduling different contexts; MIG can split a single high-performance GPU into multiple smaller sub-GPUs and assign them to different tasks. Through hardware-level support, it splits the existing, cached, and computing cores into multiple instances, and each instance is independent of each other to avoid interference between jobs. Both improve GPU efficiency and flexibility in multi-task and multi-user scenarios through GPU sharing technology.

[0010] 3. Distributed Deep Learning Training Framework

[0011] The existing deep learning training frameworks are mainly Torch and TensorFlow. PyTorch is an open source deep learning framework developed by Facebook, which allows users to flexibly build and modify computational graphs at runtime. TensorFlow is an open source deep learning framework developed by the Google Brain team, which supports the full process of deep learning development from research to production. It has a wide range of tools and ecosystems to support visualization and service deployment, including TensorBoard, TensorFlow Lite, TensorFlow Serving, etc., which are suitable for large-scale distributed training and model deployment in production environments.

[0012] Common distributed training frameworks include Parameter Server and Distributed Data Parallel (DDP). Parameter Server is a distributed machine learning framework that stores model parameters on one or more servers, supports parallel calculation and parameter update of multiple working nodes, is suitable for large-scale machine learning tasks, and provides asynchronous and synchronous update mechanisms and fault tolerance. DDP is a distributed training technology that copies the model to multiple GPUs, processes different data batches in parallel, synchronizes gradients through efficient communication operations, and ensures that the model parameters of all GPUs are consistent. It is suitable for accelerating the training of deep learning models in multi-GPU environments.

[0013] 4. Deep learning training job operation mode based on interleaved execution

[0014] In order to improve resource utilization and reduce resource fragmentation, Muri improves resource utilization by combining different jobs and executing them in an interleaved mode based on the different resource bottleneck characteristics of deep learning training jobs at different stages. The execution of deep learning training jobs usually presents a multi-round iterative training mode. In each round of iteration, the job executes the three stages of data loading, forward and backward propagation, and communication in a fixed order. The main resources required for each of the three stages are different, usually corresponding to: disk IO and CPU, GPU, and network. In the assumption of the interleaved execution mode, each job only uses one main resource at the same time, and the other resources are idle. Interleaved execution staggers the execution stages of different jobs. For example, when job A executes forward and backward propagation calculations, job B executes gradient synchronization communication. Therefore, interleaved execution staggers the execution stages of different jobs in the same period, ensuring that the main resources occupied by different jobs are different, avoiding resource competition and improving resource utilization. By modifying the runtime framework to set barriers between different stages, it can be ensured that the main resources used by jobs in the same period are staggered. This method enables jobs to make more full use of multi-dimensional resources in the cluster and realizes the concurrent execution of multiple jobs with complementary resources of different dimensions as much as possible.

[0015] The deep learning job training system implemented by the above solution still has the following problems:

[0016] 1.Kubernetes supports the management of GPU resources through Nvidia-Device-Plugin and allocates GPU resources through scheduling plug-ins. However, the current Kubernetes scheduling framework does not support the scheduling mode of multiple job combinations. When designing job combination scheduling, we usually need to consider which jobs in a batch of jobs are suitable for combination and which node the resulting job group is suitable for allocation to. However, the Kubernetes scheduling framework allocates resources based on pod as the basic granularity. In the scheduling scenario of job combination, it is necessary to rely on global job and resource information to make scheduling decisions. The scheduler often needs to obtain information about a batch of jobs to realize job combination. The plug-in interface exposed by the Kubernetes scheduling framework usually only contains independent pod and node information, which is not enough to implement the mechanism of combined scheduling. Therefore, the native Kubernetes scheduling framework is not suitable for the scenario of job combination scheduling.

[0017] 2. Although the GPU sharing scheduling scheme based on GPU isolation technology supports resource isolation for GPU computing resources and video memory resources, and allows users to customize resource requirements and schedule according to resource requirements. However, this isolation scheme is still a coarse-grained isolation mechanism. Through MPS or MIG, multiple training tasks share the same GPU resources. Although these mechanisms solve the problem that a single training task cannot fully utilize the exclusive GPU to a certain extent. However, these methods still cannot avoid the problem of resource fragmentation caused by the periodic use of resources. In MPS, the occupation of the shared GPU by the job is still scheduled by hardware, and MIG only divides the GPU, so the job will still be trained in a mode of periodic resource use, and resource fragmentation will still occur. None of these existing methods can fundamentally solve the problem of resource fragmentation in deep learning training jobs during the training process.

[0018] 3. The current distributed training runtime framework for deep learning training jobs often only implements strategies such as model parallelism, data parallelism, and pipeline parallelism for a single job. Most jobs use frameworks such as parameter servers or DDP for distributed training. The existing deep learning framework training process still uses various resources periodically, and there is resource fragmentation during the training process. At the same time, different training jobs are not aware of each other at the framework level, so under the existing framework, different jobs cannot improve resource utilization through interleaved execution. The existing distributed training framework cannot meet the needs of multiple jobs sharing resources and fine-grained control of the use of resources by different tasks to reduce resource fragmentation. We need to implement a training framework for multi-task interleaved execution so that system resources can be more fully utilized. Summary of the invention

[0019] The purpose of the embodiments of the present disclosure is to provide a deep learning training job combination scheduling system based on interleaved execution. The main functions implemented by the system include data collection during job runtime, combined scheduling, and interleaved execution of job groups. The scheduling system collects the performance of training jobs in different scenarios, including various resource utilization characteristics such as CPU, GPU, and communication; provides users with a plug-in interface for customizing job combination scheduling strategies; and implements an interleaved execution training framework so that the job group runs in an interleaved execution mode in the container. The invention integrates job control and combined scheduling algorithms into system modules based on the Kubernetes system to implement job combination scheduling, and implements the operating mechanism of interleaved execution of job groups based on the Torch DDP distributed training framework. While improving the utilization of cluster resources, the performance interference caused by task sharing resources is minimized as much as possible to ensure the execution efficiency of training tasks and improve system throughput.

[0020] In a general aspect, a deep learning training job combination scheduling system based on interleaved execution is provided, comprising three components: a deep learning training job controller, a job combination scheduler, a job group interleaved execution controller, and an executor for the interleaved execution of the job group;

[0021] The deep learning training job controller includes a runtime data collector, the job combination scheduler includes a job combiner and a job scheduler, the job group interleaved execution executor is implemented based on the Torch DDP distributed training framework, and the training job is controlled according to the configuration of the distributed training job group to execute the training process in an interleaved execution mode;

[0022] For deep learning training jobs submitted by users, the system first creates a Profile task through the training job controller to collect the runtime resource usage feature data of the job and inputs it into the job combination scheduler. The job combination scheduler outputs the combination result of the job group configuration and the node type selected by the job group to the job group interleaved execution controller based on the resource usage characteristics through a customized combination scheduling algorithm. Finally, the job group interleaved execution controller creates a job group training task based on the scheduling result to control the job group interleaved execution executor to execute the job.

[0023] The deep learning training job controller is used to manage the execution process of jobs submitted by users. Users submit training tasks through the MLJob CRD template. The Spec template of MLJob CRD defines the type of deep learning training job submitted by the user, the hyperparameters required for training, and the system resource demand constraints; the deep learning training job instance submitted by the user needs to include the model type, batch size, learning rate, iteration rounds, learning rate, data set, GPU quantity requirements, and expected completion time for deep learning training; the Status status template of CRD defines the running status of the job submitted by the user, including the current stage of the job and the Profile task job sub-status;

[0024] The deep learning training job controller is responsible for maintaining the training jobs submitted by the user and controlling the changes of the deep learning training jobs in the waiting state, Profile state, scheduling state, and completion / failure state. The job state flow method controlled by the deep learning training job controller is as follows: after the user submits the job, the job is in the Pending state, and the state is changed according to whether there is runtime resource usage time data corresponding to the job in the Etcd database; if there is no historical resource data in the system Etcd database, the job will enter the Profiling stage, in which the job creates a Profile task to collect the runtime resource usage time information of the job; if there is historical Profile data in the system or the job completes the Profiling stage, the job enters the Scheduling stage, in which the job determines the job combination method and the node type selected by the job according to the user-defined combination scheduling algorithm; after the job is successfully scheduled to the node for execution, the job enters the Executing stage to execute the actual training task; during the training task process, if the execution is successful, the job enters the Success stage, and if an exception occurs during the execution process, it enters the Fail execution failure state.

[0025] The job combination scheduler is responsible for combining the deep learning training job resources MLJob waiting to be scheduled in the system into job group JobPack resources, allocating node resources of the cluster to the job group, and maintaining the entire job running view in the system. The job combination scheduler is divided into three modules: event distribution, combination scheduling, and resource cleanup;

[0026] The event distribution module receives the event maintenance resource view sent by the deep learning training job controller and the interleaved execution job group controller, the combined scheduling module performs combined scheduling on the jobs waiting to be scheduled in the system, and the resource cleaning module is used to clean up expired system resources;

[0027] The working process of the combined scheduler is as follows: first, receive an event and determine the event type. If it is a job enqueue event, then check whether the job meets the enqueue conditions. If so, the job enters the waitQueue and updates the relevant resource view. If it is a job group execution completion event, then synchronize the corresponding MLJob resource instance status, let the relevant job enter the finishQueue and update the relevant resource view. If it is a deletion event, delete the corresponding job and job group and update the relevant resource view.

[0028] The combined scheduler periodically or responsively executes the combined scheduling plug-in for the jobs waiting to be scheduled in the system, combines the jobs and allocates node resources according to the combined scheduling strategy defined by the user; periodically executes the combined scheduling cycle through a timer or responsively triggers the scheduling cycle according to the job queue entry and completion events; the process of the scheduling cycle is: the combined scheduler first obtains the Profile data of the job in the waitQueue; passes the Profile resource data and the waitQueue queue to the job combination plug-in to combine the deep learning training jobs into job groups; passes the combined job group list, the executing job list executeList and the system resource information to the job scheduling plug-in to obtain the job group resource allocation result; according to the scheduling result, an interleaved execution job group JobPack resource instance is created for the job group that is successfully allocated to the resources, and the jobs that are not allocated to the resources are returned to the waitQueue to wait for the next scheduling; finally, the system resource view is updated.

[0029] The job group interleaved execution controller specifies the job group interleaved execution configuration by designing the JobPack CRD Spec template. The JobPack Spec template defines the job group resource requirements and the template of a single deep learning training job in the job group, and creates a JobPack instance containing the number of nodes, number of cards, and GPU type for distributed training; and needs to create a Status template to define the running status of the job group, including the current stage of the job, the status of each worker of distributed training, and the associated MLJob resources;

[0030] The interleaved execution job group controller is used to maintain the training jobs submitted by the user, control the changes in the waiting state, running state, and completion / failure state of the deep learning training job group, and the states of JobPack and MLJob are associated and corresponded; the resource state flow process controlled by the interleaved execution job group controller and the corresponding MLJob resource state flow process is as follows: when the combined scheduler completes the combined scheduling of the job to create the JobPack resource, the JobPack resource instance is in the Pending waiting state corresponding to the scheduling state of the MLJob; JobPack then creates the training task according to the resource configuration and the job template and enters the running state corresponding to the execution stage of the MLJob; after the execution of the job group is completed, if all the tasks in the job group are successfully executed, the JobPack enters the success state corresponding to the success stage of each MLJob; if an exception occurs during the execution process, the JobPack enters the failure state and the corresponding MLJob also enters the failure stage;

[0031] In each round of coordination cycle of the interleaved job group controller, the interleaved job group controller parses the Spec template of the JobPack CRD to obtain the job distributed training configuration and parameters related to model training; if the job group training task has not been created, the interleaved job group controller starts the Job as the master node of the distributed training, creates a workerPod according to the distributed training configuration, and creates a Service for communication between training nodes; the interleaved job group controller creates or updates the ConfigMap according to the parsed training configuration and model training parameters and mounts it to the created Pod; finally, the interleaved job group controller synchronizes the status of the training node to the Status status of the JobPack resource.

[0032] The implementation method of the executor of the staggered execution of the job group is as follows: multiple jobs are controlled to complete the training tasks in a group of Pod containers according to a preset staggered execution mode. The distributed training tasks in the job group implement data parallel distributed training through the DDP distributed training framework on different nodes. Different training task subprocesses in the same node implement the staggered execution operation mechanism through the executor; the startup program of the operation framework obtains the job type, hyperparameter configuration and staggered execution order by reading the ConfigMap mounted in the container; the executor of the staggered execution of the job group starts a DDP distributed training process for each training task, and loads the training model according to the job type and hyperparameter configuration and sets the hyperparameters in the training process, and sets the startup order of the jobs according to the staggered execution order of the jobs to start the training tasks;

[0033] During the iterative training process, the executor of the interleaved execution of the job group splits the iterative process of each training job, and splits each round of iteration into three stages: data loading and preprocessing stage, forward and backward propagation calculation stage, and gradient synchronization communication stage according to the resource usage characteristics of the deep learning training job; the controller of the interleaved execution job group sets a barrier between two adjacent stages, and the training process will suspend and stop execution when it reaches the barrier. Only when all the training tasks before the same barrier complete the current training stage and reach the position of the barrier, can each sub-process enter the next training stage;

[0034] The barriers of different training processes on the same node are implemented through the inter-process pipe communication mechanism. A training process will establish a named pipe with the training processes of all other jobs on the node. When executing to the barrier, the training process will perform write operations to all other training processes. All processes except the training process will perform read operations on the training process when executing to the barrier. This mechanism is used to implement the synchronization mechanism of interleaved execution of different training jobs.

[0035] The innovation of the embodiment of the present invention lies in:

[0036] The deep learning training job combination scheduling system for interleaved execution implemented by the present invention combines jobs into job groups for execution. Compared with the distributed training mode in which a single training job monopolizes resources, the operation mechanism of interleaved execution of shared resources can significantly improve the execution efficiency of jobs. The performance of distributed training jobs reasonably combined into job groups through an interleaved execution mode on shared computing resources is significantly better than the performance of individual jobs executed individually. We combine some common deep learning tasks in pairs. The time it takes for a job to execute one round of iterations alone is extended compared to the time it takes for an interleaved execution executor to run one round of iterations in an interleaved execution manner, as shown in Table 7. It can be found that except for the A2C training job, which will result in a slight performance degradation in the two-by-two combination, all other combination types can significantly improve the overall execution efficiency of training jobs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0038] Figure 1 is a schematic diagram showing an architecture diagram of a deep learning training job combination scheduling system based on interleaved execution according to an embodiment of the present disclosure;

[0039] Figure 2 is a flowchart showing the operation of a deep learning training job according to an embodiment of the present disclosure;

[0040] Figure 3 is a diagram showing a MLJob resource state flow diagram according to an embodiment of the present disclosure;

[0041] Figure 4 is a flowchart showing the execution of an event distribution module according to an embodiment of the present disclosure;

[0042] Figure 5 is a flowchart showing a combined scheduling module scheduling cycle according to an embodiment of the present disclosure;

[0043] Figure 6 is a diagram showing the JobPack resource state flow and the corresponding MLJob resource state according to an embodiment of the present disclosure;

[0044] Figure 7 is a schematic diagram showing an implementation principle of interleaved execution according to an embodiment of the present disclosure;

[0045] Figure 8 is a diagram showing a comparison of execution time of a real cluster test training job according to an embodiment of the present disclosure;

[0046] Fig. 9 2 is a diagram showing the architecture of a deep learning training job combination scheduling system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0048] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0049] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0050] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0051] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0052] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0053] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0054] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0055] Figure 1 is a schematic diagram showing a deep learning training job combination scheduling system based on interleaved execution according to an embodiment of the present disclosure.

[0056] The present invention is based on the Kubernetes container orchestration system to implement a deep learning training job combination scheduling system based on interleaved execution. The combined scheduling system combined scheduler includes three components: a deep learning training job controller MLJob Controller, a job combination scheduler JobPack Operator, and an interleaved execution job group controller JobPack Controller. The invention implements an executor for interleaved execution of job groups based on the Torch DDP distributed training framework. The executor controls the training job according to the configuration of the distributed training job group to execute the training process in an interleaved execution mode.

[0057] The deep learning training job submitted by the user first creates a Profile task to collect the runtime resource usage characteristics of the job, outputs the combination result of the job and selects the node type of the job group based on the resource usage characteristics through a custom combination scheduling algorithm, and finally creates a job group training task to control the execution of the job based on the scheduling result. The overall system operation process is as follows: Figure 2 shown.

[0058] Deep learning training job controller:

[0059] MLJob Controller is used to manage the execution process of jobs submitted by users. Users submit training tasks through the MLJob CRD template. The Spec template design of MLJob CRD is shown in Table 1. The template defines the type of deep learning training job submitted by the user, the hyperparameters required for training, and the system resource demand constraints. The deep learning training job instance submitted by the user needs to include the model type, batch size, learning rate, iteration round, learning rate, data set, GPU quantity requirement, and expected completion time for deep learning training. The Status status template of CRD defines the running status of the job submitted by the user, mainly including the current stage of the job and the Profile job sub-status. The design of the MLJob Status of the deep learning training job is shown in Table 2.

[0060] Table 1 MLJob CRD Spec template design

[0061]

[0062] Table 2MLJob CRD Status design

[0063] Fields illustrate Phase The current stage of the deep learning training job ProfileJobStatus Profile task status table for deep learning training jobs

[0064] The deep learning training job controller MLJob Controller is responsible for maintaining the training jobs submitted by users and controlling the changes in the waiting state, profile state, scheduling state, and completion / failure state of deep learning training jobs. The job state flow diagram controlled by MLJobController is as follows: Figure 3 As shown. After the user submits the job, the job is in the Pending state, and changes state according to whether there is runtime resource usage time data corresponding to the job in the Etcd database; if there is no historical resource data in the system Etcd database, the job will enter the Profiling stage, in which the job creates a Profile task to collect the runtime resource usage time information of the job; if there is historical Profile data in the system or the job completes the Profiling stage, the job enters the Scheduling stage, in which the job determines the job combination method and the node type selected by the job according to the user-defined combination scheduling algorithm; after the job is successfully scheduled to the node for execution, the job enters the Executing stage to execute the actual training task; during the training task, if the execution is successful, the job enters the Success stage, and if an exception occurs during the execution process, it enters the Fail state.

[0065] The profile data of resource usage time when the job is running is persistently stored in the Etcd key-value database. The key of each item in the database is a string path containing the job model type, resource configuration, hyperparameter settings, and resource type, in the form of / Profile / {Model} / {BatchSize} / {GPU Request} / {GPU Type} / {Resource}. The value of each item is the time the job uses the resource in a single round of iteration. The database design is shown in Table 3. Model identifies the model type of the job, BatchSize identifies the job batch size, GPU Request / Type identifies the number of GPUs and GPU types required for the distributed training of the job; Resource identifies the resource types used in the training process, which usually include CPU, disk IO, GPU, and network communication. This string combination can uniquely identify the resource usage characteristics of a class of deep learning training jobs.

[0066] Table 3 Etcd database Profile data storage design

[0067]

[0068] The Kubernetes Controller maintains the resource status through a tuning loop to ensure that related resources can be created, updated, and deleted normally. The Controller observes the resource instances of the MLJob created by the user in the cluster through the ListAndWatch method.

[0069] For jobs in the Pending state, the controller queries the Etcd database to see whether there is resource usage time data corresponding to the job. If there is corresponding runtime resource usage time data in the database, the controller sends the job resource information to the combined scheduler, and the job enters the scheduling phase and ends this round of tuning process; if there is no corresponding runtime resource usage time data in the database, the controller initializes the Profile task according to the job resource configuration, and creates the Pod, Service, podGroup and other resources required for the distributed task. The training task creates a Pod for each worker, and provides node discovery between Pods through the Service to ensure communication between working nodes. The Pod of the training task is bound to the volcano podGroup to implement gang scheduling. After the resources are created, the job enters the Profile phase and ends this round of scheduling cycle.

[0070] For jobs in the Profile stage, the controller queries the Profile tasks of the job in the system, checks the execution status of the Profile tasks, and updates the execution status of the Profile tasks in MLJob Status. If all Profile tasks are completed, the controller sends the MLJob job resource information to the combined scheduler, and the job enters the scheduling stage, ending this round of tuning process.

[0071] For jobs in the scheduling and execution phase, the controller receives job status change information from the scheduler. Based on the data information of the combined scheduler, the job status is updated to the execution phase, successful execution, or abnormal execution. The job execution status is synchronized to MLJob Status and fed back to the user, ending the tuning cycle.

[0072] Job combination scheduler:

[0073] The job combination scheduler is responsible for combining the deep learning training job resources MLJob waiting to be scheduled in the system into job group JobPack resources, allocating node resources of the cluster to the job group, and maintaining the overall job running view in the system. The combination scheduler can be mainly divided into three modules: event distribution, combination scheduling, and resource cleanup. They are responsible for receiving events sent by the deep learning training job controller and the interleaved execution job group controller to maintain the resource view, perform combination scheduling on jobs waiting to be scheduled in the system, and clean up expired system resources. At the same time, the combination scheduler can support user-defined plugins to implement richer job combination algorithms and scheduling algorithms. By implementing specific interfaces, plugins can be connected to the combination scheduler framework to implement advanced combination scheduling algorithms.

[0074] The data structure of the view of related system resources used for maintenance in the combined scheduler is shown in Table 4. The waitQueue waiting queue is used to maintain the deep learning training job resources MLJob submitted by the user and waiting for combined scheduling; the executeList execution list is used to maintain the job group resources JobPack running in the system; the finishQueue completion queue is used to maintain the deep learning training job related resources that have been executed in the system; mlJobStatusMap records the mapping of deep learning training job names to job resource instances; packStatusMap records the mapping of job group names to job group resource instances; jobRecords records the mapping of deep learning training job names to job execution related indicators.

[0075] Table 4 Combined scheduler system resource structure design

[0076]

[0077] The event distribution module is responsible for receiving resource status change events sent by the deep learning training job controller MLJob Controller and the staggered execution job group controller JobPack Controller, and updating the system resource view in the combined scheduler according to the event type and related data of the event. The execution process of the event distribution module is as follows: Figure 4 After receiving the event, the event distribution module performs operations according to the event type: if it is an enqueue event created by the MLJob Controller, it determines whether the job meets the enqueue conditions. If it does, it adds the job to the waitQueue to wait for combined scheduling, and updates the relevant resource view; if it is a job group execution completion event, it synchronizes the execution status to the training job MLJob resource instance corresponding to the job group according to the job group execution result, and the relevant job enters the finishQueue, and updates the resource view; if it is a resource deletion event, it deletes the corresponding job or job group resources and synchronizes the relevant resource view.

[0078] The combined scheduling module periodically or responsively executes the combined scheduling plug-in for the jobs waiting to be scheduled in the system, combines the jobs and allocates node resources according to the combined scheduling strategy defined by the user. The combined scheduling module periodically executes the combined scheduling cycle through a timer or triggers the scheduling cycle responsively according to the job queue and completion events. Figure 5 As shown. The combined scheduler first obtains the Profile data of the job in the waitQueue; passes the Profile resource data and the waitQueue queue to the job combination plug-in to combine the deep learning training jobs into job groups; passes the combined job group list, the executing job list executeList, and the system resource information to the job scheduling plug-in to obtain the job group resource allocation result; creates an interleaved execution job group JobPack resource instance for the job group that is successfully allocated to the resources according to the scheduling result, and returns the jobs that are not allocated to the resources to the waitQueue to wait for the next scheduling; finally, updates the system resource view.

[0079] The resource cleanup module deletes resource instances that have been executed and are no longer needed in the system and maintains the resource view in the combined scheduler. The resource cleanup module periodically checks the resource objects in the finishQueue. If the related jobs and job group resources are judged to be expired, the related resources involved are deleted and the resource view of the combined scheduler is updated.

[0080] Job group interleaved execution controller:

[0081] The JobPack Controller is used to manage the execution process of the job group generated by the combined scheduler. The JobPack CRD Spec template specifies the job group interleaving execution configuration. The JobPack Spec defines the job group resource requirements and the template of a single deep learning training job in the job group. The JobPack Spec template design is shown in Table 5. The created JobPack instance needs to include the number of nodes, the number of cards, and the GPU type for distributed training. The JobPack Status template defines the running status of the job group, mainly including the current stage of the job, the status of each worker of distributed training, and the associated MLJob resources. The JobPack CRD Status design is shown in Table 6.

[0082] Table 5 JobPack CRD Spec template design

[0083]

[0084]

[0085] Table 6 JobPack CRD Status design

[0086]

[0087] The JobPack Controller is responsible for maintaining the training jobs submitted by users and controlling the changes in the waiting, running, and completed / failed states of the deep learning training job group. The JobPack and MLJob states are related and corresponding. The resources controlled by the JobPack Controller and the corresponding MLJob resource state flow diagram are shown in the figure below. Figure 6 As shown. When the combined scheduler completes the combined scheduling of the job and creates the JobPack resource, the JobPack resource instance is in the Pending waiting state, which corresponds to the scheduling state of the MLJob; JobPack then creates the training task according to the resource configuration and the job template and enters the running state, which corresponds to the execution stage of the MLJob; after the execution of the job group is completed, if all the tasks in the job group are successfully executed, JobPack enters the successful state, which corresponds to the successful stage of each MLJob; if an exception occurs during the execution process, JobPack enters the failed state, and the corresponding MLJobs also enter the failed stage.

[0088] The JobPack Controller is responsible for maintaining the resource instances required for the execution of the job group so that they can be created normally. In each round of coordination loop of the JobPack controller, the controller parses the Spec template of the JobPack CRD to obtain the job distributed training configuration and parameters related to model training. If the job group training task has not been created, the controller starts the Job as the master node for distributed training, creates a worker Pod according to the distributed training configuration, and creates a Service for communication between training nodes. The controller creates or updates the ConfigMap based on the parsed training configuration and model training parameters and mounts it to the created Pod. Finally, the controller synchronizes the status of the training node to the Status status of the JobPack resource.

[0089] Executor for interleaved execution of job groups:

[0090] After the job interleaved execution controller creates a task resource instance for the deep learning training job submitted by the user, distributed training is performed on multiple Pods of the training task. The interleaved execution executor controls multiple jobs in a group of Pod containers to complete the training tasks in accordance with the preset interleaved execution mode. The distributed training tasks in the job group implement data-parallel distributed training through the DDP distributed training framework on different nodes, and the different training task sub-processes in the same node implement the interleaved execution operation mechanism through the executor. The startup program of the running framework obtains the job type, hyperparameter configuration, and interleaved execution order by reading the ConfigMap mounted in the container. The executor starts a DDP distributed training process for each training task, loads the training model according to the job type and hyperparameter configuration, and sets the hyperparameters in the training process. It sets the job startup order according to the interleaved execution order of the job to start the training task. The implementation mechanism of the interleaved execution of different training sub-processes on the same node is as follows: Figure 7 shown.

[0091] During the iterative training process, the executor splits the iterative process of each training job. According to the resource usage characteristics of deep learning training jobs, each round of iteration can be divided into three stages: data loading and preprocessing, forward and backward propagation calculation, and gradient synchronization communication. The controller sets a barrier between two adjacent stages. The training process will suspend and stop execution when it reaches the barrier. Only when all training tasks before the same barrier complete the current training stage and reach the barrier position, can each sub-process enter the next training stage. For example, Figure 7When the Job1 training process reaches the gradient synchronization communication phase, the job reaches the barrier, and the training process is blocked before the barrier. At this time, the Job1 training process needs to wait for Job2 to complete the forward and backward propagation calculation phase and reach the barrier, and for Job3 to complete the data loading preprocessing phase and reach the barrier. Only then can the three training processes enter the next execution phase and repeat the above process until the training is completed.

[0092] The barriers between different training processes on the same node are implemented through the mechanism of inter-process pipe communication. The training process of Job1 will establish a named pipe with the training processes of all other jobs on the node. When executing to the barrier, the training process of Job1 will perform write operations to all other training processes, and all processes except Job1 will perform read operations on the training process of Job1 when executing to the barrier. Because the process of pipe communication requires the sender and the receiver to perform read and write operations respectively to complete, when only the sender writes the signal but the receiver does not read the signal or only the receiver reads the signal but the sender does not write the signal, the process will be blocked. Therefore, as long as one training process has not completed the current training stage, other training processes will be blocked before the next training stage because they cannot write or read signals, thus realizing the synchronization mechanism of interleaved execution of different training jobs.

[0093] Effect verification:

[0094] The present invention implements the job interleaving execution mechanism in the invention purpose section on the platform and framework respectively. The present invention integrates the corresponding modules in Kubernetes to implement a deep learning training job combination scheduling system for interleaving execution, and implements the executor of job group interleaving execution based on the job interleaving execution mechanism combined with the Torch DDP distributed training framework.

[0095] Table 7 Results of improving training efficiency by staggered execution of different jobs

[0096] job1\job2 Vgg19 Resnet50 ShuffleNet A2C Bert GPT2 Vgg19 1.571 1.18 1.323 1.34 1.509 1.441 Resnet50 1.045 1.401 1.34 1.215 1.171 ShuffleNet 1.477 1.335 1.279 1.276 A2C 0.959 1.459 1.349 Bert 1.396 1.267 GPT2 1.299

[0097] The job traces of the real production cluster are converted into MLJob resources and submitted to the invented job combination scheduling system, and Muri's multi-round flower tree job combination algorithm is implemented in the job combination plug-in. We use the experimental environment of a computing cluster containing 6 A10, 8 4090, and 6 A100 GPUs to improve the end-to-end execution efficiency through job combination. The average execution time of the job on different GPU types in the system end-to-end experiment is shown in the figure. Figure 8As shown in the figure, it can be found that the interleaved execution mechanism improves the operation efficiency of jobs in various operating environments compared to the way that jobs exclusively occupy resources and execute individually. At the same time, after combining jobs into job groups and interleaved execution, the total execution time is shortened by 13.8%.

[0098] Architecture of algorithm implementation:

[0099] The combined scheduling system scheduling controller designed by the present invention is built and implemented based on the mainstream container orchestration scheduling framework Kubernetes. The interleaved execution executor is developed and implemented based on the Torch DDP distributed training framework. The architecture of the combined scheduling system for deep learning training jobs is as follows: Fig. 9 shown.

[0100] The components involved in the system include the deep learning job controller MLJob Controller, the job combination scheduler JobPack Operator, the interleaved execution job group controller JobPack Controller, the database Etcd, the scheduler Volcano for high-performance computing scenarios, and the job interleaved execution executor. Kubernetes-related user-defined resources (CRDs) include MLJob and JobPack. MLJob corresponds to a single deep learning training job and is the smallest unit for user-submitted jobs. JobPack is a combination of one or more deep learning training jobs and is the smallest unit for actual training. The overall process from user submission of a job to the start of the job group running on the cluster is roughly divided into 9 steps:

[0101] 1. Using YAML files or other methods, users write MLJob resource type instances according to the MLJob CRD specifications and send them to the API Server to create deep learning training job resources.

[0102] 2. The MLJob Controller listens to the changes of MLJob type resources in the system through the API Server, and parses the model, hyperparameters, deadline, and other content required for the corresponding job for the newly created or updated MLJob type resources.

[0103] 3. Based on the content of MLJob, MLJob Controller creates one or more Profile jobs for obtaining job characteristics and executes them on the cluster. It collects the runtime resource information of the job and stores it persistently in the Etcd database.

[0104] 4. After the runtime resource usage time data of the MLJob is successfully collected, the MLJob Controller sends the MLJob resources to the JobPack Operator to wait for combined scheduling.

[0105] 5. The JobPack Operator calls the combined scheduling plug-in to combine one or more MLJobs into a JobPack, selects the appropriate computing resource type for the JobPack, creates the corresponding JobPack, and sends it to the API Server.

[0106] 6. The JobPack Controller monitors the changes of JobPack resources in the system from the API Server, and parses the distributed training configuration and interleaved execution training configuration of the corresponding training job group for the newly created or updated JobPack type resources.

[0107] 7. JobPack Controller creates distributed training tasks and related resources composed of multiple MLJobs based on the content of JobPack.

[0108] 8. Volcano Scheduler listens to the API Server for newly created training task resource information and schedules the job Pod to the physical node according to the gang scheduling strategy.

[0109] 9. After the scheduling is successful, the physical node will create a new Pod instance for job training and start the job distributed training process through the staggered executor to complete the training task.

[0110] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A deep learning training job combination scheduling system based on interleaved execution, characterized in that: It includes three components: deep learning training job controller, job combination scheduler, job group interleaved execution controller, and job group interleaved execution executor; The deep learning training job controller includes a runtime data collector, the job combination scheduler includes a job combiner and a job scheduler, the job group interleaved execution executor is implemented based on the Torch DDP distributed training framework, and the training job is controlled according to the configuration of the distributed training job group to execute the training process in an interleaved execution mode; For deep learning training jobs submitted by users, the system first creates a Profile task through the training job controller to collect the runtime resource usage feature data of the job and inputs it into the job combination scheduler. The job combination scheduler outputs the combination result of the job group configuration and the node type selected by the job group to the job group interleaved execution controller based on the resource usage characteristics through a customized combination scheduling algorithm. Finally, the job group interleaved execution controller creates a job group training task based on the scheduling result to control the job group interleaved execution executor to execute the job.

2. A deep learning training job combination scheduling system based on interleaved execution as claimed in claim 1, characterized in that: The deep learning training job controller is used to manage the job execution process submitted by the user; The user submits a training task through the MLJobCRD template. The Spec template of the MLJob CRD defines the type of deep learning training job submitted by the user, the hyperparameters required for training, and the system resource requirement constraints. The deep learning training job instance submitted by the user needs to include the model type, batch size, learning rate, iteration round, learning rate, data set, GPU quantity requirement, and expected completion time for deep learning training. The Status template of the CRD defines the running status of the job submitted by the user, including the current stage of the job and the Profile task job sub-status. The deep learning training job controller is responsible for maintaining the training jobs submitted by users and controlling the changes of deep learning training jobs in waiting state, Profile state, scheduling state, and completion / failure state; the job state flow method controlled by the deep learning training job controller is as follows: after the user submits the job, the job is in the Pending state, and the state changes according to whether there is runtime resource usage time data corresponding to the job in the Etcd database; if there is no historical resource data in the system Etcd database, the job will enter the Profiling stage, in which the job creates a Profile task to collect the runtime resource usage time information of the job; if there is historical Profile data in the system or the job completes the Profiling stage, the job enters the Scheduling stage, in which the job determines the job combination method and the node type selected by the job according to the user-defined combination scheduling algorithm; after the job is successfully scheduled to the node for execution, the job enters the Executing stage to execute the actual training task; During the training task process, if the execution is successful, the job enters the Success stage. If an exception occurs during the execution process, it enters the Failure state.

3. A deep learning training job combination scheduling system based on interleaved execution as claimed in claim 2, characterized in that: The job combination scheduler is responsible for combining the deep learning training job resources MLJob waiting to be scheduled in the system into job group JobPack resources, allocating node resources of the cluster to the job group, and maintaining the entire job running view in the system. The job combination scheduler is divided into three modules: event distribution, combination scheduling, and resource cleanup; The event distribution module receives the event maintenance resource view sent by the deep learning training job controller and the interleaved execution job group controller, the combined scheduling module performs combined scheduling on the jobs waiting to be scheduled in the system, and the resource cleaning module is used to clean up expired system resources; The working process of the combined scheduler is as follows: first, receive the event and determine the event type. If it is a job enqueue event, then check whether the job meets the enqueue conditions. If so, the job enters the waitQueue and updates the relevant resource view. If it is a job group execution completion event, then synchronize the corresponding MLJob resource instance status, let the relevant job enter the finishQueue and update the relevant resource view. If it is a deletion event, the corresponding job and job group are deleted and the related resource view is updated; The combined scheduler periodically or responsively executes the combined scheduling plug-in for the jobs waiting to be scheduled in the system, combines the jobs and allocates node resources according to the combined scheduling strategy defined by the user; periodically executes the combined scheduling cycle through a timer or triggers the scheduling cycle responsively according to the job queue entry and completion events; The process of the scheduling cycle is as follows: the combined scheduler first obtains the Profile data of the job in the waitQueue; passes the Profile resource data and the waitQueue queue into the job combination plug-in to combine the deep learning training jobs into job groups; passes the combined job group list, the executing job list executeList and the system resource information into the job scheduling plug-in to obtain the job group resource allocation result; creates an interleaved execution job group JobPack resource instance for the job group that is successfully allocated to the resources according to the scheduling result, and returns the jobs that are not allocated to the resources to the waitQueue to wait for the next scheduling; finally, updates the system resource view.

4. A deep learning training job combination scheduling system based on interleaved execution as claimed in claim 3, characterized in that: The job group interleaved execution controller specifies the job group interleaved execution configuration by designing the JobPack CRD Spec template. The JobPack Spec template defines the job group resource requirements and the template of a single deep learning training job in the job group, and creates a JobPack instance containing the number of nodes, number of cards, and GPU type for distributed training; and needs to create a Status template to define the running status of the job group, including the current stage of the job, the status of each worker of distributed training, and the associated MLJob resources; The interleaved execution job group controller is used to maintain the training jobs submitted by the user, control the changes in the waiting state, running state, and completion / failure state of the deep learning training job group, and the states of JobPack and MLJob are associated and corresponded; the resource state flow process controlled by the interleaved execution job group controller and the corresponding MLJob resource state flow process is as follows: when the combined scheduler completes the combined scheduling of the job to create the JobPack resource, the JobPack resource instance is in the Pending waiting state corresponding to the scheduling state of the MLJob; JobPack then creates the training task according to the resource configuration and the job template and enters the running state corresponding to the execution stage of the MLJob; after the execution of the job group is completed, if all the tasks in the job group are successfully executed, the JobPack enters the success state corresponding to the success stage of each MLJob; if an exception occurs during the execution process, the JobPack enters the failure state and the corresponding MLJob also enters the failure stage; In each round of coordination cycle of the interleaved job group controller, the interleaved job group controller parses the Spec template of the JobPack CRD to obtain the job distributed training configuration and parameters related to model training; if the job group training task has not been created, the interleaved job group controller starts the Job as the master node of the distributed training, creates a workerPod according to the distributed training configuration, and creates a Service for communication between training nodes; the interleaved job group controller creates or updates the ConfigMap according to the parsed training configuration and model training parameters and mounts it to the created Pod; Finally, the interleaved job group controller synchronizes the status of the training node to the Status status of the JobPack resource.

5. A deep learning training job combination scheduling system based on interleaved execution as claimed in claim 4, characterized in that: The implementation method of the executor of the staggered execution of the job group is as follows: multiple jobs are controlled to complete the training tasks in a group of Pod containers according to a preset staggered execution mode. The distributed training tasks in the job group implement data parallel distributed training through the DDP distributed training framework on different nodes. Different training task subprocesses in the same node implement the staggered execution operation mechanism through the executor; the startup program of the operation framework obtains the job type, hyperparameter configuration and staggered execution order by reading the ConfigMap mounted in the container; the executor of the staggered execution of the job group starts a DDP distributed training process for each training task, and loads the training model according to the job type and hyperparameter configuration and sets the hyperparameters in the training process, and sets the startup order of the jobs according to the staggered execution order of the jobs to start the training tasks; During the iterative training process, the executor of the interleaved execution of the job group splits the iterative process of each training job, and splits each round of iteration into three stages: data loading and preprocessing stage, forward and backward propagation calculation stage, and gradient synchronization communication stage according to the resource usage characteristics of the deep learning training job; the controller of the interleaved execution job group sets a barrier between two adjacent stages, and the training process will suspend and stop execution when it reaches the barrier. Only when all the training tasks before the same barrier complete the current training stage and reach the position of the barrier, can each sub-process enter the next training stage; The barriers of different training processes on the same node are implemented through the inter-process pipe communication mechanism. A training process will establish a named pipe with the training processes of all other jobs on the node. When executing to the barrier, the training process will perform write operations to all other training processes. All processes except the training process will perform read operations on the training process when executing to the barrier. This mechanism is used to implement the synchronization mechanism of interleaved execution of different training jobs.

Images