Method, system and apparatus for scheduling distributed training tasks for intelligent computing

Abstract

The present invention provides a method, system and apparatus for scheduling distributed training tasks for intelligent computing. [Solution] The system includes a model performance prediction and decomposition module, a global GPU resource scheduler, and a local GPU resource scheduler that is arranged in each computing node. After receiving the subtask request sent by the model performance prediction and decomposition module, the global GPU resource scheduler assigns each subtask to the GPU of the corresponding computing node to train according to the resource consumption information of each subtask and the GPU running status of multiple computing nodes, and establishes a communication topology between each subtask. In the process of each computing node's GPU training the corresponding subtask, the GPU of each computing node monitors the computing resource running status of each computing node, and controls the scheduling of subtasks according to the computing resource running status of the GPU of all computing nodes. The present invention can improve the utilization rate of resources such as GPU and network of a computing cluster and shorten the waiting time of subtask training.

Description

[Technical field]

[0001] The present invention relates to the field of intelligent computing, and in particular to a method, system and apparatus for scheduling distributed training tasks for intelligent computing. [Background technology]

[0002] The emergence of deep learning brings great updates to fields such as natural language processing, audio / video processing, and media convergence, but as deep learning models become larger, large models with parameters exceeding tens of billions will appear, and such large-scale models always need to complete model training by building a distributed machine learning system. At the same time, due to the limited computing power of a single GPU, it has become very common to accelerate model training by building a distributed training method on multiple machines and multiple GPU cards during model training.

[0003] In distributed training, a computing task is divided into multiple subtasks and assigned to different GPUs for execution, and different distributed training methods have different communication and computation efficiencies, so when multiple models are trained simultaneously in a computing cluster, it is obvious that a simple scheduling method cannot achieve the optimal performance of the intelligent computing cluster. When distributed training models are trained simultaneously in an intelligent computing cluster, if only relying on local resource schedulers, problems such as waiting between training tasks, GPU idle, and communication congestion may occur. Summary of the Invention [Problem to be solved by the invention]

[0004] The objective of the present invention is to provide a method, system and device for scheduling distributed training tasks for intelligent computing, which solves the problem that the single-card, single-task scheduling in the prior art cannot fully utilize the cooperative scheduling characteristics of the distributed training method and cannot fully mine the performance potential of distributed training in intelligent computing clusters. [Means for solving the problem]

[0005] The technical solutions used in the present invention are as follows:

[0006] An embodiment of the present invention provides a distributed training task scheduling system for intelligent computing, wherein a computing cluster includes a plurality of computing nodes, the plurality of computing nodes can communicate with each other, and each computing node includes at least one CPU and at least one GPU; the scheduling system includes: a model performance prediction and decomposition module, which is used to determine a distributed training method corresponding to the trained model according to a trained model, a target completion time and a target input resource input by a user, divide the trained model into a plurality of subtasks, and determine resource consumption information of each subtask, wherein the distributed training method includes one of data parallelism, pipeline parallelism and hybrid parallelism, the hybrid parallelism includes data parallelism and pipeline parallelism, and the resource consumption information includes computation consumption and memory consumption; a global GPU resource scheduler for receiving the subtask request sent by the model performance prediction and decomposition module, and assigning each subtask to a GPU of a corresponding computing node according to resource consumption information of each subtask and GPU running status of the multiple computing nodes to perform training, and building a communication topology between each subtask; and monitoring the GPU computing resource running status of each computing node during the GPU of each computing node training the corresponding subtask, and controlling the scheduling of subtasks according to the GPU computing resource running status of all computing nodes, wherein the subtask request includes a distributed training method corresponding to the model to be trained, a multiple number of the subtasks, and resource consumption information of each subtask; and a local GPU resource scheduler disposed on each computing node for locally scheduling subtasks assigned to the computing node based on the distributed training scheme.

[0007] An embodiment of the present invention further provides a method for scheduling distributed training tasks for intelligent computing, wherein a computing cluster includes a plurality of computing nodes, the plurality of computing nodes can communicate with each other, and each computing node includes at least one CPU and at least one GPU; the scheduling method includes: The model performance prediction and decomposition module determines a distributed training method corresponding to the trained model according to the trained model, the target completion time and the target input resource input by the user, divides the trained model into a plurality of subtasks, and determines resource consumption information of each subtask, wherein the distributed training method includes one of data parallelism, pipeline parallelism and hybrid parallelism, the hybrid parallelism includes data parallelism and pipeline parallelism, and the resource consumption information includes computation consumption and memory consumption; After receiving the subtask request sent by the model performance prediction and decomposition module, the global GPU resource scheduler assigns each subtask to a GPU of a corresponding computing node according to the resource consumption information of each subtask and the GPU running status of the multiple computing nodes to perform training, and establishes a communication topology between each subtask. In the process of each computing node's GPU training the corresponding subtask, monitors the GPU's computing resource running status of each computing node, and controls the scheduling of subtasks according to the GPU's computing resource running status of all computing nodes, and the subtask request includes a distributed training method corresponding to the model to be trained, a multiple of the subtasks, and the resource consumption information of each subtask; A local GPU resource scheduler, which is disposed in each computing node, locally schedules subtasks assigned to the computing node based on the distributed training scheme.

[0008] An embodiment of the present invention further provides a scheduling device for distributed training tasks for intelligent computing, comprising a storage device and one or more processors, wherein executable code is stored in the storage device, and when the one or more processors execute the executable code, the device is used to execute the scheduling method for distributed training tasks for intelligent computing described in any one of the above.

[0009] An embodiment of the present invention further provides a computer-readable storage medium, storing a program thereon, the program, when executed by a processor, performing the method for scheduling distributed training tasks for intelligent computing described in any one of the above. Effect of the Invention

[0010] The beneficial effect of the present invention is that by setting up a global GPU resource scheduler, it allocates subtasks, monitors the communication topology between each subtask, and the execution status of the GPU computing resources of each computing node, and schedules subtasks, thereby improving the utilization rate of resources such as GPUs and networks in the computing cluster, and shortening the waiting time for subtask training. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a structural schematic diagram of a computing cluster according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a structural schematic diagram of a distributed training task scheduling system for intelligent computing according to another embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart of an execution method in which a model performance prediction and decomposition module of one embodiment of the present invention determines a distributed training method corresponding to a model to be trained based on a training target model, a target completion time, and a target input resource input by a user. [Figure 4] FIG. 4 is a flowchart of an execution method in which a global GPU resource scheduler controls sub-task scheduling according to the execution status of GPU computing resources of all computing nodes according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a schematic diagram illustrating the function of a global GPU resource scheduler and the interaction between the global GPU resource scheduler and local resource schedulers according to one embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart illustrating a method for executing a local GPU resource scheduler, which is disposed in each computing node, to locally schedule subtasks assigned to the computing node based on a distributed training scheme in one embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart of a first scheduling policy according to an embodiment of the present invention. [Figure 8]FIG. 8 is a flowchart of the second scheduling policy according to an embodiment of the present invention. [Figure 9] FIG. 9 is a flowchart of a method for scheduling distributed training tasks for intelligent computing according to an embodiment of the present invention. [Figure 10] FIG. 10 is a block diagram of an apparatus for scheduling distributed training tasks for intelligent computing according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the technical solutions of the embodiments of the present invention will be described clearly and completely with reference to the drawings of the embodiments of the present invention, and of course, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, any other embodiments obtained by a person skilled in the art without inventive efforts are all within the protection scope of the present invention.

[0013] Model parameter training methods may include data parallelism and pipeline parallelism, data parallelism copies the model to multiple GPUs, and performs gradient interaction and parameter update through collective communication or parameter server, and the training process may be divided into two stages of gradient calculation and gradient synchronization, where the calculation effect of gradient calculation is higher, the GPU utilization rate is higher, and there is basically no communication, but the calculation overhead is lower and the communication overhead is very high in the gradient synchronization stage. Pipeline parallelism divides the model into multiple stages for each layer, and each stage is placed on a GPU, and forward calculation is performed in the execution order in multiple stages, and the loss function is calculated in the last stage, and then the backward calculation is performed in sequence from the last stage to the first stage, and the idle waiting time between the forward calculation and the backward calculation of different stages in the whole process is different.

[0014] In current distributed training, GPU computing resource scheduling always takes a single card as the scheduling unit, which cannot fully utilize the cooperative scheduling characteristics of the distributed training method and cannot fully mine the performance potential of distributed training in intelligent computing clusters.

[0015] In response to the above, the present invention sets up a global GPU resource scheduler 20 to allocate subtasks, monitor the communication topology between each subtask, and monitor the execution status of the GPU computing resources of each computing node, and schedule subtasks, thereby improving the utilization rate of resources such as GPUs and networks in the computing cluster, and shortening the waiting time for subtask training.

[0016] In addition, unless contradictory, the features of the following examples and embodiments can be combined with each other.

[0017] 1, a computing cluster in an embodiment of the present invention may include a plurality of computing nodes, the plurality of computing nodes may communicate with each other, and each computing node may include at least one CPU and at least one GPU. As shown in FIG. 1, the computing cluster may include computing node 1, computing node 2, ..., computing node N, where N is a positive integer and N is 3 or more.

[0018] It should be noted that the method, system and apparatus for distributed training task scheduling for intelligent computing in the embodiment of the present invention are applied to distributed task scheduling in the computing cluster shown in FIG.

[0019] Specifically, referring to FIG. 2, the distributed training task scheduling system for intelligent computing according to an embodiment of the present invention may include a model performance prediction and decomposition module 10, a global GPU resource scheduler 20, and a local GPU resource scheduler 30. In the embodiment of the present invention, a local GPU resource scheduler is configured on each computing node. As shown in FIG. 2, a computing cluster may include computing node 1, computing node 2, ..., computing node N, where N is a positive integer and N is greater than or equal to 3, and computing node 1, computing node 2, ..., computing node N are respectively configured with local GPU resource scheduler 31, local GPU resource scheduler 32, ..., local GPU resource scheduler 3N.

[0020] The model performance prediction and decomposition module 10 is used to determine a distributed training method corresponding to the model to be trained according to the trained model, the target completion time and the target input resources input by the user, and to divide the trained model into multiple subtasks and determine resource consumption information of each subtask, where the distributed training method includes one of data parallelism, pipeline parallelism and hybrid parallelism, where the hybrid parallelism includes data parallelism and pipeline parallelism, and the resource consumption information includes computation consumption and memory consumption.

[0021] After receiving the subtask request sent by the model performance prediction and decomposition module 10, the global GPU resource scheduler 20 assigns each subtask to the GPU of the corresponding computing node for training according to the resource consumption information of each subtask and the GPU execution status of multiple computing nodes, establishes a communication topology between each subtask, and in the process of the GPU of each computing node training the corresponding subtask, monitors the computational resource execution status of the GPU of each computing node, and controls the scheduling of subtasks according to the computational resource execution status of the GPU of all computing nodes, where the subtask request includes a distributed training method corresponding to the model to be trained, a plurality of subtasks, and resource consumption information of each subtask.

[0022] A local GPU resource scheduler, which is disposed in each computing node, is used to locally schedule the subtasks assigned to the computing node according to a distributed training scheme.

[0023] In an embodiment of the present invention, the trained model may be a neural network model or may be another type of model, for example a mathematical model to be trained.

[0024] Furthermore, the model to be trained may include one model, or may include multiple models.

[0025] The target completion time length may be determined by the time required to complete training of the predicted trained model, for example, the target completion time length may be equal to the time required to complete training of the predicted trained model, or the target completion time length may be slightly longer than the time required to complete training of the predicted trained model. The time required to complete training of the predicted trained model may be predicted by experience, for example predicted by historical training data.

[0026] The target input resource may be determined by the amount of resources required to complete the training of the predicted trained model, for example, the amount of target input resource may be equal to the amount of resources required to complete the training of the predicted trained model, or the amount of target input resource may be slightly greater than the amount of resources required to complete the training of the predicted trained model. The amount of resources required to complete the training of the predicted trained model may be predicted by experience, for example, predicted by historical training data.

[0027] In a possible execution manner, referring to FIG. 3, the model performance prediction and decomposition module 10 may use the following steps S11 to S13 to determine a distributed training manner corresponding to the model to be trained based on the model to be trained, the target completion time, and the target input resources input by the user.

[0028] In step S11, the model to be trained is pre-trained to determine the computation time and memory overhead required for parameters of each layer in the model to be trained.

[0029] Because the time difference in model training is smaller, the model performance prediction and decomposition module 10 obtains the computation time and memory overhead required for the parameters of each layer in the trained model by pre-training on one device. Note that the pre-training in this step does not mean to complete the training, but to repeat the training of the trained model multiple times and then take the average value to predict the computation time and memory overhead of the parameters of each layer.

[0030] In a specific implementation, the process of pre-training the model to be trained and determining the computation time and memory overhead required for parameters of each layer in the trained model may include: training each layer parameter in the trained model multiple times, respectively, to determine the computation time and memory overhead for each training iteration of the parameters of each layer; determining the computation time required for the parameters of each layer based on an average value of the computation time for the multiple training iterations of the parameters of each layer; and determining the memory overhead required for the parameters of the layer based on an average value of the memory overhead for the multiple training iterations of the parameters of each layer.

[0031] For example, the trained model includes three layers, a first layer, a second layer, and a third layer, and the model performance prediction and decomposition module 10 first repeats the training of the first layer parameters, the second layer parameters, and the third layer parameters in the trained model 10 times, respectively, to calculate the calculation time T 1i and memory overhead R 1i (where i is the number of iterations), the computation time for each training iteration of the second layer parameters T 2i and memory overhead R 2i (where i is the number of iterations), and the computation time T 3i and memory overhead R 3i (where i is the iteration number, i=0, 1, 2, ..., 9).

[0032] The average computation time for 10 iterations of training the first layer parameters is T 1 =(T 10 +T 11 +…+T 19 ) / 10, and the average memory overhead for training the first layer parameters for 10 iterations is R 1 =(R 10 +R 11 +…+R 19) / 10, and the calculation methods for the average calculation time and average memory overhead for 10 repeated training of the second and third layer parameters are similar, so detailed explanations are omitted.

[0033] The length of calculation time required for the parameters of each layer may be equal to the average value of the calculation time for the parameters of the layer through multiple repeated training, or the length of calculation time required for the parameters of each layer may be slightly greater than the average value of the calculation time for the parameters of the layer through multiple repeated training. 1 or T 1 It may be a little larger than that.

[0034] The memory overhead required for each layer parameter may be equal to the average memory overhead due to multiple iterations of training the parameters of that layer, or the memory overhead required for each layer parameter may be slightly larger than the average memory overhead due to multiple iterations of training the parameters of that layer. 1 or R 1 It may be a little larger than that.

[0035] In step S12, based on the computation time and memory overhead required for the parameters of each layer, the GPU resources and task completion times required when using different distributed training methods are determined.

[0036] In addition, each iterative training in step S11 may use data parallelism or pipeline parallelism, and the parameters of each layer may use one iterative training method or may use data parallelism and pipeline parallelism (a training method including data parallelism and pipeline parallelism may be referred to as a hybrid training method).

[0037] In step S12, first, a permutation combination is performed on the repeated training of parameters of multiple layers of the trained model, and each type of permutation combination corresponds to a type of distributed training method. Then, the GPU resources required and the task completion time are evaluated for each type of permutation combination of parameters of multiple layers of the trained model.

[0038] In step S13, based on the target completion time and the target input resources, the distributed training scheme with the shortest task completion time is selected as the distributed training scheme corresponding to the model to be trained.

[0039] As an option, eliminate the permutation combination whose GPU resource exceeds the target input resource in the above step S12, and then select the permutation combination whose task completion time is the shortest from the remaining permutation combinations as the distributed training method for the model to be trained, thereby ensuring optimal training efficiency. Of course, in other embodiments, eliminate the permutation combination whose GPU resource exceeds the target input resource in the above step S12, and select the permutation combination whose task completion time is the second shortest from the remaining permutation combinations as the distributed training method for the model to be trained, thereby satisfying different training requirements.

[0040] After selecting the distributed training manner of the trained model, the model performance prediction and decomposition module 10 further divides the trained model into multiple subtasks according to the GPUs of multiple computing nodes in the computing cluster. In the case of data parallelism, each subtask is a complete model, and the subtasks in each GPU perform gradient exchange and parameter update through collective communication or parameter server; in the case of pipeline parallelism, the subtasks in each GPU are a submodel containing multiple layers of parameters, and the submodels in each GPU perform intermediate parameter communication in a point-to-point manner.

[0041] The model performance prediction and decomposition module 10 sends descriptive information, such as the assigned subtasks and the resource consumption information of each subtask, to the global GPU resource scheduler 20, which searches for GPUs on suitable computing nodes to execute and build the communication topology.

[0042] After receiving the subtask request sent from the model performance prediction and decomposition module 10, the global GPU resource scheduler 20 assigns the subtasks to the appropriate GPUs for execution according to the current GPU execution status of the computing cluster (i.e., the GPU execution status of each computing node in the computing cluster) in combination with the computation time and memory requirements of all the subtasks of the model (i.e., the resource consumption information of each subtask), and constructs a communication topology between each subtask. Then, the GPU of each computing node trains the subtasks assigned to it. That is, the global GPU resource scheduler 20 according to the embodiment of the present invention has a global resource allocation function.

[0043] As an option, the global GPU resource scheduler 20 maps the subtasks decomposed by the model performance prediction and decomposition module 10 to specific GPUs, allowing subtasks of multiple models to share the GPU and minimizing the latency between multiple subtasks of one model.

[0044] In an embodiment of the present invention, the computational resource execution status may include subtask latency and GPU utilization rate, and it can be understood that the computational resource execution status is not limited to the above subtask latency and GPU utilization rate, but may further include others, such as the CPU utilization rate of the computation node.

[0045] The means by which the global GPU resource scheduler 20 monitors the GPU computing resource execution status of each computing node may be selected as necessary. For example, in some embodiments, the global GPU resource scheduler 20 actively obtains the GPU computing resource execution status from each computing node in the computing cluster, and in other embodiments, each computing node in the computing cluster actively reports the GPU computing resource execution status obtained by the computing node to the global GPU resource scheduler 20.

[0046] Alternatively, in some embodiments, the global GPU resource scheduler 20 periodically obtains the GPU computing resource execution status of each computing node. For example, the global GPU resource scheduler 20 periodically receives the GPU computing resource execution status of each computing node that is fed back to the global GPU resource scheduler 20 by the computing node, i.e., each computing node periodically reports the GPU computing resource execution status of the computing node to the global GPU resource scheduler 20. For further example, the global GPU resource scheduler 20 periodically obtains the GPU computing resource execution status from each computing node in the computing cluster, i.e., the global GPU resource scheduler 20 actively and periodically obtains the GPU computing resource execution status from each computing node in the computing cluster. The length of the period of obtaining the GPU computing resource execution status of the computing node may be set as required, for example, 10 minutes.

[0047] It should be understood that in some other embodiments, the global GPU resource scheduler 20 aperiodically obtains the GPU computing resource execution status of each computing node. In this embodiment, the global GPU resource scheduler 20 can obtain the GPU computing resource execution status of each computing node when necessary.

[0048] The global GPU resource scheduler 20 according to the embodiment of the present invention has a function of sub-task cooperative scheduling. In a possible execution manner, referring to Fig. 4, when the global GPU resource scheduler 20 controls the scheduling of sub-tasks according to the GPU computing resource execution status of all computing nodes, the execution manner may include, but is not limited to, the following steps S21 and S22:

[0049] In step S21, a backup node is added to a subtask whose waiting time is equal to or greater than a preset time threshold, and the backup node is a computing node other than a current computing node corresponding to a subtask whose waiting time is equal to or greater than a preset time threshold among multiple computing nodes, and the GPU utilization rate of the backup node is equal to or less than a preset utilization rate threshold.

[0050] The magnitudes of the preset time threshold, the preset utilization threshold, etc. may be set by the user according to actual requirements.

[0051] For example, the preset time threshold is 5 minutes, the preset utilization threshold is 70%, the GPU of computing node 1 executes subtasks 11 and 12, computing node 2 executes subtask 13, computing node 3 executes subtasks 14 and 15, the waiting time of subtask 12 is longer than 5 minutes, the GPU utilization of computing node 2 is less than 70%, and the utilization of computing node 3 is greater than 70%, then computing node 2 may be the backup node.

[0052] In step S22, the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are copied to the backup node, so that the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are added to the backup node in a data parallel manner to participate in the training of the task in the next iteration.

[0053] Using the above example, it is noted that before scheduling subtask 12 on computation node 2, computation node 1 may have already performed correspondence training for subtask 12, and therefore the latest model parameters of subtask 12 are now copied to computation node 2.

[0054] A backup node is added to the subtask whose latency is equal to or greater than the preset time threshold, and the latest model parameters corresponding to the subtask whose latency is equal to or greater than the preset time threshold are copied to the backup node, so that the latest model parameters corresponding to the subtask whose latency is equal to or greater than the preset time threshold are added to the backup node in a data parallel manner to participate in the training of the task in the next iteration, thereby shortening the training latency of the subtask whose latency is equal to or greater than the preset time threshold, making full use of the idle time of the backup node, and finally shortening the overall training latency and improving the training efficiency.

[0055] In this step, the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are added to the backup node in a data parallel manner to participate in the training of the task in the next iteration; specifically, a narrow-range data parallel is established for two nodes, the current computing node and the backup node, so that the current computing node only needs to train half of the data in the next iteration, thereby reducing the load on the current computing node.

[0056] When performing step S22, as an option, the global GPU resource scheduler 20 sends first scheduling information to the local GPU resource scheduler of the backup node, where the first scheduling information includes latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold, and after receiving the first scheduling information, the local GPU resource scheduler of the backup node adds the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold to the backup node in a data parallel manner to participate in the training of the task in the next iteration.

[0057] Using the above embodiment, the global GPU resource scheduler 20 sends first scheduling information to the local GPU resource scheduler of the computing node 2, where the first scheduling information includes the latest model parameters of the subtask 12. After receiving the first scheduling information, the local GPU resource scheduler of the computing node 2 adds the latest model parameters of the subtask 12 to the computing node 2 in a data parallel manner, and participates in the training of the task in the next iteration as a new node in data parallel.

[0058] When the global GPU resource scheduler 20 controls the scheduling of the subtasks according to the execution status of the GPU computing resources of all the computing nodes, the global GPU resource scheduler 20 may further take into account the distributed training method corresponding to the subtasks. For example, in some embodiments, if the distributed training method corresponding to the subtasks is data parallel, the training process of the subtasks includes a gradient calculation stage and a gradient synchronization stage. When the global GPU resource scheduler 20 controls the scheduling of the subtasks according to the execution status of the GPU computing resources of all the computing nodes, the global GPU resource scheduler 20 controls the prefetching of model parameters and intermediate variables of the corresponding subtasks according to the execution status of the GPU computing resources of the computing nodes on which all the data parallel subtasks are located. Specifically, when the global GPU resource scheduler 20 controls the prefetching of model parameters and intermediate variables of corresponding subtasks according to the execution status of GPU computing resources of the computing nodes where all data parallel subtasks are located, after receiving that the parameter server starts to calculate global gradient information of all data parallel subtasks, the global GPU resource scheduler 20 sends second scheduling information to the computing nodes corresponding to the data parallel subtasks, thereby suggesting that the computing nodes corresponding to the data parallel subtasks should preferentially execute the corresponding data parallel subtasks according to the second scheduling information, and copies the latest model parameters and intermediate variables corresponding to the corresponding data parallel subtasks from the GPU main memory of the computing node corresponding to the data parallel subtasks to the GPU graphic memory of the computing node. When the corresponding data parallel subtasks wait for other calculation results depending on the subtasks, if the expected waiting time exceeds the CPU-GPU memory copy time of the corresponding data parallel computing node, the corresponding data parallel computing node temporarily moves the model parameters and intermediate variables corresponding to the corresponding data parallel subtasks from the GPU graphic memory of the corresponding data parallel computing node to the CPU main memory of the corresponding data parallel computing node.Taking advantage of the larger memory feature of CPU main memory, when a subtask in a model GPU waits for other calculation results that depend on the task, if the expected waiting time exceeds the CPU-GPU memory copy time, the model parameters and intermediate variables of the subtask are temporarily moved to the CPU main memory, and then pre-fetched before the next calculation, thus improving the utilization rate of resources such as GPU and network in the computing cluster. The CPU-GPU memory copy is transmitted through the PCI-E channel, and the transmission speed is relatively constant, therefore the CPU-GPU memory copy time can be calculated by dividing the size of the amount of data to be transmitted by the transmission speed of the PCI-E channel. When the global GPU resource scheduler 20 receives that the parameter server of the computing cluster starts to calculate global gradient information, it sends the second scheduling information to the computing node of the corresponding subtask (the computing node corresponding to the above-mentioned data parallel subtask), and after receiving the second scheduling information, the computing node of the corresponding subtask presents the subtask to be executed preferentially, and copies its model parameters and intermediate variables from the CPU main memory to the GPU graphic memory as soon as possible, thereby improving the calculation efficiency of the GPU and shortening the waiting time of the subtask calculation as much as possible. The global GPU resource scheduler 20 in this embodiment has a calculation resource adjustment function. Note that the global gradient information is determined by the gradient information of each subtask, and the global gradient information optionally includes the gradient information of each subtask, and the global gradient information optionally is obtained by processing based on the gradient information of each subtask. The gradient information includes gradient calculation information and gradient synchronization information.

[0059] In some embodiments, the global GPU resource scheduler 20 further includes a task resource reclaim function. Specifically, after the training of the model to be trained is completed, the global GPU resource scheduler 20 further determines the computing node on which each subtask is located based on the historical allocation information of each subtask of the model to be trained, and controls the computing node on which each subtask is located to reclaim the local resources used when training the corresponding subtask on the computing node. After determining that resource reclaiming by all computing nodes is completed, the global GPU resource scheduler 20 releases the resources used when training the model to be trained.

[0060] Referring to FIG. 5, in one embodiment of the present invention, the global GPU resource scheduler 20 integrates the functions of global resource allocation, subtask cooperative scheduling, computation resource adjustment and task resource recovery.

[0061] Referring to FIG. 6, when the local GPU resource scheduler disposed in each computing node locally schedules the subtasks assigned to the computing node based on the distributed training method, the scheduler includes, but is not limited to, the following steps S31 to S33.

[0062] In step S31, a training type of the locally allocated subtask is determined based on a distributed training scheme, where the training type includes a data parallel task and a pipeline parallel task.

[0063] For example, the training type of the subtask local to the corresponding computing node in the permutation combination determined in step S13 may be mentioned.

[0064] In step S32, a local scheduling policy for the locally assigned subtask is determined based on a training type of the locally assigned subtask; In step S33, the locally assigned subtasks are locally scheduled based on the local scheduling policy.

[0065] If the training type is a data parallel task, the local scheduling policy of the subtask is the first scheduling policy, and if the training type is a pipeline parallel task, the local scheduling policy of the subtask is the second scheduling policy.

[0066] The first scheduling policy may be set as needed. For example, in some embodiments, when the training type is a data parallel task, the training process of the subtask includes a gradient calculation stage and a gradient synchronization stage. The gradient calculation stage has high computation efficiency and very low communication overhead, while the gradient synchronization stage has lower computation efficiency and higher communication overhead. In the first scheduling policy, the subtasks are scheduled and managed based on the above characteristics to achieve optimal scheduling, improve the utilization of resources such as GPU and network of the computing cluster, and reduce the latency of the subtask training. Specifically, referring to FIG. 7, the first scheduling policy includes obtaining a first computation demand in the gradient calculation stage of the current subtask and a second computation demand of other subtasks in the current computing node, and determining the training order of all subtasks in the current computing node based on the computation efficiency according to the first computation demand and the second computation demand. It should be understood that the computation efficiency is negatively correlated to the magnitude of the computation demand, i.e., the greater the computation demand, the lower the computation efficiency, and the smaller the computation demand, the higher the computation efficiency. As an alternative, the more computationally efficient the training order, the higher up, and the less computationally efficient the training order, the lower up.

[0067] If the local computing resources of the current computing node are no longer able to meet the computing demand of the new task and thus may affect the task completion time predicted by the user, the current computing node's judgment result together with the GPU computing resource usage status of the current computing node is fed back to the global GPU resource scheduler 20, and the global GPU resource scheduler 20 is inquired as to whether there are other computing nodes that can meet the task completion time predicted by the user.

[0068] If the global GPU resource scheduler 20 is placed on another computing node, the scheduling of the new task will be terminated. Further, referring to Fig. 7, the first scheduling policy further includes, when the local computing resource of the current computing node exceeds the computing demand of the current subtask, feeding back the GPU computing resource execution status of the current computing node to the global GPU resource scheduler 20, inquiring whether there is another computing node whose computing resource does not exceed the computing demand of the current subtask. When the local computing resource of the computing node exceeds the computing demand of the current subtask, the training time of the current subtask exceeds the task completion time expected by the user, and when the local computing resource of the computing node is less than the computing demand of the current subtask, the training time of the current subtask is shorter than the task completion time expected by the user.

[0069] If the global GPU resource scheduler 20 is not deployed on other computing nodes, further referring to FIG. 7, the first scheduling policy further includes: when the global GPU resource scheduler 20 feedbacks that there are no other computing nodes, the current computing node builds a high priority queue and a low priority queue, and puts the gradient calculation phase task of the current subtask into the high priority queue and the gradient synchronization phase task of the current subtask into the low priority queue; the GPU of the current computing node executes the gradient calculation phase task, and the CPU of the current computing node executes the gradient synchronization phase task. Further, the first scheduling policy further includes: when the execution of the gradient calculation phase task is completed, copying the model parameters and intermediate variables corresponding to the gradient calculation phase task to the CPU main memory of the current computing node; when both the gradient calculation phase task and the gradient synchronization phase task are completed, copying the model parameters and intermediate variables of the corresponding subtask to the GPU graphic memory of the current computing node; and / or, after the current computing node receives the first scheduling information sent by the global GPU resource scheduler 20, the current computing node marks the subtasks in the low priority queue as prefetched, and the GPU of each computing node executes the subtasks marked with prefetch in a preferential manner. For example, when the global GPU resource scheduler 20 feedbacks that there are no other computing nodes, the current computing node builds a queue with two priorities, divides the data parallel task into a gradient calculation phase task and a gradient synchronization phase task, puts the gradient calculation phase task in the high priority queue, and puts the gradient synchronization phase task in the low priority queue. When the computing resources of the GPU of the current computing node are tight, the CPU of the current computing node takes charge of the gradient synchronization phase task. After the current computing node receives the first scheduling information sent by the global GPU resource scheduler 20, it marks the current subtask in the low priority queue as prefetch.The local scheduling policy of the current computing node will prioritize execution of subtasks marked for prefetch when selecting a task, and if the gradient synchronization of the current subtask is completed, copy the model parameters and intermediate variables of the current subtask from the CPU host memory of the current computing node to the GPU graphic memory of the current computing node; if not, continue to monitor and start copying until the gradient synchronization is completed.

[0070] In some other embodiments, when the training type is a pipeline parallel task, the current subtask includes multiple task stages, the calculation task of the last stage of the current subtask is a complete calculation task, and the calculation tasks of other stages of the current subtask include forward calculation tasks and backward calculation tasks. The pipeline parallel task divides the training process into multiple stages, and is a bidirectional pipeline parallel, performing forward calculation from the first stage to the last stage, then calculating the loss function, and then performing backward calculation from the last stage to the first node, and the idle times of different stages are different. The idle time of the first stage is the longest, and then it is shortened in sequence, and the forward calculation and backward calculation of the last stage are in a continuum and have no idle time. In the second scheduling policy, the subtasks are scheduled and managed based on this feature to realize optimal scheduling, improve the utilization rate of resources such as GPU and network of the computing cluster, and reduce the waiting time of subtask training.

[0071] Alternatively, after receiving the scheduling request of the GPU resource global scheduler, the current computing node determines whether to divide the current subtask into two forward and backward computing tasks according to the stage to which the current subtask belongs. If it is the last stage, the current subtask is treated as one complete computing task, and the other stages are divided into two forward and backward computing tasks. The current computing node further determines whether it can meet the computing demand of the current subtask according to the current local GPU resource execution status of the current computing node, and if NO, queries the global GPU resource scheduler 20 whether there are other computing nodes that can meet the task completion time expected by the user. If the global GPU resource scheduler 20 is located on another computing node, it ends the scheduling of the model subtask. If the global GPU resource scheduler 20 feedbacks that there are no other computing nodes, the current computing node puts the current subtask into a high priority queue. As an option, referring to FIG. 8 , the second scheduling policy includes: the current computing node judges whether the GPU resource of the current computing node can satisfy the computation demand of the current subtask according to the local GPU resource execution status; if NO, queries the global GPU resource scheduler 20 whether there are other computing nodes that can satisfy the computation demand of the current subtask; and if the global GPU resource scheduler 20 feedbacks that there are no other computing nodes, the current computing node puts the current subtask into a high priority queue.In addition, in order to fully utilize the GPU calculation efficiency, the calculation tasks of other tasks may be inserted in the idle time of the forward and backward calculation stages. Referring to FIG. 8, the second scheduling policy of this embodiment may further include: inserting the calculation tasks of other subtasks according to the idle time of the forward calculation task stage and the backward calculation task stage of the current subtask; and / or copying the model parameters and intermediate variables corresponding to the forward calculation task of the current subtask from the GPU of the current calculation node to the CPU main memory of the current calculation node after the forward calculation task of the current subtask is completed, and marking the backward calculation task related to the forward calculation task of the current subtask with a pre-execution time according to the expected idle time; and after the pre-execution time is finished, if the related backward calculation task has not yet started to be executed, copying the model parameters and intermediate variables corresponding to the corresponding forward calculation task of the current subtask from the CPU main memory to the GPU of the current calculation node again, thereby reducing the latency of the backward calculation.

[0072] An embodiment of the present invention further provides a method for scheduling distributed training tasks for intelligent computing. Referring to FIG. 9, the method for scheduling distributed training tasks for intelligent computing in an embodiment of the present invention includes: Step S100: the model performance prediction and decomposition module determines a distributed training method corresponding to the model to be trained according to the trained model, the target completion time and the target input resource input by the user, and divides the trained model into a plurality of subtasks, and determines resource consumption information of each subtask, where the distributed training method includes one of data parallelism, pipeline parallelism and hybrid parallelism, where the hybrid parallelism includes data parallelism and pipeline parallelism, and the resource consumption information includes computation consumption and memory consumption; Step S200: after receiving the subtask request sent by the model performance prediction and decomposition module, the global GPU resource scheduler allocates each subtask to the GPU of the corresponding computing node according to the resource consumption information of each subtask and the GPU running status of multiple computing nodes to perform training, and establishes a communication topology between each subtask; and in the process of each computing node's GPU training the corresponding subtask, monitors the GPU's computing resource running status of each computing node, and controls the scheduling of subtasks according to the GPU's computing resource running status of all computing nodes, where the subtask request includes a distributed training method corresponding to the model to be trained, a plurality of subtasks, and the resource consumption information of each subtask; The method may include a step S300 in which a local GPU resource scheduler disposed in each computing node locally schedules subtasks assigned to the computing node based on a distributed training scheme.

[0073] Corresponding to the embodiment of the method for scheduling distributed training tasks for intelligent computing described above, the present invention further provides an embodiment of an apparatus for scheduling distributed training tasks for intelligent computing.

[0074] Referring to Figure 10, a scheduling device for distributed training tasks for intelligent computing according to an embodiment of the present invention includes a storage device and one or more processors, wherein executable code is stored in the storage device, and when the one or more processors execute the executable code, it is used to execute the scheduling method for distributed training tasks for intelligent computing in the above embodiment.

[0075] The embodiment of the device for scheduling distributed training tasks for intelligent computing according to the embodiment of the present invention may be applied to any device having data processing capability, and the device having data processing capability may be a device or equipment such as a computer. The device embodiment may be realized by software, hardware, or a combination of software and hardware. Taking software as an example, a device in a logical sense is formed by loading corresponding computer program instructions in non-volatile memory into memory and executing them by the processor of the device having data processing capability located therein. From the hardware aspect, FIG. 10 is a hardware structure diagram of the device having data processing capability located in the device for scheduling distributed training tasks for intelligent computing according to the embodiment of the present invention. In addition to the processor, memory, network interface, and non-volatile memory shown in FIG. 10, the device having data processing capability located in the embodiment may generally further include other hardware according to the actual function of the device having data processing capability, and detailed description thereof will be omitted.

[0076] The process of implementing the functions and actions of each unit of the above device may specifically refer to the process of implementing the corresponding steps in the above method, and detailed descriptions thereof will be omitted here.

[0077] The device embodiments basically correspond to the method embodiments, so that the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely schematic, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or may be located in multiple network units. According to actual needs, some or all of the modules may be selected to implement the purpose of the proposed invention. Those skilled in the art can understand and implement it without any creative effort.

[0078] An embodiment of the present invention further provides a computer-readable storage medium, storing a program thereon, the program, when executed by a processor, performing the method for scheduling distributed training tasks for intelligent computing in the above embodiment.

[0079] The computer-readable storage medium may be an internal storage unit of any device having data processing capability as described in any one of the above embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device having data processing capability, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a flash card, etc., disposed in the device. Furthermore, the computer-readable storage medium may include an external storage device in addition to the internal storage unit of any device having data processing capability. The computer-readable storage medium may be used to store the computer program and other programs and data required by any device having data processing capability, and may also be used to temporarily store data that has already been output or is to be output. [Explanation of symbols]

[0080] 10. Model Performance Prediction and Decomposition Module 20 Global GPU Resource Scheduler 30 Local GPU Resource Scheduler

Claims

1. A distributed training task scheduling system for intelligent computing, comprising: The computing cluster includes a plurality of computing nodes, the plurality of computing nodes can communicate with each other, and each computing node includes at least one CPU and at least one GPU; the system includes a model performance prediction and decomposition module, a global GPU resource scheduler, and a local GPU resource scheduler, each of which is located on each computing node; The model performance prediction and decomposition module is used for determining a distributed training method corresponding to the trained model according to the trained model, the target completion time and the target input resource input by a user, dividing the trained model into a plurality of subtasks, and determining resource consumption information of each subtask, wherein the distributed training method includes one of data parallelism, pipeline parallelism and hybrid parallelism, the hybrid parallelism includes data parallelism and pipeline parallelism, and the resource consumption information includes computation consumption and memory consumption; The global GPU resource scheduler receives a subtask request sent by the model performance prediction and decomposition module, and assigns each subtask to a GPU of a corresponding computing node according to resource consumption information of each subtask and GPU running status of the multiple computing nodes to perform training, and establishes a communication topology between each subtask. In the process of the GPU of each computing node training the corresponding subtask, the global GPU resource scheduler monitors the computing resource running status of the GPU of each computing node, and controls the scheduling of subtasks according to the computing resource running status of the GPU of all computing nodes, where the subtask request includes a distributed training method corresponding to the model to be trained, a multiple number of the subtasks, and resource consumption information of each subtask; The local GPU resource scheduler is used for locally scheduling subtasks assigned to the computing nodes according to the distributed training scheme; The computing resource execution status includes subtask waiting time and GPU utilization rate; When the global GPU resource scheduler controls the scheduling of subtasks according to the execution status of the computing resources of the GPUs of all computing nodes, the global GPU resource scheduler specifically: Adding a backup node to a subtask whose latency is equal to or greater than a preset time threshold, the backup node being a computation node other than a current computation node corresponding to the subtask whose latency is equal to or greater than a preset time threshold among the plurality of computation nodes, and the GPU utilization rate of the backup node being equal to or less than a preset utilization rate threshold; The latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are copied to the backup node, and the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are added to the backup node in a data parallel manner for use in participating in training of the subtasks whose latency is equal to or greater than the preset time threshold in the next iteration. A distributed training task scheduling system for intelligent computing, characterized by:

2. The global GPU resource scheduler is specifically used for sending first scheduling information to a local GPU resource scheduler of the backup node when the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are added to the backup node in a data parallel manner to participate in training of the subtasks whose latency is equal to or greater than the preset time threshold in a next iteration by copying the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold to the backup node, the first scheduling information including the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold; After receiving the first scheduling information, the local GPU resource scheduler of the backup node adds the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold to the backup node in a data parallel manner, and participates in training of the subtasks whose latency is equal to or greater than the preset time threshold in a next iteration. The distributed training task scheduling system for intelligent computing according to claim 1.

3. When the distributed training method corresponding to the subtask is data parallel, the training process of the subtask includes a gradient calculation step and a gradient synchronization step. The global GPU resource scheduler controls the scheduling of the subtask according to the execution status of the computing resources of the GPUs of all computing nodes. Specifically, It is used to control the prefetching of model parameters and intermediate variables of the corresponding subtasks according to the execution status of the GPU computing resources of the computing nodes on which all data-parallel subtasks are located. The distributed training task scheduling system for intelligent computing according to claim 1.

4. The global GPU resource scheduler controls the prefetching of model parameters and intermediate variables of the corresponding subtasks according to the execution status of the GPU computing resources of the computing nodes on which all data parallel subtasks are located, specifically by: After receiving that the parameter server of the computing cluster starts to calculate global gradient information of all data parallel subtasks, sending second scheduling information to the computing nodes corresponding to the data parallel subtasks, so that the computing nodes are suggested to execute the corresponding data parallel subtasks preferentially according to the second scheduling information, and copying the latest model parameters and intermediate variables corresponding to the corresponding data parallel subtasks from the GPU main memory of the computing node to the GPU graphic memory of the computing node; For the latest model parameters and intermediate variables corresponding to the corresponding data-parallel subtask, when the corresponding data-parallel subtask waits for other calculation results depending on the subtask, if the expected waiting time exceeds the CPU-GPU memory copy time of the computing node, the computing node temporarily moves the model parameters and intermediate variables corresponding to the corresponding data-parallel subtask from the GPU graphic memory of the computing node to the CPU main memory of the computing node. The distributed training task scheduling system for intelligent computing according to claim 3.

5. The global GPU resource scheduler further comprises: After the training of the trained model is completed, determining a computing node on which each subtask is located based on historical allocation information of each subtask of the trained model; Controlling the computing node on which each subtask is located to reclaim local resources used in training the corresponding subtask on the computing node; and After it is determined that resource collection by all computing nodes has been completed, the global GPU resource scheduler is used to release resources used in training the training target model. The distributed training task scheduling system for intelligent computing according to claim 1.

6. The local GPU resource scheduler, which is disposed in each computing node, locally schedules the subtasks assigned to the computing node according to the distributed training method, specifically: determining a training type of the locally assigned subtask based on the distributed training scheme, the training type including a data parallel task and a pipeline parallel task; determining a local scheduling policy for the locally assigned sub-task based on a training type of the locally assigned sub-task; and for locally scheduling the locally assigned subtasks based on the local scheduling policy; If the training type is a data parallel task, the local scheduling policy of the subtask is a first scheduling policy; If the training type is a pipeline parallel task, the local scheduling policy of the subtask is a second scheduling policy. The distributed training task scheduling system for intelligent computing according to claim 1.

7. If the training type is a data parallel task, the training process of the subtask includes a gradient calculation step and a gradient synchronization step; The first scheduling policy is: Obtaining a first computation demand in the gradient computation stage of a current subtask and a second computation demand of another subtask in a current computation node; determining a training order of all subtasks of the current computing node based on computational efficiency according to the first computational demand and the second computational demand; When the local computing resource of the current computing node exceeds the computing demand of the current subtask, feed back the computing resource execution status of the GPU of the computing node to the global GPU resource scheduler, so that the global GPU resource scheduler queries whether there is another computing node whose computing resource does not exceed the computing demand of the current subtask, where, when the local computing resource of the computing node exceeds the computing demand of the current subtask, the training time of the current subtask exceeds the task completion time expected by a user; when the local computing resource of the computing node is less than the computing demand of the current subtask, the training time of the current subtask is shorter than the task completion time expected by a user; When the global GPU resource scheduler feedbacks that there are no other computing nodes, the current computing node builds a high priority queue and a low priority queue, and puts the gradient calculation phase task of the current subtask into the high priority queue and puts the gradient synchronization phase task of the current subtask into the low priority queue; A GPU of the current computing node executes the gradient calculation stage task, and a CPU of the current computing node executes the gradient synchronization stage task; When the execution of the gradient calculation stage task is completed, copying model parameters and intermediate variables corresponding to the gradient calculation stage task to a CPU main memory of the current computing node, and when both the gradient calculation stage task and the gradient synchronization stage task are completed, copying model parameters and intermediate variables of corresponding subtasks to a GPU graphic memory of the current computing node; and / or, after the current computing node receives the first scheduling information sent by the global GPU resource scheduler, marking the subtasks in the low priority queue for prefetching, and causing the GPU of each computing node to preferentially execute the subtasks marked for prefetching. The distributed training task scheduling system for intelligent computing according to claim 6.

8. If the training type is a pipeline parallel task, the current subtask includes a plurality of task stages, where the calculation tasks of the plurality of task stages of the current subtask include a forward calculation task and a backward calculation task, respectively, and the calculation task of the last stage of the current subtask is a complete calculation task in which the forward calculation and the backward calculation are continuous and there is no idle time; The second scheduling policy is: The current computing node determines whether the GPU resource of the computing node can satisfy the computation demand of the current subtask according to a local GPU resource execution status, and if NO, queries the global GPU resource scheduler whether there is another computing node that can satisfy the computation demand of the current subtask; If the global GPU resource scheduler feedbacks that there are no other computing nodes, the current computing node puts the current subtask into a high priority queue; and Inserting a calculation task of another subtask based on the idle time of the forward calculation task stage and the backward calculation task stage of the current subtask; and / or After the forward calculation task of the current subtask is completed, copying model parameters and intermediate variables corresponding to the forward calculation task of the current subtask from the GPU of the current calculation node to a CPU main memory of the current calculation node; and marking a pre-execution time for a backward calculation task associated with the forward calculation task of the current subtask based on an expected idle time, where, after the pre-execution time is finished, if the associated backward calculation task has not yet started to be executed, copying the model parameters and intermediate variables corresponding to the corresponding forward calculation task of the current subtask from the CPU main memory again to the GPU of the current calculation node. The distributed training task scheduling system for intelligent computing according to claim 6.

9. When the model performance prediction and decomposition module is used to determine a distributed training method corresponding to the model to be trained according to the model to be trained, the target completion time and the target input resource input by the user, specifically, Pre-training the trained model to determine the calculation time and memory overhead required for each layer parameter in the trained model, specifically, training each layer parameter in the trained model multiple times to determine the calculation time and memory overhead required for each training iteration of each layer parameter, where the calculation time required for each layer parameter is determined based on an average value of the calculation time for each layer parameter through multiple iterations of training, and the memory overhead required for each layer parameter is determined based on an average value of the memory overhead for each layer parameter through multiple iterations of training; Determining the required GPU resources and task completion times when using different distributed training methods based on the computation time and memory overhead required for the parameters of each layer; and According to the target completion time and the target input resource, a distributed training scheme with the shortest task completion time is selected as a distributed training scheme corresponding to the training target model. The distributed training task scheduling system for intelligent computing according to claim 1.

10. A method for scheduling distributed training tasks for intelligent computing, comprising: The computing cluster includes a plurality of computing nodes, the plurality of computing nodes being capable of communicating with each other, each computing node including at least one CPU and at least one GPU, and the scheduling method includes: The model performance prediction and decomposition module determines a distributed training method corresponding to the trained model according to the trained model, the target completion time and the target input resource input by the user, divides the trained model into a plurality of subtasks, and determines resource consumption information of each subtask, wherein the distributed training method includes one of data parallelism, pipeline parallelism and hybrid parallelism, the hybrid parallelism includes data parallelism and pipeline parallelism, and the resource consumption information includes computation consumption and memory consumption; After receiving the subtask request sent by the model performance prediction and decomposition module, the global GPU resource scheduler assigns each subtask to a GPU of a corresponding computing node according to the resource consumption information of each subtask and the GPU running status of the multiple computing nodes to perform training, and establishes a communication topology between each subtask. In the process of the GPU of each computing node training the corresponding subtask, the global GPU resource scheduler monitors the computing resource running status of the GPU of each computing node, and controls the scheduling of subtasks according to the computing resource running status of the GPU of all computing nodes, and the subtask request includes a distributed training method corresponding to the model to be trained, a multiple number of the subtasks, and the resource consumption information of each subtask; A local GPU resource scheduler disposed in each computing node locally schedules subtasks assigned to the computing node according to the distributed training method; The computing resource execution status includes subtask waiting time and GPU utilization rate; When the global GPU resource scheduler controls the scheduling of subtasks according to the execution status of the computing resources of the GPUs of all computing nodes, the global GPU resource scheduler specifically: Add a backup node to a subtask whose latency is equal to or greater than a preset time threshold, the backup node being a computation node other than a current computation node corresponding to the subtask whose latency is equal to or greater than a preset time threshold among the plurality of computation nodes, and the GPU utilization rate of the backup node being equal to or less than a preset utilization rate threshold; The latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are copied to the backup node, and the latest model parameters corresponding to the subtasks whose latency is equal to or greater than the preset time threshold are added to the backup node in a data parallel manner for use in participating in training of the subtasks whose latency is equal to or greater than the preset time threshold in the next iteration. A method for scheduling distributed training tasks for intelligent computing, comprising:

Images