Heterogeneous cluster GPU calling method and system for distributed training tasks

Abstract

The invention discloses a heterogeneous cluster GPU calling method and system for distributed training tasks. The method comprises the steps that a to-be-called GPU set is determined; obtaining a processing performance parameter of each GPU in the to-be-called GPU set; for each piece of training fragment data, calculating predicted transmission time of each piece of training fragment data transmitted from the current storage position to each GPU in the to-be-called GPU set; by taking minimization of predicted total completion time of the distributed training task as an optimization target, generating a scheduling mapping plan for mapping each training data fragment to a target GPU of the training data fragment; and scheduling and transmitting each training data fragment from the current storage position to the corresponding target GPU according to the scheduling mapping plan so as to execute the distributed training task. According to the method, the fragment transmission cost of the training data can be considered, and the overall training efficiency of the overall distributed training task can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of GPU resource scheduling, in particular to a heterogeneous cluster GPU calling method and system for distributed training tasks. BACKGROUND

[0002] With the rapid expansion of deep learning model size, the demand for parallel computing power continues to grow. To meet this demand, model training generally adopts distributed deployment. However, when building an AI training platform, enterprises often face the challenge of a shortage of high-performance GPU resources, prompting them to deploy GPUs of different brands and models in the same cluster to integrate and utilize heterogeneous computing resources.

[0003] This heterogeneous GPU cluster not only relieves the computing power bottleneck, but also brings significant complexity to task scheduling. This complexity mainly comes from two aspects: first, in large-scale distributed training scenarios, massive training data is usually distributed in a complex multi-level cache / storage hierarchy consisting of GPU local memory, node memory, local NVMe storage, and remote object storage. The delay and bandwidth of data access vary greatly. Second, the various GPUs deployed in the cluster have inherent differences in key performance indicators such as computing power, memory capacity, and bandwidth.

[0004] Traditional task scheduling methods mostly only focus on the performance differences between GPUs and attempt to balance the load accordingly, but they generally ignore the high transmission cost caused by the location differences of training data in multi-level storage. In data-intensive training tasks, if a data shard stored in a remote object storage is scheduled to a high-performance GPU, the data transmission time may far exceed the computation itself, forming a serious I / O bottleneck, making it difficult for the high-performance GPU to perform optimally, and thus making it difficult to generate a globally optimal mapping plan of training data shards to GPUs, resulting in low overall training efficiency and a significant bucket effect. SUMMARY

[0005] The present application provides a heterogeneous cluster GPU calling method and system for distributed training tasks that can consider training data shard transmission costs and improve the overall training efficiency of the overall distributed training task.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is as follows: A heterogeneous cluster GPU calling method for distributed training tasks, comprising the following steps: S101, determining a set of GPUs in the cluster that can be used to execute the current distributed training task, and taking it as a set of GPUs to be called; S102, obtaining a processing performance parameter of each GPU in the set of GPUs to be invoked; obtaining a current storage location of each shard in the set of training data shards of the distributed training task, and calculating an expected transmission time of each training shard data from the current storage location to each GPU in the set of GPUs to be invoked; S103, taking minimizing the expected overall completion time of the distributed training task as an optimization objective, jointly optimizing based on the obtained processing performance parameter of each GPU in the set of GPUs to be invoked and the expected transmission time of each training data shard to each GPU in the set of GPUs to be invoked, and generating a scheduling mapping plan of mapping each training data shard to its target GPU based on the joint optimization; S104, scheduling and transmitting each training data shard from its current storage location to its corresponding target GPU according to the scheduling mapping plan, to execute the distributed training task.

[0007] Optionally, the processing performance parameter is obtained by periodic sampling and updated by an exponential smoothing algorithm.

[0008] Optionally, the processing performance parameter includes a GPU processing rate.

[0009] Optionally, the expression of the expected overall completion time of the distributed training task is: , wherein, denotes the set of GPUs to be invoked, denotes one GPU in the set of GPUs to be invoked, denotes the set of training data shards allocated to the GPU , denotes one training data shard in the set of training data shards allocated to the GPU , denotes a sample number of the training data shard , denotes a processing rate of the GPU , denotes an expected time of transmitting the training data shard to the GPU .

[0010] In step S103, when generating the scheduling mapping plan of mapping each training data shard to its target GPU, a training data shard with an expected transmission time exceeding a preset threshold is also identified, and a prefetch instruction is generated for the identified training data shard; the prefetch instruction is used to instruct to start transmitting the training data shard with the expected transmission time exceeding the preset threshold to its corresponding target GPU in advance before the start of training calculation.

[0011] Step S104 continuously monitors the running state of each target GPU in executing the distributed training task, and if the running state of any target GPU does not meet the preset running requirement, executes the following steps: S301, marking the target GPU whose running state does not meet the preset running requirement as a faulty GPU, updating the set of GPUs to be invoked, and triggering the distributed training task to check point and pause; S302, determining a set of affected training data shards on the faulty GPU, the set of affected training data shards including training data shards that are participating in training on the faulty GPU and training data shards that are not participating in training; for each affected training data shard, obtaining the locations of all available data sources storing data thereof; based on the locations of all available data sources of each affected training data shard, reassigning target GPUs for each affected data shard from the set of GPUs to be invoked, and transferring the affected training data shard from the corresponding selected available data source to the newly assigned target GPU; S303, after data transfer is completed, resuming execution of the distributed training task from the checkpoint.

[0012] In step S302, the affected training data shard is transferred from the corresponding selected available data source to the newly assigned target GPU in a batch transfer manner.

[0013] In addition, the present application also provides a cluster GPU invocation system for a distributed training task, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the heterogeneous cluster GPU invocation method for a distributed training task.

[0014] In addition, the present application also provides a computer readable storage medium, which stores a computer program or instructions, the computer program or instructions being programmed or configured to execute the heterogeneous cluster GPU invocation method for a distributed training task by a processor.

[0015] In addition, the present application also provides a computer program product, comprising a computer program or instructions, the computer program or instructions being programmed or configured to execute the heterogeneous cluster GPU invocation method for a distributed training task by a processor.

[0016] Compared with the prior art, the present application mainly has the following advantages: The scheduling mapping plan for mapping each training data shard of the distributed training task to the target GPU thereof in the present application considers the processing performance parameter of each GPU in the GPU set to be invoked and the predicted transmission time of each training data shard transmitted to each GPU in the GPU set to be invoked, and obtains the scheduling mapping plan with the optimization objective of minimizing the predicted overall completion time of the distributed training task, thereby reducing the data transmission time consumption, enabling the calculation advantage of the high-performance GPU to be better exerted, and improving the training efficiency of the distributed training task. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An execution flow diagram of the heterogeneous cluster GPU invocation method for a distributed training task of the present application. DETAILED DESCRIPTION

[0018] The technical solutions of the present application will be further described in detail below with reference to the accompanying drawings.

[0019] As shown in the figure, the heterogeneous cluster GPU invocation method for a distributed training task of the present application includes the following steps: Figure 1 S101, determining a GPU set in the cluster that can be used to execute the current distributed training task and taking it as a GPU set to be invoked; S102, obtaining the processing performance parameter of each GPU in the GPU set to be invoked; obtaining the current storage location of each shard in the training data shard set of the distributed training task, and respectively calculating the predicted transmission time of each training shard data from the current storage location to each GPU in the GPU set to be invoked; S103, taking the optimization objective of minimizing the predicted overall completion time of the distributed training task, jointly optimizing and solving based on the obtained processing performance parameter of each GPU in the GPU set to be invoked and the predicted transmission time of each training data shard transmitted to each GPU in the GPU set to be invoked, and generating a scheduling mapping plan for mapping each training data shard to the target GPU thereof based on the joint optimization and solving; S104, according to the scheduling mapping plan, scheduling and transmitting each training data shard from its current storage location to its corresponding target GPU to execute the distributed training task.

[0020] ​The heterogeneous cluster GPU calling method for distributed training tasks in the embodiment maps each training data shard of the distributed training task to a scheduling mapping plan of a target GPU thereof, considers the processing performance parameter of each GPU in a set of GPUs to be called and the predicted transmission time of each training data shard to each GPU in the set of GPUs to be called, and obtains an optimization target of minimizing the predicted overall completion time of the distributed training task, thereby reducing the data transmission time consumption, enabling the calculation advantages of high-performance GPUs to be better played, and improving the training efficiency of the distributed training task.

[0021] Further, in the embodiment, the processing performance parameter is obtained by periodic sampling and is updated by using an exponential smoothing algorithm. This is advantageous in avoiding the reduction of the training efficiency of the distributed training task and improving the reliability due to the use of outdated information.

[0022] Further, in the embodiment, the processing performance parameter includes the GPU processing rate, and of course, in other embodiments, it can also include the GPU video memory utilization rate, network bandwidth and delay, etc.

[0023] Further, in the embodiment, the expression of the predicted overall completion time of the distributed training task is: , wherein, represents the set of GPUs to be called, represents one GPU in the set of GPUs to be called, represents the set of training data shards allocated to the GPU , represents one training data shard in the set of training data shards allocated to the GPU , represents the sample number of the training data shard , represents the processing rate of the GPU , represents the predicted time of the transmission of the training data shard to the GPU .

[0024] For the process of further joint optimization solution, a specific example is taken as an illustration: Suppose that only two heterogeneous GPUs in the cluster can execute the current distributed training task, which are and , and there are three training data shards, which are , and , wherein , , , The predicted transmission time of the three training data fragments to the two aforementioned heterogeneous GPUs (which can be calculated according to the size of the training data fragments and the data transmission rate between the training data fragment locations and the GPUs) is respectively: , First, all feasible scheduling schemes are generated. The following three feasible scheduling schemes are generated as an optimization description: Scheme A1, the three training data fragments are all assigned to GPU Execution Then the predicted overall completion time of the distributed training task is: ; Scheme A2, training data fragment is assigned to GPU Execution, training data fragment and training data fragment are assigned to GPU Execution: GPU The time for completing the corresponding training task is: , GPU The time for completing the corresponding training task is: , Therefore, the predicted overall completion time of the distributed training task is 3.56s; Scheme A3, training data fragment is assigned to GPU Execution, training data fragment and training data fragment are assigned to GPU Execution: GPU The time for completing the corresponding training task is: , GPU The time for completing the corresponding training task is: , Therefore, the predicted overall completion time of the distributed training task is 4.56s; Since the predicted overall completion time of the distributed training task of scheme A2 is the smallest, the scheduling mapping plan finally generated according to the mapping of each training data fragment to its target GPU is in accordance with the allocation mode of scheme A2.

[0025] Further, in the embodiment, when generating the scheduling mapping plan mapping each training data shard to its target GPU in step S103, the training data shard with a predicted transmission time exceeding a preset threshold is also identified, and a prefetch instruction is generated for the identified training data shard; the prefetch instruction is used to instruct to start the transmission of the training data shard with a predicted transmission time exceeding a preset threshold to its corresponding target GPU in advance before the training calculation starts. The training efficiency of the distributed training task can be improved.

[0026] Further, in the embodiment, step S104 continuously monitors the running state of each target GPU in the execution of the distributed training task, and if the running state of any target GPU does not meet the preset running requirement, the following steps are performed: S301, the target GPU with a running state not meeting the preset running requirement (for example, the GPU temperature exceeding the preset temperature threshold is detected continuously for multiple times, and the video memory usage rate exceeds the preset video memory threshold) is marked as a faulty GPU, the set of GPUs to be called is updated, the distributed training task is triggered to check point pause, and the GPU marked as faulty can no longer undertake the subsequent training task, thereby avoiding data loss in the training process; S302, the set of affected training data shards on the faulty GPU is determined, the set of affected training data shards includes the training data shards participating in training on the faulty GPU and the training data shards not participating in training; for each affected training data shard, the locations of all available data sources storing the data of the training data shard are obtained; based on the locations of all available data sources of each affected training data shard, a target GPU is re-allocated to each affected data shard from the set of GPUs to be called, and the affected training data shard is transmitted from the corresponding selected available data source to the newly allocated target GPU, with the optimization objective of minimizing the predicted overall completion time of the distributed training task; S303, after the data transmission is completed, the execution of the distributed training task is restored from the checkpoint.

[0027] As an optional implementation, in this way, the training time of the distributed training task can be affected due to the performance reduction of an individual GPU, and all training data shards do not need to be retrained, which is beneficial to improve the training efficiency of the distributed training task. In addition, in step S302, the target GPU is re-allocated to all training data shards not participating in training and training data shards participating in training from the set of GPUs to be called with the optimization objective of minimizing the predicted overall completion time of the distributed training task, because multiple GPU replacements may be encountered during the training process, so the same training data shard may have related data stored on multiple GPUs.

[0028] Furthermore, in this embodiment, in step S302, the affected training data fragments are transferred from the corresponding selected available data sources to the newly allocated target GPU in batches. This helps avoid communication congestion caused by transferring a large amount of data at once.

[0029] Furthermore, this embodiment also provides a cluster GPU invocation system for distributed training tasks, including interconnected microprocessors and memory, wherein the microprocessors are programmed or configured to execute heterogeneous cluster GPU invocation methods for distributed training tasks.

[0030] In addition, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute a heterogeneous cluster GPU invocation method for a distributed training task via a processor.

[0031] In addition, this embodiment also provides a computer program product, including a computer program or instructions that are programmed or configured to execute a heterogeneous cluster GPU invocation method for a distributed training task via a processor.

[0032] Those skilled in the art will understand that the technical solutions provided by the embodiments of this application may be in the form of a method, system, or computer program product. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an implementation for the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0033] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A heterogeneous cluster GPU invocation method for distributed training tasks, characterized in that, The method comprises the following steps: S101, determining a set of GPUs available for performing a current distributed training task in a cluster, and taking the set of GPUs as a set of GPUs to be invoked; S102, obtaining a processing performance parameter of each GPU in the set of GPUs to be invoked; obtaining a current storage location of each shard in a set of training data shards of the distributed training task, and calculating an estimated transmission time of each training data shard from the current storage location to each GPU in the set of GPUs to be invoked; S103, taking minimization of an estimated overall completion time of the distributed training task as an optimization objective, jointly optimizing based on the obtained processing performance parameter of each GPU in the set of GPUs to be invoked and the estimated transmission time of each training data shard to each GPU in the set of GPUs to be invoked, and generating a scheduling and mapping plan for mapping each training data shard to a target GPU based on the joint optimization; S104, scheduling and transmitting each training data shard from its current storage location to its corresponding target GPU according to the scheduling and mapping plan to perform the distributed training task.

2. The heterogeneous cluster GPU invocation method for distributed training tasks according to claim 1, wherein, The processing performance parameter is obtained by periodic sampling and is updated by using an exponential smoothing algorithm.

3. The heterogeneous cluster GPU invocation method for distributed training tasks according to claim 1, wherein, The processing performance parameter comprises a GPU processing rate.

4. The heterogeneous cluster GPU invocation method for distributed training tasks according to claim 1, wherein, An expression of the estimated overall completion time of the distributed training task is: ， wherein, denotes a set of GPUs to be invoked, denotes one GPU in the set of GPUs to be invoked, denotes a set of training data shards allocated to a GPU, denotes one training data shard in the set of training data shards allocated to GPU denotes a number of samples of training data shard denotes a processing rate of GPU denotes an estimated time for training data shard to be transferred to GPU .​​​ 5. The heterogeneous cluster GPU invocation method for distributed training tasks according to claim 4, wherein, In step S103, when the scheduling and mapping plan for mapping each training data shard to a target GPU is generated, training data shards with an estimated transmission time exceeding a preset threshold are also identified, and a prefetch instruction is generated for the identified training data shards; the prefetch instruction is used to instruct to start transmission of the training data shards with an estimated transmission time exceeding a preset threshold to their corresponding target GPUs in advance before training calculation starts.

6. The heterogeneous cluster GPU invocation method for distributed training tasks according to claim 1, wherein, Step S104 continuously monitors the running state of each target GPU during execution of the distributed training task, and if the running state of any target GPU does not meet a preset running requirement, the following steps are performed: S301, marking the target GPU whose running state does not meet the preset running requirement as a faulty GPU, updating the set of GPUs to be invoked, and triggering a checkpoint pause of the distributed training task; S302, determining a set of affected training data shards on the faulty GPU, the set of affected training data shards comprising training data shards participating in training and training data shards not participating in training on the faulty GPU; For each affected training data shard, obtaining locations of all available data sources storing data of the training data shard; Based on the locations of all available data sources of each affected training data shard, reassigning a target GPU to each affected data shard from the set of GPUs to be invoked, and transmitting the affected training data shard from the corresponding selected available data source to the newly assigned target GPU, taking minimization of the estimated overall completion time of the distributed training task as the optimization objective; S303, after data transmission is completed, resuming execution of the distributed training task from the checkpoint.

7. The heterogeneous cluster GPU invocation method for distributed training tasks according to claim 6, wherein, In step S302, the affected training data shards are transferred from the respective selected available data source to the newly assigned target GPU using batched transfers.

8. A cluster GPU invocation system for distributed training tasks, comprising microprocessors and memories interconnected, characterized in that, The microprocessor is programmed or configured to perform the method for invoking heterogeneous cluster GPUs for distributed training tasks according to any one of claims 1-7.

9. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the method for invoking heterogeneous cluster GPUs for distributed training tasks according to any one of claims 1-7 by a processor.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the method for invoking heterogeneous cluster GPUs for distributed training tasks according to any one of claims 1-7 by a processor. The computer program or instructions are programmed or configured to perform the method for invoking heterogeneous cluster GPUs for distributed training tasks according to any one of claims 1-7 by a processor.

Images