Distributed task execution method and system under GPU (Graphics Processing Unit) mixing

Abstract

The embodiment of the invention provides a distributed task execution method and system under GPU (Graphics Processing Unit) mixing, which are applied to a platform comprising a plurality of GPUs, a first class of tasks and a second class of tasks are deployed, and the priority of the second class of tasks is lower than that of the first class of tasks. The task execution method comprises the steps that a target application calls a set communication library and requests to execute a target communication operation between GPUs, and the target application belongs to a second type of tasks. Then, the set communication library determines a sub-task list involved in the target communication operation, and the sub-task list comprises a plurality of sub-tasks distributed in different GPUs; and enabling the plurality of subtasks to synchronously submit communication kernel l to a GPU driving environment through an inter-process synchronous operation. And under the condition that the GPU driving environment determines that the communication kernel l starts synchronous submission, the communication kernel l is submitted to the GPU immediately.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to hardware computing in the computer field, and more particularly, to distributed task execution in the case of mixed GPU deployment. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the demand for computing power has increased exponentially, and the supply of GPUs is relatively scarce. At the same time, there is usually a lot of waste in the use of GPUs. For example, there is an obvious tidal phenomenon in the traffic of GPU inference services. When the GPU is used exclusively, the utilization rate is low, and the scarce GPU resources cannot be fully utilized. Therefore, a GPU hybrid solution has been proposed recently, which deploys latency-sensitive online inference services and heavy-throughput offline tasks on one card to share GPU resources. By adopting a certain computing power and video memory scheduling mechanism, one card can be used for two purposes, peak shaving and valley filling, and limited GPU resources can be used to support more business needs while improving GPU utilization efficiency.

[0003] The rapid development of large models in recent years has led to an increasing demand for distributed deployment of multiple GPU cards. When deploying distributed offline tasks, the inconsistent execution progress on different cards causes long waiting times for collaborative communication, which occupies a large amount of GPU time slices and affects online service performance.

[0004] Therefore, an improved solution is needed that can better perform collaborative communication and improve GPU resource utilization when GPUs are deployed in mixed distributed tasks. Summary of the invention

[0005] One or more embodiments of this specification describe a distributed task execution method and system under GPU hybrid deployment, which can avoid communication waiting and resource waste caused by communication operations of distributed applications under conventional circumstances, and improve service performance and GPU resource utilization.

[0006] According to a first aspect, a distributed task execution method under GPU colocation is provided, which is applied to a platform including multiple GPUs, wherein first-category tasks and second-category tasks are deployed, and the priority of the second-category tasks is lower than that of the first-category tasks; the method comprises:

[0007] The target application sends a call request to the collective communication library to request execution of a target communication operation between GPUs, wherein the target application belongs to the second type of task;

[0008] The collective communication library determines a subtask list involved in the target communication operation according to the call request, including multiple subtasks distributed on different GPUs; through inter-process synchronization operations, the multiple subtasks synchronously submit communication kernels to the GPU driver environment;

[0009] When determining that the communication kernel 1 enables synchronous submission, the GPU driver environment immediately submits the communication kernel 1 to the GPU.

[0010] According to one implementation, before the target application calls the collective communication library, the target application loads the collective communication library and initializes the GPU driver environment; the GPU driver environment obtains the current configuration parameters related to the task deployment, and determines whether the target application belongs to a multi-card container deployment distributed on multiple GPUs based on the configuration parameters; in response to the GPU driver environment determining the multi-card container deployment, the collective communication library determines to start synchronous submission and notifies the GPU driver environment.

[0011] In one embodiment of the above implementation, the GPU driver environment calls the GPU virtualization driver module to obtain the configuration parameters of the current vGPU, including the number of vGPUs and the corresponding physical GPUs.

[0012] According to one implementation, the execution method further includes: the collective communication library determines whether the data size corresponding to the target communication operation exceeds a preset threshold; if it exceeds the preset threshold, the target communication operation is divided into multiple sub-communication operations.

[0013] According to one implementation, the collective communication library determines a subtask list involved in a target communication operation, including: if the target communication operation is point-to-point communication, obtaining two subtasks parsed during kernel preprocessing to form the subtask list.

[0014] Alternatively, if the target communication operation is a collective communication operation, all subtasks involved in the target application are added to the subtask list.

[0015] Furthermore, in different examples, the collective communication operation may include one of the following: an all-reduce operation, a broadcast operation, a reduce operation, and an aggregate operation.

[0016] According to one implementation, the collective communication library performs inter-process synchronization operations, including: determining an identification code for this synchronization based on the cumulative number of times the subtask set in the subtask list jointly performs communication operations of the same type as the target communication operation; and calling an inter-process synchronization algorithm based on the identification code.

[0017] In a specific embodiment, determining the identification code of this synchronization may include: if the target communication operation is point-to-point communication, encoding the two subtask identifiers in the subtask list and the cumulative number of point-to-point communication kernels submitted by the two subtasks into the identification code; if the target communication operation is a first type of collective communication operation, encoding the cumulative number of submissions of the first type of communication kernel into the identification code.

[0018] According to one implementation, the execution method further includes: the communication library records the time of synchronously submitting the communication kernel to the GPU driver environment; monitoring the execution status of the communication kernel through a monitoring thread, and ending the corresponding process if the communication kernel is not completed after exceeding a predetermined time threshold.

[0019] According to one implementation, the execution method further includes: the GPU driver environment receives a computing kernel submitted by the target application; the GPU driver environment submits the computing kernel to the GPU within a permitted time slice, wherein the permitted time slice is a time slice that is determined based on the performance indicator requirements of the first type of task and that the target application can use for computing.

[0020] According to a second aspect, a distributed task execution system under GPU colocation is provided, which is applied to a platform including multiple GPUs, wherein first-class tasks and second-class tasks are deployed, and the priority of the second-class tasks is lower than that of the first-class tasks; the system comprises:

[0021] The collective communication library is used to receive a call request from a target application to execute a target communication operation between GPUs, determine a subtask list involved in the target communication operation according to the call request, including multiple subtasks distributed on different GPUs; and enable the multiple subtasks to synchronously submit communication kernels to the GPU driver environment through inter-process synchronization operations; wherein the target application belongs to the second type of task;

[0022] The GPU driving environment is used to immediately submit the communication kernel 1 to the GPU when it is determined that the communication kernel 1 enables synchronous submission.

[0023] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0024] According to a fourth aspect, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0025] According to a fifth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method of the first aspect is implemented.

[0026] In the embodiments of this specification, when distributed BE type applications and GT type applications are mixed and deployed on the same GPU, the communication operation of the BE application may cause uncontrollable communication waiting. Through the coordination between the collective communication library and the GPU driver environment, the subtasks on each GPU involved in the communication operation can submit the communication kernel to the GPU driver environment synchronously, which greatly reduces the communication waiting time on the GPU, thereby avoiding the performance impact on the GT service. Furthermore, for communication operations with a relatively large data scale, the schemes of some embodiments reduce the probability of long kernels by data segmentation. In addition, after the collective communication library submits the kernel, the monitoring thread can be started to detect the execution time of the kernel in real time, and the risk of computing power suppression failure can be avoided by exiting the process when necessary. In this way, through the scheme of the embodiments of this specification, the execution performance of GT tasks can be better guaranteed and the utilization rate of GPU resources can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1 The following is a schematic diagram showing the execution of distributed BE tasks in the case of GPU colocation;

[0029] Figure 2 A schematic diagram of a system architecture according to an embodiment is shown;

[0030] Figure 3 A schematic diagram of a distributed task execution process according to an embodiment is shown;

[0031] Figure 4 Shows the detailed operation flow in the NCCL collective communication library during the application running phase;

[0032] Figure 5 Shows the execution and scheduling logic of kernel in Cuda. DETAILED DESCRIPTION

[0033] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0034] First, some terms used in this article as technical background are introduced and explained.

[0035] SLA: service-level agreement, service-level agreement, refers to the service level that the supplier promises to provide to customers. In the GPU scenario, it generally refers to the user's demand for resource quality.

[0036] vGPU: Virtual GPU, which uses virtualization technology to virtualize a GPU into multiple small-sized GPUs, allowing multiple tasks to share the same physical GPU.

[0037] GT (Guaranteed) tasks: tasks with higher SLA requirements, usually have clear requirements for indicators such as request pressure (QPS) or access latency (RT), and have stricter requirements on resource quality.

[0038] BE (Best Effort) tasks: tasks with lower SLA requirements, usually some computing-intensive task applications, with good fault tolerance and retry capabilities, and relatively loose requirements on resource quality. In practice, some AI large model training tasks and offline reasoning tasks are BE tasks.

[0039] Colocation: Colocation is to deploy multiple containers on the same physical GPU through vGPU to improve hardware utilization. The applications in these containers include both GT and BE types.

[0040] Kernel: Function code executed on the GPU.

[0041] Rank: In a distributed task, each subtask or instance executed in parallel can be identified by a rank. Specifically, when using GPUs for parallel computing, rank is usually used to refer to a unique identifier for each process in a communication group. The process running on each GPU card can be regarded as an independent rank.

[0042] It is understandable that GPUs, as a type of acceleration hardware used on a large scale in AI scenarios, are generally scarce and have low utilization. GPU virtualization technology can be used to implement multi-container and multi-task co-location on a single card, improving resource utilization and better reusing GPU resources. In addition, the rapid development of large models has led to an increasing demand for multi-card distributed deployment of GPUs.

[0043] It should be noted that GPU colocation and distributed deployment are concepts of different dimensions. Colocation refers to the deployment of different types of loads or tasks of different priorities on a single GPU card from the perspective of the GPU card, such as deploying GT tasks and BE tasks at the same time. Distributed deployment, on the other hand, is a description from the perspective of application tasks. Distributed deployment of tasks means that an application task is deployed on multiple GPUs for execution.

[0044] When colocation and distribution coexist, for example, when multi-card distributed BE workloads are colocation with other GT workloads, it is difficult to guarantee the SLA requirements of GT workloads due to the uncontrollable GPU communication operations. Specifically, the execution progress of distributed BE workloads on different GPU cards may be inconsistent, and there is a problem that the communication kernel on a certain card waits for a long time for other cards, thus affecting the execution of GT workloads.

[0045] Figure 1 The following figure shows the execution diagram of distributed BE tasks in the case of GPU colocation. Figure 1 In the example, it is assumed that a BE task is distributed and deployed on two physical GPUs in the hardware platform, namely GPU-0 and GPU-1. A subtask of the BE task is executed in GPU-0, which is recorded as Rank0, and another subtask of the BE task is executed in GPU-1, which is Rank1. At the same time, GPU-0 and GPU-1 are co-located GPUs, in which, in addition to the above BE tasks, other GT tasks are also deployed.

[0046] It is understandable that, since the execution priority of GT tasks is higher than that of BE tasks, the GPU scheduling unit will adjust the time slices or time ratios of GT tasks and BE tasks using GPU computing power according to the SLA indicators of GT tasks, and in this process will suppress the execution of BE tasks. The GT task loads of different GPUs are different, and the degree of suppression of different ranks of BE tasks is also different.

[0047] like Figure 1 As shown, Rank0 on GPU-0 first performs calculations in the calculation stream, and then needs to communicate with Rank1 to exchange data before subsequent operations can be performed. Therefore, Rank0 submitted the communication kernel to the GPU at time T0. However, Rank1 on GPU-1, due to a large degree of suppression, may perform calculations later and not submit the communication kernel to the GPU until time T1. Therefore, Rank0 has to wait for a long time for Rank1 to submit the communication kernel. The waiting time T1-T0 may even far exceed the time required to execute the communication kernel (T2-T1). Since there can only be one task process context being executed on the GPU at a time, during the above waiting time, GPU-0 is occupied by the context of the BE task for a long time, which seriously affects the performance of the GT service on it and also affects the GPU resource utilization.

[0048] In the related technologies, the research and optimization of GPU colocation technology mainly starts from the dimension of single physical GPU sharing, focusing on the refined allocation and scheduling strategy of GPU computing power, and does not consider the difference between distributed load colocation and single card load colocation. This makes it difficult to deploy distributed BE applications and GT applications on the same GPU. Figure 1 As shown in the figure, the communication operations of BE applications may seriously disrupt the carefully designed computing power allocation and scheduling strategies, thereby breaking the system's SLA guarantee for GT services.

[0049] In view of this, this manual proposes a solution for the scenario of distributed task GPU co-location. In this solution, the multi-card collaborative communication operations of distributed BE tasks are synchronously scheduled to avoid the problem of long waiting time and GPU occupation of communication operations between GPU cards, effectively guarantee the online service SLA, support the needs of co-location of multi-card offline BE tasks and online GT services, and improve GPU resource utilization.

[0050] Figure 2 FIG. 2 shows a schematic diagram of a system architecture according to an embodiment. Figure 2 As shown in the figure, the bottom layer of the system architecture is the hardware layer, which includes multiple physical GPUs, or multiple GPU cards. Above the GPU hardware layer is the GPU driver environment. Figure 2 The example shown is a Cuda driver.

[0051] Cuda driver is the programming environment for Nvidia's GPU hardware. Cuda can abstract the operations and functions of GPU hardware and provide them as programming interfaces. The upper-level application's task execution in the GPU must be implemented through the interface provided by Cuda.

[0052] The GPU virtualization module is a functional module implemented based on Cuda. It is used to virtualize a single physical GPU into multiple small-sized virtual GPUs, namely vGPUs, according to the needs of the task container through virtualization technology, and deploy the task container in the corresponding vGPU.

[0053] On top of the GPU virtualization module, multiple services or tasks with different priorities are run and deployed. These services or tasks can specifically include the first type of tasks with higher priority, namely GT services, and the second type of tasks with relatively lower priority, namely BE tasks. Among them, BE tasks are distributed and deployed on multiple GPUs. Since it is a distributed task, the execution of the upper-level BE tasks also depends on the collective communication library. Figure 2 The NCCL communication library is shown as an example.

[0054] NCCL is a collective communication library designed for Nvidia GPUs. It is widely used in distributed GPU tasks to implement collective communication and point-to-point communication between GPU cards. Point-to-point communication can be understood as communication between two specified GPUs (more specifically, their ranks). Collective communication is applied to joint communication operations between all GPUs involved in a distributed task. Specifically, the NCCL library provides a variety of collective communication operations, such as all-reduce, broadcast, reduce, scatter, and gather.

[0055] For example, a full reduce operation combines (reduces) data sets on all GPUs into a single value, and then broadcasts this value back to all GPUs. A broadcast operation is where data on one GPU is copied to all other GPUs. This may involve kernels for data copying and transfer. A reduce operation combines data on all GPUs into a single value, usually an aggregation operation such as sum, maximum, or minimum. A scatter operation is where data is spread from one GPU to multiple GPUs, with each GPU getting a portion of the data. A gather operation is where data fragments from multiple GPUs are gathered onto one GPU.

[0056] Through these collective communication operations of NCCL, data parallelism and synchronization between GPUs can be achieved.

[0057] In order to synchronize the multi-card cooperative communication of distributed BE tasks in the GPU colocation scenario, in the embodiments of this specification, the existing GPU driver environment and collective communication library are improved. The improved part is Figure 2 Specifically, based on the existing collective communication library, the following functional logic is added: for distributed BE tasks, coordinate the communication tasks to be submitted on each GPU so that each rank submits the communication kernel synchronously. Furthermore, if the communication data scale is too large, it is divided into multiple smaller-scale communications. If the execution time of the submitted communication kernel is too long, the process exit is triggered to avoid affecting the performance of the GT service.

[0058] Improved GPU driver environment, for multi-card distributed BE tasks, distinguishes between communication kernel and computing kernel. For communication kernel, it is quickly submitted to GPU hardware to complete execution as soon as possible. For computing kernel, task scheduling is planned according to the SLA requirements of GT tasks, and computing power is dynamically suppressed.

[0059] The following describes the execution of distributed BE tasks in conjunction with the Cuda driver environment (hereinafter referred to as Cuda driver or Cuda) and NCCL collective communication library corresponding to Nvidia GPU products. However, it should be understood that in the case of GPU hardware products from other manufacturers, the corresponding GPU driver environment and collective communication library can be used interchangeably.

[0060] Figure 3 FIG. 2 shows a schematic diagram of a distributed task execution process according to an embodiment. Figure 3 As shown in the figure, the overall execution process can be divided into the initialization phase and the running phase. They are described below.

[0061] In the initialization phase, when the process of the target application is started, the NCCL collective communication library is loaded in step S11, and the Cuda driver is initialized in S12. It can be understood that the target application here is a BE class task, and its execution priority is lower than that of a GT class task. The target application initializes the Cuda driver to obtain its own execution environment.

[0062] Next, in step S13, the Cuda driver obtains the current configuration parameters related to task deployment during initialization, and determines whether the target application is a multi-card deployment distributed on multiple GPUs based on the configuration parameters, which is used to determine whether it is necessary to enable communication collaborative submission optimization for distributed BE tasks, i.e., synchronous submission. Specifically, the Cuda driver can call the GPU virtualization driver module to obtain the configuration parameters of the current vGPU, including, for example, the number of vGPUs, the corresponding physical GPU, etc., so as to determine whether the current target application is a distributed multi-card deployment.

[0063] Next, in step S14, the NCCL communication library queries the Cuda driver about the multi-card deployment, and the Cuda driver returns a response message to the NCCL communication library indicating whether the target application is multi-card deployed. In step S15, NCCL determines whether to enable synchronous submission based on the response information and notifies the Cuda driver. Specifically, if the response message indicates that the target application is a BE task deployed with distributed multi-cards, the NCCL collective communication library determines to enable synchronous submission of the communication kernel for the target application and notifies the Cuda driver. The Cuda driver accordingly marks the target application as enabling synchronous submission.

[0064] It should be noted that the synchronous submission of the communication kernel by the NCCL layer may cause certain performance losses and even the risk of deadlock. Therefore, in the NCCL initialization phase, it is determined whether to enable the synchronous submission mode based on the specific scenario, so that the synchronous submission of the communication kernel is only enabled for distributed BE tasks, which can avoid unnecessary performance losses and improve GPU performance overall.

[0065] Next, enter the running phase of the application.

[0066] In the running stage, in step S21, the target application sends a call request to the collective communication library to request to execute the target communication operation between GPUs. That is, the target application submits a communication task request by calling the collective communication library.

[0067] Optionally, the collective communication library first determines whether the data size of the target communication operation exceeds a preset threshold. If it does not exceed the preset threshold, the subsequent processing is directly performed. If it exceeds the preset threshold, it is divided into multiple sub-communication operations. In the case of segmentation, the target communication operation mentioned below refers to each sub-communication operation obtained by segmenting the original target communication operation.

[0068] Next, the collective communication library determines the list of subtasks (rank list) involved in the target communication operation, which includes multiple subtasks distributed on different GPUs; and through inter-process synchronization operations, multiple subtasks synchronously submit communication kernels to the GPU driver environment. Figure 2 It is simplified as step S22, which synchronizes the ranks participating in the communication and submits the communication kernel to Cuda.

[0069] After the communication kernel is submitted to the Cuda driver, in step S23, Cuda detects the communication kernel and determines whether execution permission is required. For the communication kernel with synchronous submission enabled, the communication kernel is immediately executed in the GPU without waiting for execution permission.

[0070] In addition, after submitting the communication kernel to Cuda, the collective communication library repeatedly queries whether the communication kernel has been executed in step S24. If the execution has not been completed for a certain period of time, the corresponding process is forcibly exited to avoid affecting the execution of the GT task.

[0071] The following describes in detail the execution logic in the collective communication library and GPU driver environment.

[0072] Figure 4 The following figure shows the detailed operation flow in the NCCL collective communication library during the application running phase. Figure 4As shown, first, in step S41, the upper target application calls the NCCL communication library. It can be understood that the NCCL communication library provides a plurality of communication-related interface APIs for the upper application, and the upper application can call the corresponding APIs as needed to implement inter-GPU task communication without paying attention to the underlying details such as the implementation of the communication kernel. In one embodiment, the target application can issue a call request to the NCCL communication library, which includes at least the API name and usually also includes necessary parameters, such as the GPU group participating in the communication, the data buffer to be transmitted, etc. Usually, the API name can indicate what kind of communication operation is requested to be performed, for example, API ncc lAll lReduce indicates a full-reduce operation, API ncc lBroadcast indicates a broadcast operation, and so on.

[0073] Next, the NCCL communication library may determine a list of subtasks involved in the communication operation to be executed according to the call request of the target application.

[0074] Specifically, in step S42, the NCCL communication library determines whether the communication operation indicated by the call request is point-to-point communication. This can be known by the name of the API called. If it is point-to-point communication, in step S43, when preprocessing the kernel, the two ranks (i.e., subtasks) participating in the communication are parsed from the parameters of the call request, and the two ranks are recorded in the list of subtasks to be synchronized. Then proceed to step S44, NCCL starts loading the communication kernel corresponding to the target communication operation.

[0075] If the communication operation indicated by the call request is not point-to-point communication, it is assumed to be a collective communication operation, and the above rank parsing is not required. The ranks on all GPUs constitute the above subtask list by default, and step S44 is directly entered. As described above for the NCCL collective communication library, the collective communication operation includes one of the following: full reduction operation, broadcast operation, reduction operation, and aggregation operation.

[0076] After loading the communication kernel in step S44, in step S45, it is determined whether synchronous submission is enabled for the current application. This can be understood as a query on the judgment result of step S15 in the initialization phase. If it is not enabled, in step S46, the communication kernel is directly submitted to Cuda. If it is determined to enable synchronous submission, then as shown in the right branch, in step S47, the NCCL communication library calls the inter-process synchronization operation so that each subtask rank synchronously submits the communication kernel. Regarding inter-process synchronization, Figure 4 The example in this article uses the Barrier mechanism. A barrier is a synchronization mechanism that ensures that all processes will not continue to execute until they reach a certain point. It is often used to synchronize the execution phases of multiple processes in parallel computing.

[0077] In order to achieve synchronization between subtasks, we can first count the number of ranks managed by the current thread. If it is greater than 1, a new thread is created for each managed rank to perform subsequent synchronization operations to avoid blocking existing threads.

[0078] Then obtain the rank list to be synchronized this time, and calculate the identification code for this synchronization. This is because there may be multiple communication combinations that need to be synchronized at the same time. In order to avoid mutual interference between different combinations during synchronization, an identification code needs to be determined for each synchronization combination. In one embodiment, the identification code for this synchronization can be determined based on the cumulative number of times the subtask set in the above rank list jointly executes the same type of communication operation as the target communication operation.

[0079] Specifically, for a point-to-point communication operation, the identifiers of the two ranks participating in the communication and the cumulative number of point-to-point communication kernels submitted between them can be encoded into an identification code. For example, if the aforementioned target communication operation is a point-to-point communication operation performed between rank0 and rank1, and 10 point-to-point communication operations have been performed between rank0 and rank1, then the identifiers of rank0, rank1, and the number of times 10 can be encoded into an identification code, such as (01)-10. In this way, through the identification code, it can be accurately and uniquely located that the combination to be synchronized is the 11th point-to-point communication kernel to be submitted between rank0 and rank1.

[0080] For collective communication operations, the cumulative submission times of the same type of communication operation kernels can be encoded into the identification code. For example, assuming that the target communication operation to be executed is a certain type of collective communication operation, such as a reduction operation, and the overall reduction operation has been executed 8 times, that is, the cumulative number of communication kernels for the reduction operation is 8, then the number can be encoded into the identification code. In this way, it can be uniquely located that the combination to be synchronized is the 9th communication kernel for the reduction operation to be submitted between all ranks.

[0081] After determining the identification code of the combination to be synchronized, the inter-process synchronization algorithm can be called based on the identification code to make the corresponding ranks synchronously submit the communication kernel indicated by the identification code. There are already a variety of inter-process synchronization mechanisms and synchronization algorithms, which can be selected and used as needed to achieve process synchronization. For example, a dissemination algorithm can be used to ensure that the state copies of all processes are consistent through socket message passing to achieve synchronization between processes.

[0082] Through inter-process synchronization, it is ensured that in step S48, each rank of the target application synchronously submits the communication kernel to Cuda. In addition, as shown in step S49, the collective communication library also records the time of synchronously submitting the communication kernel to the GPU driver environment, and monitors the execution status of the above communication kernel through the monitoring thread. If the communication kernel is not completed after exceeding the predetermined time threshold, the corresponding process (corresponding to Figure 3 Step S24) to avoid long-term occupation of GPU resources and affecting GT tasks deployed on the same GPU.

[0083] The following describes the execution and scheduling logic in Cuda after any kernel is submitted to Cuda. The process is as follows: Figure 5 shown.

[0084] like Figure 5 As shown in FIG. 1 , when the Cuda driver receives a kernel to be submitted, first in step S51, the corresponding kernel is started and loaded, and in step S52, the priority of the corresponding task is determined, that is, whether it is a GT task or a BE task. If it is a GT task, it directly jumps to step S57 and directly submits the kernel to the GPU. If it is a BE application task, in step S53, by parsing the name of the kernel, it is determined whether the kernel is an NCCL communication kernel.

[0085] If the result of step S53 shows that the current kernel is not a communication kernel, that is, a computing kernel, then in step S54, wait for execution permission. This is because, for the computing kernel, the GPU virtualization module will dynamically adjust the time slice of the BE task using the GPU computing power according to the SLA indicator of the GT load. Only within the scheduled time slice will the execution permission of the BE task computing kernel be granted, allowing it to submit its computing kernel. Therefore, while waiting for the execution permission, the process proceeds to step S57 to submit the kernel to the GPU.

[0086] If the result of step S53 shows that the current kernel is a communication kernel, then step S55 continues to determine whether NCCL collaborative submission is used, that is, whether the aforementioned synchronous submission function is enabled for the current BE task. Figure 3 In the initialization phase of step S15, the NCCL collection communication library determines to enable synchronous submission of the communication kernel for a certain application and notifies Cuda. Cuda accordingly marks the application as enabling synchronous submission. In step S55 here, Cuda determines whether synchronous submission is enabled by reading the mark corresponding to the current application.

[0087] If synchronous submission is enabled for the current communication kernel, Cuda immediately submits the kernel to the GPU for execution. If the NCCL layer is not synchronized, it is determined to be an abnormal situation, and in step S56, the process is exited to avoid affecting the performance of the GT service.

[0088] Figure 5 It is the execution logic for any kernel. It can be understood that for the target application belonging to the BE task, when Cuda receives the communication kernel submitted synchronously through the NCCL collective communication library, the process executes the branch of steps S51, S52, S53, S55 to S57, that is, once it is determined that it is a communication kernel of a distributed BE task with synchronous submission enabled, the communication kernel is immediately submitted to the GPU for execution. For the calculation kernel submitted by the target application, the process executes the branch of steps S51, S52, S53, S54 to S57, and Cuda submits the calculation kernel to the GPU within the permitted time slice, where the permitted time slice is the time slice that the target application can use for calculation as determined by the GPU virtualization module according to the performance index requirements of the GT task.

[0089] It should be understood that, for the sake of clarity, the above embodiments are specifically described in conjunction with the NCCL collective communication library and Cuda driver environment for Nivida GPU. However, those skilled in the art understand that GPUs from other manufacturers may also have different collective communication libraries and driver environments. The technical concepts in the above embodiments can be equally extended to other collective communication libraries and driver environments.

[0090] Looking back at the above overall process, when distributed BE type applications and GT type applications are mixed and deployed on the same GPU, the communication operation of the BE application may seriously break the system's SLA guarantee for GT services. In this regard, in the solution of the embodiment of this specification, the collective communication library and the GPU driver environment coordinate and cooperate with each other so that the communication kernels on all GPUs are submitted synchronously, which greatly reduces the communication waiting time on the GPU, thereby avoiding the performance impact on the GT service. Furthermore, for communication operations with relatively large data scales, the solutions of some embodiments reduce the probability of long kernels by data segmentation. In addition, after the collective communication library submits the kernel, the monitoring thread can be started to detect the execution time of the kernel in real time, and when necessary, the risk of computing power suppression failure can be avoided by exiting the process. In this way, through the solution of the embodiment of this specification, the execution performance of GT tasks can be better guaranteed and the utilization rate of GPU resources can be improved.

[0091] According to another embodiment, a distributed task execution system under GPU colocation is provided, which is applied to a platform including multiple GPUs, wherein first-class tasks and second-class tasks are deployed, and the priority of the second-class tasks is lower than that of the first-class tasks. The system includes:

[0092] The collective communication library is used to determine the subtask list involved in the target communication operation, including multiple subtasks distributed on different GPUs, when called by the target application to request to perform the target communication operation between GPUs; and to enable the multiple subtasks to synchronously submit the communication kernel to the GPU driver environment through inter-process synchronization operation; wherein the target application belongs to the second type of task;

[0093] The GPU driving environment is used to immediately submit the communication kernel 1 to the GPU when it is determined that the communication kernel 1 enables synchronous submission.

[0094] For examples of specific implementations of the system, please refer to the examples and descriptions of the aforementioned combined execution method, which will not be repeated here.

[0095] According to another aspect of the embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the aforementioned method.

[0096] According to yet another embodiment, a computer program product is provided, comprising a computer program / instruction, which implements the aforementioned method when executed by a processor.

[0097] According to yet another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the aforementioned method is implemented.

[0098] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0099] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed task execution method under GPU colocation, applied to a platform including multiple GPUs, wherein first-class tasks and second-class tasks are deployed, and the priority of the second-class tasks is lower than that of the first-class tasks; the method comprises: The target application sends a call request to the collective communication library to request execution of a target communication operation between GPUs, wherein the target application belongs to the second type of task; The collective communication library determines a subtask list involved in the target communication operation according to the call request, including multiple subtasks distributed on different GPUs; through inter-process synchronization operations, the multiple subtasks synchronously submit communication kernels to the GPU driver environment; The GPU driver environment immediately submits the communication kernel to the GPU when determining that synchronous submission is enabled for the communication kernel.

2. The method according to claim 1, wherein: Before the target application issues a call request to the collective communication library, it also includes: The target application loads the collective communication library and initializes the GPU driver environment; The GPU driver environment obtains the current configuration parameters related to task deployment, and determines whether the target application belongs to multi-card container deployment distributed on multiple GPUs based on the configuration parameters. In response to the GPU driver environment determining the multi-card container deployment, the collective communication library determines to enable synchronous submission and notifies the GPU driver environment.

3. The method according to claim 2, wherein: The GPU driver environment obtains the current configuration parameters related to task deployment, including: The GPU driver environment calls the GPU virtualization driver module to obtain the current vGPU configuration parameters, including the number of vGPUs and the corresponding physical GPUs.

4. The method according to claim 1, further comprising: The collective communication library determines whether the data size corresponding to the target communication operation exceeds a preset threshold; If the preset threshold is exceeded, the target communication operation is divided into multiple sub-communication operations.

5. The method according to claim 1, wherein: The collective communication library determines a list of subtasks involved in the target communication operation according to the call request, including: If the communication operation indicated by the call request is point-to-point communication, two subtasks parsed during kernel preprocessing are obtained to form the subtask list.

6. The method according to claim 1, wherein: The collective communication library determines a list of subtasks involved in the target communication operation according to the call request, including: If the communication operation indicated by the call request is a collective communication operation, all subtasks involved in the target application are added to the subtask list.

7. The method according to claim 6, wherein: The collective communication operation includes one of the following: an all-reduce operation, a broadcast operation, a reduce operation, and an aggregate operation.

8. The method according to claim 1, wherein: The inter-process synchronization operation includes: Determine the identification code of this synchronization according to the accumulated number of times that the subtask set in the subtask list jointly performs the communication operation of the same type as the target communication operation; Based on the identification code, an inter-process synchronization algorithm is called.

9. The method according to claim 8, wherein: Determine the identification code for this synchronization, including: If the target communication operation is point-to-point communication, two subtask identifiers in the subtask list and the cumulative number of point-to-point communication kernels submitted by the two subtasks are encoded into an identification code; If the target communication operation is a first type of collective communication operation, the number of times the first type of communication kernel is cumulatively submitted is encoded into the identification code.

10. The method according to claim 1, further comprising: The collective communication library records the time of synchronously submitting the communication kernel to the GPU driver environment; The execution state of the communication kernel is monitored by a monitoring thread, and when the communication kernel is not completed after exceeding a predetermined time threshold, the corresponding process is terminated.

11. The method according to claim 1, further comprising: The GPU driver environment receives the computing kernel submitted by the target application; The GPU driver environment submits the computing kernel to the GPU within a permitted time slice, wherein the permitted time slice is a time slice that is determined based on the performance indicator requirements of the first type of tasks and that is available for computing by the target application.

12. A distributed task execution system under GPU colocation, applied to a platform including multiple GPUs, wherein first-class tasks and second-class tasks are deployed, and the priority of the second-class tasks is lower than that of the first-class tasks; the system comprises: A collective communication library, configured to receive a call request from a target application requesting to execute a target communication operation between GPUs, and determine a subtask list involved in the target communication operation according to the call request, including a plurality of subtasks distributed on different GPUs; and through inter-process synchronization operations, the plurality of subtasks are caused to synchronously submit communication kernels to the GPU driver environment; wherein the target application belongs to the second type of task; The GPU driving environment is used to immediately submit the communication kernel to the GPU when it is determined that the communication kernel enables synchronous submission.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

14. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.

Images