A communication and computing collaborative scheduling method and system for large language model specialist parallelism

Abstract

The application discloses a communication and calculation cooperative scheduling method and system for large language model experts in parallel, and through optimization of a task issuing and execution process on a GPU, fine-grained parallelism of Dispatch communication and grouped GEMM calculation is realized: after a Dispatch data sending task is completed, a Dispatch data receiving task and a grouped GEMM task are simultaneously started, and the two tasks are run in parallel; the receiving task takes an expert granularity as a unit, and whenever all tokens of a certain expert are received from all source GPUs, a completion signal is sent to the grouped GEMM task, triggering GEMM calculation of the expert, and there is no need to wait for all expert data to be ready. The application further optimizes queue pair configuration and work request scheduling strategies of cross-host RDMA communication, combines fine allocation of calculation resources and a CUDA graph recording and replay mechanism, and significantly reduces communication overhead and scheduling delay.

Description

Technical Field

[0001] This invention relates to the fields of network communication and high-performance technology, specifically to a communication and computational collaborative scheduling method and system for parallel operation of large language model experts. Background Technology

[0002] Many advanced large language models currently employ a MoE (Mixture of Experts) architecture. A MoE model contains many small neural networks, each called an expert. During training or inference, each token activates only a subset of experts for computation. Because large language models have a massive number of parameters, they may require multiple GPUs, or even multiple servers containing multiple GPUs, for parallel computation. A popular parallel computing approach is called expert parallelism, which distributes different experts across different GPUs for parallel computation.

[0003] The inference process of a large language model in the MoE architecture can be divided into several steps: Attention, Dispatch, Gate-upGEMM, Activation, Combine, and Down-GEMM. Dispatch is a communication operation that distributes each token to one or more experts it selects. When expert parallelism is enabled, these experts may reside on different GPUs. Gate-upGEMM is a computational operation that performs matrix multiplication between the token received by each local expert in the Dispatch and the corresponding expert weight matrix. Because there are multiple expert weight matrices, this step actually consists of a series of independent matrix multiplications, called Group GEMM; each expert and its received tokens constitute a group.

[0004] Because Gate-up GEMM depends on Dispatch, in current inference system implementations, each GPU typically only begins Gate-up GEMM computation after receiving all tokens sent to it. However, this means that during the network transmission and data copying process of Dispatch, the GPU's computing resources are idle and cannot perform computations, resulting in significant resource waste. Shortening Dispatch time could improve GPU utilization. However, Dispatch time is constrained by the physical condition of network bandwidth and cannot be shortened indefinitely. Furthermore, current advanced communication libraries, such as DeepEP, already have high network bandwidth utilization during Dispatch, thus limiting the optimization space for Dispatch at the pure communication level.

[0005] The aforementioned situation leads to a natural optimization direction: computation and communication overlap. The basic idea behind computation and communication overlap is to execute computation and communication functions simultaneously, thereby reducing the exposure time of either computation or communication. However, computation and communication overlap requires support from both communication libraries and matrix computation libraries, and faces difficulties in network transmission scheduling and computational resource allocation. Due to these difficulties, current communication libraries, such as DeepEP and DeepGEMM, do not support expert-level fine-grained overlap. Summary of the Invention

[0006] The purpose of this invention is to provide a communication and computational collaborative scheduling method and system for expert parallelism of large language models, so as to solve the problems in the prior art that it is difficult to trigger computation in an orderly manner at the expert granularity during communication, and that there is waste of computing power and resource competition when computation and communication are parallel.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a communication and computational collaborative scheduling method for parallel operation of large language model experts, comprising the following steps: S1: The application sends a Dispatch data sending task to this GPU, sending the data of each token on this GPU to the GPU where the expert selected by that token is located. S2: This GPU responds to the Dispatch data sending task and performs data sending operations: sending the data of each token to the GPU where the expert selected by that token is located; S3: The application prepares the application-layer token buffer and issues a Dispatch data receiving task and a grouped GEMM task to the local GPU; wherein, the Dispatch data receiving task writes token data received from other GPUs into the application-layer token buffer; the grouped GEMM task performs grouped GEMM calculation based on the token data in the application-layer token buffer; the Dispatch data receiving task and the grouped GEMM task are configured to start simultaneously and run in parallel after the Dispatch data sending task is completed. S4: This GPU responds to the Dispatch data reception task and performs data reception operations: monitors the reception status of tokens sent from the source GPUs to each expert of this GPU; when all tokens sent from a source GPU to a certain expert of this GPU have arrived, these token data are moved to the application layer token buffer; when all tokens of a certain expert of this GPU have been received from all source GPUs, a completion signal is sent to the group GEMM task. S5: This GPU responds to grouped GEMM tasks and performs grouped GEMM calculation operations: it monitors completion signals, and when a completion signal for a certain expert is received, it starts the GEMM calculation for that expert; it coordinates the scheduling of Dispatch data reception tasks and GEMM tasks at the expert granularity so that the two can be executed in parallel.

[0008] To optimize the above technical solution, the specific measures also include: In step S2, the Dispatch data transmission task employs different network transmission methods based on the physical location relationship between the target GPU and the local GPU: If the target GPU and this GPU are located on the same physical host and connected via NVLink, then the NVLink copy is initiated using the GPU's LD / ST instruction or TMA instruction; Furthermore, if the target GPU and this GPU are located on different physical hosts, network transmission is initiated using RDMA. This GPU only issues RDMA work requests, and the actual data transmission is handled by the network card hardware.

[0009] When initiating network transmission using RDMA, multiple queue pairs are established on each GPU for each remote GPU that needs RDMA communication, and all RDMA work requests sent to the same expert on the same remote GPU are assigned to the same queue pair. Furthermore, by taking the modulo of the expert number with the number of queue pairs, RDMA work requests from different experts destined for the same remote GPU are allocated to corresponding queue pairs, so that the RDMA work requests of experts with smaller numbers are prioritized and sent first in their respective queue pairs.

[0010] In steps S1 and S3, when the application sends tasks to the GPU, it configures the computing resource allocation ratio, specifically as follows: For Dispatch data transmission tasks, allocate the maximum proportion of computing resources available to this GPU within the current scheduling cycle; Furthermore, for the parallel execution of the Dispatch data receiving task and the grouped GEMM task, the minimum proportion of computing resources determined by the time-consuming ladder characteristics without impairing performance is allocated to the grouped GEMM task, and the remaining computing resources are allocated to the Dispatch data receiving task.

[0011] The Dispatch data receiving task further allocates computing resources internally, specifically as follows: Set the number of experts for parallel transport; The allocated computing resources are grouped according to the number of source GPUs. Each group of computing resources is responsible for processing all tokens from one source GPU. Each group of computing resources is further divided into multiple local expert groups. Each local expert group is responsible for processing tokens sent from one fixed source GPU to a fixed number of local experts.

[0012] In step S3, the task execution order is ensured through CUDA streams and events, specifically as follows: The Dispatch data sending task and the Dispatch data receiving task are sequentially sent to the first CUDA stream; Furthermore, a CUDA event is inserted into the second CUDA stream, and the event is set to wait for the Dispatch data transmission task on the first CUDA stream to complete; The grouped GEMM task is sent to the second CUDA stream. After the Dispatch data is sent, the Dispatch data reception and the grouped GEMM are executed in parallel. Furthermore, the task distribution and execution process defined by CUDA streams and events is recorded as a CUDA graph, and the entire scheduling process is executed by replaying the CUDA graph.

[0013] In step S4, when all tokens of a certain expert in this GPU have been received from all source GPUs, a completion signal is sent to the grouped GEMM task, specifically as follows: A signal array is shared between the Dispatch data reception task and the packet GEMM task; After the Dispatch data receiving task completes the writing operation of all tokens for each expert, it writes a completion flag in the signal array at the position corresponding to that expert. Furthermore, the grouped GEMM task reads the signal array through a polling method. When it finds that the completion flag corresponding to a certain expert is set, it starts the GEMM calculation for that expert.

[0014] As another important technical solution, this invention also provides a communication and computational collaborative scheduling system for parallel operation of large language model experts, comprising: The task dispatch module is used by the application to send Dispatch data dispatch tasks to this GPU, sending the data of each token on this GPU to the GPU where the expert selected by the token is located. The task execution module is used by this GPU to respond to the Dispatch data sending task and perform data sending operations: sending each token data to the GPU where the expert selected by that token is located. The receiving and computation task distribution module is used by the application to prepare an application-layer token buffer and distribute Dispatch data receiving tasks and grouped GEMM tasks to the local GPU. The Dispatch data receiving task writes token data received from other GPUs into the application-layer token buffer. The grouped GEMM task performs grouped GEMM computation based on the token data in the application-layer token buffer. The Dispatch data receiving task and the grouped GEMM task are configured to start simultaneously and run in parallel after the Dispatch data sending task has been completed. The data receiving and signal triggering module is used to respond to the Dispatch data receiving task and perform data receiving operations: monitor the token receiving status from the source GPU to each expert of this GPU; when all tokens from a source GPU to a certain expert of this GPU have arrived, move these token data to the application layer token buffer; when all tokens of a certain expert of this GPU have been received from all source GPUs, send a completion signal to the group GEMM task. The grouped GEMM calculation and collaborative scheduling module is used to respond to grouped GEMM tasks and perform grouped GEMM calculation operations: it monitors completion signals, and when a completion signal for a certain expert is received, it starts the GEMM calculation for that expert; it collaboratively schedules Dispatch data receiving tasks and GEMM tasks at the expert granularity so that the two can be executed in parallel.

[0015] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a communication and computational collaborative scheduling method for parallel large language model experts as described above.

[0016] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute a communication and computational collaborative scheduling method for large language model experts as described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention establishes an expert-level completion signal mechanism between the receiving task and the grouped GEMM task, allowing each expert's GEMM calculation to begin immediately upon the arrival of all expert data, without waiting for all expert data to be received. Compared to the traditional serial mode of receiving all data before calculation, this invention significantly reduces the idle waiting time of computing resources and achieves deep parallelism between communication and computation at the expert level.

[0018] This invention achieves a fine-grained allocation of computing resources among sending, receiving, and group GEMM tasks. In particular, during the parallel receiving and computing phase, it allocates the minimum proportion of computing resources to the group GEMM based on its time-consuming step characteristics without compromising performance, and uses the remaining resources for the receiving task. This effectively avoids resource contention and improves the overall GPU utilization.

[0019] This invention addresses cross-host scenarios by employing a strategy of establishing multiple queue pairs for each remote GPU and allocating work requests by taking the modulo of the number of queue pairs based on the expert number. This allows experts with smaller numbers to send requests first, thereby better matching the subsequent GEMM computation order and reducing the impact of communication latency on computation startup.

[0020] This invention utilizes CUDA streams and events to precisely control the task execution order and records the entire scheduling process as a CUDA graph. It then executes the task by replaying the process, which reduces the CPU and GPU interaction overhead caused by multiple task distributions and improves scheduling stability and execution efficiency.

[0021] This invention maps different network interface card (NIC) physical ports to network transmission paths using multiple queue pairs, enabling NICs to process multiple work requests in parallel, distributing network load, avoiding the serial bottleneck caused by a single queue pair, and improving the bandwidth utilization efficiency of RDMA communication. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the RDMA work request distribution strategy for Dispatch sending tasks in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram showing the execution timeline of GPU tasks and the allocation of computing resources among tasks in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram illustrating the GPU computing resource allocation method within the Dispatch receiving task in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating the interaction between the Dispatch receiving task and the GEMM group in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram illustrating the execution order of multiple GPU tasks in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram illustrating the overlap efficiency under different batch sizes in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0029] In some embodiments, the present invention provides a communication and computational collaborative scheduling method for parallel processing of large language model experts, comprising the following steps: S1: The application sends a Dispatch data sending task to this GPU, sending the data of each token on this GPU to the GPU where the expert selected by that token is located. S2: This GPU responds to the Dispatch data sending task and performs data sending operations: sending the data of each token to the GPU where the expert selected by that token is located; The Dispatch data transmission task employs different network transmission methods based on the physical location relationship between the target GPU (the GPU to which the selected expert belongs) and the local GPU: In some implementations, if the target GPU and this GPU are located on the same physical host and connected via NVLink, then the NVLink copy is initiated using the GPU's LD / ST (load / store) instruction or TMA instruction; If the target GPU and this GPU are located on different physical hosts, network transmission is initiated using RDMA. This GPU only issues an RDMA work request and needs to wait for the RDMA transmission to complete. The actual data transmission is handled by the network card hardware.

[0030] When the host supports IBGDA technology, the GPU directly initiates RDMA work requests to the network card without CPU processing.

[0031] When network transmission is initiated using RDMA, multiple QPs (queue pairs) are established on each GPU for each remote GPU that needs RDMA communication. These QPs are the basic data structures for software-NIC interaction in RDMA. A QP stores the work requests sent by the software to the NIC in the form of a first-in-first-out queue. All RDMA work requests sent to the same expert on the same remote GPU are allocated to the same QP, so that the NIC can process these requests in parallel. Different QPs are mapped to different network interface card physical ports and use different network transmission paths, thereby reducing the uneven load on the network.

[0032] By taking the modulo of the expert number with the number of queue pairs, RDMA work requests from different experts destined for the same remote GPU are assigned to corresponding queue pairs, so that the RDMA work requests of experts with smaller numbers are prioritized and sent first in their respective queue pairs.

[0033] S3: The application prepares the application-layer token buffer and issues a Dispatch data receiving task and a grouped GEMM task to the local GPU; wherein, the Dispatch data receiving task writes token data received from other GPUs into the application-layer token buffer; the grouped GEMM task performs grouped GEMM calculation based on the token data in the application-layer token buffer; the Dispatch data receiving task and the grouped GEMM task are configured to start simultaneously and run in parallel after the Dispatch data sending task is completed. In some implementations, when an application assigns tasks to the GPU, it configures the computing resource allocation ratio, specifically as follows: Since the Dispatch data transmission task is not executed in parallel with other tasks, the Dispatch data transmission task is allocated the maximum proportion of computing resources available to this GPU in the current scheduling cycle. For the Dispatch data receiving task and the grouped GEMM task executed in parallel, allocate the minimum proportion of computing resources determined by the time-consuming ladder characteristics without impairing performance to the grouped GEMM task, and allocate the remaining computing resources to the Dispatch data receiving task.

[0034] In some implementations, the Dispatch data receiving task further allocates computing resources, specifically as follows: Set the number of experts for parallel transport; The allocated computing resources are grouped according to the number of source GPUs. Each group of computing resources is responsible for processing all tokens from one source GPU. Each group of computing resources is further divided into multiple local expert groups. Each local expert group is responsible for processing tokens sent from a fixed source GPU to a fixed number of local experts. Different GPU threads within each local expert group are responsible for parallel transfer of different tokens belonging to that resource group.

[0035] In some implementations, the task execution order is ensured through CUDA streams and events, specifically: The Dispatch data sending task and the Dispatch data receiving task are sequentially sent to the first CUDA stream; Insert a CUDA event into the second CUDA stream and set the event to wait for the Dispatch data transmission task on the first CUDA stream to complete; The grouped GEMM task is sent to the second CUDA stream. After the Dispatch data is sent, the Dispatch data reception and the grouped GEMM are executed in parallel. The task distribution and execution process defined by CUDA streams and events is recorded as a CUDA graph, and the entire scheduling process is executed by replaying the CUDA graph.

[0036] S4: This GPU responds to the Dispatch data reception task and performs data reception operations: monitors the reception status of tokens sent from the source GPUs to each expert of this GPU; when all tokens sent from a source GPU to a certain expert of this GPU have arrived, these token data are moved to the application layer token buffer; when all tokens of a certain expert of this GPU have been received from all source GPUs, a completion signal is sent to the group GEMM task. When all tokens for a specific expert on this GPU have been received from all source GPUs, a completion signal is sent to the grouped GEMM task, specifically: In some implementations, a signal array is shared between the Dispatch data receiving task and the packet GEMM task, and the location of the signal array is the GPU's global memory (GMEM). After the Dispatch data receiving task completes the writing operation of all tokens for each expert, it writes a completion flag in the signal array at the position corresponding to that expert. The grouped GEMM task reads the signal array in a polling manner. When it finds that the completion flag corresponding to a certain expert is set, it starts the GEMM calculation for that expert.

[0037] S5: This GPU responds to grouped GEMM tasks and performs grouped GEMM calculation operations: it monitors completion signals, and when a completion signal for a certain expert is received, it starts the GEMM calculation for that expert; it coordinates the scheduling of Dispatch data reception tasks and GEMM tasks at the expert granularity so that the two can be executed in parallel.

[0038] In some implementations... Figure 1 This invention demonstrates the RDMA work request distribution strategy during the Dispatch phase. Following general RDMA terminology, the work request will also be referred to as a WQE (work queue element). When using RDMA for communication, a WQE is established on each GPU for each remote GPU requiring RDMA communication. q There are QPs, where positive integers are... qThis is a configurable parameter. Different QPs are mapped to different network interface physical ports and use different network transmission paths, thereby reducing network load imbalance. For each destination expert, an RDMA one-sided write WQE is issued for each token sent to it to send the data of that token; an RDMA atomic add WQE is issued to notify the receiver that the transmission is complete and to inform it of the number of tokens it needs to receive.

[0039] Because RDMA atomic operations can only guarantee the write order within the same QP, for a given source GPU, all WQEs destined for a specific expert must be sent to the same QP. This is done according to the expert number. i Divide by q The remainder is used to send the WQE of each expert to the corresponding QP, so that the RDMA work request corresponding to the expert with the smaller number is arranged in a higher position in the QP, thereby realizing priority sending for experts with smaller numbers.

[0040] For a target GPU containing 4 experts, q This refers to the WQE deployment scheme for scenarios 1, 2, and 4. q When the value is 1, each token will be sent precisely in the order of the destination expert number, but the network card can only serially encapsulate and transmit the token data; and the QP can only use one path on the network, resulting in uneven network load and poor network transmission performance. q When the QP is 4, the network card processes 4 WQEs in parallel, and by changing the mapping of QP to the network card physical port and the transport layer source port, multiple network paths are used for transmission, resulting in the best network transmission performance. However, since it is impossible to precisely control which of the 4 experts on each source GPU sends first, it cannot be well matched with the computation order of the packet GEMM. q =2 is a compromise configuration that achieves relatively good overall performance.

[0041] In some implementations, such as Figure 2 As shown, when dispatching tasks on the GPU side, the amount of GPU computing resources to be used for the dispatch needs to be specified. Since the Dispatch data dispatch task is not executed in parallel with other tasks, as much GPU computing resources as possible is allocated to this task, which is represented as 100% in the figure. Since the GEMM group and Dispatch data reception are parallel, and the GEMM computation has a greater demand for computing power than the Dispatch data reception, most of the computing resources are allocated to the GEMM group, and the remaining resources are allocated to the Dispatch data reception.

[0042] Since the time consumed by grouped GEMMs decreases in a stepwise manner as computing resources decrease, reducing computing resources within a small range has almost no impact on the performance of grouped GEMMs. Therefore, when determining the specific allocation ratio, the minimum proportion of computing resources that will not impair performance should be allocated to grouped GEMMs, denoted in the figure as . x %; allocate the remaining computing resources to Dispatch data reception, represented as 1 in the diagram. -x On NVIDIA GPUs, computing resources are allocated among the three types of computing tasks based on SMs (streaming multiprocessors). For example, on an NVIDIA GPU with 78 SMs, if the measured time for matrix computation using 78 SMs and 76 SMs is approximately the same, then 78 SMs are allocated to the Dispatch sending task, and 76 and 2 SMs are allocated to the packet GEMM and Dispatch receiving tasks, respectively.

[0043] In some implementations, such as Figure 3 As shown, the Dispatch data receiving task uses computing resources by setting a number of experts for parallel data transfer. p The computing resources are divided into multiple groups based on the total number of GPUs (rounded up appropriately if the division is not exact, details omitted below). Each group is responsible for processing a token sent by one source GPU; such a group is called a source GPU group. Each source GPU group is further divided into a small number of subgroups, each called a local expert group. Each local expert group is responsible for processing a fixed source GPU. i Tokens are sent to a fixed group of local experts. Specifically, a local expert's ID is recorded as... e Then the source GPU group i Local expert group j Responsible for moving the source GPU i Send to satisfy ( e%p== j This refers to all tokens of local experts. The smallest granularity for allocating computational resources can be a thread or a larger unit. For example, on NVIDIA GPUs, local expert groups consist of one or more complete warps to reduce warp divergence. To match network transmission patterns, p The value shall not exceed the number of QPs for each target GPU. q .

[0044] In some implementations, such as Figure 4As shown, before starting computation and communication, the application prepares a receive completion signal array. This array resides in the GPU's global memory and is used for interaction between Dispatch and Gate-up GEMM. The receive completion signal array consists of E integers, where E is the number of experts on the GPU; all integers are initially set to 0. Each time the Dispatch task completes the transfer of an expert's token, it writes a 1 to the corresponding position in the signal array. The Gate-up GEMM task, following the interaction logic shown in the diagram, continuously polls and reads the signal array in global memory. When it finds that the signal value of an expert has become non-zero, it immediately begins the corresponding GEMM computation. In this way, the Gate-up GEMM does not need to wait for all expert tokens to arrive before starting the computation task.

[0045] In some implementations, such as Figure 5 As shown, two CUDA streams are used to distribute tasks: first, the data sending and receiving tasks are distributed to the first CUDA stream; then, a CUDA event is appended to the second CUDA stream, causing the event to wait for the data sending task on the first CUDA stream; finally, the GEMM packet is distributed to the second CUDA stream. To reduce the overhead of distributing GPU tasks multiple times, these steps are recorded into a CUDA graph. When executing the entire compute-communication overlap process later, the CUDA graph can be directly replayed.

[0046] like Figure 6 As shown, the implementation effect of this invention is demonstrated by measuring the overlap efficiency of the invention under different batch sizes. Here, batch size refers to the number of tokens passed from a single GPU to Dispatch, determined by the application scenario; overlap efficiency (between 0 and 1, the higher the better) characterizes the proportion of communication time successfully masked (thus freeing up resources for matrix calculations) by this invention. The test used two hosts with a total of 16 NVIDIA GPUs, connected via a RoCEv2 network. The figure shows that this invention has optimization effects under various batch sizes, and the larger the batch size, the more significant the optimization effect, achieving a maximum overlap efficiency of 50%. This result indicates that when this invention is applied to a large language model inference system, it can reduce the overall inference process time, thereby improving the throughput of large language model inference.

[0047] In another embodiment of the present invention, a communication and computational collaborative scheduling system for parallel operation of large language model experts is proposed, comprising: The task sending module is used by the application to send Dispatch data sending tasks to this GPU, sending the data of each token on this GPU to the GPU where the expert selected by the token is located. The task execution module is used by this GPU to respond to the Dispatch data sending task and perform data sending operations: sending each token data to the GPU where the expert selected by that token is located. The receiving and computation task distribution module is used by the application to prepare an application-layer token buffer and distribute Dispatch data receiving tasks and grouped GEMM tasks to the local GPU. The Dispatch data receiving task writes token data received from other GPUs into the application-layer token buffer. The grouped GEMM task performs grouped GEMM computation based on the token data in the application-layer token buffer. The Dispatch data receiving task and the grouped GEMM task are configured to start simultaneously and run in parallel after the Dispatch data sending task has been completed. The data receiving and signal triggering module is used to respond to the Dispatch data receiving task and perform data receiving operations: monitor the token receiving status from the source GPU to each expert of this GPU; when all tokens from a source GPU to a certain expert of this GPU have arrived, move these token data to the application layer token buffer; when all tokens of a certain expert of this GPU have been received from all source GPUs, send a completion signal to the group GEMM task. The grouped GEMM calculation and collaborative scheduling module is used to respond to grouped GEMM tasks and perform grouped GEMM calculation operations: it monitors completion signals, and when a completion signal for a certain expert is received, it starts the GEMM calculation for that expert; it collaboratively schedules Dispatch data receiving tasks and GEMM tasks at the expert granularity so that the two can be executed in parallel.

[0048] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a communication and computational collaborative scheduling method for parallel large language model experts as described above.

[0049] In another embodiment of the present invention, a computer-readable storage medium is provided storing a computer program that causes a computer to execute a communication and computational collaborative scheduling method for parallel large language model experts as described above.

[0050] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A communication and computational collaborative scheduling method for parallel expert training in large language models, characterized in that, Includes the following steps: S1: The application sends a Dispatch data sending task to this GPU, sending the data of each token on this GPU to the GPU where the expert selected by that token is located. S2: This GPU responds to the Dispatch data sending task and performs data sending operations: sending the data of each token to the GPU where the expert selected by that token is located; S3: The application prepares the application-layer token buffer and issues a Dispatch data receiving task and a grouped GEMM task to the local GPU; wherein, the Dispatch data receiving task writes token data received from other GPUs into the application-layer token buffer; the grouped GEMM task performs grouped GEMM calculation based on the token data in the application-layer token buffer; the Dispatch data receiving task and the grouped GEMM task are configured to start simultaneously and run in parallel after the Dispatch data sending task is completed. S4: This GPU responds to the Dispatch data reception task and performs data reception operations: monitors the reception status of tokens sent from the source GPUs to each expert of this GPU; when all tokens sent from a source GPU to a certain expert of this GPU have arrived, these token data are moved to the application layer token buffer; when all tokens of a certain expert of this GPU have been received from all source GPUs, a completion signal is sent to the group GEMM task. S5: This GPU responds to grouped GEMM tasks and performs grouped GEMM calculation operations: it monitors completion signals, and when a completion signal for a certain expert is received, it starts the GEMM calculation for that expert; it coordinates the scheduling of Dispatch data reception tasks and GEMM tasks at the expert granularity so that the two can be executed in parallel.

2. The communication and computational collaborative scheduling method for expert parallel processing of large language models according to claim 1, characterized in that: In step S2, the Dispatch data transmission task employs different network transmission methods based on the physical location relationship between the target GPU and the local GPU: If the target GPU and this GPU are located on the same physical host and connected via NVLink, then the NVLink copy is initiated using the GPU's LD / ST instruction or TMA instruction; If the target GPU and this GPU are located on different physical hosts, network transmission is initiated using RDMA. This GPU only issues RDMA work requests, and the actual data transmission is handled by the network card hardware.

3. The communication and computational collaborative scheduling method for expert parallel processing of large language models according to claim 2, characterized in that: When initiating network transmission using RDMA, multiple queue pairs are established on each GPU for each remote GPU that needs RDMA communication, and all RDMA work requests sent to the same expert on the same remote GPU are assigned to the same queue pair. By taking the modulo of the expert number with the number of queue pairs, RDMA work requests from different experts destined for the same remote GPU are assigned to corresponding queue pairs, so that the RDMA work requests of experts with smaller numbers are prioritized and sent first in their respective queue pairs.

4. The communication and computational collaborative scheduling method for expert parallel processing of large language models according to claim 1, characterized in that: In steps S1 and S3, when the application sends tasks to the GPU, it configures the computing resource allocation ratio, specifically as follows: For Dispatch data transmission tasks, allocate the maximum proportion of computing resources available to this GPU within the current scheduling cycle; For the Dispatch data receiving task and the grouped GEMM task that are executed in parallel, allocate the minimum proportion of computing resources to the grouped GEMM task and allocate the remaining computing resources to the Dispatch data receiving task.

5. The communication and computational collaborative scheduling method for expert parallel processing of large language models according to claim 1, characterized in that: The Dispatch data receiving task further allocates computing resources internally, specifically as follows: Set the number of experts for parallel transport; The allocated computing resources are grouped according to the number of source GPUs. Each group of computing resources is responsible for processing all tokens from one source GPU. Each group of computing resources is further divided into multiple local expert groups. Each local expert group is responsible for processing tokens sent from one fixed source GPU to a fixed number of local experts.

6. The communication and computational collaborative scheduling method for expert parallel processing of large language models according to claim 1, characterized in that: In step S3, the task execution order is ensured through CUDA streams and events, specifically as follows: The Dispatch data sending task and the Dispatch data receiving task are sequentially sent to the first CUDA stream; Insert a CUDA event into the second CUDA stream and set the event to wait for the Dispatch data transmission task on the first CUDA stream to complete; The grouped GEMM task is sent to the second CUDA stream. After the Dispatch data is sent, the Dispatch data reception and the grouped GEMM are executed in parallel. The task distribution and execution process defined by CUDA streams and events is recorded as a CUDA graph, and the entire scheduling process is executed by replaying the CUDA graph.

7. The communication and computational collaborative scheduling method for expert parallel processing of large language models according to claim 1, characterized in that: In step S4, when all tokens of a certain expert in this GPU have been received from all source GPUs, a completion signal is sent to the grouped GEMM task, specifically as follows: A signal array is shared between the Dispatch data reception task and the packet GEMM task; After the Dispatch data receiving task completes the writing operation of all tokens for each expert, it writes a completion flag in the signal array at the position corresponding to that expert. The grouped GEMM task reads the signal array through a polling method. When it finds that the completion flag corresponding to a certain expert is set, it starts the GEMM calculation for that expert.

8. A communication and computational collaborative scheduling system for parallel operation of large language model experts, characterized in that, include: The task dispatch module is used by the application to send Dispatch data dispatch tasks to this GPU, sending the data of each token on this GPU to the GPU where the expert selected by the token is located. The task execution module is used by this GPU to respond to the Dispatch data sending task and perform data sending operations: sending each token data to the GPU where the expert selected by that token is located. The receiving and computation task distribution module is used by the application to prepare an application-layer token buffer and distribute Dispatch data receiving tasks and grouped GEMM tasks to the local GPU. The Dispatch data receiving task writes token data received from other GPUs into the application-layer token buffer. The grouped GEMM task performs grouped GEMM computation based on the token data in the application-layer token buffer. The Dispatch data receiving task and the grouped GEMM task are configured to start simultaneously and run in parallel after the Dispatch data sending task has been completed. The data receiving and signal triggering module is used to respond to the Dispatch data receiving task and perform data receiving operations: monitor the token receiving status from the source GPU to each expert of this GPU; when all tokens from a source GPU to a certain expert of this GPU have arrived, move these token data to the application layer token buffer; when all tokens of a certain expert of this GPU have been received from all source GPUs, send a completion signal to the group GEMM task. The grouped GEMM calculation and collaborative scheduling module is used to respond to grouped GEMM tasks and perform grouped GEMM calculation operations: it monitors completion signals, and when a completion signal for a certain expert is received, it starts the GEMM calculation for that expert; it collaboratively schedules Dispatch data receiving tasks and GEMM tasks at the expert granularity so that the two can be executed in parallel.

9. An electronic device, characterized in that, include: The present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a communication and computational collaborative scheduling method for parallel operation of large language model experts as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute a communication and computational collaborative scheduling method for parallel large language model experts as described in any one of claims 1 to 7.

Images