Systems and methods for reducing latency of multi-gpu card reduction sum in distributed text inference

Abstract

The present disclosure relates to a system and method for reducing latency of multi-GPU card reduction sum in distributed text reasoning. The system comprises: a data packing component, packing data required by a local GPU card to perform multi-GPU card reduction according to a predetermined size, wherein a first part of each data packet is to-be-transmitted data segmented in sequence from shard data, and a second part is flag data; a data transmission component, transmitting the packed data packet from the local GPU card to a buffer of a remote other GPU card; and a data reception confirmation component, in the process of performing polling and waiting, when the reception of a data packet of the predetermined size is completed, completing polling for the corresponding flag data, and based on the visibility of data transmission, making the stage reception completion result visible to all GPU cards and obtaining confirmation, so as to perform multi-card reduction sum.

Description

Technical Field

[0001] This disclosure relates to the field of computer information processing, and more specifically, to a system and method for reducing the latency of multi-GPU reduction summation in distributed text inference. Background Technology

[0002] With the explosive popularity of ChatGPT, large language models have quickly attracted the attention of the contemporary internet industry, and the demand for their application across various sectors has grown accordingly. The need for text generation has also surged. However, text generation itself consumes a significant amount of computing power, and how to deploy and accelerate the inference of large language models in today's widely used device environments is a hot research topic.

[0003] Model parallelism, a common parallel method used in training large models, can naturally be applied to text generation and inference in large oracle models. AllReduce (global reduction or full reduction) multi-GPU reduction and summation is its primary communication method. Currently, mainstream deep learning frameworks in the industry use NVIDIA's NCCL communication library, calling its AllReduce implementation. However, in scenarios with low communication volume and latency sensitivity, NCCL lacks targeted optimization, resulting in a high proportion of communication time.

[0004] TensorRTT (Tensor Runtime) and vLLM (Virtual Large Language Model) are two inference frameworks that have implemented Custom AllReduce for scenarios with low communication volume. When GPUs support direct P2P access, the CUDA IPC inter-process communication mechanism allows each GPU to obtain data pointers from other GPUs, enabling multi-GPU reduction summation. During this process, GPU Barriers (GPU Barriers are synchronization mechanisms used in parallel computing to coordinate the execution order of multiple threads / processing units, ensuring data consistency and correct execution logic by forcibly waiting for all related operations to complete before continuing execution) are also needed to achieve synchronization between GPUs.

[0005] In multi-GPU or distributed training scenarios, global barriers are used to synchronize the computation progress of different devices. For example, PyTorch's `torch.distributed.barrier()` ensures that all processes complete data loading before training begins. In scenarios with low communication volume, the actual reduction and summation operation is computationally inefficient and takes little time. However, GPU synchronization incurs overhead, and the time spent on GPU synchronization may be comparable to the actual reduction and summation operation. Therefore, the key to achieving efficient, low-latency multi-GPU reduction and summation lies in efficient GPU barriers. In each reduction and summation operation, a GPU barrier needs to be inserted before the start and end. The first GPU barrier ensures that all GPUs have executed the AllReduce Kernel, while the second GPU barrier ensures that all GPUs have completed reading data from other GPUs; otherwise, subsequent operations may corrupt the data on the GPUs, leading to incorrect multi-GPU reduction and summation results.

[0006] In the vLLM implementation, the GPU Barrier is implemented by writing flags to remote GPUs locally and then reading the flags locally via a busy wait until all remote GPUs have completed writing their flags, indicating that all GPUs have completed synchronization. Although the overhead of writing flags to remote GPUs is very low (~1µs) with P2P support, the overhead of the entire GPU synchronization increases as the number of thread blocks launched by the GPU increases. Therefore, the vLLM implementation has a limit on the number of thread blocks launched by the GPU. One drawback of limiting the number of thread blocks is that when the reduction summation is large, the load on each thread block (within the same thread block, the Barrier ensures that all threads pause at a specified position and continue subsequent operations until all threads reach that point. For example, CUDA's syncthreads() instruction is used to synchronize shared memory reads and writes) is high, making it impossible to fully utilize the parallel characteristics of the GPU. This results in a high-latency state for the entire reduction summation process, which is not efficient enough.

[0007] Therefore, it is expected that in existing distributed text inference, at least the latency of multi-GPU reduction summation can be reduced, thereby improving the efficiency of multi-GPU reduction summation.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] In view of this, this disclosure provides a distributed text reasoning system for large language models that can at least reduce the current high latency of multi-card reduction summation.

[0010] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0011] According to one aspect of this disclosure, a system for reducing the latency of multi-GPU card reduction summation in distributed text inference is proposed, comprising: a data packaging component, which packages the data that needs to be reduced locally on the GPU card according to a predetermined size, wherein the first part of each data packet is the data to be transmitted sequentially segmented from the fragmented data, and the second part is flag bit data; a data transmission component, which transmits the packaged data packets from the local GPU card to buffers of other remote GPU cards; and a data reception confirmation component, which, during the polling waiting process, when the reception of a data packet of the predetermined size is completed, polls for the corresponding flag bit data, and, based on the verification of the flag bit data, makes the phased reception completion result visible to all GPU cards and confirmed through atomic visibility or sequential visibility, so as to perform multi-card reduction summation.

[0012] According to the system for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each data packet has a first part consisting of 32 bits of data to be transmitted sequentially segmented from the fragmented data, a second part consisting of 32 bits of flag data, and a buffer for the data transmission component to transmit dual data packets from the local GPU card to other remote GPU cards each time. Furthermore, during the polling waiting process, when the reception of a 64-bit data packet is completed, the data reception confirmation component of each GPU card polls the corresponding flag data. Based on the 64-bit atomic visibility in the CUDA memory model, the phased reception completion result is made visible to all GPU cards and confirmed, so as to perform multi-card reduction summation.

[0013] According to the system for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each data packet has a first part consisting of 120 bytes of data to be transmitted, sequentially segmented from the fragmented data, a second part consisting of 8 bytes of flag data, and a buffer for the data transmission component to transmit the data packet from the local GPU card to other remote GPU cards each time based on the NVLink data transmission channel. Furthermore, each GPU card's data reception confirmation component performs a polling waiting process so that upon completion of receiving the 128-byte data packet, the corresponding flag data is polled for completion. Based on the sequential visibility in the CUDA memory model, the phased reception completion result is made visible to all GPU cards and confirmed for multi-GPU reduction summation.

[0014] According to the system for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each GPU card's data transmission component transmits consecutively packaged data packets of the local full data from the local GPU card to the buffers of other remote GPU cards in one go, so that each GPU card can perform multi-GPU reduction summation locally.

[0015] According to the system for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each GPU card's data packaging component divides the local full data into sub-shards according to the number of GPU cards before packaging, and then packages each sub-shard. The data transmission component only transmits consecutive data packets of the corresponding sub-shard from the local GPU card to the corresponding buffer of the remote GPU card. After each GPU card receives all its corresponding sub-shard data and performs multi-card reduction summation on all the received sub-shard data locally to obtain the reduction summation result sub-shard data, each GPU card sends its local reduction summation result sub-shard data to other GPU cards.

[0016] According to another aspect of this disclosure, a method for reducing the latency of multi-GPU card reduction summation in distributed text inference is provided, comprising: packaging data that needs to be reduced locally on the GPU card according to a predetermined size using a data packaging component, wherein the first part of each data packet is data to be transmitted sequentially segmented from the fragmented data, and the second part is flag bit data; transmitting the packaged data packets from the local GPU card to buffers of other remote GPU cards using a data transmission component; and performing a polling waiting process using a data reception acknowledgment component, so that when the reception of data packets of the predetermined size is completed, the polling for the corresponding flag bit data is completed, and based on the verification of the flag bit data, the phased reception completion result is made visible to all GPU cards and acknowledged through atomic visibility or sequential visibility, so as to perform multi-GPU reduction summation.

[0017] According to the method for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each data packet has a first part consisting of 32 bits of data to be transmitted, sequentially segmented from the fragmented data, a second part consisting of 32 bits of flag data, and a buffer for the data transmission component to transmit dual data packets from the local GPU card to other remote GPU cards each time. Furthermore, each GPU card's data reception confirmation component performs a polling waiting process so that upon completion of the reception of a 64-bit data packet, the corresponding flag data is polled for completion. Based on the 64-bit atomic visibility in the CUDA memory model, the phased reception completion result is visible to all GPU cards and confirmed, enabling multi-card reduction summation.

[0018] According to the method for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each data packet has a first part consisting of 120 bytes of data to be transmitted, sequentially segmented from the fragmented data, a second part consisting of 8 bytes of flag data, and a buffer for the data transmission component to transmit the data packet from the local GPU card to other remote GPU cards each time based on the NVLink data transmission channel. Furthermore, each GPU card's data reception confirmation component performs a polling waiting process so that upon completion of receiving the 128-byte data packet, the corresponding flag data is polled for completion. Based on the sequential visibility in the CUDA memory model, the phased reception completion result is seen and confirmed by all GPU cards for multi-GPU reduction summation.

[0019] According to the method for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each GPU card's data transmission component transmits continuously packaged data packets of the local full data from the local GPU card to the buffers of other remote GPU cards at one time, so that each GPU card can perform multi-card reduction summation locally.

[0020] According to the method for reducing the latency of multi-GPU card reduction summation in distributed text inference disclosed herein, each GPU card's data packaging component divides the local full data into sub-shard data according to the number of GPU cards before packaging, and then packages each sub-shard data. The data transmission component only transmits the continuous data packets of the corresponding sub-shard data from the local GPU card to the corresponding buffer of the remote GPU card. After each GPU card receives all its corresponding sub-shard data and performs multi-card reduction summation on all the received sub-shard data locally to obtain the reduction summation result sub-shard data, each GPU card sends its local reduction summation result sub-shard data to other GPU cards.

[0021] According to the system and method disclosed herein for reducing the latency of multi-GPU reduction summation in distributed text inference, considering that CUDA memory models are 64-bit atomically visible, a 64-bit data packet is formed by combining 32-bit data and a 32-bit Flag bit into a single 62-bit data packet. When the 32-bit Flag bit can be verified, it also means that the previous 32 bits of data have been transmitted. Each thread in CUDA supports a maximum of 128 bits of read and write, so two data packets can be written to the remote buffer at once. The remote GPU reads the buffer via busy wait and checks if the Flag bit value is correct; if it is, the data has been successfully transmitted. This disclosure utilizes the CUDA memory operation feature of sequential visibility of 128 bytes and the CUDA feature of each thread supporting a maximum of 16 bytes of read and write. Therefore, the data to be transmitted is divided into 128-byte groups, and each data block is divided into 8 threads of 16 bytes each, so that each group exactly meets the CUDA 128-byte transmission limit. For the last thread in each thread group, it stores 8 bytes of data and an 8-byte Flag bit. This 8-byte Flag bit is exactly 64 bits, which meets the CUDA memory model feature of 64-bit atomic visibility. Thus, packet verification is only performed by this thread, while the other threads in the thread group store 16 bytes of valid data. Therefore, when the Flag bit in the last thread of each thread group meets the expected value, it means that the previous 120 bytes of data have been successfully transmitted. This greatly utilizes the low-latency, high-bandwidth optimization protocol designed for inter-GPU collective communication (such as All Reduce and Broadcast) in NVIDIA NCCL (Cluster Communication Library), achieving a protocol bandwidth utilization rate of over 90%. This also improves the rate of multi-GPU reduction summation and reduces its latency from a bandwidth utilization perspective. Furthermore, reducing the amount of data transmitted during the reduction summation process through two-stage reduction summation also reduces the latency of the reduction summation.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0023] The above and other objects, features, and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This is a block diagram illustrating a first embodiment of a system for reducing the latency of multi-GPU card reduction summation in distributed text inference according to an exemplary embodiment.

[0025] Figure 2 The diagram illustrates the multi-GPU reduction summation process in a system designed to reduce latency in distributed text inference.

[0026] Figure 3 The figure shown is a comparison of system test results for reducing the latency of multi-GPU card reduction summation in distributed text inference according to this disclosure.

[0027] Figure 4 The diagram shown illustrates a second implementation of a system for reducing latency in multi-GPU reduction summation in distributed text inference, illustrating the multi-GPU reduction summation process.

[0028] Figure 5 The figure shown is a comparison of system test results for reducing the latency of multi-GPU card reduction summation in distributed text inference based on the LL128 protocol according to this disclosure.

[0029] Figure 6 The diagram illustrates a one-stage multi-GPU reduction summation process in a system for reducing latency in multi-GPU reduction summation in distributed text inference according to the present disclosure.

[0030] Figure 7 The diagram shown is a schematic of a two-stage multi-GPU reduction summation process in a system for reducing the latency of multi-GPU reduction summation in distributed text inference according to the present disclosure.

[0031] Figure 8 The figure shown is a comparison of test results for first-order and second-order systems in distributed text inference according to this disclosure, which reduces the latency of multi-GPU card reduction summation. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first computing device discussed below may be referred to as the second computing device without departing from the teachings of this disclosure. As used herein, the term "and / or" includes all combinations of any and more of the associated listed items.

[0037] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing this disclosure, and therefore cannot be used to limit the scope of protection of this disclosure.

[0038] Figure 1 This is a block diagram illustrating a first embodiment of a system 100 for reducing the latency of multi-GPU reduction summation in distributed text inference, according to an exemplary embodiment. Figure 1As shown, a system 100 for reducing the latency of multi-GPU card reduction summation in distributed text inference includes: a data packaging component 110, which packages the data that needs to be reduced locally on the GPU card according to a predetermined size, wherein the first part of each data packet is the data to be transmitted in order from the fragmented data, and the second part is the flag bit data; a data transmission component 120, which transmits the packaged data packets from the local GPU card to the buffers of other remote GPU cards; and a data reception confirmation component 130, which, during the polling waiting process, when the reception of a data packet of the predetermined size is completed, polls for the corresponding flag bit data, and, based on the verification of the flag bit data, makes the phased reception completion result visible to all GPU cards and confirmed through atomic visibility or sequential visibility, so as to perform multi-card reduction summation. Although Figure 1 The data packaging components 110 are labeled as 110-1, 110-2...110-N, but when not distinguishing between them, they are all referred to as 110. Similarly, the data transmission component 120 and the data reception confirmation component 130 are also referred to as 120 and 130 respectively when they are in different GPU cards.

[0039] like Figure 1 As shown, the system 100 for reducing the latency of multi-GPU card reduction summation in distributed text inference of this disclosure utilizes the 64-bit atomic visibility in the CUDA memory model to combine 32-bit data and a 32-bit Flag bit into a 64-bit data packet each time. When a GPU card receives this data packet, the data reception confirmation component 130 checks the 32-bit Flag bit by reading the data packet in the buffer through a busy wait. When the 32-bit Flag bit is verified, it also means that the previous 32 bits of data have been transmitted. Figure 2The diagram illustrates the multi-GPU reduction summation process in a system designed to reduce latency in distributed text inference. It should be noted that each thread in CUDA supports a maximum of 128 bits of read / write; therefore, the data transmission component 120 of this disclosure can write two data packets to the buffer of the remote GPU card at once. The data reception confirmation component 130 of the remote GPU card reads the data buffer through a busy wait. For example, the data reception confirmation component 130-2 of GPU2 polls the newly received data in data buffer 2 to check if it contains a 32-bit Flag bit and verifies if it matches the Flag bit value. If it does, it indicates that the valid data corresponding to that Flag bit has been successfully transmitted. The 64-bit atomic visibility ensures not only the atomicity of the operation for 64-bit data types but also guarantees the immediate visibility of the operation result to other threads through a hardware mechanism. Atomic operations for 64-bit data types refer to the uninterrupted completion of a read-write-modify-write-back process of memory by a single instruction in a multi-threaded environment, avoiding data races. "Visibility" refers to the visibility of the operation result to other threads, an extended characteristic of atomic operations. Once a thread completes an atomic operation, other threads accessing the same memory location must immediately observe the latest value without explicit synchronization instructions. Figure 2 The diagram illustrates the first implementation of a system for reducing the latency of multi-GPU reduction summation in distributed text inference, specifically focusing on the multi-GPU reduction summation process. Figure 2 As shown, each GPU, such as GPU0, continuously packages the local fragment data to be reduced and summed into a first part of valid data D and a second part of flag data F, thus forming consecutive data packets D0-1+F0-1, D0-2+F0-2, etc. These are then directly written to the caches of other GPUs, such as cache 0, cache 1, cache 2, and cache 3, via memory access. The data receiving confirmation components 130 on the remote GPUs 1, 2, and 3 poll and wait, making the polling verification results atomically visible to other GPUs, thus achieving synchronization. This eliminates the defect in the prior art where barriers ensure that all threads pause execution at a specified position, while also guaranteeing the integrity of all data.

[0040] As described above, the present invention primarily aims to accelerate the reduction summation process by preventing delays caused by mutual waiting in existing multi-GPU reduction summation. Instead, it uses this technology to address the latency problem of multi-GPU reduction summation. The 64-bit atomic visibility itself is prior art and will not be described in detail here. This disclosure, by utilizing 64-bit atomic visibility to combine 32-bit data and a 32-bit Flag bit into a 64-bit data packet each time, significantly reduces the overall GPU synchronization overhead, thus essentially eliminating the limitation on the number of thread blocks that can be started by the GPU.

[0041] Figure 3 The image shown is a comparison of system test results regarding reducing the latency of multi-GPU card reduction summation in distributed text inference according to this disclosure. Figure 3 As shown, the distributed data processing system uses eight H800 GPU cards and employs different reduction and summation systems to test the reduction and summation time for different amounts of transmitted data requiring reduction and summation. Figure 3 As shown, under the same amount of transmitted data, the existing NCCL reduction summation takes the longest time, which means the longest delay. The vLLM custom reduction summation method takes the second longest time, while the cross-node custom reduction summation method adopted in this application takes the shortest time, which means the reduction summation delay is the smallest.

[0042] Although the above-mentioned packetization method can reduce latency in multi-GPU reduction summation under the LL protocol, in the GPU synchronization mechanism based on the LL protocol, half of the transmitted data packets are used to store the Flag bit, and the other half stores the valid data. This means that half of the bandwidth is wasted during transmission. As the communication volume increases, the multi-GPU reduction summation operation gradually changes from latency-bound to memory-bound. The effective bandwidth utilization of the LL protocol is not high; that is, the bandwidth utilization efficiency is only half. This objectively means that, in this method, if the existing bandwidth is fully utilized, the latency can be further reduced. The multi-GPU reduction summation synchronization mechanism based on the LL128 protocol for large data volumes is even less efficient. The LL128 protocol is a low-latency, high-bandwidth optimized protocol designed in NVIDIA NCCL (Cluster Communication Library) for collective communication between GPUs (such as AllReduce and Broadcast). Of course, in the case of small data volume reduction summation, under the LL protocol mechanism, Figure 2 The method shown can significantly reduce latency.

[0043] To fully utilize the high-bandwidth optimized protocol based on LL128, a proposed... Figure 4The diagram shown illustrates a second implementation of a system for reducing latency in multi-GPU reduction summation in distributed text inference, representing a second embodiment of the multi-GPU reduction summation process. For this purpose, a... Figure 4 This illustrates methods to fully utilize bandwidth. For example... Figure 4 As shown, when the GPU cluster used in the deployed distributed data processing system is configured with NVLink, memory operations are visible in 128-byte sequential blocks. NVLink is a high-speed interconnect technology developed by NVIDIA, specifically designed to improve communication efficiency between GPUs and CPUs, and between GPUs and other devices (such as memory and other GPUs), especially serving high-performance computing (HPC) and artificial intelligence (AI) scenarios. This disclosure merely utilizes this technology, rather than improving it, and therefore will not elaborate further. This disclosure leverages the 128-byte sequential visibility of memory operations when NVLink is configured within the GPU cluster to split the 128-byte data packet into 120-byte data and an 8-byte flag. Since each CUDA thread supports a maximum of 16 bytes of read / write, this disclosure packages the fragmented data into 128-byte blocks and divides these 128-byte blocks into 8 thread groups based on the thread's maximum support of 16 bytes, ensuring that each group perfectly meets the 128-byte transmission limit. For the last thread in each thread group, this disclosure allows it to store 8 bytes of data and an 8-byte Flag bit. Packet verification is performed solely by the last thread, while the other threads in the thread group store 16 bytes of valid data. Therefore, as Figure 4As shown, the data packaging component 110 packages the fragmented data to be reduced and summed into 128 bytes as described above, and groups them into 8 groups within the package according to the maximum supported thread size of 16 bytes. Thus, the first part of each data packet is 120 bytes of data to be transmitted, sequentially segmented from the fragmented data. This 120 bytes of valid data is divided into 7 half-threads, including data D0-0-D0-14. The valid data D14 of the last half-thread is combined with the 8 bytes of flag data F0-0 in the second part to form a thread 8. The data transmission component 120 transmits the data packets from the local GPU0 card to the buffers 1-3 of other remote GPU cards according to the thread group order based on the NVLink data transmission channel. The data reception confirmation component 130 of each GPU card performs a polling waiting process so that when the 128-byte data packet is received, the corresponding flag data F0-0 is polled for completion. Based on the sequential visibility in the CUDA memory model, the phased reception completion result is seen and confirmed by all GPU cards for multi-card reduction and summation. Other GPU cards perform the same operation. When the Flag bit in the last thread of each thread group meets the expected value, it means that the previous 120 bytes of data have been successfully transmitted.

[0044] use Figure 4 The reduction and summation are performed as shown, and the effective bandwidth utilization of the LL128 protocol is [percentage missing].

[0045] 120B / 128B = 93.75%

[0046] Therefore, the bandwidth utilization of the LL128 protocol is greatly improved compared to the LL protocol, resulting in lower time consumption for multi-card reduction summation under high traffic volumes. Figure 5 The figure shown is a comparison of system test results regarding the reduction of latency in multi-GPU card reduction summation in distributed text inference based on the LL128 protocol according to this disclosure. Figure 5 As shown, the distributed data processing system uses eight H800 GPU cards and employs LL and LL128 reduction and summation protocols. The reduction and summation time was tested for different amounts of transmitted data requiring reduction and summation. Figure 5 As shown, under the same data transmission volume, the time spent on reduction and summation based on the LL protocol is significantly longer than that based on the LL128 protocol. This is a significant effect resulting from leveraging the 128-byte sequential visibility of memory operations in the NVLink configuration within the GPU cluster and fully employing packet-based verification to improve bandwidth utilization efficiency. It should be noted that data transmission is performed in a single shot, rather than in stages for reduction and summation.

[0047] One-stage data transfer is a one-stop process. That is, when the GPU supports P2P, the latency of directly accessing the data pointers of each card and sending data is relatively low. Based on this premise, the data transfer component 120 of each GPU card in this disclosure transmits continuously packaged data packets containing all local data from the local GPU card to the buffers of other remote GPU cards in one go, so that each GPU card can perform multi-card reduction summation locally. This is the method of one-stage multi-card reduction summation: each GPU sends data packets to the buffers on other GPUs based on the LL / LL128 communication protocol. After the GPU obtains the data at all ranks, it performs the reduction summation operation locally and writes it back to the result. Figure 6 The diagram illustrates a one-stage multi-GPU reduction summation process according to the system disclosed herein for reducing latency in multi-GPU reduction summation in distributed text inference. Figure 6 As shown, assume there are a total of N GPUs ( Figure 6 (This corresponds to 4, but there can be more or fewer). If the data volume on each GPU is M, then the communication volume per GPU is:

[0048] CommSize = (N-1) * M

[0049] The GPU's local access to the GPU's cache does not require communication. Although one-stage multi-GPU reduction summation can reduce latency, the communication volume is relatively large because each GPU needs to transfer its entire local data to the GPU. Therefore, the one-stage multi-GPU reduction summation method is only suitable for scenarios with low communication volume.

[0050] To this end, this disclosure reduces the communication volume of each GPU through a two-stage multi-GPU reduction summation, thereby reducing the latency of the reduction summation. Figure 7 The diagram illustrates a two-stage multi-GPU reduction summation process according to the system disclosed for reducing latency in distributed text inference using multi-GPU cards. Specifically, before packaging, the data packaging component 110 of each GPU card divides the local full data into sub-shards according to the number of GPU cards, and then packages each sub-shard. For example... Figure 7As shown, in a scenario with four GPUs deployed in a distributed manner, the data volume on each GPU is M. We divide the data requiring reduction and summation into four sub-parts, for example, [M0, M1, M2, M3]. After this reduction and summation data is evenly divided into sub-parts according to the number of GPUs, each GPU data transmission component 120 transmits the corresponding sub-part's continuous data packets from the local GPU card to the corresponding buffer on the remote GPU card. For example, data transmission component 120-0 sends [M1, M2, M3] to buffer 1 of GPU1, buffer 2 of GPU2, and buffer 3 of GPU3, respectively; data transmission component 120-1 sends [M0, M2, M3] to buffer 0 of GPU0, buffer 2 of GPU2, and buffer 3 of GPU3, respectively, and so on. In this first reduction and summation data transmission process, the data transmission volume for each GPU card is:

[0051] CommSize1 = (N-1) * M / N.

[0052] exist Figure 7 With only four GPUs, the data transfer volume is 3M / 4. Each GPU's data transfer component 120 sends one sub-slice to the buffers of the other corresponding GPUs. Each GPU is only responsible for the reduction and summation of the data within one sub-slice. For example, GPU0 is responsible for the reduction and summation of the M0 data on each GPU (four cards, meaning four different M0 sub-slice data), GPU1 is responsible for the reduction and summation of the M1 data on each GPU (four cards, meaning four different M1 sub-slice data), and so on. Therefore, each GPU needs to send its local data sub-slice to its corresponding GPU: GPU0 sends M0 to GPU0, M1 to GPU1, M2 to GPU2, and M3 to GPU3. At this time, the data reception confirmation component 130 of each GPU also performs a polling waiting process, and after verifying the flag data, it uses atomic visibility or sequential visibility to let other GPUs know the reception result.

[0053] After completing the reduction summation within the sub-slice, the data transmission component 120 sends its local reduction summation result sub-slice data to other GPU cards, ensuring that each GPU has a complete reduction summation result. This result transmission process is performed by the data transmission component 120, and the amount of data transmitted is also:

[0054] CommSize2 = (N-1) * M / N.

[0055] Therefore, the total data transmission volume in both stages is

[0056] CommSize = 2(N-1) * M / N.

[0057] Compared to the one-phase reduction summation method, the communication per card is (N-1)*M.

[0058] Since N>=2, it's easy to see that the communication volume of the two-phase reduction summation method is always less than or equal to that of the one-phase reduction summation method. In scenarios with large communication volumes, its communication time is lower than that of the one-phase reduction summation method. It should be noted that because the two-phase reduction summation method involves two communication operations, this latency is slightly higher. When the communication volume is extremely small, the latency caused by these two communication operations can offset the advantage of reducing communication volume, and it may sometimes be slightly higher than that of the one-phase reduction summation method. Figure 8 The figure shown is a comparison of test results for first-order and second-order systems in distributed text inference according to this disclosure, focusing on reducing the latency of multi-GPU card reduction summation. For example... Figure 8 As shown, under low communication volume, the communication latency of first-order reduction summation is actually lower than that of second-order reduction summation. Therefore, in distributed processing systems, although theoretically the communication volume of second-order reduction summation is lower than that of first-order reduction summation, its actual reduction summation latency is negatively affected by the two communication operations involved. However, when the amount of communication data is large, the effect of reducing the communication volume of second-order reduction summation will outweigh the negative effect of the two communication operations, thus significantly reducing the overall latency.

[0059] Optionally, since multi-GPU reduction summation requires copying the input data to a buffer pre-created using the CUDA IPC inter-process communication mechanism, the data and flags are written to the buffer. To distinguish each multi-GPU reduction summation, the buffer needs to be reset or the flags incremented after each summation. Without these processes, when entering the next reduction summation operation, the GPU might mistakenly believe that synchronization between the GPUs has been completed because the flags from the previous summation are still in the buffer, leading to incorrect summation results. To address this, existing vLLM writes a 1 to the buffer flag before starting the reduction summation to indicate the start of synchronization, and resets the flag to 0 after the summation is complete. TensorRT, on the other hand, increments the flags before each multi-GPU reduction summation to ensure that a different flag is used for each summation. Furthermore, considering the CUDA Graph scenario, once the graph is captured, the CUDA Kernel input parameters are fixed and will not be changed during actual calls. In this case, TensorRT's method of incrementing flags will fail. To address this, existing techniques require changing the flags to separate data pointers, incrementing the content within each pointer. All of the aforementioned operations require additional processing of the flags, thus increasing the latency of conversions between each multi-GPU reduction and summation operation.

[0060] To address this, this disclosure employs a double-buffer mechanism to mark each reduction summation. Two buffers, Buffer1 and Buffer2, are created, each using flags Flag1 and Flag2, respectively. A global counter is maintained during model inference, incrementing by 1 with each multi-GPU reduction summation call. Modulo-2 operations are performed on the counter, alternating between the two buffers. Because each buffer uses different flags, different multi-GPU reduction summations can be distinguished. This double-buffer mechanism eliminates the need for buffer resets or additional flag processing, resulting in lower latency.

[0061] In summary, based on the system and method disclosed herein for reducing the latency of multi-GPU reduction summation in distributed text inference, considering that CUDA memory models are 64-bit atomically visible, by combining 32-bit data and a 32-bit Flag bit into a 64-bit data packet each time, when the 32-bit Flag bit can be verified, it also means that the previous 32 bits of data have been transmitted. Each thread in CUDA supports a maximum of 128-bit read / write operations, so two data packets can be written to the remote buffer at once. The remote GPU reads the buffer through busy wait and checks if it matches the Flag bit value; if it does, it indicates that the data has been successfully transmitted. This disclosure utilizes the CUDA memory operation feature of sequential visibility of 128 bytes and the CUDA feature of each thread supporting a maximum of 16 bytes of read and write. Therefore, the data to be transmitted is divided into 128-byte groups, and each data block is divided into 8 threads of 16 bytes each, so that each group exactly meets the CUDA 128-byte transmission limit. For the last thread in each thread group, it stores 8 bytes of data and an 8-byte Flag bit. This 8-byte Flag bit is exactly 64 bits, which meets the CUDA memory model feature of 64-bit atomic visibility. Thus, packet verification is only performed by this thread, while the other threads in the thread group store 16 bytes of valid data. Therefore, when the Flag bit in the last thread of each thread group meets the expected value, it means that the previous 120 bytes of data have been successfully transmitted. This greatly utilizes the low-latency, high-bandwidth optimization protocol designed for inter-GPU collective communication (such as All Reduce and Broadcast) in NVIDIA NCCL (Cluster Communication Library), achieving a protocol bandwidth utilization rate of over 90%. This also improves the rate of multi-GPU reduction summation and reduces its latency from a bandwidth utilization perspective. Furthermore, reducing the amount of data transmitted during the two-stage reduction summation process also reduces the latency of the reduction summation.

[0062] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified to be uniquely different from one or more devices in this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0063] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0064] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A system for reducing the latency of multi-GPU reduction summation in distributed text inference, comprising: The data packaging component packages the data that needs to be executed by the multi-GPU card protocol locally on the GPU card according to a predetermined size. The first part of each data packet is the data to be transmitted, which is divided into segments in order from the fragmented data, and the second part is the flag bit data. The data transmission component is a buffer that transmits the packaged data packets from the local GPU card to other GPU cards at a remote location. as well as The data reception confirmation component, during the polling waiting process, when a data packet of a predetermined size is received, polls the corresponding flag bit data for which the flag bit data has been verified. Based on the verification of the flag bit data, the phased reception completion result is made visible to all GPU cards and confirmed through 64-bit atomic visibility or sequential visibility in the CUDA memory model, so as to perform multi-card reduction summation.

2. The system for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in claim 1, wherein the first part of each data packet is 32 bits of data to be transmitted, sequentially segmented from the fragmented data, the second part is 32 bits of flag data, and the data transmission component transmits the dual data packets from the local GPU card to other remote GPU cards each time, and the data reception confirmation component of each GPU card, during the polling waiting process, when the reception of the 64-bit data packet is completed, polls the corresponding flag data for the flag data that has been verified, and based on the 64-bit atomic visibility in the CUDA memory model, makes the phased reception completion result visible to all GPU cards and confirmed, so as to perform multi-card reduction summation.

3. The system for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in claim 1, wherein the first part of each data packet is 120 bytes of data to be transmitted, sequentially segmented from the fragmented data, the second part is 8 bytes of flag data, and the data transmission component transmits the data packet from the local GPU card to other remote GPU cards each time based on the NVLink data transmission channel, and the data reception confirmation component of each GPU card performs a polling waiting process so that when the reception of the 128-byte data packet is completed, the polling of the corresponding flag data that has been verified for the flag data is completed, and based on the sequential visibility in the CUDA memory model, the phased reception completion result is seen by all GPU cards and confirmed, so as to perform multi-card reduction summation.

4. The system for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in any one of claims 1-3, wherein the data transmission component of each GPU card transmits continuously packaged data packets of the local full data from the local GPU card to the buffers of other GPU cards at the remote end at one time, so that each GPU card can perform multi-card reduction summation locally.

5. The system for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in any one of claims 1-3, wherein each GPU card's data packaging component divides the local full data into sub-shard data according to the number of GPU cards before packaging, and packages each sub-shard data; the data transmission component only transmits the continuous data packets corresponding to the sub-shard data from the local GPU card to the corresponding buffer of the remote GPU card; and after each GPU card receives all its corresponding sub-shard data and performs multi-card reduction summation on all the received sub-shard data locally to obtain the reduction summation result sub-shard data, each GPU card sends its local reduction summation result sub-shard data to other GPU cards.

6. A method for reducing the latency of multi-GPU reduction summation in distributed text inference, comprising: The data packaging component packages the data that needs to be executed by the multi-GPU card protocol locally on the GPU card according to a predetermined size. The first part of each data packet is the data to be transmitted, which is divided into segments in order from the fragmented data, and the second part is the flag bit data. The data transmission component transmits the packaged data packets from the local GPU card to a buffer on another GPU card at a remote location. as well as The data reception confirmation component performs a polling waiting process so that when a data packet of a predetermined size is received, the corresponding flag bit data for which the flag bit data has been verified is polled to complete. Based on the verification of the flag bit data, the phased reception completion result is made visible to all GPU cards and confirmed through 64-bit atomic visibility in the CUDA memory model or sequential visibility in the CUDA memory model, so as to perform multi-card reduction summation.

7. The method for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in claim 6, wherein the first part of each data packet is 32 bits of data to be transmitted, sequentially segmented from the fragmented data, the second part is 32 bits of flag data, and the data transmission component transmits the dual data packets from the local GPU card to other remote GPU cards each time, and the data reception confirmation component of each GPU card performs a polling waiting process so that when the reception of the 64-bit data packet is completed, the polling of the corresponding flag data that has been verified for the flag data is completed, and based on the 64-bit atomic visibility in the CUDA memory model, the phased reception completion result is seen by all GPU cards and confirmed, so as to perform multi-card reduction summation.

8. The method for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in claim 6, wherein the first part of each data packet is 120 bytes of data to be transmitted, sequentially segmented from the fragmented data, the second part is 8 bytes of flag bit data, and the data transmission component transmits the data packet from the local GPU card to other remote GPU cards each time based on the NVLink data transmission channel, and the data reception confirmation component of each GPU card performs a polling waiting process so that when the reception of the 128-byte data packet is completed, the polling of the corresponding flag bit data that has been verified for the flag bit data is completed, and based on the sequential visibility in the CUDA memory model, the phased reception completion result is seen by all GPU cards and confirmed, so as to perform multi-card reduction summation.

9. The method for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in any one of claims 6-8, wherein the data transmission component of each GPU card transmits continuously packaged data packets of the local full data from the local GPU card to the buffers of other GPU cards at the remote end at one time, so that each GPU card can perform multi-card reduction summation locally.

10. The method for reducing the latency of multi-GPU card reduction summation in distributed text inference as described in any one of claims 6-8, wherein each GPU card's data packaging component divides the local full data into sub-shard data according to the number of GPU cards before packaging, and packages each sub-shard data; the data transmission component only transmits the continuous data packets corresponding to the sub-shard data from the local GPU card to the corresponding buffer of the remote GPU card; and after each GPU card receives all its corresponding sub-shard data and performs multi-card reduction summation on all the received sub-shard data locally to obtain the reduction summation result sub-shard data, each GPU card sends its local reduction summation result sub-shard data to other GPU cards.

Images