Communication optimization method, device, medium and system for tensor parallel training

Abstract

The application discloses a communication optimization method and device for tensor parallel training, a medium and a system, relates to the field of artificial intelligence, and is used for improving tensor parallel system resource utilization and compression and communication efficiency. At least one piece of tensor data to be executed in set communication is acquired, and the tensor data is divided into multiple communication data blocks; compression and communication transmission are respectively performed on each communication data block, so that a receiving end reconstructs the tensor data according to all the communication data blocks; wherein the compression and communication transmission processes of at least two communication data blocks are time overlapped. Through fusion of key technologies such as compression communication overlap execution, compression execution, Chunk pipeline processing and tensor parallel perception scheduling, deep collaborative optimization of the compression process and the communication transmission process is realized, GPU and network resource utilization is significantly improved, and compression calculation overhead and display memory access overhead are reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a communication optimization method, apparatus, medium and system for Tensor Parallelism (TP) training. Background Technology

[0002] With the continuous growth in the parameter scale of large language models, hybrid expert models, and multimodal base models, tensor parallelism has become an important technique for overcoming the limitations of single-GPU memory capacity. During tensor parallel training, matrix computations in a single-layer Transformer are distributed across multiple GPUs for collaborative execution. To recover the complete computation results, frequent ensemble communication operations such as AllGather, ReduceScatter, and AllReduce are required between the GPUs. Therefore, as tensor parallelism continues to increase, communication overhead gradually becomes the main bottleneck limiting training scaling efficiency.

[0003] Known solutions typically employ communication compression techniques to reduce communication overhead. These techniques compress and transmit data at the tensor level, resulting in idle network resources during compression and idle GPU computing resources during transmission. This leads to low utilization of both GPU and network resources and high communication latency. Furthermore, tensor parallel training involves high communication frequencies, making these methods ill-suited to meet the demands of high-frequency calls, strong synchronization, and error sensitivity inherent in tensor parallel communication. Summary of the Invention

[0004] The purpose of this invention is to provide a communication optimization method, apparatus, medium, and system for tensor parallel training, addressing all or part of the aforementioned problems, in order to improve the resource utilization and compression and communication efficiency of tensor parallel systems.

[0005] The technical solution adopted in this invention is as follows: In a first aspect, this application provides a communication optimization method for tensor parallel training, comprising: Obtain at least one tensor data point for the set communication to be executed, and divide the tensor data into multiple communication data blocks; Each of the communication data blocks is compressed and transmitted separately, so that the receiving end can reconstruct the tensor data based on all the communication data blocks; wherein the compression and transmission processes of at least two of the communication data blocks overlap in time.

[0006] Optionally, the compression and communication transmission includes a compression processing stage and a communication transmission stage; both the compression processing stage and the communication transmission stage of any of the communication data blocks adopt a pipelined execution mechanism.

[0007] Optionally, the compression and communication transmission processes of at least two of the communication data blocks overlap in time, including: In at least two of the aforementioned communication data blocks, the communication transmission phase of the first communication data block and the compression processing phase of the second communication data block overlap in time.

[0008] Optionally, the compression processing stage of the second communication data block and the communication transmission stage of the first communication data block are consecutive in time.

[0009] The optional compression processing stage includes a statistical calculation sub-stage, a scaling processing sub-stage, a data transformation sub-stage, and a quantization compression sub-stage; the same communication data block shares intermediate data in at least two sub-stages of the compression processing stage.

[0010] Optionally, the quantization compression sub-stage processes the communication data block using a low-bit quantization format.

[0011] In a second aspect, this application also provides a communication optimization apparatus for tensor parallel training, comprising a processor and a storage medium; the storage medium stores a computer program, and the processor runs the computer program to execute the above-described communication optimization method for tensor parallel training.

[0012] In a third aspect, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the aforementioned communication optimization method for tensor-oriented parallel training.

[0013] In a fourth aspect, this application provides another communication optimization method for tensor parallel training, which includes: Obtain at least one tensor data point for the set communication to be executed, and divide the tensor data into multiple communication data blocks; Each of the communication data blocks is processed in three stages: compression, communication transmission, and recovery. The tensor data is then reconstructed based on all the recovered communication data blocks. The compression-communication transmission-recovery processes of at least two of the communication data blocks overlap in time.

[0014] In a fifth aspect, this application provides a communication optimization system for tensor parallel training, comprising: The data acquisition module acquires at least one tensor data point for the set communication to be executed; The data block partitioning module divides the tensor data into multiple communication data blocks; The scheduling module performs a three-stage processing of compression, communication transmission, and recovery on each of the communication data blocks; wherein the three-stage processing of compression, communication transmission, and recovery of at least two of the communication data blocks overlaps in time. The reconstruction module reconstructs the tensor data based on all recovered communication data blocks.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: This application achieves deep collaborative optimization of the compression and communication transmission processes by integrating key technologies such as compressed communication overlapping execution, compressed execution, chunk pipeline processing, and tensor parallel perceptual scheduling. Compared with existing technologies, this application significantly improves GPU and network resource utilization, reduces communication latency, increases training throughput, reduces compression computational overhead and GPU memory access overhead, and can adapt to tensor parallel training scenarios of large models of different scales, demonstrating good engineering application value and industrial promotion prospects. Attached Figure Description

[0016] The present invention will be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of an implementation of a communication optimization method for tensor parallel training in a scenario targeting the sending end.

[0017] Figure 2 This is a construction diagram of a communication optimization device for tensor parallel training in one embodiment.

[0018] Figure 3 This is a flowchart of an implementation of a communication optimization method for tensor parallel training in a context of the overall system.

[0019] Figure 4 This is a construction diagram of a communication optimization system for tensor parallel training in one embodiment. Detailed Implementation

[0020] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0021] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0022] With the ever-increasing parameter size of large-scale foundational models such as GPT, LLaMA, Qwen, DeepSeek, and Mixtral, tensor parallel training has become a crucial technique for overcoming the limitations of single-device memory capacity. During tensor parallel training, large-scale matrix operations are distributed across multiple computing devices. To recover complete computational results, frequent ensemble communication operations such as AllGather, ReduceScatter, and AllReduce are required between different devices. As tensor parallelism increases, communication overhead gradually surpasses single-layer computational overhead, ultimately becoming the main bottleneck limiting system scalability.

[0023] Communication compression techniques primarily reduce the amount of communication data through quantization compression, data transformation, or sparsity processing. Known communication compression techniques mainly include the following categories: (1) Low-bit quantization communication method.

[0024] Low-bit quantization is currently the most mainstream communication compression technique in distributed training. Its core idea is to reduce the amount of communication data by reducing the precision of data representation. Specifically, this type of method uses low-bit formats such as FP8, INT8, or INT4 to replace FP16 or BF16 to represent communication tensors, thereby reducing the storage and transmission overhead per unit of data. Representative methods include QSGD, FP8 Communication, DeepSpeed ​​Compression, SDP4Bit, and TahQuant. This type of method has been widely used in data parallelism (DP) and pipeline parallelism (PP) scenarios, effectively reducing communication bandwidth requirements while ensuring model convergence. However, in tensor parallelism, due to the high communication frequency and sensitivity to numerical errors, directly applying low-bit quantization can easily introduce error accumulation problems, thus affecting training stability.

[0025] (2) Non-quantization communication compression methods.

[0026] In addition to quantization methods, various non-quantization communication compression techniques have been proposed, including sparsification methods, low-rank approximation methods, and coding methods based on data redundancy.

[0027] Sparsity methods significantly reduce the amount of communication data by transmitting only the most important elements of the tensor (such as the Top-k elements or elements exceeding a threshold). Typical methods include Top-k Gradient and Deep Gradient Compression, usually combined with error compensation mechanisms to ensure convergence. Low-rank approximation methods utilize the low-rank structure of matrices to decompose high-dimensional tensors into multiple low-dimensional matrices for transmission, such as PowerSGD. In addition, there are compression methods based on differential coding or entropy coding, which further reduce communication overhead by utilizing the temporal correlation or statistical redundancy of data. The above methods are effective in scenarios with low communication frequency, such as data parallelism and pipelined parallelism. However, due to the high-frequency calls, strong synchronization, and error sensitivity of tensor parallel communication, these methods are often difficult to apply directly in tensor parallel scenarios, easily leading to information loss accumulation or training instability.

[0028] (3) Communication compression system and communication optimization technology.

[0029] To reduce communication overhead and improve overall system efficiency, some research has focused on jointly optimizing communication compression and the communication process at the system level. On one hand, compression operations are integrated with the communication framework to form a communication compression system, reducing additional data copying and scheduling overhead. On the other hand, network utilization is improved by optimizing communication mechanisms. Typical techniques include NCCL-based Ring AllReduce, Hierarchical AllReduce, communication scheduling optimization, and Communication-Compute Overlap. These methods primarily improve performance from the perspectives of communication protocols and execution scheduling, alleviating communication bottlenecks to some extent.

[0030] However, most of the above methods design compression and communication as relatively independent modules, lacking a collaborative optimization mechanism for the high-frequency communication characteristics of tensor parallelism, making it difficult to fully leverage the performance benefits brought by communication compression while ensuring training stability.

[0031] Furthermore, research has revealed that while known tensor parallel communication compression methods can reduce the amount of communication data, their overall execution flow still follows a serial processing pattern of "statistics-scale calculation-transformation-quantization-communication-reduction." The significant computational overhead introduced by the compression process itself still offsets some of the communication gains. Especially in current mainstream implementations, statistical analysis, scale calculation, data transformation, and quantization compression are often completed by multiple independent computational stages. Each stage requires a separate GPU computation task, accompanied by memory read / write operations and thread synchronization.

[0032] In summary, current methods for reducing the amount of data in tensor parallel communication have the following shortcomings: 1) Compression and serial communication execution lead to resource idleness.

[0033] Known methods typically initiate communication only after the entire tensor compression is complete. This results in a situation where the GPU works during the compression phase and is idle during the communication phase, while the network is idle during the compression phase and works during the communication phase. This leads to underutilization of the GPU and network, limiting the overall system efficiency.

[0034] 2) Large tensor communication cannot form a pipeline.

[0035] Known methods typically process the entire tensor as the smallest unit. Network transmission is only triggered after the entire tensor is compressed, resulting in network inactivity during compression. As the tensor size increases, the waiting time lengthens, network utilization decreases, and ultimately, communication latency increases further.

[0036] 3) Multi-stage compression increases GPU startup overhead.

[0037] Since statistical computation, scaling, data transformation, and quantization compression are typically performed by multiple independent GPU kernels, each additional compression stage requires launching an additional GPU kernel. As tensor size and communication frequency increase, the overhead of kernel launches accumulates continuously.

[0038] 4) Multiple accesses to video memory increase compression costs.

[0039] In known methods, each kernel needs to perform a "tensor read-computation-memory write-back" process, which is then repeated in the next stage. Therefore, with the increase in processing stages, the number of memory read / write operations increases, leading to increased memory bandwidth consumption and higher compression costs. This problem is particularly pronounced when the communication tensor reaches tens or even hundreds of MB.

[0040] 5) Lack of a unified collaborative optimization framework for tensor parallelism.

[0041] Existing compression technologies mainly focus on improving compression ratio and reducing quantization error, while communication optimization technologies focus on communication protocols and communication scheduling. Compression systems and communication systems are usually designed independently and are isolated from each other, which means that the benefits of compression cannot be fully converted into an increase in training throughput.

[0042] To address the aforementioned shortcomings, this application proposes a communication optimization method, apparatus, medium, and system for tensor parallel training. By coordinating the optimization of compression and communication transmission (and possibly including a recovery phase), it improves the overall utilization of GPU computing resources and network transmission resources, reducing end-to-end latency for single communications. Furthermore, by integrating and scheduling the compression phase, it improves compression efficiency and reduces overall training time.

[0043] Tensor parallelism has three main stages: compression processing, communication transmission, and recovery / reconstruction. The compression processing and communication transmission stages occur at the sending end, while the recovery / reconstruction stage occurs at the receiving end. The communication optimization scheme proposed in this application is applicable not only to the optimization of the sending end but also to the overall optimization of the system (sender + receiver). In the embodiments of this application, these two optimization targets will be described separately.

[0044] In optimization schemes targeting the sending end, such as Figure 1 As shown, the communication optimization method for tensor parallel training proposed in this application includes the following two stages: Phase 1: Obtain at least one tensor data from the set communication to be executed, and divide the tensor data into multiple communication data blocks.

[0045] In the tensor parallel training process, a large-scale matrix is ​​divided into multiple local tensors for computation on multiple GPUs. The method proposed in this application can be applied to any (all) of these tensors.

[0046] After the tensor parallel layer completes the local matrix calculation, it intercepts the data stream that was originally sent directly to the ensemble communication library and redirects it to the communication optimization framework proposed in this application. In a typical implementation, the tensor data to be communicated can originate from the activation value synchronization process in the Transformer layer or from the gradient reduction process in the backpropagation phase. This application does not limit the source of the tensor data and is applicable to any data object that needs to perform ensemble communication.

[0047] After capturing tensor data, it is divided into multiple communication data chunks according to a preset scale. This design differs from known methods that treat tensors as the smallest processing unit. Large-scale communication tensors typically reach tens or even hundreds of MB in size. If communication cannot be initiated until the entire tensor is compressed, the network link will be idle for a long time during the compression phase. This waiting time will continue to increase as the tensor size further increases. Therefore, this application divides the complete tensor into multiple communication data chunks, each of which can be independently compressed and transmitted (and recovered). In this way, the communication process can be initiated in advance without waiting for the entire tensor to be processed, thereby significantly improving network resource utilization.

[0048] Phase 2: Compress and transmit each communication data block separately.

[0049] After the communication data blocks are divided, each block can be compressed and transmitted separately, allowing the receiving end to reconstruct the tensor data based on all blocks. It's important to note that at least two communication data blocks experience time overlap during compression and transmission. This means that at least one block can perform compression / transmission in parallel while other blocks are being compressed or transmitted. This allows the GPU to avoid waiting for all blocks to be fully transmitted before compression, and the network to avoid waiting for all blocks to be fully compressed before transmission. GPU computing resources and network transmission resources can be utilized simultaneously, reducing overall latency, improving communication efficiency, and increasing overall resource utilization.

[0050] Depending on the specific processing stage, the aforementioned compression and communication transmission includes a compression processing stage and a communication transmission stage. As an optional implementation, both the compression processing stage and the communication transmission stage of any communication data block employ a pipelined execution mechanism. For any communication data block, the compression processing stage and the communication transmission stage are executed sequentially, and, where GPU computing resources and network transmission resources permit, the two stages are sequential in time. That is, for any communication data block, if network transmission resources are idle after compression processing is completed, communication transmission is immediately initiated. In this way, each communication data block can complete compression communication at the fastest speed, improving the overall efficiency of tensor compression communication and ensuring that GPU computing resources and network transmission resources are always available, thus improving overall resource utilization.

[0051] As an optional implementation, in stage two, in at least two communication data blocks, the communication transmission phase of the first communication data block and the compression processing phase of the second communication data block overlap in time. That is, in two communication data blocks where the compression and communication transmission processes overlap, the first communication data block (i.e., the first communication data block) is transmitting data while the second communication data block (i.e., the second communication data block) is simultaneously undergoing compression processing. Since the communication transmission and compression processing times of the two communication data blocks may not be identical, the communication transmission process of the first communication data block and the compression processing process of the second data block may not completely overlap in time. However, whether they completely overlap does not affect the overall compression communication efficiency of the tensor or the overall resource utilization rate.

[0052] In the above implementation, as a preferred design, the compression processing stage of the second communication data block is configured to be sequential with the compression processing stage of the first communication data block. That is, after the first communication data block is compressed, the GPU immediately compresses the second communication data block. For multiple communication data blocks, according to this design principle, regardless of whether the compressed communication data block has already been transmitted, the GPU immediately compresses the next communication data block after completing the compression processing of the previous one.

[0053] The above method ensures that compression and communication tasks are always present within the system during tensor compression communication. After compressing the previous communication data block, the next communication data block is immediately compressed, and simultaneously, if network transmission resources are idle, the previous communication data block is immediately transmitted; after transmitting the previous communication data block, the next communication data block is immediately transmitted. This allows tensor compression communication to be performed without interruption of GPU computing resources and network transmission resources, preventing prolonged periods of idle GPU computing resources and network transmission resources, significantly improving the efficiency and resource utilization of tensor compression communication. It should be noted that even if only a portion of the communication data blocks (e.g., at least two communication data blocks) are time-overlapped in their compression and transmission processes, compared to the method described above that uses tensors as the smallest unit for compression communication, there is still a certain degree of improvement in overall efficiency and resource utilization.

[0054] Furthermore, regarding the compression processing stage alone, the compression processing stage for tensors (and similarly for communication data blocks) includes a statistical calculation sub-stage, a scaling sub-stage, a data transformation sub-stage, and a quantization compression sub-stage. Traditionally, each sub-stage is completed by multiple independent GPU kernels; that is, each additional sub-stage requires the launch of an additional GPU kernel. A large number of tensors directly leads to the continuous accumulation of GPU kernel overhead. In addition, each time a sub-stage is executed, the initial state or data from the previous sub-stage needs to be read from video memory, processed, and then written back to video memory. As the number of processing sub-stages increases, the frequency of video memory read / write operations increases accordingly, leading to increased memory bandwidth consumption and higher compression costs.

[0055] To address the aforementioned issues, in one optional implementation, this application eliminates / reduces the additional storage and data transfer overhead in traditional compression processing stages by integrating some or all sub-stages within the compression processing stage and optimizing data paths.

[0056] Specifically, in this application, the design involves at least two sub-stages of the compression processing phase sharing intermediate data for the same communication data block, thereby integrating at least two sub-stages into the same execution flow. Since the four sub-stages are executed in a pipeline, these at least two sub-stages are consecutive sub-stages.

[0057] As a preferred implementation, all four sub-stages can be integrated into the same execution flow. Under this mechanism, the compression processing of any communication data block no longer depends on data write-back and reloading between sub-stages, but is completed continuously within a single execution context. After the communication data block is loaded, its related statistical calculations and subsequent processing (scaling, data transformation, and quantization compression) are all completed in the same execution process. The intermediate results (intermediate data) of each sub-stage are directly transferred in registers or on-chip shared memory, without writing to global video memory or repeatedly reading and writing to video memory. This effectively reduces intermediate storage overhead and read / write overhead, meaning that no additional intermediate buffers need to be allocated for data storage between sub-stages during compression. Simultaneously, by avoiding repeated write-back and read-back of intermediate data, the number of global video memory accesses is significantly reduced, thereby reducing video memory bandwidth usage. At the execution level, since multiple sub-stages are integrated into the same execution flow, the scheduling and synchronization overhead caused by multi-sub-stage execution is reduced, improving overall execution efficiency.

[0058] Furthermore, this fusion execution method enables communication data blocks to complete the entire compression process continuously after loading and be directly output, effectively improving the continuity and locality of compression processing and avoiding the data flow interruption problem of traditional staged execution modes. Therefore, this application achieves efficient execution of the communication compression process without introducing additional video memory resources, ensuring that the compression process itself does not become a new performance bottleneck.

[0059] In addition, in order to effectively reduce the overhead of the communication transmission stage, in an optional implementation, the communication data blocks are processed using a low-bit quantization format during the compression processing stage (quantization compression sub-stage). By reducing the data precision, the amount of data to be transmitted is reduced, thereby improving the communication transmission efficiency.

[0060] Based on the ideas of this application, this application embodiment also provides a communication optimization device for tensor parallel training, such as... Figure 2 As shown, it includes a processor and a storage medium. The storage medium stores a computer program, which the processor runs to execute the aforementioned communication optimization method for tensor-oriented parallel training.

[0061] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the aforementioned communication optimization method for tensor-oriented parallel training.

[0062] In optimization schemes targeting the overall system, such as Figure 3 As shown, the communication optimization method for tensor parallel training proposed in this application includes the following two stages: Phase 1: Obtain at least one tensor data from the set communication to be executed, and divide the tensor data into multiple communication data blocks.

[0063] Phase 1 is exactly the same as the optimization scheme for the sending end mentioned earlier, so it will not be repeated here.

[0064] Phase 2: Perform three-stage processing on each communication data block: compression, communication transmission, and recovery. Reconstruct the tensor data based on all recovered communication data blocks.

[0065] Unlike optimization schemes targeting the sending end, optimization schemes for the entire system consider not only the compression and transmission of communication data blocks but also the recovery process. In stage two, the compression-transmission-recovery three-stage processing of at least two communication data blocks overlaps in time.

[0066] Similar to the optimization scheme for the sending end, in this embodiment, at least one communication data block does not need to wait for other data blocks to complete compression, communication transmission, and recovery before compression / communication transmission / recovery. Instead, compression / communication transmission / recovery is performed in parallel while other communication data blocks are undergoing compression, communication transmission, or recovery. This allows GPU computing resources and network transmission resources to be utilized simultaneously in time, and also enables the receiving end to receive the compressed communication data block for recovery as quickly as possible, rather than recovering all the tensor data at once after receiving the entire tensor data. This improves the reconstruction efficiency of tensor data. The recovery stage is the inverse stage of the compression process, including three sub-stages: inverse quantization, inverse transform, and scale recovery (inverse scaling). All recovered communication data blocks are concatenated according to the partitioned construction relationship, thus completing the reconstruction of the tensor. The reconstructed tensor can then be returned to the subsequent neural network computation process.

[0067] The three-stage compression-communication transmission-recovery process includes not only the compression and communication transmission processes described above, but also a recovery stage at the receiving end. Therefore, in optimization scenarios for the overall system, the compression, communication transmission, and recovery stages of any communication data block all employ a pipelined execution mechanism. That is, for any communication data block, after compression is completed, if network transmission resources are idle, communication transmission is immediately initiated; after transmission is completed, the recovery module immediately performs recovery processing.

[0068] As an optional implementation, in stage two, in at least two communication data blocks, the communication transmission stage of the first communication data block and the compression processing stage of the second communication data block overlap in time.

[0069] Alternatively, the recovery phase of the first communication data block overlaps with at least one of the compression phase or communication transmission phase of the second communication data block.

[0070] The above methods can effectively improve overall resource utilization, compressed communication efficiency, and tensor recovery efficiency.

[0071] As an optional implementation, the compression processing stage of the second communication data block is configured to be sequential in time with the compression processing stage of the first communication data block. Alternatively, the communication transmission stage of the second communication data block is configured to be sequential in time with the communication transmission stage of the first communication data block. Alternatively, the recovery stage of the second communication data block is configured to be sequential in time with the recovery stage of the first communication data block. Alternatively, the compression processing stage of the second communication data block is configured to be sequential in time with the communication transmission stage of the first communication data block.

[0072] Any of the above implementation methods can ensure that at least two tasks (compression task, communication transmission task and recovery task) always exist simultaneously within the system, thereby improving the processing efficiency and resource utilization of tensor parallelism.

[0073] In the compression processing stage, the optimization scheme for the overall system can be consistent with the optional implementation of the optimization scheme for the sending end. That is, at least two sub-stages of the multiple sub-stages of the compression processing stage can be integrated into the same execution flow. Alternatively, in the quantization compression sub-stage of the compression processing stage, a low-bit quantization format can be used to process the communication data blocks.

[0074] Furthermore, for the recovery phase, if its three sub-phases originally needed to be completed by multiple independent GPU Kernels, and each GPU Kernel also needed to perform the "memory read-process-memory write-back" process, then the optimized design of the compression processing phase can be referenced to integrate at least two sub-phases (preferably all sub-phases) in the recovery phase into the same execution flow to achieve a similar effect to integrating the sub-phases of the compression processing phase mentioned above.

[0075] It is worth noting that, regardless of the implementation method used, this application does not rely on a specific communication protocol during the communication transmission phase. Instead, it can work in conjunction with mainstream collection communication libraries, including implementations such as NCCL, RCCL, and MPI. For different communication primitives such as AllGather, ReduceScatter, and AllReduce, this application adopts a unified communication data block scheduling and pipeline management mechanism. Therefore, this application has good versatility and scalability.

[0076] Based on the ideas in this application, this application also provides a communication optimization system for tensor parallel training, such as... Figure 4 As shown, the system includes: The data acquisition module acquires at least one tensor data from the set communication to be executed.

[0077] The data block partitioning module divides the tensor data into multiple communication data blocks.

[0078] The scheduling module performs a three-stage processing of compression, communication transmission, and recovery on each of the communication data blocks; wherein the three-stage processing of compression, communication transmission, and recovery on at least two communication data blocks overlaps in time.

[0079] The reconstruction module reconstructs the tensor data based on all recovered communication data blocks.

[0080] The optional implementation methods of each module in the above system can be optimized and configured by referring to the optional implementation methods for the same function described above, provided that there is no conflict. They will not be described in detail here.

[0081] Through the above technical solution, this application achieves deep collaborative optimization of the communication compression process and the collective communication process. Compared with known methods, this application not only reduces the GPU kernel startup overhead and memory access overhead in the compression process, but also realizes pipelined overlapping execution of compression, communication (and recovery) processes, improving the utilization of GPU and network resources. This allows the bandwidth benefits brought by communication compression to be effectively converted into an increase in training throughput, making it more suitable for future ultra-large-scale tensor parallel training systems.

[0082] In summary, compared with existing tensor parallel communication compression methods, this application achieves coordinated optimization of the compression process and the communication transmission process by constructing a fusion communication compression framework, a communication data block pipeline scheduling mechanism, and a compression communication overlap mechanism (i.e., time overlap), thereby significantly improving the overall efficiency of large-scale model training systems.

[0083] First, existing compression communication schemes typically divide steps such as statistical analysis, scaling calculation, data transformation, and quantization compression into multiple independent execution sub-stages. Since each sub-stage requires launching an independent GPU kernel, as the communication frequency increases, the overhead of launching numerous GPU kernels accumulates, causing the compression benefits to be partially offset by the additional computational cost. This application integrates multiple compression sub-stages into a single execution flow through a fusion compression mechanism, thereby significantly reducing the number of GPU kernel launches, lowering synchronization and scheduling overhead during execution, and improving compression computation efficiency.

[0084] Secondly, traditional compression stages require frequent reads and writes to global memory between different compression sub-stages. After each compression stage, intermediate results must be written back to memory, and subsequent stages must reread the data for further processing. As the size of the communication tensor increases, numerous memory accesses consume GPU memory bandwidth resources and further increase compression latency. This application addresses this by employing a fused execution mechanism, enabling the transfer and processing of intermediate data generated during compression within the same execution unit. This reduces the number of times intermediate results are written back to memory, thereby lowering global memory access overhead and improving GPU memory system utilization efficiency.

[0085] Furthermore, existing communication compression schemes typically employ a serial execution mode of "compression completed before communication begins." During the compression phase, the GPU is active while the network is idle; during the communication phase, the network is active while the GPU is waiting. This mode prevents computing and network resources from being utilized simultaneously, reducing overall system utilization. This application utilizes a chunk-level pipeline mechanism to divide the tensor to be communicated into multiple independent data blocks, and immediately initiates communication transmission after any data block is compressed. This achieves overlapping execution of the compression and communication processes, enabling GPU computing and network resources to work simultaneously and improving overall resource utilization.

[0086] Furthermore, for processing large-scale communication tensors, existing technologies typically use the entire tensor as the processing unit, requiring the entire tensor to be compressed before transmission can begin. As the tensor size increases, communication initiation time is significantly delayed, leading to a continuous decline in network utilization. This application employs a Chunk pipelined processing mechanism, dividing the entire tensor into multiple communication data blocks. Each communication data block undergoes compression, communication transmission, and recovery stages, forming a continuous pipelined execution mode. While the current communication data block is being compressed, the previous communication data block can perform communication transmission, and earlier data blocks can perform recovery operations. This achieves continuous overlapping execution of compression, communication transmission, and recovery processes, significantly reducing end-to-end latency in a single communication session.

[0087] On the other hand, the theoretical benefit of communication compression technology comes from the reduction in the amount of communication data. However, in existing technologies, due to the additional computational overhead introduced by the compression process itself, the benefits of reducing the amount of communication data often cannot be fully translated into improved training performance. This application reduces compression overhead by fusing compression mechanisms and hides communication latency through overlapping execution mechanisms, enabling the bandwidth benefits of communication compression to be directly translated into improved training throughput. Therefore, this application not only reduces the amount of communication data but also effectively reduces the overall training time.

[0088] Furthermore, this application abstracts the compression, communication transmission, and recovery processes into a unified collaborative framework, which can simultaneously support multiple collective communication operations such as AllGather, ReduceScatter, and AllReduce. Therefore, this application is not only applicable to single communication scenarios, but also covers all core communication processes in current mainstream tensor parallel training frameworks, exhibiting good versatility and scalability.

[0089] This application is not limited to the specific embodiments described above. This application extends to any new features or combinations disclosed in this specification, as well as any new steps or combinations of any new methods or processes disclosed.

Claims

1. A communication optimization method for tensor parallel training, characterized in that, include: Obtain at least one tensor data point for the set communication to be executed, and divide the tensor data into multiple communication data blocks; Each of the communication data blocks is compressed and transmitted separately, so that the receiving end can reconstruct the tensor data based on all the communication data blocks; wherein the compression and transmission processes of at least two of the communication data blocks overlap in time.

2. The communication optimization method for tensor parallel training as described in claim 1, characterized in that, The compression and communication transmission include a compression processing stage and a communication transmission stage; both the compression processing stage and the communication transmission stage of any of the communication data blocks adopt a pipelined execution mechanism.

3. The communication optimization method for tensor parallel training as described in claim 2, characterized in that, The compression and transmission processes of at least two of the aforementioned communication data blocks overlap in time, including: In at least two of the aforementioned communication data blocks, the communication transmission phase of the first communication data block and the compression processing phase of the second communication data block overlap in time.

4. The communication optimization method for tensor parallel training as described in claim 3, characterized in that, The compression processing stage of the second communication data block is sequential in time with the communication transmission stage of the first communication data block.

5. The communication optimization method for tensor-oriented parallel training as described in any one of claims 2-4, characterized in that, The compression processing stage includes a statistical calculation sub-stage, a scaling processing sub-stage, a data transformation sub-stage, and a quantization compression sub-stage; the same communication data block shares intermediate data in at least two sub-stages of the compression processing stage.

6. The communication optimization method for tensor parallel training as described in claim 5, characterized in that, The quantization and compression sub-stage processes the communication data block using a low-bit quantization format.

7. A communication optimization device for tensor parallel training, characterized in that, It includes a processor and a storage medium; the storage medium stores a computer program, and the processor runs the computer program to perform the communication optimization method for tensor-oriented parallel training as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The medium stores a computer program, which, when executed by a processor, performs the communication optimization method for tensor-oriented parallel training as described in any one of claims 1-6.

9. A communication optimization method for tensor parallel training, characterized in that, include: Obtain at least one tensor data point for the set communication to be executed, and divide the tensor data into multiple communication data blocks; Each of the communication data blocks is processed in three stages: compression, communication transmission, and recovery. The tensor data is then reconstructed based on all the recovered communication data blocks. The compression-communication transmission-recovery processes of at least two of the communication data blocks overlap in time.

10. A communication optimization system for tensor parallel training, characterized in that, include: The data acquisition module acquires at least one tensor data point for the set communication to be executed; The data block partitioning module divides the tensor data into multiple communication data blocks; The scheduling module performs a three-stage processing of compression, communication transmission, and recovery on each of the communication data blocks; wherein the three-stage processing of compression, communication transmission, and recovery of at least two of the communication data blocks overlaps in time. The reconstruction module reconstructs the tensor data based on all recovered communication data blocks.

Images