Tensor-based compression communication optimization method, system, device and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-09
Smart Images
Figure CN122174896A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a method, system, device, and medium for optimizing compressed communication based on tensor parallelism. Background Technology
[0002] With the rapid development of artificial intelligence technology, large-scale language models, such as LLaMA-3-70B, Grok-3, and DeepSeek-V3, have hundreds of billions or even trillions of parameters. These models have demonstrated outstanding performance in natural language processing, multimodal tasks, and other fields, but they have also directly led to bottlenecks in memory capacity and computing power for single GPUs. Taking the H100 GPU as an example, its 80GB HBM3 memory is insufficient to accommodate the weights, activation values, and key-value caches of a 70B parameter model. Especially in long sequence inference or large-scale training scenarios, memory overflow and computational latency issues become increasingly prominent, which not only limits the further expansion of the model but also increases the economic cost and carbon footprint of training and inference, urgently requiring the support of efficient distributed training strategies.
[0003] Distributed training has evolved from single-data parallelism to multi-dimensional hybrid parallelism. Among these, tensor parallelism has become the mainstream technology for training and inference of models with billions of nodes due to its ability to achieve fine-grained partitioning and near-linear scaling of intra-layer computations. Tensor parallelism breaks through the single-card memory wall by partitioning matrix operations within the Transformer layer along the hidden dimension and aggregating the results through All-Reduce operations. However, this process also introduces significant communication overhead: each layer requires two All-Reduce synchronization activation values during forward propagation, and the same applies to backward propagation. Taking LLaMA-3-70B inference as an example, when the sequence length reaches 8192, All-Reduce communication can account for more than 40%. In a training scenario with a TP degree of 8, communication time accounts for nearly 40% of the total time of a single iteration. This communication bottleneck stems from both the physical limitations of the interconnect bandwidth between GPUs and the influence of message size and the efficiency of aggregate communication algorithms, and is more easily amplified in multi-node clusters.
[0004] To alleviate the communication pressure between GPUs, academia and industry have proposed various compression techniques, including TopK sparsity, group quantization, and random k-sparseness. However, some of these techniques mainly target gradient communication during backpropagation, without addressing the compression of activation values during forward propagation, which also involves a huge amount of communication for activation values. Some techniques employ a hybrid quantization strategy combining TopK and group quantization, requiring data to be grouped and processed separately, resulting in complex software logic and underutilization of hardware features. Other techniques focus on parameter synchronization in data-parallel scenarios, dynamically determining whether to compress data based on data block sparsity, without optimizing for tensor-parallel intra-layer activation value synchronization scenarios. More importantly, existing technologies have not resolved a core contradiction: NVIDIA Hopper architecture (H100 / H800) has fully supported FP8 precision TensorCore operations at the hardware level, with a single-card FP8 matrix multiplication throughput 1.8 to 2 times higher than FP16. Furthermore, frameworks such as Transformer Engine and PyTorch 2.1+ have provided FP8 computing interfaces. However, the distributed communication primitives of mainstream deep learning frameworks do not yet support the FP8 data type. This means that existing FP8 training schemes need to upgrade FP8 to FP16 before communication and then downgrade it back to FP8 after communication. This makes it impossible to reduce communication volume by utilizing the FP8 format, which severely weakens the hardware acceleration advantage and creates a "weakest link" effect.
[0005] In addition, existing compression schemes for activation values are mostly general-purpose designs that cover multiple objects such as activation values, model parameters, and gradients. They are not specifically optimized for communication scenarios of tensor parallel forward activation values, making it difficult to balance communication efficiency, hardware compatibility, and numerical accuracy. Summary of the Invention
[0006] Therefore, it is necessary to provide a tensor-parallel-based compressed communication optimization method, system, device, and medium that can reduce communication overhead in tensor-parallel scenarios to address the aforementioned technical problems.
[0007] A compressed communication optimization method based on tensor parallelism, the method comprising: Step 1: In the tensor parallel phase of distributed training, each computing device obtains the activation value of the current layer and stores it in half-precision floating-point format as the input activation value for the current layer communication. Step 2: Determine if the total number of computing devices participating in tensor parallel communication is equal to 1; if yes, proceed to step 9, sending the input activation value as the output activation value to the next network layer; if no, proceed to step 3. Step 3: Perform lossy compression on the input activation value in low-precision floating-point format to obtain the first compressed activation tensor of each computing device; Step 4: Within the tensor parallel group, perform All_to_All set communication on the first compressed activation tensor to exchange the fragments of activation tensors held by each computing device to the corresponding target device; Step 5: Each computing device performs local decompression and reduction summation on the received partial activation tensor fragments to obtain partial summation results; Step 6: Perform lossy compression on the partial summation result again in low-precision floating-point format to obtain the second compressed activation tensor; Step 7: Within the tensor parallel group, perform Allgather collection communication on the second compressed activation tensor to broadcast and aggregate the partial summation results of each computing device to all computing devices in the group; Step 8: Each computing device decompresses the aggregated partial summation results into a half-precision floating-point format to obtain a complete tensor and outputs the activation value after the precision is restored. Step 9: Send the output activation value to the next network layer to continue subsequent calculations.
[0008] On the other hand, a tensor-parallel-based compressed communication optimization device is also provided, comprising: The activation value acquisition and storage module is used in the tensor parallel phase of distributed training, where each computing device acquires the current layer activation value and stores it in half-precision floating-point format as the input activation value for current layer communication. The parallelism determination module is used to determine whether the total number of computing devices participating in tensor parallel communication is equal to 1; if yes, then execute step 9, and send the input activation value as the output activation value to the next network layer; if no, then execute step 3. The first compression module is used to perform lossy compression on the input activation value in low-precision floating-point format to obtain the first compressed activation tensor of each computing device. The All_to_All communication module is used to perform All_to_All set communication on the first compressed active tensor within the tensor parallel group, and to exchange the fragments of the active tensor held by each computing device to the corresponding target device. The decompression, reduction, and summation module is used by each computing device to perform local decompression and reduction summation on the received partial active tensor fragments to obtain partial summation results. The second compression module is used to perform lossy compression on the partial summation result again in low-precision floating-point format to obtain the second compression activation tensor. The Allgather communication module is used to perform Allgather collection communication on the second compressed activation tensor within the tensor parallel group, broadcasting and aggregating the partial summation results of each computing device to all computing devices in the group; The final decompression and precision recovery module is used by each computing device to decompress the aggregated partial summation results to a half-precision floating-point format, obtain the complete tensor, and output the activation value after precision recovery. The inter-layer transfer calculation module is used to send the output activation value to the next network layer for further calculation.
[0009] On another front, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described tensor parallel-based compressed communication optimization method.
[0010] Furthermore, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described tensor-parallel-based compressed communication optimization method.
[0011] Compared with existing technologies, the tensor-parallel-based compressed communication optimization method, system, device, and medium provided by this invention have the following beneficial effects: 1. By performing two lossy compressions on activation values using a low-precision floating-point format, combined with a two-stage communication mode of All_to_All and Allgather, the amount of communication data is reduced by approximately 50% compared to the traditional FP16 full-precision communication scheme. This significantly reduces communication overhead, alleviates the bottleneck of tensor parallel communication, and improves training and inference efficiency. Furthermore, the reduction in communication overhead and the improvement in scalability directly reduce hardware resource consumption, energy consumption, and carbon footprint during the training of large models, fully releasing the potential for hardware acceleration.
[0012] 2. It does not rely on the native low-precision floating-point format AllReduce support of mainstream deep learning frameworks, avoiding the error or program crash problems caused by directly using framework communication primitives. It achieves low-precision compressed communication without modifying the underlying communication primitives of the framework, and has strong compatibility.
[0013] 3. The core operation of cross-device reduction and summation is offloaded to the local computing device for execution, and only part of the summation result is transmitted through communication, which greatly reduces the amount of data and the frequency of interaction in cross-device communication. Especially in multi-node cluster scenarios, it can significantly reduce the latency amplification problem caused by interconnect bandwidth limitations.
[0014] 4. While compressing the communication volume, the design of dynamic scaling quantization and precision recovery after decompression keeps the accuracy of the activation value within an acceptable range, ensuring a balance between training accuracy and scalability. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other related drawings based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart illustrating a tensor-parallel-based compressed communication optimization method in one embodiment. Figure 2 This is a schematic diagram illustrating the compression of the FP16 activation value tensor to FP8 in one embodiment; Figure 3 This is a schematic diagram illustrating the All_to_All data fragment swapping of the compressed FP8 activation value tensor in one embodiment; Figure 4 This is a schematic diagram illustrating how the FP8 activation value is dequantized to FP16 and summed locally in one embodiment; Figure 5 This is a schematic diagram illustrating the Allgathering of FP8 activation values obtained by local summation and subsequent compression in one embodiment; Figure 6 This is a block diagram of a tensor-parallel-based compressed communication optimization device in one embodiment; Figure 7 This is an internal structural diagram of a computer device in one embodiment.
[0017] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0019] It should be noted that in this invention, the use of terms such as "first," "second," etc., is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0020] It is understood that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0022] Example 1 like Figure 1 As shown, this embodiment provides a compressed communication optimization method based on tensor parallelism, including the following steps: Step 1: In the tensor parallel phase of distributed training, each computing device obtains the activation value of the current layer and stores it in half-precision floating-point format as the input activation value for the current layer communication.
[0023] Step 2: Determine if the total number of computing devices participating in tensor parallel communication is equal to 1; if yes, proceed to step 9 and send the input activation value as the output activation value to the next network layer; if no, proceed to step 3.
[0024] Step 3: Perform lossy compression on the input activation value in low-precision floating-point format to obtain the first compressed activation tensor for each computing device.
[0025] Step 4: Within the tensor parallel group, perform All_to_All set communication on the first compressed active tensor to exchange the fragments of the active tensor held by each computing device to the corresponding target device.
[0026] Step 5: Each computing device performs local decompression and reduction summation on the received partial activation tensor fragments to obtain partial summation results.
[0027] Step 6: Perform lossy compression on the partial summation result again in low-precision floating-point format to obtain the second compressed activation tensor.
[0028] Step 7: Within the tensor parallel group, perform Allgather set communication on the second compressed activation tensor to broadcast and aggregate the partial summation results from each computing device to all computing devices within the group.
[0029] Step 8: Each computing device decompresses the aggregated partial summation results into a half-precision floating-point format, obtains the complete tensor, and outputs the activation value after precision is restored.
[0030] Step 9: Send the output activation value to the next network layer to continue subsequent calculations.
[0031] The tensor-parallel-based compressed communication optimization method provided by this invention performs two lossy compressions on activation values using a low-precision floating-point format. Combined with a two-stage communication mode of All_to_All and Allgather, it eliminates the need for the native low-precision floating-point format AllReduce support of mainstream deep learning frameworks. Without altering the underlying communication primitives of the framework, it reduces the amount of communication data by approximately 50% compared to the traditional FP16 full-precision communication scheme, significantly reducing communication overhead and alleviating the bottleneck of tensor-parallel communication. Simultaneously, it offloads the core operation of cross-device reduction and summation to the local computing device, reducing the amount of data and interaction frequency of cross-device communication, fully releasing the hardware acceleration potential. Furthermore, through designs such as dynamic scaling quantization and precision restoration after decompression, it maintains the numerical precision of activation values within an acceptable range, ensuring a balance between training accuracy and scalability, improving the efficiency of distributed training and inference for large models, and reducing hardware resource consumption, energy consumption, and carbon footprint. It is worth noting that in the distributed communication process, the all_to_all_single API provided by the PyTorch distributed communication library (torch.distributed) is used to implement the All-to-All communication primitive, and the all_gather API is used to implement the AllGather communication primitive.
[0032] In step 1, the half-precision floating-point format is mainly FP16 format, the activation value refers to the intermediate output data generated by the neural network layer through forward propagation calculation, which is the core input data supporting the calculation of subsequent network layers; the computing device refers to the hardware processing unit participating in distributed training.
[0033] Specifically, each computing device obtains the current layer activation value through a forward propagation computation process and immediately stores the obtained current layer activation value in FP16 format as the input activation value for current layer communication. This ensures the uniformity and standardization of data storage and provides a standardized data foundation for subsequent compression, communication, and other operations. The computing devices are GPUs with native FP8 computing support, such as H800GPU or H100GPU.
[0034] It is understandable that this step stores the activation value in FP16 format, which is compatible with the computing interfaces of mainstream deep learning frameworks, avoids processing anomalies caused by data format incompatibility, and provides a smooth connection for subsequent conversion between FP16 and low-precision floating-point formats, ensuring the continuity of the entire data processing flow.
[0035] In step 2, the total number of computing devices participating in tensor parallel communication is also called tensor parallelism (world_size), which refers to the size of the computing device cluster participating in this tensor parallel distributed training and directly determines whether cross-device data communication is required.
[0036] Specifically, the tensor parallelism parameter can be directly obtained through the communication interface of the distributed training framework, and then it can be determined whether the parameter is equal to 1. If the result is yes, it indicates that only a single device participates in training and there is no need for cross-device data interaction. The subsequent compressed communication process can be skipped directly to simplify the processing logic. If the result is no, it indicates that it is a multi-device distributed training scenario, and cross-device interaction overhead needs to be reduced through compressed communication optimization. Therefore, the compressed communication process in step 3 is entered.
[0037] It is understandable that this step uses scenario-based branching to achieve differentiated processing for single-device and multi-device training scenarios. This avoids unnecessary compression communication operations in single-device scenarios, improves single-device training efficiency, and accurately adapts to the communication optimization needs of multi-device scenarios, ensuring the versatility of the solution.
[0038] In step 3, the low-precision floating-point format is mainly FP8 format, including two sub-formats: E4M3 and E5M2. Lossy compression refers to a processing method that reduces data storage and transmission volume by means of quantization while allowing a small loss of precision. In this embodiment, it is specifically the conversion process from FP16 format to FP8 format.
[0039] Specifically, the input activation value is the FP16 activation tensor. For example... Figure 2 As shown, taking the FP16 activation tensor of computing device A (rankA) as an example, the FP16 activation tensor is divided into four fragments: a1_fp16, a2_fp16, a3_fp16, and a4_fp16. Lossy compression is performed using either a quantization method based on maximum absolute value dynamic scaling or a native FP8 tensor quantization method based on E4M3 / E5M2 format. Each FP16 fragment corresponds to FP8 format fragments such as a1_fp8, a2_fp8, a3_fp8, and a4_fp8. These FP8 format fragments constitute the first compressed activation tensor. Simultaneously, a scaling factor s corresponding to the first compressed activation tensor is generated. The scaling factor s is used for precision recovery during the decompression process in subsequent step 5.
[0040] It is understandable that this step fully utilizes the native FP8 support provided by modern GPUs such as H800 / H100 during the compression process to compress the FP16 format activation value to the FP8 format, reducing the data volume from 2 bytes / element to 1 byte / element, theoretically achieving a reduction of about 50% in communication volume. This reduces data transmission overhead without adding extra processing latency. At the same time, it relies on the GPU's Tensor Core computing units to achieve efficient quantization processing, avoiding performance loss caused by complex software processing.
[0041] In step 4, All_to_All set communication is a set communication operation in distributed computing, which refers to each computing device in the tensor parallel group exchanging local data fragments according to the preset tensor parallel partitioning dimension to other corresponding target devices in the group, thereby realizing the distributed reorganization of data; the tensor parallel group refers to a logical communication cluster composed of computing devices participating in the same tensor parallel task.
[0042] Specifically, such as Figure 3 As shown, taking four computing devices (rankA~rankD) as an example, before the All_to_All operation is executed, each computing device holds a locally generated FP8 format compressed activation tensor fragment, such as... Figure 3 As shown on the left, the compressed activation tensor fragments of different computing devices are distinguished by different colors.
[0043] During operation, each computing device swaps its portion of the active tensor fragments to the corresponding target device according to the tensor parallel partitioning dimension; after the swap is completed, as follows: Figure 3 As shown on the right, rankA holds the first column of fragments (a1, b1, c1, d1) of all computing devices, rankB holds the second column of fragments (a2, b2, c2, d2) of all computing devices, rankC holds the third column of fragments (a3, b3, c3, d3) of all computing devices, and rankD holds the fourth column of fragments (a4, b4, c4, d4) of all computing devices. In this way, each computing device can obtain all fragments of the same output channel.
[0044] It is understandable that this step achieves precise and targeted exchange of activation tensor fragments through All_to_All set communication, gathering the necessary data for subsequent local reduction and summation operations. At the same time, only compressed data in FP8 format is transmitted, which significantly reduces the amount of data in a single communication compared to traditional FP16 full-precision communication, effectively alleviating the communication pressure caused by the physical limitation of interconnect bandwidth between GPUs.
[0045] In step 5, local decompression and reduction summation refer to the combined operation of the computing device first restoring the format of the received cross-device fragments, and then performing the summation operation locally. Specifically, reduction summation is a cross-fragment addition operation on the tensor parallel dimension.
[0046] Specifically, such as Figure 4As shown, taking rankA as an example, after the All_to_All operation, it receives four partial activation tensor fragments (FP8 fragments) a1_fp8, a2_fp8, a3_fp8, and a4_fp8 from different original devices. First, based on the scaling factor s generated in step 3, each partial activation tensor fragment is decompressed to FP16 precision, restoring it to the format a1_fp16, a2_fp16, a3_fp16, and a4_fp16. Then, FP16 precision vector addition is performed locally on the computing device to obtain the partial summation result S1 (S1=a1_fp16+a2_fp16+a3_fp16+a4_fp16) of the output channel that the computing device is responsible for. Other computing devices complete the decompression and summation of the locally received fragments according to the same logic to obtain their respective corresponding partial summation results.
[0047] Understandably, this step offloads most of the reduction and summation operations that would otherwise need to be completed across devices to the local computing device, significantly reducing the amount of data and frequency of subsequent cross-device communication. At the same time, it restores the precision after decompression based on the scaling factor, ensuring the numerical accuracy of some summation results and avoiding excessive precision loss due to decompression.
[0048] In step 6, the second lossy compression refers to performing FP16 to FP8 compression on the partial result after local reduction and summation. The compression logic is consistent with step 3 to ensure the continuity of data processing.
[0049] Specifically, using the same low-precision floating-point format (FP8 format) and lossy compression method as in step 3, the FP16 format partial summation result obtained in step 5 is compressed a second time to generate a second compressed activation tensor. This second compression process also fully utilizes the native FP8 operation support of GPUs such as H800 / H100 to ensure that the compression efficiency remains consistent with the first compression. Furthermore, step 6 generates a scaling factor s corresponding to the second compressed activation tensor, which is used for precision restoration during the subsequent decompression process in step 8.
[0050] It is understandable that this step further reduces the amount of data transmitted in the subsequent Allgather communication process through secondary FP8 compression, while maintaining the uniformity of the compression method, reducing the cost of software and hardware adaptation, relying on hardware acceleration to ensure the efficiency of the compression process, and avoiding secondary compression becoming a performance bottleneck.
[0051] In step 7, Allgather is a type of collection communication operation in distributed computing, which means that each computing device in the tensor parallel group broadcasts its local second compressed activation tensor to all other computing devices in the group, so that each device in the group eventually holds the second compressed activation tensor set of all computing devices.
[0052] Specifically, such as Figure 5 As shown, each computing device broadcasts the partial summation results (S1~S4) obtained by secondary compression in FP8 format to all devices in the group through the Allgather set communication operation; after the communication is completed, rankA, rankB, rankC, and rankD all hold the FP8 format data of the four partial summation results S1, S2, S3, and S4; during this process, the data is always transmitted in FP8 compression format without format rollback, thus maintaining the communication volume advantage.
[0053] It is understandable that this step uses Allgather set communication to achieve global aggregation of partial summation results, providing comprehensive data support for the subsequent generation of the complete activation tensor. At the same time, relying on FP8 compression format transmission, it further saves communication bandwidth compared with traditional FP16 full-precision aggregation, especially in multi-node cluster scenarios, which can effectively reduce communication time caused by interconnection delay.
[0054] In step 8, the final decompression and accuracy restoration refers to the process of converting the partial summation results of the FP8 format obtained from Allgather communication back to the FP16 format and reorganizing them into a complete active tensor. This is a key step to ensure the accuracy and integrity of the output data.
[0055] Specifically, for scenarios where the tensor parallelism (world_size) is greater than 1, each computing device, based on the scaling factor generated in step 6, decompresses all the FP8 format partial summation results obtained by aggregation into FP16 format in fragments one by one, and then obtains the complete FP16 activation tensor that has been reduced and summed in the tensor parallelism dimension through local splicing operation; for scenarios where the tensor parallelism (world_size) is equal to 1, this decompression step is not required, and the FP16 format activation value stored in step 1 is used directly.
[0056] It is understandable that this step restores the original numerical precision of the activation values to the greatest extent possible through decompression operations based on scaling factors, ensuring the accuracy of subsequent network layer calculations. At the same time, it completes the reorganization of the full activation tensor, providing standardized FP16 format input data for the next network layer, avoiding the impact of data format or integrity issues on subsequent calculations.
[0057] In step 9, the complete FP16 format output activation values obtained in step 8 are directly sent to the computation unit of the next network layer through the inter-layer data transmission interface of the neural network framework; if the current stage is forward propagation, input data is provided for the forward computation of the next layer to ensure the continuity of the training process.
[0058] It is understandable that this step enables the smooth transfer of activation values between network layers, ensuring the integrity of the distributed training process. At the same time, the output activation values in FP16 format can be directly adapted to the inter-layer computation logic of mainstream deep learning frameworks without the need for additional data format conversion operations, reducing processing latency and further improving training and inference efficiency.
[0059] It should be understood that, although this embodiment Figure 1 The steps are shown sequentially as indicated by the arrows, but they are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0060] Example 2 Based on the tensor-parallel-based compressed communication optimization method in Example 1, this example discloses a tensor-parallel-based compressed communication optimization device, such as... Figure 6 As shown, the tensor-parallel-based compressed communication optimization device includes: an activation value acquisition and storage module 401, a parallelism determination module 402, a first compression module 403, an All_to_All communication module 404, a decompression, reduction, and summation module 405, a second compression module 406, an Allgather communication module 407, a final decompression and precision recovery module 408, and an inter-layer transfer calculation module 409, wherein: The activation value acquisition and storage module 401 is used in the tensor parallel phase of distributed training, whereby each computing device acquires the current layer activation value and stores it in half-precision floating-point format as the input activation value for current layer communication.
[0061] Parallelism determination module 402 is used to determine whether the total number of computing devices participating in tensor parallel communication is equal to 1; if yes, then execute step 9, and send the input activation value as the output activation value to the next network layer; if no, then execute step 3.
[0062] The first compression module 403 is used to perform lossy compression on the input activation value in low-precision floating-point format to obtain the first compressed activation tensor of each computing device.
[0063] The All-to-All communication module 404 is used to perform All-to-All set communication on the first compressed active tensor within the tensor parallel group, and to exchange the fragments of the active tensor held by each computing device to the corresponding target device.
[0064] The decompression, reduction, and summation module 405 is used by each computing device to perform local decompression and reduction summation on the received partial active tensor fragments to obtain partial summation results.
[0065] The second compression module 406 is used to perform lossy compression on the partial summation result again in low-precision floating-point format to obtain the second compressed activation tensor.
[0066] Allgather communication module 407 is used to perform Allgather collection communication on the second compressed activation tensor within the tensor parallel group, broadcasting and aggregating partial summation results from each computing device to all computing devices within the group.
[0067] The final decompression and precision recovery module 408 is used by each computing device to decompress the aggregated partial summation results to a half-precision floating-point format, obtain a complete tensor, and output the activation value after precision recovery.
[0068] The inter-layer transfer calculation module 409 is used to send the output activation value to the next network layer to continue subsequent calculations.
[0069] In this embodiment, the specific working process and working principle of the activation value acquisition and storage module 401, parallelism judgment module 402, first compression module 403, All_to_All communication module 404, decompression reduction and summation module 405, second compression module 406, Allgather communication module 407, final decompression and precision recovery module 408, and inter-layer transfer calculation module 409 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above unit modules.
[0070] Example 3 like Figure 7 The diagram illustrates a terminal device disclosed in this embodiment, comprising a transmitter, a receiver, a memory, and a processor. The transmitter transmits instructions and data, the receiver receives instructions and data, the memory stores computer-executed instructions, and the processor executes the computer-executed instructions stored in the memory to implement the method described in Embodiment 1 above.
[0071] It is important to note that the aforementioned memory can be either standalone or integrated with the processor. When the memory is set up independently, the terminal device also includes a bus for connecting the memory and the processor.
[0072] Example 4 This embodiment discloses a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method in Embodiment 1 above.
[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0074] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0075] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. A compressed communication optimization method based on tensor parallelism, characterized in that, The method includes: Step 1: In the tensor parallel phase of distributed training, each computing device obtains the activation value of the current layer and stores it in half-precision floating-point format as the input activation value for the current layer communication. Step 2: Determine if the total number of computing devices participating in tensor parallel communication is equal to 1; if yes, proceed to step 9, sending the input activation value as the output activation value to the next network layer; if no, proceed to step 3. Step 3: Perform lossy compression on the input activation value in low-precision floating-point format to obtain the first compressed activation tensor of each computing device; Step 4: Within the tensor parallel group, perform All_to_All set communication on the first compressed activation tensor to exchange the fragments of activation tensors held by each computing device to the corresponding target device; Step 5: Each computing device performs local decompression and reduction summation on the received partial activation tensor fragments to obtain partial summation results; Step 6: Perform lossy compression on the partial summation result again in low-precision floating-point format to obtain the second compressed activation tensor; Step 7: Within the tensor parallel group, perform Allgather collection communication on the second compressed activation tensor to broadcast and aggregate the partial summation results of each computing device to all computing devices in the group; Step 8: Each computing device decompresses the aggregated partial summation results into a half-precision floating-point format to obtain a complete tensor and outputs the activation value after the precision is restored. Step 9: Send the output activation value to the next network layer to continue subsequent calculations.
2. The compressed communication optimization method based on tensor parallelism according to claim 1, characterized in that, In step 1, each computing device obtains the current layer activation value through forward propagation calculation.
3. The compressed communication optimization method based on tensor parallelism according to claim 1, characterized in that, The lossy compression employs a quantization method based on dynamic scaling of maximum absolute value, or a native FP8-frame quantization method based on E4M3 / E5M2 format.
4. The compressed communication optimization method based on tensor parallelism according to any one of claims 1 to 3, characterized in that, In step 5, the specific process of local decompression and reduction summation is as follows: First, the received partial activation tensor is decompressed into half-precision floating-point format. Then, the cross-fragment summation operation on the parallel dimension of the tensor is performed locally on the computing device to obtain the partial summation result in half-precision floating-point format.
5. The compressed communication optimization method based on tensor parallelism according to any one of claims 1 to 3, characterized in that, The computing device is a GPU with native FP8 computing support.
6. The compressed communication optimization method based on tensor parallelism according to any one of claims 1 to 3, characterized in that, During the lossy compression process in steps 3 and 6, scaling factors corresponding to the first compression activation tensor and the second compression activation tensor are generated; During the decompression process in steps 5 and 8, the accuracy of the activation value is restored based on the scaling factor.
7. The compressed communication optimization method based on tensor parallelism according to any one of claims 1 to 3, characterized in that, The half-precision floating-point number format is FP16 format, and the low-precision floating-point number format is FP8 format.
8. A compressed communication optimization device based on tensor parallelism, characterized in that, The device includes: The activation value acquisition and storage module is used in the tensor parallel phase of distributed training, where each computing device acquires the current layer activation value and stores it in half-precision floating-point format as the input activation value for current layer communication. The parallelism determination module is used to determine whether the total number of computing devices participating in tensor parallel communication is equal to 1; if yes, then execute step 9, and send the input activation value as the output activation value to the next network layer; if no, then execute step 3. The first compression module is used to perform lossy compression on the input activation value in low-precision floating-point format to obtain the first compressed activation tensor of each computing device. The All_to_All communication module is used to perform All_to_All set communication on the first compressed active tensor within the tensor parallel group, and to exchange the fragments of the active tensor held by each computing device to the corresponding target device. The decompression, reduction, and summation module is used by each computing device to perform local decompression and reduction summation on the received partial active tensor fragments to obtain partial summation results. The second compression module is used to perform lossy compression on the partial summation result again in low-precision floating-point format to obtain the second compression activation tensor. The Allgather communication module is used to perform Allgather collection communication on the second compressed activation tensor within the tensor parallel group, broadcasting and aggregating the partial summation results of each computing device to all computing devices in the group; The final decompression and precision recovery module is used by each computing device to decompress the aggregated partial summation results to a half-precision floating-point format, obtain the complete tensor, and output the activation value after precision recovery. The inter-layer transfer calculation module is used to send the output activation value to the next network layer for further calculation.
9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the tensor parallel-based compressed communication optimization method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the tensor-parallel-based compressed communication optimization method according to any one of claims 1 to 7.