A method and apparatus for ensemble communication compression for distributed training of large language models

Abstract

This invention proposes a ensemble communication compression method for distributed training of large language models, comprising: constructing a unified compression acceleration programming model and defining a set of core interfaces for compression acceleration; configuring a custom compression algorithm for the target training data type based on the unified compression acceleration programming model, and compiling the custom compression algorithm into a dynamically loadable link library file; submitting the environment variables configured for the custom compression algorithm through the configuration registration interface, and initializing the compression module of the compression-accelerated ensemble communication library through the environment variables; executing 3D parallel training of the large language model, and during the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations according to the preset mapping relationship between communication primitives and compression algorithms, thereby completing data transmission across GPU nodes. This invention improves communication efficiency.

Description

Technical Field

[0001] This invention relates to the field of communication compression technology, and in particular to a method, apparatus, and storage medium for ensemble communication compression for distributed training of large language models. Background Technology

[0002] Large Language Model (LLM) training relies on parallelization to scale across GPU clusters, exchanging data between processes via communication primitives (collective communication). As model parameters grow to billions, the amount of communication increases dramatically, and inefficient collective communication becomes a well-known bottleneck restricting training efficiency. Large model training employs 3D parallelism to distribute models and batches across GPU clusters for scalable efficiency, but this approach requires complex communication patterns, incurring significant overhead. For example, training a 13B GPT-2 model with 3D parallelism requires 12,496 MB of ReduceScatter communication and 6,248 MB of AllGather communication per iteration to synchronize gradients and weights for Data Parallelism (DP); 5,448 AllReduce communications (40 MB each) to aggregate partial activations for Tensor Parallelism (TP); and 256 P2P operations to transfer activations for Pipeline Parallelism (PP). At large scale, transferring such a large amount of data accounts for more than 50% of the total training time. In the 3D parallel architecture of TP, DP, and PP, data exchange in each dimension has different characteristics: DP is used to pass weights and gradients, while TP and PP are used to pass activation values. Gradients often contain outliers, weights have a wide dynamic range, and activation values ​​may contain extreme outliers. This heterogeneity makes it impossible for a single general compression method to be applicable to all dimensions.

[0003] Most current methods are designed for specific data types (e.g., gradients) and are tightly coupled to specific frameworks, limiting simultaneous deployment, reducing user transparency, and restricting broader applicability. Implementing compression within these frameworks is complex, lacks portability, and offers little user transparency. The most popular compression methods are implemented within training frameworks, requiring modifications to multiple modules and user training scripts, and relying on various programming languages. This leads to complex deployment processes, limited cross-version portability, and low user transparency. For example, ... Figure 1 As shown, Figure 1The code snippet illustrates the deployment of the SDP4Bit compression algorithm in Megatron-LM in the prior art. Different background colors represent encoding at different layers. Implementing SDP4Bit in Megatron-LM requires redirecting weight and gradient synchronization calls (lines 1-13), manually building the compression-communication pipeline (lines 16-26), and interfaced compression operations with Python (lines 28-35). Even achieving data-parallel compression via centralized communication requires significant engineering work—over 1000 lines of code encompassing 53 Python and C / C++ functions—and extending to tensor and pipeline parallel support using distributed communication adds further complexity. To overcome these limitations, a lightweight, widely compatible, and seamlessly usable solution is needed to integrate compression into training communication while maintaining minimal overhead and improving the efficiency of parallel communication across GPU clusters. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a method, apparatus, and storage medium for ensemble communication compression for distributed training of large language models. This method separates compression from communication and standardizes the integration process using a unified compression acceleration programming model. It supports simple and rapid integration and easy configuration of different customized compression algorithms and improves the efficiency of parallel communication across GPU clusters.

[0005] This invention provides a ensemble communication compression method for distributed training of large language models, applied to GPU cluster systems, comprising:

[0006] Construct a unified compression acceleration programming model and define a set of core interfaces for compression acceleration. The core interfaces include at least a compression interface, a decompression interface, and a configuration registration interface.

[0007] Based on the unified compression acceleration programming model, a custom compression algorithm is configured for the target training data type, and the custom compression algorithm is compiled into a dynamically loadable link library file.

[0008] The environment variables configured for the custom compression algorithm are submitted through the configuration registration interface, and the compression module of the compression-accelerated collection communication library is initialized through the environment variables;

[0009] The system performs 3D parallel training of a large language model. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, thus completing the data transmission across GPU nodes.

[0010] In one embodiment of the present invention, the core interface set further includes a reduction-decompression fusion interface and a reduction-decompression-recompression fusion interface, which are used to support the fusion execution of reduction operations with compression and decompression operations in distributed training.

[0011] In one embodiment of the present invention, the target training data type includes gradient data, weight data or activation value data in the training process of a large language model, and the custom compression algorithm is a customized compression algorithm adapted to the corresponding data distribution characteristics.

[0012] In one embodiment of the present invention, the environmental variables include:

[0013] The first environment variable used to enable the compression module;

[0014] The second environment variable used to configure the compression algorithm corresponding to different set communication primitives;

[0015] A third environment variable used to configure compression strategies for inter-node communication;

[0016] The fourth environment variable used to specify the compressed sensing communication algorithm.

[0017] In one embodiment of the present invention, the 3D parallel training includes three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The correspondence between the communication primitives and the parallel dimensions is as follows:

[0018] The data parallel dimension corresponds to the AllGather and ReduceScatter communication primitives;

[0019] Tensor parallelism corresponds to AllReduce communication primitives;

[0020] The pipeline parallel dimension corresponds to the Send / Recv point-to-point communication primitives.

[0021] In one embodiment of the present invention, before calling the corresponding custom compression algorithm to compress or decompress the communication data during the execution of the cluster communication operation, the method further includes: collecting cluster communication performance characteristics through an automatic tuning engine, determining the optimal compression-aware communication algorithm path, and then calling the corresponding custom compression algorithm to perform data processing.

[0022] In one embodiment of the present invention, the size of the data block partition is dynamically optimized by an automatic overlap mechanism at runtime, and the compression or decompression process and data transmission operation are executed in parallel.

[0023] In one embodiment of the present invention, the environment variables are specified through an external configuration file, and the compression-accelerated collection communication library automatically parses the configuration file and loads the corresponding custom compression algorithm during the startup phase.

[0024] Another aspect of the present invention provides a ensemble communication compression device for distributed training of large language models, deployed in a GPU cluster system, comprising:

[0025] A unified model building module is used to build a unified compression acceleration programming model and define a set of core interfaces for compression acceleration. The core interfaces include at least a compression interface, a decompression interface, and a configuration registration interface.

[0026] The compression algorithm configuration module is used to configure a custom compression algorithm for the target training data type based on the unified compression acceleration programming model, and to compile the custom compression algorithm into a dynamically loadable link library file.

[0027] An initialization configuration module is used to submit environment variables configured by a custom compression algorithm through the configuration registration interface, and to initialize the compression module of the compression-accelerated collection communication library through the environment variables;

[0028] The communication compression execution module is used to perform 3D parallel training of large language models. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, to complete the data transmission across GPU nodes.

[0029] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the ensemble communication compression method for distributed training of large language models.

[0030] In another aspect, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the ensemble communication compression method for distributed training of large language models.

[0031] As can be seen from the above solutions, the advantages of the present invention are:

[0032] This invention provides a method for ensemble communication compression for distributed training of large language models. It constructs a unified compression acceleration programming model, defining a core set of interfaces for compression acceleration. Based on this model, it configures custom compression algorithms for the target training data type and compiles these algorithms into dynamically loadable link library files. It submits environment variables configured for the custom compression algorithms through a registration interface and initializes the compression module of the compression-accelerated ensemble communication library using these environment variables. During 3D parallel training, the compression-accelerated ensemble communication library, based on a preset mapping relationship between communication primitives and compression algorithms, calls the corresponding custom compression algorithm to compress or decompress communication data during ensemble communication operations, completing data transmission across GPU nodes. This method constructs a standardized unified compression acceleration programming model, separating the compression module from communication. Furthermore, it supports simple and rapid integration and easy configuration of different customized compression algorithms, easily integrating specific compression protocols for various data types. This method reduces the overhead of distributed training of large language models in GPU clusters, improves GPU parallel communication efficiency, and maintains high-performance operation. Attached Figure Description

[0033] Figure 1 This shows a code segment demonstrating the deployment of the SDP4Bit compression algorithm in Megatron-LM in the prior art;

[0034] Figure 2 A schematic diagram of the overall process of a ensemble communication compression method for distributed training of large language models provided in an embodiment of the present invention is shown.

[0035] Figure 3 The schematic diagram of the layered design of the compression-accelerated collection communication library COCCL is shown.

[0036] Figure 4 This invention demonstrates a code segment that implements the SDP4Bit compression algorithm in COCCL.

[0037] Figure 5 A schematic diagram of the overall structure of a ensemble communication compression device for distributed training of large language models provided in an embodiment of the present invention is shown.

[0038] The attached figures are labeled as follows:

[0039] 300: A collection communication compression device for distributed training of large language models;

[0040] 310: Unified Model Building Module;

[0041] 320: Compression algorithm configuration module;

[0042] 330: Initialize configuration module;

[0043] 340: Communication compression execution module. Detailed Implementation

[0044] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0045] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0046] See Figure 2 As shown, Figure 2 The diagram illustrates the overall flow of a ensemble communication compression method for distributed training of large language models according to an embodiment of the present invention.

[0047] A ensemble communication compression method for distributed training of large language models, applied to a GPU cluster system, includes the following steps:

[0048] Step S1: Construct a unified compression acceleration programming model and define a set of core interfaces for compression acceleration. The set of core interfaces includes at least a compression interface, a decompression interface, and a configuration registration interface.

[0049] Step S2: Based on the unified compression acceleration programming model, configure a custom compression algorithm for the target training data type, and compile the custom compression algorithm into a dynamically loadable link library file.

[0050] Step S3: Submit the configuration parameters of the custom compression algorithm through the configuration registration interface, and initialize the compression module of the compression acceleration collection communication library through environment variables.

[0051] Step S4: Perform 3D parallel training of the large language model. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, to complete the data transmission across GPU nodes.

[0052] In one specific implementation, in step S1, a unified compression acceleration programming model (UCPM) is constructed, defining a core interface set for compression acceleration. This core interface set includes at least a compression interface, a decompression interface, a configuration registration interface, and reduction-decompression-recompression fusion interfaces. These interfaces support the fusion of reduction operations with compression and decompression operations during distributed training. The compression interface `compress()` and the decompression interface `decompress()` provide basic compression and decompression functions. The `compress()` interface receives an input buffer and metadata (e.g., data type, length, grouping, etc.) and returns the compressed buffer and related metadata. The decompression interface `decompress()` works symmetrically, following the same input / output flow. To integrate a custom compression algorithm into the compression acceleration collection communication library COCCL, the user needs to implement these two interfaces.

[0053] Some compressors may combine reduction operations (crucial for some set computations) with compression operations to improve performance. The reduction-decompression fusion interface `decomp_reduce()` and the reduction-decompression-recompression fusion interface `decomp_reduce_recomp()` support this fusion operation. These interfaces are optional; if omitted, the compression interface `compress()` and the decompression interface `decompress()` are invoked by default to perform the corresponding operations.

[0054] In addition, users must register configurable parameters through the configuration registration interface reg_compress_config(). reg_compress_config() takes a configuration file path as input and returns a pointer to CPU memory used to store the configuration information.

[0055] The aforementioned set of core interfaces allows developers to integrate custom compression algorithms at minimal cost.

[0056] The Unified Compression Acceleration Programming Model (UCPM) enables the integration of compression protocols into the collection communication library without requiring changes to the upper-level framework, making it transparent to users. Furthermore, the unified compression programming model allows for the easy integration and configuration of customized compression algorithms for different data types.

[0057] In this embodiment, to provide a unified and portable interface, a Unified Compression Acceleration Programming Model (UCPM) is introduced. This model enables lightweight, widely compatible, and seamless compression integration by separating compression from complex communication logic and higher-level frameworks. It directly integrates all compression methods into the communication library, rather than the framework itself. The UCPM model separates the compression module from the communication pipeline and standardizes the integration process. This design greatly simplifies the integration process of user-defined compressors and provides high configurability. It allows users to deploy efficient compression algorithms across all dimensions of 3D parallelism with minimal effort, requiring only a few environment variables.

[0058] In step S2, based on the Unified Compression Acceleration Programming Model (UCPM), a custom compression algorithm is configured for the target training data type, and the custom compression algorithm is compiled into a dynamically loadable link library file. In a specific implementation, the target training data type includes gradient data, weight data, or activation value data during the training of a large language model, and the custom compression algorithm is a customized compression algorithm adapted to the corresponding data distribution characteristics. The dynamic link library is a *.so format file under the Linux system, and the implementation code of the custom compression algorithm does not exceed 100 lines of C / C++ code.

[0059] In step S3, the environment variables configured for the custom compression algorithm are submitted through the configuration registration interface, and the compression module of the compression-accelerated collective communication library is initialized using these environment variables. Environment variables can be specified through an external configuration file. The compression-accelerated collective communication library automatically parses the configuration file and loads the corresponding custom compression algorithm during startup. By configuring environment variables, the compression-accelerated collective communication library COCCL automatically initializes the library and parses the user configuration during startup. The compression-accelerated collective communication library is a COCCL library built upon the NVIDIA Collective Communications Library (NCCL), integrating the UCPM model. By setting environment variables, the NCCL dependency path of the PyTorch framework is pointed to the COCCL library, achieving a seamless replacement of the native NCCL. The compression-accelerated collective communication library COCCL enables efficient deployment of customized communication compression for scalable distributed training, separating compression and communication, facilitating seamless integration of custom algorithms and flexible configurations.

[0060] Figure 3This paper illustrates the layered design logic of the compression-accelerated collective communication library (COCCL). The core principle is to achieve deep collaboration between compression and communication through a middle layer while maintaining compatibility with upper-layer applications. The topmost user application layer supports mainstream large language model training frameworks such as PyTorch and Megatron. The user interface layer fully complies with the NCCL standard API, ensuring that existing training code can be directly migrated and used without modification, achieving complete transparency for users. The middle COCCL layer is divided into two main functional modules: the Unified Compression Acceleration Programming Model (UCPM) provides a standardized interface for integrating compression algorithms and supports users injecting custom compression kernels via dynamic link libraries; the Comp-Comm Co-optimization module includes compression-aware communication algorithms and a runtime auto-overlapping mechanism. The former selects the optimal communication path, while the latter implements pipelined parallelism for compression computation and data transmission, maximizing hardware utilization. Furthermore, this architecture achieves hardware compatibility through an abstract device interface layer, supporting multiple high-speed communication protocols such as PCIe, RDMA, and RNIC, ensuring optimal performance across different GPU cluster topologies. User-defined compression cores, after being integrated into the system via a unified compression programming model, are automatically incorporated into the compression-communication co-optimization framework and dynamically invoked by the system during communication. This design decouples compression functionality from the training framework, avoiding intrusive modifications to the framework's source code and allowing for flexible configuration of corresponding compression algorithms to meet different communication needs in 3D parallel training (such as data-parallel AllGather / ReduceScatter and tensor-parallel AllReduce), ultimately achieving a systematic improvement in the communication efficiency of distributed training of large language models within GPU clusters.

[0061] Table 1 shows the core interface set (API) and environment variables of the UCPM model abstraction. The first environment variable, `COCCL_ENABLE_COMPRESS`, is set to initialize the compression module of the compression-accelerated collection communication library (COCCL). This first environment variable is used to enable the compression module. The second environment variable, `COCCL_PRIM*_COMPRESSOR` (e.g., `COCCL_ALLREDUCE_COMPRESSOR`), is set to configure the compression algorithm corresponding to different collection communication primitives. The third environment variable, `COCCL_PRIM*_INTER_COMPRESSOR`, specifies the compression settings for inter-node communication. This third environment variable is used to configure the compression strategy for inter-node communication. The fourth environment variable, `COCCL_PRIM*_ALGO`, is set to configure the compression-aware communication algorithm for a specific primitive. This fourth environment variable is used to specify the compression-aware communication algorithm. The environment variables include the first, second, third, and fourth environment variables mentioned above.

[0062] Table 1 shows the core interface set (API) and environment variables of the UCPM model abstraction.

[0063]

[0064] Figure 4 This section displays the code snippet demonstrating the deployment of the SDP4Bit compression algorithm in COCCL. The blue area represents C++ code, and the gray area represents environment variables in a bash script. Line 1 in the blue area represents a user-defined custom compression configuration; lines 2-7 indicate that these configurations are registered and retrieved in COCCL. Lines 8-13 show the basic compression and decompression functions required by the user to implement the compression-accelerated collection communication library COCCL. Furthermore, SDP4Bit includes quantization and compression fusion cores, which connect to the compression-accelerated collection communication library COCCL via an optional programming interface, shown in lines 15-20. All of this can be achieved by writing approximately 100 lines of C / C++ code in a single file.

[0065] In step S4, large language model 3D parallel training is performed. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, to complete the data transmission across GPU nodes.

[0066] The 3D parallel training includes three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The correspondence between the communication primitives and the parallel dimensions is as follows: the data parallelism dimension corresponds to the AllGather and ReduceScatter communication primitives; the tensor parallelism dimension corresponds to the AllReduce communication primitive; and the pipeline parallelism dimension corresponds to the Send / Recv point-to-point communication primitive. AllGather collects data from multiple nodes onto multiple nodes, equivalent to multiple Gather operations. ReduceScatter combines Reduce operations with Scatter data distribution operations, forming the communication infrastructure for data parallelism. First, on all participating GPU nodes, a specified reduction operation is performed on data blocks located at the same position or index. Then, the complete reduced result is split according to the dimension, and different data blocks are distributed to each node. Finally, each node receives only a portion of the entire reduced result, not the whole. AllReduce typically consists of ReduceScatter and AllGather operations.

[0067] Further reference Figure 3 As shown in the gray area, PyTorch requires dynamic library linking to support NCCL, while COCCL can be supported by setting NCCL_HOME to the COCCL library path. See [link to documentation]. Figure 4 The first line of the gray area. In COCCL, the compression algorithm is enabled via the first environment variable COCCL_COMPRESSORS, see lines 2-3. In 3D parallel computing, different dimensions use different communication primitives: AllGather and ReduceScatter are used for data parallelism (DP); AllReduce is used for tensor parallelism (TP); and Send / Recv is used for point parallelism (PP). Therefore, setting the compression algorithm for these primitives enables comprehensive and systematic compression of each dimension of 3D parallel computing, see lines 4-7, thereby improving scalability and efficiency. The configuration file specifying the compression parameters is provided via COCCL_COMPRESSOR_CONFIG, see line 8.

[0068] In addition, in one embodiment, before calling the corresponding custom compression algorithm to compress or decompress the communication data during the execution of the cluster communication operation in step S4, the method further includes: collecting cluster communication performance characteristics through an automatic tuning engine, determining the optimal compression-aware communication algorithm path, and then calling the corresponding custom compression algorithm to perform data processing.

[0069] Furthermore, when performing collection communication operations and invoking the corresponding custom compression algorithm to compress or decompress the communication data, a runtime automatic overlap mechanism dynamically optimizes the data block size, allowing compression or decompression to be executed in parallel with data transmission operations. The runtime automatic overlap mechanism automatically optimizes the data block size to simultaneously perform compression and communication pipeline operations.

[0070] In summary, the ensemble communication compression method for distributed training of large language models provided by this invention constructs a unified compression acceleration programming model that supports the integration of compression protocols into the ensemble communication library. It provides a unified and portable interface, achieving lightweight, widely compatible, and seamless compression integration by separating compression from complex communication logic and higher-level frameworks. The UCPM model separates the compression module from the communication pipeline and standardizes the integration process. This design greatly simplifies the integration process of user-defined compressors and provides high configurability. It enables users to deploy efficient compression algorithms across all dimensions of 3D parallelism with minimal effort, requiring only a few environment variables. Furthermore, COCCL, a compression-accelerated collection communication library integrating a unified compression acceleration programming model, is constructed. This enables efficient deployment of customized communication compression for scalable distributed training, separating compression from communication. COCCL allows any PyTorch-based framework to transparently use user-defined compression algorithms to accelerate communication through the NCCL API without modifying the framework's source code. It seamlessly supports the easy integration and configuration of customized compression protocols, truly advancing the purpose of collection communication libraries and allowing users to easily integrate compression algorithms customized for specific data types. Moreover, this method avoids the problem of reduced communication accuracy caused by compression, reduces the overhead of distributed training of large language models in GPU clusters, improves GPU parallel communication efficiency, and maintains high-performance operation.

[0071] In one embodiment, a ensemble communication compression device for distributed training of large language models is further provided. This embodiment is a device embodiment corresponding to the above method embodiment, such as... Figure 5 As shown, Figure 5 A schematic diagram of a ensemble communication compression device for distributed training of large language models according to an embodiment of the present invention is shown. This device embodiment can be implemented in conjunction with the above-described method embodiment. The relevant technical details mentioned in the above method embodiment remain valid in this device embodiment, and will not be repeated here to avoid repetition.

[0072] A ensemble communication compression device 300 for distributed training of large language models, deployed in a GPU cluster system, includes:

[0073] The unified model building module 310 is used to build a unified compression acceleration programming model and define a set of core interfaces for compression acceleration. The core interfaces include at least a compression interface, a decompression interface, and a configuration registration interface.

[0074] The compression algorithm configuration module 320 is used to configure a custom compression algorithm for the target training data type based on the unified compression acceleration programming model, and to compile the custom compression algorithm into a dynamically loadable link library file.

[0075] The initialization configuration module 330 is used to submit the environment variables configured by the custom compression algorithm through the configuration registration interface, and to initialize the compression module of the compression acceleration collection communication library through the environment variables.

[0076] The communication compression execution module 340 is used to perform parallel training of large language models in 3D. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, to complete the data transmission across GPU nodes.

[0077] It should be noted that the relevant technical details mentioned in the above embodiments are still valid in this device embodiment, and will not be repeated here to avoid repetition.

[0078] This invention also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the above-described ensemble communication compression method for distributed training of large language models, and achieve the same technical effect.

[0079] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described ensemble communication compression method for distributed training of large language models, and achieves the same technical effect.

[0080] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A ensemble communication compression method for distributed training of large language models, applied to a GPU cluster system, characterized in that, include: Construct a unified compression acceleration programming model and define a set of core interfaces for compression acceleration. The core interfaces include at least a compression interface, a decompression interface, and a configuration registration interface. Based on the unified compression acceleration programming model, a custom compression algorithm is configured for the target training data type, and the custom compression algorithm is compiled into a dynamically loadable link library file. The environment variables configured for the custom compression algorithm are submitted through the configuration registration interface, and the compression module of the compression-accelerated collection communication library is initialized through the environment variables; The system performs 3D parallel training of a large language model. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, thus completing the data transmission across GPU nodes.

2. The method according to claim 1, characterized in that, The core interface set also includes a reduction-decompression fusion interface and a reduction-decompression-recompression fusion interface, which are used to support the fusion execution of reduction operations with compression and decompression operations in distributed training.

3. The method according to claim 1, characterized in that, The target training data types include gradient data, weight data, or activation value data during the training process of a large language model, and the custom compression algorithm is a customized compression algorithm adapted to the corresponding data distribution characteristics.

4. The method according to claim 1, characterized in that, The environment variables include: The first environment variable used to enable the compression module; The second environment variable used to configure the compression algorithm corresponding to different set communication primitives; A third environment variable used to configure compression strategies for inter-node communication; The fourth environment variable used to specify the compressed sensing communication algorithm.

5. The method according to claim 1, characterized in that, The 3D parallel training includes three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The correspondence between the communication primitives and the parallel dimensions is as follows: The data parallel dimension corresponds to the AllGather and ReduceScatter communication primitives; Tensor parallelism corresponds to the AllReduce communication primitive; The pipeline parallel dimension corresponds to the Send / Recv point-to-point communication primitives.

6. The method according to claim 1, characterized in that, Before calling the corresponding custom compression algorithm to compress or decompress the communication data during the execution of the cluster communication operation, it also includes: collecting cluster communication performance characteristics through the automatic tuning engine, determining the optimal compression-aware communication algorithm path, and then calling the corresponding custom compression algorithm to perform data processing.

7. The method according to claim 1, characterized in that, The data block size is dynamically optimized through a runtime automatic overlap mechanism, and compression or decompression processing is performed in parallel with data transmission operations.

8. The method according to claim 1, characterized in that, The environment variables are specified through an external configuration file. The compression-accelerated collection communication library automatically parses the configuration file and loads the corresponding custom compression algorithm during the startup phase.

9. A ensemble communication compression device for distributed training of large language models, deployed in a GPU cluster system, characterized in that, include: A unified model building module is used to build a unified compression acceleration programming model and define a set of core interfaces for compression acceleration. The core interfaces include at least a compression interface, a decompression interface, and a configuration registration interface. The compression algorithm configuration module is used to configure a custom compression algorithm for the target training data type based on the unified compression acceleration programming model, and to compile the custom compression algorithm into a dynamically loadable link library file. An initialization configuration module is used to submit environment variables configured by a custom compression algorithm through the configuration registration interface, and to initialize the compression module of the compression-accelerated collection communication library through the environment variables; The communication compression execution module is used to perform 3D parallel training of large language models. During the training process, the compression-accelerated ensemble communication library calls the corresponding custom compression algorithm to compress or decompress the communication data when performing ensemble communication operations, based on the preset mapping relationship between communication primitives and compression algorithms, to complete the data transmission across GPU nodes.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images