Communication compression methods, devices, and electronic equipment for parallel model training

Abstract

This invention relates to the field of machine learning, and provides a communication compression method, apparatus, and electronic device for parallel model training. The communication compression method for parallel model training includes: during the forward propagation phase, performing model layer calculations based on allocated tensors to obtain activation values ​​for at least one set of channels; performing a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​from each set to obtain a first activation value for each set; performing group quantization on the unretained activation values ​​in each set to obtain a second activation value for each set; and communicating the activation values ​​based on the first and second activation values. This method, through a combination of TopK quantization and naive group quantization, achieves fast and accurate activation value compression, improving the efficiency of parallel training.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to a communication compression method, apparatus, and electronic device for parallel model training. Background Technology

[0002] In the field of deep learning, researchers have adopted various parallel strategies to accelerate the training process of large-scale models, such as Transformer models, including tensor parallelism, pipeline parallelism, and data parallelism. These parallel strategies significantly improve training efficiency by distributing computational tasks across multiple processing units, such as GPUs.

[0003] However, each parallel strategy introduces different communication overheads: (1) Tensor parallelism divides matrix multiplication into rows and columns, requiring AllReduce aggregate communication at the end of the forward and backward computations of attention blocks and feedforward neural network blocks. (2) Pipeline parallelism divides the model into different parallel stages according to layers and places them on different computing cards, requiring point-to-point send / receive communication between groups during forward and backward computations. (3) Data parallelism copies the model multiple times, with each copy training different data blocks. When updating model parameters, the gradients of different copies are synchronized through AllReduce operations. Data parallelism has variants of ZeRO-1 / 2 / 3, which reduce the GPU memory usage of each copy, but increase the communication overhead. The aggregate communication introduced by tensor parallelism and data parallelism introduces a large amount of communication overhead when the size of the parallel group is increased, making it impossible to perform distributed training on clusters without high-speed networks.

[0004] For example, the Llama-13B model can be trained on a cluster of 32 4090 GPUs using tensor parallelism of size 2, pipeline parallelism of size 8, and data parallelism of size 2. However, because the 4090 does not support high-speed interconnects such as NVLink and PCI-E P2P, the model floating-point utilization (MFU) can only reach 25%, meaning that the 4090 does not participate in matrix multiplication operations for about three-quarters of the time.

[0005] To reduce communication volume, many studies have focused on communication compression for parallel training. However, direct quantization compression can lead to large errors, while using more complex compression methods can result in significant computational overhead during compression and decompression, which can actually slow down training. Summary of the Invention

[0006] This invention provides a communication compression method, apparatus, and electronic device for parallel model training, which solves the defects of large quantization errors and large computational overhead in compression and decompression in the prior art. It achieves simple, effective, and hardware-friendly communication compression, thereby improving the efficiency of parallel model training.

[0007] This invention provides a communication compression method for parallel model training, comprising: in the forward propagation phase, performing model layer calculations based on allocated tensors to obtain activation values ​​for at least one set of channels; performing a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​in each set to obtain a first activation value for each set; performing group quantization processing on the activation values ​​that were not retained in each set to obtain a second activation value for each set; and communicating the activation values ​​based on the first activation value and the second activation value.

[0008] According to the communication compression method for parallel model training provided by the present invention, the TopK operation is performed in a cache.

[0009] According to the communication compression method for parallel model training provided by the present invention, the first quantity and the quantization precision of the group quantization processing for activation values ​​are determined according to the communication compression level, which is determined according to the stage of model training.

[0010] According to the communication compression method for parallel model training provided by the present invention, the communication of activation values ​​based on the first activation value and the second activation value includes: sending the first activation value and the second activation value to the target processing unit using a point-to-point communication method.

[0011] According to the communication compression method for parallel model training provided by the present invention, the method further includes: calculating the gradient of the activation value during the backpropagation stage; performing random rounding quantization on the activation value gradient, wherein the quantization accuracy of the random rounding quantization is determined according to the communication compression level.

[0012] According to the communication compression method for parallel model training provided by the present invention, the method further includes: performing a TopK operation on each group of model parameters, retaining a second number of model parameters in each group to obtain a first model parameter for each group; performing group quantization processing on the model parameters that are not retained in each group to obtain a second model parameter for each group; and communicating the model parameters based on the first model parameter and the second model parameter.

[0013] According to the communication compression method for parallel model training provided by the present invention, the method further includes: determining a quantization operation and a corresponding quantization precision for the gradient of model parameters based on the communication compression level, wherein the quantization operation includes random rounding quantization or a combination of TopK operation and random rounding quantization; and quantizing the gradient of model parameters according to the determined quantization operation and the corresponding quantization precision.

[0014] The present invention also provides a communication compression device for parallel model training, comprising: a computation module configured to perform model layer computation based on allocated tensors during the forward propagation phase to obtain activation values ​​of at least one set of channels; a first quantization module configured to perform a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​in each set to obtain a first activation value for each set; a second quantization module configured to perform group quantization processing on the activation values ​​that are not retained in each set to obtain a second activation value for each set; and a communication module configured to communicate the activation values ​​based on the first activation value and the second activation value.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the communication compression method for parallel model training as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a communication compression method for parallel model training as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a communication compression method for parallel model training as described above.

[0018] The communication compression method, apparatus, and electronic device for parallel model training provided by this invention achieve fast and accurate activation value compression by combining TOPK quantization with naive group quantization, thereby improving the efficiency of parallel training. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the communication compression method for parallel model training provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the communication compression process for parallel model training provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the communication compression device for parallel model training provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising,” “including,” or “including,” and similar terms mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or “connected,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0026] The terminology involved in this invention will be briefly explained below.

[0027] The following is combined with Figures 1-4 This invention describes a communication compression method, apparatus, and electronic device for parallel model training.

[0028] Figure 1 This is a flowchart illustrating the communication compression method for parallel model training provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0029] Step 101: In the forward propagation phase, perform calculations on the model layer based on the allocated tensors to obtain activation values ​​for at least one set of channels.

[0030] In tensor parallel training, the model's weight matrix is ​​partitioned across multiple processing units (such as GPUs). Each processing unit is responsible for computing its assigned segment of the weight matrix.

[0031] Step 102: For each group of activation values ​​in at least one group of channels, perform the TopK operation to select the top K largest values ​​and retain the first number of activation values ​​in each group to obtain the first activation value of each group.

[0032] In this embodiment, the first quantity, K, can be determined based on the required accuracy. This embodiment is based on the observation that outliers in the activation values ​​are few and concentrated in a few channels. Therefore, by grouping the channels of each token and performing TopK quantization, the accuracy of most outliers can be guaranteed.

[0033] Step 103: Perform group quantization processing on the activation values ​​that were not retained in each group to obtain the second activation value for each group.

[0034] In this embodiment, group quantization is a method to reduce the precision of data representation by mapping continuous numerical values ​​to a discrete set of numerical values, thereby reducing the storage and transmission requirements of the data. Group quantization can employ various methods, such as uniform quantization, non-uniform quantization, or learning-based methods. These methods reduce data size by decreasing the precision of the data representation. During quantization, a balance needs to be struck between model performance and computational efficiency. The choice of quantization precision depends on the model's sensitivity to precision loss and the available computational resources. Naive quantization compression of the unretained activation values ​​in each group helps reduce the computational overhead associated with compression.

[0035] See Figure 2 , Figure 2 This is a schematic diagram of the communication compression process for parallel model training provided by the present invention. Figure 2 The process of TopK quantization and group quantization is shown, with Scale and index serving as identifiers for different quantized data.

[0036] Step 104: Communicate the activation values ​​based on the first activation value and the second activation value.

[0037] In this embodiment, cross-processing unit aggregation communication can be performed through communication operations, such as AllReduce.

[0038] Traditional TopK quantization is primarily used for the AllReduce approximation in data-parallel gradient synchronization. However, for tensor parallelism, applying TopK to each tensor as a whole incurs significant performance overhead and ignores the accuracy of values ​​outside the TopK, resulting in a large overall L2 error. Since tensor parallelism demands high precision, directly using group quantization or TopK will lead to a decrease in training accuracy. This implementation innovatively combines these two methods into an efficient fusion operator. TopK ensures high accuracy for outliers, while extracting the TopK makes the data distribution of the quantized groups tend towards a normal distribution, greatly reducing quantization error. Therefore, it can be used in latency-sensitive, high-communication-volume parallel training scenarios such as tensor parallelism. The communication compression method for parallel model training provided in this invention achieves fast and accurate activation value compression by combining TopK quantization with naive group quantization, improving the efficiency of parallel training.

[0039] In some alternative implementations, the TopK operation is performed in the cache. Since the TopK operation is equivalent to K maximum value operations, the TopK quantization process can be completed in the cache, avoiding multiple reads and writes to slow video memory and further improving system efficiency.

[0040] In some optional implementations, communication of activation values ​​based on a first activation value and a second activation value includes sending the first and second activation values ​​to the target processing unit using point-to-point communication. Point-to-point communication includes point-to-point reduceScatter (P2P ReduceScatter). Both AllReduce and ReduceScatter operations require summing the compressed data, and compression errors accumulate during this process. Unlike Ring ReduceScatter, which communicates with adjacent computing units each time, P2P ReduceScatter sends data packets directly to the target computing unit. For the error accumulation problem in Reduce operations, using P2P ReduceScatter reduces the number of error accumulations while maintaining the same communication volume as Ring ReduceScatter. For AllReduce operations, compressed global aggregation (AllGather) can be used after the P2P ReduceScatter operation. Thus, each data packet undergoes only two compression operations during AllReduce, reducing compression errors.

[0041] Previous P2P ReduceScatter implementations were primarily used in low-bit-gradient AllReduce. This implementation utilizes tensor parallelism, offering better accuracy than RingReduceScatter, while maintaining comparable performance in PCIe (Peripheral Component Interconnect Express) interconnect scenarios. Combining TopK quantization and P2P ReduceScatter enables efficient tensor parallel forward activation value communication compression. TopK quantization can also be used for pipelined parallel activation value communication.

[0042] In some optional implementations, the first quantity and the quantization precision of the group quantization processing for activation values ​​are determined according to the communication compression level, which is determined according to the stage of model training. Referring to Table 1, this invention constructs a 3D parallel communication compression framework (3D-CommPress) and proposes three levels of communication compression intensity, O1, O2, and O3, where A represents the activation value. The gradient represents the activation value, and W represents the model parameters. The gradient represents the model parameters, 8 bits and 4 bits represent the quantization precision, top2 represents the TopK operation with a K value of 2, and SR represents random rounding.

[0043] Table 1: Quantitative Information Table for Different Communication Compression Levels

[0044]

[0045] As an example, for fine-tuning scenarios, where the model is highly tolerant of compression, O2 communication compression can be used. For pre-training scenarios, where the model is not yet fully trained to exhibit features such as outliers, communication compression can be omitted initially, and O1 compression can be used for the remaining training after approximately 300 steps.

[0046] In some optional implementations, the method further includes: calculating the gradient of the activation value during the backpropagation phase; and performing random rounding quantization on the activation value gradient, wherein the quantization precision of the random rounding quantization is determined according to the communication compression level. In this implementation, random rounding quantization is used to compress the gradient, ensuring that the expected value of the gradient is the same as the original gradient, allowing the optimizer to function normally.

[0047] In some optional implementations, the method further includes: performing a TopK operation on each group of model parameters, retaining the second number of model parameters in each group to obtain the first model parameters of each group; performing group quantization processing on the model parameters that were not retained in each group to obtain the second model parameters of each group; and communicating the model parameters based on the first model parameters and the second model parameters.

[0048] In some optional implementations, the method further includes: determining the quantization operation and corresponding quantization precision for the model parameter gradient based on the communication compression level; the quantization operation includes random rounding quantization or a combination of TopK operation and random rounding quantization; and quantizing the model parameter gradient according to the determined quantization operation and corresponding quantization precision. For AllGather communication of model parameters in data parallelism, although it has no outliers, TopK quantization can be used for compression to ensure accuracy. For model gradient communication, since its time consumption is not significant in 3D parallelism, higher bit compression can be used for P2P ReduceScatter.

[0049] Compared to existing technologies, the above implementation first compresses the various parts of 3D parallelism, especially the AllReduce communication in tensor parallelism. The TopK quantization and P2P ReduceScatter techniques used ensure accuracy while being very hardware-friendly, introducing very little additional overhead, thus guaranteeing the high efficiency of the entire network training.

[0050] On a 4x8 RTX 4090 cluster, the above implementation achieved a 2x speedup for LLAMA-2-13B instruction fine-tuning. In fine-tuning tasks for OLMo-1B, LLAMA-2-7B, and LLAMA-2-13B, the performance loss was less than 5%, and the model's floating-point computation utilization reached over 50%. In pre-training tasks, the above implementation improved the floating-point computation utilization of OLMo-7B from 56% to 68%, and after training 12 billion words, the loss on the validation set was only about 1% different compared to the uncompressed trained model. This implementation combined with full communication compression for 3D parallelism allows for distributed training of large language models on low-bandwidth clusters.

[0051] The communication compression device for parallel model training provided by the present invention will be described below. The communication compression device for parallel model training described below can be referred to in correspondence with the communication compression method for parallel model training described above.

[0052] Figure 3This is a schematic diagram of the communication compression device for parallel model training provided in an embodiment of this application, as shown below. Figure 3 As shown, it specifically includes: a calculation module 301, configured to perform model layer calculations based on the allocated tensors during the forward propagation phase to obtain activation values ​​for at least one set of channels; a first quantization module 302, configured to perform a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​in each set to obtain the first activation value of each set; a second quantization module 303, configured to perform group quantization processing on the activation values ​​that were not retained in each set to obtain the second activation value of each set; and a communication module 304, configured to communicate activation values ​​based on the first and second activation values.

[0053] In some alternative implementations, the TopK operation is performed in the cache.

[0054] In some alternative implementations, the first quantity and the quantization precision of the group quantization for the activation values ​​are determined according to the communication compression level, which is determined according to the stage of model training.

[0055] In some optional implementations, the communication module 304 is further configured to send the first activation value and the second activation value to the target processing unit using a point-to-point communication method.

[0056] In some alternative implementations, the device also includes a third quantization module configured to: calculate the gradient of the activation value during the backpropagation phase; and perform random rounding quantization on the activation value gradient, wherein the quantization precision of the random rounding quantization is determined according to the communication compression level.

[0057] In some optional implementations, the device also includes a fourth quantization module configured to: perform a TopK operation on each group of model parameters, retain a second number of model parameters in each group to obtain the first model parameters of each group; perform group quantization processing on the model parameters that are not retained in each group to obtain the second model parameters of each group; and communicate the model parameters based on the first model parameters and the second model parameters.

[0058] In some alternative implementations, the device further includes a fifth quantization module configured to: determine the quantization operation and corresponding quantization precision for the model parameter gradient based on the communication compression level, wherein the quantization operation includes random rounding quantization or a combination of TopK operation and random rounding quantization; and quantize the model parameter gradient based on the determined quantization operation and corresponding quantization precision.

[0059] The communication compression device for parallel model training provided by this invention achieves fast and accurate activation value compression by combining TOPK quantization with naive group quantization, thereby improving the efficiency of parallel training.

[0060] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a communication compression method for parallel model training. This method includes: in the forward propagation phase, performing model layer calculations based on the allocated tensors to obtain activation values ​​for at least one set of channels; performing a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​in each set to obtain a first activation value for each set; performing group quantization processing on the activation values ​​that were not retained in each set of activation values ​​to obtain a second activation value for each set; and communicating the activation values ​​based on the first activation value and the second activation value.

[0061] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the communication compression method for parallel model training provided by the above methods. The method includes: in the forward propagation phase, performing calculations of the model layer based on the allocated tensors to obtain activation values ​​of at least one set of channels; performing a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​in each set of activation values ​​to obtain a first activation value for each set; performing group quantization processing on the activation values ​​that are not retained in each set of activation values ​​to obtain a second activation value for each set; and communicating the activation values ​​based on the first activation value and the second activation value.

[0063] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a communication compression method for parallel model training provided by the methods described above. This method includes: during the forward propagation phase, performing calculations of model layers based on allocated tensors to obtain activation values ​​for at least one set of channels; performing a TopK operation on each set of activation values ​​in the at least one set of channels, retaining a first number of activation values ​​in each set to obtain a first activation value for each set; performing group quantization processing on the activation values ​​that were not retained in each set of activation values ​​to obtain a second activation value for each set; and communicating the activation values ​​based on the first and second activation values.

[0064] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication compression method for parallel model training, characterized in that, include: During the forward propagation phase, the model layer is computed based on the assigned tensors to obtain the activation values ​​of at least one set of channels; For each group of activation values ​​in the at least one group of channels, perform the TopK operation to select the top K largest values, and retain the first number of activation values ​​in each group to obtain the first activation value of each group; wherein, the first number is the K value; For each group of activation values ​​that were not retained, perform group quantization processing on the activation values ​​to obtain the second activation value for each group; Communication of activation values ​​is performed based on the first activation value and the second activation value; The first quantity and the quantization precision of the group quantization processing for the activation values ​​are determined according to the communication compression level, which is determined according to the stage of model training.

2. The communication compression method for parallel model training according to claim 1, characterized in that, The TopK operation is performed in the cache.

3. The communication compression method for parallel model training according to claim 1, characterized in that, The communication of activation values ​​based on the first activation value and the second activation value includes: The first activation value and the second activation value are sent to the target processing unit using a point-to-point communication method.

4. The communication compression method for parallel model training according to claim 1, characterized in that, The method further includes: During the backpropagation phase, the gradient of the activation values ​​is calculated; The activation value gradient is subjected to random rounding quantization, and the quantization precision of the random rounding quantization is determined according to the communication compression level.

5. The communication compression method for parallel model training according to claim 1, characterized in that, The method further includes: For each group of model parameters, perform the TopK operation and retain the second number of model parameters in each group to obtain the first model parameters of each group; For the model parameters that were not retained in each group of model parameters, group quantization processing was performed on the model parameters to obtain the second model parameters for each group; The model parameters are communicated based on the first model parameters and the second model parameters.

6. The communication compression method for parallel model training according to claim 1, characterized in that, The method further includes: The quantization operation and corresponding quantization precision for the model parameter gradient are determined based on the communication compression level. The quantization operation includes random rounding quantization or a combination of TopK operation and random rounding quantization. The gradient of the model parameters is quantized based on the determined quantization operation and the corresponding quantization precision.

7. A communication compression device for parallel model training, characterized in that, include: The computation module is configured to perform computations on the model layer based on the allocated tensors during the forward propagation phase, and obtain activation values ​​for at least one set of channels. The first quantization module is configured to perform a TopK operation on each group of activation values ​​in the at least one group of channels, and retain a first number of activation values ​​in each group to obtain the first activation value of each group; wherein, the first number is the K value. The second quantization module is configured to perform group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain the second activation value for each group. The communication module is configured to communicate activation values ​​based on the first activation value and the second activation value; The first quantity and the quantization precision of the group quantization processing for the activation values ​​are determined according to the communication compression level, which is determined according to the stage of model training.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the communication compression method for parallel model training as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the communication compression method for parallel model training as described in any one of claims 1 to 6.

Images