Communication compression method and device for model parallel training and electronic equipment

Abstract

The invention relates to the field of machine learning, and provides a communication compression method and device for model parallel training and electronic equipment, and the method comprises the steps: executing the calculation of a model layer based on a distributed tensor in a forward propagation stage, and obtaining an activation value of at least one group of channels; executing TopK operation for each group of activation values in the activation values of the at least one group of channels, and reserving a first number of activation values in each group of activation values to obtain a first activation value of each group; performing group quantization processing on the activation values for unreserved activation values in each group of activation values to obtain a second activation value of each group; and performing communication of the activation values based on the first activation value and the second activation value. According to the method, through combination of TOPK quantization and naive group quantization, rapid and accurate activation value compression is realized, and the parallel training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a communication compression method, device and electronic equipment for parallel model training. Background Art

[0002] In the field of deep learning, in order to accelerate the training process of large-scale models such as Transformer models, researchers have adopted a variety of parallel strategies, including tensor parallelism, pipeline parallelism, and data parallelism. These parallel strategies significantly improve training efficiency by distributing computing tasks on multiple processing units (such as GPUs).

[0003] However, each parallel strategy introduces different communication overheads: (1) Tensor parallelism divides matrix multiplication into rows and columns, requiring AllReduce collective communication at the end of the forward and backward calculations of the attention block and the feedforward neural network block. (2) Pipeline parallelism divides the model into different parallel stages according to the layer and places them in different computing cards. Point-to-point send / receive communication between groups is required during forward and backward calculations. (3) Data parallelism copies the model into multiple copies, each training a different data block. When updating the model parameters, the gradients of different copies are synchronized through AllReduce operations. Data parallelism has variants of ZeRO-1 / 2 / 3, which reduce the memory usage of each copy and increase the communication volume. The collective communication introduced by tensor parallelism and data parallelism will introduce a lot of communication overhead when the parallel group size is expanded, making it impossible to perform distributed training on clusters without high-speed networks.

[0004] For example, the Llama-13B model can be trained on a cluster of 32 4090 GPUs using tensor parallelism of 2, pipeline parallelism of 8, and data parallelism of 2. However, since the 4090 does not support high-speed interconnection methods such as NvLink and PCI-E P2P, the model floating-point computing utilization (Model FLOPS Utilization, MFU) can only reach 25%, that is, about three-quarters of the time the 4090 is not involved in matrix multiplication operations.

[0005] In order to reduce the amount of communication, many works are currently focusing on communication compression for parallel training. However, direct quantization compression will lead to large errors. Using more complex compression methods, compression and decompression itself will generate a lot of computational overhead, which will in turn reduce the training speed. Summary of the invention

[0006] The present invention provides a communication compression method, device and electronic device for model parallel training, which are used to solve the defects of large quantization error and large computing overhead generated by compression and decompression in the prior art, realize simple, effective and hardware-friendly communication compression, and improve the efficiency of model parallel training.

[0007] The present invention provides a communication compression method for model parallel training, comprising: in the forward propagation stage, performing calculations of the model layer based on the allocated tensor to obtain activation values ​​of at least one group of channels; performing a TopK operation for each group of activation values ​​in the at least one group of channel activation values, retaining a first number of activation values ​​in each group of activation values ​​to obtain a first activation value for each group; performing group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain a second activation value for each group; and communicating the activation values ​​based on the first activation values ​​and the second activation values.

[0008] According to the communication compression method for model parallel training provided by the present invention, the TopK operation is completed in the cache.

[0009] According to the communication compression method for model parallel training provided by the present invention, the first number and the quantization accuracy of the group quantization processing for the activation value are determined according to the communication compression level, and the communication compression level is determined according to the stage of model training.

[0010] According to the communication compression method for model parallel training provided by the present invention, the communication of activation values ​​based on the first activation value and the second activation value includes: sending the first activation value and the second activation value to a target processing unit in a point-to-point communication manner.

[0011] According to the communication compression method for model parallel training provided by the present invention, the method also includes: in the back-propagation stage, calculating the gradient of the activation value; performing random rounding quantization processing on the activation value gradient, and the quantization accuracy of the random rounding quantization processing is determined according to the communication compression level.

[0012] According to the communication compression method for model parallel training provided by the present invention, the method also includes: performing a TopK operation on each group of model parameters, retaining a second number of model parameters in each group of model parameters to obtain first model parameters for each group; performing group quantization processing on model parameters that are not retained in each group of model parameters to obtain second model parameters for each group; and communicating model parameters based on the first model parameters and the second model parameters.

[0013] According to the communication compression method for model parallel training provided by the present invention, the method also includes: determining the quantization operation and the corresponding quantization precision for the model parameter gradient according to the communication compression level, the quantization operation includes a random rounding quantization process or a combination of a TopK operation and a random rounding quantization process; and quantizing the model parameter gradient according to the determined quantization operation and the corresponding quantization precision.

[0014] The present invention also provides a communication compression device for model parallel training, comprising: a computing module, configured to perform model layer calculations based on the allocated tensors in the forward propagation stage to obtain activation values ​​of at least one group of channels; a first quantization module, configured to perform a TopK operation on each group of activation values ​​of the at least one group of channels, retain a first number of activation values ​​in each group of activation values ​​to obtain a first activation value for each group; a second quantization module, configured to perform group quantization processing on activation values ​​that are not retained in each group of activation values ​​to obtain a second activation value for each group; and a communication module, configured to communicate activation values ​​based on the first activation value and the second activation value.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a communication compression method for parallel model training as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a communication compression method for parallel model training as described in any one of the above.

[0017] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the communication compression methods for parallel model training described above.

[0018] The communication compression method, device and electronic device for model parallel training provided by the present invention realize fast and accurate activation value compression through the combination of TOPK quantization and naive group quantization, thereby improving the efficiency of parallel training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the communication compression method for model parallel training provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the communication compression process for model parallel training provided by the present invention.

[0022] Figure 3 It is a structural schematic diagram of a communication compression device for parallel model training provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0026] The following is a brief explanation of the terms involved in the present invention.

[0027] Combine the following Figure 1-Figure 4 The present invention describes a communication compression method, device and electronic device for model parallel training.

[0028] Figure 1 is a flow chart of the communication compression method for model parallel training provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: In the forward propagation stage, the calculation of the model layer is performed based on the allocated tensor to obtain the activation value of at least one set of channels.

[0029] In tensor parallel training, the model's weight matrix is ​​split across multiple processing units (such as GPUs). Each processing unit is responsible for computing its assigned piece of the weight matrix.

[0030] Step 102: For each group of activation values ​​in the activation values ​​of at least one group of channels, perform a TopK operation, retain the first number of activation values ​​in each group of activation values ​​to obtain the first activation value of each group.

[0031] In this embodiment, the first number, i.e., the K value, can be determined according to the accuracy requirement. This embodiment is based on an observation that there are few outliers in the activation value and they are concentrated in a few channels. Therefore, by grouping the channels of each token and performing TopK quantization, the accuracy of most outliers can be guaranteed.

[0032] Step 103: Perform group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain a second activation value for each group.

[0033] In this embodiment, group quantization is a method of reducing the precision of data representation, which maps continuous values ​​into discrete sets of values, thereby reducing the storage and transmission requirements of data. Group quantization processing can adopt a variety of methods, such as uniform quantization, non-uniform quantization, or learning-based methods. These methods reduce the size of data by reducing the precision of data representation. During the quantization process, it is necessary to balance model performance and computational efficiency. The choice of quantization accuracy depends on the sensitivity of the model to precision loss and the available computing resources. Simple quantization compression of the activation values ​​that are not retained in each group of activation values ​​helps to reduce the computational overhead caused by compression.

[0034] See also Figure 2 , Figure 2 is a schematic diagram of a communication compression process for model parallel training provided by the present invention, Figure 2 The process of TopK quantization and group quantization is shown, and Scale and index are identifiers of different quantized data.

[0035] Step 104: Communicate activation values ​​based on the first activation value and the second activation value.

[0036] In this embodiment, aggregate communication across processing units may be performed through communication operations such as global reduction (AllReduce).

[0037] In the past, TopK quantization was mainly used for AllReduce approximation when synchronizing gradients in data parallelism. However, for tensor parallelism, using TopK for each tensor as a whole will result in a huge performance overhead, while ignoring the accuracy of values ​​other than TopK, resulting in a large overall L2 error. Since tensor parallelism has very high requirements for precision, directly using group quantization or TopK will lead to a decrease in training accuracy. This embodiment innovatively implements the two methods as an efficient fusion operator. TopK ensures the high precision of outliers. At the same time, extracting TopK makes the data distribution of the quantization group tend to a conventional normal distribution, greatly reducing the quantization error, so that it can be used for parallel training scenarios such as tensor parallelism that are sensitive to delays and have large communication volumes. The communication compression method for model parallel training provided by the present invention achieves fast and accurate activation value compression through the combination of TOPK quantization and naive group quantization, thereby improving the efficiency of parallel training.

[0038] In some optional implementations, the TopK operation is completed in the cache. Since the TopK operation is equivalent to K maximum value operations, the TopK quantization process can be completed in the cache, avoiding multiple reads and writes to the low-speed video memory, further improving system efficiency.

[0039] In some optional implementations, the communication of activation values ​​based on the first activation value and the second activation value includes: sending the first activation value and the second activation value to the target processing unit in a point-to-point communication mode. The point-to-point communication mode includes point-to-point reduction scatter (P2P ReduceScatter). AllReduce and ReduceScatter operations require the compressed data to be added, and the compression error will also accumulate in the process. Unlike Ring ReduceScatter, which communicates with adjacent computing units each time, P2PReduceScatter sends the data packet directly to the target computing unit. For the error accumulation problem of the Reduce operation, using P2P ReduceScatter can reduce the number of error accumulations while ensuring the same communication volume as Ring ReduceScatter. For the AllReduce operation, a compressed global aggregation (AllGather) can be used after the P2P ReduceScatter operation. As a result, each data packet only undergoes two compression operations when performing AllReduce, reducing the compression error.

[0040] In the past, P2P ReduceScatter was mainly used in low-bit gradient AllReduce. This implementation is used in tensor parallelism, and it has better accuracy than RingReduceScatter. At the same time, it has the same performance as RingReduceScatter in PCIE (Peripheral Component Interconnect Express, peripheral high-speed serial bus) interconnection scenarios. Combining TopK quantization and P2P ReduceScatter, efficient tensor parallel forward activation value communication compression can be achieved. For pipeline parallel activation value communication, TopK quantization can also be used.

[0041] In some optional implementations, the first number and the quantization accuracy of the group quantization process for the activation value are determined according to the communication compression level, and the communication compression level is determined according to the stage of model training. Referring to Table 1, the present invention constructs a 3D parallel communication compression framework (3D-CommPress) and proposes three levels of communication compression strength, O1, O2 and O3, where A represents the activation value, represents the gradient of the activation value, W represents the model parameters, Represents the gradient of the model parameters, 8bit and 4bit represent the quantization accuracy, top2 represents the TopK operation with a K value of 2, and SR represents random rounding.

[0042] Table 1: Quantization information table of different communication compression levels As an example, for fine-tuning scenarios, the model has a high tolerance for compression and can use O2 communication compression level. For pre-training scenarios, the model has not yet been fully trained to exhibit characteristics such as outliers. Communication compression can be omitted in the early stages, and O1 compression can be used for the rest of the training after about 300 steps.

[0043] In some optional implementations, the method further includes: in the back-propagation phase, calculating the gradient of the activation value; performing random rounding quantization processing on the activation value gradient, wherein the quantization accuracy of the random rounding quantization processing is determined according to the communication compression level. In this implementation, random rounding quantization is used to compress the gradient, ensuring that the expected value of the gradient is the same as the original gradient, so that the optimizer can operate normally.

[0044] In some optional implementations, the method also includes: performing a TopK operation on each group of model parameters, retaining a second number of model parameters in each group of model parameters to obtain first model parameters for each group; performing group quantization processing on the model parameters that are not retained in each group of model parameters to obtain second model parameters for each group; and communicating model parameters based on the first model parameters and the second model parameters.

[0045] In some optional implementations, the method further includes: determining a quantization operation and a corresponding quantization precision for the model parameter gradient according to the communication compression level, the quantization operation including a random rounding quantization process or a combination of a TopK operation and a random rounding quantization process; and quantizing the model parameter gradient according to the determined quantization operation and the corresponding quantization precision. For the AllGather communication of model parameters in data parallelism, although there are no outliers, TopK quantization can also be used for compression to ensure accuracy. For model gradient communication, since its time share is not large in 3D parallelism, higher bit compression can be used for P2P ReduceScatter.

[0046] Compared with the existing technology, the above implementation method first compresses the various parts of 3D parallelism, especially the AllReduce communication in tensor parallelism. The TopK quantization and P2P ReduceScatter technology used are very friendly to actual hardware while ensuring accuracy, and introduce very little additional overhead, thereby ensuring the efficiency of the entire network training.

[0047] On a 4x8 RTX 4090 cluster, the above implementation achieves a two-fold acceleration of LLAMA-2-13B instruction fine-tuning. In the fine-tuning tasks of OLMo-1B, LLAMA-2-7B, and LLAMA-2-13B, the performance loss is less than 5%, and the model floating point computing utilization rate has reached more than 50%. In the pre-training task, the above implementation increases the model floating point computing utilization rate of OLMo-7B from 56% to 68%, and after training 12 billion words, the Loss on the validation set is only about 1% different from the uncompressed training model. Combined with the above implementation, all communication compression of 3D parallelism is achieved, allowing large language models to be distributed trained in low-bandwidth clusters.

[0048] The communication compression device for model parallel training provided by the present invention is described below. The communication compression device for model parallel training described below and the communication compression method for model parallel training described above can be referenced to each other.

[0049] Figure 3A schematic diagram of the structure of a communication compression device for model parallel training provided in an embodiment of the present application, such as Figure 3 As shown, it specifically includes: a computing module 301, which is configured to perform model layer calculations based on the allocated tensors in the forward propagation stage to obtain activation values ​​of at least one group of channels; a first quantization module 302, which is configured to perform TopK operations on each group of activation values ​​of at least one group of channels, retain the first number of activation values ​​in each group of activation values ​​to obtain the first activation value of each group; a second quantization module 303, which is configured to perform group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain the second activation value of each group; a communication module 304, which is configured to communicate activation values ​​based on the first activation value and the second activation value.

[0050] In some optional implementations, the TopK operation is performed in the cache.

[0051] In some optional implementations, the first quantity and the quantization accuracy of the group quantization process for the activation values ​​are determined according to a communication compression level, and the communication compression level is determined according to a stage of model training.

[0052] In some optional implementations, the communication module 304 is further configured to: send the first activation value and the second activation value to the target processing unit in a point-to-point communication manner.

[0053] In some optional implementations, the device also includes a third quantization module, which is configured to: calculate the gradient of the activation value in the back-propagation stage; perform random rounding quantization processing on the activation value gradient, and the quantization accuracy of the random rounding quantization processing is determined according to the communication compression level.

[0054] In some optional implementations, the device also includes a fourth quantization module, which is configured to: perform a TopK operation on each group of model parameters, retain a second number of model parameters in each group of model parameters to obtain first model parameters for each group; perform group quantization processing on the model parameters that are not retained in each group of model parameters to obtain second model parameters for each group; and communicate model parameters based on the first model parameters and the second model parameters.

[0055] In some optional implementations, the device also includes a fifth quantization module, which is configured to: determine the quantization operation and the corresponding quantization precision for the model parameter gradient according to the communication compression level, the quantization operation including random rounding quantization processing or a combination of TopK operation and random rounding quantization processing; quantize the model parameter gradient according to the determined quantization operation and the corresponding quantization precision.

[0056] The communication compression device for model parallel training provided by the present invention realizes fast and accurate activation value compression through the combination of TOPK quantization and naive group quantization, thereby improving the efficiency of parallel training.

[0057] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the communication compression method for model parallel training, the method comprising: in the forward propagation stage, performing the calculation of the model layer based on the allocated tensor to obtain the activation value of at least one group of channels; for each group of activation values ​​in the activation value of at least one group of channels, performing the TopK operation, retaining the first number of activation values ​​in each group of activation values ​​to obtain the first activation value of each group; for the activation values ​​that are not retained in each group of activation values, performing group quantization processing on the activation value to obtain the second activation value of each group; and communicating the activation value based on the first activation value and the second activation value.

[0058] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0059] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the communication compression method for model parallel training provided by the above methods, the method including: in the forward propagation stage, performing calculations of the model layer based on the allocated tensors to obtain activation values ​​of at least one group of channels; for each group of activation values ​​in the activation values ​​of at least one group of channels, performing a TopK operation, retaining a first number of activation values ​​in each group of activation values ​​to obtain a first activation value for each group; for each group of activation values ​​that are not retained, performing group quantization processing on the activation values ​​to obtain a second activation value for each group; and communicating activation values ​​based on the first activation values ​​and the second activation values.

[0060] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the communication compression method for model parallel training provided by the above-mentioned methods, the method comprising: in the forward propagation stage, performing calculations of the model layer based on the allocated tensors to obtain activation values ​​of at least one group of channels; performing a TopK operation on each group of activation values ​​in the activation values ​​of at least one group of channels, retaining a first number of activation values ​​in each group of activation values ​​to obtain a first activation value for each group; performing group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain a second activation value for each group; and communicating activation values ​​based on the first activation values ​​and the second activation values.

[0061] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication compression method for model parallel training, characterized in that: include: In the forward propagation phase, the model layer calculations are performed based on the assigned tensors to obtain the activation values ​​of at least one set of channels; For each group of activation values ​​in the activation values ​​of the at least one group of channels, a TopK operation is performed, and a first number of activation values ​​in each group of activation values ​​is retained to obtain a first activation value of each group; Performing group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain a second activation value of each group; An activation value is communicated based on the first activation value and the second activation value.

2. The communication compression method for model parallel training according to claim 1, characterized in that: The TopK operation is performed in the cache.

3. The communication compression method for model parallel training according to claim 1, characterized in that: The first number and the quantization accuracy of the group quantization process for the activation value are determined according to a communication compression level, and the communication compression level is determined according to a stage of model training.

4. The communication compression method for model parallel training according to claim 3, characterized in that: The communicating of activation values ​​based on the first activation value and the second activation value includes: The first activation value and the second activation value are sent to a target processing unit in a point-to-point communication manner.

5. The communication compression method for model parallel training according to claim 3, characterized in that: The method further comprises: In the back-propagation phase, the gradient of the activation value is calculated; The activation value gradient is subjected to random rounding quantization processing, wherein the quantization accuracy of the random rounding quantization processing is determined according to the communication compression level.

6. The communication compression method for model parallel training according to claim 1, characterized in that: The method further comprises: For each group of model parameters, perform a TopK operation, retain the second number of model parameters in each group of model parameters to obtain the first model parameters of each group; Performing group quantization processing on the model parameters that are not retained in each group of model parameters to obtain second model parameters of each group; Model parameters are communicated based on the first model parameters and the second model parameters.

7. The communication compression method for model parallel training according to claim 1, characterized in that: The method further comprises: Determining a quantization operation for a model parameter gradient and a corresponding quantization precision according to the communication compression level, wherein the quantization operation includes a random rounding quantization process or a combination of a TopK operation and a random rounding quantization process; The model parameter gradient is quantized according to the determined quantization operation and the corresponding quantization accuracy.

8. A communication compression device for model parallel training, characterized in that: include: A computing module is configured to perform a computation of a model layer based on the assigned tensor in a forward propagation phase to obtain activation values ​​of at least one set of channels; A first quantization module is configured to perform a TopK operation on each group of activation values ​​in the at least one group of channel activation values, retain a first number of activation values ​​in each group of activation values ​​to obtain a first activation value of each group; A second quantization module is configured to perform group quantization processing on the activation values ​​that are not retained in each group of activation values ​​to obtain a second activation value of each group; The communication module is configured to communicate activation values ​​based on the first activation value and the second activation value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the communication compression method for model parallel training as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the communication compression method for model parallel training as described in any one of claims 1 to 7 is implemented.

Images