A gradient optimization compression method based on ternary distributed deep learning communication

Abstract

The application discloses a gradient optimization compression method based on three-value distributed deep learning communication, which comprises the following steps: obtaining a parameter list, and obtaining an optimization gradient list corresponding to the parameter; using a randomized three-value compression algorithm to encode the gradient, compressing the gradient component into a two-bit binary number, storing the bit number of each component in the encoding, and selecting an expansion value; performing AllReduce synchronization between nodes on the complete data after encoding, using a three-value summation method to perform incremental bit iteration during transmission; expanding the buffer area of the receiving party according to the complete data after encoding; decoding the complete data after encoding; calculating the average value of each gradient after decoding; and calculating the updated parameter. The application can improve the throughput of gradient transmission per unit time and reduce the transmission delay of the gradient while ensuring that the iteration interval waiting time of the distributed deep learning calculation node does not exceed the baseline.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of AllReduce communication, and particularly relates to a gradient optimization compression method for distributed deep learning communication based on ternary. BACKGROUND

[0002] With the vigorous development of deep learning in various fields, the model size and sample size of deep learning training are also expanding. In the case that single GPU training cannot meet the computing performance demand and model storage demand, using multiple nodes and multiple GPUs for distributed deep learning has become a research hotspot at present. However, using distributed deep learning will inevitably integrate some defects of previous distributed computing. The parallel model can ideally achieve linear speedup ratio improvement of the serial model, but in the real situation, the linear speedup ratio is a very difficult target to achieve. The reason for the deviation of the speedup ratio is that the communication between nodes in the distributed computing model will occupy a large amount of time overhead.

[0003] At present, the parallel mode of distributed deep learning can be divided into data parallelism and model parallelism. Data parallelism has become the preferred mode of most distributed deep learning developers due to its excellent codeability and model fusion speed. When using model parallelism for distributed deep learning, two different synchronization modes can be used according to the architecture of the system. One is the parameter server (Parameter Server) architecture, and the other is the AllReduce architecture. The parameter server generally uses a computing node other than the GPU node as a control server that collects the gradients calculated by all other nodes and distributes the updated parameters. It is a centralized communication architecture. AllReduce performs data transmission between all GPU computing nodes, and uses the gradients calculated by all other nodes to update the parameters on each computing node. It is a typical decentralized communication architecture. Due to the excellent performance and mature ecological support of the AllReduce synchronization mode, it has become the mainstream training method at present.

[0004] When using the AllReduce synchronization model for data parallel distributed deep learning, training a large model will introduce a large communication data overhead, that is, the data synchronization time between iterations of the node will occupy a large part of the overall training time, causing a serious drop in training efficiency.

[0005] Existing researches focus on using compression algorithms to compress the data synchronized between nodes to the greatest extent to reduce the amount of data during synchronization. However, the existing compression algorithm researches are mostly developed based on the parameter server architecture. When the AllReduce architecture is used, due to the difference in synchronization methods, the node obtains part of the compressed data of the previous node, sums the data corresponding to the node, and then compresses again, which causes further loss of precision. Therefore, a compression algorithm for reducing the loss of precision for the AllReduce communication architecture is needed. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a gradient optimization compression method for distributed deep learning communication based on ternary values to solve the problems of the prior art. The method uses randomized ternary compression, constructs an incremental iterative synchronization summation algorithm, and cooperates with an expansion algorithm for cache space to compress the amount of communication data. The method can improve the throughput of transmitted gradients per unit time and reduce the transmission delay of gradients while ensuring that the inter-iteration waiting time of the distributed deep learning computing node does not exceed the baseline, and can solve the problems of large communication data volume when using AllReduce for distributed deep learning training and the superimposed gradient precision loss introduced by AllReduce compression.

[0007] To achieve the above technical purposes, the technical solution adopted by the present application is as follows:

[0008] A gradient optimization compression method for distributed deep learning communication based on ternary values, comprising:

[0009] Step 1: Obtain the parameter list used by Backward and Optimizer, and obtain the corresponding optimization gradient list from the parameters;

[0010] Step 2: Use a randomized ternary compression algorithm to encode the gradient, compress the gradient components into two-bit binary numbers, store the number of bits occupied by each component in the encoding, and select an expansion value;

[0011] Step 3: Perform AllReduce synchronization between nodes for the complete data encoded in step 2, and use a ternary summation method for incremental bit iteration during transmission;

[0012] Step 4: After the AllReduce synchronization is completed, each node has all the encoded complete data, and the buffer area of the receiving party is expanded according to the encoded complete data;

[0013] Step 5: Call the decompression algorithm to decode the encoded complete data in combination with the expansion value in step 2;

[0014] Step 6: call the average method of AllReduce to obtain the average value of each gradient after decoding;

[0015] Step 7: according to the average value of each gradient in step 6, use the parameter optimization method of PyTorch or TensorFlow to calculate the updated parameters.

[0016] To optimize the above technical solutions, the specific measures taken also include:

[0017] The gradient described above is represented by 32 floating-point numbers.

[0018] The above step 2 is as follows:

[0019] Step 2.1: all nodes broadcast the maximum component value calculated in this round of iteration to all other nodes;

[0020] Step 2.2: after each node obtains the maximum component value of other nodes, the maximum component value of the entire system in this round of iteration is selected as the expansion value for storage;

[0021] Step 2.3: use the expansion value to perform probability operation, and the result of the probability operation represents the compressed value of each component in this round of iteration;

[0022] Step 2.4: use the probability operation result and the sign function of each component to perform multiplication operation to obtain the compressed component value;

[0023] Step 2.5: encode the compression result in step 2.4, store the binary encoding of all components using UINT8 data type, merge into UINT32 data and store in the buffer, and take a UINT8 format space into the component encoding bit number.

[0024] In the above step 2.1, all nodes use the Broadcast communication method to broadcast the maximum component value calculated in this round of iteration to all other nodes.

[0025] In the AllReduce synchronization described in the above step 3, each node is both a sender and a receiver.

[0026] The sum method of three values described in the above step 3 is as follows:

[0027] First, decode: get the number of bits occupied by each component in the encoding, read the buffer data in UINT32 format, and use the mask to obtain the component data by AND operation with the data block;

[0028] Second, sum and encode: after the component data is summed, it is written into the UINT32 format buffer by bit and encoded;

[0029] Finally, the addition result and the mask are shifted, and the decoding, addition, and encoding of the next data component are performed in a loop.

[0030] The buffer expansion method used in step 4 is as follows:

[0031] The buffer size of the receiver is expanded according to the number of bits occupied by each component in the encoding and the number of components, and the entry pointer of each gradient is updated.

[0032] The buffer expansion method is built in the Horovod code, and all the information required for expansion is encoded into the binary data of the specific data, and finally the entry pointer of the gradient is updated to ensure maximum compatibility with the original system.

[0033] The decompression algorithm used in step 5 is as follows:

[0034] Step 5.1: Read the number of bits occupied by the last stored component in the encoding;

[0035] Step 5.2: Construct the bitwise AND operation number required for each shift according to the number of bits occupied by the component;

[0036] Step 5.3: Perform bitwise AND and shift operations in sequence according to the component order;

[0037] Step 5.4: Expand the number of bits of the component value obtained after shifting;

[0038] Step 5.5: Fill the component value into the corresponding result gradient according to the shape and size of the gradient;

[0039] Step 5.6: Multiply the gradient obtained in step 5.5 by the expansion value to return the final decoding result.

[0040] The decompression algorithm is built in the Compression.py file, and the GPU method of Pytorch or TensorFlow is used to process the gradient array.

[0041] The present application has the following beneficial effects:

[0042] The application is based on the idea of reducing communication overhead by compressing communication volume, proposes two optimization targets of gradient throughput and transmission delay, and improves the throughput of gradient transmission per unit time and reduces the transmission delay of gradient under the condition that the inter-iteration waiting time of distributed deep learning computing nodes does not exceed the baseline. In order to reduce the communication data volume, the gradient component represented by 32-bit floating point number is compressed into two-bit binary number by using the randomization ternary compression algorithm of gradient, the gradient precision loss superposition problem introduced by AllReduce synchronization model is considered, and the synchronization method with increasing number of bits according to the number of Reduce-Scatter is adopted. The corresponding gradient compression and decompression algorithm, the summation algorithm of incremental iteration and the cache expansion algorithm of the receiver are constructed. The throughput of gradient transmission per unit time is improved and the transmission delay of gradient is reduced under the condition that the inter-iteration waiting time of distributed deep learning computing nodes does not exceed the baseline.

[0043] 1. The implementation method of the compression algorithm in the AllReduce architecture is improved, the ternary compression is adapted to the AllReduce architecture, and the incremental iteration transmission method is used to avoid multiple precision losses. The problem that the traditional compression algorithm is mostly developed for parameter server architecture, and under the parameter server architecture, the gradients calculated by each node can be sent to the parameter server for aggregation after one compression, while the AllReduce architecture needs multiple summation and compression operations of the nodes, which introduces the superposition loss of precision.

[0044] 2. The incremental iteration summation can avoid the precision loss introduced by the compression of the nodes after summation, the data after compression of the last node and the data after compression of the current node are summed, and are encoded into extended precision, supplemented by multiple encoding and decoding operations of the sender and receiver, the redundant compression operation after summation is removed, and multiple precision losses are avoided.

[0045] 3. The difference from the traditional distributed deep learning communication compression algorithm is:

[0046] (1) Different synchronization architecture: the traditional distributed deep learning compression algorithm mainly considers the parameter server architecture, i.e. the centralized synchronization method, and does not consider the precision loss problem of the decentralized point-to-point transmission method. The incremental iteration method is used in the application to optimize the precision loss, which can achieve good results.

[0047] (2) Different gradient summation algorithm: the summation algorithm of the traditional distributed deep learning compression algorithm uses the ordinary summation method for the 32-bit floating point number component without compression, and uses the summation method after decompression for the compressed gradient. The summation method used in the application uses the bit-by-bit and bit shift method to simply decode and sum the compressed gradient, and performs incremental encoding operation according to the value range of the result after summation. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 Flowchart of the method of the present application;

[0049] Figure 2 Pseudocode of the compression algorithm for a single node;

[0050] Figure 3 Diagram of the number of bits occupied by the encoded value range of the summation result;

[0051] Figure 4 Diagram of the timeline after AllReduce compression;

[0052] Figure 5 Diagram of the value range after initial compression and summation;

[0053] Figure 6 Diagram of the binary encoding after summation;

[0054] Figure 7 Flowchart of the implementation of the method of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application will be further described below with reference to the accompanying drawings.

[0056] The application provides a gradient optimization compression method based on a three-value distributed deep learning communication, using a randomized three-value compression algorithm of a gradient to compress a gradient component represented by a 32-bit floating point number into a two-bit binary number, considering a gradient precision loss superposition problem introduced by an AllReduce synchronization model, and adopting a synchronization method with an increasing number of bits according to a Reduce-Scatter number of times. A corresponding gradient compression and decompression algorithm, an incremental iteration summation algorithm and a cache expansion algorithm of a receiver are constructed. The gradient throughput in a unit time is improved and the gradient transmission delay is reduced under the condition that the inter-iteration waiting time of the distributed deep learning calculation node is not more than a baseline. The application is constructed on a Horovod basic platform, and the Horovod is a communication middleware for Pytorch, TensorFlow, MXNet and the like, which uses high-performance communication libraries such as MPI, Gloo and NCCL to perform AllReduce gradient synchronization between calculation nodes. The compression algorithm and the decompression algorithm are constructed in a Compression.py file of the Horovod code in an object-oriented manner, ensuring compatibility with the original system, using a gradient matrix processing method of Pytorch and TensorFlow, optimizing the GPU and other high-performance processors, avoiding redundant memory copy operations, and directly compressing the gradient by using the calculation node. The summation algorithm is constructed on the high-performance communication library Gloo, a new summation function is constructed for the three-value gradient form, and a coding and decoding algorithm used by the summation function is written.

[0057] As shown in Figure 1 and 7 The gradient optimization compression method comprises the following steps:

[0058] Step 1: obtaining a parameter list used by Backward and Optimizer of Pytorch or TensorFlow, obtaining an optimization gradient list corresponding to the parameters from the parameters;

[0059] Step 1: obtaining a parameter list used by Backward and Optimizer, obtaining an optimization gradient list corresponding to the parameters from the parameters;

[0060] Step 2: using a randomized three-value compression algorithm to encode the gradient, compressing the gradient component into a two-bit binary number, storing the number of bits occupied by each component in the encoding, and selecting an expansion value;

[0061] The compression method and the encoding method are called to encode the gradient stored by the 32-bit floating point number into the three-value form, and the number of bits occupied by each component is encoded into the final result;

[0062] Step 3: After the AllReduce synchronization of the encoded complete data in step 2, use the three-value summation method for incremental bit number iteration during transmission;

[0063] Step 4: After the AllReduce synchronization is completed, each node has all the encoded complete data, and the buffer of the receiving party is expanded according to the encoded complete data;

[0064] After the AllReduce synchronization is completed, each node has all the encoded complete data, and the buffer is expanded to the component bit number multiplied by the component number using the buffer expansion method;

[0065] Step 5: Call the decompression algorithm to decode the encoded complete data into 32-bit floating-point numbers combined with the expansion value in step 2;

[0066] Step 6: Call the average method of AllReduce to calculate the average value of each gradient;

[0067] Step 7: Use the parameter optimization method of PyTorch or TensorFlow to calculate the updated parameters.

[0068] Preferably, the step 2 is specifically as follows:

[0069] Step 2.1: All nodes use the Broadcast communication method to broadcast the maximum component value calculated in this round of iteration to all other nodes;

[0070] Step 2.2: After each node obtains the maximum component value of other nodes, select the maximum component value of the entire system in this round of iteration as the expansion value for storage;

[0071] Step 2.3: Use the expansion value for probability operation, and the result of the probability operation represents the probability that each component is 0 or other after compression in this round of iteration;

[0072] Step 2.4: Use the probability operation result and the sign function of each component to perform multiplication operation to obtain the compressed component value;

[0073] Step 2.5: Encode the compression result in step 2.4, use UINT8 data type to store the binary encoding of all components, merge into UINT32 data and store in the buffer, and take a UINT8 format space into the component encoding bit number.

[0074] The Broadcast method used in step 2.1 is a built-in communication method in Horovod. When using three-value compression, the maximum component value is extracted before the gradient compression method is called. After extraction, Broadcast is used for broadcasting, and the results broadcast by other nodes are obtained to store the maximum component value of this round of iteration. Since in the AllReduce synchronization of the gradient, each node is both a sender and a receiver, and also saves the maximum component value of this round of iteration, the decompression can be directly used.

[0075] The three-value summation method of step 3 is specifically:

[0076] First, decoding: obtain the number of bits occupied by each component in the encoding, read the buffer data in UINT32 format, use the mask to perform AND operation with the data block to obtain the component data.

[0077] Second, addition and encoding: after the component data is added, it is written into the UINT32 format buffer in bits and encoded.

[0078] Finally, shift the addition result and the mask, and loop the decoding, addition, and encoding of the next data component.

[0079] The summation method is built on the Gloo library. Gloo is a high-performance parallel computing communication library developed by Facebook, which provides an MPI-compatible interface and can provide better performance than OpenMPI. The specific summation method is built in the sum operation file of the Gloo library, adding a three-value-oriented summation method and writing encode and decode functions.

[0080] The buffer expansion method used in step 4 is specifically:

[0081] According to the number of bits occupied by each component in the encoding and the number of all components, the buffer size of the receiver is expanded, and the entry pointer of each gradient is updated.

[0082] The buffer expansion method is also built in the Horovod code. Buffer expansion is to ensure that after each round of AllReduce method, the result has enough storage space to store all complete data. Due to the incremental iteration method, the final result will be several times larger than the address space occupied by the sender after the first compression and sending of data, so the buffer needs to be expanded to ensure that data is not lost. All the information needed for expansion is encoded into the binary data of the specific data, and finally the entry pointer of the gradient is updated to ensure maximum compatibility with the original system.

[0083] The decompression algorithm used in step 5 is the inverse operation of the compression algorithm in step 2, and its specific steps are as follows:

[0084] Step 5.1: Read the number of bits occupied by the component stored in the last bit of the ternary encoding;

[0085] Step 5.2: Construct the bitwise AND operation number required for each shift according to the number of bits occupied by the component;

[0086] Step 5.3: Perform bitwise AND and shift operations in sequence according to the component order;

[0087] Step 5.4: Perform bit number expansion on the component values obtained after shifting to convert them to 32-bit floating point numbers;

[0088] Step 5.5: Fill the above component values into the corresponding result gradient according to the shape and size of the gradient;

[0089] Step 5.6: Multiply the gradient obtained in step 5.5 by the expansion value stored in step 2.2 to return the final decoding result.

[0090] The decompression algorithm is constructed in the Compression.py file, and the GPU method of Pytorch or TensorFlow is used to process the gradient array.

[0091] As Figure 2 shown is the pseudo code representation of the ternary compression algorithm, and the specific steps are described above.

[0092] As Figure 3 shown is the number of binary bits occupied by the range of values obtained by summing different components after each incremental iteration, and in the case where the number of bits occupied by each component does not exceed 8 bits, up to 127 nodes are supported for AllReduce operation.

[0093] As Figure 4 shown is the waiting process timeline between iterations after ternary compression, i.e., after the completion of the backward propagation of the nodes and before the start of the forward propagation.

[0094] As Figure 5 shown is a schematic diagram of compression and summation results, the upper half represents that each 32-bit floating point component will be compressed to 2-bit binary number during the first compression, with a value range from -1 to 1, and the lower half represents the value range after summation, the value range after the first summation is -2 to 2, the value range of the operation number for the second summation is different, and it needs to be decoded separately according to the number of bits occupied by the component, and the value range after summation is from -3 to 3.

[0095] As Figure 6The UINT32 encoding form after summation is shown, the last 8 bits of gray represent the mark of the number of bits occupied by each component, and the first 24 bits are the specific component data, according to the increase of the number of iterations, the number of bits occupied by each component gradually increases.

[0096] The present application studies a method for reducing the communication data volume and reducing the precision loss of distributed deep learning training under the AllReduce architecture using a ternary compression method and an incremental iteration method. In actual distributed deep learning training, the software ecology used is various, and the compression algorithm constructed based on the Horovod architecture of the present application inherits the API support of the Horovod platform for Pytorch and TensorFlow, and therefore has good compatibility and migratability.

[0097] The present application uses a randomized ternary gradient compression method to compress each component in the gradient from the original 32-bit floating point number to a binary number. In addition, the present application considers the precision loss superposition problem caused by the AllReduce synchronization architecture, and proposes an incremental iteration transmission method. Finally, the present application integrates the above algorithm based on the Horovod system to realize the optimization of the communication gradient throughput and transmission delay.

[0098] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments only, and any technical solution falling within the idea of the present application belongs to the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application should be regarded as the protection scope of the present application.

Claims

1. A gradient optimization compression method based on ternary distributed deep learning communication, characterized in that, include: Step 1: Obtain the parameter list of distributed deep learning communication, and obtain the corresponding optimization gradient list from the parameters; Step 2: Encode the gradient using a randomized ternary compression algorithm, compressing the gradient components into two binary digits and storing the number of bits occupied by each component in the encoding. Simultaneously, select an expansion value, as follows: Step 2.1: All nodes broadcast the maximum component value calculated in this iteration to all other nodes; Step 2.2: After each node obtains the maximum component value of other nodes, it selects the maximum component value of the entire system in this iteration and stores it as the expansion value; Step 2.3: Perform probability calculations using the expanded values. The result of the probability calculations represents the compressed value of each component in this iteration. Step 2.4: Multiply the probability calculation result with the sign function of each component to obtain the compressed component value; Step 2.5: Encode the compression result in Step 2.

4. Use the UINT8 data type to store the binary code of all components, merge them into UINT32 data and store them in the buffer. At the same time, take a UINT8 format space to store the number of bits of component code. Step 3: Perform AllReduce synchronization between nodes on the complete data encoded in Step 2, using a three-value summation method to iterate the incremental bit count during transmission; Step 4: After AllReduce synchronization is complete, each node has all the encoded complete data and expands the receiver's buffer according to the encoded complete data; Step 5: Call the decompression algorithm and, combined with the expansion value from Step 2, decode the encoded complete data; Step 6: Call the averaging method of AllReduce to calculate the average value of each gradient after decoding; Step 7: Calculate the updated parameters based on the average value of each gradient from Step 6.

2. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 1, characterized in that, The gradient is represented as a 32-bit floating-point number.

3. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 1, characterized in that, In step 2.1, all nodes use the Broadcast communication method to broadcast the maximum component value calculated in this iteration to all other nodes.

4. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 1, characterized in that, In the AllReduce synchronization described in step 3, each node is both a sender and a receiver.

5. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 1, characterized in that, The specific method for summing the three values ​​described in step 3 is as follows: First, decode the data: obtain the number of bits occupied by each component in the encoding, read the buffer data according to the UINT32 format, and use the mask to AND the data block to obtain the data of that component. Next, summation and encoding are performed: the component data are summed and then written bit by bit into a UINT32 format buffer for encoding. Finally, the summation result and the mask are shifted, and the process of decoding, summing, and encoding the next data component is repeated.

6. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 1, characterized in that, The buffer expansion method used in step 4 is as follows: The receiver's buffer size is expanded according to the number of bits occupied by each component in the encoding and the total number of components, while the entry pointer of each gradient is updated.

7. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 6, characterized in that, The buffer expansion method is built into the Horovod code. All the information required for expansion is encoded into the binary data of the specific data. Finally, the gradient entry pointer is updated to ensure maximum compatibility with the original system.

8. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 1, characterized in that, The specific steps of the decompression algorithm used in step 5 are as follows: Step 5.1: Read the number of bits occupied by the last stored component in the encoding; Step 5.2: Construct the bitwise AND operands required for each shift based on the number of bits occupied by the components; Step 5.3: Perform bitwise AND and shift operations sequentially according to the component order; Step 5.4: Expand the number of bits in the component value obtained after shifting; Step 5.5: Fill the component values ​​into the corresponding result gradient according to the shape and scale of the gradient; Step 5.6: Multiply the gradient obtained in step 5.5 by the expansion value and return it as the final decoding result.

9. The gradient optimization compression method based on ternary distributed deep learning communication according to claim 8, characterized in that, The decompression algorithm is built in the Compression.py file and uses GPU methods of PyTorch or TensorFlow to process the gradient array.

Images