A method for optimizing large-scale parallel communication based on lossy compression

Abstract

The present disclosure relates to communication optimization for parallel training of large-scale models of deep learning, and specifically to a large-scale parallel communication optimization method based on lossy compression, including: dividing the original data into N×N block data; each process non-blockingly receives the block data from the previous process, and compresses the block data received by the previous round of this process based on the SZX algorithm, and then non-blockingly sends the compressed block data to the next process, and decompresses the received block data and performs reduction operations until each process holds the result of the reduced sub-block; each process communicates in parallel in a ring manner until each process holds the compressed sub-block result of all processes; each process begins to decompress all the compressed sub-block results received and store data. This method reduces the communication overhead in parallel training of large model tensors, realizes the mutual masking of the time overhead of sending and receiving and compression and decompression, and saves time cost.

Description

Technical Field

[0001] The present invention relates to deep learning large-scale model parallel training communication optimization, and specifically to a large-scale parallel communication optimization method based on lossy compression. Background Art

[0002] The SZx error-bounded lossy compression algorithm compresses the input data according to a given error e. Its basic idea is to organize the entire data into small one-dimensional data blocks and calculate the average value of the minimum and maximum values ​​of these data blocks (denoted as μ). When the deviation of μ from all values ​​in the block is within the specified error range, it can be used to represent all values ​​in the block. If the deviation of all data in the block from μ is less than the specified error, then this block is called a constant block, and only μ of this data block needs to be stored; otherwise, the IEEE 754 floating-point representation of the data in the block is analyzed according to the specified error limit to compress all data in the block.

[0003] Distributed training methods achieve large model tensor parallel training by splitting the model layers among multiple computing devices. Specifically, this involves splitting the large matrix (tensor) operations in the network layer into small blocks, and then computing these small blocks in parallel on different computing devices. For the Transformer network structure, efficient tensor parallelism for the Transformer is achieved by flexibly using parallel row and column parallelism. However, in tensor parallel training, Allreduce communication is required for the activations generated by the model. Due to the hundreds of billions of parameters in large models, the Allreduce communication for activations in tensor parallelism has a huge amount of communication volume, which affects the computational efficiency and training throughput of large model training. Summary of the invention

[0004] To solve the above problems, the present invention discloses a large-scale parallel communication optimization method based on lossy compression, the method comprising:

[0005] The original data is divided into N×N blocks of data, each process is divided into N blocks, each block holds one block of data, and the processes are run simultaneously in different computing devices;

[0006] Each process receives the block data from the previous process in a non-blocking manner, and compresses the block data received by the current process in the previous round based on the error-bounded lossy data compression algorithm and stores the compressed data size, and then sends the compressed block data to the next process in a non-blocking manner, and decompresses the received block data and performs a reduction operation, performing N-1 iterative communications until each process completes N-1 reduction operations and holds the result of the reduced sub-block.

[0007] Each process is synchronized with each other and communicates in parallel in a ring manner with a fixed buffer size. It receives the reduced sub-block results from the previous process and compresses the reduced sub-block results it holds based on the error-bounded lossy data compression algorithm and sends them to the next process. It performs N-1 iterative communications until each process holds the compressed reduced sub-block results of all processes.

[0008] After all communications are completed, each process starts to decompress all the compressed protocol sub-block results received, and stores all the protocol results obtained after decompression in the local receiving buffer of each computing device.

[0009] The block data is compressed and stored in the local send buffer of each process, and memory space is pre-allocated in front of the buffer to store the size of the compressed data;

[0010] The block start position pointer is maintained according to the size of the stored compressed data. When decompressing, the block start position pointer is used to determine where to start decompression.

[0011] The compression based on the error-bounded lossy data compression algorithm also includes:

[0012] Set the error limit e;

[0013] For each process's block data, calculate the maximum and minimum values ​​in the block, and calculate the average value μ of the block based on the minimum and maximum values;

[0014] Traverse each data point in the block and calculate its deviation from μ. If the deviations of all data points in the block are less than the error limit e, the block is considered to be a constant block. If the deviations of one or more data points in the block are greater than or equal to the error limit e, the block is considered to be a non-constant block.

[0015] If the data block is judged as a constant block, the average value μ of the block is stored;

[0016] If the data block is determined to be a non-constant block, each data point is compressed according to the IEEE 754 floating point representation;

[0017] Stores compressed data and data size.

[0018] The method of compressing each data point according to the IEEE 754 floating point representation includes:

[0019] The difference between each data in the data block and the average value μ of the block is calculated, and then the difference is encoded, and the mantissa part in the floating point representation is clipped according to the error limit e.

[0020] The compressed size of the resultant sub-block data is stored in the local integer array compressed_sizes.

[0021] When decompressing the block data compressed by the error-bounded lossy data compression algorithm, the method includes:

[0022] Determine whether the result of each reduction sub-block is a constant block;

[0023] If it is a constant block, all data points are directly restored using the average value μ;

[0024] If it is a non-constant block, the floating point number is restored based on the stored difference and the information of the clipped mantissa to ensure that the error does not exceed the given error limit e.

[0025] The present invention introduces an error-bounded lossy data compression algorithm, combines the ring-compression Allgather communication method with the ring-compression Reduce-Scatter communication method, designs a ring-compression Allreduce communication method, uses non-blocking communication in the ring-compression Reduce-Scatter, compresses block data by using the error-bounded lossy data compression algorithm, realizes mutual masking of sending and receiving time overheads and compression and decompression time overheads, saves time cost, reduces communication overhead in parallel training of large model tensors, and improves computing efficiency and training throughput in the process of parallel training of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the large-scale parallel communication optimization method based on lossy compression of the present invention;

[0027] Figure 2 A schematic diagram of compressed Allgather ring communication in one embodiment of the present invention;

[0028] Figure 3 is a timing diagram of a circular Reduce-Scatter algorithm in one embodiment of the present invention;

[0029] Figure 4 Schematic diagram of inter-process ring communication in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] The best mode of implementation of the present invention is described below by way of example. It should be understood that the specific embodiments herein are used to explain the present invention in detail and should not be construed as limiting the present invention. It should be noted that various changes and modifications may be made under the premise of following the principles and core scope of the present invention, and these changes should all be deemed to be within the scope of protection of the present invention. In conjunction with the accompanying drawings, the specific implementation steps of the present invention are described in detail.

[0031] Distributed training methods achieve large-scale model tensor parallel training by splitting the model layers among multiple computing devices. Large-scale model tensor parallel training is a technology that achieves ultra-large-scale model distributed training by splitting the model's tensors and distributing them to multiple devices. It can break through the memory limitations of a single device and allow training of models with extremely large parameter counts. It is widely used in large-scale model training in deep learning, such as GPT, BERT, etc. However, in large-scale model tensor parallel training, each round of forward and backward propagation requires frequent communication between devices. Especially when the tensors shared between devices are large, the communication overhead will significantly affect performance.

[0032] In order to solve the above problems, the present invention realizes a ring-compressed Allreduce communication optimization method for parallel training of large model tensors by combining the ring-compressed Allgather communication method with the ring-compressed Reduce-Scatter communication method.

[0033] In Reduce-Scatter communication, data is first distributed to multiple processes, each of which holds a piece of data. Then each process exchanges and reduces (for example, sums, maximizes, etc.) the data it holds, and each process finally retains only a portion of the reduced result.

[0034] By compressing data, the ring compression method can effectively reduce the amount of data transmitted between processes, thereby shortening the communication time, especially in large-scale parallel computing.

[0035] In Reduce-Scatter communication, data from all processes in the same communication domain need to be reduced and calculated with the data of other processes. Compression and decompression are required in each round of communication. Therefore, compression operations can be processed in parallel with communication through non-blocking communication to maximize the use of computing resources and network bandwidth.

[0036] In Allgather communication, each process holds a part of the data and sends this part of the data to all other processes. Eventually, each process collects the data blocks of all processes to form global complete data.

[0037] The present invention introduces the error-bounded lossy data compression algorithm SZx algorithm, thereby well maintaining the quality and accuracy of data on the basis of reducing communication overhead.

[0038] For Allgather communication, each process in the same communication domain needs to communicate with each other to exchange data. Since the transmitted data is not modified during the communication, there is no need to compress and decompress in each round of communication. It is only necessary to decompress all the compressed data received after all the communications are completed.

[0039] In Allreduce communication, each process holds a data block, and all processes perform reduction operations through communication. The reduction results are broadcast among all processes, and finally each process has a complete reduction result. By optimizing the Allgather communication method and the Reduce-Scatter communication method, a ring compression Allreduce communication optimization method can be obtained.

[0040] Each process is synchronized with each other and communicates in parallel in a ring manner with a fixed buffer size. It receives the reduced sub-block results from the previous process and compresses the reduced sub-block results it holds based on the error-bounded lossy data compression algorithm and sends them to the next process. It performs N-1 iterative communications until each process holds the compressed reduced sub-block results of all processes.

[0041] After all communications are completed, each process starts to decompress all the compressed protocol sub-block results received, and stores all the protocol results obtained after decompression in the local receiving buffer of each computing device.

[0042] The process of the ring compression Allreduce communication optimization method for parallel training of large model tensors disclosed in the present invention is as follows: Figure 1 As shown, specifically:

[0043] Step 1: Divide the original data into N×N blocks of data, divide each process into N blocks, each block holds an original sub-block of data, and the processes are run simultaneously in different computing devices;

[0044] In a specific embodiment, each computing device used for distributed training includes a process, and the processes of different computing devices operate simultaneously;

[0045] Step 2: How to set the processes Figure 4 As shown, a circular communication is performed, each process receives the block data from the previous process in a non-blocking manner, and at the same time compresses the block data received by the current process in the previous round based on the error-bounded lossy data compression algorithm and stores the compressed data size, and then sends the compressed block data to the next process in a non-blocking manner, and at the same time decompresses the received block data and performs a reduction operation, and performs N-1 iterative communications until each process completes N-1 reduction operations and holds the result of the reduced sub-block;

[0046] Specifically, the error-bounded lossy compression algorithm includes the following steps:

[0047] 301: For each reduction sub-block result, calculate the maximum value and the minimum value in the block, and calculate the average value μ of the block based on the minimum value and the maximum value.

[0048] 302: traverse each data point in the block and calculate its deviation from μ. If the deviations of all data points in the block are less than a given error e, the block is considered to be a constant block. If the deviations of one or more data points in the block are greater than or equal to e, the block is considered to be a non-constant block.

[0049] 303: If the block is determined to be a constant block, only the average value μ of the block is stored, and there is no need to store other data points in the block;

[0050] If the block is determined to be a non-constant block, each data point is compressed according to the IEEE 754 floating point representation.

[0051] In a specific embodiment, the difference between each data in the data block and the average value μ of the block is calculated, and then the difference is encoded, and the mantissa part in the floating point representation is clipped according to the error limit e.

[0052] 304: Store the compressed data and the data size.

[0053] In a specific embodiment, the communication timing of this step is as follows: Figure 3 As shown in the figure, Irecv is a non-blocking receive, from T0 to T recv During the time period of T, the program is allowed to execute the compression task while waiting for the message to arrive, thereby realizing asynchronous communication. Isend is a non-blocking send, which is used to send messages in parallel computing. recv to T send The program is allowed to continue to perform the decompression task while sending messages during the time period of T, thereby achieving higher parallelism and efficiency. send to T reduce The reduction operation is performed in the time period until the reduction is completed. Finally, each process will hold a reduction sub-block result after the global reduction.

[0054] Since the storage space occupied by each block of data after compression is uncertain, it will lead to the inability to correctly decompress the compressed data, so the compressed data of each block cannot be simply combined together. In a specific embodiment, the present invention stores the compressed data of all blocks in the local sending buffer of each computing device, and pre-allocates sufficient memory space (4 bytes per block) in front of the buffer to store the size of the compressed data. Putting these blocks together instead of storing them with the compressed data blocks is essentially an index, which is more cache-friendly and has lower overhead.

[0055] The block start position pointer is maintained according to the size of the stored compressed data. When decompressing, the block start position pointer is used to determine where to start decompression.

[0056] A compression step is added to the communication process. Instead of compressing first and then dividing into blocks and then communicating, it is divided into blocks first and then compressed. The compression speed of a single block is fast, and non-blocking communication is used to reduce communication overhead while saving time costs.

[0057] Step 3: Each process synchronizes with each other and communicates in parallel in a ring manner with a fixed buffer size, receives the reduced sub-block result from the previous process and compresses the reduced sub-block result held by itself based on the error-bounded lossy data compression algorithm and sends it to the next process, performing N-1 iterative communications until each process holds the compressed reduced sub-block results of all processes;

[0058] Each process compresses the reduced sub-block results it holds based on an error-bounded lossy data compression algorithm and stores the compressed data size;

[0059] In a specific embodiment, the ring communication process is as follows Figure 2 As shown in the figure, the red squares represent the data compression process, and the green squares represent the data decompression process.

[0060] In the first round of communication, process 0 compresses the data A it holds to obtain A c Send to process 1, and receive compressed data E sent by process 4 c , process 1 compresses the data B it holds to obtain B c Send to process 2, and receive compressed data A sent by process 0 c , and so on. After the communication is completed, each process holds two blocks of data completed by the protocol;

[0061] In the next few rounds of communication, each process sends the data received in the previous round to the next process in this round. That is, in the second round of communication, process 0 sends E c to process 1, and receive D sent by process 4 c , process 1 sends A c to process 2, and receive E sent by process 0 at the same time c , and so on. After the communication is completed, each process has three blocks of data completed by the protocol;

[0062] After the third round of communication, each process has four blocks of data with completed protocols. After the fourth round of communication, each process has five blocks of data with completed protocols. After all communications are completed, each process will decompress the compressed protocol results it holds. In this algorithm, N-1 rounds are required to obtain the collection results of each process, where N is the number of processes in the communication domain.

[0063] Step 5: After all communications are completed, each process starts to decompress all the compressed protocol sub-block results received, and stores all the protocol results obtained after decompression in the local receiving buffer of each computing device.

[0064] Ultimately, all processes will eventually get the reduced result of each process.

[0065] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large-scale parallel communication optimization method based on lossy compression, characterized in that: include: The original data is divided into N×N blocks of data, each process is divided into N blocks, each block holds one block of data, and the processes are run simultaneously in different computing devices; Each process receives the block data from the previous process in a non-blocking manner, and compresses the block data received by the current process in the previous round based on the error-bounded lossy data compression algorithm and stores the compressed data size, and then sends the compressed block data to the next process in a non-blocking manner, and decompresses the received block data and performs a reduction operation, performing N-1 iterative communications until each process completes N-1 reduction operations and holds the result of the reduced sub-block. Each process is synchronized with each other and communicates in parallel in a ring manner with a fixed buffer size. It receives the reduced sub-block results from the previous process and compresses the reduced sub-block results it holds based on the error-bounded lossy data compression algorithm and sends them to the next process. It performs N-1 iterative communications until each process holds the compressed reduced sub-block results of all processes. After all communications are completed, each process starts to decompress all the compressed protocol sub-block results received, and stores all the protocol results obtained after decompression in the local receiving buffer of each computing device; The compression based on the error-bounded lossy data compression algorithm also includes: setting an error limit e; for the block data of each process, calculating the maximum value and the minimum value in the block, and calculating the average value μ of the block according to the minimum value and the maximum value; traversing each data point in the block, calculating its deviation from μ, if the deviation of all data points in the block is less than the error limit e, then the block is considered to be a constant block, if the deviation of one or more data points in the block is greater than or equal to the error limit e, then the block is considered to be a non-constant block; if the data block is determined to be a constant block, the average value μ of the block is stored; if the data block is determined to be a non-constant block, each data point is compressed according to the representation of IEEE 754 floating point numbers; and the compressed data and data size are stored; the compression of each data point according to the representation of IEEE 754 floating point numbers includes: calculating the difference between each data in the data block and the average value μ of the block, then encoding the difference, and cutting off the mantissa part in the floating point representation according to the error limit e.

2. The large-scale parallel communication optimization method according to claim 1, characterized in that: Also includes: The block data is compressed and stored in the local send buffer of each process, and memory space is pre-allocated in front of the buffer to store the size of the compressed data; The block start position pointer is maintained according to the size of the stored compressed data. When decompressing, the block start position pointer is used to determine where to start decompression.

3. The large-scale parallel communication optimization method according to claim 1, characterized in that: Also includes: The compressed size of the resultant sub-block data is stored in the local integer array compressed_sizes.

4. The large-scale parallel communication optimization method according to claim 1, characterized in that: When decompressing the block data compressed by the error-bounded lossy data compression algorithm, the method includes: Determine whether the result of each reduction sub-block is a constant block; If it is a constant block, all data points are directly restored using the average value μ; If it is a non-constant block, the floating point number is restored based on the stored difference and the information of the clipped mantissa to ensure that the error does not exceed the given error limit e.

Images