Distributed training communication acceleration method for sparse model

Abstract

The invention discloses a distributed training communication acceleration method for a sparse model, and the method comprises the steps: block-based sparseness: calculating the sparseness of a whole block according to a gradient value in a locally obtained gradient block in each round, and selecting a part of gradient blocks for transmission according to a Top-k sparseness method in a sparse format and a self-defined sparse ratio, and the aggregator node and the working node are in cooperative communication based on the bitmap and are used for receiving a local bitmap array, performing statistics and predictable synchronous flow gradient aggregation based on the bitmap. A 100Gbps test platform is used for evaluation, five servers are included in total, one server serves as an aggregator node, the other servers serve as working nodes, the nodes communicate with one another through a 100Gbps RDMA network, reference models tested by the method come from a plurality of real-world deep learning models including DeepLight, LSTM, BERT and NCF, and the deep learning models are used for testing a plurality of real-world deep learning models including DeepLight, LSTM, BERT and NCF. The models are obtained by training a plurality of open source data sets, so that the aggregation performance is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of distributed training, and in particular relates to a distributed training communication acceleration method for sparse models. Background Art

[0002] Deep Neural Network (DNN) is becoming the core component of more and more applications, covering multiple research fields such as natural language processing and computer vision. DNN models need to be properly trained with a large amount of data to achieve satisfactory performance. In the field of deep learning in recent years, the size of deep learning models and training data sets has increased dramatically. Obviously, this is a very time-consuming and complex task. In order to solve this problem, distributed deep learning training technology has emerged. The distributed training process mainly consists of two parts: computing and communication. In recent years, the size of DNN models is increasing at a faster rate than the computing power of hardware. With the rapid development of hardware technology, the computing speed of nodes has been increasing by orders of magnitude, and the communication required for the aggregation of gradients at each node is increasingly becoming a bottleneck affecting the speed of model training.

[0003] At present, many works have proposed high-performance communication optimization solutions for DNN distributed training, such as reducing the amount of data that needs to be transmitted during communication through gradient compression, decoupling the dependencies between computing and communication through scheduling, etc. However, as the model size increases rapidly, the sparsity of the model gradient is also gradually increasing - most of the gradients generated by the nodes in each iteration during the training process are zero, and only a few are non-zero. This feature is widely present in large-scale deep neural network training tasks, but has not received sufficient attention. This feature fundamentally distinguishes sparse model training from dense model deep training. For models with high sparsity, the most common practice is to use key-value pairs to store non-zero gradients, explicitly notifying the parameter server of the location of non-zero gradients, thereby avoiding the transmission of zero gradients. This format dedicated to representing sparse vectors often consists of two parts (key and value), where the key refers to the index position in the matrix where the gradient is located, and the value refers to the size of the current gradient. In distributed sparse training, each worker encodes the gradient vector as a list of key-value pairs and sends only the non-zero vector to the PS for aggregation in each iteration. Therefore, the parameters involved in the gradients transmitted by different workers may not overlap.

[0004] In addition to applying sparse format gradients, most of the other acceleration solutions are based on the Ri ng-All Reduce architecture for gradient transmission. AGsparse is a transmission method based on All Gather. It is based on the Ri ng-All Reduce architecture, calls All Gather twice to collect the keys and values ​​of sparse tensors, and performs local aggregation in each process. SparCML is also a communication library based on the Ri ng-All Reduce architecture. On the basis of AGsparse, it describes different situations of gradients by building a delay-bandwidth cost model, thereby making a trade-off between small messages and large messages and deciding whether the gradient format should remain sparse or become dense. Ky lix is ​​also a sparse All Reduce method based on All Gather. It uses the Butterfly network to implement cross-layer network communication of the cluster, thereby reducing the total communication volume of the system.

[0005] Although the All Reduce architecture is widely used in practice, it also has some problems and challenges, one of the main problems is the communication overhead. In the All Reduce operation, all nodes need to communicate globally to transmit partial results of local calculations to other nodes, which will lead to a lot of network bandwidth consumption and delay, especially in large-scale distributed environments. As the scale of distributed systems increases, the overhead of All Reduce operations will increase significantly. Due to the increase in communication volume and delay, its performance may be limited and cannot be effectively expanded to thousands or even tens of thousands of nodes. Summary of the invention

[0006] The purpose of the present invention is to propose an efficient sparse model distributed training acceleration solution, which focuses on improving the communication performance in sparse model training through a block-based sparsification algorithm and a bitmap-based predictable synchronous streaming gradient aggregation algorithm.

[0007] The object of the present invention can be achieved by the following technical solutions: a distributed training communication acceleration method for sparse models, including a block-based sparsification algorithm and a bitmap-based predictable synchronous streaming gradient aggregation algorithm;

[0008] Block-based sparsification algorithm: The sparsity of the entire block is calculated based on the gradient values ​​in the local gradient block in each round. Based on the Top-k sparsification method in the sparse format, a part of the gradient block is selected for transmission according to the custom sparsity ratio.

[0009] Predictable synchronous streaming gradient aggregation algorithm based on bitmap: It provides advance coordination between aggregator nodes and working nodes, and records the transmission status of gradients through bitmaps, so that all nodes can communicate in a planned manner during aggregation.

[0010] Preferably, the block-based sparseness algorithm is specifically:

[0011] Step 1: Calculate the sparsity of the entire block based on the gradient values ​​in the local gradient block obtained in each round. The specific calculation method is the number of non-zero gradient values ​​in a gradient block divided by the total number of gradient values.

[0012] Step 2: According to the sparse ratio Krat io, the gradient blocks with sparsity lower than the threshold are marked as gradient blocks that need to be transmitted;

[0013] Step 3: Mark the gradient blocks that need to be transmitted and record them in the local view maintained by the working node. Unmarked gradient blocks do not need to be transmitted, thereby reducing the amount of communication data.

[0014] Preferably, the predictable synchronous streaming gradient aggregation algorithm based on bitmap is specifically:

[0015] Consider the aggregation of a stream. Assume that a stream has N gradient blocks;

[0016] The working node first performs the sparseness of the gradient blocks of the entire stream, and then records the transmission status of all gradient blocks in the entire stream through a bitmap array with a length of N bits. Before the aggregation begins, the working node first sends the local bitmap array to the aggregator node. After the aggregator node receives the bitmaps from all working nodes, it accumulates the total aggregation times of each gradient block and records the results in the aggcount array of the local aggregator node. When the gradient blocks are aggregated, the aggregator node can query the array storing the above calculation results and use it as the number of times the gradient needs to be received. After the aggregator receives all the gradient block data, it returns the aggregation results to all working nodes.

[0017] Preferably, the result of calculating the number of aggregations of the gradient block is recorded specifically as follows: a count operation is performed on each bit in the bitmap, and if the bit is 1, the count is incremented by one.

[0018] Preferably, for the gradient blocks that are not sent by all working nodes, it is necessary to instruct the working nodes to ignore them in advance when sending.

[0019] The beneficial effects of the present invention are as follows:

[0020] The present invention uses a 100Gbps test platform for evaluation, which includes a total of 5 servers, one of which is used as an aggregator node and the remaining servers are used as working nodes. The nodes communicate through a 100Gbps RDMA network. The benchmark models tested in this paper are derived from several real-world deep learning models, including DeepL i ght, LSTM, BERT and NCF. These models are trained using many open source data sets. This paper uses the above-mentioned applications and the gradient data actually generated in the model training to test the aggregation performance, which greatly improves the aggregation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with the accompanying drawings;

[0022] Figure 1 This is the overall workflow diagram of this method.

[0023] Figure 2 This is an algorithm introduction diagram of this method.

[0024] Figure 3 This is the aggregate performance test result of this method and the dense collective communication library NCCL.

[0025] Figure 4 This is the aggregation performance test result of this method and the sparse set communication library Omni Reduce. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Example:

[0028] like Figures 1 to 4 ,This scheme discloses a distributed training communication acceleration method for sparse models, including a block-based sparsification algorithm and a bitmap-based predictable synchronous streaming gradient aggregation algorithm;

[0029] Block-based sparsification algorithm: Calculate the sparsity of the entire block based on the gradient value in the local gradient block in each round, select a part of the gradient block for transmission based on the custom sparsity ratio according to the Top-k sparsification method in the sparse format, and predictable synchronous streaming gradient aggregation algorithm based on bitmap: Provides advance coordination between aggregator nodes and working nodes, records the transmission status of gradients through bitmaps, and enables all nodes to communicate in a planned manner during aggregation, solving the unpredictable problems in synchronous streaming aggregation with almost zero communication overhead.

[0030] The block-based sparseness algorithm is specifically:

[0031] Step 1: Calculate the sparsity of the entire block based on the gradient values ​​in the local gradient block obtained in each round. The specific calculation method is the number of non-zero gradient values ​​in a gradient block divided by the total number of gradient values.

[0032] Step 2: According to the sparse ratio Krat io, the gradient blocks with sparsity lower than the threshold are marked as gradient blocks that need to be transmitted;

[0033] Step 3: Mark the gradient blocks that need to be transmitted and record them in the local view maintained by the working node. Unmarked gradient blocks do not need to be transmitted, thereby reducing the amount of communication data.

[0034] The predictable synchronous streaming gradient aggregation algorithm based on bitmap is specifically:

[0035] Consider the aggregation of a stream. Assume that a stream has N gradient blocks;

[0036] The working node first performs the sparseness of the gradient blocks of the entire stream, and then records the transmission status of all gradient blocks in the entire stream through a bitmap array of length N. Before the aggregation begins, the working node first sends the local bitmap array to the aggregator node. After the aggregator node receives the bitmaps from all working nodes, it accumulates the total aggregation times of each gradient block and records the results in the aggcount array of the local aggregator node. When the gradient blocks are aggregated, the aggregator node can query the array storing the above calculation results and use it as the number of times the gradient needs to be received. After the aggregator receives all the gradient block data, it returns the aggregation results to all working nodes.

[0037] The result of calculating the number of aggregation times of the gradient block is recorded specifically as follows: a count operation is performed according to each bit in the bitmap, and if the count is 1, the count is increased by one.

[0038] For the gradient blocks that are not sent by all working nodes, it is necessary to instruct the working nodes to ignore them in advance when sending them. Therefore, the aggregator node that has the vision of all working nodes needs to take the responsibility of calculating the positions of these gradient blocks. The specific operation is that the aggregator node traverses the aggcount array, and sets the bitmap to 1 when it finds a position with a value of zero, which is the position of the gradient block that is not sent by all nodes. The gradient blocks at these positions do not need to be concerned by all nodes during aggregation, and they can directly jump to the aggregation of the next gradient block. Finally, the aggregator node sends the bitmap to all working nodes.

[0039] Figure 1 The overall workflow diagram of the method is given. Here, two working nodes w1 and w2 and an aggregator node Aggregator are taken as examples. The colored gradient blocks of the working nodes are gradient blocks to be transmitted, and the others are sparse gradient blocks that are not transmitted. The overall workflow of the method is mainly divided into three steps: a) block-based gradient sparsification; b) bitmap-based collaborative communication between aggregator nodes and working nodes; c) predictable synchronous streaming gradient aggregation.

[0040] First, the system executes the sparseness algorithm of the gradient block. After calculating the sparsity of all gradient blocks, it selects a certain proportion (using the sparse ratio Krat io constraint) of the gradient blocks to wait for transmission, and then records the selected results in the local bitmap array (see ①); after the calculation is completed, the working node sends the local bitmap array to the aggregator node for statistics (see ②). The aggregator adds bits and stores the aggregation times of each position in the local aggcount array (i.e., the cnt array in the figure). At the same time, it calculates the bitmap view of the gradient block position that the working node is concerned about and does not need to be updated (see ③), and feeds it back to the working node to complete the collaboration between the two (see ④). Finally, the system applies a streaming synchronous aggregation algorithm. The aggregator node determines whether it has received the aggregation results of all nodes based on the aggcount array when aggregating each gradient block.

[0041] Figure 3 The performance comparison of this method with the dense collective communication library NCCL under different sparsity, 1 aggregator node, 4 working nodes in a 100Gbps network environment (based on BERT load) is shown. The results show that the aggregation throughput performance of this method is not much different from that of NCCL when the gradient is dense; it is significantly better than NCCL when the sparsity is higher than 92%, and can achieve a speedup ratio of up to 5.7x.

[0042] Figure 4The performance comparison of this method with the current best sparse collection communication library Omn iReduce (based on BERT load) is shown under different sparsities, 1 aggregator node, 4 worker nodes in a 100Gbps network environment. The results show that the aggregation throughput performance of SMReduce is better than Omn iReduce under any sparsity, and the maximum speedup ratio can reach 1.6x. When the gradient sparsity is not high, since the proportion of non-zero blocks to all gradient blocks is large and the communication volume is also large, the metadata sent by Omn iReduce during communication becomes a pure additional overhead, while this method does not have these overheads.

[0043] The algorithm introduction procedure is as follows:

[0044] Algorithm 1 Collaborative Aggregation Algorithm

[0045]

[0046]

[0047] In summary, compared with existing schemes, this method shows better results in polymerization performance.

[0048] The above formulas are obtained by collecting a large amount of data for software simulation and selecting a formula that is close to the real value. The coefficients in the formula are set by technical personnel in this field according to actual conditions. The above are only preferred specific implementations of the present invention, but the protection scope of the present invention is not limited to this. Any technical personnel familiar with the technical field within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, make equivalent replacement or change, which should be covered within the protection scope of the present invention.

Claims

1. A distributed training communication acceleration method for sparse models, characterized in that: include: Block-based sparsification: The sparsity of the entire block is calculated based on the gradient values ​​in the local gradient block obtained in each round, and a portion of the gradient blocks with lower sparsity are selected for transmission based on the custom sparsity ratio; Predictable synchronous streaming gradient aggregation based on bitmap: It provides advance coordination between aggregator nodes and worker nodes, records the transmission of gradients through bitmaps, and enables all nodes to communicate in a planned manner during aggregation.

2. According to claim 1, a distributed training communication acceleration method for sparse models is characterized in that: The block-based sparseness algorithm is specifically: Step 1: Calculate the sparsity of the entire block based on the gradient values ​​in the local gradient block obtained in each round. The specific calculation method is the number of non-zero gradient values ​​in a gradient block divided by the total number of gradient values; Step 2: According to the sparse ratio Kratio, the gradient blocks with sparsity lower than the threshold are marked as gradient blocks that need to be transmitted; Step 3: Mark the gradient blocks that need to be transmitted and record them in the local view maintained by the working node. Unmarked gradient blocks do not need to be transmitted.

3. The distributed training communication acceleration method for sparse models according to claim 1, characterized in that: The predictable synchronous streaming gradient aggregation algorithm based on bitmap is specifically: Consider the aggregation of a stream. Assume that a stream has N gradient blocks; The working node first performs the sparseness of the gradient blocks of the entire stream, and then records the transmission status of all gradient blocks in the entire stream through a bitmap array of length N. Before aggregation begins, the working node first sends the local bitmap array to the aggregator node. After the aggregator node receives the bitmaps of all working nodes, it accumulates the total aggregation times of each gradient block and records the results in the aggcount array of the local aggregator node. When the gradient blocks are aggregated, the aggregator node can query the array storing the above calculation results and use it as the number of times the gradient needs to be received. After the aggregator receives all the gradient block data, it returns the aggregation results to all working nodes.

4. The distributed training communication acceleration method for sparse models according to claim 3 is characterized in that: The result of calculating the number of aggregation times of the gradient block is recorded specifically as follows: a count operation is performed according to each bit in the bitmap, and if the count is 1, the count is increased by one.

5. The distributed training communication acceleration method for sparse models according to claim 3, characterized in that: For gradient blocks that are not sent by all worker nodes, it is necessary to instruct the worker nodes to ignore them in advance when sending.

Images