Distributed training communication optimization method and system for bandwidth-constrained environments

Abstract

The present invention provides a distributed training communication optimization method and system for a bandwidth-constrained environment, which relates to the fields of deep learning and distributed system technology, including: step S1: adopting a distributed training platform and using a data parallel mode; step S2: establishing a deep learning model, and having all workers in the distributed training platform perform data loading, forward propagation, backpropagation, parameter communication, and parameter update operations in sequence to train the deep learning model; step S3: based on the above steps, comprehensively using delayed communication, asynchronous communication, and layered communication optimization methods to iterate the deep learning model. The present invention fully utilizes the heterogeneity of device interconnection through a layered strategy, while the local asynchronous strategy alleviates the network communication bottleneck between nodes while ensuring high convergence efficiency of model training.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and distributed system technology, and in particular to a distributed training communication optimization method and system for bandwidth-constrained environments. Background Art

[0002] In recent years, with the development of increasingly large training datasets and increasingly complex models, deep learning has achieved unprecedented success in various practical applications, such as computer vision, natural language processing, and speech recognition. These tremendous advances have brought new challenges to deep learning systems while also ushering in a continuous stream of system innovations. Distributed training is one of the most important innovations, and distributed training systems are becoming the "new normal" for deep learning systems.

[0003] There are two main reasons for the popularity of distributed deep learning. First, the need for high training efficiency. Although the computing power of modern hardware (such as GPUs and TPUs) has rapidly improved in recent years, training complex models on a single device can still be prohibitively expensive. For example, training ResNet-50 on the ImageNet dataset takes approximately two days on an Nvidia V100 GPU, and training the popular NLP model BERT on a single GPU can take several months. Therefore, scaling deep learning systems to a large number of computing devices to leverage larger computing resources and accelerate learning is an inevitable trend. Second, the memory requirements of a single device are insufficient to accommodate the growing number of models and train them in batches. Deep learning tasks are memory-intensive, and the memory requirements of a deep learning system are typically determined by the model size and batch size. Today, some widely used mainstream models have millions to billions of parameters. For example, the recently released language model GPT-2 has 1.5 billion parameters, and a single copy of the model requires approximately 6GB of memory. Therefore, to address the memory shortage issue, the need to split model training across multiple devices becomes apparent. Given the necessity and importance of distributed deep learning, we have seen significant efforts from both academia and industry to explore and develop this field over the past few years. Supported by well-researched distributed learning algorithms and high-performance communication primitives, several efficient systems (e.g., PyTorch-DDP, Tensorflow, Horovod, DeepSpeed) have made significant progress, driving a wide range of machine learning-related research and applications. Together, these newly proposed systems have shaped the trends in distributed deep learning:

[0004] Trend 1: Distributed deep learning theory and algorithms are flourishing. Following the simplest data-parallel algorithms, researchers have been actively exploring more efficient system communication algorithms, such as asynchronous training, decentralized training, and compressed training. Each of these algorithms has opened up new research topics in distributed deep learning algorithms. All of this work has demonstrated, both theoretically and practically, that algorithms combined with distributed training systems can effectively reduce communication overhead while ensuring accurate convergence.

[0005] Trend 2: Communication overhead remains the primary factor limiting the scalability of distributed machine learning systems. Currently, the scalability of mainstream deep learning distributed systems remains far from satisfactory, especially for communication-intensive tasks such as VGG and GPT. Although existing systems have been carefully optimized for performance, their communication overhead typically increases rapidly with model size and the number of devices. Further reducing communication costs and achieving better scalability are key challenges facing existing systems.

[0006] Communication overhead remains the primary factor limiting the scalability of distributed machine learning systems. Currently, the scalability of mainstream data-parallel deep learning distributed systems still faces numerous bottlenecks, especially for communication-intensive neural networks such as VGG and BERT. Although existing systems have been optimized for performance, with the increasing complexity of models and the number of devices, distributed deep learning training tasks still struggle to achieve ideal performance in practical application scenarios. The main challenges are:

[0007] (1) Severe dependence on network equipment. In a multi-machine multi-GPU scenario, machines communicate with each other via Ethernet. Generally speaking, network bandwidth will become the main reason restricting the scalability of distributed training. In actual application scenarios, configuring high-performance network cards for GPU clusters requires high costs.

[0008] (2) Unable to fully utilize the heterogeneity of cluster interconnection. In actual clusters, high-performance interconnection devices such as PCB and NVLink are usually used within computing nodes, while Ethernet communication is used between nodes. Under the current mainstream data parallel architecture, due to the limited communication bandwidth between nodes, the high-performance interconnection within the nodes will be greatly wasted.

[0009] Existing methods cannot simultaneously optimize both communication overhead and statistical efficiency. Synchronous communication algorithms have high convergence efficiency but high communication overhead, while asynchronous communication algorithms can solve communication bottlenecks but cannot guarantee statistical efficiency. Summary of the Invention

[0010] In view of the deficiencies in the prior art, the present invention provides a distributed training communication optimization method and system for a bandwidth-constrained environment.

[0011] According to the present invention, a distributed training communication optimization method and system for bandwidth-constrained environments are provided, and the solution is as follows:

[0012] In a first aspect, a method for optimizing distributed training communication in a bandwidth-constrained environment is provided, the method comprising:

[0013] Step S1: Use a distributed training platform and a data parallel model;

[0014] Step S2: Establish a deep learning model, and perform data loading, forward propagation, back propagation, parameter communication, and parameter update operations on all workers in the distributed training platform in sequence to train the deep learning model;

[0015] Step S3: Based on the above steps, the optimization methods of delayed communication, asynchronous communication and layered communication are comprehensively used to iterate the deep learning model.

[0016] Preferably, each worker runs three components: a computation process, a communication process, and a parameter optimizer.

[0017] The computation process uses the PyTorch framework to perform forward propagation and backward propagation on the deep learning model and calculate the gradient of the model;

[0018] The communication process cooperates with the calculation process to transfer model parameters between workers and synchronize the overall model according to different strategies to ensure the convergence efficiency of the overall model.

[0019] The parameter optimizer updates the model parameters using the gradients calculated by the computation process.

[0020] Preferably, the worker internal operation includes:

[0021] 1) Initialize the model so that all workers start from the same model W (0) Start training, split the data set according to the total number of workers, read the data, and start iterative training;

[0022] 2) In the tth iteration, the computation process and the communication process are started in parallel:

[0023] Calculation process: Based on the data set read in, using the model W (t) Perform forward propagation and back propagation to calculate the gradient G (t) ;

[0024] Communication process: Model W (t)Send to the internal network and external network for parameter communication, and calculate and return according to different communication strategies

[0025] 3) The model obtained by parameter communication of the communication process And the gradient G obtained by the calculation process (t) , update the model in the parameter optimizer to complete this round of iteration.

[0026] Preferably, the delayed communication optimization method in step S3 is based on a model averaging method, uses local model parameters to calculate the model gradient, and updates the global model parameters after the last iterative synchronization.

[0027] Preferably, the delayed communication optimization method specifically includes:

[0028] Assume that there are K workers participating in the communication. In the tth iteration, the following processes are started in parallel:

[0029] Calculation process: Calculate the worker local gradient G through data reading, forward propagation, and back propagation (t) ;

[0030] Communication process: Using local model W (t) Perform aggregation and averaging operations with other workers to obtain

[0031] Wait for the end of this iteration calculation process and use the communication result of the previous iteration To update the parameters, you need to wait for the communication process of the previous iteration to end, and the result of this communication It is used for the next iteration.

[0032] Preferably, the asynchronous communication optimization method, for scenarios with limited bandwidth, where the communication time of the next iteration is much longer than the calculation time, decouples the calculation process from the communication process. The calculation process can start the next round of calculation without waiting for the previous round of communication to be completed. After each round of communication is completed, the local model is updated and the next round of communication is started.

[0033] Preferably, the layered communication optimization method abstracts the overall communication into two layers: communication within a single machine and communication between machines. Different communication strategies are adopted according to the bandwidth characteristics of each layer of communication to fully utilize the bandwidth within and between machines.

[0034] Based on layered communication, a local asynchronous approach is adopted, that is, parameter communication is performed synchronously within the machine and asynchronously outside the machine. The process is as follows:

[0035] 1) Given a cluster with N machines, each with n GPUs, build a hierarchical network with each machine in the cluster as a node and each GPU computing device as a rank. Select Node-k and Rank-j devices for analysis.

[0036] 2) At the tth iteration, the computation process and the communication process are started in parallel; the computation process calculates the gradient of this round The communication process is divided into two parts: internal communication and external communication;

[0037] 3) Internal communication: In the local ring network built inside Node-k, Rank-j uses the local model Perform Ring-Allreduce operations with other ranks to obtain a local average model

[0038] 4) External communication:

[0039] Off-machine aggregate averaging: In the external ring network built between nodes, Node-k_Rank-0 uses a local average model Conduct external Ring-Allreduce communication with other Node_Rank-0 to obtain the global average model

[0040] In-machine broadcast: Each Node’s Rank-0 is broadcasted Synchronize to all Ranks;

[0041] Process locking and releasing: When starting external communication, a lock m is acquired at the same time jk , until released after the external communication is completed;

[0042] 5) Internal parameter update: At the end of each iteration, based on the delayed communication, the local average model of Node-k in the previous round is used Perform local parameter updates;

[0043] 6) Global parameter update, check lock m at the end of each iteration jk Whether to release, if released, it means the global average model Ready, will Participate in model updating as a model correction item.

[0044] Preferably, the optimization method of step S3 further includes parameter quantization, that is, during the communication process, the model parameters are quantized using half-precision floating point numbers FP16;

[0045] The parameter quantization first calculates the difference Δ between the two iterative models t =W(t) -W (t-1) , for Δ t After quantizing the model parameters, communication is performed and the final communication result is This is done by taking a delayed communication approach and superimposing it on the updates of the local model.

[0046] Preferably, the optimization method of step S3 further includes decentralization, including two decentralized communication strategies, Ring and Round-Robin, using a decentralized mode of two-to-two communication, so that the communication overhead of a single machine remains unchanged as the distributed scale expands;

[0047] Among them, Ring's decentralized communication strategy: the communication method adopts direct transmission between adjacent nodes, and each node performs synchronous Allreduce internally. However, nodes directly adopt asynchronous one-way communication. After the communication is initiated, each node sends parameters to the next adjacent node and receives parameters sent from the previous adjacent node. The received parameters are averaged with the local parameters and updated according to the delayed communication.

[0048] Round-Robin decentralized communication strategy: Nodes are dynamically connected in pairs. Each communication is conducted only between two nodes. Synchronous Allreduce is performed within the node, while asynchronous point-to-point communication is used outside the node. Before each communication is initiated, it is necessary to wait for RoundRobinSelector to select peer nodes and select the next peer node of each node. Subsequently, each node sends its own model parameters to the peer node and receives the model parameters of the peer node at the same time. After averaging the two, delayed communication is used for iterative updates.

[0049] In a second aspect, a distributed training communication optimization system for a bandwidth-constrained environment is provided, the system comprising:

[0050] Model M1: uses a distributed training platform and a data-parallel model;

[0051] Model M2: Build a deep learning model and train all workers in the distributed training platform by sequentially performing data loading, forward propagation, backpropagation, parameter communication, and parameter update operations on the deep learning model.

[0052] Model M3: Based on the above modules, it uses a combination of delayed communication, asynchronous communication, and layered communication optimization methods to iterate the deep learning model.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention optimizes the communication / computation overlap ratio by rationally scheduling the time relationship between parameter communication and parameter update;

[0055] 2. This invention proposes a local asynchronous strategy to decouple communication and computation at this level and achieve non-blocking iteration;

[0056] 3. The present invention adopts a hierarchical strategy to optimize the algorithm for heterogeneous interconnection bandwidth according to the actual interconnection equipment conditions of the cluster, making full use of the interconnection bandwidth at each level;

[0057] 4. The present invention explores semi-precision quantization optimization and decentralized optimization of the proposed algorithm, which can alleviate the communication bottleneck in the synchronous strategy and reduce the iteration / communication ratio in the asynchronous strategy, thereby ensuring the training convergence effect.

[0058] Other beneficial effects of the present invention will be explained through the introduction of specific technical features and technical solutions in the specific implementation methods. Those skilled in the art should be able to understand the beneficial technical effects brought about by the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0060] Figure 1 This is a schematic diagram of the AS-SGD algorithm architecture;

[0061] Figure 2a-2b Schematic diagram of the communication mechanism of gradient averaging SGD and delayed communication SGD;

[0062] Figure 3a-3b Ring-Allreduce and layered Allreduce distributed training architectures;

[0063] Figure 4 This is a diagram of Ring's decentralization strategy;

[0064] Figure 5 Schematic diagram of the Round-Robin decentralization strategy;

[0065] Figure 6 This is the round-robin selection table for the Round-Robin decentralization strategy. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0067] The embodiment of the present invention provides a distributed training communication optimization method for a bandwidth-constrained environment, referring to Figure 1 As shown, the method specifically includes the following contents:

[0068] Step S1: Use a distributed training platform and use a data parallel model.

[0069] Step S2: Establish a deep learning model, and perform data loading, forward propagation, back propagation, parameter communication, and parameter update operations on all workers in the distributed training platform in sequence to train the deep learning model.

[0070] Each worker runs three components: a computation process, a communication process, and a parameter optimizer. The computation process uses the PyTorch framework to perform forward and backward propagation on the deep learning model and calculate the model's gradients. The communication process, in conjunction with the computation process, is responsible for transferring model parameters between workers and synchronizing the overall model according to different strategies to ensure the convergence efficiency of the overall model. The parameter optimizer uses the gradients calculated by the computation process to update the model parameters.

[0071] The internal operations of the worker include:

[0072] 1) Initialize the model so that all workers start from the same model W (0) Start training, split the data set according to the total number of workers, read the data, and start iterative training;

[0073] 2) In the tth iteration, the computation process and the communication process are started in parallel:

[0074] Calculation process: Based on the data set read in, using the model W (t) Perform forward propagation and back propagation to calculate the gradient G (t) ;

[0075] Communication process: Model W (t) Send to the internal network and external network for parameter communication, and calculate and return according to different communication strategies

[0076] 3) The model obtained by parameter communication of the communication process And the gradient G obtained by the calculation process (t) , update the model in the parameter optimizer to complete this round of iteration.

[0077] Step S3: Based on the above steps, the optimization methods of delayed communication, asynchronous communication and layered communication are comprehensively used to iterate the deep learning model.

[0078] The delayed communication optimization method in step S3 uses model averaging to calculate the model gradient using local model parameters and updates the global model parameters synchronized at the previous iteration. By delaying parameter synchronization for one iteration, the overlap between computation time and communication time is further increased, thereby reducing communication overhead.

[0079] The delayed communication optimization method specifically includes:

[0080] Assume that there are K workers participating in the communication. In the tth iteration, the following processes are started in parallel:

[0081] Calculation process: Calculate the worker local gradient G through data reading, forward propagation, and back propagation (t) ;

[0082] Communication process: Using local model W (t) Perform aggregation and averaging operations with other workers to obtain

[0083] Wait for the end of this iteration calculation process and use the communication result of the previous iteration To update the parameters, you need to wait for the communication process of the previous iteration to end, and the result of this communication It is used for the next iteration.

[0084] Asynchronous communication optimization addresses bandwidth-constrained scenarios where the communication time for the next iteration is significantly longer than the computation time. This decouples the computation and communication processes, allowing the computation process to begin the next round of computation without waiting for the previous round of communication to complete. After each round of communication completes, the local model is updated and the next round of communication begins. By eliminating the computation waiting time, the training system's throughput is further improved.

[0085] A layered communication optimization approach addresses the significant differences in communication bandwidth between GPUs within a machine and between GPUs across machines in a general GPU cluster. This approach abstracts overall communication into two layers: intra-machine communication and inter-machine communication. Based on the bandwidth characteristics of each layer, different communication strategies are adopted to fully utilize the bandwidth within and between machines.

[0086] Based on layered communication, a local asynchronous approach is adopted, that is, parameter communication is performed synchronously within the machine and asynchronously outside the machine. The process is as follows:

[0087] 1) Given a cluster with N machines, each with n GPUs, build a hierarchical network with each machine in the cluster as a node and each GPU computing device as a rank. Select Node-k and Rank-j devices for analysis.

[0088] 2) At the tth iteration, the computation process and the communication process are started in parallel; the computation process calculates the gradient of this round The communication process is divided into two parts: internal communication and external communication;

[0089] 3) Internal communication: In the local ring network built inside Node-k, Rank-j uses the local model Perform Ring-Allreduce operations with other ranks to obtain a local average model

[0090] 4) External communication:

[0091] Off-machine aggregate averaging: In the external ring network built between nodes, Node-k_Rank-0 uses a local average model Conduct external Ring-Allreduce communication with other Node_Rank-0 to obtain the global average model

[0092] In-machine broadcast: Each Node’s Rank-0 is broadcasted Synchronize to all Ranks;

[0093] Process locking and releasing: When starting external communication, a lock m is acquired at the same time jk , until released after the external communication is completed;

[0094] 5) Internal parameter update: At the end of each iteration, based on the delayed communication, the local average model of Node-k in the previous round is used Perform local parameter updates;

[0095] 6) Global parameter update, check lock m at the end of each iteration jk Whether to release, if released, it means the global average model Ready, will Participate in model updating as a model correction item.

[0096] The optimization method of step S3 also includes parameter quantization, that is, during the communication process, the model parameters are quantized using half-precision floating point numbers FP16; thereby further reducing the amount of transmitted data and lowering the requirements for system bandwidth without almost affecting the model performance.

[0097] Parameter quantization first calculates the difference Δ between the two iterative models t =W (t) -W (t-1) , for Δ t After quantizing the model parameters, communication is performed and the final communication result is This is done by taking a delayed communication approach and superimposing it on the updates of the local model.

[0098] The optimization method of step S3 also includes decentralization. To address the problems of high communication overhead and poor scalability of current mainstream synchronous communication methods in large-scale distributed training scenarios, two decentralized communication strategies, Ring and Round-Robin, are used. A decentralized mode of two-to-two communication is used, so that the communication overhead of a single machine remains unchanged as the distributed scale expands, further enhancing the scalability of distributed training.

[0099] Among them, Ring's decentralized communication strategy: the communication method adopts direct transmission between adjacent nodes, and each node performs synchronous Allreduce internally. However, nodes directly adopt asynchronous one-way communication. After the communication is initiated, each node sends parameters to the next adjacent node and receives parameters sent from the previous adjacent node. The received parameters are averaged with the local parameters and updated according to the delayed communication.

[0100] Round-Robin decentralized communication strategy: Nodes are dynamically connected in pairs. Each communication is conducted only between two nodes. Synchronous Allreduce is performed within the node, while asynchronous point-to-point communication is used outside the node. Before each communication is initiated, it is necessary to wait for RoundRobinSelector to select peer nodes and select the next peer node of each node. Subsequently, each node sends its own model parameters to the peer node and receives the model parameters of the peer node at the same time. After averaging the two, delayed communication is used for iterative updates.

[0101] The present invention also provides a distributed training communication optimization system for bandwidth-constrained environments. The distributed training communication optimization system for bandwidth-constrained environments can be implemented by executing the process steps of the distributed training communication optimization method for bandwidth-constrained environments. That is, those skilled in the art can understand the distributed training communication optimization method for bandwidth-constrained environments as a preferred embodiment of the distributed training communication optimization system for bandwidth-constrained environments. The system includes:

[0102] Model M1: uses a distributed training platform and a data-parallel model;

[0103] Model M2: Build a deep learning model and train all workers in the distributed training platform by sequentially performing data loading, forward propagation, backpropagation, parameter communication, and parameter update operations on the deep learning model.

[0104] Model M3: Based on the above modules, it uses a combination of delayed communication, asynchronous communication, and layered communication optimization methods to iterate the deep learning model.

[0105] Next, the present invention will be described in more detail.

[0106] The present application provides a distributed training communication optimization method for bandwidth-constrained environments, which optimizes the communication / computation overlap ratio by reasonably scheduling the time relationship between parameter communication and parameter update. At the same time, on this basis, according to the actual situation of the interconnected devices in the cluster, a hierarchical strategy is adopted to optimize the algorithm for the heterogeneous interconnection bandwidth, making full use of the interconnection bandwidth of each level. For the level with a large communication bandwidth bottleneck, a local asynchronous strategy is proposed to decouple the communication and calculation of this level to achieve non-blocking iteration. Finally, the present invention also explores semi-precision quantization optimization and decentralized optimization of the proposed algorithm, with the aim of alleviating the communication bottleneck in the synchronous strategy and reducing the iteration / communication ratio in the asynchronous strategy, thereby ensuring the training convergence effect.

[0107] like Figure 1This is the AS-SGD algorithm architecture diagram (Asynchronous and Synchronous (AS) - Stochastic Gradient Descent (SGD)), with each machine in the cluster as a node, and within each node, a deep learning training task runs in parallel on each GPU computing device Rank. All ranks on all nodes, that is, all workers, perform distributed training of the model in a data-parallel manner, and perform data loading, forward propagation, backpropagation, parameter communication, and parameter update operations in sequence. During the parameter communication process, each node is interconnected through high-performance PCIe (Peripheral Component Interconnection Express (PCIe)) / Nvlink (a set of buses and communication protocols developed and proposed by NVIDIA) to build an internal communication network, while the nodes are connected through Ethernet to build an external communication network.

[0108] The overall process of the AS-SDG algorithm is as follows. Each worker runs three components: the computation process, the communication process, and the parameter optimizer. The overall process of the algorithm consists of three parts:

[0109] (1) Initialize the model so that all workers start from the same model W (0) Start training, split the data set according to the total number of workers, read the data, and start iterative training.

[0110] (2) In the tth iteration, the computation process and the communication process are started in parallel:

[0111] Calculation process: Based on the data set read in, using the model W (t) Perform forward propagation and back propagation to calculate the gradient G (t) ;

[0112] Communication process: Model W (t) Send to the internal network and external network for parameter communication, and calculate and return according to different communication strategies

[0113] (3) The model obtained by parameter communication of the communication process And the gradient G obtained by the calculation process (t) , update the model in the parameter optimizer to complete this round of iteration.

[0114] Figure 2a-2bThis is a diagram of the communication mechanism of the gradient averaging SGD and delayed communication SGD algorithms. In the distributed training of deep neural networks, parallel mini-batch SGD is widely used in multi-machine and multi-card training scenarios to accelerate the training process. It uses multiple GPUs to sample local stochastic gradients in parallel, aggregates all gradients in a single server to obtain the average value, and uses SGD updates with average gradients to update the local model of each GPU. Ideally, compared with the SGD of a single worker, parallel mini-batch SGD can achieve a linear speedup in training time (relative to the number of workers). However, as the number of workers increases, the demand for gradient communication continues to grow, and this linear scalability is significantly limited in practice. One of the main reasons is that when taking this average gradient approach, the communication can only be started after waiting for the gradient calculation results during the calculation process. Although mainstream work such as Pytorch (an open source Python machine learning library) has proposed a method for parallel backpropagation and gradient communication, which can hide the communication overhead under backpropagation, such as Figure 2a This method can achieve a certain degree of communication / computation overlap, but when the communication overhead is greater than the backpropagation time, the computation process still needs to be blocked after the backpropagation ends and wait for the communication process to complete gradient aggregation before the gradient update can be performed. Therefore, there is still a large bottleneck.

[0115] Therefore, the present invention proposes Figure 2b The model-averaged delayed communication SGD strategy shown in the figure essentially replaces the object of aggregate communication with model parameters instead of gradients, and parameter updates are completed using the aggregated results of the previous iteration. By delaying parameter communication for one iteration, the parameters of the previous iteration can achieve maximum computation / communication overlap in the current iteration, thereby reducing communication overhead and allowing the next iteration of training to begin as soon as the current iteration ends.

[0116] The process of the delayed communication SGD algorithm is as follows, assuming that there are K workers participating in the communication. In the tth iteration, the following processes are started in parallel:

[0117] Calculation process: Calculate the worker local gradient G through data reading, forward propagation, and back propagation (t) ;

[0118] Communication process: Using local model W (t) Perform aggregation and averaging operations with other workers to obtain

[0119] Wait for the end of this iteration. At this time, the communication process of this iteration is usually not completed. In order to avoid the bottleneck, we use the communication result of the previous iteration. Update the parameters (need to wait for the communication process of the previous iteration to end), and the result of this communication It is used for the next iteration.

[0120] Figure 3a-3b It is a distributed training architecture that combines Ring-Allreduce (ring allreduce operation) and layered Allreduce (a type of collective communication operation, the goal is to efficiently integrate data from different machines and then distribute the results to each machine). Generally speaking, real cluster environments often have different communication bandwidths at multiple levels. For example, a multi-machine multi-card GPU cluster usually has high-performance interconnection bandwidth within the machine, while the interconnection bandwidth between machines is relatively low. The distributed training architecture proposed in the current mainstream deep learning framework is usually like this: Figure 3a The Ring-Allreduce architecture shown in Figure 1 constructs a communication network across all computing GPUs in a ring, ignoring the heterogeneity of communication links across the entire system. This approach treats all computing and interconnected resources as devices with uniform performance. While this improves the schedulability of computing power in a fragmented cluster and facilitates resource management and utilization, communication performance is limited by the weakest interconnection bandwidth between nodes in the cluster.

[0121] Therefore, the present invention proposes Figure 3b The layered communication architecture shown in the figure is based on the typical cluster machine composition, with the internal and external layers of the machines in the cluster as the layered basis, to build a two-layer Ring-Allreduce interconnection network. One is the internal network using PCB / NVLink interconnection within the machine, and the other is the external network using Ethernet interconnection between machines.

[0122] Since the communication overhead between machines is huge, while the communication overhead within machines is not large, in order to ensure the statistical efficiency of model training while maximizing the alleviation of the communication bandwidth bottleneck, the algorithm proposed in this invention will be implemented in a locally asynchronous manner based on the layered strategy, that is, parameter communication is performed in a synchronous manner within the machine and asynchronously outside the machine.

[0123] The specific process of the algorithm of the local asynchronous layering strategy is as follows:

[0124] (1) Given a cluster with N machines, each with n GPU devices, a hierarchical network is constructed, with each machine in the cluster as a node and each GPU computing device as a Rank. Node-k and Rank-j devices are selected for analysis.

[0125] (2) At the tth iteration, the computation process and the communication process are started in parallel. The computation process calculates the gradient of this round The communication process is divided into two parts: internal communication and external communication.

[0126] (3) Internal communication: In the local ring network built inside Node-k, Rank-j uses the local model Perform Ring-Allreduce operations with other ranks to obtain a local average model

[0127] (4)External communication:

[0128] Off-machine aggregate averaging: In the external ring network built between nodes, Node-k_Rank-0 uses a local average model Conduct external Ring-Allreduce communication with other Node_Rank-0 to obtain the global average model

[0129] In-machine broadcast: Each Node’s Rank-0 is broadcasted Synchronize to all ranks.

[0130] Process locking and releasing: When starting external communication, a lock m is acquired at the same time jk , until released after external communication ends.

[0131] (5) Internal parameter update: At the end of each iteration, according to the delayed communication strategy, the local average model of Node-k in the previous round is used. Perform local parameter updates.

[0132] (6) Global parameter update, check lock m at the end of each iteration jk Whether to release, if released, it means the global average model Ready, will Participate in model updating as a model correction item.

[0133] The following describes the half-precision quantization strategy. In the AS-SGD algorithm, this hierarchical strategy divides the communication problem into two levels: intra-node and inter-node. A synchronous, delayed communication strategy within a node allows for near-complete overlap of communication and computation. An asynchronous strategy for inter-node communication decouples the communication and computation processes, enabling non-blocking computation iterations. However, this strategy also presents a challenge: If the inter-node asynchronous communication strategy is too costly, multiple iterations may be required to complete a single communication. A high iteration / communication ratio can also degrade the model's final convergence performance. Therefore, even with the asynchronous strategy, it is still necessary to reduce the inter-node communication overhead and minimize the iteration / communication ratio to ensure convergence while achieving high speedup.

[0134] The so-called half-precision floating-point number (Float 16, FP16) is stored using 2 bytes (16 bits). Compared with 8-bit or 16-bit integers, half-precision floating-point numbers have the advantage of a high dynamic range; compared with single-precision floating-point numbers, it can save half the storage space and bandwidth. Usually in the application scenario of deep learning distributed training, we will use 32-bit single-precision floating-point numbers to calculate and save models and gradients. However, in reality, the accuracy requirements of model training are not that high. Therefore, in many research applications, in order to reduce computing and storage requirements, half-precision floating-point numbers are often used to represent certain parameters, such as gradients. By the same token, if half-precision quantization is adopted in the parameter communication in distributed training, the size of communication parameters can be effectively reduced, the communication bandwidth bottleneck can be alleviated, and the communication overhead can be reduced.

[0135] Since AS-SGD adopts the model averaging strategy, the object of communication is the model parameters. If the model parameters are directly quantized with half precision, the accuracy of the model after communication will be seriously lost. Therefore, the half-precision quantization scheme proposed in this section will first calculate the difference Δ between the two iterations of the model. t =W (t) -W (t-1) , for Δ t After half-precision quantization, communication is performed, and the final communication result is The delayed communication method proposed in the previous section is used to superimpose the update on the local model.

[0136] Figures 4 to 6 Two decentralized approaches based on the AS-SGD algorithm of this invention are presented. In a typical Ring-Allreduce aggregation operation, after parameter communication, each worker obtains a globally unified model. This communication scheme, which periodically maintains a global model, can be considered a centralized approach. However, many studies have shown that when performing distributed deep learning training aggregation communication, each communication node does not need to periodically maintain a global model. This new communication method is categorized as decentralized communication. Decentralized stochastic gradient descent is a data-parallel distributed learning algorithm that eliminates the requirement for a consistent global model across all communication nodes. This makes its communication model significantly different from Ring-Allreduce or parameter server-based algorithms. Using decentralized stochastic gradient descent, each communication node only needs to exchange data with one or a few specific communication nodes, rather than periodically aggregating data globally. In other words, in a decentralized approach, each communication node needs to exchange data with fewer nodes than in Ring-Allreduce.

[0137] Therefore, based on the above-mentioned synchronous and asynchronous cooperative communication algorithm, the present invention further proposes two decentralized attempt schemes: the Ring decentralized scheme and the Round-Robin decentralized scheme.

[0138] Figure 4 Figure 1 is a schematic diagram of the Ring decentralization strategy. This strategy is only used for inter-node communication within a hierarchical structure. The inter-node communication topology maintains the same connection structure as the Ring-Allreduce architecture, but the communication method is direct transmission between adjacent nodes, rather than global aggregation and averaging. The detailed process of the Ring decentralization strategy is as follows: Each node performs a synchronous Allreduce internally, but nodes communicate directly with each other using asynchronous one-way communication. After communication is initiated, each node sends parameters to the next adjacent node and receives parameters sent from the previous adjacent node. The received parameters are averaged with the local parameters and then updated according to the delayed communication strategy proposed in this invention.

[0139] Figure 5 This is a schematic diagram of the Round-Robin decentralization scheme. This strategy is also only used for communication between nodes in a hierarchical structure. The method cancels the ring connection between nodes and adopts a dynamic two-by-two connection method. Each communication is only carried out between two nodes. The detailed process of the Round-Robin decentralization strategy is as follows: Taking N nodes Node as an example, similar to the Ring decentralization strategy, synchronous Allreduce is performed inside the Node, while asynchronous point-to-point communication is used outside the Node. Before each communication is initiated, it is necessary to wait for the RoundRobinSelector (Round-Robin selector, used to select the next node for communication) to select a peer node. This operation uses the Round-Robin polling mechanism to select the next peer node for each Node: different peer nodes are selected in turn, starting from 1, until N-1 (N is the total number of nodes), and then the cycle is restarted. Subsequently, each Node sends its own model parameters to the peer node and receives the model parameters of the peer node at the same time. After averaging the two, the delayed communication method proposed in the present invention is used for iterative update.

[0140] The embodiments of the present invention provide a distributed training communication optimization method and system for bandwidth-constrained environments. A hierarchical strategy fully utilizes the heterogeneity of device interconnection, while a local asynchronous strategy alleviates the network communication bottleneck between nodes while ensuring high convergence efficiency of model training.

[0141] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0142] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A distributed training communication optimization method for bandwidth-constrained environments, characterized in that: include: Step S1: Use a distributed training platform and a data parallel model; Step S2: Establish a deep learning model, and perform data loading, forward propagation, back propagation, parameter communication, and parameter update operations on all workers in the distributed training platform in sequence to train the deep learning model; Step S3: Based on the previous steps, the optimization methods of delayed communication, asynchronous communication, and layered communication are comprehensively used to iterate the deep learning model; The layered communication optimization method abstracts the overall communication into two layers: internal communication within a single machine and communication between machines. Different communication strategies are adopted according to the bandwidth characteristics of each layer of communication to fully utilize the bandwidth within and between machines. Based on layered communication, a local asynchronous approach is adopted, that is, parameter communication is performed synchronously within the machine and asynchronously outside the machine. The process is as follows: 1) Given a cluster of N machines, each with n GPUs, build a hierarchical network with each machine in the cluster as a node and each GPU computing device as a rank. Select Node-k, Rank-j devices for analysis. 2) At the tth iteration, the computation process and the communication process are started in parallel; The calculation process calculates the gradient of this round ,The communication process is divided into two parts: internal communication and external ,communication; 3) Internal communication: In the local ring network built inside Node-k, Rank-j uses the local model Perform Ring-Allreduce operations with other ranks to obtain a local average model ; 4) External communication: Off-machine aggregate averaging: In the external ring network built between nodes, Node-k_Rank-0 uses a local average model Conduct external Ring-Allreduce communication with other Node_Rank-0 to obtain the global average model ; In-machine broadcast: Each Node’s Rank-0 is broadcasted via the broadcast operation. Synchronize to all Ranks; Process locking and releasing: When starting external communication, a lock is acquired at the same time , until released after the external communication is completed; 5) Internal parameter update: At the end of each iteration, based on the delayed communication, the local average model of the previous round of Node-k is used Perform local parameter updates; 6) Global parameter update, check the lock at the end of each iteration Whether to release, if released, it means the global average model Ready, will Participate in model updating as a model correction item.

2. The distributed training communication optimization method for bandwidth-constrained environments according to claim 1, characterized in that: Each worker runs three components: a computation process, a communication process, and a parameter optimizer. The computation process uses the PyTorch framework to perform forward propagation and backward propagation on the deep learning model and calculate the gradient of the model; The communication process cooperates with the calculation process to transfer model parameters between workers and synchronize the overall model according to different strategies to ensure the convergence efficiency of the overall model. The parameter optimizer updates the model parameters using the gradients calculated by the computation process.

3. The distributed training communication optimization method for bandwidth-constrained environments according to claim 2, characterized in that: The internal operation of the worker includes: 1) Initialize the model so that all workers start from the same model Start training, split the data set according to the total number of workers, read the data, and start iterative training; 2) In the tth iteration, the computation process and the communication process are started in parallel: Calculation process: Based on the data set read in, the model is used Perform forward propagation and back propagation to calculate the gradient ; Communication process: Model Send to the internal network and external network for parameter communication, and calculate and return according to different communication strategies ; 3) The model obtained by parameter communication of the communication process , and the gradient obtained by the calculation process , update the model in the parameter optimizer to complete this round of iteration.

4. The distributed training communication optimization method for bandwidth-constrained environments according to claim 1, characterized in that: The delayed communication optimization method in step S3 is based on the model averaging method, which uses local model parameters to calculate the model gradient and updates it on the global model parameters after the last iterative synchronization.

5. The distributed training communication optimization method for bandwidth-constrained environments according to claim 4, characterized in that: The delayed communication optimization method specifically includes: Assume that there are K workers participating in the communication. In the tth iteration, the following processes are started in parallel: Computation process: Calculate worker local gradients through data reading, forward propagation, and back propagation ; Communication process: Using local model Perform aggregation and averaging operations with other workers to obtain ; Wait for the end of this iteration calculation process and use the communication result of the previous iteration To update the parameters, you need to wait for the communication process of the previous iteration to end, and the result of this communication It is used for the next iteration.

6. The distributed training communication optimization method for bandwidth-constrained environments according to claim 1, characterized in that: The asynchronous communication optimization method is designed to decouple the computing process from the communication process in scenarios where bandwidth is limited and the communication time for the next iteration is much longer than the computing time. The computing process can start the next round of computing without waiting for the previous round of communication to complete. After each round of communication is completed, the local model is updated and the next round of communication is started.

7. The distributed training communication optimization method for bandwidth-constrained environments according to claim 1, characterized in that: The optimization method of step S3 also includes parameter quantization, that is, during the communication process, the model parameters are quantized using half-precision floating point numbers FP16; The parameter quantization first calculates the difference between the models obtained by two iterations ,right After quantizing the model parameters, communication is performed and the final communication result is Delayed communication is used to superimpose the updates on the local model.

8. The distributed training communication optimization method for bandwidth-constrained environments according to claim 1, characterized in that: The optimization method of step S3 also includes decentralization, including two decentralized communication strategies, Ring and Round-Robin, which use a decentralized mode of two-to-two communication to ensure that the communication overhead of a single machine remains unchanged as the distributed scale expands; Among them, Ring's decentralized communication strategy: the communication method adopts direct transmission between adjacent nodes, and each node performs synchronous Allreduce internally. However, nodes directly adopt asynchronous one-way communication. After the communication is initiated, each node sends parameters to the next adjacent node and receives parameters sent from the previous adjacent node. The received parameters are averaged with the local parameters and updated according to the delayed communication. Round-Robin decentralized communication strategy: Nodes are dynamically connected in pairs. Each communication is conducted only between two nodes. Synchronous Allreduce is performed within the node, while asynchronous point-to-point communication is used outside the node. Before each communication is initiated, it is necessary to wait for RoundRobinSelector to select peer nodes and select the next peer node of each node. Subsequently, each node sends its own model parameters to the peer node and receives the model parameters of the peer node at the same time. After averaging the two, delayed communication is used for iterative updates.

9. A distributed training communication optimization system for bandwidth-constrained environments, characterized in that: include: Module M1: uses a distributed training platform and a data parallel model; Module M2: Establish a deep learning model, and train the deep learning model by sequentially performing data loading, forward propagation, backpropagation, parameter communication, and parameter update operations on all workers in the distributed training platform; Module M3: Based on the previous modules, it uses a combination of delayed communication, asynchronous communication, and layered communication optimization methods to iterate the deep learning model; The layered communication optimization method abstracts the overall communication into two layers: internal communication within a single machine and communication between machines. Different communication strategies are adopted according to the bandwidth characteristics of each layer of communication to fully utilize the bandwidth within and between machines. Based on layered communication, a local asynchronous approach is adopted, that is, parameter communication is performed synchronously within the machine and asynchronously outside the machine. The process is as follows: 1) Given a cluster of N machines, each with n GPUs, build a hierarchical network with each machine in the cluster as a node and each GPU computing device as a rank. Select Node-k, Rank-j devices for analysis. 2) At the tth iteration, the computation process and the communication process are started in parallel; The calculation process calculates the gradient of this round ,The communication process is divided into two parts: internal communication and external ,communication; 3) Internal communication: In the local ring network built inside Node-k, Rank-j uses the local model Perform Ring-Allreduce operations with other ranks to obtain a local average model ; 4) External communication: Off-machine aggregate averaging: In the external ring network built between nodes, Node-k_Rank-0 uses a local average model Conduct external Ring-Allreduce communication with other Node_Rank-0 to obtain the global average model ; In-machine broadcast: Each Node’s Rank-0 is broadcasted Synchronize to all Ranks; Process locking and releasing: When starting external communication, a lock is acquired at the same time , until released after the external communication is completed; 5) Internal parameter update: At the end of each iteration, based on the delayed communication, the local average model of the previous round of Node-k is used Perform local parameter updates; 6) Global parameter update, check the lock at the end of each iteration Whether to release, if released, it means the global average model Ready, will Participate in model updating as a model correction item.

Images