Optimization Methods for Constrained Weight Synchronous Parallel Models in Distributed Deep Learning

Abstract

This invention discloses a constrained weighted synchronous parallel model optimization method for distributed deep learning. Existing synchronous and asynchronous combined models suffer from asynchronous defects. This invention introduces a gradient aggregation process. If the model gradient aggregation is globally synchronous, the local model gradients of each computing node in this round of aggregation are summed and averaged to obtain a new global model gradient, which is then returned to all computing nodes. If the model gradient aggregation is locally synchronous, the parameter server node calculates the weight factor based on the number of times each computing node participates in local synchronization, then calculates the weighted average of the model gradients of all computing nodes participating in local synchronization, and uses this weighted average as the new local model gradient for each computing node participating in local synchronization. This invention solves the lag problem while ensuring training speed, improves the overall performance of model training, and guarantees the accuracy of model training.

Description

Technical Field

[0001] This invention belongs to the field of distributed deep learning acceleration technology, specifically a method for optimizing constrained weight synchronous parallel models for distributed deep learning. Background Technology

[0002] With the continuous development of computer hardware and software technology, deep learning technology has become one of the core research contents in the field of artificial intelligence. It has been widely used in fields such as digital information recognition and analysis, and using deep learning algorithms to process large-scale data is increasingly becoming a core scenario for many modern application services. However, for data that is shifting from TB to PB and is now approaching ZB, as well as the training of complex deep learning models on a massive scale, using a single computing node often takes days or weeks. Therefore, distributed deep learning systems have gradually become a research hotspot for solving large-scale deep learning problems in the era of big data.

[0003] Most distributed deep learning system architectures are based on iterative, parameter server, or data stream implementations. Communication is an essential part of multi-node collaborative training and plays a crucial role in improving the efficiency of distributed deep learning. The key to improving the efficiency of inter-node communication lies in increasing the proportion of computation time. Therefore, reasonable communication mechanisms and appropriate model aggregation methods are needed to train models more efficiently.

[0004] Communication mechanisms include two categories: synchronous mechanisms and asynchronous mechanisms.

[0005] A typical model employing synchronous communication mechanisms is the Bulk Synchronous Parallel (BSP) model. In this model, each computing node performs the same iterations based on a local copy of the global gradient generated in the previous iteration. After completing one iteration, each computing node uploads its local gradient to the parameter server node and waits for the parameter server to return the global gradient before proceeding to the next iteration. In the BSP model, the parameter server node must wait for the local gradients of all computing nodes, aggregate them to generate the global gradient, and then send it to each computing node. Because the parameter server node needs to wait for all computing nodes to complete the computation in this round, high-performing computing nodes may be unable to proceed to the next iteration and enter a blocked waiting state, thus wasting computational resources and resulting in a very low percentage of effective computation time. Furthermore, as the number of computing nodes increases, the probability of at least one lagging computing node in any given iteration also increases. In addition, the potential causes of computing node performance lag (such as hardware or network failures) are often unpredictable.

[0006] A typical model employing asynchronous communication mechanisms is the purely asynchronous parallel (AP) model. In this model, after each computation node completes an iteration, it immediately communicates with the parameter server, uploading its local gradient and retrieving the latest global gradient. The parameter server, in turn, does not wait for the local gradients of all computation nodes; instead, it aggregates the gradients uploaded by each node with the global gradient, generates a new global gradient, and transmits it to that node. Because of the performance differences among computation nodes, the number of iterations a node performs locally becomes uncontrollable. As time progresses, the number of iterations performed by better-performing nodes will far exceed that of slower-performing nodes, causing the deep learning model to get stuck in a local optimum, thus compromising the accuracy of its training. Since the training results are not necessarily accurate, the model gets bogged down in endless iterations in search of the optimal solution, severely impacting its convergence speed or even preventing it from converging.

[0007] From the above typical synchronous and asynchronous models, it can be concluded that purely synchronous and purely asynchronous models have serious defects and cannot meet the expectations of academia and industry for the training efficiency of distributed deep learning. Ho et al. proposed the Stale Synchronous Parallel (SSP) model to address the problems of purely synchronous and purely asynchronous models. It combines the characteristics of both: allowing some computing nodes to perform a certain number of iterations more than other lagging computing nodes, and using local gradients for iteration before participating in global synchronization. While not completely eliminating the performance lag problem of computing nodes, SSP relaxes the iteration synchronization consistency condition and the global consistency guarantee of local parameters. Although SSP improves the training speed of the model by using local gradients, it accumulates asynchronous errors and reduces the model's convergence speed due to the loss of local gradient updates from some lagging nodes. Yang et al. proposed the Limited Synchronous Parallel (LSP) model, which designs a "limited barrier" where the first few nodes to complete computation participate in limited synchronization each time. This approach reduces the asynchronous defects to some extent, but it may still write outdated local models to the parameter server. Therefore, ensuring both the accuracy and the training or convergence speed of the model becomes an urgent problem to be solved in the training of distributed deep learning models. Summary of the Invention

[0008] This invention addresses the asynchronous defect inherent in existing synchronous / asynchronous models, specifically the potential slowdown in global convergence when outdated local gradients are aggregated to generate an average gradient on parameter server nodes. It provides a constrained weighted synchronous parallel model optimization method for distributed deep learning. This method introduces a gradient aggregation process. If the model gradient aggregation is globally synchronous, the local model gradients of each computing node in that round of aggregation are summed and averaged to obtain a new global model gradient, which is then returned to all computing nodes. If the model gradient aggregation is locally synchronous, the parameter server node calculates a weight factor based on the number of times each computing node participates in local synchronization. Then, it calculates the weighted average of the model gradients of all computing nodes participating in local synchronization and uses this weighted average as the new local model gradient for each participating computing node. The weight factor calculation employs a Min-Max normalization method. This invention solves the lag problem while maintaining training speed, improving the overall performance of model training and ensuring model training accuracy.

[0009] This invention provides an optimization method for constrained weight synchronization parallel models in distributed deep learning, implemented through the following steps:

[0010] Step 1: Establish a distributed deep learning training system based on a parameter server architecture. This system consists of two types of nodes: parameter server nodes and compute nodes. The parameter server nodes are responsible for task scheduling and employ a multi-process working mode. Each process is divided into three categories based on its function:

[0011] Communication processes: There are multiple processes, equal to the number of computing nodes, used to maintain local model gradient communication with each computing node;

[0012] Gradient aggregation process: There is a process to aggregate the local model gradients sent from each computing node to the parameter server node and to process the update of the global model gradient.

[0013] Performance monitoring process: Assume that the performance changes of the computing node during the local model gradient transmission cycle are monitored based on the transmission cycle of the computing node sending local model gradients to the parameter server node; where the transmission cycle includes iteration time and communication time.

[0014] Step 2: Deploy a performance monitoring module to monitor the parameter server node and all compute nodes to record the transmission cycle of the compute nodes sending local model gradients to the parameter server node;

[0015] Step 3: Calculate the size of computing nodes participating in local synchronization: The parameter server node pulls the single iteration time of each computing node through the performance monitoring module, then divides the single iteration time of each computing node, and takes the size of the cluster with the best performance as the number of computing nodes participating in local synchronization in the next iteration.

[0016] Step 4: Determine the synchronization method between the parameter server node and the compute node:

[0017] All compute nodes send their local model gradients to the parameter server node, which then aggregates the gradients and returns the aggregated results to each compute node. If the sum of the number of local synchronizations participated in by all compute nodes reaches a preset value m, a global synchronization is performed. Otherwise, local synchronization is performed. Furthermore, the number of times each compute node participates in local synchronization is recorded. After global synchronization, the number of times each compute node participates in local synchronization is reset to zero.

[0018] If the model gradient aggregation is a global synchronization method, the local model gradients of each computing node in this round of model gradient aggregation are summed and averaged to obtain the new global model gradient, which is then returned to all computing nodes. If the model gradient aggregation is a local synchronization method, the parameter server node calculates the weight factor based on the number of local synchronizations each computing node participates in, then calculates the weighted average of the local model gradients of all computing nodes participating in the local synchronization, and uses the weighted average of the local model gradients as the new local model gradients of each computing node participating in the local synchronization.

[0019] Step 5: If the computing node participates in local or global synchronization, use the new local or global model gradient for the next iteration of training; otherwise, use the local model gradient for the next iteration of training.

[0020] Step 6: Repeat steps 3 to 5 until the predetermined number of iterations is completed.

[0021] Preferably, the CIFAR-10 color image dataset is used to establish a distributed deep learning training system based on a parameter server architecture.

[0022] Preferably, the parameter server node and the computing node use a point-to-point communication method.

[0023] Preferably, the performance monitoring module is deployed using the open-source tool Ganglia.

[0024] Preferably, the single iteration time of each computing node is divided based on the K-Means performance clustering algorithm.

[0025] The beneficial effects of this invention are as follows:

[0026] This invention adjusts the gradient aggregation method of parameter server nodes by introducing a weight factor for each iteration to each computing node. The weight factor is calculated and averaged based on the number of local synchronizations. Compared to summing and averaging, this method can reduce the asynchronous defect that computing nodes with poor computing performance may still write outdated local model gradients to parameter server nodes, resulting in slow convergence. This improves the gradient aggregation effect of parameter server nodes, speeds up the convergence of the training model, and improves the model training accuracy, thereby improving the overall performance of distributed deep learning training. Attached Figure Description

[0027] Figure 1 This is a diagram of the LWSP distributed deep learning training framework based on PyTorch, which is the basis of this invention.

[0028] Figure 2 This is a training flowchart of the present invention.

[0029] Figure 3 This is a comparison chart showing the relationship between the accuracy of different trained models and the number of computation processes when the performance of each computing node is similar.

[0030] Figure 4 This is a comparison chart showing the relationship between training time and number of computation processes for different training models when the performance of each computing node is similar.

[0031] Figure 5 This is a comparison chart showing the relationship between the accuracy of different training models and the number of computation processes when there are significant differences in the performance of each computing node.

[0032] Figure 6 This is a comparison chart showing the relationship between training time and number of computation processes for different training models when there are significant differences in the performance of each computing node. Detailed Implementation

[0033] The following figures, together with specific embodiments, further illustrate the present invention.

[0034] like Figure 2 As shown, the optimization method for the Limited Weight Synchronous Parallel (LWSP) model for distributed deep learning involves the following steps:

[0035] Step 1: Using CIFAR-10, a color image dataset that approximates a universal object, establish a distributed deep learning training system based on a parameter server (PS) architecture.

[0036] like Figure 1As shown, this invention employs a parameter server architecture combined with compute nodes operating in a multi-process manner. Each communication process on the parameter server node corresponds to a specific compute node, responsible for transmitting the gradient model computed by that compute node to the parameter server and maintaining communication between them. Simultaneously, the parameter server node has a dedicated gradient aggregation process acting as a gradient update manager. To efficiently utilize gradient information, weighting factors are introduced, accelerating the convergence speed of the training model and improving the gradient aggregation effect on the parameter server. The parameter server node also includes a performance monitoring process to monitor performance changes of the compute nodes during local model gradient transmission cycles.

[0037] Step 2: Deploy a performance monitoring module using the open-source tool Ganglia to monitor the parameter server node and all compute nodes, and record the transmission cycle of the compute nodes sending local model gradients to the parameter server node.

[0038] Step 3: As Figure 2 As shown, the parameter server node can calculate the local synchronization scale (i.e., the number of computing nodes participating in local synchronization) based on the single iteration time in the transmission cycle of the local model gradient sent by the computing node to the parameter server node, using the K-Means performance clustering algorithm.

[0039] Step 4: All computing nodes send their local model gradients to the parameter server node. After the parameter server node aggregates the gradients, it returns the aggregated results to each computing node. If the sum of the number of local synchronizations participated in by all computing nodes reaches a preset value m, then a global synchronization is performed. Otherwise, local synchronization is performed. Furthermore, the number of times each computing node participates in local synchronization is recorded. After global synchronization, the number of times each computing node participates in local synchronization is reset to zero.

[0040] If the model gradient aggregation is a global synchronization method, the local model gradients of each computing node in this round of model gradient aggregation are summed and averaged to obtain the new global model gradient, which is then returned to all computing nodes. If the model gradient aggregation is a local synchronization method, the parameter server node calculates the weight factor for each computing node's participation in local synchronization based on the number of times each computing node participates in local synchronization, obtains the weight factor set, and aggregates and updates the local model gradient based on the weight factor set during local synchronization.

[0041] The local model gradient is aggregated and updated based on the weight factor set, which can be expressed as follows: For a locally synchronized scenario with k computing nodes, the calculated weight factor set Ξ={ξ1,ξ2,…,ξ} is... k Simultaneously, design the local model gradient set G = {g1, g2, ..., g} for each computation node. k}, then the local gradient model of aggregation update as follows:

[0042]

[0043] Its vectorized model form is:

[0044]

[0045] Wherein, for any i-th computation node, the weight factor ξ i ∈Ξ, all have ξ i <1, and satisfies g i This represents the local model gradient of the i-th computation node participating in the synchronization during this round of local synchronization.

[0046] Step 5: If the compute node participated in the model gradient update of the parameter server node, it pulls the new model gradient from the parameter server node and uses the new model gradient to enter the next round of iteration calculation; otherwise, it uses the local model gradient to enter the next iteration training.

[0047] Step 6: Repeat steps 3 to 5 until the predetermined number of iterations is completed, then the model is considered to have converged.

[0048] Figure 3 and Figure 4 These are line graphs showing the relationship between the accuracy and training time of different distributed deep learning models training the same dataset, and the number of computation processes, when the performance differences of computing nodes in a simulated cluster are relatively small. Figure 3 It is evident that BSP achieves higher training accuracy because it ensures model consistency across all computing nodes. In other words, distributed machine learning algorithms trained in parallel using synchronous communication are equivalent to their corresponding single-machine optimization algorithms. For example, the BSP-SGD algorithm, utilizing K machines of identical performance for distributed training, is equivalent to a single-machine SGD algorithm using mini-batch scaling by a factor of K. However, although LSP, SSP, and LWSP... Figure 3 While the accuracy of LSP is not as high as that of BSP, LSP, SSP, and LWSP all utilize the fault-tolerance characteristics of deep learning models for limited asynchronous training, shortening the synchronization wait time of the parameter server. Their training times are all shorter than those of BSP. Figure 4As shown. Furthermore, SSP overuses fault tolerance, leading to excessive local iterations by each computing node when their performance is similar. This easily results in getting trapped in local optima, and the number of iterations increases with the number of computing nodes, causing a continuous decline in accuracy. LSP and LWSP, by implementing restricted synchronization barriers, ensure the number of global model updates, preventing the global model from getting trapped in local optima after local iterations. Therefore, LSP and LWSP maintain relatively high model accuracy with a relatively short training time as the number of processes increases, especially LWSP, whose model accuracy is significantly higher than LSP and SSP.

[0049] Figure 5 and Figure 6 These are line graphs showing the relationship between the accuracy and training time of different distributed deep learning models training the same dataset, and the number of computation processes, when there are significant differences in the performance of computing nodes in a simulated cluster. Figure 5 It is evident that when there are significant performance differences among computing nodes in the cluster, although BSP's accuracy is lower than LWSP's, it is still relatively high. However, its training time is significantly longer than SSP, LSP, and LWSP. This is because, when there are large performance differences among computing nodes, BSP's global synchronization is severely hampered by the most lagging computing node, resulting in significant latency. With increasing performance differences among computing nodes, SSP allows computing nodes to reach local synchronization conditions faster than BSP. LSP, by adjusting the scale of local synchronization when performance differences increase, mitigates the expansion of fault tolerance to some extent, allowing LSP to maintain good accuracy and training speed even in clusters with large performance differences. LWSP, by introducing a gradient aggregation strategy, makes fuller use of the local model gradients of each computing node, ensuring both the accuracy of the trained model and accelerating its convergence.

Claims

1. An optimization method for constrained weight synchronous parallel models for distributed deep learning, characterized by: The method includes the following steps: Step 1: Establish a distributed deep learning training system based on a parameter server architecture; the distributed deep learning training system based on the PS architecture consists of two types of nodes: parameter server nodes and computing nodes; the parameter server nodes adopt a multi-process working mode, and each process is divided into three categories according to its function: Communication processes: There are multiple processes, equal to the number of computing nodes, used to maintain local model gradient communication with each computing node; Gradient aggregation process: There is a process to aggregate the local model gradients sent from each computing node to the parameter server node and to process the update of the global model gradient. Performance monitoring process: Assume a process to monitor the performance changes of the computing node during the local model gradient transmission cycle based on the transmission cycle of the computing node sending local model gradients to the parameter server node; the transmission cycle includes iteration time and communication time. Step 2: Deploy a performance monitoring module to monitor the parameter server node and all compute nodes to record the transmission cycle of the compute nodes sending local model gradients to the parameter server node; Step 3: Calculate the size of computing nodes participating in local synchronization: The parameter server node pulls the single iteration time of each computing node through the performance monitoring module, then divides the single iteration time of each computing node, and takes the size of the cluster with the best performance as the number of computing nodes participating in local synchronization in the next iteration. Step 4: Determine the synchronization method between the parameter server node and the compute node: All compute nodes send their local model gradients to the parameter server node, which then aggregates the gradients and returns the aggregated results to each compute node. If the sum of the number of local synchronizations participated in by all compute nodes reaches a preset value, a global synchronization is performed. Otherwise, local synchronization is performed. Furthermore, the number of times each compute node participates in local synchronization is recorded. After global synchronization, the number of local synchronizations participated in by each compute node is reset to zero. If the model gradient aggregation is a global synchronization method, the local model gradients of each computing node in this round of model gradient aggregation are summed and averaged to obtain the new global model gradient, which is then returned to all computing nodes. If the model gradient aggregation is a local synchronization method, the parameter server node calculates the weight factor based on the number of local synchronizations each computing node participates in, then calculates the weighted average of the local model gradients of all computing nodes participating in the local synchronization, and uses the weighted average of the local model gradients as the new local model gradients of each computing node participating in the local synchronization. Step 5: If the computing node participates in local or global synchronization, use the new local or global model gradient for the next iteration of training; otherwise, use the local model gradient for the next iteration of training. Step 6: Repeat steps 3 to 5 until the predetermined number of iterations is completed.

2. The method for optimizing constrained weight synchronous parallel models for distributed deep learning according to claim 1, characterized in that: We use the CIFAR-10 color image dataset to build a distributed deep learning training system based on a parameter server architecture.

3. The method for optimizing constrained weight synchronous parallel models for distributed deep learning according to claim 1, characterized in that: The parameter server node and the computing node communicate using a point-to-point communication method.

4. The method for optimizing a constrained weight synchronous parallel model for distributed deep learning according to claim 1, characterized in that: Deploy the performance monitoring module using the open-source tool Ganglia.

5. The method for optimizing a constrained weight synchronous parallel model for distributed deep learning according to claim 1, characterized in that: The single iteration time of each computing node is divided based on the K-Means performance clustering algorithm.

Images