Cluster packet synchronization optimization method and system for distributed deep neutral network

Abstract

The invention discloses a cluster packet synchronization optimization method and system for a distributed deep neutral network. The method comprises the following steps of grouping nodes in a cluster according to the performance, allocating training data according to the node performance, utilizing a synchronous parallel mechanism in the same group, using an asynchronous parallel mechanism between the different groups and using different learning rates between the different groups. The nodes with the similar performance are divided into one group, so that the synchronization overhead can be reduced; more training data can be allocated to the nodes with the good performance, so that the resource utilization rate can be improved; the synchronous parallel mechanism is used in the groups with the small synchronization overhead, so that an advantage of good convergence effect of the synchronous parallel mechanism can be exerted; the asynchronous parallel mechanism is used between the groups with the large synchronization overhead, so that the synchronization overhead can be avoided; the different groups uses the different learning rates to facilitate model convergence. According to the method and the system, a packet synchronization method is used for a parameter synchronization process of the distributed deep neutral network in a heterogeneous cluster, so that the model convergence rate is greatly increased.

Description

technical field [0001] The invention belongs to the technical field of distributed optimization of deep neural networks, and more particularly relates to a method and system for grouping synchronization optimization of distributed deep neural network clusters. Background technique [0002] At present, Deep Neural Network (DNN) has been applied in many fields such as image, speech, and natural language processing, and has made many breakthroughs. Due to the large scale of its training data and trained model parameters, deep neural networks require sufficient computing resources and storage resources. Therefore, the traditional single-machine node training mode can no longer meet the requirements, and distributed computing modes such as clusters must be used. [0003] Distributed Deep Learning usually adopts data parallel mode for model training. Such as figure 1 As shown, data parallelism refers to splitting the training data, storing one or more split training data on eac...

AI Technical Summary

Problems solved by technology

However, the synchronization overhead between nodes is relatively high. While a node is waiting for other nodes to complete the current round of iterations, its own computing resources and network resources are idle. In heterogeneous clusters and large-scale homogeneous clusters, this The phenomenon is particularly serious

In a heterogeneous cluster, due to the large difference in the hardware configuration of the nodes, there are obvious performance differences among the nodes. Some nodes run fast, while others run slowly. Therefore, in each round of iteration, the fast All nodes need to wait for the slow nodes, resulting in idle resources of the fast nodes. The bottleneck of training lies in the slowest nodes; in a large-scale homogeneous cluster, although the performance of the nodes is the same, due to the large number of nodes, the overall The stability of the parameter server will be

Images