Cluster packet synchronization optimization method and system for distributed deep neutral network

Abstract

The invention discloses a cluster packet synchronization optimization method and system for a distributed deep neutral network. The method comprises the following steps of grouping nodes in a cluster according to the performance, allocating training data according to the node performance, utilizing a synchronous parallel mechanism in the same group, using an asynchronous parallel mechanism between the different groups and using different learning rates between the different groups. The nodes with the similar performance are divided into one group, so that the synchronization overhead can be reduced; more training data can be allocated to the nodes with the good performance, so that the resource utilization rate can be improved; the synchronous parallel mechanism is used in the groups with the small synchronization overhead, so that an advantage of good convergence effect of the synchronous parallel mechanism can be exerted; the asynchronous parallel mechanism is used between the groups with the large synchronization overhead, so that the synchronization overhead can be avoided; the different groups uses the different learning rates to facilitate model convergence. According to the method and the system, a packet synchronization method is used for a parameter synchronization process of the distributed deep neutral network in a heterogeneous cluster, so that the model convergence rate is greatly increased.

Description

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine

Login to view more

AI Technical Summary

Problems solved by technology

However, the synchronization overhead between nodes is relatively high. While a node is waiting for other nodes to complete the current round of iterations, its own computing resources and network resources are idle. In heterogeneous clusters and large-scale homogeneous clusters, this The phenomenon is particularly serious

In a heterogeneous cluster, due to the large difference in the hardware configuration of the nodes, there are obvious performance differences among the nodes. Some nodes run fast, while others run slowly. Therefore, in each round of iteration, the fast All nodes need to wait for the slow nodes, resulting in idle resources of the fast nodes. The bottleneck of training lies in the slowest nodes; in a large-scale homogeneous cluster, although the performance of the nodes is the same, due to the large number of nodes, the overall The stability of the parameter server will be

Images