Distributed training method and device for deep learning model, equipment and storage medium

Abstract

The invention discloses a distributed training method for a deep learning model. The distributed training method comprises the following steps: acquiring a network communication speed data set betweennodes in a cluster; performing network grouping on all nodes of the cluster by utilizing a clustering algorithm and the network communication speed data set; and scheduling the distributed training task of the deep learning model to the node of the network group with the highest network communication speed for training. Visibly, in the application, network grouping is carried out on nodes in advance according to the network communication speed between the nodes, so that when a distributed training task of a deep learning model is executed, the node in the network group with the highest network communication speed can be selected for training, and the problem that the training speed is too low due to the fact that the network communication speed of the node is too low is solved. The invention further discloses a distributed training device and equipment for the deep learning model and a computer readable storage medium, and the technical effects can also be achieved.

Description

technical field [0001] The present invention relates to the technical field of model training, and more specifically, to a distributed training method, device, equipment, and computer-readable storage medium of a deep learning model. Background technique [0002] As the scale of deep learning models becomes larger and more parameters need to be trained, the computing power of a single computing node can no longer meet the needs of model training, so researchers conduct distributed training for deep learning models. The meaning of distributed model training is: the training process of the model is distributed to multiple computing nodes, and the computing resources of multiple nodes are used to participate in the parameter training of the model to speed up the training speed. When multiple nodes participate in the training process of the model, according to the researcher's design, each node will calculate some parameters of the model or model parameters of a certain iteratio...

AI Technical Summary

Problems solved by technology

However, when Kubernetes dispatches the distributed model to nodes, it does not consider the network communication speed between these nodes. When the number of network transmissions between nodes is frequent or the amount of data transmitted is large, the network transmission speed is very important. Too slow network transmission Speed ​​can become the bottleneck of model training

Images