Distributed training method and device for deep learning model

Abstract

The invention discloses a distributed training method and device for a deep learning model. According to the specific implementation scheme, the method comprises the steps of obtaining training statedata corresponding to a training task sent by a deep learning platform; generating an elastic scaling strategy according to a cluster resource demand corresponding to the training task; dynamically adjusting the number of training nodes corresponding to the training task by adopting an elastic scaling strategy; and executing a training task according to the training state data and the adjusted training node. According to the method, the adaptability of the cluster resource demand corresponding to the training task is improved, the GPU or CPU resource utilization rate is improved, and it can beensured that the training task can be correctly and efficiently executed by utilizing the adjusted training node under the condition that the training node is added or deleted at any time.

Description

technical field [0001] This application relates to the field of deep learning, especially to the field of distributed training. Background technique [0002] The deep learning framework / platform supports a distributed training mode, that is, multiple devices can be used, and multiple GPUs (Graphics Processing Units) can be set on each device, and the deep learning model is parallelized on the GPUs in each device. train. Existing deep learning frameworks / platforms, for example, TensorFlow (based on data flow programming, dataflow programming) native PS (parameter service, Parameter server) architecture supports asynchronous training mode. When the deep learning framework / platform is running, the deep learning framework / platform can be deployed to a specific physical cluster. The nodes in the TensorFlow cluster are divided into two categories: parameter server and worker. The parameter server stores the parameters of the model, and the working server is responsible for calcu...

AI Technical Summary

Problems solved by technology

[0003] However, when the deep learning model is performing different training tasks, some training tasks require more GPUs, while others only require less GPUs, or some special training tasks will show a certain cycle when using GPUs. Sexual characteristics, there are peaks and valleys in use, resulting in an idle state of the GPU in some training tasks

Therefore, for different training tasks, the number of working servers cannot be adaptively adjusted, resulting in low GPU cluster utilization

Images