Resource management method and system for large-scale distributed deep learning

Abstract

The invention discloses a resource management method and system for large-scale distributed deep learning, and the method and system achieve the memory resource optimization management of parameters and gradient intermediate data during the training operation of a neural network, and guarantee the reasonable configuration of distributed communication bandwidth resources. Cross-layer memory reuse is realized again, intermediate data required by iterative computation and sparse communication are migrated into a CPU main memory and then migrated back as required, and interlayer memory consumptionis reduced; on the basis of reasonable migration of CPU-GPU data, independence of intra-layer memory reuse, intra-layer calculation mining and memory access operation is achieved, and intra-layer memory consumption is reduced as much as possible. Distributed parameter communication optimization is realized while efficient utilization of memory resources is ensured. Data access in the distributedparameter updating stage is reasonably redirected, a CPU main memory serves as a mirror image access area, the data access to parameters and gradients is completed, and the problems of gradient data missing and parameter writing border crossing are solved.

Description

technical field [0001] The invention belongs to the technical field of distributed systems, and more specifically relates to a resource management method and system for large-scale distributed deep learning. Background technique [0002] Deep learning (deep neural network) has achieved breakthroughs in fields such as computer vision, language modeling, and speech recognition. Compared with the traditional artificial neural network, the characteristics of the deep neural network are: it has more hidden layers and neurons, the calculation volume of the training and reasoning phase is huge, and a large amount of intermediate data will be generated at the same time. These characteristics make most deep learning applications have a large demand for storage space and computing resources in the training process, and the current high-performance acceleration hardware (such as GPU, etc.) cannot meet their needs well. So there has been a lot of optimization work on memory management ...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the defects of the prior art, the purpose of the present invention is to provide a resource management method and system for large-scale distributed deep learning, aiming at solving the problem of simultaneously managing and configuring computing, memory and bandwidth resources in the prior art. The method of directly combining memory optimization and communication optimization is adopted. This method cannot coordinate and manage multiple resources, resulting in low resource utilization of the training system and easily causing data loss and memory out-of-bounds problems.

Images