MPI-based neural network architecture search parallelization method and device

Abstract

The invention relates to an MPI-based neural network architecture search parallelization method and device. The method comprises the steps that MPI processes are started according to the number of GPUs in a current multi-computer environment and arranged in sequence; the started MPI process reads data from a designated position in a training set according to the serial number of the started MPI process, and gradient calculation is carried out; the GPU of each node performs gradient specification calculation according to the hierarchical structure, and summarizes calculation results to the first GPU in the GPUs; the first GPU performs gradient full-specification calculation according to an annular structure; starting from the first GPU in each node, broadcasting a gradient calculation result according to a hierarchical structure; and updating the weight and the bias value of the neural network by using the new gradient value. On the basis of guaranteeing the recognition rate of the neural network architecture search result model, the neural network architecture search training efficiency can be effectively improved, the training time is greatly shortened, and therefore the efficiency of the automatic deep learning process is improved.

Description

technical field [0001] The invention relates to the technical field of neural network search parallelization, in particular to an MPI-based neural network architecture search parallelization method and device. Background technique [0002] At present, discovering an efficient neural network architecture requires a considerable workload of deep learning experts, and it is necessary to manually build a suitable neural network architecture according to different directions. This working mode consumes a lot of energy and time of deep learning practitioners. At present, in order to solve this problem, a variety of methods for automatically searching the neural network architecture have been proposed. The algorithms with better performance include reinforcement learning and evolutionary learning. Since they are all searched in discrete spaces, they require a huge amount of calculation. More than 1,000 days of GPU working time. Some scholars have also proposed a differentiable neu...

AI Technical Summary

Problems solved by technology

[0003] Stand-alone deep neural network training is limited by the fact that local IO processing cannot be greatly accelerated, and it is difficult to further improve GPU computing power and video memory.

Images