Neural network training method, storage medium and equipment

Abstract

The embodiment of the invention discloses a neural network training method, which comprises the following steps of: constructing a training framework comprising a parameter node and a plurality of training nodes, and updating neural network parameters of the plurality of training nodes and the parameter node; training by each training node, and respectively sending neural network parameters and/orneural network cumulative gradients to the parameter nodes every other preset training steps; fusing the neural network parameters and/or the neural network cumulative gradients of the training nodesby the parameter nodes, and updating the neural network parameters and/or the neural network cumulative gradients of the parameter nodes according to the neural network parameters and/or the neural network cumulative gradients; and each training node performs training again according to the fused neural network parameters and/or the neural network cumulative gradient sent by the parameter node, and the parameter node outputs a neural network model through a preset model training termination condition. According to the neural network training method provided by the embodiment of the invention,the training efficiency of the neural network training method and the performance and training precision of the convergence model can be further improved.

Description

technical field [0001] The invention relates to the technical field of deep learning, in particular to a neural network training method, storage medium and equipment. Background technique [0002] With the increasing complexity of the problems faced by intelligent algorithms such as deep learning and deep reinforcement learning, the data scale and training calculation amount required for deep neural network training have increased dramatically, and the training time of single machines has gradually tended to explode. Although the development of GPU hardware has made great progress in recent years and provided support for the training of deep neural networks to a certain extent, the time for single-machine training of deep neural networks is still too long. Under this condition, the distributed training of deep neural network was proposed and gradually attracted the attention of researchers. [0003] There are mainly three modes of distributed training of deep neural network...

AI Technical Summary

Problems solved by technology

[0006] With the increasing complexity of the tasks to be handled by the deep neural network model, the training data that needs to be considered and used in training the deep neural network is increasing day by day, and the convergence efficiency and The performance and training accuracy of the model obtained by convergence is poor, and the data utilization efficiency is low

Images