Training method, device and chip of a neural network model

Abstract

The embodiments of the present invention relate to the field of machine learning, and in particular to a neural network model training method, device and chip, which are used to reduce the communication traffic between the server module and each working module during the training process of the neural network model. In the embodiment of the present invention, the model training method of each layer is determined according to the estimated data amount in the model parameter set of each layer and the estimated data amount of the output data. When the jth layer is a model parallel training method, due to the The second output data is the output data of the j-1 layer training of the m work modules, so the work module performs model parameter training according to the second output data, and can directly obtain the global gradient of the model parameters, compared to the work module in the prior art. The server module pushes up the local gradient of the model parameters and pulls down the global gradient of the model parameters from the server module to obtain the global gradient of the model parameters, which reduces the amount of communication between the working module and the server module.

Description

technical field [0001] Embodiments of the present invention relate to the field of neural network model training, and in particular to a neural network model training method, device, and chip. Background technique [0002] Since deep learning has achieved great success in large-scale image classification datasets, both academia, government and industry are vigorously promoting the development of deep learning, and have continuously achieved new results. As the main model form in deep learning, the feedforward neural network model has been widely used in tasks such as face recognition, image classification, target detection, and video analysis, and is being rapidly adopted by major machine vision manufacturers for intelligent Image, video processing and other products. At present, the depth of the feed-forward neural network model is getting deeper and deeper, and the structure is more and more complex. For example, in many intelligent image and video processing tasks, the d...

AI Technical Summary

Problems solved by technology

[0005] In the above scheme, since the L layer of the neural network model includes a large number of model parameters, the application of this scheme will cause each working module to push a large number of local gradients of model parameters to the server module, and pull down a large number of model parameters from the server module. Global gradient, resulting in a large amount of information communication between the server module and each working module

Images