Neural network model training method, device, chip and system

Abstract

The embodiment of the invention provides a model training method. According to the embodiment of the invention, each working module comprises a calculation process and a communication process which run in parallel, the calculation process calculates model parameters of a next training model according to local gradient and model parameters of last iteration, and the communication process pushes thelocal gradient and pull-down global gradient to a server module. The process of calculating the model parameters of the next iteration of the training model by the calculation process does not dependon the global gradient of the last iteration, so that the time windows of the calculation process and the communication process are overlapped, and the training time delay of the model parameters isshortened.

Description

technical field [0001] The embodiments of the present invention relate to the field of machine learning, and in particular to a neural network model training method, device, chip and system. Background technique [0002] With the rapid development and popularization of computer and information technology, industry application data is growing explosively. Industry and enterprise big data that can easily reach hundreds of trillion bytes (Trillionbyte, TB for short) or even petabytes (PB for short), often implies a lot of in-depth knowledge that is not available when the amount of data is small And value, data analysis led by large-scale machine learning (including deep learning) is a key technology for converting big data into useful knowledge. Google, Facebook, Microsoft, Baidu and other large Internet companies at home and abroad have established specialized big data-based machine learning and artificial intelligence research and development institutions to conduct in-depth...

AI Technical Summary

Problems solved by technology

[0011] It can be seen from the above description that during the iterative process of each model parameter, each working module first pushes the local gradient to the server module, and waits for the global gradient pulled down from the server module to the model parameter, and then updates the local gradient according to the global gradient. Model parameters, and then calculate the local gradient according to the updated local model parameters. It can be seen that the time taken by each iteration process includes the communication time for pushing up the local gradient to the server module and pulling down the global gradient from the server module, as well as updating the local model parameters and calculating The calculation time of the local gradient takes a long time for one iteration, which leads to a large delay in the training process of the model parameters

Images