Neural network model training method, device, chip and system

Abstract

Embodiments of the invention relate to the field of machine learning, in particular to a neural network model training method, device, chip and system, and aims at shortening training time delay of model parameters. The method comprises the following steps of: aiming the ith iteration, in each training period, of each working module in N working modules, executing the ith iteration in parallel byeach working module, wherein each training period comprises K iterations; calculating a model parameter of the (i+1)th iteration according to a local gradient of the ith iteration and a model parameter of the ith iteration, and when i is smaller than K, calculating the local gradient of the (i+1)th iteration according to the model parameter of the (i+1)th iteration and sample data of the (i+1)th iteration; pulling down a global gradient of the rth iteration to a server module and / or pushing up a local gradient of the fth iteration to the server module by the working module. According to the method, time windows of calculation processes and communication processes are overlapped, so that the training time delay of model parameters is shortened.

Description

technical field [0001] The embodiments of the present invention relate to the field of machine learning, and in particular to a neural network model training method, device, chip and system. Background technique [0002] With the rapid development and popularization of computer and information technology, industry application data is growing explosively. Industry and enterprise big data that can easily reach hundreds of trillion bytes (Trillionbyte, TB for short) or even petabytes (PB for short), often implies a lot of in-depth knowledge that is not available when the amount of data is small And value, data analysis led by large-scale machine learning (including deep learning) is a key technology for converting big data into useful knowledge. Google, Facebook, Microsoft, Baidu and other large Internet companies at home and abroad have established specialized big data-based machine learning and artificial intelligence research and development institutions to conduct in-depth...

AI Technical Summary

Problems solved by technology

[0011] It can be seen from the above description that during the iterative process of each model parameter, each working module first pushes the local gradient to the server module, and waits for the global gradient pulled down from the server module to the model parameter, and then updates the local gradient according to the global gradient. Model parameters, and then calculate the local gradient according to the updated local model parameters. It can be seen that the time taken by each iteration process includes the communication time for pushing up the local gradient to the server module and pulling down the global gradient from the server module, as well as updating the local model parameters and calculating The calculation time of the local gradient takes a long time for one iteration, which leads to a large delay in the training process of the model parameters

Images