Neural network model training apparatus and method

Abstract

The invention relates to a neural network model training apparatus and method. The apparatus comprises an iteration calculation used for performing iterative calculation on a weight of a path in a neural network model; and a determining and output unit used for, when conditions for stopping iteration are satisfied, stopping the iteration and outputting a weight of the path of the iteration this time as a final weight. The iteration calculation unit comprises a weight calculation unit used for calculating the weight of the path of the iteration this time; a correlation function calculation unit used for randomly selecting one sample from a sample set for training the neural network model, according to the weight of the path of the iteration this time, calculating a correlation function of the selected sample, maintaining correlation functions of other samples besides the selected sample in the sample set unchanged, wherein the correlation function of the sample is a function associated with a loss function of the sample; and a total correlation function calculation unit used for calculating a total correlation function of the iteration this time according to the correlation functions of the selected sample and other samples.

Description

technical field [0001] The present disclosure relates to the technical field of neural networks, and in particular to a neural network model training device and method. Background technique [0002] This section provides background information related to the present disclosure which is not necessarily prior art. [0003] Among the technical solutions for training neural networks, recent researchers have proposed several near-incremental gradient methods, including MISO (Minimization by Incremental Surrogate Optimization, using incremental substitution optimization to minimize), Prox-SDCA (Proximal Stochastic Dual Coordinated Ascent, Near-random double-coordinate ascent), Prox-SVRG (ProximalStochastic Variance Reduced Gradient, near-random variance reduction gradient) and SAG (Stochastic Average Gradient, random average gradient), all have reached a linear convergence rate. However, the convergence speed of these methods is still unsatisfactory when applied to training neura...

AI Technical Summary

Problems solved by technology

However, the convergence speed of these methods is still unsatisfactory when applied to training neural networks, especially when training large-scale neural networks.

Images