Quick training method of large-scale data recurrent neutral network (RNN)

Abstract

The invention relates to a quick training method of large-scale data recurrent neutral network (RNN), and belongs to the technical field of machine learning. The method comprises the steps of synchronously updating large-scale data of internal coefficients in the average gradient direction and the gradient residual main component direction to quickly train the generalized regression neural network; performing error back propagation to obtain the gradient of the target functions at each training sample to the internal coefficient; grouping the training samples; performing weighted averaging for the whole training sample set and the gradients of each group according to the target function value of each training sample; updating the internal coefficients in the global average gradient direction and the residual main component direction that and the group average gradient is orthogonal to the global average gradient. With the adoption of the method, the gradient information of each training sample can be effectively utilized with relatively small calculation cost to reduce the iteration steps can be decreased, so as to increase the calculation efficiency of RNN training process.

Description

technical field [0001] The invention belongs to the technical field of machine learning, in particular to applications such as speech recognition and natural language processing for large-scale data information processing, high-dimensional time series analysis and the like. Background technique [0002] Contemporary data acquisition technology generates a large amount of complex data, which contains rich information, and has great potential value for various application fields in production and scientific research. Extracting useful information from large-scale data requires effective data processing methods. Artificial neural network is one of the most widely used data information extraction methods, and has shown outstanding performance in computer vision, speech recognition and natural language processing. [0003] Artificial Neural Network (ANN), referred to as Neural Network (NN), is a computational model that imitates the structure and function of biological neural ne...

AI Technical Summary

Problems solved by technology

However, the update method in step 6 only uses the average gradient, and the update step size depends on the preset learning factor. Although it has a strong regularization ability, it does not make full use of the gradient and objective function information at each training sample, resulting in Convergence is slow, and the number of iteration steps is large

Images