Recurrent neural network sparse connection method based on block tensor decomposition

Abstract

The invention discloses a recurrent neural network sparse connection method based on block tensor decomposition. The recurrent neutral network sparse connection method is applied to the field of deeplearning neural network structure optimization and solves the problem that an existing method cannot perform analysis and optimization based on high-dimensional facts of input data and self redundancyof full connection and cannot quicken a training speed and maintaining or improve model accuracy at the same time. According to the method disclosed by the invention, sparsity of a full connection layer in a deep network is taken into consideration, an tensor decomposition idea is introduced, a sharing degree of network parameters is improved, a BPTT is utilized to perform model training, and themethod is adapted to most application scenes of an existing deep network; compared with an existing full-connection mode, a training speed and a convergence precision of the method disclosed by the invention are more greatly improved.

Description

technical field [0001] The invention belongs to the field of deep learning neural network structure optimization, and in particular relates to a design technology of a sparse connection method of a cyclic neural network based on block tensor decomposition. Background technique [0002] Recurrent neural network (RNN) has been widely used in the field of time series information processing, such as speech recognition, text translation, video classification, etc. In the traditional neural network model, the data flows from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes between each layer are not connected. But this kind of neural network cannot capture time series information. In the RNN network, the preorder information in the time series is memorized and applied to the calculation of the current output, that is, the nodes between the hidden layers are connected, and the input of the hidden layer includes not onl...

AI Technical Summary

Problems solved by technology

[0006] In order to solve the problem that the existing cyclic neural network structure ignores the high-dimensional fact of the input data and the redundant characteristics of the full connection, the present invention proposes a sparse connection method of the cyclic neural network based on block tensor decomposition, and simultaneously based on the input data Analysis and optimization of the high-dimensional facts and the redundant characteristics of the full connection itself, compared with the prior art, the convergence accuracy of the present invention is greatly improved

Images