A method for efficiently tensioning a fully connected neural network

Abstract

The invention relates to a method for efficiently tensioning a fully connected neural network, comprising the following steps of S1 expressing the input vector of the fully connected layer of the neural network as a tensor form; S2 expressing and tensioning the weights of all connection layers of the neural network as tensor ring decomposition form; 3 carrying out the forward propagation of the neural network by using a tensor algorithm of the whole connected layer of the neural network; S4 in the back propagation process of the neural network, updating the core matrix of the tensor ring decomposition form of the weight of the whole connecting layer of the neural network; S5 judging whether the data set completes the iteration, and if so, returning the weight tensor parameter of the network; if not, returning to the step S3. The method of the invention can reduce the parameters used by the neural network and the training time of the neural network, reduce the dependence and consumptionof the large-scale neural network on the hardware level such as the memory size of the computer, the GPU calculation ability and the like, and reduce the number of the neural network parameters without reducing the classification accuracy of the network.

Description

technical field [0001] The invention relates to the technical field of neural network deep learning, in particular to a method for efficiently tensorizing a fully connected neural network. Background technique [0002] Neural networks usually contain fully connected layers. The fully connected layer performs affine transformation y=Wx on the input vector x to obtain y, and then outputs y. [0003] The neural network has shown excellent performance in the field of image classification and detection. By increasing the number of layers of the network and expanding the width of each layer, it can fit various image data sets and complete various complex classification tasks. However, the deepening and widening of the network leads to a sharp increase in network parameters, which in turn reduces the speed of computer training neural networks. In addition, updating a large number of network parameters requires larger computer memory and a GPU with more computing power. Nowadays,...

AI Technical Summary

Problems solved by technology

[0006] In the literature "Tensorizing neural networks", Novikov et al proposed to express the network weight of the fully connected layer using TT decomposition (tensor train decomposition). This method can manually adjust the parameter compression rate of the network, but this method is sensitive to the initialization of network weight parameters. The accuracy of the compressed network is unstable

Images