Construction method of deep neural network

Abstract

The invention discloses a construction method of a deep neural network. The structure of the deep neural network is a tree structure of a shared parameter and comprises a plurality of shared parameter branches and a plurality of Softmax layers; when the convergence rate of a certain branch is lowered, a fissile node which owns a plurality of outputs is subjected to fission to obtain one fission node which has the same type with the fissile node, a new feature detector is created, different features are generated, a parent node and a child node of the fission node inherit the fissile node, and the parameter of the fission node is initialized. On a premise of multi-model combination, calculation cost can be reduced, and a plurality of high-quality models can be obtained by fission.

Description

technical field [0001] The invention relates to a deep neural network structure, in particular to a construction method of a deep neural network. Background technique [0002] With the popularity of deep learning in various fields, the scale of neural network parameters is getting larger and larger. [0003] In the structure of deep neural networks, model composition can almost always improve the performance of machine learning methods. Averaging multi-model predictions can further reduce the error rate. At present, research on neural networks is focused on neural networks with a static structure, that is, before training, the structure of the network has been completely designed and fixed, and there will be no change thereafter. The multi-column deep neural network is such a combination model, which trains multiple fixed-structure deep networks separately, and then averagely predicts the results of all deep networks. Although this model further reduces the error rate by ...

AI Technical Summary

Problems solved by technology

Although this model further reduces the error rate by 30% to 40% compared with previous methods, the calculation amount of this multi-model is almost equivalent to the sum of the individual training calculation amount of each model

As the neural network parameters of deep learning will increase and the amount of calculation will increase, the calculation cost of such multi-model combination is quite expensive.

Moreover, the quality of each model of the multi-column deep neural network may be uneven, which will also affect the accuracy of the final prediction.

Images