A network structure optimization method based on an attention mechanism

Abstract

The invention provides a network structure optimization method based on an attention mechanism, and the method comprises the steps: carrying out the reasonable weight distribution of each module of aneural network, emphasizing or weakening the contribution of certain input data to the next processing, carrying out the design in a differentiable mode, and completing an end-to-end neural network. According to the specific method, a multi-layer neural network is used for learning a weight distribution function. The learning mode is different from a common neural network training mode, firstly, only one target network is trained and then a weight distribution network is added to the network, then parameters of the target network are fixed, the weight distribution network is trained, and the target network and the weight distribution network are trained in an iteration mode until the effect is optimal.

Description

technical field [0001] The invention relates to the field of the Internet and the field of deep learning, in particular to a network structure optimization method based on an attention mechanism. Background technique [0002] The network structure optimization method based on the attention mechanism optimizes the neural network structure by assigning reasonable weights to each module of the neural network, emphasizing or weakening the contribution of some input data to the next step of processing, and designing in a differentiable manner . The technologies closest to the present invention are: [0003] (1) Regularization method based on L1 norm: L1 regularization refers to the sum of the absolute values ​​of each element in the weight vector w. By including the L1 regularization item in the loss calculation during the iterative optimization process, the regularization can be made The matrix parameters constrained by the transformation item become smaller, in order to gener...

AI Technical Summary

Problems solved by technology

[0005] (3) The method of randomly shielding neurons based on dropout: it can be approximated as a sparse constraint between neurons, and it is realized by shielding some neurons so that they do not work, but it has great randomness and cannot be obtained A globally effective information to constrain the activation or sparsity of neurons, and sparsity constraints are also important for neural network structures

However, the L1 and L2 norms are only for the weight matrix constrained by them, but for neurons between the same layer at a higher level, and a larger range of neural layers, there is no corresponding sparse constraint to realize the network structure. Sparse, can not better perform sparse constraints on each neuron and neural layer

The dropout method shields neurons based on randomness, which simply reduces the risk of over-fitting, but cannot reasonably and effectively reduce the number of neurons to achieve network sparsity.

Images