Improved method and system for deformable activation function, and storage device

Abstract

The present invention provides an improved method and system for a deformable activation function, and a storage device. Based on parameter learning, a deformable exponential function is introduced tooptimize the exponential form in the existing activation function. At the same time, a scale parameter is introduced, and the scale parameter is updated by the gradient during a training process. Byintroducing the idea of "parameterization" into the activation function, the performance of the activation function is improved, thereby improving the training efficiency and performance of the network; the method is suitable for the construction of any neural network, and has good scalability without platforms.

Description

technical field [0001] The invention relates to an improved method, system and storage device of a deformable activation function, and relates to the fields of machine learning, neural network and computer vision. Background technique [0002] In recent years, deep convolutional networks have become the most powerful weapon for large-scale visual recognition. Along the deep learning technology, researchers have continuously proposed new networks from the perspective of network structure and network depth, and achieved good performance. These networks all have a common basic module: the activation function module. The main function of the activation function is to introduce nonlinearity to the network and eliminate the problems of gradient disappearance and dispersion. At the same time, it can speed up the training process of the network during training. Activation functions are mainly divided into two categories. The first category is "rectified linear unit", which zeros t...

AI Technical Summary

Problems solved by technology

[0003] 1. The output of the rectified linear unit has no negative values, which makes the mean value of the output positive. During the network training process, as the positive mean value is accumulated layer by layer, it will lead to continuous mean shift

eventually lead to training failure

[0004] 2. The exponential linear unit output has negative values, which alleviates the mean shift problem to a certain extent, but the negative value output is a fixed exponential mapping, which does not completely solve the cumulative mean shift problem

[0005] 3. The negative semi-axis output of the exponential linear unit is a fixed exponential function output, which causes the activation function to lose its flexibility and cannot be combined with the training process of the network to find the most suitable activation function for the network

Images