Trainable piecewise linear activation function generation method

Abstract

The invention discloses a trainable piecewise linear activation function generation method, used for simplifying the computing problem of a complex non-linear activation function in a hardware acceleration process. The invention uses the piecewise linear function to replace the non-linear activation function, and constantly updates the coefficient through self-learning to achieve the purpose of using the linear function to replace the non-linear activation function. Compared with the general linear activation function, the trainable piecewise linear activation function generation method is closer to the original non-linear activation function; the relative error is less, and the coefficient is updated in reverse propagation continuously; therefore, the convergence speed of learning is accelerated, and the problems of gradient explosion and gradient disappearance are eliminated as far as possible in a certain range.

Description

technical field [0001] The invention belongs to the field of computers, in particular to a method for generating a trainable segmented linear activation function. Background technique [0002] In recent years, machine learning has been practically used in many fields such as computers and the Internet, and has played a huge role, greatly improving the success rate of functions including image recognition and language recognition. In the neural network, the result of each layer of the network is processed by the activation function before it is used as the final output. The continuous development of the activation function is an important part of the continuous improvement of the deep network. The continuous improvement of the activation function makes the output of the neural network more accurate. for precision. [0003] The learning network that only includes the convolutional layer and the fully connected layer is obtained through multi-layer operations, and the results ...

AI Technical Summary

Problems solved by technology

[0004] With the development of deep networks, a variety of activation functions are used in the network, but in terms of application, due to the huge network of machine learning, the implementation of general hardware is relatively slow, so it is inevitable to accelerate the deep network through hardware. Must do

In the process of hardware acceleration, due to the lack of computing resources, it is difficult to implement complex operations of nonlinear activation functions

The existing linear activation functions, such as ReLU function, PReLU function, etc., although the use of these linear functions simplifies the network, the use of simple linear functions still does not solve the problem of nonlinear mapping, and the output of the deep network is still stuck in the linear mapping. stage

Images