A design method of neural network activation function

Abstract

The invention discloses a design method of a neural network activation function. The method includes designing a neural network framework, selecting a saturation activation function as the neural network activation function; According to the application scenario of the neural network, selecting the initial fixed-point bit width of the data. Determining The initial fractional bit width and the initial integer bit width according to the precision requirement and the data characteristics of the neural network. Performing Binary conversion of the fractional part and the integer part; Taking the converted fixed-point format data as input, training the neural network and recording the training results. Record the training test results; Repeating steps 2 to 6 until the overall fixed-point width,decimal-bit width and integer-bit width meet the requirements are found as the final neural network fixed-point architecture. Determining saturation value of activation function of neural network; Implementing forward propagation function; Implement back propagation function. The invention saves power consumption and cost, and solves the problem of fixed-point algorithm overflow.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence neural network, related algorithms, software, hardware and chips. In particular, it relates to a design method of a neural network activation function. Background technique [0002] An artificial neural network is a computational model designed by humans to mimic the way biological neural networks work. Neuron (Neuron) is the basic unit of neural network, also known as node (Node), it receives input (Input) from the outside or other nodes, and calculates output (Output) through an activation function (Activation Function); each The input corresponds to Weight, which is the relative importance of each input received by this node; Bias can be understood as a special input. [0003] Deep learning is a field of machine learning that studies the algorithms, theory, and applications of complex artificial neural networks. Since it was proposed by Hinton in 2006, deep learning has been ...

AI Technical Summary

Problems solved by technology

However, compared with fixed-point arithmetic units, floating-point arithmetic has problems such as occupying more resources, large area, high power consumption, and high cost.

For the fixed-point neural network, there is a problem that must be solved is data overflow

Images