Softmax implementation method based on hardware platforms

Abstract

The invention discloses an implementation method based on various hardware platforms (such as CPLD, FPGA, special chips) for calculating a softmax function. The softmax function is widely applied to multi-classification tasks, attention models and the like for deep learning, wherein related e-exponent and division calculations need to consume many hardware resources. According to a design method,by conducting mathematical transformation on the function, the e-exponent calculation is simplified to one constant multiplication, one exponential operation of 2 with a fixed input range and one shift operation; n division operations are simplified to one highest nonzero digit detection operation, one reciprocal operation with a fixed input range, one shift operation and n multiplication operations. The exponential and reciprocal operations of 2 are achieved according to a specially-designed lookup table, and the same precision can be achieved with a smaller storage space. The method is usedin the attention models and the like for deep learning, the calculation speed can be greatly increased on the premise of almost not influencing the precision, and the consumption of calculation resources and storage resources is reduced.

Description

technical field [0001] The invention relates to the technical field of integrated circuits, in particular to a softmax calculation method based on a hardware platform and its architecture design. Background technique [0002] The softmax function is widely used in multi-classification tasks and attention models of deep learning. It can map the output of multiple neurons to the (0, 1) interval, and the sum of all items is 1, which can be regarded as a probability. It is often used in machine learning to calculate the proportion of multiple outputs or multiple inputs. In the case of weight. [0003] The e-index calculation and division calculation involved in Softmax consume a lot of hardware resources. The implementation efficiency and resource consumption of deep learning on the hardware platform will be affected by the calculation speed and resource requirements of this function. After a simple transformation of the function, the exponent calculation is replaced by a shif...

AI Technical Summary

Problems solved by technology

[0004] The present invention proposes a kind of design method in order to solve the problem that it is not easy to realize the softmax function calculated by fixed-point numbers on various hardware platforms (including CPLD, FPGA, special-purpose chips, etc.), which can reduce the hardware hardware area and reduce the look-up table storage The value range of the parameter, so that the resources required to store the parameters are reduced under the same calculation precision, or the parameters with higher precision are stored while consuming the same storage space, so as to achieve higher calculation precision

Images