Weight and activation value quantification method for long-term and short-term memory network

Abstract

The invention discloses a weight and activation value quantification method for a long-term and short-term memory network. The method comprises the following steps: 1) collecting a weight and activation value number set of the long-term and short-term memory network; 2) determining a corresponding target quantization range, starting threshold traversal circulation, and scaling or setting a weightvalue and an activation value as saturation values according to the calculated scaling factors and saturation values under different threshold conditions; and 3) after traversing is completed, respectively calculating the KL divergence of the initial number set of the weight value and the activation value and the KL divergence of the mapped number set, and finally respectively outputting positiveand negative number direction truncation thresholds and the minimum KL divergence value. According to the method, the long-term and short-term memory network obtained after high-precision floating-point number training is completed is converted into the fixed-point number network, a quantitative structure for the weight value and the activation value of the long-term and short-term memory networkis innovatively designed, the hardware overhead is reduced while the algorithm hardware implementation precision is guaranteed, and the operation speed is increased.

Description

technical field [0001] The invention belongs to the field of cyclic neural network quantization, and in particular relates to a quantization method for weight and activation values ​​of long-term and short-term memory networks. Background technique [0002] With the continuous improvement of the computing power of graphics processors and general-purpose central processing units, the requirements for computational complexity of artificial neural networks have been eased. After 2012, artificial intelligence algorithms based on neural networks have been continuously developed and widely used in many fields such as pattern recognition, speech processing, and image processing. However, the development of hardware performance cannot satisfy the evolution of algorithms. The SSD network proposed in 2016 introduces up to 50 billion calculations and needs to be run in large workstations. However, desktop processors and mobile processors cannot bear such a large amount of calculation...

AI Technical Summary

Problems solved by technology

However, it is difficult for desktop processors and mobile processors to bear such a large amount of calculation, which greatly limits the application scenarios of neural networks, including various terminal applications such as virtual reality technology and augmented reality technology

Images