Quantization method of improving the model inference accuracy

Abstract

The disclosure describes various embodiments for quantizing a trained neural network model. In one embodiment, a two-stage quantization method is described. In the offline stage, statically generatedmetadata (e.g., weights and bias) of the neural network model is quantized from floating-point numbers to integers of a lower bit width on a per-channel basis for each layer. Dynamically generated metadata (e.g., an input feature map) is not quantized in the offline stage. Instead, a quantization model is generated for the dynamically generated metadata on a per-channel basis for each layer. The quantization models and the quantized metadata can be stored in a quantization meta file, which can be deployed as part of the neural network model to an AI engine for execution. One or more speciallyprogrammed hardware components can quantize each layer of the neural network model based on information in the quantization meta file.

Description

technical field [0001] Embodiments of the present disclosure relate generally to artificial intelligence (AI) engines. More specifically, embodiments of the present disclosure relate to neural network quantization. Background technique [0002] A branch of artificial intelligence (AI), machine learning can perform tasks without using an application specifically programmed for that task. In contrast, machine learning learns from past examples of a given task during training, which typically involves learning weights from a dataset. [0003] A trained machine learning model (e.g., a neural network model) can perform tasks on input data by inference, and typically uses a 32-bit floating-point representation as the default representation to represent the model's metadata (e.g., weights and biases). During inference, the input feature map can be represented as a 32-bit integer. Larger bitwidths for metadata and input feature maps can severely impact the performance of neural n...

AI Technical Summary

Problems solved by technology

However, reducing the bitwidth with quantization usually leads to a drastic drop in the inference accuracy of quantized neural network models

Images