Optimized group-wise quantized matmul on neural network accelerator

Abstract

For matrix multiplication (MatMul) in large language models, group-wise quantization is used for weight compression, resulting in data quantization on both the output and input channels when converted to convolution on a deep neural network (DNN) accelerator. Unrolling the MatMul into smaller operations with the same input channel scale value (referred to as channel-wise quantization) can lead to significant increase in workload count, while the compute overhead for each workload is small. To address this issue, the MatMul operation is instead performed on a reshaped input matrix and a reshaped weight matrix and unnecessary outputs are discarded. The result still obtains the correct result for MatMul but trade off calculation complexity for workload overhead.

Images