Hardware accelerators for dot-product based lookup algorithms including hardware accelerators for transformer key-value caches

Abstract

A hardware accelerator is provided for dot-product based lookup algorithms including hardware accelerators for transformer key-value. The accelerator includes memory stacks that include memory banks (e.g., DRAM) fabricated on respective memory dies. The memory banks can be connected by through silicon vias (TSVs). The accelerator also includes arithmetic logic units configured on a processor die, and that include dedicated circuitry for the dot-product based lookup algorithm. The processor die can be arranged below the memory stacks to minimize the wire lengths from the ALUs to the memory stacks to reduce energy consumption due to moving bits between the ALUs and the memory stacks. The length of the wires from the ALUs to the memory stacks can be less than

Images