Hardware attention accelerator with on-chip memory for key and value vectors
An on-chip storage device with BRAM-based memory and validity bitmap in the attention unit addresses memory bottlenecks in autoregressive text generation, achieving a 256-fold reduction in calculations by eliminating quadratic complexity.
DE202026002048U1Undetermined Publication Date: 2026-07-02NEUROVEXON UG (HAFTUNGSBESCHRÄNKT)
Patent Information
- Application Number
- DE202026002048
- Authority / Receiving Office
- DE · DE
- Patent Type
- Utility models
- Current Assignee / Owner
- Filing Date
- 2026-05-04
- Publication Date
- 2026-07-02
- Estimated Expiration
- 2036-05-31
AI Technical Summary
Technical Problem
Autoregressive text generation in transformer models faces a quadratic complexity issue due to the need to recalculate key and value vectors for each new token, leading to memory bottlenecks and high latency in software-based caching solutions.
Method used
Implementing a dedicated on-chip storage device within the attention unit using BRAM-based memory blocks for key and value vectors, equipped with a validity bitmap and efficient store/load mechanisms, reducing access latency to one clock cycle.
Benefits of technology
Reduces computational complexity from O(n²) to O(n) by eliminating memory bottlenecks and bus overhead, enabling a 256-fold reduction in calculations for long sequences.
✦ Generated by Eureka AI based on patent content.
Smart Images
Figure 00000006_0000
Abstract
An on-chip memory device within a hardware attention unit of a neural network accelerator, comprising: a) at least one dedicated on-chip memory block (1) for storing key vectors and at least one dedicated on-chip memory block (2) for storing value vectors, the memory blocks being located within the attention unit; b) a validity bitmap (3) with one bit per token position indicating whether valid key / value vectors are stored for the respective position; c) a store logic that, when calculating new token positions, stores the calculated key / value vectors element-wise in the on-chip memory and, after completion of all dimensions, sets the corresponding bit in the validity bitmap (3); d) a load logic that, in subsequent calculations, makes stored key / value vectors available from the on-chip memory with a latency of one clock cycle;unde) a hit check logic (10) which combinatorially (without clock cycle latency) indicates, based on the validity bitmap (3), whether valid K / V vectors are available for a queried token position.;
Need to check novelty before this filing date? Find Prior Art
Citation Information
Patent Citations
Key-value cache compression and sparse attention computation method and system for large language model inference
CN120952055B
Methods and apparatus to evict tokens from a key value cache
US20250036876A1