Key-Value Caching Optimization System and Method for Reducing Memory Overhead in Long Text Inference of Large Language Models

Abstract

This invention discloses a key-value (KV) caching optimization system to reduce the memory overhead of long text inference in large language models. The system includes an offline optimization module, a pre-filling decoupling module, a caching module, and a dynamic reconstruction module. The offline optimization module determines the optimal compression parameters for each layer of the model and performs offline decomposition of static weights. The pre-filling decoupling module adaptively decouples the input lexical embeddings into sparse components and residual components, generating structured codes. The caching module stores these structured codes to replace traditional KV matrices. The dynamic reconstruction module uses a custom operator to reconstruct the codes into transient KV tensors in real time during the inference phase and releases them immediately after attention calculation. This invention preserves key feature information through an adaptive decoupling mechanism; reduces the storage requirements of residual components through structured coding and low-rank projection; and efficiently reconstructs the compressed sparse and low-rank codes through a custom fusion operator, establishing an on-demand calculation and immediate release mechanism for transient key-value tensors.

Images