Key-value cache management method and apparatus

CN121882113BActive Publication Date: 2026-06-09MOXIN ARTIFICIAL INTELLIGENCE TECH (SHENZHEN) CO LTD

Patent Information

Authority / Receiving Office: CN · China
Patent Type: Patents(China)
Current Assignee / Owner: MOXIN ARTIFICIAL INTELLIGENCE TECH (SHENZHEN) CO LTD
Filing Date: 2026-03-13
Publication Date: 2026-06-09

AI Technical Summary

Technical Problem

Existing large language models suffer from severe memory fragmentation, wasted storage and bandwidth, and inability to support low-latency dynamic sparsity and quantization group size reconfiguration when managing key-value caches with heterogeneous precision.

Method used

A key-value cache management method is adopted, which generates sparse key data, sparse index and sparse scaling factor through sparse processing and quantization processing, and stores them in a unified heterogeneous physical block. Combined with the compile-time constant offset table, it realizes integrated management of heterogeneous precision key-value cache, and supports dynamic adjustment of sparsity rate and quantization group size.

Benefits of technology

It solves the problems of memory fragmentation, storage and bandwidth waste, improves the response speed and stability of large language models, and supports low-latency dynamic sparsity and quantization group size reconfiguration.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure CN121882113B_ABST

Patent Text Reader

Abstract

This disclosure relates to a key-value cache management method and apparatus. A key-value cache management method is provided, comprising: obtaining a pair of key data K and value data V from a large language model; performing sparsification processing on the key data K to generate sparsed key data K; storing the sparsed key data K, a sparse index, and a sparse scaling factor into a unified heterogeneous physical block of the key-value cache based on a compile-time constant offset table; performing quantization processing on the key data K and value data V to generate quantized key data K and quantized value data V; and storing the quantized key data K, the key data quantization scaling factor, the quantized value data V, and the value data quantization scaling factor into the unified heterogeneous physical block based on the compile-time constant offset table, wherein the compile-time constant offset table is configured to locate multiple fields in the unified heterogeneous physical block.

Need to check novelty before this filing date? Find Prior Art