Key value cache processing method and device, equipment and storage medium

By creating a sliding window locally on the edge device to manage the key-value cache, the problems of insufficient memory and high latency are solved, and a high-performance LLM inference service with low memory and low latency is achieved, meeting the needs of edge computing.

CN121958142APending Publication Date: 2026-05-01THE HONG KONG UNIV OF SCI & TECH
0 Cites 1 Cited by

Patent Information

Application Number
CN202511875927.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-06-10
Filing Date
2025-12-12
Publication Date
2026-05-01

AI Technical Summary

Technical Problem

Existing technologies struggle to efficiently manage key-value caches on memory-constrained edge devices, leading to insufficient memory, high latency, and poor stability during LLM inference, making it difficult to meet the low latency and high performance requirements of edge computing scenarios.

Method used

A sliding window is created in the local memory of the edge device. When the capacity reaches a threshold, the old cache is removed and the new cache is retrieved from the remote storage simultaneously. The local cache content is managed through the sliding window, which reduces memory usage and communication latency.

Benefits of technology

It achieves high-performance LLM inference service with low memory and low latency on edge devices, effectively reducing memory consumption and communication volume through a sliding window strategy, and ensuring the stability and efficiency of LLM inference.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a key value cache processing method and device, equipment and a storage medium, the method is applied to edge equipment in communication connection with remote storage, when a large language model service is used for reasoning, the first capacity of a local key value cache is detected, and when a preset capacity threshold value is reached, the first capacity of the local key value cache is determined. And creating and maintaining a sliding window covering a plurality of key value cache entries in a local memory of the edge device. New tokens are continuously generated along with large language model service reasoning, and local cache content is updated through the sliding window: on one hand, the oldest key value cache entry in the sliding window is removed to release memory space; on the other hand, a first cache entry corresponding to the newly generated token is synchronously obtained from the remote storage and stored in the local memory. According to the invention, communication traffic and delay with remote storage are effectively reduced, and realization of low-memory, low-delay and high-performance LLM reasoning service is facilitated. The technical scheme of the invention can be widely applied to the technical field of artificial intelligence.
Need to check novelty before this filing date? Find Prior Art