Key value cache processing method and device, equipment and storage medium

Abstract

The invention discloses a key value cache processing method and device, equipment and a storage medium, the method is applied to edge equipment in communication connection with remote storage, when a large language model service is used for reasoning, the first capacity of a local key value cache is detected, and when a preset capacity threshold value is reached, the first capacity of the local key value cache is determined. And creating and maintaining a sliding window covering a plurality of key value cache entries in a local memory of the edge device. New tokens are continuously generated along with large language model service reasoning, and local cache content is updated through the sliding window: on one hand, the oldest key value cache entry in the sliding window is removed to release memory space; on the other hand, a first cache entry corresponding to the newly generated token is synchronously obtained from the remote storage and stored in the local memory. According to the invention, communication traffic and delay with remote storage are effectively reduced, and realization of low-memory, low-delay and high-performance LLM reasoning service is facilitated. The technical scheme of the invention can be widely applied to the technical field of artificial intelligence.

Description

[0001] Cross-referencing of related patent applications This invention claims priority to U.S. Provisional Patent Application No. 63 / 821,344, filed June 10, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for key-value caching. Background Technology

[0003] Large Language Models (LLMs) have become a core component of artificial intelligence due to their ability to understand and generate human-like text, giving rise to a range of innovative applications such as real-time virtual assistants and edge-based intelligent systems. To improve the inference efficiency of LLMs across various devices, key-value (KV) caching is a crucial module. It stores the keys and values ​​of all previous lexical units during each LLM inference process to avoid redundant computation. However, deploying LLMs on memory-constrained edge devices such as smartphones and IoT sensors presents significant challenges. These devices typically have extremely limited memory capacity; for example, smartphones use approximately 4–12 GB, while low-end devices may only have 512 MB. Unlike model weights, the size of the KV cache grows linearly with sequence length. For long context scenarios (such as 128K lexical units), even in quantized models (such as 4-bit LLaMA 3.2 3B), the KV cache can still consume 1.5 GB of memory, easily exceeding the available memory budget of edge devices.

[0004] In related technologies, existing key-value (KV) cache management solutions mainly focus on compressing the KV cache (e.g., through quantization, sparsification, or attention-score-based levitation) or offloading the KV cache to larger storage spaces (e.g., from GPU memory to CPU memory or host memory). However, these methods have significant shortcomings in resource-constrained edge environments: on the one hand, the compressed KV cache may still exceed the available memory capacity of the edge device, leading to inference interruptions or performance degradation; on the other hand, offloading the KV cache to remote storage requires the transfer of a large number of tensors (up to 10 GB), which can easily cause high latency, and the synchronous acquisition mechanism can create a communication bottleneck. Furthermore, it is sensitive to network bandwidth fluctuations, making it difficult to meet the stringent low-latency requirements of edge applications.

[0005] In summary, existing technologies struggle to achieve efficient key-value cache management under limited memory and dynamic network conditions. This leads to problems such as insufficient memory, high latency, and poor stability when deploying long-context LLM on edge devices, severely restricting the large-scale application of LLM in edge computing scenarios. Summary of the Invention

[0006] This application provides a key-value caching processing method, apparatus, device, and storage medium, which can effectively reduce the amount of communication and latency with remote storage, and is conducive to realizing low memory, low latency, and high performance LLM inference services.

[0007] One aspect of this application provides a key-value caching method applied to an edge device, wherein the edge device is communicatively connected to remote storage; the method includes: When performing inference using the large language model service, the first capacity of the local key-value cache on the edge device used for the large language model service is detected; In response to the first capacity being greater than or equal to a preset capacity threshold, a sliding window is created and maintained in the local memory of the edge device; wherein the window size of the sliding window covers a number of key-value cache entries; In response to the large language model service generating a new token, the position of the sliding window is slid; Remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve the first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry into the local memory.

[0008] Specifically, in some embodiments, the method further includes: For the large language model service, multiple candidate windows are pre-created; wherein each candidate window has a different window size; Collect memory consumption data and key-value cache communication overhead data of the large language model service when running in each candidate window; A mapping table is established based on the memory consumption data and the key-value cache communication overhead data; wherein, the mapping table is used to record the correspondence between the window size of the candidate window and the memory consumption data and the key-value cache communication overhead data; According to the mapping table, the target window is determined from the candidate windows, and the parameters of the target window are preset onto the sliding window.

[0009] Specifically, in some embodiments, determining the target window from the candidate windows according to the mapping table includes: Based on the mapping table, calculate the comprehensive cost score corresponding to each candidate window; The overall cost scores corresponding to each candidate window are compared, and the candidate window with the best overall cost score is determined as the target window.

[0010] Specifically, in some embodiments, the large language model service includes multiple Transformer layers; the creation and maintenance of a sliding window in the local memory of the edge device includes: The extent to which each Transformer layer depends on the context during inference is detected; Based on the size of the dependency range, determine the window size of the sliding window allocated to each Transformer layer, and create and maintain the sliding window in the local memory.

[0011] Specifically, in some embodiments, retrieving the first key-value cache entry corresponding to the new token from the remote storage and storing the first key-value cache entry in local memory includes: In decoding time step t, identify the index of the first key-value cache entry related to the attention computation of the current token; The index is sent to the remote storage to asynchronously initiate a request to retrieve the first key-value cache entry; While waiting to retrieve the first key-value cache entry from the remote storage, attention computation is performed on the token at decoding time step t using the historical key-value cache entry already cached locally in decoding time step t-1. Receive and store the first key-value cache entry fed back by the remote storage.

[0012] Specifically, in some embodiments, the method further includes, before sliding the position of the sliding window: Send a request to the remote scheduler; wherein the request contains index information of the first key-value cache entry to be retrieved; Receive a scheduling instruction from the remote scheduler, the scheduling instruction being used to instruct the edge device to perform one of the following two operations: Retrieve the first key-value cache entry from the remote storage; or... The current decoding task is uploaded to the cloud computing resources allocated by the remote scheduler for execution.

[0013] On the other hand, embodiments of this application also provide a key-value caching processing apparatus, applied to an edge device, wherein the edge device is communicatively connected to remote storage; the apparatus includes: The detection unit is configured to detect the first capacity of the local key-value cache for the large language model service on the edge device when performing inference using the large language model service; A creation unit is configured to create and maintain a sliding window in the local memory of the edge device in response to the first capacity being greater than or equal to a preset capacity threshold; wherein the window size of the sliding window covers a number of key-value cache entries; A sliding unit is used to slide the position of the sliding window in response to the large language model service generating a new token; An execution unit is configured to remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve the first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry into the local memory.

[0014] On the other hand, embodiments of this application provide a key-value caching processing system, including: an edge device, remote storage, and a remote scheduler; The edge device is configured to, when performing inference using the large language model service, detect a first capacity of the local key-value cache on the edge device for the large language model service; in response to the first capacity being greater than or equal to a preset capacity threshold, create and maintain a sliding window in the local memory of the edge device; wherein the window size of the sliding window covers a number of key-value cache entries; in response to the large language model service generating a new token, slide the position of the sliding window; remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve a first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry in the local memory; The remote storage is used to store key-value cache entries related to the large language model service; The remote scheduler is used to coordinate key-value cache transfers and computation offloading between the edge device and the remote storage.

[0015] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory; The memory is used to store computer programs; The processor executes the computer program to implement the aforementioned key-value caching processing method.

[0016] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned key-value caching processing method.

[0017] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the aforementioned key-value caching processing method.

[0018] The embodiments of this application include at least the following beneficial effects: This application provides a key-value caching processing method, apparatus, device, and storage medium, applied to an edge device communicatively connected to remote storage. When performing inference using a large language model service, the first capacity of the local key-value cache is detected. When the first capacity reaches a preset capacity threshold, a sliding window covering a certain number of key-value cache entries is created and maintained in the local memory of the edge device. As the large language model service continuously generates new tokens through inference, the local cache content is updated through this sliding window: on the one hand, the oldest key-value cache entry in the sliding window is removed to free up memory space; on the other hand, the first cache entry corresponding to the newly generated token is synchronously retrieved from the remote storage and stored in the local memory. This application significantly reduces the local memory usage of the edge device by retaining only the most critical key-value cache entries in the local memory through the sliding window; at the same time, the synchronous replacement strategy of new and old cache entries effectively reduces the communication volume and latency with the remote storage, thereby facilitating the realization of low-memory, low-latency, and high-performance LLM inference services under the resource constraints of the edge device. Attached Figure Description

[0019] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0020] Figure 1 This is a system architecture diagram of the key-value caching processing method provided in the embodiments of this application. Figure 2 This is a flowchart illustrating a key-value caching method provided in an embodiment of this application. Figure 3 This is a schematic diagram of a sliding window provided in an embodiment of this application; Figure 4 This is a structural block diagram of a key-value caching processing device provided in an embodiment of this application; Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0023] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0024] Before introducing the technical solution of this application, the technical terms involved in the embodiments of this application will be introduced as follows: 1) Artificial intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. AI also utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results—theories, methods, technologies, and application systems for this purpose.

[0025] 2) Large Language Model (LLM) is an artificial intelligence program based on deep learning. It learns the patterns, grammar, knowledge, and contextual relationships of human language by training on massive amounts of text data. Its core capability is to understand and generate natural language.

[0026] Large Language Models (LLMs) have become a core component of artificial intelligence due to their ability to understand and generate human-like text, giving rise to a range of innovative applications such as real-time virtual assistants and edge-based intelligent systems. To improve the inference efficiency of LLMs across various devices, key-value (KV) caching is a crucial module. It stores the keys and values ​​of all previous lexical units during each LLM inference process to avoid redundant computation. However, deploying LLMs on memory-constrained edge devices such as smartphones and IoT sensors presents significant challenges. These devices typically have extremely limited memory capacity; for example, smartphones use approximately 4–12 GB, while low-end devices may only have 512 MB. Unlike model weights, the size of the KV cache grows linearly with sequence length. For long context scenarios (such as 128K lexical units), even in quantized models (such as 4-bit LLaMA 3.2 3B), the KV cache can still consume 1.5 GB of memory, easily exceeding the available memory budget of edge devices.

[0027] In related technologies, existing key-value (KV) cache management solutions mainly focus on compressing the KV cache (e.g., through quantization, sparsification, or attention-score-based levitation) or offloading the KV cache to larger storage spaces (e.g., from GPU memory to CPU memory or host memory). However, these methods have significant shortcomings in resource-constrained edge environments: on the one hand, the compressed KV cache may still exceed the available memory capacity of the edge device, leading to inference interruptions or performance degradation; on the other hand, offloading the KV cache to remote storage requires the transfer of a large number of tensors (up to 10 GB), which can easily cause high latency, and the synchronous acquisition mechanism can create a communication bottleneck. Furthermore, it is sensitive to network bandwidth fluctuations, making it difficult to meet the stringent low-latency requirements of edge applications.

[0028] In summary, existing technologies struggle to achieve efficient key-value cache management under limited memory and dynamic network conditions. This leads to problems such as insufficient memory, high latency, and poor stability when deploying long-context LLM on edge devices, severely restricting the large-scale application of LLM in edge computing scenarios.

[0029] In view of this, this application provides a key-value caching method, apparatus, device, and storage medium, aiming to improve the problem that related technologies struggle to efficiently deploy Long Context Large Language Models (LLMs) on resource-constrained edge devices. This application is applied to an edge device communicating with remote storage. When performing inference using a large language model service, the first capacity of the local key-value cache is detected. When the first capacity reaches a preset capacity threshold, a sliding window covering a certain number of key-value cache entries is created and maintained in the edge device's local memory. As the large language model service continuously generates new tokens through inference, the local cache content is updated through this sliding window: on the one hand, the oldest key-value cache entry in the sliding window is removed to free up memory space; on the other hand, the first cache entry corresponding to the newly generated token is synchronously retrieved from the remote storage and stored in the local memory. This application significantly reduces the local memory usage of the edge device by retaining only the most critical key-value cache entries in the local memory through the sliding window; simultaneously, by utilizing a synchronous replacement strategy of old and new cache entries, the communication volume and latency with the remote storage are effectively reduced, thus facilitating the realization of low-memory, low-latency, and high-performance LLM inference services under the resource constraints of the edge device.

[0030] System architecture and scenario description used in the embodiments of this application Please refer to Figure 1 , Figure 1 The diagram shows a system architecture diagram of a key-value caching processing method provided in this application embodiment, which includes a terminal device 140, an Internet 130, a gateway 120, a backend server 110, etc.

[0031] In this embodiment, the terminal device 140 may include various forms such as a desktop computer, laptop computer, PDA (personal digital assistant), mobile phone, vehicle terminal, home theater terminal, and dedicated terminal. Furthermore, it can be a single device or a collection of multiple devices. The terminal device 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0032] Backend server 110 refers to a computer system that can provide certain services to terminal device 140. Compared with ordinary terminal device 140, backend server 110 has higher requirements in terms of stability, security, and performance. Backend server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0033] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal device 140 to backend server 110 are forwarded to the corresponding backend server 110 via gateway 120. Messages sent from backend server 110 to terminal device 140 are also forwarded to the corresponding terminal device 140 via gateway 120.

[0034] The backend server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0035] The key-value caching processing method provided in this application embodiment can be implemented on terminal device 140. Specifically, when terminal device 140 needs to run a large language model service for local inference, the method starts operating, and terminal device 140 will detect in real time the key-value cache capacity allocated to the large language model service in its local memory. Once the capacity of the local key-value cache is detected to reach or exceed a preset threshold, terminal device 140 will create and maintain a sliding window in its local memory. In subsequent inference processes, whenever the large language model service generates a new token, terminal device 140 will drive the sliding window to slide forward. Its core operations include two aspects: first, removing the oldest key-value cache entry currently within the window from local memory to free up space; second, synchronously retrieving the first key-value cache entry corresponding to the newly generated token from a remote storage system (usually deployed in the backend server 110 or associated cloud storage) connected via Internet 130 and gateway 120, and storing it in local memory. In this way, the terminal device 140 can efficiently manage context sequences that are much larger than its memory capacity with its extremely limited memory resources, ensuring low latency and stable execution of large language model inference tasks.

[0036] Of course, it is understood that the implementation environment corresponding to the method in the embodiments of this application is not limited to that of the implementation environment. Figure 1 As shown, those skilled in the art can flexibly select the specific implementation environment according to actual needs, and this application does not impose any restrictions on this.

[0037] General Description of Embodiments in this Application The key-value caching processing method in this application embodiment can be applied to systems including edge devices and remote storage. This method can be executed by an edge device that is communicatively connected to the remote storage. Specifically, the edge device is the main entity executing LLM inference and implementing the core cache management method in this application. It refers to devices that are physically close to the data source or user but have limited computing and memory resources. Generally, edge devices possess certain local computing power and memory, but may not be sufficient to independently handle complete, especially long-context, LLM inference tasks. For example, edge devices may include, but are not limited to, the following device types: mobile smart terminals: such as smartphones, tablets, augmented reality / virtual reality glasses, etc.; IoT terminals: such as central controllers (smart speakers, smart gateways) in smart homes, sensor gateways in industrial IoT, surveillance cameras, etc.; embedded devices: such as in-vehicle intelligent systems, drones, retail terminals, etc.

[0038] In this embodiment, remote storage is a key infrastructure for providing memory expansion for edge devices. It refers to a server-side storage system that is physically separated from the edge device, connected via a network, and provides large-capacity storage, possessing storage capacity and reliable persistence capabilities far exceeding the local memory of the edge device. Exemplarily, it can be deployed in the following locations: Edge network side: such as storage for edge servers deployed at base stations, shopping malls, or factories. This deployment method, being close to the edge device, helps reduce access latency. Core cloud data center: i.e., traditional cloud storage services, such as object storage or high-performance distributed file systems. They can provide near-unlimited elastic storage space.

[0039] Of course, it is understandable that the specific implementation of the actual edge devices and remote storage can be flexibly configured according to requirements, and this application does not impose any restrictions on this.

[0040] Please refer to Figure 2 , Figure 2 A flowchart illustrating a key-value caching method provided in an embodiment of this application is shown. Figure 2 As shown, a key-value caching processing method according to an embodiment of this application includes, but is not limited to, the following steps: Step 210: When performing inference using the large language model service, detect the first capacity of the local key-value cache on the edge device used for the large language model service; Step 220: In response to the first capacity being greater than or equal to a preset capacity threshold, a sliding window is created and maintained in the local memory of the edge device; wherein the window size of the sliding window covers a number of key-value cache entries; Step 230: In response to the large language model service generating a new token, slide the position of the sliding window; Step 240: Remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve the first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry into the local memory.

[0041] This application provides a method for processing key-value caches. This method is applied to an edge device, which is communicatively connected to remote storage. When the first capacity of the local key-value cache used for a large language model service on the edge device reaches or exceeds a preset capacity threshold, a sliding window is created and maintained in local memory. The window size of this sliding window covers a number of key-value cache entries. As the inference process of the large language model service generates new tokens, the method dynamically manages the local cache content by sliding the position of this sliding window. Specifically, each time the window is slid, the oldest key-value cache entry currently in the window is removed from local memory. Simultaneously, the first key-value cache entry corresponding to the new token is retrieved from remote storage and stored in local memory. Through this synchronous replacement mechanism based on a sliding window, efficient management of the key-value cache is achieved on resource-constrained edge devices, effectively reducing local memory usage and remote communication overhead.

[0042] In this embodiment, the large language model service deployed on the edge device refers to a service model that directly integrates and runs the inference capabilities of a large language model in the local environment of the edge device. This service enables the edge device to independently execute LLM inference tasks such as text generation, question answering, summarization, and translation without relying on a continuous and stable connection to a remote cloud, thereby meeting the edge computing requirements of low latency and high data privacy protection. Generally, a large language model service typically includes optimized model weight files, a dedicated inference engine, and runtime libraries. Considering the limited computing resources and memory capacity of edge devices, the deployed models are often optimized versions using model compression techniques such as pruning, quantization, or knowledge distillation, such as 4-bit or 8-bit quantized models, to significantly reduce model size and memory usage while ensuring certain performance, enabling them to be loaded and executed in resource-constrained edge environments.

[0043] For example, the operation of the large language model service is specifically manifested as follows: the edge device receives user input (such as voice commands, textualized results of sensor data, etc.) triggered or generated locally, and the service calls the local computing unit (such as CPU, GPU, or NPU) to perform forward inference computation of LLM, and gradually generates the output sequence autoregressively. In this process, the key-value cache generated to maintain the inference state and avoid repeated computation is efficiently managed by the key-value cache processing method provided in this application, so as to overcome the cache bloat caused by the growth of the context sequence.

[0044] The following section provides a detailed description and explanation of a key-value caching method provided in this application embodiment, along with specific process steps.

[0045] In step 210, when the edge device performs inference using the large language model service, it detects the capacity of the local key-value cache used for the large language model service. In this embodiment, this capacity is referred to as the first capacity.

[0046] It is easy to understand that when an edge device runs a large language model service and performs generative inference tasks, due to the autoregressive nature of LLM, its key-value cache data will grow linearly with the increase in the number of generated tokens. In this embodiment, the first capacity refers to the space occupied by all key-value caches corresponding to the currently generated token sequences in the local memory of the edge device.

[0047] The specific implementation of detecting the initial capacity of the local key-value cache for the large language model service can be flexibly implemented according to requirements. For example, in some embodiments, the memory management interface provided by the operating system or the underlying inference framework can be utilized. When initializing the large language model service, a dedicated memory area is allocated for the current inference session to store the key-value cache. Subsequently, its current usage can be obtained by querying the memory statistics of this specific memory area. In some embodiments, a capacity calculator can be built into the key-value cache management module. This calculator can perform real-time estimation based on the data structure characteristics of the key-value cache. For example, the initial capacity can be dynamically calculated using a predefined formula: Capacity = Batch size × Sequence length × Number of layers × 2 (Key and Value) × Number of attention heads × Dimension of each head × Number of bytes occupied by each parameter. With this formula, the management system can immediately calculate the theoretical memory consumption of the current key-value cache after each new token is generated and the sequence length increases, and use it as the initial capacity.

[0048] It should be noted that the above detection behavior can be periodic or triggered after each new token is generated. Its purpose is to monitor the usage status of local memory in real time, determine whether the current cache capacity has reached the device resource boundary, and provide a decision basis for the subsequent activation of the sliding window management mechanism.

[0049] In step 220, the detected first capacity can be compared with a preset capacity threshold to determine their relative sizes. In this embodiment, the preset capacity threshold is a configurable parameter whose value can be determined comprehensively based on the total memory size of the edge device, the memory requirements of the operating system and other applications, and the safe memory boundary reserved for LLM services, to ensure stable system operation. For example, the threshold can be set to an absolute value (e.g., 1GB) or a percentage relative to the total available memory (e.g., 80% of available memory).

[0050] In this embodiment, the detection logic runs continuously. Once the measured first capacity reaches or exceeds the preset capacity threshold, the subsequent sliding window creation and maintenance process is immediately triggered. This ensures that local memory usage remains within a safe boundary, preventing inference service interruptions or device crashes due to memory exhaustion. Conversely, if the first capacity does not exceed the preset capacity threshold, it indicates that the remaining space in the local memory is still sufficient to accommodate the continuously growing full key-value cache. In this case, the system will maintain the standard full-cache inference mode without creating or activating the sliding window management mechanism. Specifically, the system will continue the normal autoregressive inference process of the large language model. Each time a new token is generated, all the calculated key-value caches (including those generated by the new token itself and those of all historical tokens required to maintain the attention mechanism) will be completely retained in the local memory of the edge device. At this time, there is no need to perform operations such as evicting old cache entries from local memory or asynchronously retrieving cache entries from remote storage. The inference process is directly based on the continuously growing sequence of full key-value caches in local memory, which can achieve the best model inference accuracy because all contextual information is immediately available, and there is no information loss that may be introduced due to cache unloading.

[0051] When the initial capacity is greater than or equal to a preset capacity threshold, a sliding window will be created and maintained in the local memory of the edge device to initiate an efficient cache management mode. In this embodiment, the creation of the sliding window is an initialization process. Specifically, a fixed storage area can be allocated in the local memory, the size of which is determined by a predefined window size, which specifies the number of key-value cache entries that the window can cover (e.g., a cache corresponding to the most recent N tokens). The creation process includes initializing the window's data structures, such as pointers or indices pointing to the start and end positions of the current window, and allocating the necessary memory space to accommodate these cache entries. At this point, it is necessary to determine which critical key-value cache entries in the current lengthy sequence should be retained within this initial window.

[0052] For example, please refer to Figure 3 , Figure 3 A schematic diagram of a sliding window provided in an embodiment of this application is shown. Figure 3 In the middle, the sliding window covers 5 key-value cache entries, and its size can be flexibly adjusted according to needs in practical applications.

[0053] In step 230, after the sliding window is created, the cache management process begins. The sliding of the window position is a key dynamic operation in this process. This operation is directly triggered by the event that the large language model service successfully generates a new token each time. Essentially, the cached content that the sliding window focuses on shifts forward to reflect the progress of the sequence and concentrate limited local memory resources on the context that may be needed for subsequent computations.

[0054] Specifically, the sliding window can be logically viewed as a fixed-length view over a lengthy sequence of tokens. When a new token is generated, the end of this sequence is moved forward by one position. To maintain the timeliness of the window's coverage, the start and end positions of the window need to be moved synchronously by one unit in the direction of the sequence update. This means that the token that was originally at the very front of the window, i.e., the first one included in the window, will be moved out of the window's view.

[0055] From an implementation perspective, the sliding operation is typically accomplished by updating the data structure that maintains the window boundaries, such as incrementing a pointer or index pointing to the window's starting position. This action itself is lightweight and instantaneous; it does not immediately involve the physical movement or deletion of large-scale data. However, this logical position sliding directly triggers the specific cache content update strategy in subsequent step 240. Therefore, the sliding of the window position is the core control signal connecting the new token generation event and the specific cache replacement action, ensuring that cache management remains strictly synchronized with the model's inference process.

[0056] In step 240, the physical data operations required after the sliding window is slid are specifically performed. The core of this is the synchronous replacement strategy of "one out and one in", which aims to ensure the relevance of the window content to the current inference context while maintaining a constant local memory usage.

[0057] Specifically, in this step, the system locates the oldest key-value cache entry in the current window—that is, the entry corresponding to the earliest token included in the window—based on the window's metadata (such as pointers or indexes). This entry, because its associated context is furthest from the newly generated token, is generally considered to have relatively low importance for subsequent computations. Subsequently, the system performs a removal operation, which is not a simple data deletion but an orderly resource release process. This may include, for example, unlinking the entry from the cache index and notifying the memory management unit that the memory block it occupies has been released and can be reallocated, thereby ensuring that local memory space is safely and effectively reclaimed.

[0058] Simultaneously with the removal operation, the system initiates a fetch request to remote storage. This request aims to retrieve the first key-value cache entry corresponding to the newly generated token. This first key-value cache entry can be a key-value vector generated by the new token itself, which is necessary for subsequent attention calculations. This process is synchronous, meaning that the input / output operations for removing the old entry and fetching the new entry overlap in time, rather than being executed sequentially. This concurrency is crucial for reducing latency, allowing the time required to fetch data from the relatively slow remote storage to be partially or completely masked by the local removal operation and any other possible computations.

[0059] Once the required first key-value cache entry is successfully retrieved from remote storage, the system immediately writes it to the local memory space previously freed up by removing the oldest entry. After storage is complete, the sliding window's metadata (such as its end position) is updated to formally confirm that the new entry has become part of the window. Through this series of coordinated operations, step 240 ensures that the local cache content is always a dynamic, cache subset focused on the most recent context, thereby enabling continuous support for long-context, large-language-model inference under strictly limited edge device memory conditions.

[0060] It is understood that the key-value caching method provided in this application embodiment detects the first capacity of the local key-value cache. When the first capacity reaches a preset capacity threshold, a sliding window covering a certain number of key-value cache entries is created and maintained in the local memory of the edge device. As the Large Language Model Service continuously generates new tokens for inference, the local cache content is updated through this sliding window: on the one hand, the oldest key-value cache entry in the sliding window is removed to free up memory space; on the other hand, the first cache entry corresponding to the newly generated token is synchronously retrieved from remote storage and stored in local memory. This application significantly reduces the local memory usage of the edge device by retaining only the most critical key-value cache entries in local memory through the sliding window; at the same time, the synchronous replacement strategy of new and old cache entries effectively reduces the communication volume and latency with remote storage, thereby facilitating the realization of low-memory, low-latency, and high-performance LLM inference services under the resource constraints of the edge device.

[0061] Specifically, in some embodiments, the method further includes: For the large language model service, multiple candidate windows are pre-created; wherein each candidate window has a different window size; Collect memory consumption data and key-value cache communication overhead data of the large language model service when running in each candidate window; A mapping table is established based on the memory consumption data and the key-value cache communication overhead data; wherein, the mapping table is used to record the correspondence between the window size of the candidate window and the memory consumption data and the key-value cache communication overhead data; According to the mapping table, the target window is determined from the candidate windows, and the parameters of the target window are preset onto the sliding window.

[0062] This application provides a pattern recognition method for window size optimization, aiming to address the challenge of determining the optimal window size in a sliding window mechanism. Generally, the relationship between the sliding window size, memory consumption, and key-value cache communication overhead remains stable across different scenarios. Based on this stability, this application applies the optimized configuration obtained through offline analysis in the early stages of decoding to the entire decoding process.

[0063] Specifically, this method first performs a pre-analysis of the system. For the large language model service, a series of different candidate window sizes are pre-defined to form a complete test range. Subsequently, in a controlled environment, data on the actual memory consumption and key-value cache communication overhead of the system are collected when the large language model service runs under each candidate window size. These data objectively reflect the impact of different window sizes on the two key resources.

[0064] After data acquisition, a mapping table is established based on the obtained memory consumption and communication overhead data. The core function of this mapping table is to accurately record the correspondence between each candidate window size and its resulting system overhead (i.e., memory consumption and communication overhead), thereby quantifying complex trade-offs. Based on this mapping table, an optimal target window can be determined from all candidate window sizes according to a predefined optimization objective (e.g., minimizing communication overhead while meeting memory upper limit constraints, or seeking the minimum combined overhead of memory and communication). This target window is the globally optimal or near-optimal solution determined through a data-driven approach during the pre-analysis phase. It will be preset as the parameters of the sliding window and applied to subsequent actual decoding tasks that may involve long contexts.

[0065] It is understood that, in this embodiment of the application, this optimization method based on offline pattern recognition can lock an efficient window size before decoding begins, thereby ensuring a better balance between local memory usage and remote communication overhead throughout the inference process, without the need for complex dynamic adjustments online, thus taking into account both optimization effect and operating efficiency.

[0066] Specifically, in some embodiments, determining the target window from the candidate windows according to the mapping table includes: Based on the mapping table, calculate the comprehensive cost score corresponding to each candidate window; The overall cost scores corresponding to each candidate window are compared, and the candidate window with the best overall cost score is determined as the target window.

[0067] In this embodiment of the application, the process of determining the target window from the candidate window can be an optimization decision-making process based on quantitative evaluation. By introducing a computable comprehensive cost score, the two different dimensions of memory consumption and communication cost are unified into a comparable scalar value, thereby providing objective and accurate data basis for selecting the optimal window size.

[0068] Specifically, when determining the target window, a comprehensive cost score can be calculated for each candidate window size recorded in a pre-established mapping table. This score is a quantitative value, calculated to comprehensively reflect the total resource overhead of the system under the given window size configuration. For example, this comprehensive cost score can be a function of memory consumption data and key-value cache communication overhead data. Generally, this function can be a weighted sum function, i.e., comprehensive cost score = α * standardized memory consumption data + β * standardized key-value cache communication overhead data, where α and β are weighting coefficients, representing the system designer's emphasis on memory efficiency and bandwidth efficiency, respectively, and their sum is 1. The weighting coefficients can be adjusted according to the specific resource constraints of the edge device and the sensitivity to latency and cost in the application scenario. For example, on devices with extremely tight memory, memory consumption can be given a higher weight (α > β). This application does not limit the specific values ​​of these two coefficients.

[0069] After calculating the overall cost score corresponding to all candidate window sizes, the decision-making process enters the comparison phase. The system iterates through and compares the overall cost scores corresponding to each candidate window, and sorts them according to the optimization objective (usually minimizing overall cost). Finally, the candidate window with the optimal overall cost score (i.e., the smallest value, representing the minimum total cost) is officially determined as the target window.

[0070] It is understood that the decision-making mechanism based on score comparison in this application transforms the complex multi-objective trade-off problem into an intuitive single-objective optimization problem, making the decision logic clear and reproducible, and able to adapt to different resource preference settings, thereby quickly selecting the most balanced and efficient window size configuration under specific constraints.

[0071] Specifically, in some embodiments, the large language model service includes multiple Transformer layers; the creation and maintenance of a sliding window in the local memory of the edge device includes: The extent to which each Transformer layer depends on the context during inference is detected; Based on the size of the dependency range, determine the window size of the sliding window allocated to each Transformer layer, and create and maintain the sliding window in the local memory.

[0072] This application provides a fine-grained sliding window management method based on differences between Transformer layers. Generally, large language model services can be built based on Transformer layers. However, in practical applications, it has been found that the dependencies of multiple Transformer layers in a large language model service on context information vary significantly, and not all layers require equal long-range context support. Based on this finding, this application breaks away from the traditional approach of setting a uniform window size for all layers, instead adopting a layer-by-layer customization strategy to achieve more refined control over resource allocation.

[0073] Specifically, when establishing the sliding window, we can first proceed to the analysis phase of the Transformer layers, aiming to detect the extent of context dependence of each Transformer layer during inference. This dependence can be quantified early in the decoding process, such as when processing a calibration sequence, by analyzing the distribution characteristics of the attention weights of each layer. For example, layers with attention weights highly concentrated on the most recent tokens can be judged to have a smaller dependence; while layers with more evenly distributed attention weights and a broad focus on historical context can be judged to have a larger dependence. This hierarchical dependence pattern is inherent in the model structure and remains stable throughout the decoding process; therefore, the results of early analysis have global guiding significance.

[0074] After obtaining the dependency scope characteristics of each layer, the configuration phase begins. Based on the detected dependency scope of each Transformer layer, the system independently determines and allocates a dedicated sliding window size. For layers with smaller dependency scopes, the system allocates a smaller window size, sufficient to capture their key local context while minimizing local memory usage. Conversely, for layers with larger dependency scopes requiring a broader context view, the system allocates a larger window size to ensure sufficient critical historical context is retained locally, reducing the frequency of requests to remote storage due to missing context and lowering communication latency. Finally, a sliding window with its independently optimized window size is created and maintained in the local memory of the edge device for each Transformer layer.

[0075] It is understood that the adaptive mechanism of layer-by-layer customized windows in this application embodiment can intelligently balance memory efficiency and computing performance, and accurately allocate valuable memory resources to the model layer that needs the most long context, thereby optimizing resource utilization efficiency and bandwidth requirements as a whole.

[0076] Specifically, in some embodiments, retrieving the first key-value cache entry corresponding to the new token from the remote storage and storing the first key-value cache entry in local memory includes: In decoding time step t, identify the index of the first key-value cache entry related to the attention computation of the current token; The index is sent to the remote storage to asynchronously initiate a request to retrieve the first key-value cache entry; While waiting to retrieve the first key-value cache entry from the remote storage, attention computation is performed on the token at decoding time step t using the historical key-value cache entry already cached locally in decoding time step t-1. Receive and store the first key-value cache entry fed back by the remote storage.

[0077] This application provides an asynchronous key-value cache prefetching method that hides data retrieval latency through overlapping communication and computation, thereby significantly improving the efficiency of the decoding process. In large language model services, the important contextual information relied upon by adjacent tokens during attention computation often has a high degree of similarity. Utilizing this principle of temporal locality, this application innovatively introduces an asynchronous retrieval paradigm, breaking the resource idleness and latency problems caused by the traditional synchronous retrieval method, which requires waiting for data to be ready before computation.

[0078] Specifically, the execution process of this embodiment is tightly coupled with the model's decoding time step. At the start of decoding time step t (t is a positive integer), the edge device first identifies the indices of the key-value cache entries necessary for attention computation of the current token. These indices are immediately sent to remote storage to initiate an asynchronous data retrieval request. Unlike synchronous operations, the system does not block waiting for data to return after this, but immediately performs a crucial operation: directly using historical key-value cache entries that were cached locally in the previous decoding time step (t-1), which may be slightly outdated but highly relevant, to perform attention computation for the current token at time step t. This design cleverly utilizes the similarity of adjacent token contexts, allowing the computation task to continue while waiting for the latest cache entries to be retrieved, thereby transforming the communication and computation processes that should occur sequentially into parallel processing.

[0079] In this embodiment, the asynchronous overlap mechanism effectively hides the communication latency caused by retrieving data from remote storage behind the local computing task. When the decoding process enters the next time step (t+1) and needs to process the next layer of computation, the key-value cache entry requested at time step t has theoretically been transmitted and stored locally, making it available for use. This ensures the smooth progress of the decoding process and avoids waiting delays.

[0080] It should be noted that in the first decoding step, since there is no local cache, a synchronous initialization is required to fill the basic cache. All subsequent steps can benefit from this asynchronous prefetching mechanism. In this way, the embodiments of this application greatly reduce decoding latency while paying only a minimal cost in accuracy, achieving efficient and low-latency large language model inference on edge devices.

[0081] Specifically, in some embodiments, the method further includes, before sliding the position of the sliding window: Send a request to the remote scheduler; wherein the request contains index information of the first key-value cache entry to be retrieved; Receive a scheduling instruction from the remote scheduler, the scheduling instruction being used to instruct the edge device to perform one of the following two operations: Retrieve the first key-value cache entry from the remote storage; or... The current decoding task is uploaded to the cloud computing resources allocated by the remote scheduler for execution.

[0082] This application provides a local-remote collaborative dynamic scheduling mechanism designed to address extreme scenarios such as drastic network fluctuations or severely limited local resources, ensuring the overall robustness of the inference service. Its main application involves introducing a centralized intelligent scheduler, enabling edge device decoding strategies to be dynamically adjusted based on real-time global resource conditions, thereby achieving optimal utilization of system resources.

[0083] Specifically, before the edge device prepares to slide its local sliding window to obtain a new key-value cache entry, this embodiment adds a decision-making step: the edge device first sends a request to a remote scheduler deployed at a remote location, which includes at least the index information of the first key-value cache entry to be obtained. This request is essentially the edge device consulting the remote scheduler on how to proceed with the next step of decoding.

[0084] As the decision-making brain, the remote scheduler, upon receiving a request, comprehensively evaluates multiple real-time factors, primarily including the availability and stability of current network bandwidth, the load status of the remote storage server, and the idle status of GPU resources in the cloud computing cluster. Based on this evaluation, the remote scheduler generates a specific scheduling instruction and sends it to the edge device. This instruction instructs the edge device to perform one of two different operations: The first operation is the normal path, which instructs the edge device to directly retrieve the required first key-value cache entry from the remote storage and continue decoding locally. This is suitable when the network and remote storage are in good condition. The second operation is the degraded or offload path. When the remote scheduler determines that the network bandwidth is too congested, or that the current decoding task is very complex and inefficient to execute locally, it instructs the edge device to upload the entire current decoding task to a more powerful cloud computing resource (such as a GPU instance) dynamically allocated by the scheduler for execution. This means that subsequent decoding work will be completed entirely in the cloud, and the edge device only needs to wait and receive the final inference result.

[0085] Through this collaborative scheduling, the embodiments of this application enable the system to have an adaptive capability: when resources are sufficient, edge computing is prioritized to reduce latency and protect privacy; when resources are scarce, cloud computing power can be seamlessly used to ensure task completion and service quality, thereby maintaining a reliable and efficient large language model service in various complex real-world deployment environments.

[0086] In this embodiment of the application, a key-value caching processing system is also provided, including: an edge device, remote storage, and a remote scheduler; The edge device is configured to, when performing inference using the large language model service, detect a first capacity of the local key-value cache on the edge device for the large language model service; in response to the first capacity being greater than or equal to a preset capacity threshold, create and maintain a sliding window in the local memory of the edge device; wherein the window size of the sliding window covers a number of key-value cache entries; in response to the large language model service generating a new token, slide the position of the sliding window; remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve a first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry in the local memory; The remote storage is used to store key-value cache entries related to the large language model service; The remote scheduler is used to coordinate key-value cache transfers and computation offloading between the edge device and the remote storage.

[0087] In this application embodiment, a key-value caching system for deploying large language models in resource-constrained environments is also provided. This system constructs an efficient, adaptive, and robust inference framework through the collaborative work of edge devices, remote storage, and a remote scheduler.

[0088] The core of this system is the edge device, which acts as the direct executor of the LLM inference task and integrates the core management module of this application. An adaptive sliding KV cache window manager running on the edge device is responsible for maintaining a subset of key KV cache entries in its limited local memory. This manager can dynamically adjust the window size based on different context dependencies of the Transformer layer, selectively evicting old entries and fetching new ones from the remote storage, thereby transforming the memory footprint of the KV cache from a linearly growing variable into a controllable constant. Furthermore, the asynchronous KV cache prefetcher on the edge device identifies key cache entries and cleverly performs local computation while asynchronously fetching the subsequently needed data from remote storage, achieving an overlap of communication and computation. This transforms the original serial waiting process into a parallel pipeline, significantly reducing the decoding latency caused by data acquisition.

[0089] As the system's capacity storage device, the remote storage system is responsible for storing complete long context sequence KV cache entries, providing edge devices with almost unlimited cache space expansion, thereby breaking through the hard limitations of the edge devices' own physical memory.

[0090] The remote scheduler acts as the system's intelligent command center, serving as a local-remote collaborative service scheduler. It does not directly store data but dynamically coordinates data flow and computational tasks between edge devices and remote storage based on real-time task requirements (such as computational complexity) and network conditions (such as available bandwidth). Its core function is to make optimal scheduling decisions: whether to instruct edge devices to retrieve data from remote storage and continue local inference, or, in cases of poor network conditions or excessive workload, to directly offload computational tasks to powerful cloud computing resources. This dynamic resource allocation mechanism ensures that the system can meet the requirements of low-latency, high-precision inference services even under various fluctuating environments.

[0091] It is understood that the system in this application embodiment, by organically combining adaptive cache management, asynchronous prefetching and intelligent cooperative scheduling, has successfully enabled the efficient operation of large language models requiring long contexts on memory-constrained edge devices, providing key infrastructure support for the popularization of core AI capabilities at the edge.

[0092] Reference Figure 4 In this embodiment of the application, a key-value cache processing apparatus is also provided, which includes: Detection unit 410 is used to detect the first capacity of the local key-value cache for the large language model service on the edge device when performing inference using the large language model service; Creation unit 420 is used to create and maintain a sliding window in the local memory of the edge device in response to the first capacity being greater than or equal to a preset capacity threshold; wherein the window size of the sliding window covers a number of key-value cache entries; Sliding unit 430 is used to slide the position of the sliding window in response to the generation of a new token by the large language model service; The execution unit 440 is configured to remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve the first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry into the local memory.

[0093] It is understandable that, such as Figure 2 The content of the key-value caching processing method embodiments shown are all applicable to the key-value caching processing device embodiments of this invention. The specific functions implemented by the key-value caching processing device embodiments of this invention are the same as those shown in the examples. Figure 2 The key-value caching processing method shown in the embodiment is the same, and the beneficial effects achieved are the same as those described above. Figure 2 The beneficial effects achieved by the illustrated key-value caching processing method embodiment are also the same.

[0094] Reference Figure 5 This application provides an electronic device, including: At least one processor 510; At least one memory 520 is used to store at least one program; When at least one program is executed by at least one processor 510, the at least one processor 510 enables... Figure 2 This illustrates a key-value caching method.

[0095] Similarly, the content of the above method embodiments is applicable to the embodiments of this electronic device. The specific functions implemented by the embodiments of this electronic device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0096] This application embodiment also provides a computer-readable storage medium storing a program executable by a processor 510, which, when executed by the processor 510, is used to perform the above-described key-value cache processing method.

[0097] It is understandable that, such as Figure 2The content of the key-value caching processing method embodiment shown is applicable to the embodiment of this computer-readable storage medium. The specific functions implemented by the embodiment of this computer-readable storage medium are the same as those shown below. Figure 2 The key-value caching processing method shown is the same as that embodiment, and the beneficial effects achieved are the same as those shown. Figure 2 The beneficial effects achieved by the illustrated key-value caching processing method embodiment are also the same.

[0098] This application also discloses a computer program product or computer program, which includes computer instructions stored in the aforementioned computer-readable storage medium. Figure 5 The processor of the illustrated electronic device can read the computer instructions from the aforementioned computer-readable storage medium, and the processor executes the computer instructions, causing the electronic device to perform... Figure 2 The illustrated example shows a method for processing key-value caches.

[0099] Understandable Figure 2 The key-value caching processing methods shown in the embodiments are all applicable to this computer program product or computer program embodiment. The specific functions implemented by this computer program product or computer program embodiment are the same as those described above. Figure 2 The key-value caching processing method shown in the embodiment is the same, and the beneficial effects achieved are the same. Figure 2 The beneficial effects achieved by the illustrated key-value caching processing method embodiment are also the same.

[0100] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0101] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0102] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0104] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0105] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0106] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0107] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0108] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for processing key-value caching, characterized in that, The method is applied to an edge device, which is communicatively connected to remote storage; the method includes: When performing inference using the large language model service, the first capacity of the local key-value cache on the edge device used for the large language model service is detected; In response to the first capacity being greater than or equal to a preset capacity threshold, a sliding window is created and maintained in the local memory of the edge device; wherein the window size of the sliding window covers a number of key-value cache entries; In response to the large language model service generating a new token, the position of the sliding window is slid; Remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve the first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry into the local memory.

2. The key-value caching processing method according to claim 1, characterized in that, The method further includes: For the large language model service, multiple candidate windows are pre-created; wherein each candidate window has a different window size; Collect memory consumption data and key-value cache communication overhead data of the large language model service when running in each candidate window; A mapping table is established based on the memory consumption data and the key-value cache communication overhead data; wherein, the mapping table is used to record the correspondence between the window size of the candidate window and the memory consumption data and the key-value cache communication overhead data; According to the mapping table, the target window is determined from the candidate windows, and the parameters of the target window are preset onto the sliding window.

3. The key-value caching processing method according to claim 2, characterized in that, The step of determining the target window from the candidate windows according to the mapping table includes: Based on the mapping table, calculate the comprehensive cost score corresponding to each candidate window; The overall cost scores corresponding to each candidate window are compared, and the candidate window with the best overall cost score is determined as the target window.

4. The key-value caching processing method according to claim 1, characterized in that, The large language model service includes multiple Transformer layers; the creation and maintenance of a sliding window in the local memory of the edge device includes: The extent to which each Transformer layer depends on the context during inference is detected; Based on the size of the dependency range, determine the window size of the sliding window allocated to each Transformer layer, and create and maintain the sliding window in the local memory.

5. The key-value caching processing method according to claim 1, characterized in that, The step of retrieving the first key-value cache entry corresponding to the new token from the remote storage and storing the first key-value cache entry in local memory includes: In decoding time step t, identify the index of the first key-value cache entry related to the attention computation of the current token; The index is sent to the remote storage to asynchronously initiate a request to retrieve the first key-value cache entry; While waiting to retrieve the first key-value cache entry from the remote storage, attention computation is performed on the token at decoding time step t using the historical key-value cache entry already cached locally in decoding time step t-1. Receive and store the first key-value cache entry fed back by the remote storage.

6. The method for processing key-value cache according to any one of claims 1-5, characterized in that, Before sliding the position of the sliding window, the method further includes: Send a request to the remote scheduler; wherein the request contains index information of the first key-value cache entry to be retrieved; Receive a scheduling instruction from the remote scheduler, the scheduling instruction being used to instruct the edge device to perform one of the following two operations: Retrieve the first key-value cache entry from the remote storage; or... The current decoding task is uploaded to the cloud computing resources allocated by the remote scheduler for execution.

7. A key-value caching processing apparatus, characterized in that, Applied to an edge device, the edge device and remote storage are connected in communication; the device includes: The detection unit is configured to detect the first capacity of the local key-value cache for the large language model service on the edge device when performing inference using the large language model service; A creation unit is configured to create and maintain a sliding window in the local memory of the edge device in response to the first capacity being greater than or equal to a preset capacity threshold; wherein the window size of the sliding window covers a number of key-value cache entries; A sliding unit is used to slide the position of the sliding window in response to the large language model service generating a new token; An execution unit is configured to remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve the first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry into the local memory.

8. A key-value caching processing system, characterized in that, include: Edge devices, remote storage, and remote schedulers; The edge device is configured to, when performing inference using the large language model service, detect a first capacity of the local key-value cache on the edge device for the large language model service; in response to the first capacity being greater than or equal to a preset capacity threshold, create and maintain a sliding window in the local memory of the edge device; wherein the window size of the sliding window covers a number of key-value cache entries; in response to the large language model service generating a new token, slide the position of the sliding window; remove the oldest key-value cache entry currently in the sliding window from the local memory, synchronously retrieve a first key-value cache entry corresponding to the new token from the remote storage, and store the first key-value cache entry in the local memory; The remote storage is used to store key-value cache entries related to the large language model service; The remote scheduler is used to coordinate key-value cache transfers and computation offloading between the edge device and the remote storage.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the key-value caching processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the key-value caching processing method according to any one of claims 1 to 6.

Images