LLM decoding stage KV cache offloading methods, systems, storage media, and electronic devices.

Abstract

This invention discloses a method, system, storage medium, and electronic device for unloading KV Cache during the LLM decoding phase. The method includes: writing the KV Cache data items to be unloaded to a ZNS SSD; maintaining the mapping relationship in a globally extended hash index structure; obtaining the current valid physical location information of the target KV Cache data; and performing garbage collection on zones in the ZNS SSD that meet preset reclamation conditions. The system includes: a data writing module, an index maintenance module, a query readback module, and a garbage collection module. This avoids forcibly unloading the KV Cache to the SSD during the computationally intensive pre-filling phase, focusing on optimizing the KV Cache capacity expansion and on-demand readback process during the decoding phase. It reduces index maintenance overhead and device write management overhead, improving cache management efficiency in long context and high-concurrency inference scenarios.

Description

Technical Field

[0001] This invention relates to the fields of large language model inference optimization and computer storage technology, specifically to a method, system, storage medium, and electronic device for unloading KV cache during the LLM decoding stage. Background Technology

[0002] Large Language Models (LLMs) typically include a pre-filling stage and a decoding stage during inference. The pre-filling stage mainly performs parallel computation on the input context and generates an initial key-value cache (KV Cache). The decoding stage generates output token by token and repeatedly accesses the KV Cache corresponding to the historical tokens at each step. As the context length, the number of model layers, the number of attention heads, and the number of concurrent requests increase, the capacity of the KV Cache expands rapidly, becoming an important factor limiting the inference throughput and context length of large language models.

[0003] In existing systems, key-value caches typically reside in GPU memory or CPU memory to reduce read latency. However, GPU memory capacity is limited. When historical contexts are long or multiple requests are executed simultaneously, the system needs to balance capacity, access latency, and recalculation overhead. If historical key-value caches are discarded directly, subsequent recalculation may be required, increasing inference costs. If all key-value caches are kept in the high-speed storage layer, it is difficult to support long contexts and high-concurrency scenarios.

[0004] One feasible approach is to offload part of the KV cache to low-cost, high-capacity storage media such as solid-state drives (SSDs). However, ordinary SSDs are mainly block device interfaces, and their internal address translation and garbage collection are managed in a hidden manner, making it difficult to fully match the characteristics of KV cache, such as append writes, on-demand reads, and concentrated failures throughout its lifecycle. Zoned namespace SSDs (ZNS SSDs) provide sequential write and explicit reset capabilities in units of zones, making them more suitable for handling append-write data. However, their sequential write constraints require the upper-layer system to precisely manage write addresses, validity, and the garbage collection process.

[0005] Furthermore, the pre-filling stage is typically a computationally intensive stage. Forcing the KVCache being generated to be written to the SSD and frequently read back during this stage may increase additional I / O paths and slow down the pre-filling process. Therefore, a more reasonable optimization target is the decoding stage, in which the historical KVCache has already been generated and will be repeatedly read during the subsequent token generation process. The system is more prone to capacity pressure and needs to perform on-demand offloading and reading back between the high-speed storage layer and the SSD.

[0006] The existing KV Cache offloading solution still has several problems:

[0007] (1) The relationship between the index structure and the actual physical write location is not clear enough, which can easily lead to problems such as long search paths and high metadata maintenance overhead;

[0008] (2) Existing storage solutions based on ZNS SSDs organize data by logical partitions and then bind them to physical zones. However, this will over-couple the hash organization method with the ZNS physical layout, which is not conducive to the write allocation and recycling scheduling of KV Cache.

[0009] (3) The handling of old copies of KV Cache is not clear enough and is easily misunderstood as requiring multiple historical versions to be read during query, while the actual system should only read the current valid physical location;

[0010] (4) If it is not possible to determine whether the data in the Zone is still referenced by the current index during garbage collection, invalid data may be migrated incorrectly or valid data may be collected incorrectly. Summary of the Invention

[0011] This invention aims to provide a method, system, storage medium, and electronic device for unloading KV cache during the LLM decoding stage, in order to solve the problems of high index maintenance overhead, mismatch between write and recycling paths, difficulty in determining the validity of old physical copies, and insufficient capacity expansion efficiency during the existing KV cache unloading process.

[0012] Therefore, this invention discloses a method for unloading the KV cache during the LLM decoding stage, comprising the following steps:

[0013] During the decoding phase of large language model inference, the KV Cache data items to be unloaded are determined;

[0014] The KV Cache data items to be unloaded are written to the target Zone in the ZNS SSD in an append-only manner.

[0015] During or after writing, the mapping relationship between the key of the KVCache data item and the current valid physical location information is maintained in the global extended hash index structure in memory;

[0016] Based on the query key, the global extended hash index structure is searched to obtain the current valid physical location information of the target KV Cache data, and the target KV Cache data is read.

[0017] Based on the dirty data statistics of each Zone in the ZNS SSD, garbage collection is performed on the Zones in the ZNS SSD that have reached the preset recycling conditions.

[0018] Furthermore, in the decoding stage of the large language model inference, the KV Cache data items to be unloaded are determined as follows:

[0019] The KV Cache data generated during the pre-filling stage is retained in the high-speed storage layer;

[0020] Once the inference process enters the decoding phase, the KV Cache data items that need to be offloaded from the high-speed storage layer to the ZNS SSD are determined based on the video memory capacity, water level threshold, request concurrency, context length, or cache residency strategy.

[0021] Furthermore, the key of the KV Cache data item includes at least:

[0022] One or more of the following can be used to uniquely identify a KV Cache data item: request identifier, sequence identifier, layer identifier, attention header identifier, token block range, and KV type identifier.

[0023] Furthermore, the current valid physical location information of the target KV Cache data includes at least:

[0024] The Zone identifier, the offset within the Zone, the data length, and one or more combinations of version number, write generation, verification information, or status bits.

[0025] Furthermore, the KV Cache data items to be unloaded are sequentially written to the target Zone in the ZNSSSD using an append-only writing method, specifically as follows:

[0026] Select the currently writable target Zone based on the write allocation strategy;

[0027] The KV Cache data items to be unloaded are appended and written in the order of the write pointers of the target Zone;

[0028] After the write operation is completed, the current valid physical location information corresponding to the KV Cache data item to be unloaded is written to or updated in the global extended hash index structure.

[0029] Furthermore, when the KV Cache data item corresponding to the same Key is rewritten, migrated, evicted, or deleted, only the mapping of the Key in the global extended hash index structure is updated to the current valid physical location or the mapping is deleted, and the old physical copy is recorded as invalid data in the index structure or Zone metadata. During the query, only the current valid physical location information pointed to by the global extended hash index structure is read.

[0030] Furthermore, based on the dirty data statistics of each Zone in the ZNS SSD, garbage collection is performed on the Zones in the ZNS SSD that have reached the preset collection conditions, specifically as follows:

[0031] Calculate the amount or proportion of invalid data in each zone;

[0032] When the amount or proportion of invalid data in the target zone reaches a preset threshold, scan the data items in the target zone;

[0033] By comparing the current valid physical location information of the corresponding Key in the global extended hash index structure, it is determined whether the data item in the target Zone is still valid data;

[0034] Migrate the data items that are still valid to the new writable Zone, update the global extended hash index structure, and perform a Reset operation on the target Zone.

[0035] Furthermore, this invention discloses an LLM decoding stage KV cache offloading system, comprising:

[0036] The index maintenance module is used to establish and maintain the global extended hash index structure in memory, and record the mapping relationship between the key of the KV Cache data item and the current valid physical location information;

[0037] The data writing module is used to write the KV Cache data items to be unloaded into the target Zone in ZNS SSD in an append-only manner during the decoding stage of large language model inference, and to trigger the index maintenance module to update the global extended hash index structure during or after the writing process.

[0038] The query and readback module is used to look up the global extended hash index structure based on the query key, obtain the current valid physical location information of the target KVCache data, and read the target KVCache data.

[0039] The garbage collection module is used to perform garbage collection on the Zones of the ZNS SSD that have reached the preset collection conditions, based on the statistical information of dirty data in each Zone of the ZNS SSD.

[0040] In addition, the present invention discloses a storage medium storing a computer program or program instructions, the computer program or program instructions being used to execute the LLM decoding stage KV Cache offloading method.

[0041] Finally, this invention discloses an electronic device, comprising: a memory and a processor, wherein,

[0042] The memory stores a computer program or program instructions, and the processor implements the LLM decoding stage KV Cache offloading method when executing the computer program or program instructions.

[0043] The LLM decoding stage KV Cache offloading method, system, storage medium, and electronic device disclosed in this invention have at least the following beneficial effects:

[0044] This invention focuses its optimization efforts on the decoding stage, avoiding unnecessary SSD I / O during the computationally intensive pre-filling phase; it maintains the mapping between the key and the current valid physical location through a globally extended memory hash index structure, making the query path independent of physical zone partitions; it reduces random writes and internal device management overhead by sequentially appending actual KV cache data to ZNS SSDs; and it improves the accuracy and space utilization of garbage collection by marking old physical copies that are deleted, evicted, migrated, or replaced as invalid data and verifying whether they are still referenced by the current index during garbage collection.

[0045] Other features and advantages of the invention will be further set forth in the description below, or may be understood through practice of the invention. Attached Figure Description

[0046] Other features and advantages of the present invention will be further described in the following embodiments, wherein:

[0047] Figure 1 This is a flowchart illustrating the KV Cache offloading method during the LLM decoding stage of Embodiment 1 of the present invention.

[0048] Figure 2This is a flowchart illustrating the KV Cache offloading method during the LLM decoding stage of Embodiment 1 of the present invention.

[0049] Figure 3 This is a schematic diagram comparing the end-to-end decoding performance of Embodiment 1 of the present invention under a long context real task load;

[0050] Figure 4 This is a schematic diagram comparing the P99 decoding latency caused by KV Cache unloading and readback under different context lengths in Embodiment 1 of the present invention;

[0051] Figure 5 This is a comparative diagram of different KV Cache offloading schemes in Embodiment 1 of the present invention in terms of host side write amplification, garbage collection overhead, space utilization and index lookup latency.

[0052] Figure 6 This is a schematic diagram of the structure of the KV Cache offloading system in the LLM decoding stage of Embodiment 2 of the present invention;

[0053] Figure 7 This is a schematic diagram of the structure of the electronic device according to Embodiment 3 of the present invention;

[0054] Figure 8 This is a schematic diagram of the structure of the storage medium in Embodiment 4 of the present invention. Detailed Implementation

[0055] Existing large language model inference systems typically store key-value (KV) caches in GPU memory or CPU memory. When the context length and number of concurrent requests are small, this approach can provide low access latency; however, when the system needs to support long contexts, multi-session concurrency, or a larger batch processing scale, the KV cache capacity will quickly exceed the capacity of the high-speed storage layer.

[0056] While some existing offloading methods can migrate KV Cache to SSD, they usually do not distinguish between the different performance bottlenecks of the pre-filling stage and the decoding stage. The pre-filling stage is mainly based on large-scale matrix calculations, and forced offloading at this stage may introduce additional I / O overhead. The decoding stage requires repeated access to the historical KV Cache, which is more susceptible to capacity and readback path. Therefore, this invention limits offloading optimization to the decoding stage to avoid unnecessary interference with the pre-filling calculation path.

[0057] Some existing methods may use a key-based partitioning structure and bind partitions to physical zones, but this approach couples index organization with physical write layout, complicating zone allocation, load balancing, and garbage collection scheduling. This invention uses a memory-globally extended hash index structure, where the hash index only maintains the mapping relationship between the key and the current valid physical location, without requiring a one-to-one correspondence between the hash bucket or logical partition and the physical zone of the ZNS SSD.

[0058] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] like Figure 1 As shown, the process of the KV Cache offloading method in the LLM decoding stage of Embodiment 1 of the present invention includes the following steps:

[0061] S1: In the decoding stage of large language model inference, determine the KV Cache data items to be unloaded;

[0062] In this embodiment, the large language model inference process includes a pre-filling stage and a decoding stage. The pre-filling stage is mainly responsible for parallel computation of the input context and generating an initial KV Cache. This stage is not the target stage for forced unloading in this invention. After entering the decoding stage, the system determines the KV Cache data items that need to be unloaded from the high-speed storage layer to the ZNS SSD based on the video memory capacity, water level threshold, request concurrency, context length, cache residency time, or cache scheduling strategy.

[0063] It should be noted that in this invention, a KV Cache data item can be KV data corresponding to a Token, or it can be a segment of KV data organized according to a continuous Token range, attention layer, attention head, or block granularity. The system can select an appropriate data item granularity based on the data layout of the inference framework.

[0064] S2: Write the KV Cache data items to be unloaded to the target Zone in the ZNS SSD in an append-only manner;

[0065] In this embodiment, the ZNS SSD provides sequential write capability in units of Zone. The data writing module selects the target Zone based on the current writable Zone, remaining space, write pointer position, concurrent write pressure, or recycling status, and sequentially appends the actual KVCache data to the target Zone.

[0066] It should be noted that the selection of the target zone is determined by the write allocation strategy, and is not required to be directly determined by the hash result of the key. The global extended hash index structure does not establish a fixed one-to-one correspondence with the physical zones in the ZNS SSD.

[0067] S3: During or after writing, maintain the mapping relationship between the key of KVCache data items and the current valid physical location information in the global extended hash index structure in memory;

[0068] In this embodiment, a global extended hash index structure is maintained in memory. The entries of the global extended hash index structure are used to record the key of the KV Cache data item and the current valid physical location information corresponding to the key. The key can be obtained by combining fields such as request identifier, sequence identifier, layer identifier, attention header identifier, token block range, KV type identifier, etc. The current valid physical location information includes at least the Zone identifier, the offset within the Zone, and the data length, and may also include the version number, write generation, verification information, or status bit.

[0069] Once the actual KV Cache data has been written, the system writes the physical location of the data item into the global extended hash index structure. When the data corresponding to the same key is migrated or rewritten, the system only updates the mapping of the key to the new current valid physical location. The old physical copy is no longer queried as a valid version, but is marked as invalid data and awaits subsequent garbage collection.

[0070] S4: Based on the query key, look up the global extended hash index structure, obtain the current valid physical location information of the target KV Cache data, and read the target KV Cache data;

[0071] In this embodiment, when the decoding stage needs to access a certain historical KV Cache data item, the query readback module first searches the global extended hash index structure according to the target key. If the corresponding table entry is hit, the current valid physical location information of the target KV Cache data is obtained, and the corresponding data is read from the ZNS SSD according to the Zone identifier, the offset within the Zone, and the data length.

[0072] It should be noted that the old physical copy in this invention is not a logical multiple version for querying. When querying, it is neither necessary nor advisable to read multiple old versions of the same key. Instead, it only reads the valid physical location currently pointed to by the global extended hash index structure. This can avoid repeatedly reading invalid data and reduce readback latency.

[0073] S5: Based on the dirty data statistics in each Zone, perform garbage collection on Zones that meet the preset collection conditions;

[0074] In this embodiment, the garbage collection module maintains dirty data statistics for each Zone. Dirty data includes old physical copies that are no longer referenced by the current valid mapping of the global extended hash index structure due to deletion, eviction, migration, or rewriting. Dirty data statistics may include the number of invalid data entries, the number of invalid data bytes, or the percentage of invalid data.

[0075] When the amount or proportion of dirty data in a target zone reaches a preset threshold, the garbage collection module scans the data items in the target zone and determines whether the data item is still valid by comparing the current valid physical location information of the corresponding key in the global extended hash index structure. For data items that are still valid, the garbage collection module migrates them to a new writable zone and updates the global extended hash index structure. Data items that are no longer referenced by the current index are not migrated. After the valid data migration is completed, the system performs a Reset operation on the target zone to make it re-enter a writable state.

[0076] like Figure 2 As shown, the workflow of the KV Cache offloading method in the LLM decoding stage of Embodiment 1 of the present invention includes:

[0077] During the decoding phase, the KV Cache manager is responsible for identifying data items that need to be unloaded under capacity pressure. The data writing module appends the actual KV Cache to the ZNS SSD. The globally extended hash index structure maintains the mapping from the key to the current valid physical location. The query and readback module looks up the table through the key and reads the corresponding data. The garbage collection module performs valid data migration and Reset collection based on the Zone dirty data statistics.

[0078] Specifically, the feasibility and effectiveness of the present invention will be further verified through experiments. The experiments can select open-source large language models as the test models, such as Llama-2-7B, Llama-2-13B, Qwen2.5-7B-Instruct, or Transformer decoding models of equivalent size. The inference service can adopt mainstream inference frameworks that support KV Cache management, and configure GPU, high-speed host memory, and ZNS SSD in the server. The experimental workload can include a set of long-context real-world tasks such as LongBench, multi-turn dialogue request trajectories such as ShareGPT, and synthetic long-context workloads for controlling variables.

[0079] In experiments, the method of this invention can be compared with the following schemes:

[0080] (1) Pure high-speed storage layer scheme, that is, KV Cache mainly resides in GPU video memory or host memory, and uses eviction or recomputation when the capacity is insufficient;

[0081] (2) KV Cache offloading scheme based on ordinary non-volatile memory express (NVMe) SSD, that is, offloading and reading back through block device interface;

[0082] (3) The ZNS SSD offloading scheme, which does not use a globally extended hash index and a Zone validity check, was not adopted;

[0083] (4) The KVCache offloading scheme for the decoding stage based on global extended hash index and ZNS SSD sequential append writing proposed in this invention.

[0084] Experimental indicators may include at least:

[0085] The decoding phase includes end-to-end throughput, single-token decoding latency, P99 decoding latency, maximum concurrent requests under different context lengths, average latency of KV Cache readback, index lookup overhead, ZNS SSD sequential write throughput, write amplification, space utilization, number of Zone Resets, and the blocking time of background garbage collection on foreground decoding requests.

[0086] like Figure 3As shown, Embodiment 1 of the present invention compares end-to-end decoding performance under a real task load with a long context. The experiment uses Llama-2-7B and Llama-2-13B as test models, and sets the context length to 8K, 16K, 32K and 64K tokens. The results compare CPU DRAM offloading, ordinary NVMe SSD offloading, a scheme that only uses ZNSSSD sequential writing but does not combine global extended hash and effective address maintenance, and the scheme of the present invention.

[0087] Experimental results show that as the context length increases, the decoding throughput of ordinary NVMe SSD Offloading decreases more significantly, while the solution of this invention can maintain more stable decoding performance.

[0088] On Llama-2-7B, the solution of this invention achieves a decoding throughput of 46, 44, 39, and 31 tokens / s at context lengths of 8K, 16K, 32K, and 64K, respectively, which is an improvement of approximately 12.2%, 25.7%, 50.0%, and 72.2% compared to ordinary NVMe SSD offloading.

[0089] On Llama-2-13B, the solution of this invention achieves a Decode throughput of 34, 31, 28, and 20 Tokens / s for the corresponding context lengths, respectively, which is an improvement of approximately 17.2%, 34.8%, 64.7%, and 100.0% compared to ordinary NVMe SSD Offloading.

[0090] The above results demonstrate that, during the long context decoding stage, as the KV Cache capacity pressure increases, this invention can effectively reduce the performance loss caused by cache unloading and readback through the global extended hash index and ZNS SSD sequential append write mechanism.

[0091] like Figure 4 As shown, Embodiment 1 of the present invention compares the P99 decoding latency caused by KV Cache unloading and readback under different context lengths. The experiment also uses Llama-2-7B and Llama-2-13B as test models, and sets the context length to 8K, 16K, 32K and 64K Tokens. The P99 Decode latency is used to measure the change of tail latency during long context decoding.

[0092] Experimental results show that, compared with ordinary NVMe SSD offloading, the solution of the present invention can significantly reduce the tail decoding latency in long context scenarios;

[0093] On Llama-2-7B, the P99 Decode latency of this invention at context lengths of 8K, 16K, 32K, and 64K is 31, 35, 42, and 57ms / Token, respectively, which are approximately 11.4%, 23.9%, 38.2%, and 48.2% lower than that of ordinary NVMe SSD offloading.

[0094] On Llama-2-13B, the P99 Decode latency of the proposed solution at the corresponding context lengths is 46, 53, 63, and 92 ms / Token, respectively, which is approximately 11.5%, 23.2%, 38.2%, and 45.9% lower than that of ordinary NVMe SSD offloading.

[0095] The results demonstrate that by maintaining the current valid physical address in memory and avoiding reading old copies or performing additional scans during the query process, the present invention can effectively reduce the impact of SSD readback and background recycling on decoding tail latency.

[0096] like Figure 5 As shown, in Embodiment 1 of the present invention, different KV Cache offloading schemes are compared in terms of host side write amplification, garbage collection overhead, space utilization and index lookup latency. The experiment compares the scheme of ordinary NVMe SSD offloading, the scheme of using only ZNS SSD basic sequential write and the scheme of the present invention.

[0097] The experimental results above show that this invention does not simply move the KV Cache from memory to SSD, but rather performs collaborative optimization around maintaining the current valid physical address, sequential append writing, and zone reclamation during the decoding stage. This approach can reduce recomputation and readback costs in long context and high-concurrency inference scenarios, improve capacity expansion capabilities, and reduce space reclamation overhead on ZNS SSDs.

[0098] Experimental results show that the solution of the present invention has significant advantages in storage layer performance.

[0099] Regarding write amplification on the host side, the write amplification of ordinary NVMe SSD Offloading is about 2.4×, the basic sequential write scheme of ZNS SSD is about 1.5×, and the solution of this invention reduces it to 1.12×, which is about 53.3% lower than that of ordinary NVMe SSD Offloading and about 25.3% lower than that of the basic sequential write scheme of ZNS SSD.

[0100] In terms of space utilization, the effective space utilization of ordinary NVMe SSD Offloading is about 62%, the basic sequential write solution of ZNSSSD is about 76%, and the solution of this invention reaches about 88%, which is about 41.94% higher than the ordinary NVMe SSD Offloading write solution and about 15.79% higher than the basic sequential write solution of ZNS SSD.

[0101] Regarding garbage collection blocking, the P99 GC blocking time of ordinary NVMe SSD Offloading is about 145ms, and the basic sequential write scheme of ZNS SSD is about 72ms. The solution of this invention reduces it to 24ms, which is about 83.4% lower than that of ordinary NVMe SSD Offloading and about 66.7% lower than that of the basic sequential write scheme of ZNS SSD.

[0102] Regarding index lookup latency, the index lookup latency of ordinary NVMe SSD Offloading is about 28μs, the basic sequential write scheme of ZNSSSD is about 12μs, and the scheme of this invention is about 4μs, which is about 85.7% lower than that of ordinary NVMe SSD Offloading and about 66.7% lower than that of the basic sequential write scheme of ZNS SSD.

[0103] The above results demonstrate that the present invention maintains the current valid physical address through a globally extended hash index and performs validity verification based on this index during background garbage collection. This can simultaneously reduce write amplification, improve space utilization, and reduce the blocking impact of garbage collection on the decoding process.

[0104] In summary, Embodiment 1 of the present invention focuses on optimization during the decoding stage, avoiding unnecessary disk I / O introduced during the pre-filling stage; it maintains the relationship between the Key and the current valid physical location through a globally extended hash index structure, avoiding strong binding between logical partitions and physical zones; it improves the efficiency of writing, querying, and space reclamation in KV Cache unloading scenarios through ZNS SSD sequential append writing and zone-based garbage collection mechanisms; and it demonstrates through experiments under long context tasks and real-world dialogue request trajectories that the present invention has optimization effects in terms of decoding throughput, tail latency, space utilization, and garbage collection overhead.

[0105] Example 2

[0106] like Figure 6 As shown, the LLM decoding stage KV Cache offloading system of Embodiment 2 of the present invention includes:

[0107] The system includes a data writing module 100, an index maintenance module 200, a query readback module 300, and a garbage collection module 400.

[0108] The data writing module 100 is used in the decoding stage of large language model inference to write the KV Cache data items to be unloaded into the target Zone in ZNS SSD in the order of append writing, and trigger the index maintenance module to update the global extended hash index structure during or after the writing process.

[0109] The index maintenance module 200 is used to establish and maintain a global extended hash index structure in memory, recording the mapping relationship between the key of KVCache data items and the current valid physical location information;

[0110] The query readback module 300 is used to look up the global extended hash index structure based on the query key, obtain the current valid physical location information of the target KVCache data, and read the target KVCache data.

[0111] The garbage collection module 400 is used to perform garbage collection on the zones of the ZNS SSD that have reached the preset collection conditions, based on the statistical information of dirty data in each zone of the ZNS SSD.

[0112] In this embodiment, the index maintenance module 200 does not divide the hash index into multiple fixed partitions according to the physical zone, but maintains a global extended hash index structure in memory. The global extended hash index structure can expand the directory or bucket structure according to the data scale, and update the current valid physical location information corresponding to the key when writing, migrating, deleting or evicting.

[0113] In summary, Embodiment 2 of the present invention implements KV Cache unloading, index maintenance, query readback and space reclamation in the decoding stage through a modular approach. The modules coordinate with each other based on a globally extended hash index structure, thereby decoupling the upper-layer query logic from the physical zone allocation strategy of the underlying ZNS SSD.

[0114] Example 3

[0115] like Figure 7 As shown, in Embodiment 3 of the present invention, the electronic device includes a processor 51 and a memory 52 coupled to the processor 51.

[0116] The memory 52 is used to store program instructions for implementing the KV Cache unloading method in the LLM decoding stage, and the processor 51 is used to run the program instructions in the memory 52 to implement the KV Cache unloading method in the LLM decoding stage.

[0117] The processor 51 can also be called a CPU. The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0118] Example 4

[0119] like Figure 8 As shown, Embodiment 4 of the present invention provides a storage medium for storing program instructions for executing the KV Cache unloading method in the LLM decoding stage.

[0120] The storage medium of this embodiment stores a program file 61 capable of implementing all the above methods. The program file 61 can be stored in the storage medium in the form of a software product and includes several instructions to cause a computer device or processor to execute all or part of the steps of the methods of various embodiments of this invention.

[0121] The aforementioned storage media include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. These storage media can exist independently or be installed in electronic devices such as computers, servers, mobile phones, and tablets.

[0122] In summary, the LLM decoding stage KV Cache offloading method, system, storage medium, and electronic device disclosed in this invention have at least the following beneficial effects:

[0123] This invention focuses its optimization efforts on the decoding stage, avoiding unnecessary SSD I / O during the computationally intensive pre-filling phase; it maintains the mapping between the key and the current valid physical location through a globally extended memory hash index structure, making the query path independent of physical zone partitions; it reduces random writes and internal device management overhead by sequentially appending actual KV cache data to ZNS SSDs; and it improves the accuracy and space utilization of garbage collection by marking old physical copies that are deleted, evicted, migrated, or replaced as invalid data and verifying whether they are still referenced by the current index during garbage collection.

[0124] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0125] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

[0126] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0127] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those skilled in the art without any creative effort are all within the scope of protection of the present invention.

Claims

1. A method for unloading the KV cache during the LLM decoding stage, characterized in that, Includes the following steps: During the decoding phase of large language model inference, the KV Cache data items to be unloaded are determined; The KV Cache data items to be unloaded are written to the target Zone in the ZNS SSD in an append-only manner. During or after writing, the mapping relationship between the key of the KVCache data item and the current valid physical location information is maintained in the global extended hash index structure in memory; Based on the query key, the global extended hash index structure is searched to obtain the current valid physical location information of the target KV Cache data, and the target KV Cache data is read. Based on the dirty data statistics of each Zone in the ZNS SSD, garbage collection is performed on the Zones in the ZNS SSD that have reached the preset recycling conditions.

2. The LLM decoding stage KV cache offloading method according to claim 1, characterized in that, During the decoding phase of large language model inference, the KV Cache data items to be unloaded are determined as follows: The KV Cache data generated during the pre-filling stage is retained in the high-speed storage layer; Once the inference process enters the decoding phase, the KV Cache data items that need to be offloaded from the high-speed storage layer to the ZNS SSD are determined based on the video memory capacity, water level threshold, request concurrency, context length, or cache residency strategy.

3. The KV Cache offloading method in the LLM decoding stage according to claim 1, characterized in that, The key of the KV Cache data item includes at least: One or more of the following can be used to uniquely identify a KV Cache data item: request identifier, sequence identifier, layer identifier, attention header identifier, token block range, and KV type identifier.

4. The LLM decoding stage KV cache unloading method according to claim 1, characterized in that, The current valid physical location information of the target KVCache data includes at least: The Zone identifier, the offset within the Zone, the data length, and one or more combinations of version number, write generation, verification information, or status bits.

5. The LLM decoding stage KV cache offloading method according to claim 1, characterized in that, The KV Cache data items to be unloaded are written sequentially to the target Zone in the ZNS SSD using an append-only method, specifically as follows: Select the currently writable target Zone based on the write allocation strategy; The KV Cache data items to be unloaded are appended and written in the order of the write pointers of the target Zone; After the write operation is completed, the current valid physical location information corresponding to the KV Cache data item to be unloaded is written to or updated in the global extended hash index structure.

6. The LLM decoding stage KV cache offloading method according to claim 1, characterized in that, When the KV Cache data item corresponding to the same Key is rewritten, migrated, evicted, or deleted, only the mapping of the Key in the global extended hash index structure is updated to the current valid physical location or the mapping is deleted, and the old physical copy is recorded as invalid data in the index structure or Zone metadata. During the query, only the current valid physical location information pointed to by the global extended hash index structure is read.

7. The LLM decoding stage KV cache offloading method according to claim 1, characterized in that, Based on the dirty data statistics of each Zone in the ZNS SSD, garbage collection is performed on the Zones in the ZNS SSD that meet the preset collection conditions, specifically as follows: Calculate the amount or proportion of invalid data in each zone; When the amount or proportion of invalid data in the target zone reaches a preset threshold, scan the data items in the target zone; By comparing the current valid physical location information of the corresponding Key in the global extended hash index structure, it is determined whether the data item in the target Zone is still valid data; Migrate the data items that are still valid to the new writable Zone, update the global extended hash index structure, and perform a reset operation on the target Zone.

8. A KV Cache offloading system for the LLM decoding stage, characterized in that, The method for performing the LLM decoding stage KV Cache offloading method according to any one of claims 1 to 7 includes: The index maintenance module is used to establish and maintain the global extended hash index structure in memory, and record the mapping relationship between the key of the KVCache data item and the current valid physical location information; The data writing module is used to write the KV Cache data items to be unloaded into the target Zone in ZNS SSD in an append-only manner during the decoding stage of large language model inference, and to trigger the index maintenance module to update the global extended hash index structure during or after the writing process. The query and readback module is used to look up the global extended hash index structure based on the query key, obtain the current valid physical location information of the target KV Cache data, and read the target KV Cache data. The garbage collection module is used to perform garbage collection on the Zones of the ZNS SSD that have reached the preset collection conditions, based on the statistical information of dirty data in each Zone of the ZNS SSD.

9. A storage medium, characterized in that, The storage medium stores a computer program or program instructions, which are used to execute the LLM decoding stage KV Cache offloading method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory and processor, among which, The memory stores a computer program or program instructions, and when the processor executes the computer program or program instructions, it implements the LLM decoding stage KV Cache offloading method according to any one of claims 1 to 7.

Images