Method and apparatus for kv cache optimization

Abstract

The application relates to a KV cache optimization method and device. The optimization method comprises the following steps: collecting metadata associated with a cache block in a KV cache; establishing a hotness model of the cache block based on the metadata of the cache block; executing an optimization strategy for the cache block in response to a trigger condition based on the hotness model of the cache block; and updating a mapping structure associated with the cache block. The optimization method according to the application embodiment can reduce invalid occupation of the KV cache, thereby improving cache utilization, inference efficiency and system stability.

Description

Technical Field

[0001] This application generally relates to the field of artificial intelligence, and more specifically, to a KV cache optimization method and apparatus. Background Technology

[0002] In the inference process of Large Language Models (LLMs), to improve the efficiency of autoregressive generation, the common practice is to use key-value cache (KV cache) technology to store attention key-value pairs of historical tokens. The KV cache is allocated and managed in GPU memory in blocks or pages, with dynamic expansion and reuse achieved through mapping structures such as block tables and cache slot mappings. Existing cache reclamation technologies mainly include passive reclamation, eviction, or unloading triggered by the end of a request or GPU memory usage approaching a preset threshold. Regarding cache eviction strategies, existing technologies typically employ common strategies such as LRU (Least Recently Used) and FIFO (First In First Out), or directly perform coarse-grained cache release based on request granularity. Summary of the Invention

[0003] Embodiments of this application provide a heat-aware KV cache optimization method, apparatus, computer-readable storage medium, and computer program product that can reduce invalid KV cache usage, thereby improving cache utilization, inference efficiency, and system stability.

[0004] According to one aspect of this application, a key-value (KV) cache optimization method is provided, comprising: collecting metadata associated with cache blocks in the KV cache; establishing a popularity model of the cache blocks based on the metadata of the cache blocks; executing an optimization strategy for the cache blocks in response to a triggering condition based on the popularity model of the cache blocks; and updating a mapping structure associated with the cache blocks.

[0005] According to another aspect of this application, a KV cache optimization apparatus is provided, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the optimization method described above.

[0006] According to another aspect of this application, a non-transitory computer-readable storage medium is provided, storing instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described optimization method.

[0007] According to another aspect of this application, a computer program product is provided, including instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described optimization method. Attached Figure Description

[0008] Figure 1 A flowchart illustrating a KV cache optimization method according to an embodiment of this application is shown.

[0009] Figure 2 Another flowchart of the KV cache optimization method according to an embodiment of this application is shown.

[0010] Figure 3 A schematic diagram of a KV cache optimization apparatus according to an embodiment of this application is shown. Detailed Implementation

[0011] The features and exemplary embodiments of various aspects of this application will now be described in detail. Numerous specific details are provided in the following detailed description to provide a thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be implemented without some of these specific details. The following description of embodiments is merely intended to provide a better understanding of this application by illustrating examples. This application is by no means limited to any specific configurations and algorithms described below, but rather covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of this application. Well-known structures and techniques are not shown in the accompanying drawings and the following description in order to avoid unnecessarily obscuring this application.

[0012] As mentioned earlier, in the reasoning process of large language models, to improve the efficiency of autoregressive generation, the common practice is to use key-value caching technology to store attention key-value pairs of historical tokens, and to allocate and manage the KV cache in video memory in units of blocks or pages. Dynamic expansion and reuse of the KV cache are achieved through mapping structures such as block tables and cache slot mappings. Existing cache reclamation technologies mainly include passive reclamation, eviction, or unloading triggered by the end of a request or when video memory usage approaches a preset threshold. Regarding cache eviction strategies, existing technologies typically employ general strategies such as LRU and FIFO, or directly perform coarse-grained cache release based on request granularity.

[0013] The aforementioned existing technologies have the following significant shortcomings in practical application scenarios with long contexts, large differences in request lengths, continuous batch processing, and high-concurrency mixed loads: 1. Lack of fine-grained awareness of KV cache access patterns. For example, access to KV caches has obvious structural and phased characteristics (e.g., intensive access to one or more KV caches only in a few decoding steps), but existing caching strategies do not consider information such as the access frequency, temporal distribution, and lifecycle characteristics of KV caches, resulting in a large number of low-value or rarely accessed cache blocks occupying video memory for a long time, causing resource waste; 2. Passive triggering mechanisms lead to performance jitter. For example, passive migration or eviction based on video memory usage is often triggered when the system is under high load, and large-scale data migration or synchronization operations under high system load introduce additional overhead, causing a decrease in cache throughput and long-tail latency jitter; 3. Increased cache fragmentation and decreased memory access locality. For example, frequent cache block reallocation and release easily generate video memory fragmentation, and hot data is scattered in different physical areas, reducing cache access efficiency and affecting cache hit rate; 4. Lack of proactive optimization mechanisms. For example, these techniques only take action when cache resources are strained, lacking proactive optimization mechanisms and failing to preventively alleviate cache resource pressure. Therefore, they often suffer from low cache utilization and throughput.

[0014] Embodiments of this application provide a method, apparatus, computer-readable storage medium, and computer program product for optimizing key-value (KV) caches based on popularity awareness. The method according to embodiments of this application can sense the popularity of KV caches online and perform dynamic, preventative cache optimization based on popularity, reducing invalid KV cache usage and thereby improving cache utilization, inference efficiency, and system stability.

[0015] Specifically, this application proposes a closed-loop KV cache heat awareness and optimization framework, including one or more of the following features:

[0016] 1. Online cache monitoring and metadata collection: During the inference process, features such as access frequency, timing information, and lifecycle information associated with cache blocks or cache pages are collected in units, and a comprehensive metadata architecture is constructed.

[0017] 2. Popularity Modeling and Classification: Based on the collected feature data, the popularity value of each cache block is calculated, and an online updatable popularity model is constructed. The cache blocks are classified into different levels such as hot cache blocks, warm cache blocks, and cold cache blocks by combining lifecycle information, which realizes fine-grained cache state characterization.

[0018] 3. Heat-driven optimization strategy: Based on the heat model, one or more optimization strategies are executed at system safety points, including cache reordering (aggregating hot cache blocks into contiguous physical regions to improve memory access locality), early reclamation (releasing cache blocks whose lifecycle has ended or which are consistently low in heat in advance to reduce invalid occupancy), hot and cold tiering (migrating cold cache blocks to low-priority storage layers and keeping hot cache blocks in high-bandwidth video memory), or other similar strategies;

[0019] 4. Closed-loop triggering and mapping maintenance: Optimization strategies are dynamically triggered based on system status (e.g., memory usage, fragmentation rate, latency, etc.), and the mapping structure is updated synchronously when the optimization strategy is executed to ensure the consistency and correctness of cache access.

[0020] The KV cache optimization method according to the embodiments of this application realizes the transformation from passive response to proactive prevention. Through cache heat perception and preventive optimization, low-value cache can be released in advance before cache resources are scarce, and the cache layout is optimized, thereby improving the overall inference efficiency and stability.

[0021] The present application will now be described in further detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application to these specific forms.

[0022] Figure 1 A flowchart illustrating a KV cache optimization method 100 according to an embodiment of this application is shown. Figure 1 As shown, the optimization method 100 may include the following steps S101-S104.

[0023] In step S101, method 100 may include collecting metadata associated with cache blocks in the KV cache.

[0024] For example, during the inference process of a large language model, the access behavior of the KV cache can be monitored in real time using cache blocks or cache pages as management units, and metadata associated with each cache block or cache page in the KV cache can be collected. For example, the metadata associated with each cache block or cache page can be stored on a host computer or a device computer. In the following, this application mainly describes corresponding embodiments based on cache blocks, but it will be understood that the description of cache blocks is equally applicable to cache pages.

[0025] In some embodiments, the metadata associated with each cache block may include, but is not limited to, at least one of the following: the access frequency of the cache block, the timing information of the cache block, and the lifecycle information of the cache block.

[0026] In some embodiments, the access frequency of a cache block may include, but is not limited to, the read or write count of the cache block within one or more statistical windows, or other similar access frequency metrics. In some embodiments, the timing information of a cache block may include, but is not limited to, the access sequence number or timestamp of the most recent access to the cache block, or statistical characteristics of the access interval for the cache block, such as decay statistics, or other similar timing information metrics. In some embodiments, the lifecycle information of a cache block may include, but is not limited to, some or more of the following: an identifier of the request or sequence associated with the cache block, an identifier indicating whether the cache block is active, an identifier indicating the end or cancellation of the request or sequence associated with the cache block, a branch pruning flag associated with the cache block, or other similar lifecycle information metrics.

[0027] In some embodiments, the collection of metadata associated with cache blocks can be performed in batches for multiple cache blocks. For example, the collection of metadata can be performed in batches for the corresponding cache block set when the attention call boundary or the attention scheduler knows, for example, "the current decoding step will access the cache block set," thereby avoiding fine-grained recording every time a cache block is accessed, thus reducing runtime overhead.

[0028] In step S102, method 100 may include establishing a hotness model of cache blocks based on the metadata of cache blocks.

[0029] In some embodiments, establishing a cache block popularity model may include: calculating a cache block popularity value; and classifying the cache block into one of a first type of cache block, a second type of cache block, or a third type of cache block based on a threshold and the popularity value. In this application, the cache block popularity value can refer to any metric that reflects the frequency or value of a cache block being accessed.

[0030] In some embodiments, establishing a cache block heat model may include calculating a cache block heat value based on at least one of the cache block's access frequency, timing information, and lifecycle information. More specifically, in some embodiments, the heat value Heat(b) of cache block b may be calculated according to the following equation (1):

[0031] Heat(b) = (α×freq_score(b) + β×recency_score(b)) × lifecycle_score(b)

[0032] Equation (1)

[0033] The frequency term `freq_score(b)` indicates the number of accesses to cache block `b` within one or more statistical windows. The timing information term `recency_score(b)` is obtained based on a decay function of the most recent access time / step difference of cache block `b`. The lifecycle information term `lifecycle_score(b)` indicates whether the requests or sequences associated with cache block `b` are still active. For example, when the requests or sequences associated with cache block `b` have ended, or the branches associated with cache block `b` have been pruned, `lifecycle_score(b)` can be a value very close to 0 or can be 0, and in this case, cache block `b` can be directly considered a "reclaimable" cache block. Furthermore, α and β are weighting coefficients corresponding to the frequency term `freq_score(b)` and the timing information term `recency_score(b)`, respectively. These can be constants or dynamically adjusted according to system load, without limitation here.

[0034] In some embodiments, the first type of cache block may correspond to a cache block with a high heat value (also referred to as a hot cache block in this application), the second type of cache block may correspond to a cache block with an intermediate heat value (also referred to as a warm cache block in this application), and the third type of cache block may correspond to a cache block with a low heat value (also referred to as a cold cache block in this application).

[0035] In this application, the high popularity value, intermediate popularity value, and low popularity value are related to the threshold used in the optimization method. If a fixed threshold is used, the corresponding values ​​of the high popularity value, intermediate popularity value, and low popularity value can also be fixed. For example, the high popularity value can correspond to 70% or more of the maximum popularity value of all cache blocks, the intermediate popularity value can correspond to 30% to 70% of the maximum popularity value of all cache blocks, and the low popularity value can correspond to 30% or less of the maximum popularity value of all cache blocks. These numerical ranges for popularity values ​​are merely examples, and other ranges are possible. For example, the low popularity value can also correspond to 50% or less of the maximum popularity value of all cache blocks, and the high popularity value can also correspond to 80% or more of the maximum popularity value of all cache blocks. Any suitable numerical range for popularity values ​​is within the scope of this application. Furthermore, if a dynamic threshold is used, the corresponding values ​​of the high popularity value, intermediate popularity value, and low popularity value can also be dynamically adjusted, as described below.

[0036] In some embodiments, the threshold for dividing cache blocks may include an upper threshold T_hot and a lower threshold T_cold, and in some embodiments, one or both of the upper threshold T_hot and the lower threshold T_cold may be dynamically adjusted.

[0037] In some embodiments, for example, when the calculated heat value Heat(b) of cache block b is greater than the upper threshold T_hot, cache block b can be classified as a first-class cache block; when the heat value Heat(b) of cache block b is less than the upper threshold T_hot and greater than the lower threshold T_cold, cache block b can be classified as a second-class cache block; when the heat value Heat(b) of cache block b is less than the lower threshold T_cold, cache block b can be classified as a third-class cache block. That is, in this application, heat values ​​greater than the upper threshold T_hot can be considered high heat values, heat values ​​less than the upper threshold T_hot and greater than the lower threshold T_cold can be considered intermediate heat values, and heat values ​​less than the lower threshold T_cold can be considered low heat values.

[0038] In some embodiments, the initial values ​​of the thresholds (upper threshold T_hot and / or lower threshold T_cold) can be determined based on the statistical distribution of the heat values ​​Heat(b) of all cache blocks within the statistical window. For example, the value corresponding to 30% of the maximum heat values ​​of all cache blocks within the statistical window can be used as the initial lower threshold T_cold, and the value corresponding to 70% of the maximum heat values ​​of all cache blocks can be used as the initial upper threshold T_hot. The values ​​of 30% and 70% mentioned above are merely examples; other values ​​are also possible and are all within the scope of this application. For example, the value corresponding to 20% or 40% of the maximum heat values ​​of all cache blocks within the statistical window can be used as the initial lower threshold T_cold, and the value corresponding to 80% or 60% of the maximum heat values ​​of all cache blocks can be used as the initial upper threshold T_hot.

[0039] In some embodiments, the upper threshold T_hot and / or lower threshold T_cold used to partition cache blocks can be dynamically adjusted based on system status (including, but not limited to, actual memory usage U and target memory usage U_target). More specifically, in some embodiments, the upper threshold T_hot and lower threshold T_cold can be calculated according to the following equations (2)-(3):

[0040] T_hot = T_hot_ref + k_h×(U-U_target); Equation (2)

[0041] T_cold = T_cold_ref + k_c×(U-U_target) Equation (3)

[0042] Wherein, T_hot_ref and T_cold_ref are the initial values ​​of the upper threshold T_hot and the lower threshold T_cold, respectively. They can be determined based on the statistical distribution of the heat values ​​Heat(b) of all cache blocks within the statistical window, as mentioned above; k_h and k_c are adjustment coefficients associated with the upper threshold T_hot and the lower threshold T_cold, respectively, and can be configured to make the dynamic adjustment of the upper threshold T_hot and the lower threshold T_cold smoother; U_target can be a pre-set constant term indicating the target amount of video memory usage that the system is expected to maintain smoothly. For example, U_target can typically be 80% to 90% of the total video memory capacity.

[0043] For example, when memory usage U exceeds the target memory usage U_target and continues to rise, the upper threshold T_hot and / or the lower threshold T_cold can be increased. This allows more cache blocks to be classified as warm / cold cache blocks, which can then be reclaimed or released, enabling more aggressive cache space reclamation and release. Conversely, when memory usage U is below the target memory usage U_target and continues to decrease, the upper threshold T_hot and / or the lower threshold T_cold can be decreased. This results in fewer cache blocks being classified as warm / cold cache blocks, thereby reducing cache block migration and jitter.

[0044] In step S103, method 100 may include: executing an optimization strategy for cache blocks based on a cache block heat model in response to a triggering condition.

[0045] In some embodiments, executing an optimization strategy for a cache block in response to a triggering condition may include, but is not limited to, at least one of the following: rearranging the cache block; prematurely reclaiming the cache block; and stratifying the cache block.

[0046] In some embodiments, rearranging cache blocks may include placing first-type cache blocks in contiguous cache regions and updating the physical addresses of the first-type cache blocks. For example, one or more first-type cache blocks scattered across multiple non-contiguous regions, i.e., cache blocks with high-frequency values, can be identified, and these first-type cache blocks can be clustered into more contiguous or more efficient cache regions based on their physical addresses, such as the same larger cache block or cache pool, or contiguous cache page ranges. In this way, KV cache fragmentation can be reduced and address jump costs can be lowered.

[0047] Furthermore, according to embodiments of this application, rearranging cache blocks may also include updating the physical address of the rearranged cache block so that the logical address of the cache block can be mapped to the new physical address, so that the cache block can still be correctly located after being rearranged.

[0048] In some embodiments, the rearrangement of cache blocks can be performed at a scheduling safe point to ensure the correctness of cached data, and atomic switching and rollback in case of switching failure can also be implemented using version number mechanisms / dual table switching to ensure the consistency of cached data.

[0049] In some embodiments, cache blocks that meet the early reclamation conditions can be released early or placed in a candidate reclamation queue to reduce invalid key-value cache occupancy. Specifically, in some embodiments, early reclamation of cache blocks can be performed when at least one of the following is met: the lifetime of the cache block ends; the cache block is classified as a third-category cache block within multiple consecutive statistical windows; and the cache block is determined to be no longer referenced; etc.

[0050] In some embodiments, the termination of a cache block's lifecycle may include, but is not limited to, at least one of the following: the request / sequence associated with the cache block has ended / completed or been canceled, the beam branch associated with the cache block has been pruned, or the verification of the speculative branch associated with the cache block has failed. In some embodiments, classifying a cache block as a third-category cache block within multiple consecutive statistical windows may indicate that the cache block's heat value has consistently been below the lower threshold T_cold within multiple statistical windows, meaning the cache block has consistently remained low-heat, and combined with the cache block's lifecycle information or attention range, it can be determined that the probability of subsequent access to the cache block is extremely low. Furthermore, in some embodiments, before prematurely reclaiming a cache block, it is necessary to confirm that the cache block is no longer mapped to any references, which can be achieved through active set verification, reference counting, or mapping version consistency checks.

[0051] In some embodiments, tiering cache blocks may include, but is not limited to, migrating third-type cache blocks (i.e., cold cache blocks) to a low-priority cache layer, and migrating first-type cache blocks (i.e., hot cache blocks) to or retaining them in a high-priority cache layer. For example, a low-priority cache layer may include a low-priority video memory pool, compressed region, or main memory, while a high-priority cache layer may include a high-bandwidth cache layer.

[0052] In some other embodiments, tiering cache blocks may also include migrating second-type cache blocks (i.e., warm cache blocks) to a lower-priority cache layer or migrating them to a higher-priority cache layer, or alternatively, temporarily not migrating second-type cache blocks, depending on the actual memory usage U and the target memory usage U_target. For example, when the actual memory usage U is close to or higher than the target memory usage U_target, second-type cache blocks may be migrated to a lower-priority cache layer, while when the actual memory usage U is significantly lower than the target memory usage U_target, second-type cache blocks may not be migrated or may be migrated to a higher-priority cache layer.

[0053] In some embodiments, tiering cache blocks may further include migrating third-class cache blocks to a higher-priority cache layer when a third-class cache block is reclassified as a first-class or second-class cache block, or when a third-class cache block request enters an active state. In other words, when the popularity value of a cold cache block increases or a cold cache block request re-enters the active decoding phase, a back-migration / prefetch operation for the cold cache block can be performed.

[0054] In some embodiments, the above migration of cache blocks can be performed in batches for multiple cache blocks. Furthermore, the migration can be performed in parallel with the calculation of cache block popularity values, where migration budgets and migration trigger thresholds can be set to control migration bandwidth and avoid performance fluctuations caused by excessive migration.

[0055] In some embodiments, the triggering conditions may include, but are not limited to, at least one of the following: the usage or growth rate of video memory exceeds a threshold; the fragmentation rate of video memory exceeds a threshold; the dispersion of cache blocks with high heat values ​​exceeds a threshold; queue congestion; and long-tail latency exceeds a threshold; etc.

[0056] In some embodiments, different triggering conditions may have different priorities, and executing an optimization strategy for a cache block in response to a triggering condition may include: executing the optimization strategy for the cache block based on the priority of the triggering condition. According to embodiments of this application, for example, when multiple triggering conditions occur simultaneously, the optimization strategy for the cache block may be executed based on a preset priority of the multiple triggering conditions.

[0057] For example, when both the usage or growth rate of video memory exceeds a threshold and the fragmentation rate of video memory exceeds a threshold, an optimization strategy can be implemented to prioritize addressing the issue of video memory usage or growth rate exceeding the threshold, and then address the issue of fragmentation rate exceeding the threshold. In this case, for example, early reclamation of cache blocks can be prioritized to reduce video memory usage or growth rate, and then cache block rearrangement can be performed to reduce video memory fragmentation rate.

[0058] For example, when both the memory fragmentation rate and the dispersion of cache blocks with high activity levels exceed a threshold, an optimization strategy can be implemented to prioritize addressing the memory fragmentation issue and then address the dispersion of cache blocks with high activity levels. In this case, for example, cache block rearrangement can be prioritized to reduce the memory fragmentation rate, followed by cache block stratification to reduce the dispersion of cache blocks with high activity levels.

[0059] For example, when the usage or growth rate of video memory exceeds a threshold, the fragmentation rate of video memory exceeds a threshold, and the dispersion of cache blocks with high activity values ​​exceeds a threshold simultaneously, corresponding optimization strategies can be implemented to prioritize addressing the issue of video memory usage or growth rate exceeding the threshold, then addressing the issue of video memory fragmentation exceeding the threshold, and finally addressing the issue of the dispersion of cache blocks with high activity values. In this case, for example, early reclamation of cache blocks can be prioritized to reduce video memory usage or growth rate, followed by rearranging cache blocks to reduce fragmentation rate, and then stratifying cache blocks to reduce the dispersion of cache blocks with high activity values.

[0060] Furthermore, in some embodiments, multiple optimization strategies for the same triggering condition may also have corresponding priorities, and executing an optimization strategy for a cache block in response to a triggering condition may include: executing the optimization strategy for the cache block according to the priority of the optimization strategy for the triggering condition.

[0061] For example, in response to the usage of video memory or the rate of increase in usage exceeding a threshold, early reclamation of cache blocks can be prioritized. Furthermore, if early reclamation of cache blocks cannot meet the requirements for video memory usage or the rate of increase in usage, cache block tiering or rearrangement can be performed.

[0062] For example, in response to a memory fragmentation rate exceeding a threshold, cache blocks can be prioritized for rearrangement. Furthermore, if rearranging cache blocks cannot meet the memory fragmentation requirements, cache block tiering or early cache block reclamation can be performed.

[0063] For example, in response to the dispersion of cache blocks with high popularity values ​​exceeding a threshold, cache block rearrangement can be prioritized to cluster cache blocks with high popularity values. Furthermore, if cache block rearrangement cannot meet the dispersion requirements, cache block stratification or early cache block reclamation can be performed.

[0064] For example, in response to queue congestion or long-tail latency exceeding a threshold, the early reclamation of cache blocks can be prioritized, followed by the rearrangement and / or tiering of cache blocks.

[0065] The above descriptions of the priorities of triggering conditions and optimization strategies are merely exemplary, and the embodiments of this application are not limited to the specific priorities described above. For example, in response to the dispersion of cache blocks with high popularity values ​​exceeding a threshold, the cache blocks can be prioritized for stratification, and then the cache blocks can be rearranged, as long as the dispersion requirement of cache blocks with high popularity values ​​can be met.

[0066] Furthermore, in some embodiments, the various optimization strategies for cache blocks may have default priorities. For example, early cache block reclamation may have the highest priority, cache block stratification may have a lower priority than early cache block reclamation, and cache block rearrangement may have a lower priority than cache block stratification. The priority order of the various optimization strategies described herein is merely an example; other orders are also possible and within the scope of this application, as long as they can meet the balance requirements between cache throughput, latency, and stability.

[0067] Therefore, the KV cache optimization method according to the embodiments of this application can execute different optimization strategies or strategy combinations for the same triggering condition, or execute different optimization strategies or strategy combinations for a specific combination of triggering conditions.

[0068] In step S104, the mapping structure associated with the cache block can be updated. This update can be performed, for example, while executing the optimization strategy for the cache block, or after executing the optimization strategy for the cache block. For example, when rearranging or stratifying cache blocks, the physical address of the cache block can be updated, or when reclaiming a cache block, the block table and cache slot mapping associated with the cache block can be updated, thereby ensuring the consistency and correctness of cache accesses after executing the optimization strategy.

[0069] The KV cache optimization method according to embodiments of this application, through heat awareness and preventative optimization, can reduce the long-term occupation of low-value KV cache, lower the cache fragmentation rate, improve the locality of access to hot cached data, and reduce the burst migration and synchronization overhead associated with passive cache usage triggering mechanisms, thereby improving inference throughput and long-tail latency stability. The KV cache optimization method according to embodiments of this application is particularly suitable for typical workloads such as continuous batch processing and long contexts.

[0070] Figure 2 Another flowchart of a KV cache optimization method 200 according to an embodiment of this application is shown. Method 200 can correspond to the above combination. Figure 1 Method 100 is described. For example... Figure 2 As shown, the method 200 may include steps S201-S208.

[0071] In step S201, system initialization is performed, including but not limited to loading configuration parameters, allocating metadata storage space, and starting cache block monitoring threads.

[0072] In step S202, the attention scheduler determines the KV buffer block or set of KV buffer blocks that the current decoding step needs to access.

[0073] In step S203, the metadata of the KV buffer blocks involved in the current decoding step is collected.

[0074] In step S204, the heat value of the KV buffer block involved in the current decoding step is calculated and a heat model of the KV buffer block is established.

[0075] In step S205, the triggering conditions are checked to determine whether one or more optimization strategies should be executed for the KV cache block.

[0076] If it is determined that no optimization strategy needs to be executed, proceed to step S208 and the current decoding step ends; if it is determined that an optimization strategy needs to be executed, proceed to step S206 and execute one or more selected optimization strategies at the scheduling safe point.

[0077] In step S207, the mapping relationship of the KV cache blocks is updated to ensure the correctness of subsequent KV cache block accesses.

[0078] In step S208, at the end of the current decoding step, the metadata of the KV cache blocks involved in the current decoding step is cleared.

[0079] After executing step S208, method 200 can return to step S202 to continue executing the next decoding step. For specific details regarding method 200, please refer to the above description of method 100, which will not be repeated here. Furthermore, method 200 can achieve the same beneficial effects as method 100.

[0080] Figure 3 A schematic diagram of a KV cache optimization apparatus according to an embodiment of this application is shown. The apparatus is shown as a computing device 300, which can be used to perform the above-described KV cache optimization method. Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more hardware processors 304 coupled to bus 302 for processing information. The one or more hardware processors 304 may include, for example, one or more general-purpose microprocessors.

[0081] like Figure 3As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302. Main memory 306 is used to store information and instructions executed by one or more processors 304, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in storage media accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0082] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0083] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0084] Each of the processes, methods, and algorithms described above may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented partially or fully in dedicated circuit systems.

[0085] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0086] Specific embodiments further provide an apparatus including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the apparatus to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0087] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0088] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0089] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0090] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0091] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0092] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0093] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0094] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0095] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0096] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0097] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the term "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A key-value caching optimization method, characterized in that, include: Collect metadata associated with cache blocks in the KV cache; A heat model of the cache block is established based on the metadata of the cache block; Based on the hotness model of the cache block, an optimization strategy for the cache block is executed in response to the triggering condition; as well as Update the mapping structure associated with the cache block; The establishment of the heat model of the cache block includes: calculating the heat value of the cache block; and classifying the cache block into one of the first type of cache block, the second type of cache block, or the third type of cache block based on the threshold and the heat value; The first type of cache block corresponds to a cache block with a high popularity value, the second type of cache block corresponds to a cache block with a medium popularity value, and the third type of cache block corresponds to a cache block with a low popularity value. The execution of an optimization strategy for the cache block in response to a triggering condition includes at least one of the following: rearranging the cache block; prematurely reclaiming the cache block; and stratifying the cache block; The rearrangement of the cache blocks includes: placing the first type of cache blocks in a contiguous cache region and updating the physical address of the first type of cache blocks; The layering of the cache blocks includes: migrating the third type of cache blocks to a low-priority cache layer, and migrating the first type of cache blocks to a high-priority cache layer; Different triggering conditions have different priorities. When multiple triggering conditions occur simultaneously, the optimization strategy for the cache block is executed in order of the preset priority of the multiple triggering conditions.

2. The method according to claim 1, characterized in that, The threshold includes an upper threshold and a lower threshold, and the upper threshold and the lower threshold can be dynamically adjusted.

3. The method according to claim 2, characterized in that, When the popularity value of the cache block is greater than the upper threshold, the cache block is classified as the first type of cache block; When the heat value of the cache block is less than the upper threshold and greater than the lower threshold, the cache block is classified as the second type of cache block; When the popularity value of the cache block is less than the lower threshold, the cache block is classified as the third type of cache block.

4. The method according to claim 1, characterized in that, The rearrangement of the cache blocks is performed at the scheduling safe point.

5. The method according to claim 1, characterized in that, Early reclamation of the cache block is performed when at least one of the following conditions is met: The lifetime of the cache block has ended; The cache block is classified as the third type of cache block within multiple consecutive statistical windows; and The cache block is determined to be no longer referenced.

6. The method according to claim 1, characterized in that, Layering the cache blocks further includes: When the third type of cache block is reclassified as the first type of cache block or the second type of cache block, or when the third type of cache block requests to enter an active state, the third type of cache block is migrated to a high-priority cache layer.

7. The method according to claim 1, characterized in that, The migration is performed in batches for multiple cache blocks.

8. The method according to any one of claims 1-7, characterized in that, The metadata associated with the cache block includes at least one of the following: the access frequency of the cache block, the timing information of the cache block, and the lifecycle information of the cache block.

9. The method according to claim 8, characterized in that, The access frequency of the cache block includes: the read or write count of the cache block within the statistics window; The timing information of the cache block includes: the access sequence number or timestamp of the most recent access to the cache block, and attenuation statistics of the access interval for the cache block; and The lifecycle information of the cache block includes: an identifier of the request or sequence associated with the cache block, an identifier indicating whether the cache block is active, an identifier indicating the end or cancellation of the request or sequence associated with the cache block, or a branch pruning flag associated with the cache block.

10. The method according to claim 8, characterized in that, Establishing the heat model for the cache block includes: The hotness value of the cache block is calculated based on at least one of the access frequency, timing information, and lifecycle information of the cache block.

11. The method according to any one of claims 1-7, characterized in that, The collection of metadata associated with the cache blocks is performed in batches for multiple cache blocks.

12. A KV cache optimization device, characterized in that, include: One or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-11.

13. A non-transitory computer-readable storage medium, characterized in that, The system stores instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-11.

14. A computer program product, characterized in that, Includes instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-11.

Images