An end-side-based large model key-value cache dynamic optimization method and system

Abstract

The application discloses a kind of based on end side's big model key value cache dynamic optimization method and system, method includes: S1, the context resource of multiple inference sequences is centrally controlled, and corresponding context space is pre-allocated;S2, whether the currently available context space satisfies the allocation demand of inference sequence is judged, if it satisfies, then the intelligent context allocation is carried out to inference sequence, if it does not satisfy, then whether the remaining memory space of end side equipment is enough is judged, according to the judgment result, step S3 or S4 is executed;S3, if space is enough, then the available context space is increased by dynamic expansion, and the context space allocation and context update of inference sequence are re-performed;S4, if space is insufficient, first carry out context recovery, then discard and retain context by sequence perception sliding window mechanism, and execute context update after allocation is completed.The application can realize the efficient inference of big model under the condition of end side limited resource.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence large model optimization technology, specifically to a dynamic optimization method and system for large model key-value caching based on the edge side. Background Technology

[0002] With the rapid development of AI PCs (AI PCs), the deployment and application of large language models on edge devices are becoming increasingly widespread, and users are placing higher demands on the real-time performance and response speed of localized AI services. On edge devices such as AI PCs, to achieve fast response and low-latency inference, AI inference frameworks typically employ a pre-allocation mechanism to manage key-value cache resources. This involves setting a default value and allocating a fixed-size cache space before inference begins. This static memory management does not support dynamic expansion at runtime; the cache size is fixed during system initialization and cannot be adjusted according to actual processing needs. However, this pre-allocation mechanism faces significant challenges. Due to the relatively limited memory resources of edge devices, sequence lengths are often difficult to accurately predict in real-world applications. A completely fixed pre-allocation strategy either wastes memory or leads to key-value cache loss due to insufficient caching, resulting in decreased accuracy of inference results.

[0003] In multi-task concurrent batch processing, existing technologies employ a static, fixed-size allocation strategy, evenly distributing pre-allocated key-value cache space across multiple tasks. This one-size-fits-all approach fails to adapt to the varying needs of different task sequences, resulting in low cache resource utilization: short sequences consume excessive cache space, leading to waste, while long sequences suffer from insufficient cache, impacting processing performance. Furthermore, the lack of dynamic awareness of sequence activity and resource scheduling mechanisms prevents timely adjustments to resource allocation strategies when the number of sequences changes dynamically, further reducing the overall system efficiency and resource utilization value.

[0004] In summary, on-device local inference demands extremely high real-time performance and response speed. Therefore, fully dynamic allocation of key-value caches is more suitable for high-concurrency servers, while fixed pre-allocation on the device side is too rigid and cannot adapt to different contextual requirements. Thus, there is an urgent need to implement a dynamic optimization method for key-value caches on the device side for large models. This method should maximize the utilization efficiency of key-value caches while ensuring fast response times, thereby improving the service quality and user experience of large language models in AI PC-based scenarios. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a dynamic optimization method and system for large model key-value caching based on the client side, in order to achieve efficient inference of large models under client-side resource constraints and improve user experience.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A dynamic optimization method for large model key-value caching based on the client side includes the following steps: S1, start the large model inference task on the end device, centrally manage the context resources of multiple inference sequences through unified context management, and pre-allocate the corresponding context space for each inference sequence based on the preset initial context size; S2, determine whether the currently available context space meets the allocation requirements of the inference sequence: if it does, then perform intelligent context allocation for the inference sequence based on the currently allocated context space; if it does not, then continue to determine whether the remaining memory space of the edge device is sufficient, and execute step S3 or S4 according to the determination result: S3. If the remaining memory space of the terminal device is sufficient, the available context space is increased by dynamic expansion, and the context space allocation and context update of the inference sequence are performed after the dynamic expansion is completed. S4. If the remaining memory space of the edge device is insufficient, intelligent context reclamation and allocation are performed first. If this fails, the context of the inference sequence in the allocated context space is intelligently discarded and retained through the sequence-aware sliding window mechanism, and then allocated to the currently active inference sequence. After the allocation is completed, the context is updated.

[0007] Furthermore, the specific steps for pre-allocating the corresponding context space for each inference sequence based on the preset initial context size are as follows: Estimate the space required to load a large model and the amount of memory / GPU memory needed during the inference process of the large model based on the device model and memory / GPU size, and determine the initial context size based on the remaining memory space of the device. Two queues are maintained to manage the inference sequence state. For an active inference sequence, a portion of the context is pre-allocated from the allocated initial context.

[0008] Furthermore, in step S2, the specific steps for intelligently allocating the inference sequence based on the currently allocated context space are as follows: A differentiated and on-demand allocation strategy is implemented for inference sequences. Inference sequences are perceived and monitored in real time, and resources are allocated and reclaimed from the allocated context space according to actual needs, so as to improve the resource utilization of multiple sequences.

[0009] Furthermore, in step S3, the specific steps for dynamically expanding the available context space are as follows: Determine whether to enable the expansion flag. If enabled, determine the expansion size based on the total memory requirement under the current context cache size and the available memory space of the edge device, provided that the preset safety threshold is met. After determining the expansion size, based on the computation graph used in the current large model inference process, the scheduler and computation context of the current computation graph are reused by copying, and a data copying execution graph for key-value cache expansion is constructed. Adjust the size of the metadata array for managing the key-value cache based on the expansion size, and initialize the metadata parameters of the expansion portion; Create and initialize a new context data structure to store the new context key-value cache, and allocate key-value data and buffers layer by layer according to the model; Based on the data replication execution graph, key-value cache replication nodes are added to the computation graph according to the model hierarchy. After all nodes are added, the computation graph is executed uniformly to realize the replication of the original key-value cache data to the expanded context data structure. After the key-value cache is copied, update the key-value cache's data structure references, computation graph parameters, and context state to complete the dynamic resizing operation.

[0010] Furthermore, the steps for determining the expansion size are as follows: Prioritize expanding the capacity by using a preset maximum expansion factor that does not exceed the current context size. If the space is still insufficient after expansion, gradually reduce the expansion factor to a preset minimum limit. In each expansion, ensure that the expansion size meets the alignment requirements of the preset initial context size until a suitable expansion size is found.

[0011] Furthermore, the specific steps for adding key-value cache replication nodes to the computation graph according to the model hierarchy are as follows: For key data, create one-dimensional views for source key cache data and target key cache data respectively, and create corresponding replication operator nodes in the computation graph based on the one-dimensional views; For value data, the replication method is determined based on the storage layout of the value data: when the value data is in a non-transposed layout, a one-dimensional view is created for the source value cache data and the target value cache data respectively, and a corresponding replication operator node is created in the computation graph; when the value data is in a transposed layout, a two-dimensional view is created for the source value cache data and the target value cache data respectively, and a corresponding replication operator node is created in the computation graph based on the two-dimensional view. The replication operator nodes are executed concurrently under the control of the computation graph scheduler to complete the batch replication of key-value cache data.

[0012] Furthermore, in step S4, intelligent context recycling and allocation are first performed. If this fails, the context of the inference sequence in the allocated context space is intelligently discarded or retained using a sequence-aware sliding window mechanism. The specific steps are as follows: Maintain the running status and active time of the inference sequence. First, reclaim part or all of the context space from the long-term inactive inference sequences, and then allocate it to the inference sequences that need it. If expansion and reclamation are not possible, the context of the inference sequence in the allocated context space is intelligently discarded and retained using a sequence-aware sliding window mechanism, as follows: Based on the context size allocated to each inference sequence, maintain an independent window size for each inference sequence; A predetermined number of context prefix contents in the inference sequence are retained and not pruned, and the context area excluding the prefix contents is used as a sliding window area; The number of contexts to be discarded is determined according to the preset discard ratio and the minimum discard limit. Based on the number of contexts to be discarded, the foremost part of the context content in the sliding window area is deleted, and the remaining context content is moved to the starting position of the sliding window area. After the pruning process is completed, the corresponding context index information and position encoding information are updated to release key-value cache space for storing new key-value cache data.

[0013] Furthermore, the method also includes a dynamic scaling-down process, which is triggered when the edge device needs to free up memory space for a new model or inference task, and includes the following steps: Determine the reduced target context size based on the target requirements for context size; Based on the computation graph used in the current large model inference process, the corresponding scheduler and computation context are reused by copying the computation graph to construct a data copying execution graph for key-value cache reduction. Adjust the size of the metadata array used to manage the key-value cache according to the target context size, and initialize the metadata parameters corresponding to the reduced context; Create a new context data structure to store the reduced key-value cache, and allocate corresponding key-value data and buffers to each layer according to the model hierarchy; Based on the data replication execution graph, key-value cache replication nodes are added to the computation graph according to the model hierarchy. After all nodes are added, the computation graph is executed uniformly to realize the replication of the original key-value cache data to the reduced context data structure. After the key-value cache copy is complete, update the key-value cache's data structure references, computation graph parameters, and context state, and after completing the dynamic shrinkage, set the expansion permission flag to the off state.

[0014] A dynamic optimization system for key-value caching of large models based on the aforementioned method, the system being deployed within a large language model inference framework, the system comprising: The dynamic key-value cache management module is used to uniformly manage the key-value cache generated during the inference process of large models, and maintain the usage of the key-value cache according to the current context state; and to adjust the size of the key-value cache in real time according to computing resources and context requirements, and realize dynamic expansion and dynamic reduction based on pre-allocated cache; The multi-sequence intelligent context allocation module is used to reasonably allocate the key-value cache space according to the real-time context requirements of each inference sequence when there are multiple inference sequences. The sequence-aware sliding window module is used to prune the context of the inference sequence when the remaining space in the key-value cache is insufficient and expansion cannot be performed. This is done by discarding part of the historical context to free up key-value cache space.

[0015] Furthermore, the key-value cache dynamic optimization system operates with the collaborative support of the model loading and computation module, the scheduler, and the memory management system. It is used to achieve dynamic adjustment of the key-value cache and continuous execution of inference tasks in a large model inference framework under the condition of limited resources on the edge side.

[0016] Compared with the prior art, the advantages of the present invention are as follows: This invention centrally manages the key-value caches of multiple inference sequences through unified context management and pre-allocates them based on a preset initial context size, improving the initialization efficiency and management controllability of large-scale model inference on the client side. By judging the available context space and memory status in real time, it avoids memory waste and fragmentation caused by unreasonable allocation, enhancing the stability of client-side resource scheduling. Dynamic expansion when there is sufficient remaining memory increases the available context space, enabling the system to smoothly support the concurrent execution of more inference sequences, reducing inference blocking caused by insufficient context, and improving overall throughput. When memory is limited, it first reclaims the context, and then uses a sequence-aware sliding window mechanism to intelligently discard and retain context, prioritizing the continuous operation of active inference sequences and avoiding invalid context from occupying memory resources for a long time. This invention achieves on-demand allocation and dynamic optimization of the key-value cache of large-scale models on the client side by combining on-demand expansion and intelligent reclamation, which can significantly improve context utilization, multi-sequence concurrent inference capability, and system stability under limited memory conditions. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0018] Figure 2 This is a flowchart of the multi-sequence intelligent context allocation process in a specific application embodiment.

[0019] Figure 3 This is a flowchart of the dynamic expansion process in a specific application embodiment.

[0020] Figure 4 This is a flowchart of a sequence-aware sliding window in a specific application example.

[0021] Figure 5 This is a schematic diagram of a large model inference framework that includes the system of Embodiment 2 of the present invention.

[0022] Figure 6 This is a schematic diagram of the result generated using the existing fixed pre-allocation method in a specific application embodiment.

[0023] Figure 7 This is a schematic diagram of the result generated using the method of this invention in a specific application embodiment. Detailed Implementation

[0024] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0025] The following are definitions of some abbreviations and key terms in this invention: Key-value caching (kv_cache, also referring to context) is a key technique for accelerating text generation in large models by caching intermediate computation results. When generating new words, it avoids repeatedly calculating the key-value pairs of all previous words, thus significantly improving inference efficiency. However, this comes at the cost of consuming additional memory / GPU memory to store these caches, making it a typical "space-for-time" strategy.

[0026] Edge-based inference: The core of edge-based inference lies in deploying the entire inference process of the AI ​​model locally on the user's terminal device. Edge-based inference based on AI PCs specifically refers to utilizing the PC's built-in dedicated AI acceleration engines (such as NPU, CPU, and GPU) to independently complete all inference computations from data input to result generation locally. This paradigm completely eliminates dependence on cloud servers. Its core value is the realization of a highly efficient, low-latency, and data privacy-preserving local inference closed loop, which is a key cornerstone for promoting the personalization and universality of AI applications.

[0027] In large language model inference, the self-attention mechanism needs to store a large number of intermediate key-value pairs, which are cached in a dedicated key-value cache. As the number of model parameters and the length of the context increase, the key-value cache size can typically account for 30% of the entire model inference, making key-value cache management a key bottleneck affecting inference performance. In AI PC-based local inference scenarios, traditional key-value cache management techniques suffer from the following problems: Static caching management strategy: AI PC-side scenarios have extremely high requirements for fast response and low latency. Dynamic allocation of key-value caches, which are completely non-contiguous, is more suitable for high-concurrency servers. Therefore, edge-side local inference typically uses a pre-allocation mechanism to manage key-value cache resources. This static memory management offers the best performance on the edge, but the cache size is fixed during system initialization and cannot be adjusted according to actual processing needs. This pre-allocation mechanism faces significant challenges. Due to the relatively limited memory resources of edge devices, sequence lengths are often difficult to accurately predict in real-world applications. A completely fixed pre-allocation strategy either wastes memory / GPU memory, affecting the operation of other models or tasks on the device, or leads to key-value cache loss due to insufficient cache space, resulting in semantic loss, decreased inference performance, and reduced accuracy of inference results. This can lead to garbled output and incorrect results when the context size is exceeded. For example, the popular open-source local inference framework llama.cpp pre-allocates a fixed 4096 tokens of context (key-value cache) space. Exceeding this limit results in significant key-value cache loss, making it unable to fully support long texts and long dialogues.

[0028] Uneven resource allocation across multiple tasks: In systems supporting parallel processing of multiple task sequences, cache resources are typically distributed evenly, failing to adapt to the varying needs of different task sequences. The lack of dynamic awareness of sequence activity and resource scheduling mechanisms means that resource allocation strategies cannot be adjusted promptly when the number of sequences changes dynamically, further reducing the overall system efficiency and resource utilization value. For example, setting the cache size to 4 in llama.cpp will evenly distribute the entire cache size across the four channels. Even if only one task is actually running, the other channels will remain idle, drastically reducing resource utilization.

[0029] To address the aforementioned technical issues, this invention provides a dynamic optimization method for key-value cache of large models on the client side. It solves the problem of uneven resource allocation among multiple tasks by using multi-sequence intelligent context allocation technology; it dynamically manages the context by adjusting the key-value cache size in real time according to computing resources and context requirements, based on pre-allocated cache; and it manages key-value cache blocks through a sequence-aware sliding window mechanism, thereby significantly improving the utilization rate and inference efficiency of key-value cache for local inference of large models on the AI ​​PC client side.

[0030] Example 1 like Figure 1 As shown, the method in this embodiment includes the following steps: S1, start the large model inference task on the end device, centrally manage the context resources of multiple inference sequences through unified context management, and pre-allocate the corresponding context space for each inference sequence based on the preset initial context size; S2, determine whether the currently available context space meets the allocation requirements of the inference sequence: if it does, then perform intelligent context allocation for the inference sequence based on the currently allocated context space; if it does not, then continue to determine whether the remaining memory space of the edge device is sufficient, and execute step S3 or S4 according to the determination result: S3. If the remaining memory space of the terminal device is sufficient, the available context space is increased by dynamic expansion, and the context space allocation and context update of the inference sequence are performed after the dynamic expansion is completed. S4. If the remaining memory space of the edge device is insufficient, intelligent context reclamation and allocation are performed first. If this fails, the context of the inference sequence in the allocated context space is intelligently discarded and retained through the sequence-aware sliding window mechanism, and then allocated to the currently active inference sequence. After the allocation is completed, the context is updated.

[0031] In specific application embodiments, such as Figure 1 and Figure 2 As shown, the process begins with multi-sequence intelligent context allocation, followed by model inference to detect sufficiency of the context space. If insufficient, memory prediction and threshold control determine if there is enough remaining space. If not, dynamic expansion is performed, followed by on-demand allocation. If the remaining space is insufficient, the process continues with multi-sequence intelligent context allocation. If allocation fails, a sequence-aware sliding window is used to maintain normal operation within the existing cache size. If upper-layer scheduling requires memory reduction, key-value cache reduction is implemented. Finally, computation graph and context updates are performed. It can be understood that the dynamic awareness and resource scheduling mechanism for sequence activity, timely adjustment of resource allocation strategies when the number of sequences changes dynamically, and the on-demand allocation mechanism can avoid wasting resources during idle periods.

[0032] In this embodiment, the specific steps for pre-allocating the corresponding context space for each inference sequence based on a preset initial context size are as follows: Estimate the space required to load a large model and the amount of memory / GPU memory needed during the inference process of the large model based on the device model and memory / GPU size, and determine the initial context size based on the remaining memory space of the device. Two queues are maintained to manage the inference sequence state. For an active inference sequence, a portion of the context is pre-allocated from the allocated initial context.

[0033] In specific application embodiments, an initial context (key-value cache) size is pre-allocated, typically for 4096 tokens (based on 3B and 7B models). However, if the device has ample video memory, different initial context sizes can be set for different devices. Based on the device model and video memory size, the space required to load the model is calculated. Then, the space needed to store one token in the key-value cache is calculated, resulting in the space required to store n tokens. This roughly assesses the video memory size required for the entire model inference process. The initial value is then set based on the remaining device space, ensuring the entire model occupies no more than 1 / 3 of the total device space (adjusted according to needs). It can be understood that setting different initial context sizes on different devices based on the device model and memory / video memory size, and then evaluating predictions using memory / video memory (reserving space for multi-task, multi-model scenarios), is more flexible than the existing fixed default value. The context module maintains two queues (pending and active) to manage sequence states and obtain the number of active sequences. In the local client-side scenario, the maximum concurrent sequences are set to 4. The initial pre-allocated context is 4096 tokens in size. An allocation size, alloc-size, such as 512 or 1024, is set internally. The first inference sequence is pre-allocated alloc-size initial context size from the allocated initial context.

[0034] In step S2, the specific steps for intelligently allocating the inference sequence based on the currently allocated context space are as follows: A differentiated and on-demand allocation strategy is implemented for inference sequences. Inference sequences are perceived and monitored in real time, and resources are allocated and reclaimed from the allocated context space according to actual needs, so as to improve the resource utilization of multiple sequences.

[0035] In specific application embodiments, through intelligent allocation strategies, differentiated cache allocation strategies and on-demand allocation strategies are adopted based on the number of active sequences, combined with recycling and truncation mechanisms, etc., abandoning the strategy of fixed pre-allocation of multiple sequences, the inference sequence is perceived and monitored in real time, and resources are allocated and recycled from the allocated context space according to actual needs, avoiding resource waste caused by idle sequences and improving resource utilization. If the current sequence's context space is exhausted, then allocate `alloc-size` from the already allocated context, and so on. The same applies if a new sequence becomes active. If the allocated context space is insufficient, step S3 is executed. If dynamic expansion is successful, allocation continues from the already allocated space. If dynamic expansion fails, it indicates insufficient device memory. If a new sequence has not been allocated any context space, `alloc-size` must be reclaimed from other sequences to ensure its operation. The reclaimed sequence undergoes context deletion and movement, truncation in the saved generated token array, and then the context information is updated.

[0036] In this embodiment, the specific steps in step S3 to increase the available context space through dynamic expansion are as follows: Determine whether to enable the expansion flag. If enabled, determine the expansion size based on the total memory requirement under the current context cache size and the available memory space of the edge device, provided that the preset safety threshold is met. After determining the expansion size, based on the computation graph used in the current large model inference process, the scheduler and computation context of the current computation graph are reused by copying, and a data copying execution graph for key-value cache expansion is constructed. Adjust the size of the metadata array for managing the key-value cache based on the expansion size, and initialize the metadata parameters of the expansion portion; Create and initialize a new context data structure to store the new context key-value cache, and allocate key-value data and buffers layer by layer according to the model; Based on the data replication execution graph, key-value cache replication nodes are added to the computation graph according to the model hierarchy. After all nodes are added, the computation graph is executed uniformly to realize the replication of the original key-value cache data to the expanded context data structure. After the key-value cache is copied, update the key-value cache's data structure references, computation graph parameters, and context state to complete the dynamic resizing operation.

[0037] In this embodiment, the steps for determining the expansion size are as follows: Prioritize expanding the capacity by using a preset maximum expansion factor that does not exceed the current context size. If the space is still insufficient after expansion, gradually reduce the expansion factor to a preset minimum limit. In each expansion, ensure that the expansion size meets the alignment requirements of the preset initial context size until a suitable expansion size is found.

[0038] In this embodiment, the specific steps for adding key-value cache replication nodes to the computation graph according to the model hierarchy are as follows: For key data, create one-dimensional views for source key cache data and target key cache data respectively, and create corresponding replication operator nodes in the computation graph based on the one-dimensional views; For value data, the replication method is determined based on the storage layout of the value data: when the value data is in a non-transposed layout, a one-dimensional view is created for the source value cache data and the target value cache data respectively, and a corresponding replication operator node is created in the computation graph; when the value data is in a transposed layout, a two-dimensional view is created for the source value cache data and the target value cache data respectively, and a corresponding replication operator node is created in the computation graph based on the two-dimensional view. The replication operator nodes are executed concurrently under the control of the computation graph scheduler to complete the batch replication of key-value cache data.

[0039] In specific application embodiments, such as Figure 3 As shown, the steps for implementing dynamic expansion are as follows: Step S31: First, determine whether the expansion flag is enabled, then determine the expansion size. When the key-value cache space is insufficient, first expand by a maximum of twice the context size. If the space is insufficient, gradually reduce the expansion coefficient, but it cannot be lower than the minimum limit (e.g., 1.2, while also satisfying alignment in units of alloc-size). Specifically, to determine whether the remaining space is sufficient for the current expansion size, it is necessary to assess the total memory requirement under the current context cache size. The assessment method is the same as in step S1. Then, obtain the device space and set a safety threshold (e.g., retaining no less than 1 / 3 of the total space, reserving a portion for small model inference or display tasks, etc.). Avoid frequent expansions to ensure both the flexibility of the key-value cache space and low latency and fast response.

[0040] Step S32: Prepare the computation graph and scheduler by copying the computation graph and reusing the scheduler and computation context of the current computation graph.

[0041] Step S33: Adjust the size of the metadata array (which manages the key-value cache), and then initialize the metadata parameters of the expanded portion.

[0042] Step S34: Create and initialize a new context data structure to store the new context key-value cache, allocate tensor data (key K and value V) layer by layer according to the model, and allocate a new buffer for this purpose.

[0043] Step S35: Add computation graph nodes by model layer. After adding all nodes, all operations are added to the computation graph. Then, execute the computation graph together to complete the copying of the key-value cache. The following is the creation of key-value (KV) nodes: For K data, create 1D views for both the source and destination data, and then create the cpy operator. For V data, it is relatively more complex. The non-transposed case is handled in the same way as K data. The transposed case requires optimization using 2D views. Create 2D views and add nodes. The underlying implementation of operators in the computation graph will execute concurrently, so it is not necessary to copy the V data row by row, which can greatly optimize performance.

[0044] Step S36: Update the key-value cache data structure, update the computation graph and context parameters, and trigger multi-sequence intelligent context reallocation.

[0045] Understandably, by using memory prediction and threshold control, when the key-value cache space is insufficient, memory requirements are calculated based on the model layer number, and a safety threshold is set for the remaining space. If sufficient space remains, the cache will be significantly expanded, avoiding frequent expansions. This ensures both the flexibility of the key-value cache space and low latency and fast response. Non-contiguous key-value cache blocks can affect graph computation performance; for local inference, contiguous key-value cache blocks offer the highest performance. This embodiment allocates new memory and uses the cpy operator to add it to the computation graph for concurrent processing, efficiently handling the expansion of keys k and values ​​v. After expansion, the computation graph is updated, allowing for flexible handling of scenarios with long context text and extremely long dialogues. Furthermore, the infrequent and efficient expansion method results in virtually imperceptible expansion to the user in practical testing.

[0046] In this embodiment, in step S4, intelligent context recycling and allocation are first performed. If this fails, the context of the inference sequence in the allocated context space is intelligently discarded and retained through a sequence-aware sliding window mechanism. The specific steps are as follows: Maintain the running status and active time of the inference sequence. First, reclaim part or all of the context space from the long-term inactive inference sequences, and then allocate it to the inference sequences that need it. If expansion and reclamation are not possible, the context of the inference sequence in the allocated context space is intelligently discarded and retained using a sequence-aware sliding window mechanism, as follows: Based on the context size allocated to each inference sequence, maintain an independent window size for each inference sequence; A predetermined number of context prefix contents in the inference sequence are retained and not pruned, and the context area excluding the prefix contents is used as a sliding window area; The number of contexts to be discarded is determined according to the preset discard ratio and the minimum discard limit. Based on the number of contexts to be discarded, the foremost part of the context content in the sliding window area is deleted, and the remaining context content is moved to the starting position of the sliding window area. After the pruning process is completed, the corresponding context index information and position encoding information are updated to release key-value cache space for storing new key-value cache data.

[0047] In a specific application embodiment, if the context space of the active sequence is insufficient and no resources are allocated, a recycling mechanism is implemented. If a sequence's last active time exceeds a threshold (e.g., more than 40 minutes of inactivity), an alloc-size context is taken from the inactive sequences and allocated to the active sequences. Inactive sequences undergo context deletion and movement, and are truncated in the saved generated token array. If allocation is not possible, a sequence-aware sliding window is used. The active time is updated after each sequence completes execution. Finally, the cache queue and sequence context information are updated.

[0048] If the remaining space is insufficient, meaning expansion is not possible, a sequence-aware sliding window mechanism will be triggered. This mechanism discards a certain amount of old cache to store new key-value cache, thus more effectively preserving important key-value cache and ensuring the normal operation of the task. For example... Figure 4 As shown, the specific steps of the sequence-aware sliding window mechanism are as follows: Step S41: Sequence dynamic perception, based on the context size allocated for each sequence according to the multi-sequence intelligent context, and maintain an independent window size for each sequence.

[0049] Step S42: Perform intelligent discarding, preserving important context prefixes. Typically, the first four tokens are kept fixed, while the remaining window area is used as a sliding window area. Then, based on a discarding strategy of proportion (10%, not less than 60%), the number of discarded tokens is calculated. The foremost part of the remaining sliding window is deleted, and the subsequent parts are moved to the front of the sliding area. An internal index management mechanism is used here, and this process is repeated cyclically. Finally, the rotation position encoding information and context information need to be updated.

[0050] It's understandable that maintaining two queues (pending and active) in the context module to manage sequence states enables intelligent context reclamation and allocation. Each sequence maintains an independent cache limit (window), and the window size is dynamically adjusted based on sequence usage and activity. By preserving important context prefixes and then using an appropriate discarding strategy, performance impact from overly frequent operations can be prevented, and inference accuracy can be avoided from decreasing due to excessive cache discarding.

[0051] In this embodiment, the method further includes a dynamic reduction process, which is triggered when the edge device needs to free up memory space for a new model or inference task, and includes the following steps: Determine the reduced target context size based on the target requirements for context size; Based on the computation graph used in the current large model inference process, the corresponding scheduler and computation context are reused by copying the computation graph to construct a data copying execution graph for key-value cache reduction. Adjust the size of the metadata array used to manage the key-value cache according to the target context size, and initialize the metadata parameters corresponding to the reduced context; Create a new context data structure to store the reduced key-value cache, and allocate corresponding key-value data and buffers to each layer according to the model hierarchy; Based on the data replication execution graph, key-value cache replication nodes are added to the computation graph according to the model hierarchy. After all nodes are added, the computation graph is executed uniformly to realize the replication of the original key-value cache data to the reduced context data structure. After the key-value cache copy is complete, update the key-value cache's data structure references, computation graph parameters, and context state, and after completing the dynamic shrinkage, set the expansion permission flag to the off state.

[0052] In specific application embodiments, a model scheduling layer is placed on top of the inference framework to manage the concurrency of multiple models. Under normal circumstances, only one model is loaded on a single device. If a second model is loaded and there is insufficient space, context shrinking is required. The implementation method is basically the same as that for expansion, which involves sampling newly allocated memory and then using the cpy operator to add it to the computation. Figure 1 Concurrent processing is initiated, and the computation graph is updated at the end. The execution steps are basically the same as those for dynamic expansion above. The final context size is set according to the requirements of the upper layer. Other steps are basically the same. After completing this step, the expansion flag is set to off, and expansion can only be performed after it is enabled later.

[0053] like Figure 6 and Figure 7 The image shown is a demonstration scenario (simple display effect) in the open-source Longbench. Figure 6 This is the generation result before optimization (fixed pre-allocation, current state in the edge AI PC) (all results are wrong, garbled text). Figure 7 This is the result after applying the optimization method of this embodiment. It can be seen that the result obtained by applying the method of this embodiment is more accurate.

[0054] Compared with the prior art, this embodiment has the following beneficial effects: This embodiment adjusts the key-value cache size in real time according to computing resources and context requirements. It achieves dynamic expansion and dynamic reduction (including dynamic expansion and dynamic reduction) based on pre-allocated cache. It can safely expand the cache capacity according to actual needs at runtime, avoid semantic loss and inference accuracy decline due to insufficient cache, as well as the generation of garbled characters. It significantly improves the ability of edge devices to process long sequences, so as to more effectively support the needs of long text and long dialogue as well as model parallelism. By managing key-value cache blocks through a sequence-aware sliding window mechanism and an intelligent discard strategy, important contextual information can be saved more effectively, improving the accuracy of inference results within a limited context space. By using multi-sequence intelligent context allocation, dynamic perception of sequence activity status and resource scheduling mechanism, as well as on-demand allocation and recycling mechanism, compared with the static average allocation strategy of existing technologies, it can adapt to the actual needs of different task sequences. Through on-demand allocation and recycling mechanism, it can significantly improve the utilization rate of multi-sequence cache resources.

[0055] Example 2 like Figure 5 As shown, the large model key-value caching dynamic optimization system of this embodiment is deployed in a large language model inference framework, and includes: The dynamic key-value cache management module is used to uniformly manage the key-value cache generated during the inference process of large models, and maintain the usage of the key-value cache according to the current context state; and to adjust the size of the key-value cache in real time according to computing resources and context requirements, and realize dynamic expansion and dynamic reduction based on pre-allocated cache; The multi-sequence intelligent context allocation module is used to reasonably allocate the key-value cache space according to the real-time context requirements of each inference sequence when there are multiple inference sequences. The sequence-aware sliding window module is used to prune the context of the inference sequence when the remaining space in the key-value cache is insufficient and expansion cannot be performed. This is done by discarding part of the historical context to free up key-value cache space.

[0056] The multi-sequence intelligent context allocation module is applied to the context initialization (step S1 of embodiment 1), intelligent allocation (step S2 of embodiment 1), and recycling mechanism (step S4 of embodiment 1) of the inference sequence. The multi-sequence intelligent context allocation module avoids idle and wasted resources by dynamically sensing the active state of the sequence and the resource scheduling mechanism, adjusting the resource allocation strategy in a timely manner when the number of sequences changes dynamically, and the on-demand allocation mechanism.

[0057] In this embodiment, the key-value cache dynamic optimization system performs unified context management, interfaces with the model scheduling and application layer, and runs with the collaborative support of the model loading and calculation module, scheduler, and memory management system. It is used to realize the dynamic adjustment of key-value cache and the continuous execution of inference tasks in a large model inference framework under the condition of limited edge resources.

[0058] The system of the present invention corresponds to the method described above and has the same advantages as the method described above.

[0059] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. Computer-readable media include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0060] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A dynamic optimization method for large model key-value caching based on the client side, characterized in that, Including the following steps: S1, start the large model inference task on the end device, centrally manage the context resources of multiple inference sequences through unified context management, and pre-allocate the corresponding context space for each inference sequence based on the preset initial context size; S2, determine whether the currently available context space meets the allocation requirements of the inference sequence: if it does, then perform intelligent context allocation for the inference sequence based on the currently allocated context space; if it does not, then continue to determine whether the remaining memory space of the edge device is sufficient, and execute step S3 or S4 according to the determination result: S3. If the remaining memory space of the terminal device is sufficient, the available context space is increased by dynamic expansion, and the context space allocation and context update of the inference sequence are performed after the dynamic expansion is completed. S4. If the remaining memory space of the edge device is insufficient, intelligent context reclamation and allocation are performed first. If this fails, the context of the inference sequence in the allocated context space is intelligently discarded and retained through the sequence-aware sliding window mechanism, and then allocated to the currently active inference sequence. After the allocation is completed, the context is updated.

2. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 1, characterized in that, The specific steps for pre-allocating the corresponding context space for each inference sequence based on a preset initial context size are as follows: Estimate the space required to load a large model and the amount of memory / GPU memory needed during the inference process of the large model based on the device model and memory / GPU size, and determine the initial context size based on the remaining memory space of the device. Two queues are maintained to manage the inference sequence state. For an active inference sequence, a portion of the context is pre-allocated from the allocated initial context.

3. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 1, characterized in that, In step S2, the specific steps for intelligently allocating the inference sequence based on the currently allocated context space are as follows: A differentiated and on-demand allocation strategy is implemented for inference sequences. Inference sequences are perceived and monitored in real time, and resources are allocated and reclaimed from the allocated context space according to actual needs, so as to improve the resource utilization of multiple sequences.

4. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 1, characterized in that, In step S3, the specific steps for increasing the available context space through dynamic expansion are as follows: Determine whether to enable the expansion flag. If enabled, determine the expansion size based on the total memory requirement under the current context cache size and the available memory space of the edge device, provided that the preset safety threshold is met. After determining the expansion size, based on the computation graph used in the current large model inference process, the scheduler and computation context of the current computation graph are reused by copying, and a data copying execution graph for key-value cache expansion is constructed. Adjust the size of the metadata array for managing the key-value cache based on the expansion size, and initialize the metadata parameters of the expansion portion; Create and initialize a new context data structure to store the new context key-value cache, and allocate key-value data and buffers layer by layer according to the model; Based on the data replication execution graph, key-value cache replication nodes are added to the computation graph according to the model hierarchy. After all nodes are added, the computation graph is executed uniformly to realize the replication of the original key-value cache data to the expanded context data structure. After the key-value cache is copied, update the key-value cache's data structure references, computation graph parameters, and context state to complete the dynamic resizing operation.

5. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 4, characterized in that, The steps for determining the expansion size are as follows: Prioritize expanding the capacity by using a preset maximum expansion factor that does not exceed the current context size. If the space is still insufficient after expansion, gradually reduce the expansion factor to a preset minimum limit. In each expansion, ensure that the expansion size meets the alignment requirements of the preset initial context size until a suitable expansion size is found.

6. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 4, characterized in that, The specific steps for adding key-value cache copy nodes to the computation graph according to the model hierarchy are as follows: For key data, create one-dimensional views for source key cache data and target key cache data respectively, and create corresponding replication operator nodes in the computation graph based on the one-dimensional views; For value data, the replication method is determined based on the storage layout of the value data: when the value data is in a non-transposed layout, one-dimensional views are created for the source value cache data and the target value cache data respectively, and corresponding replication operator nodes are created in the computation graph; When the value data is a transposed layout, create two-dimensional views for the source value cache data and the target value cache data respectively, and create corresponding copy operator nodes in the computation graph based on the two-dimensional views; The replication operator nodes are executed concurrently under the control of the computation graph scheduler to complete the batch replication of key-value cache data.

7. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 1, characterized in that, In step S4, intelligent context recycling and allocation are first performed. If this fails, the context of the inference sequence in the allocated context space is intelligently discarded or retained using a sequence-aware sliding window mechanism. The specific steps are as follows: Maintain the running status and active time of the inference sequence. First, reclaim part or all of the context space from the long-term inactive inference sequences, and then allocate it to the inference sequences that need it. If expansion and reclamation are not possible, the context of the inference sequence in the allocated context space is intelligently discarded and retained using a sequence-aware sliding window mechanism, as follows: Based on the context size allocated to each inference sequence, maintain an independent window size for each inference sequence; A predetermined number of context prefix contents in the inference sequence are retained and not pruned, and the context area other than the prefix contents is used as a sliding window area; The number of contexts to be discarded is determined according to the preset discard ratio and the minimum discard limit. Based on the number of contexts to be discarded, the foremost part of the context content in the sliding window area is deleted, and the remaining context content is moved to the starting position of the sliding window area. After the pruning process is completed, the corresponding context index information and position encoding information are updated to release key-value cache space for storing new key-value cache data.

8. The dynamic optimization method for large model key-value caching based on the end-side as described in claim 1, characterized in that, The method also includes a dynamic scaling process, which is triggered when the edge device needs to free up memory space for a new model or inference task, and includes the following steps: Determine the scaled-down target context size based on the target requirements for context size; Based on the computation graph used in the current large model inference process, the corresponding scheduler and computation context are reused by copying the computation graph to construct a data copying execution graph for key-value cache reduction. Adjust the size of the metadata array used to manage the key-value cache according to the target context size, and initialize the metadata parameters corresponding to the reduced context; Create a new context data structure to store the reduced key-value cache, and allocate corresponding key-value data and buffers to each layer according to the model hierarchy; Based on the data replication execution graph, key-value cache replication nodes are added to the computation graph according to the model hierarchy. After all nodes are added, the computation graph is executed uniformly to realize the replication of the original key-value cache data to the reduced context data structure. After the key-value cache copy is complete, update the key-value cache data structure references, computation graph parameters, and context state, and set the expansion permission flag to the off state after dynamic shrinking is completed.

9. A dynamic optimization system for large model key-value caching based on the method of any one of claims 1-8, characterized in that, The system is deployed within a large language model inference framework, and the system includes: The dynamic key-value cache management module is used to uniformly manage the key-value cache generated during the inference process of large models, and maintain the usage of the key-value cache according to the current context state; and to adjust the size of the key-value cache in real time according to computing resources and context requirements, and realize dynamic expansion and dynamic reduction based on pre-allocated cache; The multi-sequence intelligent context allocation module is used to reasonably allocate the key-value cache space according to the real-time context requirements of each inference sequence when there are multiple inference sequences. The sequence-aware sliding window module is used to prune the context of the inference sequence when the remaining space in the key-value cache is insufficient and expansion cannot be performed. This is done by discarding part of the historical context to free up key-value cache space.

10. The large model key-value caching dynamic optimization system according to claim 9, characterized in that, The key-value cache dynamic optimization system operates with the collaborative support of the model loading and computation module, scheduler, and memory management system. It is used to achieve dynamic adjustment of the key-value cache and continuous execution of inference tasks in a large model inference framework under the condition of limited resources on the edge side.

Images