Method and device for realizing inference acceleration through computing power layered token cache resource pool by intelligent computing cloud platform

Abstract

The application provides a method and device for implementing inference acceleration by power layer Token cache resource pool of intelligent computing cloud platform, relates to the technical fields of intelligent computing center, intelligent computing center, computing power infrastructure and intelligent computing cloud, and comprises the following steps: S1, dividing the Token-KV Cache generated by the intelligent computing cloud platform when analyzing and inferring a large model into multiple KV cache blocks, and obtaining the metadata fingerprint of the KV cache block; S2, determining the cache locality comprehensive score of the inference request at the candidate node according to the KV cache block, and determining the target node of the inference request; S3, determining the un-cached KV cache block in the power layer Token cache resource pool of the target node based on the metadata fingerprint and performing hierarchical storage processing; and S4, performing full life cycle management on the KV cache block during the inference request execution process. The application can realize cache-aware request routing and greatly reduce the waste of computing power.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing power infrastructure, and smart computing cloud technologies, specifically to a method and apparatus for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] Large language model inference typically involves two stages: prefill and decode. In the prefill stage, the model performs forward computation on a large number of tokens in the input prompt or context, and generates key / value states for each attention layer. In the decode stage, the model generates output token by token or in groups of tokens, and repeatedly reads the key / value states of previously received tokens. For long contexts and high-concurrency inference, the size of the KV cache grows approximately linearly with the context length, number of layers, number of attention heads, head dimension, batch size, and precision bytes, thus easily becoming a major source of overhead for GPU-based HBM.

[0008] Existing inference engines typically support capabilities such as continuous batch processing, paged attention, prefix caching, CPU offload, memory paging management, or prefill / decode decoupling, which can reduce memory fragmentation, reduce redundant prefix calculations, or improve batch processing efficiency to some extent. However, from the perspective of intelligent computing cloud platforms, there is still an important direction for evolution: elevating the KV cache from the internal state of a single inference instance to a platform-level, measurable, tradable, and manageable token-KV resource.

[0009] It is evident that since the emergence of intelligent computing centers, there has been a lack of an effective Token-KV caching resource pool solution for intelligent computing cloud platforms. This has prevented the system from truly achieving cache-aware request routing, resulting in a significant waste of computing power. This is a problem that urgently needs to be solved in this field. Summary of the Invention

[0010] This invention provides a method and apparatus for accelerating inference in an intelligent computing cloud platform through a tiered token caching resource pool, in order to solve the problem in the prior art where intelligent computing center cloud platforms cannot truly achieve cache-aware request routing, resulting in a large waste of computing power.

[0011] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for accelerating inference through a tiered token cache resource pool in an intelligent computing cloud platform, comprising: step S1, dividing the token-KV cache generated by the intelligent computing cloud platform when parsing inference requests for large models into multiple KV cache blocks, and obtaining the metadata fingerprint of the KV cache blocks; Step S2: Based on the KV cache block, determine the cache locality comprehensive score of the inference request at the candidate node, and based on the cache locality comprehensive score, determine the target node of the inference request; Step S3: Based on the metadata fingerprint, determine the reusable historical KV cache blocks in the computing power tiered token cache resource pool of the target node and the uncached KV cache blocks among the multiple KV cache blocks, and store the uncached KV cache blocks in the computing power tiered token cache resource pool in a tiered manner. Step S4: The target node executes the inference request based on the multiple KV cache blocks cached in the computing power hierarchical Token cache resource pool, and performs full lifecycle management of the KV cache blocks during the execution process.

[0012] In one embodiment, step S1 includes: Step S1.1: Divide the input word sequence of the reasoning request into multiple logical word intervals; Step S1.2: Obtain the KV cache block for each logical term interval under each Transformer layer of the large model.

[0013] In one embodiment, step S1.2 includes: Step S1.2.1: Obtain the first large model. l The input hidden state, key projection parameters, and value projection parameters of each Transformer layer; Step S1.2.2: According to the large model... l The input hidden states, key projection parameters, and value projection parameters of the Transformer layer are used to calculate the large model's nth layer. l The complete key tensor and complete value tensor of the Transformer layer; Step S1.2.3: Based on the logical lexical interval, analyze the large model... l Tensor slicing is performed on the complete key tensor and complete value tensor of each Transformer layer to obtain the corresponding KV cache block.

[0014] In one embodiment, step S1 further includes: Step S1.3: Determine the prefix hash chain of the i-th KV cache block, wherein the order of different KV cache blocks is determined based on the logical order of the KV cache blocks in the inference request; Step S1.4: Obtain auxiliary parameters related to the fingerprint, wherein the auxiliary parameters include at least one of precision format, tenant range, shared domain, security label, key version, signature policy, and integrity verification policy; Step S1.5: Construct the metadata fingerprint corresponding to the i-th KV cache block based on the prefix hash chain of the i-th KV cache block and the auxiliary parameters.

[0015] In one embodiment, step S1.3 includes: Step S1.3.1: Obtain environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block, wherein the environment configuration parameters include at least one of the following: model identifier, model version, tokenizer identifier, vocabulary version, LoRA version, Adapter version, plugin parameter version, cache format version, and position encoding scheme and its parameters; Step S1.3.2: Determine the prefix hash chain of the i-th KV cache block based on the environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block.

[0016] In one embodiment, step S1 further includes: Step S1.6: Determine the amount of pre-filling computation to be avoided when reusing the KV cache block, and convert the amount of pre-filling computation to be avoided into basic computing power revenue based on a preset computing power value conversion coefficient. Step S1.7: Determine the cost parameters generated by the KV cache block occupying each cache level of the computing power hierarchical Token cache resource pool, wherein the cost parameters include: cache occupation cost, cross-node or cross-level network transmission cost, and additional security management costs for encryption, verification, auditing and access control. Step S1.8: Deduct the cache occupation cost, the network transmission cost, and the security management additional cost from the basic computing power revenue to calculate the net computing power revenue of the KV cache block, wherein the net computing power revenue is used to determine the storage level of the corresponding KV cache block in the computing power tiered Token cache resource pool.

[0017] In one embodiment, step S1 further includes: Step S1.9: Determine the actual number of output tokens of the large model, and the equivalent number of tokens saved by avoiding repeated pre-computation by reusing the KV cache; Step S1.10: Convert the saved equivalent word units into effective output value according to the preset conversion weight, and add the effective output value to the actual output word units to obtain the comprehensive output benefit; Step S1.11: Calculate the total resource cost based on the computing resource cost, storage resource cost, and network resource cost during system operation; Step S1.12: The ratio of the comprehensive output revenue to the total resource cost is taken as the effective word element output efficiency.

[0018] In one embodiment, step S2 includes: Step S2.1: Obtain the scoring-related parameters of each candidate node, wherein the scoring-related parameters include net cache hit benefit, predicted future reuse probability, popularity of similar services or session families, memory pressure, decoding queue pressure, security risk, additional cost of cross-node transmission, and reward items that meet the first token delay, TPOT, or tenant priority. Step S2.2: Based on preset weights, perform a weighted summation on each of the scoring-related parameters to obtain the cache locality comprehensive score of the inference request at the corresponding candidate node.

[0019] In one embodiment, step S2.1 includes: Step S2.1.1: Determine the reusable hit status of the candidate KV cache block in the candidate node, wherein the value of the reusable hit status is positively correlated with the storage level of the candidate KV cache block; Step S2.1.2: Calculate the estimated recalculation cost of the candidate KV cache block and the transmission cost of transmitting the candidate KV cache block to the candidate node, and determine the difference between the estimated recalculation cost and the transmission cost. Step S2.1.3: When the candidate KV cache block meets the matching condition, multiply the difference by the reusable hit state, and accumulate the product of all the candidate KV cache blocks required by the request to obtain the net cache hit benefit of the candidate node.

[0020] In one embodiment, the candidate KV cache block is allowed to be reused when the key field of the candidate KV cache block meets the matching condition with the request attribute of the inference request. The matching conditions include at least one of the following conditions: The model identifier, tokenizer identifier, plug-in parameter version, and location encoding scheme of the candidate KV cache block are completely consistent with the inference request; The precision format or recoverable precision of the candidate KV cache block meets the precision requirements of the inference request. The tenant scope or shared domain to which the candidate KV cache block belongs is within the tenant or shared scope to which the inference request is authorized to access; The security label of the candidate KV cache block is not higher than the permission level of the inference request; The current time is earlier than the set expiration time of the candidate KV cache block.

[0021] In one embodiment, the configuration objective of the preset weights is to minimize the overall cost of the system, which is derived based on a statistical evaluation of historical routing decision results; The statistical indicators of the historical routing decision results include at least one of the following: first word delay, word generation delay during decoding, actual hit benefit, cache block loading time, recalculation time, GPU pause, service level target default status, and subsequent cache reuse count. The first word delay, the word generation delay, GPU resource overhead, network transmission overhead, memory overhead, and the service level target default status are multiplied by their respective preset conversion coefficients and then summed to obtain the comprehensive cost.

[0022] In one embodiment, the tiered token caching resource pool includes at least two of the following tiers: L0, L1, L2, L3, L4, and L5; wherein, The L0 layer is a GPU HBM; The L1 layer is a cache that can be directly accessed by the DPU, smart network card, or GPU; The L2 layer is CPU DRAM; The L3 layer is either NVMe or a local SSD; The L4 layer is a remote SSD, a remote memory pool, or a distributed cache; The L5 layer is object storage, shared storage, or archive storage.

[0023] In one embodiment, step S2 further includes: Step S2.3: Obtain a set of candidate nodes that meet preset conditions, wherein the preset conditions include at least tenant isolation, model version, and security policy; Step S2.4: Determine the pre-filled queue pressure of each candidate node, and the additional switching cost required to switch the inference request from the current session sticky node to the candidate node; Step S2.5: Based on the comprehensive score of cache locality of the inference request at each of the candidate nodes, the penalty item related to the pre-filled queue pressure, and the penalty item related to the additional switching cost, calculate the final routing score of each of the candidate nodes. Step S2.6: Select the candidate node corresponding to the maximum value of the final route score as the target node, and route the inference request to the target node.

[0024] In one embodiment, step S4 includes: Step S4.1: During the process of the target node executing the inference request based on the KV cache blocks in the computing power tiered token cache resource pool, the value parameters of each KV cache block in the computing power tiered token cache resource pool are obtained in real time. The value parameters include at least the future reuse probability, recalculation cost, capacity cost of continued storage, expected movement cost, risk parameters, and strategy reward parameters. Step S4.2: Calculate the corresponding lifetime value based on the value parameters of the KV cache block; Step S4.3: Determine the initial level of the KV cache block in the computing power tiered token cache resource pool and the changing trend of its lifecycle value; Step S4.4: Perform full lifecycle management on the corresponding KV cache block based on the initial level and the changing trend of the lifecycle value.

[0025] In one embodiment, step S4 further includes: Step S4.5: During the decoding phase, obtain the prefetch related parameters of the inference request, wherein the prefetch related parameters include the current decoding speed, the loading time of the KV cache block, the safety jitter margin, and the size of the KV cache block, and the loading time is the time required to prefetch the KV cache block from the current cache level to the target level; Step S4.6: Calculate the pre-fetched wavefront distance based on the pre-fetched relevant parameters; Step S4.7: Determine the interval between the expected consumption position and the current consumption position of the KV cache block; Step S4.8: When the interval does not exceed the prefetch wavefront distance, prefetch the KV buffer block.

[0026] In one embodiment, step S4.8 includes at least one of the following: Step S4.8.1: When the target service level of the inference request is greater than the preset level threshold, increase the security jitter margin or increase the prefetch priority; Step S4.8.2: When the high bandwidth memory pressure of the target level is greater than the preset pressure threshold, reduce the prefetch window.

[0027] In one embodiment, step S4 further includes: Step S4.9: Determine the set of KV cache blocks for the inference request, and divide the set of KV cache blocks into a first set that is hit and reusable, and a second set that is missed or not reusable; Step S4.10: Re-perform pre-filling calculation on the missing parts of the KV cache block set, and concatenate the hit KV cache blocks in the first set with the KV cache blocks obtained by re-pre-filling calculation according to their positions to construct a complete KV context; Step S4.11: Determine the prefix consistency check results of all hit KV cache blocks in the first set; Step S4.12: When the prefix consistency check result is passed, determine the recalculation cost of the hit KV cache block and the transmission cost of obtaining the hit KV cache block across levels; Step S4.13: If the transmission cost is higher than the recalculation cost, then the contribution of the corresponding hit KV cache block to the effective hit rate is determined to be negative, and the hit KV cache block is recalculated.

[0028] In one embodiment, step S4 further includes: Step S4.14: Identify the hit and reusable prefix data, and directly reuse the corresponding KV cache block; Step S4.15: Monitor the access progress during the decoding phase and downgrade KV cache blocks that are not in the current decoding wavefront to a lower-level cache medium, wherein the lower-level cache medium includes dynamic random access memory or non-volatile memory. Step S4.16: Promote the KV cache block that will be accessed in a short period of time to the higher-level cache of the computing power hierarchical Token cache resource pool; Step S4.17: Perform compression or eviction operations on the KV cache blocks whose lifetime value is lower than a preset value threshold; Step S4.18: Perform cache-aware routing on the inference requests so that inference requests with the same prefix share the higher-level cache.

[0029] In one embodiment, the KV cache block is bound to security metadata and security checks are performed during lookup, routing, migration, reuse, degradation, archiving, and eviction processes. The security mechanisms used for the security checks include at least: tenant-scope binding, security label binding, key isolation, integrity verification, failure policy, and audit trail.

[0030] In one embodiment, the metadata fingerprint includes a version vector, wherein the version vector is determined based on at least one of model weights, tokenizer, position encoding, adapter, quantization parameters, and attention kernel.

[0031] In one embodiment, step S3 includes: Step S3.1: Based on the metadata fingerprint, determine the matching result between the KV cache block of the inference request and the KV cache block in the computing power tiered token cache resource pool; Step S3.2: Based on the matching result, determine the KV cache blocks that are not stored in the target node's computing power hierarchical Token cache resource pool; Step S3.3: Determine the net computing power benefit of the unstored KV cache block; Step S3.4: Based on the net computing power revenue, store the unstored KV cache blocks in the computing power tiered Token cache resource pool in a tiered manner.

[0032] Secondly, the present invention also provides an apparatus for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform, comprising: The processing module is used to divide the Token-KVCache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks, and obtain the metadata fingerprint of the KV cache blocks; The first determining module is used to determine the cache locality comprehensive score of the inference request at the candidate node based on the KV cache block, and to determine the target node of the inference request based on the cache locality comprehensive score. The second determining module is used to determine, based on the metadata fingerprint, reusable historical KV cache blocks in the computing power hierarchical token cache resource pool of the target node and uncached KV cache blocks among the multiple KV cache blocks, and store the uncached KV cache blocks hierarchically in the computing power hierarchical token cache resource pool. The management module is used to execute the inference request on the target node based on the multiple KV cache blocks cached in the computing power hierarchical token cache resource pool, and to perform full lifecycle management of the KV cache blocks during the execution process.

[0033] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the method for accelerating inference through a tiered token caching resource pool in the intelligent computing cloud platform described in the first aspect above.

[0034] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the method for accelerating inference through a tiered token caching resource pool in the intelligent computing cloud platform described in the first aspect above.

[0035] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the method for accelerating inference through a tiered token caching resource pool in the intelligent computing cloud platform described in the first aspect above.

[0036] In this invention, step S1 involves dividing the Token-KV Cache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks and obtaining the metadata fingerprints of the KV cache blocks; step S2 involves determining the cache locality comprehensive score of the inference request on the candidate nodes based on the KV cache blocks, and determining the target node of the inference request based on the cache locality comprehensive score; step S3 involves determining the reusable historical KV cache blocks and the uncached KV cache blocks in the computing power tiered Token cache resource pool of the target node based on the metadata fingerprints, and storing the uncached KV cache blocks hierarchically in the computing power tiered Token cache resource pool; step S4 involves executing the inference request on the target node based on the multiple KV cache blocks cached in the computing power tiered Token cache resource pool, and performing full lifecycle management of the KV cache blocks during the execution process. By introducing metadata fingerprints and cache-aware routing, the KV Cache is transformed from a temporary state within a single machine into a shared resource at the cloud platform level, thereby enabling cross-request and cross-session reuse of prefix computation results among multiple nodes, effectively reducing the computing power waste caused by repeated pre-filling. Attached Figure Description

[0037] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a method for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform, as provided by this invention. Figure 2 This is a structural diagram of a device for accelerating inference through a hierarchical token caching resource pool in an intelligent computing cloud platform provided by the present invention. Figure 3 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0039] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] The “computing power” mentioned in this invention refers to the ability of computer equipment or computing / data centers to process information. It is the ability of computer hardware and software to work together to perform a certain computing requirement. It is the computing power to achieve the target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0041] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0042] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capabilities of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capabilities.

[0043] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB), and the disaster recovery ratio is an important indicator of security and reliability.

[0044] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0045] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0046] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0047] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0048] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0049] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0050] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0051] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0052] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0053] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0054] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0055] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0056] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0057] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0058] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0059] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0060] The "Token" mentioned in this invention refers to a basic unit of measurement when large language models or multimodal models process text, code, image descriptions, document content, or other serialized inputs and outputs. It is important to emphasize that different models use different tokenizers. The same text may correspond to different numbers of tokens in different models.

[0061] The "inference request" mentioned in this invention refers to a call instruction sent by a user or upstream business system to a large model, requesting it to generate text or results based on input prompts.

[0062] The “Token-KV Cache” mentioned in this invention refers to the key and value tensor states of each attention layer generated during the forward propagation process when a large model processes input text.

[0063] The "Token-KV block" described in this invention refers to a key / value cache block generated by one or more tokens on one or more Transformer layers, a set of attention heads, or a specified tensor shard. A Token-KV block can be divided according to continuous token ranges, layer numbers, head groups, tensor parallel sharding, or expert branches.

[0064] The “metadata fingerprint” mentioned in this invention refers to a set of structured identifiers used to determine whether a certain Token-KV block can be securely and correctly reused, including but not limited to prompt prefix hash, model identifier, tokenizer identifier, adapter identifier, location encoding scheme, tenant range, security label, version number, generation time, expiration time, precision format, cache level, and physical location.

[0065] The "candidate node" mentioned in this invention refers to an idle or available server in the resource pool that can be allocated.

[0066] The "target node" mentioned in this invention refers to the server that is ultimately designated to execute the inference request after evaluation by the scheduling algorithm.

[0067] The "Token-KV locality score" described in this invention refers to the comprehensive locality score calculated by the scheduler for a request when executing on a candidate node, candidate GPU, or candidate inference instance. This score not only expresses "whether the cache is hit," but also the computing power saved after a hit, the cost required to move the cache, the impact of migration on memory pressure, SLO constraints, and security policies.

[0068] The “Hierarchical Token Cache Resource Pool (KV Lake)” mentioned in this invention refers to a cloud platform-level Token-KV cache resource pool, which is not the same as a single storage system, but a hierarchical resource management plane composed of HBM, DRAM, NVMe, remote SSD, object storage, shared storage, DPU / smart NIC side cache, etc.

[0069] The "full lifecycle management" mentioned in this invention refers to the dynamic management and control of the entire process of KV cache blocks from generation, storage, reuse, hierarchical migration to final deletion (elimination).

[0070] The “input lexical sequence” mentioned in this invention refers to a computer-readable discrete numeric ID string that is transformed from the original text of the reasoning request after being processed by a tokenizer.

[0071] The "logical lexical interval" mentioned in this invention refers to: artificially dividing the input lexical sequence into multiple segments according to a fixed length or semantic boundary in order to perform block-based computation and caching.

[0072] The "Transformer layer" mentioned in this invention refers to the basic neural network structure unit inside a large model, which is usually composed of multiple layers of Transformers stacked together.

[0073] The "input hidden state" mentioned in this invention refers to the high-dimensional vector feature representation of lexical units in a specific neural network layer of the model.

[0074] The "Key / Value projection parameter" mentioned in this invention refers to a part of the model weight matrix, which is used to calculate the corresponding Key and Value tensors from the input hidden states through linear transformation.

[0075] The "tensor slicing" described in this invention refers to the operation of cutting the calculated complete multidimensional tensor data into small blocks according to logical intervals in physical memory (video memory).

[0076] The “prefix hash chain” described in this invention refers to an encrypted linked list structure based on a hash algorithm, which binds the content of the current block to the hash values ​​of all previous blocks to ensure that the context order of the cache sequence is absolutely correct and cannot be tampered with.

[0077] The “auxiliary parameters / security tags / tenant range” mentioned in this invention refer to attribute tags attached to cache blocks, which are used to define the precision format, security level, and which enterprise customers (tenants) are allowed to access the cache block.

[0078] The "environment configuration parameters" mentioned in this invention refer to the snapshot of the underlying hardware and software environment when the cache is generated, such as the model version, word segmenter version, etc.

[0079] The “LoRA version / Adapter version” mentioned in this invention refers to a fine-tuned version of the large model. The model behavior is different under different fine-tuned versions, and the generated caches cannot be used interchangeably.

[0080] The "position encoding scheme" described in this invention refers to a mathematical algorithm (such as RoPE) that determines how the model understands the order of tokens in a sentence. Different parameters will result in completely different tensor values.

[0081] The “prefill computation” mentioned in this invention refers to the number of floating-point operations (FLOPs) required by the Large Language Model (LLM) to process the complete input prompt during the inference prefill phase.

[0082] The "basic computing power benefit" mentioned in this invention refers to the economic value converted from the computing power saved by reusing the cache.

[0083] The "net computing power revenue" mentioned in this invention refers to the actual net profit after deducting the storage and network relocation costs from the basic computing power revenue.

[0084] The "effective token production efficiency" mentioned in this invention refers to a comprehensive indicator for evaluating the energy efficiency of a system, reflecting how many tokens are actually generated effectively per unit of resource cost (including GPU, storage, and network).

[0085] The “video memory pressure / decoding queue pressure” mentioned in this invention refers to an indicator that measures the current remaining GPU video memory space of a computing node and the degree of congestion of requests waiting in the queue for processing.

[0086] The "First Token Delay (TTFT) / TPOT" mentioned in this invention refers to a key indicator for measuring the model's response speed. TTFT refers to the time taken to generate the first token, and TPOT (Time Per Output Token) refers to the average generation time for each subsequent token.

[0087] The "matching condition" mentioned in this invention refers to the rigid preconditions (such as consistent permissions) that must be met for the system to reuse a certain cache block.

[0088] The "recoverable precision" mentioned in this invention means that if the cache is compressed during storage (e.g., compressed from FP16 to INT8), it can be recovered and meet the minimum precision requirements of the request when loaded.

[0089] The "historical routing decision results" mentioned in this invention refer to the records of past task allocations by the system and the results generated (such as whether there were timeouts or stutters), which are used by machine learning algorithms to automatically fine-tune future scheduling weights.

[0090] The "L0 to L5 storage" described in this invention refers to the physical storage hierarchy divided by the system. L0 (GPU HBM) is the fastest and most expensive; L1 (DPU direct-attached cache, etc.) is next; L2 (CPU DRAM) is next; L3 / L4 (SSD) is slower but has a large capacity; L5 (object / archive storage) is the slowest and cheapest, and is used for cold data.

[0091] The “session sticky node” mentioned in this invention refers to a server that has previously processed a user’s session and has accumulated a lot of cache for that user locally.

[0092] The "additional switching cost" mentioned in this invention refers to the performance loss caused by forcibly assigning a user's continuous conversations to another unprepared new server.

[0093] The "lifetime value" mentioned in this invention refers to a comprehensive quantitative score used to dynamically evaluate how likely a cache block is to be reused in the future and whether it is worthwhile to keep it in high-speed video memory.

[0094] The "prefetch wavefront distance" mentioned in this invention refers to a safety buffer zone that the system needs to send subsequent cache fetch instructions to the slow hard drive in advance to prevent the model from calculating too fast and waiting for data.

[0095] The "safe jitter margin" mentioned in this invention refers to the amount of data (margin) that is preloaded in advance to cope with sudden network lag.

[0096] The “token-wavefront prefetching” described in this invention refers to a mechanism that, based on the current decoding speed, the subsequent token consumption rhythm, and network and storage access latency, pre-moves the Token-KV block that is about to be accessed by attention computation from a lower-level cache to a higher-level cache or a location that can be directly accessed by the GPU.

[0097] The "prefetch window" mentioned in this invention refers to the range (window) of data that the system is allowed to move to video memory in a single operation.

[0098] The "prefix consistency check result" mentioned in this invention refers to the final mathematical check performed by the system before concatenating multiple cache blocks together, ensuring that the concatenated context logic is coherent and correct.

[0099] The "key isolation" mentioned in this invention refers to encrypting the data of different tenants with different passwords.

[0100] The "audit trail" mentioned in this invention refers to a log that records who accessed the cache and when.

[0101] The "version vector" mentioned in this invention refers to an array that records version numbers.

[0102] The "unstored KV cache block" mentioned in this invention refers to a completely new context fragment that the system has never seen before, calculated after processing a request. The system will evaluate its value and store it in the database.

[0103] It is important to emphasize that KV cache is not a regular text cache. Building a token-KV cache resource pool for cloud platforms is an extremely complex systems engineering challenge, which will be explained in detail below: 1. Fragmentation and compatibility barriers caused by strong binding characteristics.

[0104] Traditional cloud caching (such as Redis caching HTML or JSON) is platform-independent; as long as the content is consistent, it can be reused across the entire network. However, key-value (KV) caches are deeply bound to model weights, tokenizers, positional encodings (such as RoPE), adapters (such as LoRA), precision, and even underlying operators. Therefore, intelligent computing cloud platforms cannot simply match caches based on text hashes; instead, they must maintain an extremely large and fine-grained multidimensional metadata index. If the same public system prompt is invoked by five models with five different parameter configurations in the cloud, five completely different KV tensors need to be stored. This significantly increases the storage fragmentation rate of the resource pool and the complexity of addressing and routing.

[0105] 2. Strict isolation and security risk control in a multi-tenant environment.

[0106] In enterprise-level large-scale model services, KV Cache carries highly sensitive business data (such as proprietary prompt structures, confidential documents retrieved by RAG, and users' private session history). Therefore, intelligent computing cloud platforms must establish high-standard access control and isolation mechanisms. If cache penetration or cross-tenant hash collisions occur (e.g., two different tenants happen to send the same prefix text, but with different subsequent logic), misscheduling can lead to serious data leaks. Achieving strong isolation similar to VPC at the distributed memory (HBM) level while ensuring extremely low latency is technically extremely difficult.

[0107] 3. Scheduling game triggered by high mobility costs.

[0108] Key-value caches are extremely large (often ranging from several gigabytes to tens of gigabytes). Retrieving such massive tensor data between distributed cloud nodes over the network (even with RDMA or dedicated lines) is physically time-consuming. At this point, the scheduler faces a dynamic trade-off between transmission time and local recalculation time. The system not only needs to know the cache location but also needs to be aware of network bandwidth congestion and the computing load of each node in real time, and quickly calculate whether routing requests to a remote cache is more cost-effective than having the GPU directly refill the cache locally. Implementing such a global scheduling algorithm with both network and computing power awareness is extremely difficult.

[0109] 4. Cache eviction failure caused by multi-dimensional popularity characteristics.

[0110] Traditional Least Recently Used (LRU) algorithms are too simplistic for key-value (KV) caches. The reuse value of a KV string depends not only on time but also heavily on business flow (e.g., templates for specific workflows), session activity, and even the token's position. Therefore, intelligent computing cloud platforms need to design intelligent, multi-dimensional lifecycle management strategies. This requires resource pools not only to store data but also to understand the contextual relationships within that data, predicting which prefixes will be frequently accessed concurrently in the next few minutes and which are merely transient elements in a single session. Designing and maintaining such a low-overhead, high-accuracy dynamic eviction algorithm is a significant challenge in the industry.

[0111] 5. Precise prefetching synchronization is required for decoding wavefront features.

[0112] During the decoding phase, GPUs generate tokens one by one, and the demand for historical key-value pairs progresses gradually, like waves, rather than being consumed all at once. The resource pool also has strict requirements on the timing of distributing cached data to inference nodes: distributing it too quickly will overwhelm the GPU's precious HBM, leading to OOM (Out of Memory); distributing it too slowly will cause the GPU's computing power to idle, waiting for network data returns, resulting in severe token generation stuttering (increased latency after the first letter). Therefore, the platform must establish an extremely precise "streaming backpressure and on-demand prefetching" mechanism between distributed storage and the GPU inference kernel to achieve perfect synchronization between data supply and computing power consumption.

[0113] In summary, elevating KV Cache from a single-machine internal state to a cloud platform-level resource is not just a simple storage problem, but a cross-disciplinary challenge that integrates tensor computation, distributed network routing, strong security sandbox, dynamic operations research scheduling, and hardware-software co-design. This invention solves this pressing problem in the field of computing power.

[0114] Please see Figure 1 , Figure 1This is a flowchart of a method for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform, as provided by this invention. Figure 1 As shown, the method includes: Step S1: Divide the Token-KV Cache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks, and obtain the metadata fingerprint of the KV cache blocks.

[0115] It's important to note that in large-scale cloud-based inference scenarios, user requests often contain a large amount of repetitive prefix content (such as system prompts, fixed templates, etc.). Traditional inference systems typically treat the generated context cache (KV Cache) as a large and indivisible whole, lacking fine-grained segmentation and identification mechanisms. This coarse-grained management approach prevents the system from accurately identifying and extracting reusable common fragments, leading to a large amount of invalid and redundant computation, wasted expensive GPU memory, and increased cross-node scheduling costs.

[0116] Therefore, it is urgent to divide the massive, contiguous KV cache into fine-grained standard cache blocks and extract metadata fingerprints containing hardware and software configuration information. This step transforms the previously difficult-to-manage internal tensor states of the model into cloud assets that can be accurately retrieved, verified, and securely isolated, thus laying the core foundation for subsequent implementation of cross-request reuse, intelligent routing and distribution, and hierarchical scheduling of global resources.

[0117] Optionally, in some embodiments, the metadata fingerprint may be implemented using SHA series, BLAKE series, HMAC with key, Merkle tree, content-addressable object ID, or platform-defined irreversible digest.

[0118] Optionally, in some embodiments, the technical solutions disclosed herein can be applied to large text models, large code models, visual language models, speech multimodal models, long context reasoning models, MoE models, RAG systems, Agent tool invocation systems, and enterprise private model reasoning platforms.

[0119] The beneficial technical effect achieved by step S1 is that by dividing the continuous and large Token-KV Cache into multiple independent cache blocks and extracting exclusive metadata fingerprints, an accurate identification foundation can be laid for achieving precise feature matching and data reuse across sessions in subsequent massive concurrent requests.

[0120] Step S2: Based on the KV cache block, determine the cache locality comprehensive score of the inference request on the candidate node, and based on the cache locality comprehensive score, determine the target node of the inference request.

[0121] It's important to note that in traditional large-scale model cluster scheduling, gateway distribution typically relies solely on conventional load balancing strategies such as node idle time or round-robin. The scheduler is essentially blind to the physical distribution of the underlying massive key-value cache across the nodes. This blind scheduling, lacking data awareness, can easily dispatch requests containing common prefixes to nodes without relevant caches, forcing the GPU to spend a significant amount of time on ineffective pre-filling computations from scratch. Alternatively, in order to force reuse, it may trigger extremely bandwidth-intensive cross-node tensor fetching, further exacerbating inference latency and wasting computing power.

[0122] Therefore, there is an urgent need to introduce an intelligent routing mechanism based on a comprehensive scoring system for cache locality. This step allows new inference requests to proactively seek out existing data. By accurately evaluating the matching benefits of new requests with the local caches of each candidate node before dispatching requests, it ensures that tasks are directed to the optimal node with the highest hit rate. This not only maximizes the utilization of local high-speed GPU memory but also fundamentally avoids cross-node network transmission bottlenecks and redundant computation overhead, making it a key scheduling hub for achieving low-latency, high-concurrency inference on cloud platforms.

[0123] The beneficial technical effects achieved by step S2 are as follows: By introducing an intelligent routing mechanism based on cache locality scoring, inference requests with repeated prefixes can be accurately dispatched to computing nodes that already hold the relevant context, thereby maximizing the local data hit rate and significantly reducing network congestion and transmission latency caused by fetching large tensors across nodes.

[0124] Step S3: Based on metadata fingerprints, determine the reusable historical KV cache blocks and the uncached KV cache blocks in the target node's computing power tiered token cache resource pool, and store the uncached KV cache blocks in a tiered manner in the computing power tiered token cache resource pool.

[0125] It's important to note that the context cache (KV cache) of large models is extremely large. Even if requests are accurately dispatched to the target node via front-end routing, the system will still face severe resource bottlenecks if there is a lack of fine-grained storage and management mechanisms within the node: if all newly calculated caches are crammed into GPU high-speed memory (HBM), the extremely expensive memory resources will be exhausted instantly (OOM); if cache misses are discarded directly to save space, the next time the same prefix is ​​encountered, computing power must be wasted to recalculate. This unplanned, flat storage model severely limits the total amount of historical context that a single node can carry and retain.

[0126] Therefore, there is an urgent need to introduce a storage scheduling mechanism based on fingerprint comparison and computing power tiering.

[0127] Optionally, in some embodiments, the hierarchy of the computing power tiered token cache resource pool KV Lake can be adjusted according to hardware conditions, and may include HBM, SRAM, GPU L2 hint cache, CPU DRAM, CXL memory pool, local NVMe, remote NVMe, object storage, DPU cache, smart switch cache or other accessible storage.

[0128] The beneficial technical effects achieved by step S3 are as follows: Based on the accurate fingerprint comparison mechanism, existing historical computing results are effectively reused, and the new computing KV cache blocks that are not matched are distributed to different levels of storage media according to factors such as cost and access frequency. This can directly eliminate the consumption of computing power for repeated pre-filling, while taking into account storage capacity and access speed, and achieve the optimization of the economic benefits of global hardware resources.

[0129] Step S4: Execute inference requests on multiple KV cache blocks cached in the target node's tiered token cache resource pool based on computing power, and perform full lifecycle management of the KV cache blocks during the execution process.

[0130] It should be noted that even if the system achieves accurate request routing and hierarchical storage through the preceding steps, it will still quickly get stuck if there is a lack of dynamic management of this data during the actual inference execution phase. As concurrent requests continue to pour in, the most expensive and fastest GPU memory (HBM) has limited space; if it only stores data without deleting it or adopts a rigid cleanup strategy (such as static LRU), the memory will soon be filled with cold data that has only been used once, leaving no place for the truly high-frequency hot cache, ultimately leading to memory overflow (OOM) or a sharp drop in the overall hit rate.

[0131] Therefore, it is urgent to introduce a dynamic management mechanism for the entire lifecycle of KV cache blocks during the inference execution process.

[0132] The beneficial technical effects achieved by step S4 are: it can ensure that computing nodes can quickly and smoothly call multi-level cache data to accelerate model decoding during the actual inference execution phase. At the same time, through dynamic management throughout the entire life cycle of operation, it can perform upgrade, downgrade or eviction operations on cache blocks in a timely manner to prevent memory accumulation and fragmentation, and ensure that the node cache resource pool can maintain a high hit rate and high turnover efficiency for a long time.

[0133] In this invention, step S1 involves dividing the Token-KV Cache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks and obtaining the metadata fingerprints of the KV cache blocks; step S2 involves determining the cache locality comprehensive score of the inference request on the candidate nodes based on the KV cache blocks, and determining the target node of the inference request based on the cache locality comprehensive score; step S3 involves determining the reusable historical KV cache blocks and the uncached KV cache blocks in the computing power tiered Token cache resource pool of the target node based on the metadata fingerprints, and storing the uncached KV cache blocks hierarchically in the computing power tiered Token cache resource pool; step S4 involves executing the inference request on the target node based on the multiple KV cache blocks cached in the computing power tiered Token cache resource pool, and performing full lifecycle management of the KV cache blocks during the execution process. By introducing metadata fingerprints and cache-aware routing, the KV Cache is transformed from a temporary state within a single machine into a shared resource at the cloud platform level, thereby enabling cross-request and cross-session reuse of prefix computation results among multiple nodes, effectively reducing the computing power waste caused by repeated pre-filling.

[0134] In one embodiment, step S1 includes: Step S1.1: Divide the input word sequence of the reasoning request into multiple logical word intervals.

[0135] Optionally, in some embodiments, when dividing the input token sequence of the inference request into multiple logical token intervals, the input token sequence of a request can be denoted as... The cache is divided into G based on the size B of the key-value block (KV). n / B There are several logical token ranges. The g-th range is: .

[0136] Optionally, in some embodiments, semantic boundary segmentation can also be used for inputs that naturally possess semantic boundaries, such as enterprise system prompts, contract terms, code files, and knowledge base paragraphs. The system can combine "fixed token number segmentation" and "semantic paragraph segmentation" while ensuring correct positional encoding and causal attention. To be compatible with PagedAttention-like kernels, the system can also continue mapping to fixed physical pages within semantic blocks.

[0137] Optionally, in some embodiments, the granularity of the Token-KV block can be a fixed number of tokens, semantic paragraphs, attention head groups, hierarchical groups, tensor parallel fragments, RAG document fragments, or a combination of multiple granularities.

[0138] Step S1.2: Obtain the KV cache blocks for each logical term interval under each Transformer layer of the large model.

[0139] The beneficial technical effects achieved by steps S1.1 and S1.2 are as follows: by dividing the input tokens of long sequences into logical intervals and establishing a mapping with the physical cache blocks of each layer of Transformer, the system can more flexibly match and reuse local common prefixes or segments, thereby improving the hit probability and granularity of cache reuse.

[0140] In one embodiment, step S1.2 includes: Step S1.2.1: Obtain the large model's first... l The input hidden state, key projection parameters, and value projection parameters of each Transformer layer.

[0141] Step S1.2.2: Based on the large model... l The input hidden states, key projection parameters, and value projection parameters of the Transformer layer are used to calculate the largest model's... l The complete key tensor and complete value tensor of the Transformer layer.

[0142] Step S1.2.3: Based on logical lexical intervals, the large model... l Tensor slicing is performed on the complete key tensor and complete value tensor of each Transformer layer to obtain the corresponding KV cache block.

[0143] The beneficial technical effects achieved by steps S1.2.1 to S1.2.3 are as follows: By directly generating and extracting complete tensor data of a specific interval based on the model's hidden state and projection parameters, the accurate execution and correct alignment of logical word segmentation in the underlying mathematical matrix operations and physical storage of the video memory can be guaranteed.

[0144] For example, for the first l The key / value tensor of a layer can be represented as:

[0145] Z l For the first l Layer input hidden state. and These are the key / value projection parameters, respectively. The g-th Token-KV block can be defined as:

[0146] In tensor parallel, pipeline parallel, or expert parallel scenarios, the Token-KV block can further include a shard number s to form... In scenarios involving multiple query attention or grouped query attention, blocks can also be divided into key-value head groups, thereby reducing unnecessary granularity of movement.

[0147] In one embodiment, step S1 further includes: Step S1.3: Determine the prefix hash chain of the i-th KV cache block, wherein the order of different KV cache blocks is determined based on the logical order of the KV cache blocks in the inference request.

[0148] Step S1.4: Obtain auxiliary parameters related to the fingerprint, wherein the auxiliary parameters include at least one of the following: precision format, tenant range, shared domain, security label, key version, signature policy, and integrity verification policy.

[0149] Step S1.5: Construct the metadata fingerprint corresponding to the i-th KV cache block based on the prefix hash chain and auxiliary parameters of the i-th KV cache block.

[0150] The beneficial technical effects achieved by steps S1.3 to S1.5 are as follows: by combining a prefix hash chain with a strict logical order and multi-dimensional security and tenant auxiliary parameters, hash collisions are fundamentally prevented, and cross-tenant isolation and security control of cached data are ensured.

[0151] Optionally, in some embodiments, the metadata fingerprint may also include generation time, last access time, access count, reference count, compression algorithm, checksum, physical location, level, node number, GPU number, DPU cache address, object storage key, failure reason, and audit serial number. The physical location can be stored as a variable attribute in the metadata index, while the logical fingerprint remains stable, so that the Token-KV block does not need to change its logical identity when migrating between different levels.

[0152] In one embodiment, step S1.3 includes: Step S1.3.1: Obtain environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block. The environment configuration parameters include at least one of the following: model identifier, model version, tokenizer identifier, vocabulary version, LoRA version, Adapter version, plugin parameter version, cache format version, and location encoding scheme and its parameters.

[0153] Step S1.3.2: Determine the prefix hash chain of the i-th KV cache block based on the environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block.

[0154] The beneficial technical effects achieved by steps S1.3.1 to S1.3.2 are as follows: By forcibly incorporating the underlying environment configurations such as model version, scaling parameters (such as RoPE), and fine-tuning weights (such as LoRA) into the calculation of the hash chain, it can be ensured that cache reuse is only allowed under the premise that the operator and model environment are completely consistent, thus avoiding calculation logic errors.

[0155] Understandably, to ensure the correctness and security of cache reuse, this embodiment establishes a metadata fingerprint for each Token-KV block. Let c be the token content digest of the g-th block. g Its prefix hash chain is p g Then we can define:

[0156] Where m represents the model identifier and model version, and τ represents the tokenizer identifier and vocabulary version. a Indicates the version of the LoRA, Adapter, or plugin parameter. ρ This represents the location coding scheme and its parameters. v This indicates the cache format version. Based on the prefix hash chain, the final Token-KV block's reuse fingerprint can be represented as:

[0157] in, Indicates the precision format, such as BF16, FP16, FP8, or quantization format; q Indicates tenant scope and shared domain; r Indicates a safety label; κ This indicates the key version, signature policy, or integrity verification policy.

[0158] In one embodiment, step S1 further includes: Step S1.6: Determine the amount of pre-filling computation to be avoided when reusing KV cache blocks, and convert the amount of pre-filling computation to be avoided into basic computing power revenue based on the preset computing power value conversion coefficient.

[0159] Step S1.7: Determine the cost parameters generated by each cache level of the KV cache block's computing power tiered Token cache resource pool. The cost parameters include: cache occupancy cost, cross-node or cross-level network transmission cost, and additional security management costs for encryption, verification, auditing, and access control.

[0160] Step S1.8: Deduct cache occupation cost, network transmission cost and security management additional cost from the basic computing power revenue to calculate the net computing power revenue of the KV cache block. The net computing power revenue is used to determine the storage level of the corresponding KV cache block in the computing power tiered Token cache resource pool.

[0161] The beneficial technical effects achieved by steps S1.6 to S1.8 are as follows: by quantifying the computing power benefits brought by caching and deducting the costs of cache occupation, network transmission and security management, a net benefit index is calculated, and the KV cache block should be placed in a certain storage level based on the net benefit index, which can effectively avoid blind storage and ineffective network migration.

[0162] In other words, in this embodiment of the disclosure, the system can record the avoidable computation, memory usage time, and caching service cost corresponding to each Token-KV block as a computing power ledger. Let's assume a reused block... b The amount of prefill computation to be avoided is F save (b) The duration it occupies at cache level z is T z (b) The unit capacity cost is p z The transmission cost is p ne If t(b), then the net computing power gain of this block can be defined as:

[0163] Where α F p is the conversion factor for computing power value. sec (b) Costs for encryption, verification, auditing, and access control. The platform can calculate U based on tenant, model, business line, template, or session. KV This allows us to identify which system prompts, enterprise documents, or code contexts are most worthy of being permanently cached at a high level.

[0164] In one embodiment, step S1 further includes: Step S1.9: Determine the actual number of output tokens of the large model, and the equivalent number of tokens saved by avoiding repeated pre-computation through KV cache reuse.

[0165] Step S1.10: Convert the saved equivalent word units into effective output value according to the preset conversion weight, and add the effective output value to the actual output word units to obtain the comprehensive output benefit.

[0166] Step S1.11: Calculate the total resource cost based on the computing resource cost, storage resource cost, and network resource cost during system operation.

[0167] Step S1.12: The ratio of comprehensive output revenue to total resource cost is taken as the effective word element output efficiency.

[0168] The beneficial technical effects achieved by steps S1.9 to S1.12 are as follows: by converting the saved equivalent word elements into output and calculating the effective word element output efficiency in combination with the overall system resource cost, the obtained effective word element output efficiency can enable the cloud platform to intuitively quantify the real performance improvement and cost savings brought about by the global caching strategy.

[0169] In some embodiments, the effective token production efficiency can be defined as:

[0170] in N out This represents the actual number of tokens output. N prefill_saved To avoid duplicate prefilling through Token-KV reuse, the equivalent number of tokens is required. C GPU , C mem , C net These are the costs for GPU, storage, and network resources, respectively. λ This metric is used to convert saved prefill tokens into effective output. Through this indicator, the platform can directly incorporate KV cache reuse into its computing power management metrics, rather than just as a runtime optimization.

[0171] In one embodiment, step S2 includes: Step S2.1: Obtain the scoring-related parameters of each candidate node. The scoring-related parameters include net cache hit benefit, predicted future reuse probability, popularity of similar services or session families, memory pressure, decoding queue pressure, security risks, additional cross-node transmission costs, and reward items that meet the first token delay, TPOT, or tenant priority.

[0172] Step S2.2: Based on the preset weights, perform a weighted summation on each of the scoring-related parameters to obtain the comprehensive cache locality score of the inference request at the corresponding candidate node.

[0173] The beneficial technical effects achieved by steps S2.1 to S2.2 are: when selecting a target node, multiple factors such as cache hit benefit, memory pressure, queue congestion and first token delay can be comprehensively considered to ensure that the request is routed to the globally optimal computing node.

[0174] Optionally, in some embodiments, for the arriving inference request r The system first obtains its candidate Token-KVblock set B(r). For candidate node n, let... h b,n This indicates the reusable hit status of block b at node n. A higher value is taken if it is located in the local node's HBM or DRAM, a lower value is taken if it is located in a remote lower-level storage, and 0 if it does not exist. Taking into account the net cache hit benefit, the predicted future reuse probability, the popularity of similar services or session families, memory pressure, decoding queue pressure, security risks, additional costs of cross-node transmission, and rewards for meeting the first token latency, TPOT, or tenant priority, the cache locality comprehensive score Token KV locality score can be expressed as:

[0175] in, P future ( r，n This indicates the probability that the new Token-KV generated after the request is executed will be reused on this node in the future. H hot ( r，n This indicates that the node already has similar prefixes, business templates, or session families in terms of popularity. M press (n) represents the HBM memory pressure; Q decode (n) represents the decode queue pressure; R risk (r,n) represents a security risk or strategy conflict; C net (r,n) represents the additional cost of cross-node transmission; A SLO (r,n) represents the reward for satisfying the first token delay, TPOT, or tenant priority.

[0176] Memory pressure can be defined as:

[0177] In one embodiment, step S2.1 includes: Step S2.1.1: Determine the reusable hit status of the candidate KV cache block in the candidate node. The value of the reusable hit status is positively correlated with the storage level of the candidate KV cache block.

[0178] Step S2.1.2: Calculate the estimated recalculation cost of obtaining the candidate KV cache block and the transmission cost of transmitting the candidate KV cache block to the candidate node, and determine the difference between the estimated recalculation cost and the transmission cost.

[0179] Step S2.1.3: When a candidate KV cache block meets the matching condition, multiply the difference by the reusable hit status, and sum the product of all candidate KV cache blocks required by the request to obtain the net cache hit benefit of the candidate node.

[0180] The beneficial technical effects achieved by steps S2.1.1 to S2.1.3 are as follows: By directly introducing the difference between the estimated recalculation cost and the transmission cost into the calculation of the net hit benefit, the scheduling system can dynamically perceive the network physical bottleneck and only incentivize cross-node reuse when the transmission time is indeed less than the recalculation time.

[0181] In one embodiment, the candidate KV cache block is allowed to be reused when the key field of the candidate KV cache block matches the request attribute of the inference request. The matching criteria include at least one of the following: The model identifier, tokenizer identifier, plug-in parameter version, and location encoding scheme of the candidate KV cache block are completely consistent with the inference request; The precision format or recoverable precision of the candidate KV cache block meets the precision requirements of the inference request; The tenant scope or shared domain to which the candidate KV cache block belongs is within the tenant or shared scope that the inference request is authorized to access; The security label of the candidate KV cache block is not higher than the permission level of the inference request; The current time is earlier than the set expiration time of the candidate KV cache block.

[0182] The beneficial technical effects achieved by this embodiment are: by requiring that accuracy, model configuration, tenant permissions, security labels, and lifecycle all match, it can provide underlying security verification and correctness barriers for multi-tenant shared computing environments.

[0183] It is understood that in this embodiment, the system only allows reuse when the key fields of the candidate cache block meet the matching conditions:

[0184] in, b ≥ r This indicates that the cached precision or recoverable precision meets the request requirements. Indicates the tenant or shared scope that is requesting authorized access, r b ≤r r This indicates that the security label of the block is no higher than the requested permission level, t exp (b) indicates the expiration time. This matching condition explicitly incorporates security isolation and correctness conditions into the cache reuse process, rather than relying solely on the text hash.

[0185] It is understood that in this embodiment, security isolation can be achieved using tenant-level keys, session-level keys, trusted execution environments, DPU-side encryption and decryption, zero-trust access control, audit log on-chaining, or hardware security module management keys.

[0186] In one embodiment, the objective of configuring the preset weights is to minimize the overall cost of the system, which is derived from a statistical evaluation of historical routing decisions. Among them, the statistical indicators of historical routing decision results include at least one of the following: first word delay, word generation delay in the decoding stage, actual hit benefit, cache block loading time, recalculation time, GPU pause, service level target default status, and subsequent cache reuse count; The overall cost is obtained by multiplying the initial word delay, word generation delay, GPU resource overhead, network transmission overhead, memory overhead, and service level target default status by their respective preset conversion coefficients and then summing them.

[0187] The beneficial technical effects achieved by this embodiment are as follows: by retrospectively evaluating past statistical indicators such as latency, stuttering, hit rate, and SLO default rate to adjust the weight configuration, it can be ensured that the overall scheduling cost of the system can continuously converge to the optimal state as the running time progresses.

[0188] In other words, in this embodiment, the weights in the Token-KV locality score w 1 to w 8. This can be configured manually or updated by the strategy learning module based on historical feedback. The system can record the results of each routing decision, including the initial token delay, decode TPOT, actual hit benefit, loading time, recalculation time, GPU stall, whether the SLO is breached, and the number of subsequent cache reuses. The goal is to minimize the overall cost.

[0189] The system can adjust weights through online regression, contextual bandit, reinforcement learning, or Bayesian optimization. However, in a preferred stable implementation, even without complex learning algorithms, regularized weights and sliding window statistics can be used. Adaptive learning is an enhancement implementation method and does not affect the core technical solution of this invention.

[0190] Optionally, in some embodiments, the weights of the Token-KV locality score can be statically configured or adaptively adjusted by online learning, offline playback, reinforcement learning, contextual bandit, or business strategy engine.

[0191] In one embodiment, the tiered token caching resource pool includes at least two of the following tiers: L0, L1, L2, L3, L4, and L5; wherein, The L0 layer is a GPU HBM; The L1 layer is a cache that can be directly accessed by DPUs, smart network cards, or GPUs; The L2 layer is the CPU DRAM; The L3 layer is either NVMe or a local SSD; Layer 4 is a remote SSD, a remote memory pool, or a distributed cache; Layer L5 is for object storage, shared storage, or archive storage.

[0192] The beneficial technical effects achieved by this embodiment are: storage resources can be extended from GPU HBM to object archive storage, significantly increasing the overall capacity limit of the token cache and providing a hardware foundation for the hierarchical management of hot and cold data.

[0193] In other words, the KVLake in this embodiment is not a single cache pool, but a tiered resource system designed for different latency, bandwidth, capacity, and cost. The preferred tiers are as follows: Layer L0: GPUHBM. This layer has the lowest latency and highest bandwidth, making it suitable for hot token-KV blocks that are being decoded or about to be accessed. Layer L0 is typically managed directly by the inference engine, but in this embodiment, its state is exposed to the platform scheduler through an adaptation module.

[0194] Layer L1: DPUs, smart NICs, or GPUs can directly access the cache. This layer is suitable for storing hot data shared across GPUs or nodes, and can also handle verification, encryption / decryption, RDMA registration, and prefetch queue management.

[0195] L2 layer: CPU DRAM. This layer has a large capacity and is suitable for storing warm token-KV blocks that may be reused in the near future. It can interact with the GPU via PCIe, CXL, NVLink-C2C, or RDMA.

[0196] L3 layer: Local NVMe or local SSD. This layer is suitable for storing single-node session history, long document prefixes, and recently used low-frequency multiplexed blocks.

[0197] Layer 4: Remote SSDs, remote memory pools, or distributed caches. This layer is suitable for enterprise templates, common system prompts, and popular document snippets shared across multiple nodes.

[0198] Layer L5: Object storage, shared storage, or archive storage. This layer has a large capacity and low cost, making it suitable for cold token key-value blocks, compressed blocks, or blocks that require audit retention but are not suitable for occupying higher-level caches.

[0199] In addition, each level z can be represented by a triple ( C z ，B z ， z ) indicates that among them C z For capacity, B z For effective bandwidth, z For access latency. For a block b, if its size is |b|, the estimated cost of migrating from level z to the target level z′ is:

[0200] Where C decode (b) Cost of decompression or format conversion, C verify (b) Costs for verification, decryption, or authorization checks. These costs are used for routing, promotion, and demotion decisions.

[0201] In one embodiment, step S2 further includes: Step S2.3: Obtain a set of candidate nodes that meet the preset conditions, wherein the preset conditions include at least tenant isolation, model version and security policy.

[0202] Step S2.4: Determine the pre-filled queue pressure of each candidate node, and the additional switching cost required to switch inference requests from the current session sticky node to the candidate node.

[0203] Step S2.5: Calculate the final routing score for each candidate node based on the comprehensive score of cache locality of inference requests at each candidate node, the penalty item related to pre-filled queue pressure, and the penalty item related to additional switching costs.

[0204] Step S2.6: Select the candidate node corresponding to the maximum value of the final route score as the target node, and route the inference request to the target node.

[0205] The beneficial technical effects achieved by steps S2.3 to S2.6 are as follows: the introduction of tenant isolation hard conditions, pre-filled queue congestion penalties, and switching costs that break session stickiness during scheduling effectively prevents system crashes caused by request overload of a single node with high-frequency hot spot cache.

[0206] Optionally, in some embodiments, the final request route may be represented as:

[0207] in, N allow ( r Q is a set of candidate nodes that meet the requirements of tenant isolation, model version, and security policies. prefill ( n ) represents the prefill queue pressure, C switch ( r，n To switch the request from its session sticky node to the node n Additional costs.

[0208] The key to this scoring mechanism is that node selection is not based solely on idle GPUs or cache hits, but rather on comparing "the computing power saved by reusing Token-KV" with "the movement, security, and memory costs incurred in acquiring these Token-KV".

[0209] Optionally, in some embodiments, for non-prefix reuse scenarios, local recalculation, attention correction, position remapping, or block fusion mechanisms may be introduced, but the generation quality and security policy requirements must be met.

[0210] In one embodiment, step S4 includes: Step S4.1: During the process of the target node executing the inference request based on the KV cache block in the computing power tiered token cache resource pool, the value parameters of each KV cache block in the computing power tiered token cache resource pool are obtained in real time. The value parameters include at least the future reuse probability, recalculation cost, capacity cost of continued storage, expected movement cost, risk parameters, and strategy reward parameters.

[0211] Step S4.2: Calculate the corresponding lifetime value based on the value parameters of the KV cache block.

[0212] Step S4.3: Determine the initial level and lifecycle value trend of the KV cache block in the computing power tiered Token cache resource pool.

[0213] Step S4.4: Perform full lifecycle management on the corresponding KV cache blocks based on the changing trends of the initial level and lifecycle value.

[0214] The beneficial technical effects achieved by steps S4.1 to S4.4 are as follows: by calculating the comprehensive lifetime value of cache blocks under reuse probability, holding cost and migration cost in real time, the decisions on cache retention, degradation or eviction can be made more accurately and reliably.

[0215] Optionally, in some embodiments, lifecycle value It can be represented as:

[0216] Among them, P reuse (b) represents the probability of future reuse, C rec (b) indicates recalculated cost, C store (b) represents the capacity cost of continuing to store the block, C move (b) represents the expected movement cost, C risk (b) Indicates a security, compliance, or disclosure risk, A strategic (b) indicates policy rewards for high-value tenants, critical business templates, or high-priority sessions.

[0217] When V(b) remains positive in the higher-level cache and the block is expected to be accessed in the near future, the system performs promotion; when V(b) decreases but is still potentially reusable, the system performs degradation; when V(b) is negative and there are no compliance retention requirements, the system performs eviction. Promotion and degradation can specifically include the following actions: HBM Promotion: Copy the block being decoded or about to be decoded from DRAM / NVMe to the GPU HBM and establish a mapping that the inference engine can directly read.

[0218] DRAM Promotion: Move warm blocks on remote or local NVMe to CPU DRAM in advance as a near-end cache for the next batch of requests.

[0219] DPU-side placement: Place frequently reused blocks across nodes in the DPU or smart NIC-side cache to reduce CPU involvement and network protocol overhead.

[0220] NVMe Degradation: Degrade blocks that have not been accessed recently but may still be reused from HBM / DRAM to local NVMe and record the object index.

[0221] Remote archiving: Compress and encrypt cold blocks before writing them to shared storage or object storage, and free up high-cost cache space.

[0222] Policy eviction: Delete blocks that have expired, whose model version has become invalid, whose tenants have revoked their authorization, whose verification has failed, or whose value is below the threshold.

[0223] Regarding the elimination strategy, this embodiment does not employ a simple LRU (Least Recently Used) approach, but rather a benefit-aware elimination approach. For the candidate elimination set E, the block that minimizes the total value loss and satisfies the release space requirement is preferentially eliminated:

[0224] Where R free This represents the space that needs to be freed. This formula can be implemented using a greedy algorithm, a quantile queue, a multilevel heap, or an approximate knapsack algorithm.

[0225] Optionally, in some embodiments, the promotion, demotion, and elimination strategies may employ reward-aware greedy algorithms, multi-queue algorithms, hierarchical LFU, LRU with TTL, approximate knapsack algorithms, minimum SLO default cost algorithms, or multi-objective optimization algorithms.

[0226] In one embodiment, step S4 further includes: Step S4.5: Obtain the prefetch-related parameters of the inference request during the decoding stage. The prefetch-related parameters include the current decoding speed, the loading time of the KV cache block, the safety jitter margin, and the size of the KV cache block. The loading time is the time required to prefetch the KV cache block from the current cache level to the target level.

[0227] Step S4.6: Calculate the pre-fetched wavefront distance based on the pre-fetched relevant parameters.

[0228] Step S4.7: Determine the interval between the expected consumption position and the current consumption position of the KV cache block.

[0229] Step S4.8: When the interval does not exceed the prefetch wavefront distance, prefetch the KV buffer block.

[0230] The beneficial technical effects achieved by steps S4.5 to S4.8 are as follows: by calculating the prefetch wavefront distance based on the real-time decoding speed of the model and the network loading time, it is possible to achieve asynchronous preloading of historical KV cache blocks on demand and at specific points.

[0231] Optionally, in some embodiments, during the decoding phase, the request generates a token at a certain speed. Suppose the request... r The current decoding speed is v r The unit is tokens / second; the time required to prefetch a block from level z to the target level is... T load (b,z); the safety margin for jitter is... δThe block size is B. The system can calculate the pre-fetched wavefront distance:

[0232] When the expected consumption position of a block is no more than the current consumption position D wave When a block is reached, the system initiates a prefetch. If the request has a high SLO level, the delta value can be increased or the prefetch priority can be raised; if the HBM pressure is high, the prefetch window can be reduced to avoid prematurely occupying video memory.

[0233] For multiple concurrent requests, the prefetch module sorts them by "earliest consumption time" and "delay cost". Let the estimated consumption time of blockb be t. need (b), the loading time is t ready (b) Then its stall risk can be defined as:

[0234] in π r Priority is given to requests or the cost of defaulting on SLOs. The prefetch scheduler prioritizes these requests. R stall High priority blocks, while avoiding consuming network bandwidth with low priority prefetching.

[0235] It is understood that the token-wavefront prefetching in this embodiment also supports hierarchical pipelines. For example, a block is first prefetched from object storage to a remote SSD, then from the remote SSD to the local NVMe, then from the local NVMe to DRAM, and finally enters HBM just before consumption. The system calculates multiple arrival times based on the bandwidth and latency of each level to form a pipeline schedule, rather than waiting for the GPU to request data before reading synchronously.

[0236] Optionally, in some embodiments, token-wavefront prefetching may be based on token speed, or on attention kernel consumption events, CUDA events, inference engine scheduling ticks, network completion queues, or storage I / O completion events.

[0237] In one embodiment, step S4.8 includes at least one of the following: Step S4.8.1: When the target service level of the inference request is greater than the preset level threshold, increase the security jitter margin or increase the prefetch priority.

[0238] Step S4.8.2: When the high-bandwidth memory pressure at the target level exceeds the preset pressure threshold, reduce the prefetch window.

[0239] The beneficial technical effects achieved by steps S4.8.1 to S4.8.2 are: actively widening the prefetch margin when a high-guarantee SLO request arrives, and promptly reducing the prefetch window when the GPU memory is under pressure, thereby enhancing the robustness of the inference engine in extreme concurrency scenarios.

[0240] In one embodiment, step S4 further includes: Step S4.9: Determine the set of KV cache blocks for the inference request, and divide the set of KV cache blocks into a first set that is hit and reusable, and a second set that is missed or not reusable.

[0241] Step S4.10: Re-perform pre-filling calculation on the missing parts of the KV cache block set, and concatenate the hit KV cache blocks in the first set with the KV cache blocks obtained by re-pre-filling calculation according to their positions to construct a complete KV context.

[0242] Step S4.11: Determine the prefix consistency check results of all hit KV cache blocks in the first set.

[0243] Step S4.12: When the prefix consistency check result is passed, determine the recalculation cost of the hit KV cache block and the transmission cost of obtaining the hit KV cache block across levels.

[0244] Step S4.13: If the transmission cost is higher than the recalculation cost, then the corresponding hit KV cache block is determined to have a negative contribution to the effective hit rate, and the hit KV cache block is recalculated.

[0245] The beneficial technical effects achieved by steps S4.9 to S4.13 are as follows: In the process of splicing the local hit cache and the new calculation cache, if a sudden network congestion causes the transmission cost of some caches to exceed the recalculation cost, the system can stop the loss in time and perform on-site recalculation of the cache that should have been pulled, thereby ensuring the optimal solution for the overall inference latency.

[0246] In other words, in this embodiment, not all blocks will be hit for a new request. Let the set of requested blocks be B(r), and the set of hit and reusable blocks be the first set B. hit (r), the set of missing or unreusable items is the second set B. miss If (r), the system will only perform the necessary prefill for the missing parts and concatenate the hit block with the newly computed block according to their positions to form a complete KV context.

[0247] Because Transformer causal attention and positional encoding are sensitive to token order, partial hits should satisfy prefix consistency. For strict prefix reuse, the system requires all hit blocks to be constructed from 1 to 1. gThe continuous prefix. For RAG multi-fragment or non-prefix multiplexing scenarios, the system may introduce correction calculations or local recalculations in alternative implementations, but the independent claims preferably focus on prefix and session history multiplexing to improve protection stability.

[0248] The system can define an effective hit rate as:

[0249] This metric differs from the standard hit rate. The standard hit rate only calculates the number of blocks or tokens that are hit, while the effective hit rate calculates the actual computational cost saved after reuse. If some remote blocks are hit but the transmission cost exceeds the recalculation cost, their contribution to the Heeff can be negative, thus avoiding blind reuse.

[0250] In one embodiment, step S4 further includes: Step S4.14: Identify the hit and reusable prefix data, and directly reuse the corresponding KV cache block.

[0251] Step S4.15: Monitor the access progress during the decoding phase and downgrade KV cache blocks that are not in the current decoding wavefront to a lower-level cache medium, wherein the lower-level cache medium includes dynamic random access memory or non-volatile memory.

[0252] Step S4.16: Promote KV cache blocks that are about to be accessed in a short period of time to the higher-level cache of the computing power tiered Token cache resource pool.

[0253] Step S4.17: Perform compression or eviction operations on KV cache blocks whose lifetime value is lower than a preset value threshold.

[0254] Step S4.18: Perform cache-aware routing on inference requests so that inference requests with the same prefix share the higher-level cache.

[0255] The beneficial technical effects achieved by steps S4.14 to S4.18 are as follows: the operation specifications for prefix reuse, wavefront outer data degradation, wavefront inner data promotion, and low-value data cleaning are clarified, and routing intervention is used to enable same-source requests to share high-level high-speed video memory, thereby achieving full utilization of runtime resources.

[0256] In other words, in this embodiment, without a Token-KV resource pool, the number of concurrent requests a single GPU can handle is limited by model weights, workspace, activation, KV cache, and fragmentation. Roughly speaking:

[0257] in M HBMThis refers to the total amount of GPU video memory. M model For model weight allocation, M workspace For runtime workspace, M reserve For safety reasons, This represents the average active key-value occupancy per request.

[0258] Reduce in the following ways Alternatively, improve its effective utilization: avoid regenerating reusable prefixes; downgrade blocks not currently decoded to DRAM / NVMe; promote blocks that will be accessed soon to HBM; compress or evict low-value blocks; implement cache-aware routing for requests, allowing requests with the same prefix to share higher-level caches. If resource pooling results in an average HBM active key-value occupancy per request... Then the concurrency boosting factor can be approximated as:

[0259] This formula illustrates that the concurrency improvement of this solution comes not only from faster prefill, but also from retaining only the truly hot and soon-to-be-used Token-KV in the higher-level video memory, enabling more requests to enter the same decode batch.

[0260] In one embodiment, security metadata is bound to the KV cache block, and security checks are performed during lookup, routing, migration, reuse, degradation, archiving, and eviction. Among them, the security mechanisms used for security checks include at least: tenant scope binding, security label binding, key isolation, integrity verification, failure policy, and audit trail.

[0261] The beneficial technical effects achieved in this embodiment are as follows: by deeply binding security metadata with KV cache blocks and forcibly performing security and integrity checks at every lifecycle node (lookup, migration, degradation, etc.) during its flow, a defense-in-depth system that prevents tampering and leakage is constructed.

[0262] In other words, in this embodiment, the Token-KV block may contain intermediate representations of user input, enterprise documents, and session history projected onto the model. Although it is not the original text, it can still be considered sensitive derived data. This embodiment binds security metadata to each Token-KV block and performs security checks during lookup, routing, migration, reuse, degradation, archiving, and eviction processes.

[0263] Preferred security mechanisms include: Tenant-scoped binding: The shared scope of a block can be a single request, a single session, a single user, a single tenant, a tenant group, a public template, or a platform-level common template. By default, blocks for private requests are not allowed to be reused across tenants.

[0264] Security label binding: For contexts containing personal information, corporate secrets, code repositories, financial data, or regulated content, a higher security label is set, and the scheduler only searches and routes among candidate nodes that meet the permission level.

[0265] Key isolation: When a block leaves HBM or a process boundary, it can be encrypted using a tenant key, session key, or storage tier key, and the key version is written to the metadata fingerprint.

[0266] Integrity verification: Before block migration, decompression, and import into the inference engine, checksum, MAC, or signature verification is performed to prevent erroneous or tampered data from entering attention calculations.

[0267] Invalidation policy: When the model version, Adapter version, tenant authorization, session validity period, or compliance retention policy changes, the relevant block will automatically become invalid and will no longer be reused.

[0268] Audit trail: Records every block lookup, hit, rejection, migration, reuse, and deletion event for easy security auditing and fault tracing.

[0269] Through the above mechanism, this embodiment embeds the security boundary into the Token-KV resource pool, rather than relying on coarse-grained ACLs from the storage system after the fact. This avoids the same text template being mistakenly treated as the same cached object under different tenants, and also prevents shared prefix blocks from being indirectly read by unauthorized requests.

[0270] In one embodiment, the metadata fingerprint includes a version vector, which is determined based on at least one of model weights, a tokenizer, a positional encoding, an adapter, quantization parameters, and an attention kernel.

[0271] The beneficial technical effect achieved by this embodiment is that by incorporating quantization parameters and underlying operator (AttentionKernel) details into the version vector, silent calculation errors in cross-request reuse caused by subtle kernel implementation differences or precision loss are effectively prevented.

[0272] In one embodiment, step S3 includes: Step S3.1: Based on the metadata fingerprint, determine the matching result between the KV cache block of the inference request and the KV cache block in the computing power tiered Token cache resource pool.

[0273] Step S3.2: Based on the matching results, determine the KV cache blocks that are not stored in the target node's computing power hierarchical Token cache resource pool.

[0274] Step S3.3: Determine the net computing power benefit of the unstored KV cache blocks.

[0275] Step S3.4: Based on the net computing power revenue, store the unstored KV cache blocks in the computing power tiered Token cache resource pool in a tiered manner.

[0276] The beneficial technical effects achieved by steps S3.1 to S3.4 are as follows: by evaluating the net computing power benefit of the newly calculated KV cache blocks that failed to hit, it is ensured that only data with real future reuse value will be accepted and stored in the appropriate resource pool level, thus avoiding the pollution of the system storage pool by invalid data.

[0277] In other words, the KV cache in this embodiment is strongly correlated with the model version. Even if the prompt is exactly the same, the old KV cache may become unusable if the model weights, tokenizer, position encoding, adapter, quantization parameters, or the semantics of the attention kernel change. Therefore, this disclosure introduces a version vector into the metadata fingerprint:

[0278] When requesting version vector v r With block version vector v b Compatibility relationship not satisfied v b v r When this happens, the system refuses to reuse the block. Compatibility can be complete equality or a platform-predeclared compatibility mapping. For example, some precision formats can be downgraded from high precision to low precision, and some adapters can declare inheritance of common prefixes, but such compatibility must be explicitly approved by the policy service. Failure control includes time-based failure, version failure, authorization failure, data quality failure, and capacity policy failure. For blocks that experience validation errors, abnormal output associations, or security policy changes, the system can set their status to quarantine. Blocks in quarantine do not participate in new request reuse until validation is completed or they are deleted. Optionally, in one embodiment, cache-aware request routing includes the following steps: 1. The inference gateway receives requests and parses the tenant, model, tokenizer, adapter, SLO, context length, session identifier, and security label.

[0279] 2. The request parsing and token segmentation module generates a prefix block plan and calculates the logical fingerprint of each block.

[0280] 3. The KV Lake metadata indexing module returns a candidate hit list, which includes the block's level, node, GPU, object key, compression format, remaining validity period, and security license status.

[0281] 4. The cache-aware request routing module calculates the Token-KV locality score for each candidate node and filters out nodes that do not meet the tenant isolation or model consistency conditions.

[0282] 5. If the best node already has enough HBM / DRAM hits, the request is routed directly to that node; if the best node only has hits in the lower-level cache, the system determines whether prefetching or migration is better than recalculation.

[0283] 6. The inference engine adaptation module imports a reusable Token-KV block before prefilling and only performs incremental prefilling on the missing or non-reusable parts.

[0284] 7. New Token-KV blocks generated during the request execution process are registered in KV Lake, and policies determine whether to retain, compress, encrypt, share, or make them available only within the session.

[0285] Through the above process, when the same system prompt, enterprise template, or long document prefix is ​​reused by multiple requests, the system will naturally route subsequent requests to nodes that already have Token-KV, thereby reducing redundant calculations. Even if a node's GPU load is not the lowest, as long as its caching benefits are sufficient to offset queuing costs, the system will still prioritize that node; conversely, if a node has a cache but its transmission and GPU memory pressure is too high, the system can choose to recalculate or select a suboptimal cache node.

[0286] Understandably, to ensure the engineering feasibility of this disclosed technical solution, the inference engine adaptation module can provide the following interfaces: Register KV: After completing the prefill, the inference engine registers the generated Token-KV block and its physical address, size, layer number, fragment, precision, and checksum into KVLake.

[0287] Lookup KV: Before routing, the inference gateway or scheduler queries the list of reusable blocks based on the request fingerprint.

[0288] Import KV: Before execution, the inference engine imports the hit block from HBM, DRAM, NVMe, or remote storage into the current execution context.

[0289] Pin KV: Set a short-term non-removal flag for blocks that are about to enter decoding or have high SLO requirements.

[0290] Move KV: Moves blocks between levels and updates the metadata index synchronously.

[0291] Evict KV: Deletes or downgrades blocks based on lifecycle policies.

[0292] Verify KV: Checks version, tenant, security label, checksum, and expiration time before importing.

[0293] The aforementioned interface can be implemented as an inference engine plugin, a sidecar process, a DPU agent, or a cloud platform runtime extension. The focus of this invention is on the token-KV resource pooling and scheduling logic behind the interface, rather than relying on any specific open-source framework.

[0294] Based on the above embodiments, this disclosure can achieve at least the following technical effects: (1) Reduce redundant prefills. For repetitive content such as system prompts, long document prefixes, enterprise templates, code repository contexts, and session history, the system can directly reuse the already generated Token-KV blocks to avoid consuming GPU computing power again for each request.

[0295] (2) Improve the response speed of the first token. After a reusable token-KV is hit, the request only needs to perform incremental prefill on the missing part, which significantly reduces the TTFT, especially suitable for long contexts and multi-turn sessions.

[0296] (3) Increase the concurrency limit. By keeping hot blocks in the high-level cache, degrading warm and cold blocks, and allowing requests with the same prefix to share the cache, the average active HBM usage per request can be reduced, allowing more requests to be executed on the same GPU or in the same cluster.

[0297] (4) Avoid incorrect scheduling. The locality score of this disclosed technical solution takes into account both hit benefits and transmission costs. It will not blindly fetch remotely just because "there is a cache somewhere", nor will it ignore the computing power saved by reusable KV just because "a GPU is idle".

[0298] (5) Reduce GPU stall. Token-wavefront prefetching allows subsequent token-KV blocks in the lower-level cache to reach the higher-level cache before decoding and consumption, reducing GPU wait times for network or storage responses.

[0299] (6) Supports multi-tenant security isolation. By binding the tenant scope, security label, expiration time and key version to each Token-KV block, enterprise customer data isolation can be guaranteed while cache reuse.

[0300] (7) Develop computing power asset management capabilities. The platform can measure the prefill FLOPs saved, the token cost saved, and the cache resources occupied by each Token-KV block, thereby transforming the KV cache from an engineering optimization object into an operable computing power asset.

[0301] (8) Compatible with existing inference frameworks. This disclosure can be implemented through plugins, sidecars, or control planes, and can be used in conjunction with technologies such as PagedAttention, continuous batch processing, prefix caching, PD disaggregation, CPU offload, and quantized inference.

[0302] For example, in a large-scale enterprise customer service model, each request contains a fixed system prompt and business rule template with approximately 4096 tokens, while the actual user question only has an average of 200 tokens. Traditional inference systems re-execute the complete prefill every time a request arrives, resulting in a significant waste of GPU computing power on repetitive templates.

[0303] When the system first processes the enterprise template using the disclosed technical solution, it divides it into multiple Token-KV blocks based on B=128 tokens, generates a metadata fingerprint containing the enterprise tenant identifier, model version, tokenizer, adapter, location code, and security tag, and registers the generated KV blocks in KVLake. Since this template is a single-tenant, high-frequency private template, the policy module sets its future reuse probability to a high level and promotes the most recently accessed high-frequency blocks to the DRAM of multiple inference nodes and the HBM of the target node.

[0304] When subsequent requests from the same enterprise arrive, the inference gateway first calculates their prefix fingerprint and checks for existing hits. The scheduler finds that node A already has a majority of blocks in its HBM, node B has a shorter queue but no cache, and node C has a remote NVMe hit but high transmission latency. After calculating the locality score, node A has the highest net benefit, and the request is routed to node A. The inference engine only performs incremental prefill on the part of the user's actual question and reuses the imported systemprompt token-KV during the decode phase. As a result, the initial token latency and prefill GPU usage are significantly reduced.

[0305] If the enterprise template is upgraded, or the model or adapter version changes, the version vector of the old block will no longer be compatible, and new requests will not incorrectly reuse the old block. The old block will be downgraded or phased out according to the strategy, and the new template block will be re-registered after its first generation and gradually promoted.

[0306] For example, in long document Q&A scenarios, users upload the same contract, prospectus, or technical manual and then ask multiple questions consecutively. The document length could reach 64,000 tokens. If the entire document were refilled for each question, it would cause severe TTFT latency and HBM usage.

[0307] When adopting the technical solution disclosed herein, the system generates a Token-KV block for each document segment during the initial document processing. The first few high-frequency segments are retained in DRAM, the complete document KV is stored locally on NVMe, and cold segments are compressed and written to remote shared storage. When subsequent issues arise, the scheduler searches for reusable blocks based on the document fingerprint, session identifier, and tenant security tag. For matched consecutive prefixes, the system imports them directly; for non-consecutive segments or segments requiring correction, recalculation or partial correction can be selected according to the implementation strategy.

[0308] Once the request enters the decoding phase, the system monitors the current decoding speed as 40 tokens per second, the average loading time from local NVMe to DRAM to HBM as 80 milliseconds, the safety margin as 30 milliseconds, and the block size as 128 tokens. The prefetch wavefront distance is:

[0309] The system therefore initiates prefetching at least one block in advance. If NVMe queue congestion is detected or the SLO request is stricter, the safety margin is increased, and prefetching is initiated even more blocks in advance. In this way, GPU attention can typically obtain data from HBM or DRAM when reading historical key-value pairs, instead of synchronously waiting for lower-level storage.

[0310] For example, in a code assistant scenario, numerous requests reference the README, interface definitions, common utility functions, and dependency configurations of the same code repository. Different developers have different questions, but the repository context is highly repetitive. The system generates a Token-KV block from the repository-level context and sets its sharing scope to the same code repository project within the same enterprise tenant. Due to the sensitivity of the code data, the block's security label is set to "Enterprise Code Private," and it is encrypted and stored in the DRAM / NVMe layer using the tenant key.

[0311] When a developer initiates a request for code completion, unit test generation, or error explanation, the scheduler first checks the token-key-value match status of the repository context. If a node has recently served the same repository, it is likely that there are already hot blocks in its HBM / DRAM. Even if the current GPU utilization of this node is slightly higher than other nodes, as long as the benefit of cache reuse exceeds the queuing cost, the system can still route the request to this node. If another node is about to handle more requests for the same repository, the system can copy the relevant DRAM blocks to this node during off-peak hours to achieve proactive preheating.

[0312] When an enterprise revokes a developer's permissions or rotates code repository keys, the security isolation and auditing module immediately invalidates the relevant blocks for that user or the old key version. Invalidated blocks are no longer reused, preventing code context leaks.

[0313] For example, some requests contain both platform-wide templates and tenant-specific contexts. For instance, platform-unified tool call instructions and output format constraints can be shared across multiple tenants, while tenant business knowledge bases, customer data, and session history cannot be shared across tenants. This disclosed technical solution allows different Token-KV blocks for the same request to have different sharing scopes.

[0314] The system sets platform-level public security labels and public key versions for public template blocks, and sets single-tenant sharing scope for tenant private blocks. During routing, public blocks can be reused among multiple tenants, while private blocks can only be reused within authorized tenants. The Locality score calculates the revenue for public and private blocks separately, and ultimately selects the node that matches both the public template and the private context.

[0315] This block-based security strategy is more computationally efficient than "keeping the entire request private" and more secure than "sharing only by text hash". Its key lies in the tenant scope and security label binding at the token-KV block granularity.

[0316] For example, in a high-performance inference cluster, multiple GPU nodes are connected via a high-speed network. For a frequently used system prompt shared by multiple nodes, loading it from remote storage to the CPU and then copying it to the GPU each time would consume CPU and PCIe resources. In this embodiment, the platform sets up an L1Token-KV cache on the DPU or smart NIC side, and the DPU agent performs RDMA registration, data verification, tenant label checking, and prefetch queue management.

[0317] When a request is routed to a GPU node, if the required block is located on an adjacent node or in the DPU-side cache, the DPU agent can directly transfer the block to the target GPU's accessible region via registered memory or GPU Direct. Metadata indexes record the block's DPU location and verification information. The scheduler treats DPU-side hits as a higher tier than remote SSDs but slightly lower than local HBMs. This significantly reduces the cost of sharing key-value pairs across nodes, decreases CPU intervention, and makes prefetching easier to overlap with decode computations.

[0318] For example, for daily report analysis, customer service peaks, or batch code review tasks that occur at fixed times, the system can preheat the Token-KV block based on historical reuse probabilities. The strategy learning module discovered that a certain type of enterprise template experiences a significant increase in access volume between 9:00 and 10:00 AM on weekdays, and that each reuse can save a large amount of prefill. Therefore, before the peak, the system promotes the Token-KV block of this template from object storage to a remote SSD, then to DRAM in the target region, and finally places the first few blocks into the HBM or DPU-side cache before the predicted requests arrive.

[0319] Active warm-up is not simply copying popular data; instead, it selects the blocks and nodes with the highest expected net benefit based on V(b) and Sloc calculations. If the HBM level exceeds the threshold during warm-up, the system only warms up to the DRAM layer, waiting for real requests to arrive before performing HBM promotion. This reduces peak TTFT and avoids prolonged occupation of expensive video memory when there are no requests.

[0320] For example, when a GPU node fails, its Token-KV block within its HBM may be lost. Since the KVLake metadata index also records the location of the block's replica in DRAM, NVMe, or remote storage, the system can route new requests to other nodes with available replicas or rebuild the necessary block on a replacement node. If the block only has a replica in the failed node's HBM and no persistent replica, the system marks it as missing and refills the corresponding prefix.

[0321] For high-value templates, the strategy module can maintain multiple tiers of replicas to reduce the impact of node failures on hit rates. The number of replicas can be determined by future reuse probability, security level, and recovery costs, rather than being statically configured.

[0322] Please see Figure 2 , Figure 2 This is a structural diagram of a device for accelerating inference through a hierarchical token caching resource pool in an intelligent computing cloud platform provided by the present invention. Figure 2 As shown, the device 200 for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform includes: The processing module 201 is used to divide the Token-KVCache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks and obtain the metadata fingerprint of the KV cache blocks. The first determining module 202 is used to determine the cache locality comprehensive score of the inference request in the candidate node based on the KV cache block, and to determine the target node of the inference request based on the cache locality comprehensive score. The second determining module 203 is used to determine, based on metadata fingerprint, reusable historical KV cache blocks and uncached KV cache blocks in the computing power hierarchical Token cache resource pool of the target node, and store the uncached KV cache blocks hierarchically in the computing power hierarchical Token cache resource pool. The management module 204 is used to execute inference requests on multiple KV cache blocks cached in the target node's computing power-tiered Token cache resource pool, and to perform full lifecycle management of the KV cache blocks during the execution process.

[0323] In one embodiment, the processing module 201 includes: The first processing unit is used to divide the input word sequence of the inference request into multiple logical word intervals; The first acquisition unit is used to acquire the KV cache blocks of each logical term interval under each Transformer layer of the large model.

[0324] In one embodiment, the first acquisition unit 2012 is specifically used for: Obtain the large model l The input hidden state, key projection parameters, and value projection parameters of each Transformer layer; According to the large model l The input hidden states, key projection parameters, and value projection parameters of the Transformer layer are used to calculate the largest model's... l The complete key tensor and complete value tensor of the Transformer layer; Based on the logical word interval of the large model l Tensor slicing is performed on the complete key tensor and complete value tensor of each Transformer layer to obtain the corresponding KV cache block.

[0325] In one embodiment, the processing module 201 further includes: The first determination is used to determine the prefix hash chain of the i-th KV cache block, wherein the order of different KV cache blocks is determined based on the logical order of the KV cache blocks in the inference request; The second acquisition form is used to acquire auxiliary parameters related to the fingerprint, wherein the auxiliary parameters include at least one of the following: precision format, tenant range, shared domain, security label, key version, signature policy, and integrity verification policy; The fingerprint construction unit is used to construct the metadata fingerprint corresponding to the i-th KV cache block based on the prefix hash chain and auxiliary parameters of the i-th KV cache block.

[0326] In one embodiment, the first determining unit is specifically used for: Obtain the environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block. The environment configuration parameters include at least one of the following: model identifier, model version, tokenizer identifier, vocabulary version, LoRA version, Adapter version, plugin parameter version, cache format version, and position encoding scheme and its parameters. The prefix hash chain of the i-th KV cache block is determined based on the environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block.

[0327] In one embodiment, the processing module 201 further includes: The second determining unit is used to determine the amount of pre-filling computation avoided when reusing KV cache blocks, and to convert the amount of pre-filling computation avoided into basic computing power revenue based on a preset computing power value conversion coefficient. The third determining unit is used to determine the cost parameters generated by the KV cache block occupying computing power in each cache level of the tiered token cache resource pool. The cost parameters include: cache occupation cost, cross-node or cross-level network transmission cost, and additional security management costs for encryption, verification, auditing and access control. The fourth determining unit is used to deduct cache occupation costs, network transmission costs, and additional security management costs from the basic computing power revenue to calculate the net computing power revenue of the KV cache block. The net computing power revenue is used to determine the storage level of the corresponding KV cache block in the computing power tiered Token cache resource pool.

[0328] In one embodiment, the processing module 201 further includes: The fifth determining unit is used to determine the actual number of output tokens of the large model, as well as the equivalent number of tokens saved by avoiding redundant pre-computation through KV cache reuse; The first calculation unit is used to convert the saved equivalent word units into effective output value according to the preset conversion weight, and add the effective output value to the actual output word units to obtain the comprehensive output benefit. The second calculation unit is used to calculate the total resource cost based on the computing resource cost, storage resource cost, and network resource cost during system operation. The sixth determining unit is used to use the ratio of comprehensive output revenue to total resource cost as the effective word element output efficiency.

[0329] In one embodiment, the first determining module 202 includes: The third acquisition unit is used to acquire the scoring-related parameters of each candidate node. The scoring-related parameters include net cache hit benefit, predicted future reuse probability, popularity of similar services or session families, memory pressure, decoding queue pressure, security risks, additional cross-node transmission costs, and reward items that meet the first token delay, TPOT, or tenant priority. The third calculation unit is used to perform a weighted summation of each item in the scoring-related parameters based on preset weights to obtain the comprehensive cache locality score of the inference request at the corresponding candidate node.

[0330] In one embodiment, the third acquisition unit is specifically used for: Determine the reusable hit status of candidate KV cache blocks in candidate nodes, where the value of the reusable hit status is positively correlated with the storage level of the candidate KV cache block; Calculate the estimated recalculation cost of acquiring candidate KV cache blocks and the transmission cost of transmitting candidate KV cache blocks to candidate nodes, and determine the difference between the estimated recalculation cost and the transmission cost. When a candidate KV cache block meets the matching condition, the difference is multiplied by the reusable hit status, and the product of all candidate KV cache blocks required by the request is accumulated to obtain the net cache hit benefit of the candidate node.

[0331] In one embodiment, the candidate KV cache block is allowed to be reused when the key field of the candidate KV cache block matches the request attribute of the inference request. The matching criteria include at least one of the following: The model identifier, tokenizer identifier, plug-in parameter version, and location encoding scheme of the candidate KV cache block are completely consistent with the inference request; The precision format or recoverable precision of the candidate KV cache block meets the precision requirements of the inference request; The tenant scope or shared domain to which the candidate KV cache block belongs is within the tenant or shared scope that the inference request is authorized to access; The security label of the candidate KV cache block is not higher than the permission level of the inference request; The current time is earlier than the set expiration time of the candidate KV cache block.

[0332] In one embodiment, the objective of configuring the preset weights is to minimize the overall cost of the system, which is derived from a statistical evaluation of historical routing decisions. Among them, the statistical indicators of historical routing decision results include at least one of the following: first word delay, word generation delay in the decoding stage, actual hit benefit, cache block loading time, recalculation time, GPU pause, service level target default status, and subsequent cache reuse count; The overall cost is obtained by multiplying the initial word delay, word generation delay, GPU resource overhead, network transmission overhead, memory overhead, and service level target default status by their respective preset conversion coefficients and then summing them.

[0333] In one embodiment, the tiered token caching resource pool includes at least two of the following tiers: L0, L1, L2, L3, L4, and L5; wherein, The L0 layer is a GPU HBM; The L1 layer is a cache that can be directly accessed by DPUs, smart network cards, or GPUs; The L2 layer is the CPU DRAM; The L3 layer is either NVMe or a local SSD; Layer 4 is a remote SSD, a remote memory pool, or a distributed cache; Layer L5 is for object storage, shared storage, or archive storage.

[0334] In one embodiment, the first determining module 202 further includes: The fourth acquisition unit is used to acquire a set of candidate nodes that meet preset conditions, wherein the preset conditions include at least tenant isolation, model version and security policy; The seventh determining unit is used to determine the pre-filled queue pressure of each candidate node, as well as the additional switching cost required to switch the inference request from the current session sticky node to the candidate node; The fourth calculation unit is used to calculate the final routing score of each candidate node based on the comprehensive score of cache locality of inference requests at each candidate node, the penalty item related to pre-filled queue pressure, and the penalty item related to additional switching costs. The routing unit is used to select the candidate node corresponding to the maximum value of the final routing score as the target node and route the inference request to the target node.

[0335] In one embodiment, the management module 204 includes: The fifth acquisition unit is used to acquire the value parameters of each KV cache block in the computing power tiered token cache resource pool in real time during the process of the target node executing an inference request based on the KV cache block in the computing power tiered token cache resource pool. The value parameters include at least the future reuse probability, recalculation cost, capacity cost of continued storage, expected movement cost, risk parameters, and strategy reward parameters. The fifth calculation unit is used to calculate the corresponding lifetime value based on the value parameters of the KV cache block; The eighth determining unit is used to determine the initial level and lifecycle value change trend of the KV cache block in the computing power tiered token cache resource pool; The full lifecycle management unit is used to manage the corresponding KV cache blocks throughout their entire lifecycle based on the initial level and the changing trend of their lifecycle value.

[0336] In one embodiment, the management module 204 further includes: The sixth acquisition unit is used to acquire the prefetch related parameters of the inference request during the decoding stage. The prefetch related parameters include the current decoding speed, the loading time of the KV cache block, the safety jitter margin, and the size of the KV cache block. The loading time is the time required to prefetch the KV cache block from the current cache level to the target level. The sixth calculation unit is used to calculate the pre-fetched wavefront distance based on the pre-fetched relevant parameters; The ninth determining unit is used to determine the interval between the expected consumption position and the current consumption position of the KV cache block; The prefetch unit is used to prefetch the KV buffer block when the interval does not exceed the prefetch wavefront distance.

[0337] In one embodiment, the prefetch unit is configured to perform at least one of the following steps: When the target service level of an inference request is greater than the preset level threshold, increase the security jitter margin or increase the prefetch priority. When the high-bandwidth memory pressure at the target level exceeds the preset pressure threshold, the prefetch window is reduced.

[0338] In one embodiment, the management module 204 further includes: The tenth determining unit is used to determine the set of KV cache blocks for the inference request, and divide the set of KV cache blocks into a first set that is hit and reusable, and a second set that is hit or not reusable; The seventh computation unit is used to perform necessary pre-filling computations on the second set and to concatenate the hit KV cache blocks in the first set with the newly computed KV cache blocks in the second set according to their positions to construct a complete KV context. The eleventh determining unit is used to determine the prefix consistency check results of all hit KV cache blocks in the first set; The twelfth determining unit is used to determine the recalculation cost of the hit KV cache block and the transmission cost of obtaining the hit KV cache block across levels when the prefix consistency check result is passed. The recalculation unit is used to determine that the contribution of the corresponding hit KV cache block to the effective hit rate is negative if the transmission cost is higher than the recalculation cost, and to perform recalculation on the hit KV cache block.

[0339] In one embodiment, the management module 204 further includes: The identification unit is used to identify the hit and reusable prefix data and directly reuse the corresponding KV cache block; The degradation processing unit is used to monitor the access progress during the decoding stage and degrade KV cache blocks that are not in the current decoding wavefront to a lower-level cache medium, wherein the lower-level cache medium includes dynamic random access memory or non-volatile memory. The promotion processing unit is used to promote KV cache blocks that will be accessed in a short period of time to the higher-level cache of the computing power tiered token cache resource pool; The compression and eviction processing unit is used to perform compression or eviction operations on KV cache blocks whose lifetime value is lower than a preset value threshold. The cache-aware routing unit is used to perform cache-aware routing on inference requests, enabling inference requests with the same prefix to share the higher-level cache.

[0340] In one embodiment, security metadata is bound to the KV cache block, and security checks are performed during lookup, routing, migration, reuse, degradation, archiving, and eviction. Among them, the security mechanisms used for security checks include at least: tenant scope binding, security label binding, key isolation, integrity verification, failure policy, and audit trail.

[0341] In one embodiment, the metadata fingerprint includes a version vector, which is determined based on at least one of model weights, a tokenizer, a positional encoding, an adapter, quantization parameters, and an attention kernel.

[0342] In one embodiment, the second determining module 203 includes: The thirteenth determining unit is used to determine the matching result between the KV cache block of the inference request and the KV cache block in the computing power tiered Token cache resource pool based on the metadata fingerprint. The fourteenth determining unit is used to determine the KV cache blocks not stored in the target node's computing power tiered Token cache resource pool based on the matching results; The fifteenth determination unit is used to determine the net computing power benefit of the unstored KV cache blocks; Storage units are used to store unstored KV cache blocks in a hierarchical manner within the computing power tiered token cache resource pool, based on net computing power revenue.

[0343] In this way, by introducing metadata fingerprints and cache-aware routing, the KV Cache is transformed from a temporary state within a single machine into a shared resource at the cloud platform level, thereby enabling cross-request and cross-session reuse of prefix computation results across multiple nodes, effectively reducing the waste of computing power caused by repeated pre-filling.

[0344] It should be noted that the device for accelerating inference through the computing power tiered token caching resource pool in the intelligent computing cloud platform of this invention can be a device, or a component, integrated circuit, or chip in an electronic device.

[0345] The present invention also provides an electronic device, see [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 301, a processor 302, and a program or instructions stored in the memory 301 that run on the memory. When the program or instructions are executed by the processor 302, they can achieve the following: Figure 1 The corresponding intelligent computing cloud platform achieves the same beneficial effect through any step in the method embodiment of accelerating inference by using a tiered token caching resource pool, and will not be elaborated here.

[0346] The processor 302 can be a CPU, ASIC, FPGA or GPU.

[0347] Those skilled in the art will understand that all or part of the steps of the above-described method embodiment for accelerating inference through a tiered token caching resource pool in intelligent computing cloud platform can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0348] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1The corresponding intelligent computing cloud platform accelerates inference through a tiered token caching resource pool, and any step in the method embodiment achieves the same technical effect. To avoid repetition, it will not be elaborated further here. The storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0349] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The corresponding intelligent computing cloud platform achieves the same technical effect by using a tiered computing power token caching resource pool to accelerate inference through various processes in its implementation method. To avoid repetition, these processes will not be elaborated here.

[0350] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this invention indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0351] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0352] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of the present invention.

[0353] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for accelerating inference in an intelligent computing cloud platform through a tiered token caching resource pool, characterized in that: include: Step S1: Divide the Token-KV Cache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks, and obtain the metadata fingerprint of the KV cache blocks; Step S2: Based on the KV cache block, determine the cache locality comprehensive score of the inference request at the candidate node, and based on the cache locality comprehensive score, determine the target node of the inference request. Specifically, when determining the net cache hit benefit of the candidate node for determining the cache locality comprehensive score, the estimated recalculation cost of the candidate KV cache block and the transmission cost of transmitting the candidate KV cache block to the candidate node are calculated. If the transmission cost is higher than the recalculation cost, the corresponding candidate KV cache block is determined to have a negative contribution to the effective hit rate, and the candidate KV cache block is recalculated. Step S3: Based on the metadata fingerprint, determine the reusable historical KV cache blocks in the computing power tiered token cache resource pool of the target node and the uncached KV cache blocks among the multiple KV cache blocks, and store the uncached KV cache blocks in the computing power tiered token cache resource pool in a tiered manner. Step S4: The target node executes the inference request based on the multiple KV cache blocks cached in the computing power hierarchical Token cache resource pool, and performs full lifecycle management of the KV cache blocks during the execution process; Step S4 includes: Step S4.5: During the decoding phase, obtain the prefetch related parameters of the inference request, wherein the prefetch related parameters include the current decoding speed, the loading time of the KV cache block, the safety jitter margin, and the size of the KV cache block, and the loading time is the time required to prefetch the KV cache block from the current cache level to the target level; Step S4.6: Calculate the pre-fetched wavefront distance based on the pre-fetched relevant parameters; Step S4.7: Determine the interval between the expected consumption position and the current consumption position of the KV cache block; Step S4.8: When the interval does not exceed the prefetch wavefront distance, the KV cache block is prefetched; when the KV cache block is prefetched, multiple arrival times are calculated based on the bandwidth and latency of each level to form a pipeline schedule, and the KV cache block that is about to be accessed by attention calculation is moved from the lower-level cache to the higher-level cache or a location that can be directly accessed by the GPU in advance.

2. The method as described in claim 1, characterized in that, Step S1 includes: Step S1.1: Divide the input word sequence of the reasoning request into multiple logical word intervals; Step S1.2: Obtain the KV cache block for each logical term interval under each Transformer layer of the large model.

3. The method as described in claim 2, characterized in that, Step S1.2 includes: Step S1.2.1: Obtain the first large model. l The input hidden state, key projection parameters, and value projection parameters of each Transformer layer; Step S1.2.2: According to the large model... l The input hidden states, key projection parameters, and value projection parameters of the Transformer layer are used to calculate the large model's nth layer. l The complete key tensor and complete value tensor of the Transformer layer; Step S1.2.3: Based on the logical lexical interval, analyze the large model... l Tensor slicing is performed on the complete key tensor and complete value tensor of each Transformer layer to obtain the corresponding KV cache block.

4. The method as described in claim 1, characterized in that, Step S1 further includes: Step S1.3: Determine the prefix hash chain of the i-th KV cache block, wherein the order of different KV cache blocks is determined based on the logical order of the KV cache blocks in the inference request; Step S1.4: Obtain auxiliary parameters related to the fingerprint, wherein the auxiliary parameters include at least one of precision format, tenant range, shared domain, security label, key version, signature policy, and integrity verification policy; Step S1.5: Construct the metadata fingerprint corresponding to the i-th KV cache block based on the prefix hash chain of the i-th KV cache block and the auxiliary parameters.

5. The method as described in claim 4, characterized in that, Step S1.3 includes: Step S1.3.1: Obtain environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block, wherein the environment configuration parameters include at least one of the following: model identifier, model version, tokenizer identifier, vocabulary version, LoRA version, Adapter version, plugin parameter version, cache format version, and position encoding scheme and its parameters; Step S1.3.2: Determine the prefix hash chain of the i-th KV cache block based on the environment configuration parameters, the content digest of the i-th KV cache block, and the prefix hash chain of the (i-1)-th KV cache block.

6. The method as described in claim 1, characterized in that, Step S1 further includes: Step S1.6: Determine the amount of pre-filling computation to be avoided when reusing the KV cache block, and convert the amount of pre-filling computation to be avoided into basic computing power revenue based on a preset computing power value conversion coefficient. Step S1.7: Determine the cost parameters generated by the KV cache block occupying each cache level of the computing power hierarchical Token cache resource pool, wherein the cost parameters include: cache occupation cost, cross-node or cross-level network transmission cost, and additional security management costs for encryption, verification, auditing and access control. Step S1.8: Deduct the cache occupation cost, the network transmission cost, and the security management additional cost from the basic computing power revenue to calculate the net computing power revenue of the KV cache block, wherein the net computing power revenue is used to determine the storage level of the corresponding KV cache block in the computing power tiered Token cache resource pool.

7. The method as described in claim 1, characterized in that, Step S1 further includes: Step S1.9: Determine the actual number of output tokens of the large model, and the equivalent number of tokens saved by avoiding repeated pre-computation by reusing the KV cache; Step S1.10: Convert the saved equivalent word units into effective output value according to the preset conversion weight, and add the effective output value to the actual output word units to obtain the comprehensive output benefit; Step S1.11: Calculate the total resource cost based on the computing resource cost, storage resource cost, and network resource cost during system operation; Step S1.12: The ratio of the comprehensive output revenue to the total resource cost is taken as the effective word element output efficiency.

8. The method as described in claim 1, characterized in that, Step S2 includes: Step S2.1: Obtain the scoring-related parameters of each candidate node, wherein the scoring-related parameters include net cache hit benefit, predicted future reuse probability, popularity of similar services or session families, memory pressure, decoding queue pressure, security risk, additional cost of cross-node transmission, and reward items that meet the first token delay, TPOT, or tenant priority. Step S2.2: Based on preset weights, perform a weighted summation on each of the scoring-related parameters to obtain the cache locality comprehensive score of the inference request at the corresponding candidate node.

9. The method as described in claim 8, characterized in that, Step S2.1 includes: Step S2.1.1: Determine the reusable hit status of the candidate KV cache block in the candidate node, wherein the value of the reusable hit status is positively correlated with the storage level of the candidate KV cache block; Step S2.1.2: Calculate the estimated recalculation cost of the candidate KV cache block and the transmission cost of transmitting the candidate KV cache block to the candidate node, and determine the difference between the estimated recalculation cost and the transmission cost. Step S2.1.3: When the candidate KV cache block meets the matching condition, multiply the difference by the reusable hit state, and accumulate the product of all the candidate KV cache blocks required by the request to obtain the net cache hit benefit of the candidate node.

10. The method as described in claim 9, characterized in that, When the key field of the candidate KV cache block meets the matching condition with the request attribute of the inference request, the candidate KV cache block is allowed to be reused; The matching conditions include at least one of the following conditions: The model identifier, tokenizer identifier, plug-in parameter version, and location encoding scheme of the candidate KV cache block are completely consistent with the inference request; The precision format or recoverable precision of the candidate KV cache block meets the precision requirements of the inference request. The tenant scope or shared domain to which the candidate KV cache block belongs is within the tenant or shared scope to which the inference request is authorized to access; The security label of the candidate KV cache block is not higher than the permission level of the inference request; The current time is earlier than the set expiration time of the candidate KV cache block.

11. The method as described in claim 8, characterized in that, The objective of configuring the preset weights is to minimize the overall cost of the system, which is derived from a statistical evaluation of historical routing decisions. The statistical indicators of the historical routing decision results include at least one of the following: first word delay, word generation delay during decoding, actual hit benefit, cache block loading time, recalculation time, GPU pause, service level target default status, and subsequent cache reuse count. The first word delay, the word generation delay, GPU overhead, network transmission overhead, memory overhead, and the service level target default status are multiplied by their respective preset conversion coefficients and then summed to obtain the comprehensive cost.

12. The method according to any one of claims 1-11, characterized in that, The tiered token caching resource pool includes at least two of the following tiers: L0, L1, L2, L3, L4, and L5; wherein, The L0 layer is a GPU HBM; The L1 layer is a cache that can be directly accessed by the DPU, smart network card, or GPU; The L2 layer is CPU DRAM; The L3 layer is either NVMe or a local SSD; The L4 layer is a remote SSD, a remote memory pool, or a distributed cache; The L5 layer is object storage, shared storage, or archive storage.

13. The method according to any one of claims 1-11, characterized in that, Step S2 further includes: Step S2.3: Obtain a set of candidate nodes that meet preset conditions, wherein the preset conditions include at least tenant isolation, model version, and security policy; Step S2.4: Determine the pre-filled queue pressure of each candidate node, and the additional switching cost required to switch the inference request from the current session sticky node to the candidate node; Step S2.5: Based on the comprehensive score of cache locality of the inference request at each of the candidate nodes, the penalty item related to the pre-filled queue pressure, and the penalty item related to the additional switching cost, calculate the final routing score of each of the candidate nodes. Step S2.6: Select the candidate node corresponding to the maximum value of the final route score as the target node, and route the inference request to the target node.

14. The method according to any one of claims 1-11, characterized in that, Step S4 includes: Step S4.1: During the process of the target node executing the inference request based on the KV cache blocks in the computing power tiered token cache resource pool, the value parameters of each KV cache block in the computing power tiered token cache resource pool are obtained in real time. The value parameters include at least the future reuse probability, recalculation cost, capacity cost of continued storage, expected movement cost, risk parameters, and strategy reward parameters. Step S4.2: Calculate the corresponding lifetime value based on the value parameters of the KV cache block; Step S4.3: Determine the initial level of the KV cache block in the computing power tiered token cache resource pool and the changing trend of its lifecycle value; Step S4.4: Perform full lifecycle management on the corresponding KV cache block based on the initial level and the changing trend of the lifecycle value.

15. The method as described in claim 1, characterized in that, Step S4.8 includes at least one of the following: Step S4.8.1: When the target service level of the inference request is greater than the preset level threshold, increase the security jitter margin or increase the prefetch priority; Step S4.8.2: When the high bandwidth memory pressure of the target level is greater than the preset pressure threshold, reduce the prefetch window.

16. The method as described in claim 1, characterized in that, Step S4 further includes: Step S4.9: Determine the set of KV cache blocks for the inference request, and divide the set of KV cache blocks into a first set that is hit and reusable, and a second set that is missed or not reusable; Step S4.10: Re-perform pre-filling calculation on the missing parts of the KV cache block set, and concatenate the hit KV cache blocks in the first set with the KV cache blocks obtained by re-pre-filling calculation according to their positions to construct a complete KV context; Step S4.11: Determine the prefix consistency check results of all hit KV cache blocks in the first set; Step S4.12: When the prefix consistency check result is passed, determine the recalculation cost of the hit KV cache block and the transmission cost of obtaining the hit KV cache block across levels; Step S4.13: If the transmission cost is higher than the recalculation cost, then the contribution of the corresponding hit KV cache block to the effective hit rate is determined to be negative, and the hit KV cache block is recalculated.

17. The method as described in claim 1, characterized in that, Step S4 further includes: Step S4.14: Identify the hit and reusable prefix data, and directly reuse the corresponding KV cache block; Step S4.15: Monitor the access progress during the decoding phase and downgrade KV cache blocks that are not in the current decoding wavefront to a lower-level cache medium, wherein the lower-level cache medium includes dynamic random access memory or non-volatile memory. Step S4.16: Promote the KV cache block that will be accessed in a short period of time to the higher-level cache of the computing power hierarchical Token cache resource pool; Step S4.17: Perform compression or eviction operations on the KV cache blocks whose lifetime value is lower than a preset value threshold; Step S4.18: Perform cache-aware routing on the inference requests so that inference requests with the same prefix share the higher-level cache.

18. The method as described in claim 1, characterized in that, The KV cache block is bound to security metadata and security checks are performed during lookup, routing, migration, reuse, degradation, archiving and eviction processes; The security mechanisms used for the security checks include at least: tenant-scope binding, security label binding, key isolation, integrity verification, failure policy, and audit trail.

19. The method as described in claim 1, characterized in that, The metadata fingerprint includes a version vector, which is determined based on at least one of the following: model weights, tokenizer, position encoding, adapter, quantization parameters, and attention kernel.

20. The method as described in claim 1, characterized in that, Step S3 includes: Step S3.1: Based on the metadata fingerprint, determine the matching result between the KV cache block of the inference request and the KV cache block in the computing power tiered token cache resource pool; Step S3.2: Based on the matching result, determine the KV cache blocks that are not stored in the target node's computing power hierarchical Token cache resource pool; Step S3.3: Determine the net computing power benefit of the unstored KV cache block; Step S3.4: Based on the net computing power revenue, store the unstored KV cache blocks in the computing power tiered Token cache resource pool in a tiered manner.

21. A device for accelerating inference in an intelligent computing cloud platform through a tiered token caching resource pool, characterized in that, include: The processing module is used to divide the Token-KV Cache generated by the intelligent computing cloud platform during large model parsing and inference requests into multiple KV cache blocks, and obtain the metadata fingerprint of the KV cache blocks; The first determining module is used to determine the cache locality comprehensive score of the inference request on the candidate node based on the KV cache block, and to determine the target node of the inference request based on the cache locality comprehensive score. Specifically, when determining the net cache hit benefit of the candidate node for determining the cache locality comprehensive score, the estimated recalculation cost of the candidate KV cache block and the transmission cost of transmitting the candidate KV cache block to the candidate node are calculated. If the transmission cost is higher than the recalculation cost, the corresponding candidate KV cache block is determined to have a negative contribution to the effective hit rate, and the candidate KV cache block is recalculated. The second determining module is used to determine, based on the metadata fingerprint, reusable historical KV cache blocks in the computing power hierarchical token cache resource pool of the target node and uncached KV cache blocks among the multiple KV cache blocks, and store the uncached KV cache blocks hierarchically in the computing power hierarchical token cache resource pool. The management module is used to execute the inference request on the target node based on the multiple KV cache blocks cached in the computing power hierarchical token cache resource pool, and to perform full lifecycle management of the KV cache blocks during the execution process. Specifically, the management module is used to: obtain prefetch-related parameters of the inference request during the decoding phase, wherein the prefetch-related parameters include the current decoding speed, the loading time of the KV cache block, the security jitter margin, and the size of the KV cache block, and the loading time is the time required to prefetch the KV cache block from the current cache level to the target level; calculate the prefetch wavefront distance based on the prefetch-related parameters; determine the interval between the expected consumption position of the KV cache block and the current consumption position; prefetch the KV cache block when the interval does not exceed the prefetch wavefront distance; and calculate multiple arrival times based on the bandwidth and latency of each level during the prefetching of the KV cache block to form a pipeline scheduling, and pre-move the KV cache blocks that are about to be accessed by attention computing from the lower-level cache to the higher-level cache or the position that can be directly accessed by the GPU.

22. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform as described in any one of claims 1 to 20.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for accelerating inference through a tiered token caching resource pool in the intelligent computing cloud platform as described in any one of claims 1 to 20.

24. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, implement the steps of the method for accelerating inference through a tiered token caching resource pool in an intelligent computing cloud platform as described in any one of claims 1 to 20.

Images